29 thoughts on “How to remove backups?”

1. This is not an answer to your question, but may be interesting: At one stage I had MythTV set up to record a large number of TV shows, and delete old ones automatically. The files were very large — more than 1GB — and I was using ext3, and it fairly quickly became apparent that ext3 sucks at deleting big files — it would severely degrade other IO for big chunks of time, wreaking havoc. The advice on MythTV forums etc was to switch to JFS, which has good delete performance. It worked wonderfully — very fast, smooth deletes (no idea if JFS any good for PostgreSQL in general, just though I’d share this anecdote!)

2. I have also experienced this when deleting MythTV recordings on a test machine running Ext3. My production MythTV backend at the time ran Reiser3 which did not have this issue. Today I am using NFS to my Mac Pro running HFSX which is also immune.

   MythTV added sometime ago an option to slowly delete recordings using many truncations. As you have discovered, this works.

   Anybody using this method in production may wish to rename the file before commencing to prevent confusion should the process be interrupted.

3. ext3 is known to be terrible for this.

   The following benchmarks are 2 years old, but already showed ext4 to be much lower-impact at deletes than ext3:

   http://www.bullopensource.org/ext4/20070404/FileDeletion.html

   Jon

4. @Macdice:
   thanks, I heard also a lot of good stuff about XFS, but unfortunately FS change on large databases server is not a thing that you can easily do.

5. @Jason:
   Thanks about information. It seems MythTV guys worked it out quite well.

6. @Jon:
   Thanks for link.

   Time for me is not really important – file can even take multiple hours to remove, but impact on concurrent operations should be minimal.

   From the benchmark it looks like XFS is clear winner. ext3 with extents takes shorter time than ext3, but still writes *very* comparable amount of data. Reads way less, though.

7. probowales albo usunac katalog z plikiem (przenies go tam najpierw), albo zmienic ustawienia ext3, albo kolejne rozwiazanie, to osobny naped na te pliki.

8. @gregj:
   usuniecie katalogu z plikiem nic nie da – przeciez pokazalem, ze istotne jest to, ze jest zwalniane miejsce – wiec co za roznica czy usuwasz plik z katalogiem czy bez.

   co do ustawien ext3 – nie. znasz jakies ustawienia ktore moglyby pomoc?

   co do osobnego napedu – dodawanie nowych dyskow do pracujacego serwera jest dosyc skomplikowane.

9. any filesystem that uses extents will be better at this. ext4, and xfs I know both use extents (I don’t know about jfs, or reiserfs).

   the reason being is that in ext3 every block used gets recorded block1 is for this file, block2, block3 or whatever… at 4096 blocks for a 100G file this takes quite a while to remove all those entries. When you are using extents this entry is block1..block100G or whatever. It uses a range instead of listing each block sequentially.

   What IO scheduler are you using? you might get better performance out of deadline. I don’t think ionice works with deadline at all though.. nicing is for cfq only (I think, was when it first came out).

   I don’t know that it would help… but perhaps ‘really nicing’ your program would help

   http://xenoterracide.blogspot.com/2008/04/really-nice.html

   this is of course just combining nice and ionice giving it less processor and IO time.

10. I’m at least as in the dark as you are but I don’t *think* ext3 would do defrag like that.

    My guess is that the kernel is trying to do the delete in one step and as a result is locking the file system for that whole interval. OTOH this could in theory be solvable by making the system call code release it’s locks when it’s preempted by a context switch. Given that no one has implemented that, I might be wrong about the root cause (or that fix might be to hard to implement).

11. Being the person who forced the author to use ext3, to satisify support requirements for RHEL5 + qlogic + brocade + NetApp, as NetApp, or most other companies, will not support just your ordinary (no offense to other unsupported distros) linux distro, and this becomes important when you spend $x on your storage solution, I thought I might offer the fruits of the authors suggestions to purging multi-hundred gig files from ext3 file systems, at least from my point of view. The author suggested that we truncate the logs rather than just purge them in one shot. This seemed like a great idea except that only the truncate C libs were included with Centos5 and RHEL5, and not the “truncate” command included with more recent core-utils rpm packages. Other options such as perl/python/name you home made hack existed, but a bet that the truncate command from a newer fedora(12) was based on the same libs for truncate as the current Cent5/RHEL5, a simple copy of the truncate binary from Fedora 12 to Cent 5 worked perfectly. (not yet tested on RHEL5) But having that command made this script possible. And may buy some time until we can migrate to ext4, or do away with OEM support of any kind. (unlikely)

    Again, thanks to the author for his wide and deep knowledge of systems, and excellent understanding of databases, operating systems and storage. And so I give to you, “truncatenator”, a replacement for rm in large file removal crons:

    #!/bin/bash
    # yes I watch Phineas AND Ferb ON disney, dont ask.
    trunc="/truncatenator/truncate"
    file=$1
    SIZE=`ls -l $file |awk '{print $5}'`
    while [ $size -gt 1000000000 ]; do
            SIZE=`echo $size-100000000|bc`;
            $trunc -c -s -100000000 $file ;
            sleep 0.25;
    done
    rm -f $file                
    #end

    This has proved itself on Centos5 with a reasonably fast disk subsystem to be able to remove large files without blocking other processes from disk access. You can adjust to any needed values to allow for your system to not block on file system access. Again, thanks to the author, who has time and again proven his chops to this old dog.
    Cheers
    R

12. I would love to see these same tests run on other filesystems, same criteria.

13. @Allspaw:
    well, I would love to run those tests, but I don’t have hardware to put it on, and running it on virtual box doesn’t make much sense.

14. we could probably convince phoronix to do them…. they already love doing all kinds of benchmarks…

15. Out of curiosity, what kind of hardware is PostgreSQL running on?

16. @Aszeszo:
    beefy. really beefy. sorry – can’t tell anything more.

17. Slow deletion on ext3 is simply because it touches every inode of the file, which turns into a bunch of random writes all over the disk for a giant file. There’s no defragmentation or similar work going on. A volume that’s also holding pg_xlog is a really bad combination because that’s going to force all the inode writes to disk each time a fsync comes in from a WAL write, rather than grouping them up into more efficient, sorted chunks before writing as would normally be the case.

18. One thought wanders through my mind:

    why truncate/delete the file? If the backups are simmilar in size you could try to overwrite previous backup file contents with a command simmilar to this one:

    pg_dump … | dd of=backup_file conv=notrunc

19. @CRG:
    1. it’s pretty obscure, and not a lot of people know you can do it
    2. in case next backup is smaller – it leads to errors on restore. these errors can be ignored, but the sole fact that uncompressing is not clean – makes me worried.

20. @depesz:

    ad. 1)

    Yeah, it’s not so obvious.

    ad. 2)

    why would that lead to errors? Let’s assume that we use custom backup file format. As far as I understand custom is a binary format that uses file offsets and it’s not a stream like a regular SQL dump.

    If you’re in doubt you could try it on some test db…

21. @CRG:
    it’s tar.gz of $PGDATA, not pg_dump.

    use this technique to overwrite tar, and you’ll see errors at the end of uncompression.

22. The article says “As You can see, I used -c2 -n7 options to ionice, which seem sane.” which is true but if you want no disruption then -c3 ‘idle’ would be a better choice than -c2 ‘best-effort’. I use -c3 to build in the background and have found it to work well without causing the build to wait for ever. If you really do have 100% io usage then -c3 will not let the delete ever complete but I don’t expect that is what you have based on the worked test.

    On Serverfault the “use ionice” answer is not very helpful if none of the options are explained, for starters your default process will already have some ionice (equivalent) settings granted by the OS.

23. No doubt, I am going to seem stupid (sigh, not the first time) but I would just try normal practice for low priority I/O interfering with high priority I/O. Isolate the backup files on their own physical disk(s), preferable SLOW ones, with their own mount point(s). Unless Linux is a lot more stupid than I think, this ought to prevent the rm from eating too much in the way of resources, and I cannot imagine locking of anything important being an issue. If you have REALLY beefy hardware, as you say, the CPU and memory usage for this delete should be negligible. The only thing I would double check is that Linux does not decide to increase disk cache substantially during the rm. I doubt it will.

24. @Tim:
    the case is long gone, but backups were on separate set of disks, afair.

25. If the backup dir is on a different disk or lv, why not just write a script to drop it, put it back and then format it?

26. @Jules:
    because you might have for example 4 backups on it, and you want to drop only one? Though it is interesting idea to have separate lv/partition for each backup. not such how well it would scale, though.

27. Yes, of course.
    I have a system that is a disk staging area for an RMAN job. A weekly full followed by daily incrementals. All sent off to tape on a daily basis. Once a week we needed to delete all the files and start again. 4TB took 6 hours. Much faster for us drop the disk and then put it back.

28. @Depesz, sorry for gravedigging here a bit but you said, depend son how it scales… according to your parameters back then, it required either 3 (or 4 for redundancy?) LV partitions .. hardly a huge scaling issue. And since you have a fixed size more or less, you don’t need ot worry about LV resizing either. So like osmeone said… instead of deleting, simply release the whole set of pointers, by a quick format or so 🙂

29. Thanks for research – your method help me with task!

Comments are closed.