Efficiently delete a million files on Linux servers

Last updated on 06 Sep 2023, 13:14:56.
Category: Cloud servers

CentOS Debian Linux

It happens to the best: some script rockets to the skyline resulting in an instand system administrator headache because some folder - typically, sessions - was stuffed with millions of files. Linux is not quite happy with that, deleting the folder is not an option and the loyal "rm -rf" command decides to call it a day. To make things even worse: you want to remove only files of some days ago... what are the options?

Find is you friend

The Linux "find" command is a possible solution, many will go for:

find /yourmagicmap/* -type f -mtime +3 -exec rm -f {} \;

The command above will give a list of files older than 3 days and will pass each found file to the rm command. The rule above has one problem though: it will take some time, since calling the rm command a million times is not exactly what you can call efficient.

A better option is:

 find /yourmagicmap/* -type f -mtime +3 -delete

This adds the delete flag to the find command giving it the command to throw it all away. Do the right thing and pass along your command in a cronjob if you need to clean out the folder on a regular basis.

The rsync alternative!

rsync is without doubt one of the most handy commands when it comes to file operations. Rsync can do any kind of volume sync - as you may know - but it also provides a way to empty a folder.The example below assumes you have a folder named /tmp/empty/ which contains no files, and a folder /tmp/session/ that contains too much crap. The rule below allows you to remove those files:
rsync -a --delete /tmp/empty /tmp/session/

Which is the fastest?

rm: deleting millions of file is a no-can-do!

find -exec: an option, but slower!

find -delete: fast and easy way to remove loads of files.

rsync --delete: without doubt the quickest!

Related articles

Managed services in the spotlight

Our system administrators have lifted a couple of cargo ships filled with containers of work. (yes, that pun was on...

Read more

What service level guarantees (SLA) are available for a server housing (co-location)?

Server housing or co-location customers can fall back on an excellent service level guarantee. A Kinamo server housing service comes...

Read more

Discover here how we’ve fixed a blocking error when upgrading ESXi 6.5 to 7.0 update 2 through vSphere Lifecycle Manager.

Kinamo maintains multiple VMware vSphere clusters, not only for our own infrastructure, but also for a variety of customers. As...

Read more

Need extra help?

Were not all your questions answered?
Don't worry, we will be happy to help you via a support request!


Select your language

All languages: