Have Geek Brain

Limiting the number of files in one directory on unix / linux using python

is it bad to output many files to the same directory on unix / linux? I run thousands of jobs in the cluster and each one dumps a file into one directory. The upper bound here is about 50,000 files. Can IO limit speed in light of this? If so, does the problem go away with the nested directory structure?

Thanks.

+2


a source to share


3 answers


Cm:



How many files can you put in a directory?

+2


a source


I believe that most filesystems store the names of the contained files in a list (or some other linear data structure), so keeping a lot of files in one directory can cause slowness for simple operations like enumeration. Having a nested structure can alleviate this problem by creating a tree structure (or even Trie , if that makes sense) of names, which can shorten the time it takes to get file statistics.



0


a source


My suggestion is to use a nested directory structure (i.e. categorization). You can name them with timestamps, specific prefixes for each application, etc. It gives you a sense of order when you need to search for specific files and make managing your files easier.

0


a source






More articles:


All Articles