What's the fastest way to get directory and subdirectory size in unix using Perl?
I am using Perl stat () to get the size of a directory and its subdirectories. I have a list of about 20 parent directories that have several thousand recursive subdirectories, and each subdirectory has several hundred entries. The main computational part of the script looks like this:
sub getDirSize {
my $dirSize = 0;
my @dirContent = <*>;
my $sizeOfFilesInDir = 0;
foreach my $dirContent (@dirContent) {
if (-f $dirContent) {
my $size = (stat($dirContent))[7];
$dirSize += $size;
} elsif (-d $dirContent) {
$dirSize += getDirSize($dirContent);
}
}
return $dirSize;
}
The script takes more than one hour to execute and I want to make it faster.
I tried using shell command du
but the output du
(passed in bytes) is inaccurate. And it also takes a long time. I am working on HP-UNIX 11i v1.
a source to share
I faced a similar problem and used the parallelization method to speed it up. Since you have ~ 20 top level directories, this might be a pretty simple approach for you to try. Divide your top-level directories into multiple groups (how many groups are best is a rule of thumb), invoke fork()
and parse directories in child processes multiple times . At the end of the child processes, write your results into some temporary files. When all the children are done, read the results from the files and process them.
a source to share
With some help from sfink and samtregar on perlmonks try this:
#!/usr/bin/perl
use warnings;
use strict;
use File::Find;
my $size = 0;
find( sub { $size += -f $_ ? -s _ : 0 }, shift(@ARGV) );
print $size, "\n";
Here we recurse all subdirectories of the specified directory, getting the size of each file, and we reuse stat from the file test using the special "_" syntax for the size test.
I am inclined to believe that du will be reliable enough.
a source to share
I see a couple of problems. One @dirContent is explicitly set to <*> this will reset every time you enter getDirSize. The result will be an infinite loop, at least until you run out of the stack (since this is a recursive call). Second, there is a special file descriptor entry for extracting information from a stat call, the underscore (_). See: http://perldoc.perl.org/functions/stat.html . Your as-is code calls stat three times with essentially the same information (-f, stat, and -d). Since file I / O is expensive, you really want to call stat once and then refer to the data with "_". Here is some sample code that I believe accomplishes what you are trying to do
#!/usr/bin/perl
my $size = 0;
getDirSize(".",\$size);
print "Size: $size\n";
sub getDirSize {
my $dir = shift;
my $size = shift;
opendir(D,"$dir");
foreach my $dirContent (grep(!/^\.\.?/,readdir(D))) {
stat("$dir/$dirContent");
if (-f _) {
$$size += -s _;
} elsif (-d _) {
getDirSize("$dir/$dirContent",$size);
}
}
closedir(D);
}
a source to share
Whenever you want to speed up something, you must first figure out what is slow. Use a profiler such as Devel :: NYTProf to analyze your program and find out where you should focus your efforts.
In addition to reusing that data from the last stat, I would get rid of the recursion as Perl is terrible at this.I would build a stack (or queue) and work on it until there was nothing left to process.
a source to share
If your home directory is overwhelmingly the largest consumer of directories and inodes, then don't compute it. Calculate the other half of the system and deduce the size of the rest of the system from it. (you can use disk space from df
within a few ms). You may need to add a small "fiction" factor to get the same numbers. (also remember that if you count some free space like root
you will have more than other users 5% in ext2 / ext3 on Linux, don't know about HPUX).
a source to share
Below is another variant of getDirSize (), which does not require a reference to a variable holding the current size and takes a parameter indicating whether subdirectories should be considered or not:
#!/usr/bin/perl
print 'Size (without sub-directories): ' . getDirSize(".") . " bytes\n";
print 'Size (incl. sub-directories): ' . getDirSize(".", 1) . " bytes\n";
sub getDirSize
# Returns the size in bytes of the files in a given directory and eventually its sub-directories
# Parameters:
# $dirPath (string): the path to the directory to examine
# $subDirs (optional boolean): FALSE (or missing) = consider only the files in $dirPath, TRUE = include also sub-directories
# Returns:
# $size (int): the size of the directory contents
{
my ($dirPath, $subDirs) = @_; # Get the parameters
my $size = 0;
opendir(my $DH, $dirPath);
foreach my $dirEntry (readdir($DH))
{
stat("${dirPath}/${dirEntry}"); # Stat once and then refer to "_"
if (-f _)
{
# This is a file
$size += -s _;
}
elsif (-d _)
{
# This is a sub-directory: add the size of its contents
$size += getDirSize("${dirPath}/${dirEntry}", 1) if ($subDirs && ($dirEntry ne '.') && ($dirEntry ne '..'));
}
}
closedir($DH);
return $size;
}
a source to share