Large file upload problem

I recently took over a file hosting site (similar to rapidshare, megaupload, etc.) and am currently averaging around 75k visitors per day. After the migration, we wrote our own system from scratch. Now it's time to upgrade the infrastructure and work with web servers and scale. Now the problem is that after the file download is complete, the web server does not "flush" the used memory. We uploaded several 300MB + files and before uploading, we had about 3.7GB of free memory and after downloading 3 files of 300M each, the memory usage is about 1.2GB. We originally thought it might be an apache issue, but upgrading to Litespeed didn't fix it either. Here's "memory usage" from "top"

Mem: 4015996k total, 1206036k used, 2809960k free, 114984k buffers

I am scratching my head to figure out how to "clean up" the memory used by the web server or php at boot time because that way the system will run out of memory and crash in no time.

Here are my php.ini values:

max_execution_time = 6000 max_input_time = 6000 memory_limit = 2048M post_max_size = 2047M upload_max_filesize = 2047M

In Apache, php runs on fcgi and on Litespeed, this is Litespeed SAPI.

Thanks Bill