Most efficient way to save binary from internet using Python 2.6?

I am trying to download (and save) a binary from the internet using Python 2.6 and urllib.

As I understand it, read (), readline () and readlines () are 3 ways to read a file-like object. Since binaries are not actually broken on new lines, read () and readlines () read the entire file into memory.

Is choosing the random read size () the most efficient way to limit memory usage during this process?

i.e.

import urllib
import os

title = 'MyFile'
downloadurl = 'http://somedomain.com/myfile.avi'
webFile = urllib.urlopen(downloadurl)
mydirpath = os.path.join('c:', os.sep,'mydirectory',\
                         downloadurl.split('/')[-1])

if not os.path.exists(mydirpath):
    print "Downloading...%s" % title
    localFile = open(mydirpath, 'wb')
    data = webFile.read(1000000) #1MB at a time
    while data:
        localFile.write(data)
        data = webFile.read(1000000) #1MB at a time
    webFile.close()
    localFile.close()
    print "Finished downloading: %s" % title
else:
    print "%s already exists." % mydirypath

      

I chose read (1,000,000) arbitrarily because it worked and kept using RAM. My guess is that if I were working with a raw network buffer, choosing a random sum would be bad, as the buffer could run dry if the baud rate was too low. But it looks like urllib is already handling lower level buffering for me.

With this in mind, chooses an arbitrary penalty number? Is there a better way?

Thanks.

+1


a source to share


2 answers


For this you have to use urllib.urlretrieve

. It will handle everything for you.



+2


a source


Instead of using your own read-write loop, you should probably check the module shutil

. The method copyfileobj

will allow you to define buffering. The most effective method varies from situation to situation. Even copying the same source file to the same destination can be affected by network issues.



+1


a source







All Articles