Geek Answers Handbook

How to upload files using Python?

HI, everyone. I am new to Python and use Python 2.5 on CentOS.

I need to download files like WGET do.

I did some searching, and there are some solutions, the obvious way:

 import urllib2 mp3file = urllib2.urlopen("http://www.example.com/songs/mp3.mp3") output = open('test.mp3','wb') output.write(mp3file.read()) output.close()

It works great. But I want to know if the mp3 file is VERY large, for example, 1Gb, 2Gb or even more. Can this piece of code work? Are there any better ways to load large files in Python, perhaps with a progress bar like WGET do.

Thanks a lot!

+4
source share
4 answers

It’s easier there:

 import urllib urllib.urlretrieve("http://www.example.com/songs/mp3.mp3", "/home/download/mp3.mp3")
+15
source

For really large files, your code will use a lot of memory, since you immediately load the entire file into memory. It might be better to read and write data in chunks:

 from __future__ import with_statement import urllib2 mp3file = urllib2.urlopen("http://www.example.com/songs/mp3.mp3") with open('test.mp3','wb') as output: while True: buf = mp3file.read(65536) if not buf: break output.write(buf)
+3
source

Why not just call wget , then?

 import os os.system ("wget http://www.example.com/songs/mp3.mp3")
+2
source

your current code will read the entire stream in memory before writing to disk. Therefore, for cases when the file is larger than the available memory, you are faced with problems.

to resolve this, you can read the fragments at a time and write them to a file.

(copied from blob stream from urllib2 to file )

 req = urllib2.urlopen(url) CHUNK = 16 * 1024 with open(file, 'wb') as fp: while True: chunk = req.read(CHUNK) if not chunk: break fp.write(chunk)

"experiment a bit with the various sizes of CHUNK to find a" sweet spot "for your requirements."

+1
source

More articles:


All Articles