Python crash when calculating SHA-1 hashes for large files on Windows

Question

I am wondering if I can have fresh eyes on this python script. It works great with small and medium sized files, but with large files (4-8 GB or so) these are unexplained crashes after launching for several minutes.

Zipped script here

Or:

import sys import msvcrt import hashlib #Print the file name (and its location) to be hashed print 'File: ' + str(sys.argv[1]) #Set "SHA1Hash" equal to SHA-1 hash SHA1Hash = hashlib.sha1() #Open file specified by "sys.argv[1]" in read only (r) and binary (b) mode File = open(sys.argv[1], 'rb') #Get the SHA-1 hash for the contents of the specified file SHA1Hash.update(File.read()) #Close the file File.close() #Set "SHA1HashBase16" equal to the hexadecimal of "SHA1Hash" SHA1HashBase16 = SHA1Hash.hexdigest() #Print the SHA-1 (hexadecimal) hash of the file print 'SHA-1: ' + SHA1HashBase16 #Make a blank line print ' ' #Print "Press any key to continue..." print 'Press any key to continue...' #"Press any key to continue..." delay char=0 while not char: char=msvcrt.getch() 

* Updated *

A working python script to calculate the SHA-1 hash of large files. Thanks to Ignacio Vasquez-Abrams for what's wrong, and Tom Zich for the code.

Source as zip here

To just drag and drop a file, you need to hash on top of the script. Alternatively, you can use the command line using:

 SHA-1HashGen.py Path&File 

If SHA-1HashGen.py is the name of the script file, and Path & File is the path and file name for hashing.

Or release the script into the SendTo folder (on Windows, shell: sendto) to get it as the right mouse button.

Answer 1

Stop reading a file at a time; you consume all the memory in the system. Read in 16 MB or so pieces.

 data = File.read(16 * 1024 * 1024) 

Answer 2

(In response to Peter’s comment, 2 GB is left.)

I suspect Ignacio is right. Try replacing the read / update line as follows:

 while True: buf = File.read(0x100000) if not buf: break SHA1Hash.update(buf)