How to read a file with NUL separators?

Question

I usually use the following Python code to read lines from a file:

f = open('./my.csv', 'r') for line in f: print line 

But what about if the file is a string divided by "\ 0" (not "\ n")? Is there a Python module that could handle this?

Thanks for any advice.

Answer 1

If your file is small enough for you to read all of this in memory, you can use split:

 for line in f.read().split('\0'): print line 

Otherwise, you can try this recipe from a discussion of this function request :

 def fileLineIter(inputFile, inputNewline="\n", outputNewline=None, readSize=8192): """Like the normal file iter but you can set what string indicates newline. The newline string can be arbitrarily long; it need not be restricted to a single character. You can also set the read size and control whether or not the newline string is left on the end of the iterated lines. Setting newline to '\0' is particularly good for use with an input file created with something like "os.popen('find -print0')". """ if outputNewline is None: outputNewline = inputNewline partialLine = '' while True: charsJustRead = inputFile.read(readSize) if not charsJustRead: break partialLine += charsJustRead lines = partialLine.split(inputNewline) partialLine = lines.pop() for line in lines: yield line + outputNewline if partialLine: yield partialLine 

---

I also noticed that your file has the extension "csv". Python has a built-in CSV module (import csv). There is the Dialect.lineterminator attribute, however, it is not currently implemented in the reader:

Dialect.lineterminator

The string used to complete the lines created by the author. The default is '\ r \ n'.

Note. The reader is hard-coded to recognize "\ r" or "\ n" as the end of a line and ignores the liner-determinant. This may change in the future.

Answer 2

I changed the Mark Byers sentence so that we could read a READLINE file with NUL separator strings in Python. This approach reads a potentially large file line by line and should be more memory efficient. Here is the Python code (with comments):

 import sys # Variables for "fileReadLine()" inputFile = sys.stdin # The input file. Use "stdin" as an example for receiving data from pipe. lines = [] # Extracted complete lines (delimited with "inputNewline"). partialLine = '' # Extracted last non-complete partial line. inputNewline="\0" # Newline character(s) in input file. outputNewline="\n" # Newline character(s) in output lines. readSize=8192 # Size of read buffer. # End - Variables for "fileReadLine()" # This function reads NUL delimited lines sequentially and is memory efficient. def fileReadLine(): """Like the normal file readline but you can set what string indicates newline. The newline string can be arbitrarily long; it need not be restricted to a single character. You can also set the read size and control whether or not the newline string is left on the end of the read lines. Setting newline to '\0' is particularly good for use with an input file created with something like "os.popen('find -print0')". """ # Declare that we want to use these related global variables. global inputFile, partialLine, lines, inputNewline, outputNewline, readSize if lines: # If there is already extracted complete lines, pop 1st llne from lines and return that line + outputNewline. line = lines.pop(0) return line + outputNewline # If there is NO already extracted complete lines, try to read more from input file. while True: # Here "lines" must be an empty list. charsJustRead = inputFile.read(readSize) # The read buffer size, "readSize", could be changed as you like. if not charsJustRead: # Have reached EOF. if partialLine: # If partialLine is not empty here, treat it as a complete line and copy and return it. popedPartialLine = partialLine partialLine = "" # partialLine is now copied for return, reset it to an empty string to indicate that there is no more partialLine to return in later "fileReadLine" attempt. return popedPartialLine # This should be the last line of input file. else: # If reached EOF and partialLine is empty, then all the lines in input file must have been read. Return None to indicate this. return None partialLine += charsJustRead # If read buffer is not empty, add it to partialLine. lines = partialLine.split(inputNewline) # Split partialLine to get some complete lines. partialLine = lines.pop() # The last item of lines may not be a complete line, move it to partialLine. if not lines: # Empty "lines" means that we must NOT have finished read any complete line. So continue. continue else: # We must have finished read at least 1 complete llne. So pop 1st llne from lines and return that line + outputNewline (exit while loop). line = lines.pop(0) return line + outputNewline # As an example, read NUL delimited lines from "stdin" and print them out (using "\n" to delimit output lines). while True: line = fileReadLine() if line is None: break sys.stdout.write(line) # "write" does not include "\n". sys.stdout.flush() 

Hope this helps.