Python method for storing a list of bytes in a network (big-endian) byte order for a file (little-endian)

Question

My current task is to parse the tcpdump data containing the P2P messages, and I am having problems with the chunk data that I receive and write to a file on my x86 machine. My suspicion is that I have a simple problem with the content, with the bytes that I write to the file.

I have a list of bytes containing a P2P video fragment that is read and processed using the python-pcapy BTW package.

bytes = [14, 254, 23, 35, 34, 67, etc... ]

I am looking for a way to save these bytes, which are currently stored in a list in a Python application, to a file.

I am currently writing fragments as follows:

 def writePiece(self, filename, pieceindex, bytes, ipsrc, ipdst, ts): file = open(filename,"ab") # Iterate through bytes writing them to a file if don't have piece already if not self.piecemap[ipdst].has_key(pieceindex): for byte in bytes: file.write('%c' % byte) file.flush() self.procLog.info("Wrote (%d) bytes of piece (%d) to %s" % (len(bytes), pieceindex, filename)) # Remember we have this piece now in case duplicates arrive self.piecemap[ipdst][pieceindex] = True # TODO: Collect stats file.close() 

As you can see from the for loop, I write the bytes to the file in the same order as I process them from the explorer (i.e. network or large order).

Suffice it to say that the video, which is the payload of the parts, does not play well in VLC: -D

It seems to me that I need to convert them to low order byte order, but I'm not sure what the best way to approach this in Python.

UPDATE

The solution that was developed for me (by writing P2P parts that handle the final problems accordingly):

 def writePiece(self, filename, pieceindex, bytes, ipsrc, ipdst, ts): file = open(filename,"r+b") if not self.piecemap[ipdst].has_key(pieceindex): little = struct.pack('<'+'B'*len(bytes), *bytes) # Seek to offset based on piece index file.seek(pieceindex * self.piecesize) file.write(little) file.flush() self.procLog.info("Wrote (%d) bytes of piece (%d) to %s" % (len(bytes), pieceindex, filename)) # Remember we have this piece now in case duplicates arrive self.piecemap[ipdst][pieceindex] = True file.close() 

The key to the solution was to use the Python structural module as a suspect and, in particular:

  little = struct.pack('<'+'B'*len(bytes), *bytes) 

Thanks to those who responded with helpful suggestions.

Answer 1

You can also use array.array :

 from array import array f.write(array('B', bytes)) 

instead

 f.write(struct.pack('<'+'B'*len(bytes), *bytes)) 

which at a small size

 f.write(struct.pack('B' * len(bytes), *bytes)) # the < is redundant; there is NO ENDIANNESS ISSUE 

if len (bytes) is "big", maybe better than

 f.write(struct.pack('%dB' % len(bytes), *bytes)) 

Answer 2

To save you some work, you can use bytearray (Python 2.6 and later):

 b = [14, 254, 23, 35] f = open("file", 'ab') f.write(bytearray(b)) 

This does all the conversion of your 0-255 values ​​to bytes without the need for all loops.

I do not see your problem otherwise without additional information. If the data is really byte-wise, then assertion is not a problem, as others have said.

(By the way, using bytes and file as variable names is not good, as it hides built-in modules with the same name).

Answer 3

This may have been answered earlier in Python File Slurp w / endian conversion .