How to write a long integer as binary in Python?

Question

In Python, long integers have unlimited precision. I would like to write an integer of 16 bytes (128 bits) to a file. struct from the standard library only supports up to 8 byte integers. array has the same limitation. Is there a way to do this without disguising and shifting each integer?

Some explanation here: I am writing to a file that will be read from programs other than Python, so there is no pickle. All 128 bits are used.

Answer 1

Two possible solutions:

• Just pickle your long integer. This will write the integer in a special format that allows it to be read again if that is all you want.

• Use the second code snippet in this answer to convert a long int to a large endian line (which can be easily changed to a small endian if you prefer) and write this line to your file.

The problem is that the internal representation of bigints does not directly contain the binary data that you request.

Answer 2

I think for unsigned integers (and ignoring the entity) something like

 import binascii def binify(x): h = hex(x)[2:].rstrip('L') return binascii.unhexlify('0'*(32-len(h))+h) >>> for i in 0, 1, 2**128-1: ... print i, repr(binify(i)) ... 0 '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' 1 '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01' 340282366920938463463374607431768211455 '\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff' 

can technically satisfy the requirements of having non-Python-specific output without using an explicit mask, and (I assume) not using any non-standard modules. Not particularly elegant, though.

Answer 3

The PyPi bitmap module combined with the built-in bin() function seems like a good combination of a simple and flexible solution.

 bytes = bitarray(bin(my_long)[2:]).tobytes() 

The sequence number can be controlled with a few lines of code. You will have to evaluate the effectiveness.

Answer 4

Why not use a struct with an unsigned long long type twice?

 import struct some_file.write(struct.pack("QQ", var/(2**64), var%(2**64))) 

This is described here (scroll down to get the table with Q): http://docs.python.org/library/struct.html

Answer 5

This may not escape the requirement of “mask and shift of each whole”. I'm not sure what to avoid using mask and shift in the context of long Python values.

These bytes are:

 def bytes( long_int ): bytes = [] while long_int != 0: b = long_int%256 bytes.insert( 0, b ) long_int //= 256 return bytes 

Then you can pack this list of bytes with struct.pack( '16b', bytes )

Answer 6

With Python 3.2 and later, you can use int.to_bytes and int.from_bytes : https://docs.python.org/3/library/stdtypes.html#int.to_bytes

Answer 7

You can split the object into binary, use protocol buffers (I don't know if they allow you to serialize unlimited precision integers), or BSON if you don't want to write code.

But to write a function that unloads 16 byte integers, shifting it, you should not do so hard if it is not a critical time.

Answer 8

It may be a little late, but I do not understand why you cannot use struct:

 bigint = 0xFEDCBA9876543210FEDCBA9876543210L print bigint,hex(bigint).upper() cbi = struct.pack("!QQ",bigint&0xFFFFFFFFFFFFFFFF,(bigint>>64)&0xFFFFFFFFFFFFFFFF) print len(cbi) 

bigint itself is rejected, but if you mask it with & 0xFFFFFFFFFFFFFFFF, you can reduce it to 8 bytes of int instead of 16. Then the top is shifted and masked. You may need to play a bit with the byte. I used! to tell him to create a serial byte address of the network. In addition, you may need to change the value of msb and lsb (upper and lower bytes). I will leave this as an exercise for user definition. I would say saving things, since the network endian will be more secure, so you always know what the endianess of your data is.

No, don’t ask me if the network endian is big or small endian ...

Answer 9

Based on @DSM's answer and support for negative integers and byte size, I created the following improved code snippet:

 def to_bytes(num, size): x = num if num >= 0 else 256**size + num h = hex(x)[2:].rstrip("L") return binascii.unhexlify("0"*((2*size)-len(h))+h) 

This will handle negative integers correctly and allow the user to set the number of bytes