Difference between Bytearray and List in Python

Question

I'm curious to see how memory management differs between Bytearray and a list in Python.

I found several questions, such as The difference between bytearray and a list , but didn’t exactly answer my question.

My question is for sure ...

from array import array >>> x = array("B", (1,2,3,4)) >>> x.__sizeof__() 36 >>> y = bytearray((1,2,3,4)) >>> y.__sizeof__() 32 >>> z = [1,2,3,4] >>> z.__sizeof__() 36 

As we can see, there is a size difference between the list /array.array (36 bytes for 4 elements) and byte (32 bytes for 4 elements). Can someone explain to me why this is? For a byte array, it makes sense that it takes 32 bytes of memory for 4 elements ( 4 * 8 == 32 ) , but how can this be interpreted for a list and array.array?

 # Lets take the case of bytearray ( which makes more sense to me at least :p) for i in y: print(i, ": ", id(i)) 1 : 499962320 2 : 499962336 #diff is 16 units 3 : 499962352 #diff is 16 units 4 : 499962368 #diff is 16 units 

Why the difference between two adjacent elements here is different from 16 units, when each element takes up only 8 bytes. Does this mean that each memory address pointer points to a nibble?

Also what are the criteria for allocating memory for an integer? I read that Python will allocate more memory based on the value of an integer (correct me if I am wrong), the larger the number, the more memory.

For instance:

 >>> y = 10 >>> y.__sizeof__() 14 >>> y = 1000000 >>> y.__sizeof__() 16 >>> y = 10000000000000 >>> y.__sizeof__() 18 

What are the criteria by which Python allocates memory?

And why is Python taking up much more memory, and C takes up only 8 bytes (mine is a 64-bit machine)? when they are perfectly in the range of integers (2 ** 64) ?

Metadata:

Python version: '3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:43:06) [MSC v.1600 32 bit (Intel)]'

Machine Architecture: 64-bit

PS : Please bring me to a good article where Python memory management is better explained. I spent almost an hour to understand these things, and ended up asking this Question in SO .:(

Answer 1

I am not saying this is the complete answer, but there are some tips for understanding this.

bytearray is a sequence of bytes, and list is a sequence of references to objects. Thus, [1,2,3] actually stores pointers to cells for those integers that are stored in memory elsewhere. To calculate the total memory consumption by the list structure, we can do this (I use sys.getsizeof everywhere, call __sizeof__ plus GC overhead)

 >>> x = [1,2,3] >>> sum(map(getsizeof, x)) + getsizeof(x) 172 

The result may vary on different machines.

Also, look at this:

 >> getsizeof([]) 64 

This is because lists are mutable. To be fast, this structure allocates some memory range for storing references to objects (plus some storage for meta, for example, the length of the list). When you add items, the following memory cells are filled with links to those items. When there is no place to store new items, a new, larger range is allocated, there is data copied there, and old ones released. This is called dynamic arrays.

You can observe this behavior by running this code.

 import sys data=[] n=15 for k in range(n): a = len(data) b = sys.getsizeof(data) print('Length: {0:3d}; Size in bytes: {1:4d}'.format(a, b)) data.append(None) 

My results:

 Length: 0; Size in bytes: 64 Length: 1; Size in bytes: 96 Length: 2; Size in bytes: 96 Length: 3; Size in bytes: 96 Length: 4; Size in bytes: 96 Length: 5; Size in bytes: 128 Length: 6; Size in bytes: 128 Length: 7; Size in bytes: 128 Length: 8; Size in bytes: 128 Length: 9; Size in bytes: 192 Length: 10; Size in bytes: 192 Length: 11; Size in bytes: 192 Length: 12; Size in bytes: 192 Length: 13; Size in bytes: 192 Length: 14; Size in bytes: 192 

We can see that 64 bytes are used to store 8 memory addresses (64 bits each).

Almost the same goes for bytearray() (change the second line to data = bytearray() and add 1 to the last).

 Length: 0; Size in bytes: 56 Length: 1; Size in bytes: 58 Length: 2; Size in bytes: 61 Length: 3; Size in bytes: 61 Length: 4; Size in bytes: 63 Length: 5; Size in bytes: 63 Length: 6; Size in bytes: 65 Length: 7; Size in bytes: 65 Length: 8; Size in bytes: 68 Length: 9; Size in bytes: 68 Length: 10; Size in bytes: 68 Length: 11; Size in bytes: 74 Length: 12; Size in bytes: 74 Length: 13; Size in bytes: 74 Length: 14; Size in bytes: 74 

The difference is that memory is now used to store actual byte values, not pointers.

Hope this helps you explore further.