Why does a python thread consume so much memory?

Question

Why does a python thread consume so much memory?

I measured that spawning one thread consumes 8 megabytes of memory, almost as big as the whole new python process!

OS: Ubuntu 10.10

Edit: due to popular demand, I will give some extraneous examples, here it is:

from os import getpid from time import sleep from threading import Thread def nap(): print 'sleeping child' sleep(999999999) print getpid() child_thread = Thread(target=nap) sleep(999999999) 

In my pmap pid box will give 9424K

Now run the child thread:

 from os import getpid from time import sleep from threading import Thread def nap(): print 'sleeping child' sleep(999999999) print getpid() child_thread = Thread(target=nap) child_thread.start() # <--- ADDED THIS LINE sleep(999999999) 

Now pmap pid will give 17620K

So, the cost of the additional flow is 17620K - 9424K = 8196K

T. 87% launch of a completely new separate process!

Now this is not so, is it wrong?

Answer 1

This is independent of Python and relates to a separate stack that is allocated by the OS for each thread. The default maximum stack size for your OS is 8 MB.

Note that 8 MB is just a piece of address space that is discarded, with very little memory that was originally executed. Additional memory gets bound to the stack when required, up to a limit of 8 MB.

The limit can be configured using ulimit -s , but in this case I see no reason for this.

Aside, pmap shows the use of address space. This is not the best way to evaluate memory usage. The two concepts are very different if they are related.