Python: fine for sleeping topics

Question

this question refers to performance issues that may or may not occur due to the large number of python sleeping threads on the web server.

Reference Information. I am implementing an online store using django / satchmo. The requirement relates to late payment. The customer can reserve the product and allow a third party to pay for it later (using a random and unique URL).

To handle the unordering of an item, I create a thread that will sleep during the reservation time and then delete the reservation / mark the product as sold when it wakes up. It looks like this:

#Reserves a product when it is placed in the cart def reserve_cart_product(product): log.debug("Reserving %s" % product.name) product.active = False product.featured = False product.save() from threading import Timer Timer(CART_RESERVE_TIME, check_reservation, (product,)).start() 

I use the same method when filtering unique URLs after they expire, only the timer sleeps much longer (usually 5 days).

So my question for you SO is as follows:

Does a large number of sleeping threads seriously affect performance? Are there better methods for planning a single event in the future. I would like to keep this in python if possible; no call to at or cron via sys .

A site is not exactly high traffic; The (generous) upper limit for goods ordered per week will be around 100. Combined with trolley reservations, this may mean that there are more than 100 sleeping threads at any given time. Will I regret scheduling tasks this way?

thanks

Answer 1

I see no reason why this should not work. The base code for Timer (in threading.py) just uses time.sleep. As soon as he waited some time, he basically starts the loop with time.sleep (0.05). This should result in CPU usage being basically 0%, even with hundreds of threads. Here is a simple example where I noticed 0% CPU usage for the python process:

 import threading def nothing(): pass def testThreads(): timers = [threading.Timer(10.0, nothing) for _ in xrange(881)] print "Starting threads." map(threading.Thread.start, timers) print "Joining threads." map(threading.Thread.join, timers) print "Done." if __name__ == "__main__": testThreads() 

The real problem is that you cannot start too many threads. On my 64-bit 4 GB system, I can only start 881 streams before I get an error. If you really only have a few hundred, I cannot imagine that this will not work.

Answer 2

100 threads is not a problem, but how did tgray indicate what would happen if the server went down (Power cut, Planned Maintenance, Hardware failure, etc.)?

You need to store the reservation information in your database somewhere.

You can then force the cron job to run an unreservation script periodically, for example, and you do not need all these threads to sit.

If you really don't want to use cron, just one worker thread that sleeps for a minute and then checks to see if any of the holds are required.

Answer 3

Typically, sleeping threads have no overhead other than the memory allocated for their stacks and other private data. Modern operating system scheduling algorithms have O (1) complexity, so even a working thread does not introduce overhead except memory. At the same time, it is difficult to imagine an efficient design that requires many threads. The only case I can imagine is communicating with many other peers. In this case, use asynchronous IO.