"Unable to start new thread error" in Python

Question

I run a script that extracts information from debian packages and stores it in a database.

After retrieving information from approximately 100 packets, an error occurs. Error: "Unable to start a new thread" Why did I encounter this error? What could be this possible solution?

This is the code used to save data:

for i in list_pack: if not i in oblist: #Creating Packages slu=slugify(i) ob=Gbobject() ob.title=i ob.slug=slu ob.content='' ob.tags=tagname #with reversion.create_revision(): ob.save() gb=Gbobject.objects.get(title=i) gb.objecttypes.add(Objecttype.objects.get(title='Packages')) gb.sites.add(Site.objects.get_current()) #with reversion.create_revision(): gb.save() gd=Gbobject.objects.get(title=i) print 'The Object created was',gd #Decsription try: atv=package_description_dict[i] atvalue=unicode(atv,'utf-8') except UnicodeDecodeError: pass try: lo=Gbobject.objects.get(title=i) loid=NID.objects.get(id=lo.id) except Gbobject.DoesNotExist: pass if atvalue<>'': slu=slugify(atvalue) at=Attribute() at.title=atvalue at.slug=slu at.svalue=atvalue at.subject=loid att=Attributetype.objects.get(title='Description') at.attributetype=att #with reversion.create_revision(): at.save() print 'yeloow13' 

Like Description , there are about 2 additional package properties that are saved in a similar way.

This is the full trace that I get when an error occurs: -

  error Traceback (most recent call last) /home/radhika/Desktop/dev_75/gnowsys-studio/demo/<ipython console> in <module>() /usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/gstudio/Harvest/debdata.py in <module>() 1086 # create_attribute_type() 1087 # create_relation_type() -> 1088 create_objects() 1089 #create_sec_objects() 1090 #create_relations() /usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/gstudio/Harvest/debdata.py in create_objects() 403 ob.sites.add(Site.objects.get_current()) 404 #with reversion.create_revision(): --> 405 ob.save() 406 #time.sleep(10) 407 #gd=Gbobject.objects.get(title=i) /usr/local/lib/python2.6/dist-packages/django_reversion-1.6.0-py2.6.egg/reversion/revisions.pyc in do_revision_context(*args, **kwargs) 298 try: 299 try: --> 300 return func(*args, **kwargs) 301 except: 302 exception = True /usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/objectapp/models.pyc in save(self, *args, **kwargs) 658 @reversion.create_revision() 659 def save(self, *args, **kwargs): --> 660 super(Gbobject, self).save(*args, **kwargs) # Call the "real" save() method. 661 662 /usr/local/lib/python2.6/dist-packages/django_reversion-1.6.0-py2.6.egg/reversion/revisions.pyc in do_revision_context(*args, **kwargs) 298 try: 299 try: --> 300 return func(*args, **kwargs) 301 except: 302 exception = True /usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/gstudio/models.pyc in save(self, *args, **kwargs) 327 @reversion.create_revision() 328 def save(self, *args, **kwargs): --> 329 super(Node, self).save(*args, **kwargs) # Call the "real" save() method. 330 331 /usr/local/lib/python2.6/dist-packages/django/db/models/base.pyc in save(self, force_insert, force_update, using) 458 if force_insert and force_update: 459 raise ValueError("Cannot force both insert and updating in model saving.") --> 460 self.save_base(using=using, force_insert=force_insert, force_update=force_update) 461 462 save.alters_data = True /usr/local/lib/python2.6/dist-packages/django/db/models/base.pyc in save_base(self, raw, cls, origin, force_insert, force_update, using) 568 if origin and not meta.auto_created: 569 signals.post_save.send(sender=origin, instance=self, --> 570 created=(not record_exists), raw=raw, using=using) 571 572 /usr/local/lib/python2.6/dist-packages/django/dispatch/dispatcher.pyc in send(self, sender, **named) 170 171 for receiver in self._live_receivers(_make_id(sender)): --> 172 response = receiver(signal=self, sender=sender, **named) 173 responses.append((receiver, response)) 174 return responses /usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/objectapp/signals.pyc in wrapper(*args, **kwargs) 65 if inspect.getmodulename(fr[1]) == 'loaddata': 66 return ---> 67 signal_handler(*args, **kwargs) 68 69 return wrapper /usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/objectapp/signals.pyc in ping_external_urls_handler(sender, **kwargs) 90 from objectapp.ping import ExternalUrlsPinger 91 ---> 92 ExternalUrlsPinger(gbobject) 93 94 /usr/local/lib/python2.6/dist-packages/django_gstudio-0.3.dev-py2.6.egg/objectapp/ping.pyc in __init__(self, gbobject, timeout, start_now) 153 threading.Thread.__init__(self) 154 if start_now: --> 155 self.start() 156 157 def run(self): /usr/lib/python2.6/threading.pyc in start(self) 472 _active_limbo_lock.release() 473 try: --> 474 _start_new_thread(self.__bootstrap, ()) 475 except Exception: 476 with _active_limbo_lock: error: can't start new thread 

I am not writing any code to handle threads.

Answer 1

Sorry for the incomplete solution here, I do not have a rating for publication in the comments section.

One thing to check is the total number of threads you use. I have code that checks the number of cores (using sys) and then starts threads and checks the kernel load to check how the OS handles thread allocation, and I found out that Windows 7 (for example) seems to throw 32 error thread on an 8 (logical) core processor. [This is in Python 2.7, 32 bits in Win 7 64-bit, etc., YMMMV]. On other machines, I can get more than 1000 threads.

So, I assume the short version is: how many threads do you already have when you get this error? You can check with

 threading.active_count() 

Besides the fact that you are not actually passing part of the code to enter text, I would point you to this excellent Python Central page .

You can also use the previous discussion when counting the number of large threads and how to approach them.

Again, my apologies that this is more a view than a decision, I think more information is likely to be needed to help us understand what you are working on.

Answer 2

You are using a 32-bit system and running out of virtual memory. One of your libraries probably spawns threads and doesn't fix them correctly. As a workaround, try reducing the default stack size to threading.stack_size .