Challenge Heroku Celery

Question

We have a simple task working with django celery on Heroku. Sort of:

@task Simple_task(): for line in csv.reader(origin): process_line(line) process_line(line): fields = parse_line(line) reg = Model1() # Django model reg.field1 = fields[0] reg.field2 = fields[1] reg.field3 = fields[2] reg.save() 

Where the origin is the csv file. When the file is large (more than 50,000 lines), the task takes up all the memory, which leads to R14 errors until it is canceled by the system (with 150% of the available memory 512 MB). The memory is never released, and we must restart the task manually.

Running on a Linux machine or with wizards on a development machine without any problems (all 170,000 lines). It seems that the memory leak is ONLY on Heroku. By the way, we start with DEBUG = False.

Did something break with Heroku’s celery task? Anything we can't lose? This became a demo congestion when deploying to Heroku.

Any help would be greatly appreciated.

Answer 1

I agree with JoshB that in your case it takes up more than 512 MB of memory.

• What if you do the process_line task and create a queue for them instead of the task to process the entire file. In this case, your memory on Heroku will not be overloaded.

• Another possible solution for you could be a new service from Heroku, where you can use 1 GB of RAM on your speakers. Link: 2x dynos beta

Answer 2

Django leak error when DEBUG set to True because it saves a copy of every SQL statement that is executed.

You can test locally using a virtual machine with the same characteristics as your hosting. Or use ulimit to limit process memory. This way you can check if your code works locally with only 512 MB of RAM.