How to properly capture and process RQ timeouts in Python?

Question

Trying to find a good way to catch the RQ job timeout, so you can request it after the timeout.

Basically, the right solution would provide a way (for example, an exception handler in the worker or something of its kind) to request the expiration of jobs. Also, if the job returns to the failed queue, this is a good answer.

Many thanks! Any help would be appreciated!

Answer 1

It looks like you want to use exception handling . From the docs:

Operation may fail due to exceptions. When your RQ employees work in the background, how do you get notified of these exceptions?

Default: failed queue. The default security system for RQ is a failed queue. Each task that does not execute is stored here, along with its exception information (type, value, trace). Although this makes sure that no unsuccessful tasks are “lost”, it is useless to receive pro-active notification of a failure.

Custom exception handlers Starting with version 0.3.1, support for RQ registration of custom exception handlers. This allows you to replace the default behavior (send the job to an unsuccessful queue) or take additional steps if an exception occurs.

You can also save jobs in a redis sorted set with the key job_id and time.time() + timeout as an estimate, and then do a working run of ZRANGEBYSCORE sorted_set 0 [current_time] and process everything that was returned as the given time.