I have ~ 250,000 repetitive tasks every day; about a fifth of which can be updated with different scheduled days every day.
Can this be done effectively in celery? - It bothers me from beat.py celery:
def tick(self): """Run a tick, that is one iteration of the scheduler. Executes all due tasks. """ remaining_times = [] try: for entry in values(self.schedule): next_time_to_run = self.maybe_due(entry, self.publisher) if next_time_to_run: remaining_times.append(next_time_to_run) except RuntimeError: pass return min(remaining_times + [self.max_interval])
source share