Is Celery timeout limit actually being used?

[pooleycodes]

Several months ago to try and stop endless spin out of tasks in the async process the timeout of 30 minutes was introduced at which point it should send a SIGKILL to the tasks, the parent ticket to this suggests it is not.

ToDo

• Investigate with a timeout of 1 minute and many comma file to see what signals are being sent if/and they being ignored?
• What can be changed to make them used.

[pooleycodes]

The 30-minute hard time limit has never been firing. Tested locally with a 1-minute limit and a blocking task: no SIGKILL was sent to the task at all — so there was nothing being "ignored," the signal simply never fires.

The cause is the Celery pool type. The worker runs with -P eventlet, which does not enforce task_time_limit — the eventlet pool has no supervisor process to kill a running task, so the limit is dead config.

Switching to -P prefork fixes this: prefork runs a supervisor (MainProcess) that tracks each task's runtime and SIGKILLs the child process on the hard limit. Confirmed locally, including killing a task wedged in a blocking psycopg2 call — i.e. the stuck-commit() scenario.

It's believed eventlet was chosen because it's well suited to high-concurrency network tasks, but CELERY_WORKER_CONCURRENCY has never been set (it defaults to 1), so the worker only ever processes one task at a time — meaning eventlet's benefits were never actually reaped. So moving to prefork is safe: no throughput is lost at the current setting, and we gain working kill-on-timeout.

Extra quick Q

Any concurrency increase (e.g. --concurrency 2 to use both vCPUs) could be worthwhile (two ECS tasks, both set to 2vCPUS) - this could double our throughput? gated on measuring per-child memory against the 4 GB ECS task limit, to avoid trading the hang for OOM kills.