Skip to content

Run celery beat in separate container#233

Draft
Charl1996 wants to merge 1 commit into
mainfrom
infra/run-beat-as-own-container
Draft

Run celery beat in separate container#233
Charl1996 wants to merge 1 commit into
mainfrom
infra/run-beat-as-own-container

Conversation

@Charl1996

Copy link
Copy Markdown
Contributor

Technical Summary

This PR results from this ticket: https://dimagi.atlassian.net/browse/CI-593

As per celery docs:

You have to ensure only a single scheduler is running for a schedule at a time, otherwise you’d end up with duplicate tasks

Currently we run the celery beat in the same process as the celery worker, although this configuration is not recommended for production (at least when running more than one worker):

... celery -A config.celery_app worker -l INFO --concurrency 2 --max-tasks-per-child 20 --beat

We've gotten away with running it this way because we only have one worker instance, but if we scale workers we might run into trouble.

Logging and monitoring

No new logs - after deploy I'll monitor the logs to ensure tasks are registered and executed.

Safety Assurance

Current celery beat is not running, and the worst case scenario is the same, ending up with beat not running. No impact expected on the workers.

Safety story

  • I am confident that this change will not break current and/or previous versions of CommCare apps

Automated test coverage

No automated tests

QA Plan

N.A

Labels & Review

  • The set of people pinged as reviewers is appropriate for the level of risk of the change

@Charl1996 Charl1996 requested a review from calellowitz May 22, 2026 12:01

@ajeety4 ajeety4 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.
I think it would be good to check that the new container does not add significant memory overhead, which I assume it does not.

Will wait for Cal's thoughts if any.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants