Skip to content

fix: make upload timeouts configurable, default to 1 hour (#942) - #943

Open
alanbchristie wants to merge 3 commits into
stagingfrom
fragalysis-backend_abc-942-upload-timeouts
Open

fix: make upload timeouts configurable, default to 1 hour (#942)#943
alanbchristie wants to merge 3 commits into
stagingfrom
fragalysis-backend_abc-942-upload-timeouts

Conversation

@alanbchristie

@alanbchristie alanbchristie commented May 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

Closes #942 (in this repo). Large file uploads (e.g. 6 GiB) were producing 504 Gateway Timeout because every hop in the request path had its own timeout, and the in-container nginx capped at 1000s (~16.7 min).

  • nginx.conf / django_nginx.conf are now envsubst templates. All timeout directives derive from STACK_NGINX_TIMEOUT_S (default 3600s) rendered by launch-stack.sh before nginx -tq runs. envsubst is scoped to that single variable so nginx's own $vars (e.g. $http_host, $; regex anchors) pass through untouched.
  • Gunicorn worker timeout is now driven by STACK_GUNICORN_TIMEOUT_S (default 3600s), replacing the hard-coded 3000s.
  • Dockerfile pulls in gettext-base (provides envsubst) and stages the nginx configs as templates under /etc/nginx/templates/.
  • Django DATA_UPLOAD_MAX_MEMORY_SIZE (default 10 GiB) and FILE_UPLOAD_MAX_MEMORY_SIZE (default 2.5 MiB, Django's own default made explicit) are now env-driven Django settings so request-body and in-memory upload thresholds are visible and tunable.

⚠️ Companion change required in fragalysis-stack-kubernetes

The Kubernetes NGINX ingress controller (v1.12.0) is configured in fragalysis-stack-kubernetes, not here. Its defaults (60s) will still produce 504s regardless of this PR. The ingress (or per-route annotations on the upload endpoint) must be raised in lockstep:

nginx.ingress.kubernetes.io/proxy-read-timeout:    "3600"
nginx.ingress.kubernetes.io/proxy-send-timeout:    "3600"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "75"
nginx.ingress.kubernetes.io/proxy-body-size:       "0"

The two new Django env vars (DATA_UPLOAD_MAX_MEMORY_SIZE, FILE_UPLOAD_MAX_MEMORY_SIZE) and the two new stack env vars (STACK_NGINX_TIMEOUT_S, STACK_GUNICORN_TIMEOUT_S) should be surfaced as stack_* Ansible variables in the playbook repo if they need to be tunable per-deployment.

Test plan

  • Python syntax check: python -m py_compile fragalysis/settings.py gunicorn.conf.py
  • Bash syntax check: bash -n launch-stack.sh
  • envsubst rendering simulated locally — both files render correctly, nginx $vars and regex anchors preserved
  • pre-commit passes on the changed files
  • docker compose up -d --build succeeds and nginx -tq (run inside launch-stack.sh) passes — please verify in a real container build; skipped here as it re-runs the full apt/texlive layer (~10+ min)
  • docker compose exec backend cat /etc/nginx/nginx.conf | grep timeout shows 3600s on all seven directives; same for /etc/nginx/sites-available/default.conf
  • Override test: set STACK_NGINX_TIMEOUT_S=120 in compose, restart, confirm rendered values change
  • docker compose exec backend python manage.py shell -c "from django.conf import settings; print(settings.DATA_UPLOAD_MAX_MEMORY_SIZE, settings.FILE_UPLOAD_MAX_MEMORY_SIZE)" returns the expected ints
  • End-to-end upload of a large file (>1 GiB) against a dev cluster with the ingress annotations above applied — confirms no 504

🤖 Generated with Claude Code

Large file uploads (e.g. 6 GiB) were producing 504 Gateway Timeout
because every hop in the request path had its own timeout and the
in-container nginx capped at 1000s.

- nginx.conf / django_nginx.conf are now envsubst templates; all
  timeout directives derive from STACK_NGINX_TIMEOUT_S (default 3600s)
  rendered by launch-stack.sh before nginx -tq runs. nginx's own $vars
  pass through untouched (envsubst is scoped to the one variable).
- gunicorn worker timeout is now driven by STACK_GUNICORN_TIMEOUT_S
  (default 3600s), replacing the hard-coded 3000s.
- Dockerfile pulls in gettext-base for envsubst and stages the nginx
  configs as templates under /etc/nginx/templates/.
- DATA_UPLOAD_MAX_MEMORY_SIZE and FILE_UPLOAD_MAX_MEMORY_SIZE are now
  env-driven Django settings so request-body and in-memory upload
  thresholds are explicit (defaults 10 GiB and 2.5 MiB).

The Kubernetes NGINX ingress controller is configured in
fragalysis-stack-kubernetes; it must be raised to >=3600s in lockstep
or this change alone won't lift the 504.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
a.b.christie and others added 2 commits May 28, 2026 12:50
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
alanbchristie added a commit that referenced this pull request May 28, 2026
Large file uploads (e.g. 6 GiB) were producing 504 Gateway Timeout
because every hop in the request path had its own timeout, and the
in-container nginx capped at 1000s.

- nginx.conf / django_nginx.conf are now envsubst templates; all
  timeout directives derive from NGINX_TIMEOUT_S (default 3600s)
  rendered by launch-stack.sh before nginx -tq runs. nginx's own $vars
  pass through untouched (envsubst is scoped to the one variable).
- Gunicorn --timeout is now driven by STACK_GUNICORN_TIMEOUT_S
  (default 3600s), replacing the hard-coded 3000s in launch-stack.sh.
- Dockerfile pulls in gettext-base for envsubst and stages the nginx
  configs as templates under /etc/nginx/templates/.
- DATA_UPLOAD_MAX_MEMORY_SIZE (default 18 GiB) and
  FILE_UPLOAD_MAX_MEMORY_SIZE (default 2.5 MiB, Django's own are now
  env-driven Django settings so request-body and in-memory upload
  thresholds are explicit and tunable.

The Kubernetes NGINX ingress controller is configured in
fragalysis-stack-kubernetes; it must be raised to >=3600s in lockstep
or this change alone won't lift the 504.

Mirrors the change applied to staging via PR #943.


EOF
)

Co-authored-by: a.b.christie <alan.christie@matildapeak.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@alanbchristie

Copy link
Copy Markdown
Collaborator Author

NOTE This has been merged to the unification branch there is (probably) no reason to merge this to staging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

504 Gateway Timeout uploading large files

1 participant