Skip to content

Fix Execution API JWT reissue TOCTOU race#70227

Open
anmolxlight wants to merge 1 commit into
apache:mainfrom
anmolxlight:fix-jwt-reissue-mid-request
Open

Fix Execution API JWT reissue TOCTOU race#70227
anmolxlight wants to merge 1 commit into
apache:mainfrom
anmolxlight:fix-jwt-reissue-mid-request

Conversation

@anmolxlight

Copy link
Copy Markdown
Contributor

closes: #70222

Problem

Intermittent ExpiredSignatureError / Failed to validate JWT on /execution/... for tasks that are not necessarily older than [execution_api] jwt_expiration_time.

Root cause is a TOCTOU race in JWTReissueMiddleware:

  1. JWTBearer validates the bearer token (still valid) and caches claims on the ASGI request scope.
  2. The request handler runs (e.g. heartbeat).
  3. Middleware re-calls avalidated_claims on the same bearer string.
  4. If the token crossed exp during the request, that second validation raises ExpiredSignatureError.
  5. The exception is swallowed, so no Refreshed-API-Token header is returned.
  6. The client's next call arrives with a fully expired token → 403, and after enough failed heartbeats the task is killed.

This matches the intermittent multi-task failures reported in #70222 and the long-running LocalExecutor case in #67939.

Fix

Reuse the claims already validated and cached by JWTBearer (request.scope["ti_token"]) instead of re-parsing/re-validating the Authorization header. If remaining lifetime is below the existing refresh threshold, issue a fresh token from those claims.

No expiry leeway is added. Workload tokens remain non-refreshable.

Tests

  • Updated threshold reissue tests to populate the ASGI scope the way production does.
  • Added a mid-request expiry regression test (custom route advances the clock past exp during the handler) asserting Refreshed-API-Token is still issued and avalidated_claims is not called by middleware.

Local verification

uv run --project airflow-core pytest airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py -xvs
ruff check airflow-core/src/airflow/api_fastapi/execution_api/app.py airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py
ruff format --check airflow-core/src/airflow/api_fastapi/execution_api/app.py airflow-core/tests/unit/api_fastapi/execution_api/versions/head/test_router.py

All 5 tests passed.

Notes

Related open PR #68499 targets the same race for #67939. This PR is based on current main, includes ashb's requested mid-request time-travel regression shape, and closes #70222.

JWTReissueMiddleware re-validated the bearer token after the handler ran.
If the token expired mid-request, ExpiredSignatureError was swallowed and
no Refreshed-API-Token was returned, so the next heartbeat failed with 403.

Reuse claims already cached on the ASGI scope by JWTBearer instead.

Closes: apache#70222
Related: apache#67939
Signed-off-by: anmolxlight <anmolx.work@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:task-sdk

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Execution API JWT expires unexpectedly in Airflow 3.2.2 (ExpiredSignatureError on /execution/...)

1 participant