Fix Execution API JWT reissue TOCTOU race#70227
Open
anmolxlight wants to merge 1 commit into
Open
Conversation
JWTReissueMiddleware re-validated the bearer token after the handler ran. If the token expired mid-request, ExpiredSignatureError was swallowed and no Refreshed-API-Token was returned, so the next heartbeat failed with 403. Reuse claims already cached on the ASGI scope by JWTBearer instead. Closes: apache#70222 Related: apache#67939 Signed-off-by: anmolxlight <anmolx.work@gmail.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #70222
Problem
Intermittent
ExpiredSignatureError/Failed to validate JWTon/execution/...for tasks that are not necessarily older than[execution_api] jwt_expiration_time.Root cause is a TOCTOU race in
JWTReissueMiddleware:JWTBearervalidates the bearer token (still valid) and caches claims on the ASGI request scope.avalidated_claimson the same bearer string.expduring the request, that second validation raisesExpiredSignatureError.Refreshed-API-Tokenheader is returned.This matches the intermittent multi-task failures reported in #70222 and the long-running LocalExecutor case in #67939.
Fix
Reuse the claims already validated and cached by
JWTBearer(request.scope["ti_token"]) instead of re-parsing/re-validating the Authorization header. If remaining lifetime is below the existing refresh threshold, issue a fresh token from those claims.No expiry leeway is added. Workload tokens remain non-refreshable.
Tests
expduring the handler) assertingRefreshed-API-Tokenis still issued andavalidated_claimsis not called by middleware.Local verification
All 5 tests passed.
Notes
Related open PR #68499 targets the same race for #67939. This PR is based on current
main, includes ashb's requested mid-request time-travel regression shape, and closes #70222.