Fix intermittent Lambda 409 when adding Function URL permissions#6933
Open
AppleDannyClegg wants to merge 1 commit into
Open
Fix intermittent Lambda 409 when adding Function URL permissions#6933AppleDannyClegg wants to merge 1 commit into
AppleDannyClegg wants to merge 1 commit into
Conversation
Deploying an SSR site (Nextjs/Astro/etc.) intermittently fails with:
operation error Lambda: AddPermission, StatusCode: 409,
ResourceConflictException: The function could not be updated due to a
concurrent update operation.
Lambda serialises mutating control-plane operations per function. During a
deploy, two resources mutate the same function but have no ordering between
them, so Pulumi runs them concurrently:
- the `aws.lambda.FunctionUrl` created in `Function.createUrl()`
(CreateFunctionUrlConfig), and
- the `aws.lambda.Permission` granting `lambda:InvokeFunctionUrl`.
For the OAC path the permission is created in `SsrSite` and depends only on
the function, not on the function URL — and the URL resource is not reachable
because it is not exposed on `Function.nodes`. The permission's AddPermission
call therefore lands while CreateFunctionUrlConfig still has the function in
`LastUpdateStatus: InProgress`, yielding the 409. Fresh stacks hit it far more
often because the function and its URL are created together in one run.
Serialise the permission after the function URL:
- expose the `FunctionUrl` resource on `Function.nodes.url`, and
- add `dependsOn` on it to every `lambda.Permission` that grants URL access,
both the ones created inside `Function.createUrl()` and the OAC/CloudFront
ones created in `SsrSite`.
`dependsOn` is metadata-only, so existing permissions are not replaced.
f2912b6 to
ce8177c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix intermittent Lambda 409 when adding Function URL permissions
Problem
Deploying an SSR site (
Nextjs,Astro,SolidStart, …) intermittently fails with:It's non-deterministic and hits fresh/ephemeral stacks (e.g. per-PR preview
environments) far more often than re-deploys. Retrying the deploy succeeds.
Cause
AWS Lambda serialises mutating control-plane operations per function: after any
update the function is briefly
LastUpdateStatus: InProgress, and a secondmutating call in that window is rejected with a 409.
During a deploy two resources mutate the same function and, with no ordering
between them, Pulumi runs them concurrently:
aws.lambda.FunctionUrl— created inFunction.createUrl()(
CreateFunctionUrlConfig), andaws.lambda.Permissionwithaction: "lambda:InvokeFunctionUrl".For the OAC path the permission is created in
SsrSiteand depends only on thefunction — not on the function URL. It can't depend on the URL because the
FunctionUrlresource isn't exposed onFunction.nodes(onlyrole,function,logGroup,eventInvokeConfig). SoAddPermissionracesCreateFunctionUrlConfigand lands mid-update. Fresh stacks hit it more oftenbecause the function and its URL are created together in a single run.
Fix
Serialise the permission after the function URL:
FunctionUrlresource onFunction.nodes.url, anddependsOnon it to everylambda.Permissionthat grants URL access —both the ones created inside
Function.createUrl()(public + router modes)and the OAC/CloudFront ones created in
SsrSite(server functions and theimage optimizer).
dependsOnonly changes the Pulumi dependency graph, so existing permissionresources are updated in place — no replacement, no downtime.
Testing
tsc --noEmitpasses.platformtest suite: all component tests pass (the only failures are apre-existing Node-version incompatibility in unrelated suites —
alb,bucket,service-alb— that error on@types/noderesolution /ERR_TRACE_EVENTS_UNAVAILABLE, unaffected by this change).oac/oac-with-edge-signingandnoneprotection modes (and a multi-regiondeploy) to confirm the 409 no longer occurs and that no permission resources
are replaced on an in-place update.
Notes / possible follow-ups
The same class of concurrent-mutation race can in principle affect other
per-function operations that run alongside URL/permission creation (e.g.
provisioned-concurrency config, or an environment update). Those are out of
scope here; happy to look at them separately if maintainers think it's worth it.