From 20d12261368ae93b330c3c206fb394bbefcb40c6 Mon Sep 17 00:00:00 2001 From: Dmitrii Creed Date: Sat, 30 May 2026 22:56:59 +0400 Subject: [PATCH] fix(docker): disable AddPreviousOutputInEnv on security-report MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Root cause for the post-v2026.5.43 E2BIG (run 26690935574 in integrail/baas). ## Wrong diagnosis (initial version of this PR) I originally hypothesised that the pulumi-command provider's `default → default_1_2_1` migration was passing the OLD serialised Create through fork/exec, and proposed `ReplaceOnChanges` + `DeleteBeforeReplace` as the fix. Reading pulumi-command's source carefully refutes that: // commandController.go func (c *Command) Update(ctx, req) { ... cmd := news.Create // uses NEW Create, not olds. ... err := run(ctx, *cmd, state.BaseInputs, ...) } The Update path runs the *new* Create. The migration carries no old Create through fork/exec. ## Actual root cause `pulumi-command/provider/pkg/provider/local/commandOutputs.go::run()`: if in.AddPreviousOutputInEnv == nil || *in.AddPreviousOutputInEnv { if out.Stdout != "" { cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", util.PulumiCommandStdout, out.Stdout)) } if out.Stderr != "" { cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", util.PulumiCommandStderr, out.Stderr)) } } `AddPreviousOutputInEnv` **defaults to TRUE**. On every Update, pulumi-command appends the previous run's stdout to envp as a single `PULUMI_COMMAND_STDOUT=<...>` entry. The security-report script writes the full rendered vulnerability table to stdout for inline build-log visibility — on a chrome-base- derived image that's a 150-200 KB markdown table (thousands of CVE rows from Trivy + Grype merged). pulumi-command captures that into the resource's `Stdout` output. On the next Update the captured stdout is re-injected as a single envp entry. Kernel ARG_MAX (~128 KB on Linux) covers argv+envp combined → exec fails with E2BIG even though our argv is now ~80 bytes thanks to tempfile staging. This explains the empirical observations: - `scan-baas--test/baas-merged` succeeded — stdout is a short summary. - `security-report-baas--test/baas` failed — stdout is the full table. - Argv visible in the error is short (~80 bytes), but envp is the bloat. - The "migration" appearance in the diff was a coincidence — provider upgrades changed identifier-style but didn't change envp semantics. ## Fix Set `AddPreviousOutputInEnv: sdk.Bool(false)` on the security-report resource. The security scripts don't read `PULUMI_COMMAND_STDOUT`, so disabling the env injection is a no-op for script behaviour. Stdout is still captured into Pulumi state for `stack output` / downstream references, just not re-injected into envp. ## Why not `ReplaceOnChanges` + `DeleteBeforeReplace` Initial draft used those options. They *happen* to work (replace creates a fresh resource with empty `out.Stdout`, so the env injection has nothing to pass), but for the wrong reason: - The replace path is wasteful — every change to the report content (i.e. every deploy with a new CVE set) replaces the resource. - It doesn't address the actual root cause; if pulumi-command ever starts passing state by other means we'd hit a regression. - Pulumi state churn (replace vs update) clutters operator-visible diff. `AddPreviousOutputInEnv: false` is surgical and addresses the actual problem. ## Tests Existing 9 tests still pass — this change only affects Pulumi resource options, not the helper's behaviour. The envp injection path is exercised end-to-end on the next baas deploy retry after this fix releases. ## Blast radius Audited every `local.NewCommand` site in SC: - `security-report-` — bug manifests here. Fix in this PR. - `scan--merged` / `scan--` — `createScanLocalCommand`. Stdout is bounded (Trivy/Grype's summary output ~few KB). Not at risk, but inconsistent default. Worth setting to false too for defense-in-depth, in a follow-up PR after we validate this lands. - `registry-login-` — short fixed-size output. Safe. - `sign-` / `verify-` / `sbom-` / `provenance-` via `newSecurityCommand` — cosign/syft output bounded. Safe. Out of scope for this PR but tracked. Signed-off-by: Dmitrii Creed --- pkg/clouds/pulumi/docker/build_and_push.go | 29 ++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/pkg/clouds/pulumi/docker/build_and_push.go b/pkg/clouds/pulumi/docker/build_and_push.go index 01343f3a..f71042c9 100644 --- a/pkg/clouds/pulumi/docker/build_and_push.go +++ b/pkg/clouds/pulumi/docker/build_and_push.go @@ -252,6 +252,35 @@ func executeSecurityOperations(ctx *sdk.Context, stack api.Stack, dockerImage *d } return fmt.Sprintf("sh %s", shellQuote(path)) }).(sdk.StringOutput), + + // Disable the default behaviour of passing the previous run's + // stdout/stderr back as PULUMI_COMMAND_STDOUT / PULUMI_COMMAND_STDERR + // env vars on Update. pulumi-command's + // `run()` builds envp with: + // + // if in.AddPreviousOutputInEnv == nil || *in.AddPreviousOutputInEnv { + // if out.Stdout != "" { + // cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", + // util.PulumiCommandStdout, out.Stdout)) + // } + // ... + // } + // + // The security-report script writes the full rendered vulnerability + // table to stdout for inline build-log visibility. On a chrome-base- + // derived image that's a 150-200 KB markdown table (thousands of CVE + // rows). pulumi-command captures that into the resource's `Stdout` + // output, which is then echoed back as a SINGLE envp entry on the + // next Update. Kernel ARG_MAX (~128 KB on Linux) covers argv+envp + // combined, so the next deploy fails at `fork/exec /bin/sh: argument + // list too long` — even though our argv is now ~80 bytes thanks to + // tempfile staging. + // + // Our security scripts don't read PULUMI_COMMAND_STDOUT; turning + // this off is purely a no-op for the script's behaviour. The + // stdout is still captured into state for `pulumi stack output` and + // downstream resource references, just not re-injected into envp. + AddPreviousOutputInEnv: sdk.Bool(false), }, sdk.DependsOn(reportDeps)) if err != nil { return nil, errors.Wrapf(err, "failed to create security report for image %q", imageName)