Add configurable S3 lifecycle rules for state bucket

[FelixHeinstadt]

Problem: Over time the s3 storage usage exceeds multiple terabytes which causes increased costs and unnecessary storage usage. There should be an option to set sensible lifecycle rules for the sst state bucket.

Summary

• Adds env-var-driven S3 lifecycle rules for the SST state bucket
• Master toggle SST_S3_LIFECYCLE=true (disabled by default)
• Per-prefix config for expiration days (SST_S3_LIFECYCLE_{PREFIX}_DAYS) and noncurrent version retention (SST_S3_LIFECYCLE_{PREFIX}_VERSIONS)
• Supports app/, lock/, eventlog/, snapshot/, update/ prefixes
• Runs on every init (not a versioned bootstrap step), so config changes take effect immediately
• Non-fatal: failure logs a warning but doesn't block deploys

Usage

export SST_S3_LIFECYCLE=true
export SST_S3_LIFECYCLE_APP_DAYS=7
export SST_S3_LIFECYCLE_APP_VERSIONS=10
export SST_S3_LIFECYCLE_LOCK_DAYS=7
export SST_S3_LIFECYCLE_EVENTLOG_DAYS=7
export SST_S3_LIFECYCLE_SNAPSHOT_DAYS=30
export SST_S3_LIFECYCLE_UPDATE_DAYS=30

Issues

Closes #6074
Closes #4521

[vimtor]

thanks for your contribution @FelixHeinstadt

this looks very promising

i believe we should surface this as a top level config in sst.config.ts instead of env vars:

export default $config({
  app(input) {
    return {
      name: "my-sst-app",
      home: "aws",
      state: {
        retention: 30, // documented as keep X number of versions
      }
    };
  },
});

i've collapsed everything into x number of versions since time-based expiration seems scary, and having different options for each part doesn't seem worth it

we'll also need to implement it for the cloudflare home provider. it should be doable since it's s3 compatible

what do you think? do you want to give it a go? i can do the config part if you want to just leave one environment variable for the retention

[FelixHeinstadt]

@vimtor If I saw it correctly only /app actually creates new object versions. The others unfortunately just create new objects, so only saying keep x versions would not help...
But 100% agree using versioned objects and keeping x versions would be a lot better.

[FelixHeinstadt]

@vimtor let me know your thoughts on how to proceed.

[vimtor]

sorry i need to learn more about this to give you a proper response

my initial intution goes towards making everything a versioned object of finding a way to "when X state version is expired, delete all of the events, snapshots, updates, etc. before it"

[vimtor]

i'm merging some other state-related PRs. i'll get back to this soon. my current idea is that we should move to keeping all the state files as versioned files. i need to investigate this more, but i feel like it's worth doing anyway

then we can just add a new retention field as I put on my first comment

[outaTiME]

While state.retention is being implemented, is there any official guidance for users on v4.13.1+ (with state.compress now active) on how to safely clean up older noncurrent versions in the state bucket?

Specifically, would applying an S3 NoncurrentVersionExpiration lifecycle rule on the state bucket interfere with Pulumi's history or rollback mechanism?

Asking because in our case, snapshot files dropped from ~2.4MB to ~210KB after enabling compress, but the noncurrent uncompressed versions remain and there's no clear interim path to reclaim that storage without risking state integrity.

[outaTiME]

Hi @vimtor, following up on this — any updates on the versioned-files + retention direction? Happy to test once there's a branch.

In the meantime, I'd really appreciate some interim guidance on safely reclaiming storage in both buckets, since usage keeps growing on every deploy:

sst-state bucket

• After enabling state.compress, current snapshots dropped significantly, but the noncurrent uncompressed versions remain.
• Is it safe to apply an S3 NoncurrentVersionExpiration lifecycle rule (e.g. keep last N noncurrent versions of app/, snapshot/, update/, eventlog/), or does Pulumi rely on older noncurrent versions for recovery/rollback?
• If not lifecycle-safe, is there a recommended manual cleanup path (CLI command, prefixes safe to prune, etc.)?

sst-asset bucket

• Each deploy uploads new Lambda zip keys, and old ones accumulate indefinitely.
• For users not relying on Pulumi rollbacks to older state versions, would it be safe to apply a time-based expiration (e.g. objects older than 60–90 days) on this bucket?
• Or are there object prefixes / tag conventions that distinguish "still referenced by current stack" from "orphaned" so we can prune more confidently?

[vimtor]

Superseded by #6925, which implements count-based state history retention across AWS, Cloudflare, and local homes.