Nebraska never deletes old instance data

[Akshats-git]

Current situation

Nebraska never deletes rows from the tables that track the fleet. These tables only grow:

• instance
• instance_application
• instance_status_history
• event
• activity / admin_activity
• instance_stats

There is no cleanup job and no retention setting. The only DELETE statements in the backend are for channels, packages, blacklists and floors.

Nebraska already knows what a dead instance is. In backend/pkg/api/internal/dbreads/groups.go:

// If an instance doesn't update its status for deadInstanceTimeSpan then the instance
// is considered dead.
deadInstanceTimeSpan = "6 months"

That constant is only used to limit how far back a query looks. The rows stay in the database forever.

instance_stats is a separate case. A background job in backend/pkg/server/server.go inserts into it every hour. No API handler reads it. The only readers of GetInstanceStats and GetInstanceStatsByTimestamp are unit tests and the Playwright fixtures.

Impact

A machine that was decommissioned two years ago is still a row in instance, with its status history, events and activity attached. Fleets with node churn hit this fast. Two users described the problem in #352:

We have multiple clusters, adding and removing flatcar nodes hourly by the 10's.

There needs to be a way to auto-remove or hide instances that are no longer reporting.

Constant growth also feeds the bloat described in #507. Autovacuum thresholds are based on table size, so the bigger these tables get the less often it runs.

Ideal future situation

Operators can set a retention window. Data older than that window is removed automatically in the background.

Retention should be off by default so existing deployments do not change behaviour on upgrade.

Implementation options

The schema already supports this. instance_application, instance_status_history, event and activity all declare references instance (id) on delete cascade. Deleting from instance cleans up the rest.

Rough plan:

• Add instance-retention, history-retention and stats-retention flags in backend/pkg/config/config.go. Default them to 0, which means disabled.
• Run the pruner from the same place as the existing hourly stats job in backend/pkg/server/server.go.
• Pick instances by their newest last_check_for_updates in instance_application, falling back to instance.created_ts.
• Delete in batches so the job does not hold a long lock on a hot table.
• Add a log-only mode that reports how many rows would be removed. This lets operators check the impact before turning retention on.
• Add a nebraska_rows_pruned_total counter and a short doc page under docs/.

Happy to work on this if it sounds useful.