Consider rejecting taskprov tasks whose task expiration exceeds the peer aggregator's report expiry age

[tgeoghegan]

Taskprov tasks have a task expiration date, past which an aggregator will refuse to participate in the task. But before a given peer aggregator can provision such tasks, Janus operators have to make a taskprov_peer_aggregators entry, which includes report_expiry_age, used to determine if artifacts are eligible for garbage collection. These two values can interact in surprising ways: suppose a peer aggregator's report expiry age is set to 14 days, but then they configure a task which expires in 21 days. If that leader sends aggregation jobs at the start of the task, it might expect to be able to collect the results of those jobs in an aggregate share 21 days later, but in fact Janus will have garbage collected them. We might want to do some validation at task provisioning time for this.

[tgeoghegan]

We don't check the two values against each other on main, and it's much the same situation on release/0.7.

I'm not certain if putting a restriction like this in place is necessarily valid in all cases. We could imagine a task that is intended to run for 28 days and sets its task expiration accordingly, and plans to collect every seven days. So it might be reasonable to set the report_expiry_age to 7 days, if you're confident about keeping up with your collection cadence.

[divergentdave]

I don't think this would make sense, as DAP tasks may be long-lived, for continual observation-style settings. Generally, the first batch and the last batch in a task could be arbitrarily far apart in time. This sort of validation would mean that we'd put a cap on the length of task intervals, and push longer-running telemetry tasks to not use task intervals at all. Our report_expiry_age parameter reflects operational constraints on how long we store anti-replay data, and is our approach to exercise the flexibility outlined in section 6.4.1 of DAP. I think the proper solution here is for leader and helper operators to make sure their state retention strategies will work well together, and that the leader completes aggregation and collection for reports and batches before either aggregator's strategy allows for dropping relevant state.

[jcjones]

Rather than a static check at provisioning time, what if we raise this via observability to the end-customer? We could add a counter for "artifacts deleted before {aggregation, collection}" and provide it via API to the consumer via one path or other, so ultimately leader and helper can give out a matrix for reports and aggregates. In a properly-tuned environment they'd be 0.

[tgeoghegan]

Yeah, or we could wire up a notification if some uncollected reports are about to be garbage collected. We could use the existing GC machinery with different parameters to check if data will expire in a week instead of whether it's expired right now, and then it'd send an email or something. Of course this requires us to wire up some notification mechanism from Janus to some other component that knows how to send emails.