prometheus: switch from time-based to size-based retention

[anguslees]

We should switch to size-based retention policy (--storage.tsdb.retention.size) since that most accurately reflects our PVC storage constraints.

[EamonKeane]

Agreed, --storage.tsdb.retention=183d ends up overloading the default 8Gi PV within 90 days depending on the cluster.

[0xshipthecode]

It seems that --storage.tsdb.retention.size is still marked experimental in the prometheus docs A second issue is that apparently the WAL size is not taken into account, so the size parameter does not actually limit the storage requirements. If both have been considered and this seems like the way to go, I can add this in.

[javsalgar]

Hi,

I just saw this PR: prometheus/prometheus#5886 It seems that the WAL size might be taken into account, am I wrong?

[optimus-kart]

Even I had an issue in produciton where the prometheus pod crashed after storage limit was reached, even though the storage.retention was configured on the server. Can anyone confirm if this is working as expected?