[Feat]: High Availability

[awoimbee]

Is your feature request related to a problem? Please describe

Right now there is trow-0 and that's it. If it goes down, the cluster can't pull anything when using the mutating webhook.

Describe the solution you'd like

HA for proxying only is easy:

sequenceDiagram
    participant cri-o
    participant trow-2
    participant trow-0
    participant trow-1

    cri-o ->> trow-2: request manifest
    trow-2 -->> trow-0: request manifest
    trow-0 -->> trow-2: not found
    trow-2 -->> trow-1: request manifest
    trow-1 -->> trow-2: found, here's the manifest and blobs
    trow-2 ->> cri-o: here's the manifest

Loading

Now, how to handle uploads and deletions ? 🤔

1. Add blob auto clean
2. Disable deletion of blobs (as permitted in the spec)
3. Every request is also sent to the other nodes
4. At first start, sync state with the other running replicas, if no replicas found, assume state is up to date (possible data loss)

Describe alternatives you've considered

Storing images on blob storage. It removes the performance benefit of being able to run Trow on nvme storage.