dockerhub rate limit - need for a local docker repo server

[parmentelat]

as per
https://docs.docker.com/docker-hub/download-rate-limit/

there is a limit on the number of pulls that can be triggered in any 6 hour time slot

in order to work around that, we need to setup a local dockerhub cache server

NOTES

• faraday seems like the right place to put this server, as it mostly targets the FIT nodes (and the sopnodes will have the option to use it too)
• Théo had already put something in place, and on faraday too, except that it was poorly (not at all?) documented

[dsaucez]

actually we also need a local repo for sopnode, for now, I deploy one every time I need it in sopnode (see ansible playbook for it https://github.com/dsaucez/SLICES/blob/main/sopnode/ansible/registry.yaml) It sets up the registry via k8s and install CA correctly such that nodes in the cluster can trust the registry. Beware that in some environments it is mandatory to have signed registries

[parmentelat]

On 19 Sep 2022, at 15:59, Damien Saucez ***@***.***> wrote: actually we also need a local repo for sopnode, for now, I deploy one every time I need it in sopnode (see ansible playbook for it https://github.com/dsaucez/SLICES/blob/main/sopnode/ansible/registry.yaml) It sets up the registry via k8s and install CA correctly such that nodes in the cluster can trust the registry. Beware that in some environments it is mandatory to have signed registries

thanks for outlining the fact that it needs to be signed I don’t feel like creating a new repo over and over again is going to cut it, because we want to avoid multiple pulls, so some stability seems required but to be honest that approach had not occurred to me, so it might work after all I’ll first try to unearth whatever Théo had done before I decide how to deal with this thanks for the input in any case !

[parmentelat]

an update on this matter; all this relates to faraday.inria.fr

• the previous duo of registries that had been setup by Téo are now dismantled because:
  • they were running on docker, which we want to eradicate
  • they were running http only, and no https
  • they were undocumented, and in particular it was unclear which part of the filesystem was being used
• the docker rpms have been uninstalled altogether
• and replaced by podman (together with podman-docker, so that one can still use the docker command)
• an instance of the docker registry is running on
  https://sopnode-registry.inria.fr/
  and is configured with a proper certificate

[parmentelat]

this means that one can do, for example, from a sopnode wired node:

see also https://docs.docker.com/registry/spec/api/

sopnode-w2 ~ (master) #
curl https://sopnode-registry.inria.fr/v2/
{}     # newline added artificially here

sopnode-w2 ~ (master) #
podman image tag docker.io/library/ubuntu sopnode-registry.inria.fr/ubuntu

sopnode-w2 ~ (master) #
podman push sopnode-registry.inria.fr/ubuntu
Getting image source signatures
Copying blob 7f5cbd8cc787 done
Copying config 2dc39ba059 done
Writing manifest to image destination
Storing signatures

sopnode-w2 ~ (master) #
curl https://sopnode-registry.inria.fr/v2/_catalog
{"repositories":["ubuntu"]}

sopnode-w2 ~ (master) #
curl https://sopnode-registry.inria.fr/v2/ubuntu/tags/list
{"name":"ubuntu","tags":["latest"]}

and then from another node

sopnode-w3 ~ (master) #
podman run -ti --rm sopnode-registry.inria.fr/ubuntu
Trying to pull sopnode-registry.inria.fr/ubuntu:latest...
Getting image source signatures
Copying blob e90119119306 done
Copying config 2dc39ba059 done
Writing manifest to image destination
Storing signatures
root@11c8d1f9142e:/# 

[parmentelat]

the material for this feature is located here:

https://github.com/fit-r2lab/r2lab-embedded/tree/master/docker-registry

[parmentelat]

HOWEVER

there is still a very strong limitation currently though (as in: it's not usable yet)
as FIT nodes are not able to connect to this registry

[root@fit01 ~]# curl https://sopnode-registry.inria.fr/v2/
... hangs stupidly there and finally issues a timeout

so of course everything points at a firewalling issue, to be investigated...

[parmentelat]

observations

when running curl https://sopnode-registry.inria.fr/v2/ from either place

from sopnode

when a request comes from sopnode-w2 (and the registry server answers fine)

• packet shows up on interface internet (from sopnode-w2 to faraday)
• it gets forwarded on the cni-podman0 interface (from sopnode-w2 to e.g. 10.88.0.6)
• and then to the veth virtual interface attached to the container (e.g. veth4a7a7410, changes each time of course) with same src and dst

from FIT

when a request comes from a FIT node (and the registry server does not answer)

• packet shows up on interface control (from 192.168.3.x to faraday)
• but it then gets forwarded to the internet interface !
  as src=faraday.inria.fr and dst=10.88.0.6
• instead of the expected behaviour which is to send this to cni-podman0 - hence the failure

[parmentelat]

temporarily reverted to using docker; same behaviour, except for the private IP range used, which is 172.17.0.2

this behaviour strongly seems to relate to the fact that we originate traffic from an otherwise NAT'ed address (fitnn) and towards an address that is local to faraday

I am suspecting the dst-type LOCAL filter used by CNI in the iptables to be part of the issue here

however I can't explain why/how the packet with the private IP ends up on internet instead of cni-podman0 given the routes

root@faraday ~ (master) # ip route | egrep 'podman|docker'
10.88.0.0/16 dev cni-podman0 proto kernel scope link src 10.88.0.1 linkdown
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1