Support Swarm Services

[ehazlett]

This will add support for Swarm services.

[jcmcote]

I'm running Interlock 1.4 with docker 17.05.0-ce and nginx 1.11.10. Both Interlock and nginx are running on the swarm manager.

I configure my services using docker compose file version 3 and deploy them with docker stack deploy.

I successfully configured Interlock so that any container started on the manager node gets added to the nginx configuration and I'm able to access the service via the nginx gateway.

However if the container starts on one of the worker nodes Interlock does not detect it and does not create any routes for it.

Is this a know issue?

Note that when I run
docker -H tcp://192.168.1.100:2375 events
I only see events from manager node not the worker nodes. Expected behaviour?
thanks!

[byrnedo]

Any idea when this support would be added?

[jcmcote]

I have looked at the code for the server.go file

In that file there is a poller function called runPoller. This function uses the docker SDK ContainerList function. However this function only returns containers on the current docker daemon not the containers in the cluster of swarm nodes.

containers, err := s.client.ContainerList(context.Background(), opts)

So I do not understand how Interlock can add routes for services which are deployed on other nodes. Is this functionality not supported ? Do the containers exposed via nginx (the load balancer) need to run on the same host where Interlock is running?

Using Java docker SDK I was able to list the swarm services, find the tasks for these services. Could Interlock use the tasks label instead of using the labels on docker containers like is done in the runPoller function? Would such a patch be welcomed?

Thanks

[ehazlett]

Thanks for the interest!

Yes, you could use the tasks however the user would need to create a service using the --container-label option instead of service labels --label. If you wanted to use the service labels, you would need to get the service info and then use that.

Another side note is that if the service is on it's own network you would need to make sure that the proxy container (nginx, etc) gets attached to the service network otherwise it won't be able to get to the backend.

[jcmcote]

I can either use the service labels or container labels. However I think I will first try with service labels, if present dig down to the task to find their specific IP, then dig down to the containers to find the ports they expose via DockerFile. I don't see any other option to support swarm services which can be deployed on any node. I'll try doing this via the polling mechanism. Agreed. Nginx and the services it proxies need to be in the same overlay network. Cheers Jean-claude

…

On Wed, Aug 16, 2017 at 9:33 AM, Evan Hazlett ***@***.***> wrote: Thanks for the interest! Yes, you could use the tasks however the user would need to create a service using the --container-label option instead of service labels --label. If you wanted to use the service labels, you would need to get the service info and then use that. Another side note is that if the service is on it's own network you would need to make sure that the proxy container (nginx, etc) gets attached to the service network otherwise it won't be able to get to the backend. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJoEwtfEyN0ifgopIpjaE-04HB9q9JCjks5sYu-pgaJpZM4OS_OV> .

[byrnedo]

Just to chime in: not 100% sure what you mean about the polling but AFAIK the managers events are now swarm-wide so one can watch those to discover new services/tasks.

[ehazlett]

I can either use the service labels or container labels. However I think I
will first try with service labels, if present dig down to the task to find
their specific IP, then dig down to the containers to find the ports they
expose via DockerFile.

Be careful as this will have a lot of API calls which will impact service at large scale. FYI, /services and /tasks returns a summary that has most of the info needed and is a lot cheaper.

[jcmcote]

Yes that's the endpoints I wanted to use /services and /tasks.

…

On Wed, Aug 16, 2017 at 11:43 AM, Evan Hazlett ***@***.***> wrote: I can either use the service labels or container labels. However I think I will first try with service labels, if present dig down to the task to find their specific IP, then dig down to the containers to find the ports they expose via DockerFile. Be careful as this will have a lot of API calls which will impact service at large scale. FYI, /services and /tasks returns a summary that has most of the info needed and is a lot cheaper. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJoEwnFwVY_A7F1EIPXyO4OHVgDEC6_cks5sYw47gaJpZM4OS_OV> .

[jcmcote]

ok I was not aware the events now include swarm-wide events. In the version of docker I have I don't see them when I do docker events Is that the command to see them? I've read a few posts describing various options for the commands but I don't know which one was finally implemented..

…

On Wed, Aug 16, 2017 at 11:18 AM, Donal Byrne ***@***.***> wrote: Just to chime in: not 100% sure what you mean about the polling but AFAIK the master events are now swarm-wide so one can watch those to discover new services/tasks. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJoEwnMfPJqsEV8Ka1M1lcypLWdV0ClMks5sYwg8gaJpZM4OS_OV> .

[byrnedo]

Which docker version you running?

…

On Wed, 16 Aug 2017, 18:32 jcmcote, ***@***.***> wrote: ok I was not aware the events now include swarm-wide events. In the version of docker I have I don't see them when I do docker events Is that the command to see them? I've read a few posts describing various options for the commands but I don't know which one was finally implemented.. On Wed, Aug 16, 2017 at 11:18 AM, Donal Byrne ***@***.***> wrote: > Just to chime in: not 100% sure what you mean about the polling but AFAIK > the master events are now swarm-wide so one can watch those to discover new > services/tasks. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#224 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AJoEwnMfPJqsEV8Ka1M1lcypLWdV0ClMks5sYwg8gaJpZM4OS_OV > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFRbq6x5PofXZlAxB2HwjhkgyE3eqqS4ks5sYxl-gaJpZM4OS_OV> .

[jcmcote]

I'm running Docker version 17.06.0-ce, build 02c1d87 When I run *docker events* and run this command docker service create --detach=false --constraint 'node.role == manager' --label "interlock.hostname=test, interlock.domain=local" --replicas 1 --name hello alpine:latest ping docker.com I get these events 2017-08-16T17:48:30.023419627Z service create fyg40a1coqkfmgs6m94th6ncp (name=hello) 2017-08-16T17:48:32.346189802Z image pull alpine:latest@sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe (name=alpine) 2017-08-16T17:48:32.381662707Z container create c069671ff5630f6c9ca328a5cf61d3073d11b3dd32d9215f633f49d70e75c866 ( com.docker.swarm.node.id=lrdb0krdeqxufy3qswi61chw0, com.docker.swarm.service.id=fyg40a1coqkfmgs6m94th6ncp, com.docker.swarm.service.name=hello, com.docker.swarm.task=, com.docker.swarm.task.id=p141f4f4gr233jbpaamne5tb5, com.docker.swarm.task.name=hello.1.p141f4f4gr233jbpaamne5tb5, image=alpine:latest@sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe, name=hello.1.p141f4f4gr233jbpaamne5tb5) 2017-08-16T17:48:32.394780897Z network connect 6ade162d08473c372e1833adf803f6ff43e34b4d879bf811532fed8290eb94c1 (container=c069671ff5630f6c9ca328a5cf61d3073d11b3dd32d9215f633f49d70e75c866, name=bridge, type=bridge) 2017-08-16T17:48:32.533265365Z container start c069671ff5630f6c9ca328a5cf61d3073d11b3dd32d9215f633f49d70e75c866 ( com.docker.swarm.node.id=lrdb0krdeqxufy3qswi61chw0, com.docker.swarm.service.id=fyg40a1coqkfmgs6m94th6ncp, com.docker.swarm.service.name=hello, com.docker.swarm.task=, com.docker.swarm.task.id=p141f4f4gr233jbpaamne5tb5, com.docker.swarm.task.name=hello.1.p141f4f4gr233jbpaamne5tb5, image=alpine:latest@sha256:1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe, name=hello.1.p141f4f4gr233jbpaamne5tb5) However running the service on a worker docker service create --detach=false --constraint 'node.role == worker' --label "interlock.hostname=test, interlock.domain=local" --replicas 1 --name hello alpine:latest ping docker.com I'll only see the service events. Not the container events. 2017-08-16T17:45:36.513901298Z service remove txtsw8cd1jyqcdwb02f5ul4h8 (name=hello) 2017-08-16T17:45:50.251105519Z service create v9e8nihpeocdlcnfpyphow3ln (name=hello) I believe that's why the current implementation of Interlock does not detect these containers and does not create routes from them unless the containers happen to run on the manager. I would like to change that to be service based (so monitoring service events might sufficient to detect changes in the topology) so that nginx can proxy containers (tasks) started across the swarm cluster. jc On Wed, Aug 16, 2017 at 1:03 PM, Donal Byrne <notifications@github.com> wrote:

…

Which docker version you running? On Wed, 16 Aug 2017, 18:32 jcmcote, ***@***.***> wrote: > ok I was not aware the events now include swarm-wide events. In the version > of docker I have I don't see them when I do > > docker events > > Is that the command to see them? I've read a few posts describing various > options for the commands but I don't know which one was finally > implemented.. > > > On Wed, Aug 16, 2017 at 11:18 AM, Donal Byrne ***@***.***> > wrote: > > > Just to chime in: not 100% sure what you mean about the polling but AFAIK > > the master events are now swarm-wide so one can watch those to discover > new > > services/tasks. > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > <#224# issuecomment-322806075 > >, > > or mute the thread > > < > https://github.com/notifications/unsubscribe-auth/ AJoEwnMfPJqsEV8Ka1M1lcypLWdV0ClMks5sYwg8gaJpZM4OS_OV > > > > . > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#224 (comment) >, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ AFRbq6x5PofXZlAxB2HwjhkgyE3eqqS4ks5sYxl-gaJpZM4OS_OV> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJoEwiMdWY6hdP17wEHou5jIlS0JA_SIks5sYyDcgaJpZM4OS_OV> .

[byrnedo]

Sorry, my bad then, I thought that had been implemented.

…

On Wed, 16 Aug 2017, 19:53 jcmcote, ***@***.***> wrote: I'm running Docker version 17.06.0-ce, build 02c1d87 When I run *docker events* and run this command docker service create --detach=false --constraint 'node.role == manager' --label "interlock.hostname=test, interlock.domain=local" --replicas 1 --name hello alpine:latest ping docker.com I get these events 2017-08-16T17:48:30.023419627Z service create fyg40a1coqkfmgs6m94th6ncp (name=hello) 2017-08-16T17:48:32.346189802Z image pull ***@***.*** :1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe (name=alpine) 2017-08-16T17:48:32.381662707Z container create c069671ff5630f6c9ca328a5cf61d3073d11b3dd32d9215f633f49d70e75c866 ( com.docker.swarm.node.id=lrdb0krdeqxufy3qswi61chw0, com.docker.swarm.service.id=fyg40a1coqkfmgs6m94th6ncp, com.docker.swarm.service.name=hello, com.docker.swarm.task=, com.docker.swarm.task.id=p141f4f4gr233jbpaamne5tb5, com.docker.swarm.task.name=hello.1.p141f4f4gr233jbpaamne5tb5, ***@***.*** :1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe, name=hello.1.p141f4f4gr233jbpaamne5tb5) 2017-08-16T17:48:32.394780897Z network connect 6ade162d08473c372e1833adf803f6ff43e34b4d879bf811532fed8290eb94c1 (container=c069671ff5630f6c9ca328a5cf61d3073d11b3dd32d9215f633f49d70e75c866, name=bridge, type=bridge) 2017-08-16T17:48:32.533265365Z container start c069671ff5630f6c9ca328a5cf61d3073d11b3dd32d9215f633f49d70e75c866 ( com.docker.swarm.node.id=lrdb0krdeqxufy3qswi61chw0, com.docker.swarm.service.id=fyg40a1coqkfmgs6m94th6ncp, com.docker.swarm.service.name=hello, com.docker.swarm.task=, com.docker.swarm.task.id=p141f4f4gr233jbpaamne5tb5, com.docker.swarm.task.name=hello.1.p141f4f4gr233jbpaamne5tb5, ***@***.*** :1072e499f3f655a032e88542330cf75b02e7bdf673278f701d7ba61629ee3ebe, name=hello.1.p141f4f4gr233jbpaamne5tb5) However running the service on a worker docker service create --detach=false --constraint 'node.role == worker' --label "interlock.hostname=test, interlock.domain=local" --replicas 1 --name hello alpine:latest ping docker.com I'll only see the service events. Not the container events. 2017-08-16T17:45:36.513901298Z service remove txtsw8cd1jyqcdwb02f5ul4h8 (name=hello) 2017-08-16T17:45:50.251105519Z service create v9e8nihpeocdlcnfpyphow3ln (name=hello) I believe that's why the current implementation of Interlock does not detect these containers and does not create routes from them unless the containers happen to run on the manager. I would like to change that to be service based (so monitoring service events might sufficient to detect changes in the topology) so that nginx can proxy containers (tasks) started across the swarm cluster. jc On Wed, Aug 16, 2017 at 1:03 PM, Donal Byrne ***@***.***> wrote: > Which docker version you running? > > On Wed, 16 Aug 2017, 18:32 jcmcote, ***@***.***> wrote: > > > ok I was not aware the events now include swarm-wide events. In the > version > > of docker I have I don't see them when I do > > > > docker events > > > > Is that the command to see them? I've read a few posts describing various > > options for the commands but I don't know which one was finally > > implemented.. > > > > > > On Wed, Aug 16, 2017 at 11:18 AM, Donal Byrne < ***@***.***> > > wrote: > > > > > Just to chime in: not 100% sure what you mean about the polling but > AFAIK > > > the master events are now swarm-wide so one can watch those to discover > > new > > > services/tasks. > > > > > > — > > > You are receiving this because you commented. > > > Reply to this email directly, view it on GitHub > > > <#224# > issuecomment-322806075 > > >, > > > or mute the thread > > > < > > https://github.com/notifications/unsubscribe-auth/ > AJoEwnMfPJqsEV8Ka1M1lcypLWdV0ClMks5sYwg8gaJpZM4OS_OV > > > > > > . > > > > > > > — > > You are receiving this because you commented. > > Reply to this email directly, view it on GitHub > > < #224 (comment) > >, > > or mute the thread > > <https://github.com/notifications/unsubscribe-auth/ > AFRbq6x5PofXZlAxB2HwjhkgyE3eqqS4ks5sYxl-gaJpZM4OS_OV> > > > . > > > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#224 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/AJoEwiMdWY6hdP17wEHou5jIlS0JA_SIks5sYyDcgaJpZM4OS_OV > > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFRbq6-MEAt-CDnZmPF236EONGy9u_UOks5sYyyqgaJpZM4OS_OV> .