Enable systemd logs only

[sboeuf]

We should be able to specify a debug mode --syslog-only so that we don't get any traces/logs printed on stderr/stdout. This is very useful if we want to debug the entire chain (runtime+shim+proxy) without getting some traces on the standards outputs supposed to receive the output from the container.

[sboeuf]

@amshinde do you want to implement it ? Or I can assign it to myself.

[dlespiau]

I'd like all logs to go to the proxy at some point so we can have a global view from there: clearcontainers/proxy#35

I think I'll have time to implement that next week.

[sboeuf]

@dlespiau are you sure we want those logs in the proxy ? I think that you will potentially miss all the logs coming from the shim before it connects to the proxy.
I think that a more global approach for clear containers would be to log everything to the same file with the same expected format and timestamps. That way, it could be very nice to see all the interactions in one log. What do you think about this ?

[dlespiau]

Using a file seems worse to me:

• In both cases, one needs to open a fd, either for a file or a domain socket. Anything that happens before that cannot be logged. File vs Socket doesn't change anything.
• Write to a file from several processes is really not fun and there will be a need to synchronise cross process writes somehow with somewhat heavy locks.

Note that logging to a service is also quite common these days (http://zipkin.io/, http://opentracing.io/, ...) I feel that having such a solution is actually a too much for us, a deployment hurdle we can't afford. We can easily have something similar, all logging through to the proxy. That should allow us to do interesting things as well, things like live querying the logs from the debug HTTP interface (please give me the logs for container with ID 234871238412758 only).

[sboeuf]

You're right, the synchronization seems quite difficult for a single file. And the proxy seems the most appropriate component to centralize all logs as it is supposed to never die. But I don't really understand the way you plan to get the traces from the runtime starting qemu for example. This is done before we connect the proxy, therefore are you planning to connect earlier to the proxy to solve this log issue ?

[dlespiau]

You can connect as soon as you want to proxy and send the log commands from the runtime (a new command I'll be doing this week).

There's still a gap though: this isn't solving how neither the qemu messages nor the early boot kernel messages are going to be collected (proxy connect to the qemu console at RegisterVM time but qemu is already starting(. I haven't looked at what would be a feasible way to solve that.

[amshinde]

@dlespiau I think #29 addresses this.
We still log to stderr in case of error logs, which I think should be retained. The shim usually exits in case of unrecoverable errors or errors due to misbehaving proxy, it is useful to log these to stderr.

[sboeuf]

Yes I think this change solve this issue. And I agree we can still print error message since we don't expect the system to be usable after that.