Implement stackdriver logging

[gregw]

Implement stackdriver logging as described in the jetty-runtime design document.

[gregw]

work in progress will be in branch https://github.com/GoogleCloudPlatform/jetty-runtime/tree/master-68

[gregw]

@meltsufin currently logging is split over openjdk-runtime (json formatters) and jetty-runtime.

Ultimately, I can see the results of this issue being split over:

• pull request to google cloud java LoggingHandler
• an non-jetty specific extension of LoggingHandler that could replace the JSON stuff (no longer needed) in openjdk-runtime and adds traceid support.
• jetty specific usage and configuration of the LoggingHandler which belongs in jetty-runtime

I propose that for simplicity, I develop all the code here in jetty-runtime and then once we are happy with how it is working, we then distribute it out to the locations above (or as then appropriate) before we merge to master/cloud/open-jdk

OK?

[meltsufin]

@gregw sounds good to develop the code here first.

[gregw]

@meltsufin progress is slow. I'm a bit concerned that the JUL to google-cloud-java approach may be a bit flawed. It is working when the log level is INFO, but once you make the level FINE, then the implementation of google-cloud-java using things like netty and HttpURLConnection that themselves log to java util logging, so infinite recursion and stack overflows result!

See googleapis/google-cloud-java#1386

We could go straight from the jetty log to google-cloud-logging, but anything that uses JUL or JCL (eg parts of the datastore used for sessions) will be lost (or end up on the console).

It may be that the google-cloud-logging developers have a solution to prevent the recursion. But I think it really needs a very simple HTTP client that does not do logging to be used to talk to the stackdrive service. Perhaps the google-cloud-java library can be used to do the auth first.

[meltsufin]

@gregw Can you use a thread local flag to signal to the handler that it's being recursively called and break the infinite recursion?

[gregw]

@meltsufin we should be able to use a thread local, although care will be needed to ensure that we don't hammer on that for every log line. I've not done that yet, as it is really something that needs to be solved at the google-cloud-java level, but may do so today just to make some progress.

I've been thinking over alternate solutions and nothing really is as satisfactory as having a JUL handler receiving all container logs from all sources and sending them to stackdriver

[gregw]

The LoggingHandler does have protection for looping in it's implementation. It maintains a list of masked loggers and removes the stackdriver logger from them. However, for some reason it is not working for my testing. I am getting assistance to analyse.

Good news is that the logs are appearing in the console... just an infinite number of them :)

[meltsufin]

@gregw It actually makes sense that the LoggingHandler would have protection against infinite recursion, but I guess the way we're trying to redirect logs is breaking it. Maybe we're somehow capturing the logs it skips and directng them back to it? Anyway, thanks for the update.