Logging using the Google Cloud Logging API

[meltsufin]

The Google Cloud Logging API or the stdout and stderr should be used to send logs to Stackdriver Logging.

[gregw]

So using the LogEntry.builder means that we can remove the JSON formatting classes?

I assume that there will be no local log file, unless configured in the users logging.properties and all logs will be consumed only by this API by default?

Should we also remove the request log file?

[joakime]

A bit concerned about the com.google.cloud.logging.HttpRequest object and what it shows in its testcase.

The Google Cloud Logging is java.util.logging based, so that means the the minimum server logging support level will be java.util.logging, not the Jetty default of StdErrLog

The setup for Google Cloud Logging API seems to be based on setting up the com.google.cloud.logging.LoggingHandler with the correct authentication and whatnot. (unless its auto initialized by the Google Cloud services layer)

Possible setup options:

1. Use slf4j-api -> slf4j-jdk14 on the server side,
   • Jetty uses its Slf4jLog implementation
2. Use JavaUtilLog as the Jetty logger implementation.

In either setup, we would need to:

• bar server org.slf4j from being used by the application.
• allow java.util.logging to be used by the application, but prevent the application from replacing, resetting, or removing the root handlers (otherwise this will break the Google Cloud API use)

[joakime]

We need to make sure that the Jetty base image does not use etc/jetty-logging.xml as that is not compatible with both the java.util.logging layer, and the Google Cloud Logging API.

[gregw]

Please consider GoogleCloudPlatform/appengine-java-vm-runtime#282 in new logging design

[meltsufin]

Would it make sense to start out by making sure that the Jetty server log is directed to standard error? Standard out/error should automatically be picked up by Stackdriver Logging.

[joakime]

we would still need java.util.logging, as without it, nothing from the google layers would ever show up in the stackdriver logs without it.

the default jetty StdErrLog is multi-line, which the Stackdriver has had problems with in the past, hence all of the effort to make logging more stable with the json format, etc..

to fix StdErrLog would mean a fork of StdErrLog to something like GoogleStackdriverSystemErrLog just to conform to the limits of the Stackdriver standard out/error support

that effort would be better spent in just implementing the java.util.logging layer properly to Google Cloud Logging.

[meltsufin]

we would still need java.util.logging, as without it, nothing from the google layers would ever show up in the stackdriver logs without it.

What do you have in mind when you say "google layers"?

the default jetty StdErrLog is multi-line, which the Stackdriver has had problems with in the past, hence all of the effort to make logging more stable with the json format, etc..

There's work going on to automatically detect multi-line java stack traces in the logs. Is there anything else that's potentially multi-line in the Jetty server logs?

to fix StdErrLog would mean a fork of StdErrLog to something like GoogleStackdriverSystemErrLog just to conform to the limits of the Stackdriver standard out/error support

What limits are you referring to?

that effort would be better spent in just implementing the java.util.logging layer properly to Google Cloud Logging.

There are java.util.logging handlers in the Logging API. Beyond those handlers, what would our logging layer need to do?

[joakime]

Multiline output it very common, and not just for stack traces.

JSP produces multi-line, JSR365 (javax.websocket) does, most datasources, Jersey, Spring, and even the javax.servlet API does too (example: exceptions during startup / deployment are collected into a multi exception which is a single logging event that reports the main deployment concern, and then lists each deployment failure and possible stacktrace for that failure as well)

This is the limits that we refer to with the naive stackdriver parsing of System.err and System.out.

When I say "google layers" I mean any of the google or gcloud classes that operate on the server (or the webapp) would produce no output if using Jetty's default StdErrLog setup (which outputs only Jetty logging events to System.err)

So, we have top level options, and multiple sub-options.

Option 1: Standard Jetty Logging

This would be Jetty setup for StdErrLog and without etc/jetty-logging.xml.

• Only Jetty logging events at INFO level are produced on System.err.
• No other output is produced.
• Nothing on System.out
• Nothing from java.util.logging
• Application logging does not exist and is not configured
• No JSP logging events
• No 3rd party events
• This setup is known to produce horrible results in the console loggers

Option 2: Jetty uses java.util.logging

Technique A: Jetty uses its JavaUtilLog implementation

• Server side would setup a Handler to output its events somewhere
• Application logging for java.util.logging works
• Application configuration of levels works
• Application configuration of custom j.u.l handlers work
• Google APIs / classes can participate in logging
• JSP logging events supported
• Limited 3rd party support (as we are stuck on java.util.logging only)

Technique B: Jetty uses its Slf4jLog implementation

• Server side would setup a Handler to output its events somewhere
• Application logging for java.util.logging works
• Application configuration of levels works
• Application configuration of custom j.u.l handlers work
• Google APIs / classes can participate in logging
• JSP logging events supported
• Full 3rd party logging support (as we can now support any logging events from anywhere: system, server classloader, jndi, applications, etc..)

In either technique, since we are using java.util.logging now, using the Google Cloud Logging API is attainable, and preferred over the System.out and System.err parsing.

However, as noted before, we still need to be careful to not allow the application to remove/reset/change the Google Cloud Logging Handlers at the java.util.logging root context. (configuring the Google Cloud Logging, such as setting the named logger levels is very much supported)

Using the Google Cloud Logging API also means we have preliminary support for associating the HttpRequest (via the traceid or similar information), something that the Stackdriver support for system.out and system.err has historically never been able to do.

[gregw]

For the sake of clarity, I think I should prepare a PR for Joakim's Option 1: all the logging in jetty stripped back to the minimal: no request log and stderr log for the container. I'll add a simple test webapp that generates errors and/or causes errors within the container and we can see how the logging looks in the cloud console.

We can then see exactly what needs to be fixed and that will guide us in choosing between status quo, Technique A, Technique B, logging API or something else.

[meltsufin]

Yes, sounds good!