Log console output to allocation logs

[Freddo3000]

This pull request implements some basic console logging by having the VM open a UNIX socket which is then read from and output to the nomad allocation log FIFO.

[chrisroberts]

Hi @Freddo3000! Sorry there was a bit of a delay getting to this PR. We were still wrapping up some other things and I needed to take some time to sit and play around with this a bit.

Thanks so much for this PR and getting things moving on task logs. After getting your changeset rebased I spun it up and it worked just as described 🎉 One side effect I encountered was that using virsh console X stopped working (though could be worked around with providing devname to use the console device). Getting the output from the serial device was great but once the system was booted, the logs would show nothing more than the final login prompt.

So I started to wonder if we could instead get cloud-init to point its logs at the serial device instead, and then consume that and stream it to Nomad. The cloud-init configuration allows for configuring multiple loggers, so I took a swing at wiring that in and found it worked rather well, which was great. While doing that I also came across the output key, which looked like it might be just the thing for logging the output of the commands defined for the task.

The output key supports splitting the output to send stdout and stderr to different locations. If two serial devices were created in the VM and bound to host sockets, we could have one for stdout and one for stderr. After adding in the extra serial device and getting the logging configuration all updated, the results turned out pretty well.

Using this example job that spins up a simple http server using python, this is what I get in the UI:

with request logs going to stderr:

which is great. Now we are getting output for the life of the command and the task is behaving more like tasks from other drivers. I found that being able to see the logs from cloud-init was really useful, but likely not something that would always want to be seen. Instead of always sending cloud-init logs, tasks can optionally enable them in the task config:

config {
  logging {
    cloudinit = true
    level     = "debug"
  }

  ...
}

If cloudinit logging is enabled, then an extra logger is added to cloudinit to write to the "stdout" serial device and we get something like this:

Would be really happy to hear any feedback you have about these changes, and thank you again for this PR!

[Freddo3000]

Took a brief look at it and it looks good to me, albeit a bit more complex than my initial implementation. Only thought I have is whether the disable toggle is needed for the logging block seeing as there's a task-wide option to disable logs already: https://developer.hashicorp.com/nomad/docs/job-specification/logs though perhaps this allows us to skip some threads?