Prometheus monitoring mixin

[thinkmassive]

The sample dashboards are great, and they seem to be used as a starting point by the majority of lndmon users. Unfortunately the fully rendered dashboards are difficult to iterate on, especially in terms of contributing changes back to this repo.

One solution could be to package the dashboards and rules+alerts as a Prometheus Monitoring Mixin, which is designed to be reusable and extendable. Grafana dashboards can be generated by grafonnet for better modularity.

[thinkmassive]

I started implementing this here. Currently it uses each existing sample dashboard as a json blob, so the next step is to start parameterizing the dashboards.

Now I'm wondering where this code should live... in this repo, possibly with the output overwriting the existing sample dashboards? If so, should I add build steps to the top-level Makefile and/or github actions?

This isn't blocking development, but any guidance would be appreciated at your convenience. @guggero @Roasbeef

[guggero]

I haven't looked into mixins that much yet, but I agree that having huge JSON blobs for the dashboards probably isn't easily maintainable. So I'd be curious to see how a parametrized jsonnet dashboard would look like.

In order to encourage users to extend the parametrized files, I think we should remove the "raw" example dashboards once we converted them into jsonnet and perhaps add the rendered dashboards as a release artifact. Or we could keep the now generated example files in the code but add a build step that make sure there is no diff when generating/overwriting them in a PR (similar to how we do it with the gRPC generated files in lnd).