Add leniency and type coercion for prom datasource options

[jcolladokuri]

Summary

This PR adds support for leniency on prom datasource settings types and type coercion where possible.

Before, jsonData was treated and unmarshalled as generic map[string]any this allowed for wrongly typed values to be part of datasource configuration (i.e. "true" instead of true

Starting with #220, strict typing was enforced for all datasource config fields. For any existing customers that are using provisioning, this effectively broke their datasources.

This PR fixes that by adding LenientTypes, type coercion when possible, and if this fails, the field gets dropped instead of crashing the whole datasource. Any fields that were previously strict, remain strict (i.e. httpMethod)

Before (blocking on go unmarshalling):

After (letting the datasource health check go through):

Detailed summary

• Adds lenient types, all of the below type coercions are attempted after trying to unmarshal from the expected type. If the type coercions still fail, the the field gets dropped without erroring the user out:
  • LenientBool: tries type coercion from string and float64
  • LenientString: tries type coercion from float64 and bool
  • LenientFloat64: tries type coercion from string
  • LenientInt64: tries type coersion from float64 and string
  • LenientExemplarTraceIDDestinations: no type coercion
• Adds logging for all coerced and dropped fields, ideally, we should move towards strict typing, this will help give us an idea of how much this is still being needed and used by customers.

Future work

Right now this will help unblock customers who had provisioned datasources working under the previous generic jsonData. However, as a follow up, we should expose to customers that specific fields were coerced or dropped so that they could potentially correct it and we move towards a strict schema completely.

Testing

Created a prom provisioned datasource with the wrong types for different fields and ensure that the datasource still loaded, and had the right value where coercion was possible. (#311)

[iwysiu]

I have a couple questions but overall looks good!

[iwysiu]

can you pipe in the prometheus logger here so it has a little more context?

[jcolladokuri]

is there anything specific from the additional context that you think would be helpful to add here? piping the prom logger has some trade-offs. Because these unmarshalling functions are called directly from encoding/json looking would have to live outside of the UnmarshalJSON functions and moved into ParsePromOptions, it requires additional parsing and we'd lose information about which path the specific value took (i.e. from int64 to string).

Would love to hear your thoughts on this

[iwysiu]

I think that if we drop an invalid int64 here we actually set lenientInt64 to 0 (since its allocated and we don't return an unmarshaling error)

[jcolladokuri]

Great callout, I have fixed this! A few things:

• unrelated to your comment but got rid of lenient int since it was only used by one field and the fronted already sets seriesLimit to a number anyway so float64 more closely matches a double
• For your call out, I added a flag for whether a value was actually readable or not ("10" vs "ten") and should be dropped. This way we are able to distinguish actual 0 values from when it should be nil.

Let me know what you think!