[RFC] [Draft] Normalized, actionable health-check errors for Grafana datasource plugins

[yesoreyeram]

Status: Draft / for review · Authors: @yesoreyeram · Owning team: group-datasources · Last updated: 2026-06-30

Home	github.com/grafana/dsconfig (new health module) + family libs
Reference implementation	#111 (working module + tests; API in Appendix B reflects it)
Related	grafana-plugin-sdk-go (ErrorSource), sqlds, grafana-aws-sdk, grafana-azure-sdk-go, grafana-google-sdk-go

---

1. Summary

Datasource health checks (the CheckHealth behind Save & test) return inconsistent, often raw or opaque error messages across the ~50 datasource plugins.

This RFC proposes a small shared library — dsconfig/health — that normalizes every health-check failure into a consistent, safe, machine-classified, and actionable result:

• a stable, machine-readable errorCode,
• a clean, secret-free human message,
• structured remediation that is context-aware across many dimensions (not just auth type), and
• a redacted verbose detail for support.

The core module depends only on grafana-plugin-sdk-go, so HTTP-only plugins incur no SDK bloat. Most plugins adopt it via a dependency bump, because the logic plugs into the shared CheckHealth paths already provided by sqlds and grafana-aws-sdk.

2. Motivation & background

CheckHealth is the first real interaction during onboarding; a confusing failure here is a major drop-off point and a top support driver. Problems today:

• Inconsistent shape — some plugins return err.Error() raw, some fmt.Sprintf a prefix, some a hardcoded generic string. No common structure across repos.
• Unsafe content — raw pass-through can leak hosts, DSNs, and tokens into a UI/log string.
• Unactionable — messages rarely say what to fix or where, and almost never link to docs.
• Mis-attributed — plugin-caused vs downstream-caused failures aren't consistently distinguished, distorting error-source SLOs.
• Not machine-readable — no stable code to drive docs, i18n, dashboards, or support tooling.

3. Goals / Non-goals

3.1 Goals

• G1 — One consistent, localizable health-error message shape across all datasource plugins.
• G2 — A stable, machine-readable errorCode taxonomy for docs, i18n, and telemetry.
• G3 — Context-aware remediation (datasource, auth method, deployment, provider sub-code, TLS kind, …).
• G4 — Redaction of secrets before anything is surfaced.
• G5 — Correct ErrorSource (plugin vs downstream) attribution.
• G6 — Adoption with minimal per-plugin change and no dependency bloat.
• G7 — Scales to 50+ plugins and hundreds of scenarios without a central bottleneck.

3.2 Non-goals

• N1 — Changing the SDK CheckHealth contract or the Save & test UX flow.
• N2 — Backend localization (the frontend maps errorCode → localized copy).
• N3 — Replacing per-plugin connection logic; only the error shaping changes.
• N4 — Covering query-time errors (a separate concern, though the taxonomy is reusable there later).

4. Requirements

4.1 Functional

• FR1 — Map any health-check error to exactly one Code from a closed taxonomy.
• FR2 — Produce a one-line, safe human Message and a structured JSONDetails.
• FR3 — Resolve remediation from a rich, multi-dimensional Diagnosis (not just auth type).
• FR4 — Let families/plugins register classifiers and remediation rules without editing the core.
• FR5 — Redact secrets from any surfaced raw error.
• FR6 — Set errorSource (plugin/downstream) per category.
• FR7 — Provide a success constructor (OK).

4.2 Non-functional

• NFR1 — Core module depends only on grafana-plugin-sdk-go (no SQL/AWS/Azure/GCP SDKs).
• NFR2 — Registration is O(1) at init; resolution is O(rules) with small N and no allocation hotspots.
• NFR3 — JSONDetails is a stable, additively-versioned contract.
• NFR4 — Registries are thread-safe.
• NFR5 — Zero behavior change for un-migrated plugins (opt-in per repo).

5. Current-state audit

Gathered via GitHub code search (fragment-based, 100/page → high-confidence approximations). Phase 0 finalizes these numbers with full file access.

Connection families (where normalization plugs in):

Family	Mechanism	~count	Shared health entry point
SQL via sqlds	database/sql behind sqlds	~8	SQLDatasource.CheckHealth (sqlds)
SQL via core sqleng	lib/pq, mysql, mssql	3	per-repo sqleng/handler_checkhealth.go
AWS SDK	grafana-aws-sdk / aws-sdk-go	~10	AsyncAWSDatasource.CheckHealth (grafana-aws-sdk)
Azure SDK	grafana-azure-sdk-go	~5	per-repo
GCP SDK	cloud.google.com/go, google-sdk	~3	per-repo
Direct HTTP	sdk httpclient / net/http	~20	per-repo (httpclient middleware already classifies source)
Bespoke protocol	e.g. MQTT	1	per-repo

Headline: ~30 plugins use an SDK/driver; ~20 use a direct HTTP connection.

Current error-message patterns (~25 sampled): raw err.Error() pass-through ~9; fmt.Sprintf + prefix ~4; hardcoded generic ~8; mixed ~6.

Reusable primitives: grafana-plugin-sdk-go (backend.ErrorSource, ErrorSourceFromHTTPStatus, httpclient ErrorSourceMiddleware) classifies source but not the message; sqlds has a shared CheckHealth + HealthChecker; grafana-aws-sdk has a shared AsyncAWSDatasource.CheckHealth.

6. Detailed design

6.1 Concepts / glossary

• Code — stable, coarse error category (the versioned public vocabulary).
• Diagnosis — Code + sub-signals (ProviderCode, HTTPStatus, TLSKind, offending Field) + Context.
• Classifier — maps a raw error into a Diagnosis. Families that own an SDK register them; a generic classifier covers net/TLS/timeout.
• Context — environment dimensions injected at the call site (datasource type, auth type, deployment, PDC, docs base URL, vars).
• Remediation — structured guidance: summary, steps[], docsUrl, fields[].
• Rule / Match — declarative (conditions → Remediation); conditions are any subset of dimensions (empty = wildcard); resolved by specificity.
• ErrorSource — SDK concept (plugin vs downstream) carried through for attribution.

6.2 Error-code taxonomy

INVALID_CONFIGURATION (plugin), AUTHENTICATION_FAILED, PERMISSION_DENIED, HOST_UNREACHABLE, CONNECTION_TIMEOUT, TLS_ERROR, NOT_FOUND, RATE_LIMITED, QUOTA_EXCEEDED, UPSTREAM_ERROR, UNSUPPORTED_VERSION, QUERY_VALIDATION_FAILED, UNEXPECTED_RESPONSE (downstream), UNKNOWN (plugin).

UNEXPECTED_RESPONSE is the fallback for "the server answered, but not with the format we expected" — an HTML error/login page where JSON was expected, or a JSON error body we couldn't interpret (see §6.4a). It is used only when classification can't refine to a more specific code.

Each code maps to a canonical headline + generic fallback remediation + default ErrorSource. Codes are additive and never repurposed.

6.3 Diagnosis dimensions (why remediation isn't only "auth type")

error category · provider sub-code (AADSTS / AWS error code / SQLSTATE / HTTP status) · datasource family · auth method · deployment (cloud/enterprise/oss) · network path (direct/PDC/proxy) · TLS sub-kind (unknown-CA/hostname/expired/client-cert) · offending config field · permission scope · server version/capability · region/cloud variant · transient-vs-persistent.

6.4 Classification pipeline

Priority, first match wins:

1. explicit Error tag →
2. registered family classifiers (set Code + sub-signals) →
3. generic Go inspection (errors.Is/As on context.DeadlineExceeded, net.DNSError, net.OpError, x509/tls) →
4. HTTP-status helper →
5. UNKNOWN.

6.4a Interpreting upstream response bodies

A large class of real-world failures isn't a transport error at all — the server answered, but with a body that isn't the JSON the plugin expected. Two recurring cases: an HTML page (proxy/LB/WAF/SSO interception, or a URL pointing at a web root) and a JSON error envelope whose shape varies per provider. Left raw, both surface as opaque parse errors like invalid character '<' looking for beginning of value.

Guiding principle — classify, don't echo. The upstream body is used to classify (pick Code + ProviderCode) and is recorded only in the redacted verbose. The human Message is always built from catalog/rule copy keyed by Code/ProviderCode, so output stays stable regardless of upstream chaos. The body is treated as untrusted: parsing is size- and depth-bounded and must degrade to a fallback code rather than crash the health check.

HTML responses. HTML is a symptom, not a category — it almost always means something between Grafana and the API answered. A response-aware helper sniffs Content-Type: text/html or a body starting with <!doctype/<html/<?xml and refines by status + context:

Observed	Likely cause	Code
HTML + 502/503/504	gateway / LB / upstream down	UPSTREAM_ERROR (or HOST_UNREACHABLE)
HTML + 401, or 200 + HTML login/SSO page	auth proxy / SSO intercept (common with PDC/proxy)	AUTHENTICATION_FAILED
HTML + 403 (WAF / block page)	firewall / WAF	PERMISSION_DENIED
HTML + 200, wrong path / web root	URL points at a web server, not the API	INVALID_CONFIGURATION (highlight URL field)
HTML, status genuinely ambiguous	unknown interceptor	UNEXPECTED_RESPONSE

The 200-OK-HTML + networkPath = pdc\|proxy combination is special-cased toward AUTHENTICATION_FAILED — it's the silent SSO-redirect trap in enterprise setups, where a JSON parse error would be especially misleading.

Inconsistent JSON error envelopes. Every provider shapes errors differently, so parsing is delegated to provider classifiers (ADR-002/007): each family extracts its own provider code authoritatively (Prometheus errorType, Elastic error.type, AWS smithy Code, Azure AADSTS#, JSON:API errors[].status, GraphQL extensions.code, SQLSTATE, …). ProviderCode — a stable machine code, not the free-text message — is the join key for remediation.

For the long tail / not-yet-classified plugins, a best-effort generic extractor walks common envelopes (case-insensitive, defensively): top-level error (string or object), message, error_description, detail, reason, title; arrays errors[0].{detail,message,reason} and error.root_cause[0].reason; codes code, errorType, error.type, extensions.code, status. It yields a hint for verbose and an optional ProviderCode; if nothing parses, classification lands on UNEXPECTED_RESPONSE rather than dumping raw JSON.

Telemetry feedback loop (see §10). When the generic path fires (no provider classifier matched), emit a sampled, redacted record. That backlog drives which provider classifiers to author next — turning upstream inconsistency into a measurable worklist.

6.4b Timeouts, cancellation, and clock skew

A single CONNECTION_TIMEOUT is too coarse — the same word covers failures with different fixes. Classifiers therefore set a TimeoutKind sub-signal:

TimeoutKind	Typical cause	Notes
dial	firewall silently dropping packets (SYN blackhole), wrong host/port	Often better framed as HOST_UNREACHABLE; remediation points at port/security-group/firewall.
read	server reachable but slow (heavy query, undersized instance)	Stays CONNECTION_TIMEOUT; remediation points at server load / increasing the timeout.
deadline	the CheckHealth gRPC deadline elapsed before the attempt finished	Plugin should respect the inbound ctx deadline and return promptly rather than hang the UI.

Because the backend usually can't tell a slow server from a silently-dropped connection, timeout remediation should name both possibilities.

Cancellation is not a timeout. context.Canceled means Grafana or the user aborted the check (e.g. navigated away); it is distinguished from context.DeadlineExceeded and returned as a benign, non-error result (HealthStatusUnknown) — never surfaced as a scary failure, never recorded as a span error. Only DeadlineExceeded maps to CONNECTION_TIMEOUT.

Error-source nuance. Timeouts default to downstream, but a too-aggressive plugin-side timeout is plugin-caused; the mapping is overridable per case.

Clock skew is a sneaky cross-cutting cause that today scatters across TLS_ERROR ("certificate not yet valid/expired") and AUTHENTICATION_FAILED (AWS RequestTimeTooSkewed / "Signature expired", JWT nbf/exp). It is recognized as a signal and yields a dedicated "check the host's clock / NTP" remediation rather than generic TLS/auth copy.

6.5 Remediation resolution

A list of Rule{When Match; Give Remediation} registered as plain Go literals. The resolver scores matching rules by specificity (more constrained dimensions = higher weight; ProviderCode/TLSKind weigh most), picks the highest, and falls back to the catalog's generic remediation. A RemediationProvider func is the escape hatch for logic-heavy cases. Rules live in the importing binary, so a family's rules are naturally scoped to that family's plugins.

6.6 Output contract

• Message — "<DS name>: <headline> <remediation summary>"; never a raw DSN/host/token.
• JSONDetails (fixed shape):

  {
    "errorCode": "...",
    "providerCode": "...",
    "errorSource": "plugin|downstream",
    "correlationId": "a1b2c3",
    "remediation": { "summary": "...", "steps": ["..."], "docsUrl": "...", "fields": ["..."] },
    "verbose": "<redacted raw error> [httpStatus=502 bodyKind=html contentType=text/html]"
  }

• Status vs codes. The SDK CheckHealthResult.Status is a 3-value enum (Ok/Error/Unknown) — not a number — and is the only top-level status (non-goal N1). The machine-readable references are errorCode (taxonomy, drives localized copy + Troubleshoot link) and providerCode (the upstream's own machine code, when known). The schema is fixed — the additional diagnostic metadata (httpStatus, tlsKind/timeoutKind/bodyKind/contentType) is folded into the verbose string rather than added as new fields, so JSONDetails never grows new keys. The offending config field is surfaced through the existing remediation.fields.
• Redaction: verbose is the raw error/body summary passed through a redactor (masks password/token/secret/api-key/authorization and credentials embedded in URLs), with the non-secret diagnostic signals appended; the existing message is preserved as the prefix.
• Correlation ID: a short per-failure reference, surfaced in the UI and stamped on the matching log line (§10). It is the trace ID when a sampled trace is present, else a generated fallback (§10.4), so the reference a user copies resolves to the full distributed trace. It lets support retrieve detail without the UI ever dumping it — and lets verbose stay hidden by default (see §14).

6.7 Frontend consumption

The UI keys off errorCode for localized copy + a "Troubleshoot" doc link, renders remediation.steps, and highlights remediation.fields in the config form. Backwards compatible: older UIs still show Message.

7. Architecture & placement

grafana/dsconfig (multi-module repo)
└── health/                      module github.com/grafana/dsconfig/health
      deps: ONLY grafana-plugin-sdk-go
      taxonomy/catalog · Diagnosis/Context · Classifier registry
      Rule/Match + resolver · redaction · Result(ctx, err, opts...)

Family libs / plugins (own heavy SDKs; register via init()):
  grafana-aws-sdk       -> AWS classifier + rules; calls health.Result in shared CheckHealth
  sqlds                 -> calls health.Result in shared CheckHealth; SQL driver classifiers
  grafana-azure-sdk-go  -> Azure classifier (AADSTS) + rules
  <plugin>              -> plugin-specific rules (e.g. Athena workgroup/S3)

8. Architecture Decision Records

ADR-001 — Dedicated light dsconfig/health module (SDK-only)

• Status: Accepted.
• Context: Core must be importable by every plugin, including HTTP-only ones, without dragging in cloud/SQL SDKs.
• Decision: Put the taxonomy/classifier/remediation core in a new module github.com/grafana/dsconfig/health whose only dependency is grafana-plugin-sdk-go.
• Consequences: Cheap for all importers; discoverable under the datasources-owned repo; adds one small module to maintain in the existing multi-module repo. To keep the graph SDK-only even for tracing, span annotation is done through an injected SpanRecorder rather than a direct OpenTelemetry import (see §10.4); the reference implementation in feat(health): normalized, actionable health-check errors (RFC #110) #111 has exactly one direct dependency — grafana-plugin-sdk-go.

ADR-002 — Register adapters at the call site / family libs, not in core

• Status: Accepted.
• Context: Provider-specific classification needs heavy SDKs (smithy, azcore, lib/pq).
• Decision: Families (grafana-aws-sdk, sqlds, grafana-azure-sdk-go) and plugins self-register classifiers + rules via init(). The core never imports those SDKs.
• Consequences: No dependency bloat; per-binary rule scoping; classification lives next to the connection code. Slight duplication of SDK deps the family libs already carry.

ADR-003 — Author remediation rules as Go literals, not YAML

• Status: Accepted.
• Context: Authors are Go developers; YAML adds a parser, embed, and templating.
• Decision: Rules are []Rule Go literals registered via RegisterRules.
• Consequences: Compile-time-checked, no new dependency, easy refactors; bulk copy edits require a rebuild (acceptable). A RemediationProvider func remains for dynamic cases.

ADR-004 — Rich, multi-dimensional Diagnosis + specificity-ranked rules

• Status: Accepted.
• Context: A flat code → message map can't express provider sub-codes, auth type, deployment, TLS sub-kind, etc.
• Decision: Classifiers fill a structured Diagnosis; remediation is matched by Match over any subset of dimensions and ranked by specificity with graceful fallback.
• Consequences: Scales by adding data; partial coverage still yields useful messages; ordering logic is centralized and testable.

ADR-005 — Split output: safe Message + structured JSONDetails, redact verbose

• Status: Accepted.
• Context: Need both human-friendly and machine-readable output without leaking secrets.
• Decision: Message is built only from catalog/rule copy + DS name; JSONDetails has a fixed shape — errorCode/providerCode/errorSource/correlationId/remediation/verbose — and additional diagnostic metadata is folded into verbose rather than added as new fields.
• Consequences: Stable, non-growing contract for frontend + telemetry; secrets never in Message; verbose redacted for support and carries the diagnostic signals as a suffix.

ADR-006 — Additive, versioned taxonomy; frontend owns localization

• Status: Accepted.
• Context: Backend strings shouldn't be the localization surface.
• Decision: errorCodes are additive and never repurposed; the frontend maps code → localized copy + doc link.
• Consequences: Backend wording changes are non-breaking; i18n is centralized in the frontend.

ADR-007 — Upstream response-body interpretation is delegated and defensive

• Status: Accepted.
• Context: Upstream errors arrive as HTML pages (proxy/WAF/SSO interception) or as JSON envelopes whose shape differs per provider. The core can't assume a body format, and the raw body is unsafe to surface.
• Decision: Provider classifiers own parsing of their own envelopes and extract a stable ProviderCode; a best-effort generic extractor covers the long tail; a response-aware helper maps HTML by status + context. The body only ever classifies — it is never echoed into Message, only into a redacted verbose — and parsing is size/depth-bounded, degrading to UNEXPECTED_RESPONSE. (See §6.4a, §11.)
• Consequences: Stable output regardless of upstream format; new envelopes are handled by adding a classifier, not changing core; "classify, don't echo" keeps secrets out of surfaced copy. Generic-path hits are logged as the backlog for new classifiers.

ADR-008 — Classify on typed errors with deterministic precedence

• Status: Accepted.
• Context: Matching on SDK error strings breaks silently when smithy/lib/pq/azcore reword errors between versions, and init() registration order across packages is non-deterministic — so two matching classifiers could resolve differently per build.
• Decision: Classifiers prefer errors.As/errors.Is against concrete SDK error types (string sniffing only as an explicit last resort), walk wrapped chains, and handle errors.Join. Resolution order is fixed by an explicit priority/specificity rule, never by registration order.
• Consequences: Robust across SDK upgrades; deterministic, testable classification; a small priority contract every classifier must honour. The generic string-based classifier is the documented fallback, not the norm.

ADR-009 — Classify once, render to every surface (UI / logs / metrics / trace)

• Status: Accepted.
• Context: A health error serves three audiences with different safety bars; treating them as one string either leaks detail to the UI or starves logs/metrics.
• Decision: Classify once into a Diagnosis, then render: a minimal safe Message+remediation+correlation ID to the UI; a fuller, secret-masked structured log line keyed by the same correlation ID; bounded-cardinality metrics (providerCode is logged, never a metric label); and a redacted record on the active trace span (SetStatus/RecordError/attributes). The log, metrics and span sinks are injected via options (WithLogger/WithMetrics/WithSpanRecorder) so the core stays dependency-light and the fan-out is testable. The correlation ID is the trace ID when sampled, else a generated fallback. Log severity is keyed on errorSource. (See §10, §10.4.)
• Consequences: No surface reclassifies; the trace/correlation ID bridges UI, logs, traces and metric exemplars so verbose can stay hidden by default; redaction extends to spans (Tempo has its own access controls); cardinality discipline keeps metrics safe and cheap; classifier-coverage becomes a monitored regression signal.

9. Rollout

• Phase 0 — authoritative audit: with org access, fetch each repo's src/plugin.json + CheckHealth; finalize connection-family counts and a per-repo conversion checklist.
• Wave 1: land dsconfig/health; wire sqlds + grafana-aws-sdk shared CheckHealth → ~18 SQL/AWS plugins via dependency bumps.
• Wave 2: Azure/GCP classifiers + rules; high-traffic HTTP plugins (loki, tempo, prometheus, elasticsearch, influxdb).
• Wave 3: long tail (bespoke HTTP, mqtt, sqleng postgres/mysql/mssql).

Each wave is independently shippable; un-migrated plugins keep working.

10. Observability — user, logs, telemetry

A single Diagnosis fans out to several surfaces with different audiences and safety bars (three consumer surfaces below, plus the trace span in §10.4). The library classifies once; no surface reclassifies.

Surface	Audience	Goal	Safety bar
UI (CheckHealthResult)	DS admin/editor	fix it now	strict — no secrets, no internal infra by default
Logs	operator / plugin dev / support	diagnose a specific failure	masked, but fuller; correlatable
Metrics	SRE / PM / DS team	trends, alerting, classifier backlog	aggregate only, bounded cardinality

10.1 What the user sees (UI)

Message, remediation, errorCode, errorSource, providerCode, and a correlation ID (§6.6); the diagnostic metadata (httpStatus/tlsKind/timeoutKind/bodyKind/contentType) rides inside the redacted verbose string, keeping the schema fixed. No raw error, stack trace, internal host/DSN, or upstream body; verbose is hidden by default (see §14) — the correlation ID is the bridge to detail.

10.2 What we log

Structured (SDK logger, key/value): errorCode, providerCode, errorSource, datasourceType, datasourceUID (prefer UID over name — names can be PII), sub-signals (timeoutKind/tlsKind/bodyKind), httpStatus, correlationId, classifierPath (tag/family/generic/unknown), check duration, and the redacted verbose; carry trace/span IDs when OTel is present.

• Level keyed on errorSource to avoid alert fatigue: downstream (user misconfig) → warn/info; plugin / UNKNOWN / classification-failed → error (our bugs).
• Log once at the boundary where health.Result is built, not at every wrap layer. Secrets always masked, even in logs; infra detail (host) may be allowed in logs but never the UI (policy knob).

10.3 How we track / monitor (metrics)

Counters + a histogram, with strict label cardinality:

datasource_healthcheck_total{datasourceType}                                   # denominator
datasource_healthcheck_failures_total{errorCode, datasourceType, errorSource}  # numerator
datasource_healthcheck_duration_seconds{datasourceType}                        # slow / timeout trends

• Bounded labels only. errorCode (closed taxonomy), datasourceType (~50), errorSource (2), small enum sub-signals. providerCode is NOT a metric label (unbounded — AADSTS/SQLSTATE/etc.); it lives in logs (and metric exemplars if a bridge is wanted).
• Coverage is first-class. Track the rate of classifierPath="generic", UNKNOWN, and UNEXPECTED_RESPONSE — it is both the classifier backlog and the regression alarm (a sudden spike usually means an SDK reworded its errors and a string-match classifier broke — the ADR-008 risk).
• Alert on signal, not baseline: spike in errorSource=plugin/UNKNOWN → likely a release regression (page); spike in one errorCode×datasourceType → upstream incident or bad release; don't alert on baseline downstream config errors — track those as an onboarding funnel (first-Save&test success rate) instead.
• Privacy: metrics are safe only while labels stay bounded and free of free text; respect telemetry opt-out (OSS) and route to usage-insights / the Cloud pipeline accordingly (see §14).

10.4 Trace propagation & correlation

A distributed trace already flows through CheckHealth; the correlation ID rides on it rather than competing with it.

Browser (Save & test)
   │  HTTP  (root span often starts here / at Grafana)
   ▼
Grafana server  ── creates/continues span ──┐
   │  gRPC  (W3C traceparent in metadata)    │  one trace
   ▼                                          │
Plugin backend  CheckHealth(ctx, req)  ◄──────┘  ctx carries trace+span
   │  outbound (httpclient / otelsql / otelaws)
   ▼
Downstream datasource  (usually NOT in your trace)

• Caller threading: the plugin passes the inbound ctx from CheckHealth(ctx, req) straight through its connection attempt into Result(ctx, …). The health.Context dimensions (datasource type/auth/network) are assembled from req separately — distinct from the Go ctx.
• Inbound context: the SDK's OTel gRPC interceptors propagate trace context, so ctx carries the trace/span; read it via tracing.TraceIDFromContext(ctx, true) / OTel SpanContextFromContext.
• Outbound: the SDK httpclient tracing middleware injects traceparent and opens a client span; otelaws does the same and captures the AWS request ID; otelsql wraps queries. The upstream rarely joins the trace, so the plugin-side client span is the unit of truth — operators shouldn't hunt for a downstream span that never existed.
• Span recording is injected. Result records the redacted classification onto the span (SetStatus(Error), RecordError, attributes health.code/error_source/provider_code/timeout_kind/classifier_path) via a caller-supplied SpanRecorder, so the core import graph stays SDK-only (ADR-001); an otel-backed recorder lives in the plugin/family lib. Redaction extends to spans — they ship to Tempo under different access controls than logs, so the same masking runs first.
• Sets correlationId = trace ID when sampled, else a generated fallback; stamps traceID/spanID on the log line.

How each scenario reads in a trace:

Scenario	What the trace shows	The tell
HTML / proxy-WAF-SSO interception	outbound client span errors/odd status; no child span from the real datasource	the gap is diagnostic — the request never reached the API
Timeout	client span duration pinned at the deadline; health.timeout_kind attr	dial vs read visible from where time went
Cancellation	span ends cancelled	treat as benign — do not SetStatus(Error) (avoids polluting error dashboards)
SQL / AWS	otelsql/otelaws client span with db.system / AWS request ID	AWS request ID is the cross-ref to vendor support, like providerCode

Stitching the three pillars: traceID/spanID on logs → Loki↔Tempo derived-field links; metric exemplars carry traceID so a spike on ..._failures_total links straight to a failing trace; the UI correlation ID resolves the whole trace.

Caveats:

• Sampling drops failures. Traces are head-sampled, so a traceID may have no stored trace. Health checks are low-volume / high-value — force sampling on CheckHealth errors (sampling priority / sample-on-error) so a failure trace is never lost.
• Tracing may be off (OSS/unconfigured): degrade gracefully — generate the correlation ID, still log and emit metrics; handle zero/un-sampled span context.

10.5 Testing & conformance

• Unit tests: table-driven Diagnose/Result cases → expected Code, ErrorSource, redacted output, resolved remediation.
• Conformance helper: every declared (datasourceType × authType × code) resolves to a non-empty remediation with a docsUrl.
• CI lint: fail any CheckHealth returning raw err.Error() instead of health.Result.

11. Security & privacy

Redaction runs on verbose; Message is built only from catalog/rule copy + datasource name, never the raw error. ErrorSource mapping keeps plugin- vs downstream-caused failures correct for SLOs.

Response bodies are never dumped raw into verbose. HTML error/login pages and JSON error envelopes frequently embed internal hostnames, stack traces, CSRF/session tokens, and form details. The redactor therefore:

• HTML: records contentType, byte length, status, and at most the <title> or first ~200 chars of visible text — e.g. HTML response (text/html, 4096 bytes, 200): "Sign in - Okta" — run through the secret masker; never the full page.
• JSON: records only the extracted message/ProviderCode hint (see §6.4a), masked — not the whole body.
• Bodies are size- and depth-capped before parsing and treated as untrusted, so a malformed/huge/non-UTF8 body degrades to UNEXPECTED_RESPONSE instead of leaking or crashing.

12. Backwards compatibility & versioning

The errorCode taxonomy is additive; codes never change meaning. JSONDetails has a fixed shape — diagnostic metadata is folded into verbose rather than added as new top-level fields, so the contract never grows new keys. Adoption is per-plugin and independent.

13. Alternatives considered

• Everything in one self-contained module (incl. all adapters): forces AWS+Azure+SQL+GCP deps onto every importer. Rejected (dependency bloat). [ADR-001/002]
• Core in grafana-plugin-sdk-go: ubiquitous, but baking opinionated copy into the core SDK is undesirable and ties cadence to the SDK. Rejected; reuse SDK primitives only.
• YAML-authored rules: flexible/i18n-friendly but adds parser/embed/templating. Rejected. [ADR-003]
• Per-repo bespoke fixes: no shared contract; doesn't scale. Rejected.

14. Open questions

• Exact module path within grafana/dsconfig (/health vs a package in the existing /dsconfig module). The reference implementation (feat(health): normalized, actionable health-check errors (RFC #110) #111) places it at /health as its own module; relocating is cheap if reviewers prefer otherwise.
• Frontend ownership of the errorCode → localized-copy + doc-link mapping.
• Standardize AuthType/Deployment enums centrally, or per family?
• verbose visibility: hide by default and rely on the correlation ID (§6.6) + logs, or gate verbose behind a debug/admin flag in the UI? (Even redacted, it can reveal upstream hostnames.)
• Telemetry destination & opt-out: where do the §10.3 metrics land (usage-insights / Cloud pipeline), and how is OSS telemetry opt-out honoured?
• Model a non-binary "warning/degraded" outcome, or stay strictly OK-vs-error? Real cases — connected but a deprecated server version, partial permissions, or TLS verification disabled — are soft failures. Options: (a) add a degraded/warning result (richer UX, new frontend contract, more surface); (b) stay binary and surface soft issues as text in the OK message (simpler, matches backend.HealthStatus today). Decide before implementation.

15. Operational & edge-case considerations

• Multiple problems → one code. A check can fail several ways at once; resolution applies a fixed precedence (config → TLS → auth → connection → permission) and reports the most actionable single Code. This precedence also governs tag selection when an error tree carries several tags (e.g. errors.Join) — Diagnose collects all tags across the unwrap tree and picks by rank, not traversal order.
• Redaction is defense-in-depth, not the primary control. Keyword/regex masking has false negatives, so "classify, don't echo" (§6.4a) stays the main safeguard; verbose favours an allowlist of known-safe fields over denylisting bad ones, plus size caps. (Surface routing — UI vs logs vs metrics — is covered in §10.)
• Rate limiting carries detail. When present, Retry-After is captured into the remediation ("retry after Ns").
• Defensive defaults. Result(nil) → OK; missing Context → generic remediation; zero/unknown Code → UNKNOWN; classification never panics on malformed input.
• Force-sample failed health checks. Traces are head-sampled, so a failure's trace can be dropped; set sampling priority / sample-on-error in CheckHealth so the high-value failure trace (§10.4) is retained.
• Frontend must tolerate unknown future codes (requirement, not aside). A newer backend errorCode an older UI doesn't recognize must fall back to rendering Message — never a blank/typed-as-unknown state.

---

Appendix A — datasource repo inventory (52)

Expand the full list

astradb-datasource, athena-datasource, azure-cosmosdb-datasource, azure-data-explorer-datasource, azure-prometheus-datasource, clickhouse-datasource, dynamodb-datasource, falconlogscale-datasource, github-datasource, google-bigquery-datasource, google-sheets-datasource, grafana-amazonprometheus-datasource, grafana-aurora-datasource, grafana-azure-monitor-datasource, grafana-cloudmonitoring-datasource, grafana-cloudwatch-datasource, grafana-csv-datasource, grafana-cube-datasource, grafana-elasticsearch-datasource, grafana-graphite-datasource, grafana-infinity-datasource, grafana-influxdb-datasource, grafana-jaeger-datasource, grafana-loki-datasource, grafana-malefico-datasource, grafana-marketplacetest-datasource, grafana-mssql-datasource, grafana-mysql-datasource, grafana-opentsdb-datasource, grafana-parca-datasource, grafana-postgresql-datasource, grafana-prometheus-datasource, grafana-pyroscope-datasource, grafana-tempo-datasource, grafana-test-datasource, grafana-zipkin-datasource, grafanacon-custom-data-source-plugin-example, hackathon-13-rapid7-ics, iot-sitewise-datasource, jenkins-datasource, mock-datasource, mqtt-datasource, opensearch-datasource, pagerduty-datasource, redshift-datasource, sentry-datasource, strava-datasource, tailscale-grafana-plugin, test-datasource, timestream-datasource, x-ray-datasource, yugabyte-datasource.

(+ monorepos grafana/plugins, grafana/plugins-private to sweep; app-bundled grafana-k6-app, asserts-app-plugin, hackathon-16-tamagrotchi-grafana-app excluded.)

Appendix B — API sketch

Public surface of the health module, matching the reference implementation in #111.

package health

import (
	"context"
	"net/http"
	"time"

	"github.com/grafana/grafana-plugin-sdk-go/backend"
)

// Code is the stable, coarse, public error vocabulary (additive, never repurposed).
type Code string

const (
	CodeInvalidConfiguration Code = "INVALID_CONFIGURATION"
	CodeAuthenticationFailed Code = "AUTHENTICATION_FAILED"
	CodePermissionDenied     Code = "PERMISSION_DENIED"
	CodeHostUnreachable      Code = "HOST_UNREACHABLE"
	CodeConnectionTimeout    Code = "CONNECTION_TIMEOUT"
	CodeTLSError             Code = "TLS_ERROR"
	CodeNotFound             Code = "NOT_FOUND"
	CodeRateLimited          Code = "RATE_LIMITED"
	CodeQuotaExceeded        Code = "QUOTA_EXCEEDED"
	CodeUpstreamError        Code = "UPSTREAM_ERROR"
	CodeUnsupportedVersion   Code = "UNSUPPORTED_VERSION"
	CodeQueryValidation      Code = "QUERY_VALIDATION_FAILED"
	CodeUnexpectedResponse   Code = "UNEXPECTED_RESPONSE"
	CodeUnknown              Code = "UNKNOWN"
)

// Error tags an error with an explicit Code and optional sub-signals (highest classifier priority).
type Error struct {
	Code         Code
	ProviderCode string
	Field        string
	Err          error
}

func (e *Error) Error() string { return e.Err.Error() }
func (e *Error) Unwrap() error { return e.Err }

func WithCode(code Code, err error) error { return &Error{Code: code, Err: err} }
func Tag(err error, code Code, providerCode, field string) error {
	return &Error{Code: code, ProviderCode: providerCode, Field: field, Err: err}
}

// Diagnosis is the structured classification result; Context carries call-site dimensions.
type TLSKind string

const (
	TLSUnknownAuthority TLSKind = "unknown_authority"
	TLSHostnameMismatch TLSKind = "hostname_mismatch"
	TLSExpired          TLSKind = "expired"
	TLSClientCert       TLSKind = "client_cert"
)

type Context struct {
	DatasourceType string
	DatasourceName string
	AuthType       string
	Deployment     string // cloud | enterprise | oss
	NetworkPath    string // direct | pdc | proxy
	DocsBaseURL    string
	Vars           map[string]string
}

// BodyKind captures the shape of an upstream response body (see §6.4a).
type BodyKind string

const (
	BodyJSON BodyKind = "json"
	BodyHTML BodyKind = "html"
	BodyText BodyKind = "text"
)

// TimeoutKind distinguishes dial vs read vs the CheckHealth deadline (see §6.4b).
type TimeoutKind string

const (
	TimeoutDial     TimeoutKind = "dial"     // connect timed out (often firewall/wrong port)
	TimeoutRead     TimeoutKind = "read"     // reachable but slow
	TimeoutDeadline TimeoutKind = "deadline" // CheckHealth ctx deadline elapsed
)

// ClassifierPath records which pipeline stage produced a Diagnosis (RFC §10.2/§10.3).
type ClassifierPath string

const (
	PathTag     ClassifierPath = "tag"
	PathFamily  ClassifierPath = "family"
	PathGeneric ClassifierPath = "generic"
	PathUnknown ClassifierPath = "unknown"
)

type Diagnosis struct {
	Code         Code
	ProviderCode string
	HTTPStatus   int
	TLSKind      TLSKind
	TimeoutKind  TimeoutKind // when Code is CONNECTION_TIMEOUT / HOST_UNREACHABLE
	BodyKind     BodyKind    // sniffed response-body shape, when known
	ContentType  string      // upstream response Content-Type, when known
	Field        string      // offending config field, when known
	Path         ClassifierPath
	Context      Context
}

// Classifier maps a raw error to a Diagnosis. Families register these via init().
type Classifier func(err error, ctx Context) (Diagnosis, bool)

// RegisterClassifier registers c at the default priority (0).
func RegisterClassifier(c Classifier)

// RegisterClassifierWithPriority registers c; higher priority wins, ties break by
// registration order — never by init() ordering across packages (ADR-008).
func RegisterClassifierWithPriority(priority int, c Classifier)

// Diagnose runs the classification pipeline (RFC §6.4). It collects every *Error
// tag across the unwrap tree (incl. errors.Join) and picks the most actionable by
// precedence (RFC §15); otherwise it tries family classifiers then generic
// net/TLS/timeout inspection. Never returns an empty Code, never panics.
func Diagnose(err error, ctx Context) Diagnosis

// ClassifyHTTPResponse inspects an HTTP response (status + Content-Type + body)
// and refines HTML/non-JSON answers by status and context (§6.4a). HTTP families
// call this before unmarshalling so headers are available to the classifier.
func ClassifyHTTPResponse(resp *http.Response, body []byte, ctx Context) (Diagnosis, bool)

// ExtractJSONError is a best-effort, never-authoritative reader for arbitrary
// JSON error envelopes (§6.4a). It tolerates error-as-string|object|array,
// numeric-or-string codes, nesting, and missing fields; parsing is size- and
// depth-bounded and never panics. Returns ok=false when nothing parses.
func ExtractJSONError(body []byte) (msg string, providerCode string, ok bool)

// Remediation is the structured, surfaced guidance.
type Remediation struct {
	Summary string   `json:"summary"`
	Steps   []string `json:"steps,omitempty"`
	DocsURL string   `json:"docsUrl,omitempty"`
	Fields  []string `json:"fields,omitempty"`
}

// Match is a declarative condition set; empty fields are wildcards. Rules are Go literals.
type Match struct {
	Code         Code
	ProviderCode string
	TLSKind      TLSKind
	TimeoutKind  TimeoutKind
	BodyKind     BodyKind
	AuthType     string
	Deployment   string
}

type Rule struct {
	When Match
	Give Remediation
}

func RegisterRules(rules ...Rule)

// Sinks are injected so the core stays dependency-light (ADR-001/009).
// backend.Logger satisfies Logger directly; SpanRecorder is backed by an
// otel adapter in the plugin so the core never imports OpenTelemetry.
type Logger interface {
	Error(msg string, args ...any)
	Warn(msg string, args ...any)
	Info(msg string, args ...any)
}
type Metrics interface {
	Observe(d Diagnosis, durationSeconds float64)
}
type SpanRecorder interface {
	Record(ctx context.Context, d Diagnosis, redactedVerbose string)
}

type Option func(*options)

func WithContext(c Context) Option       // classification/remediation dimensions
func WithLogger(l Logger) Option         // structured log sink (§10.2)
func WithMetrics(m Metrics) Option       // metrics sink (§10.3)
func WithSpanRecorder(s SpanRecorder) Option // trace span sink (§10.4)
func WithVerbose(include bool) Option    // include redacted verbose in JSONDetails (default off)
func WithDuration(d time.Duration) Option

// Result classifies err once and renders it four ways (ADR-009): the safe UI
// payload is returned; the log/metrics/span sinks fire only when injected. The
// inbound ctx supplies the trace for the correlation ID and span recording.
//   - err == nil          → OK
//   - benign cancellation → HealthStatusUnknown (RFC §6.4b)
//   - otherwise           → HealthStatusError + JSONDetails
func Result(ctx context.Context, err error, opts ...Option) *backend.CheckHealthResult

// ResultForResponse is the response-aware entry point for HTTP families (§6.4a):
// it inspects status + Content-Type + body so HTML/odd answers classify with
// their sub-signals preserved. A 2xx with a normal body and no rawErr → OK.
func ResultForResponse(ctx context.Context, resp *http.Response, body []byte, rawErr error, opts ...Option) *backend.CheckHealthResult

// OK is the success constructor.
func OK(message string) *backend.CheckHealthResult