database: add per-query Prometheus metrics

[yi-chen-roger]

Summary

Adds per-query Prometheus metrics for the Database layer (CockroachDB/postgres/mysql), which currently has none. Build-time DB failures today only surface as BundleBuilderTaskFailureRateHigh{error_type=service} (in a consumer) with no further detail on which query or operation failed.

New metrics, following the existing gitsync/http/worker pattern in internal/metrics:

• ocp_database_query_count_total{operation, state} — counter, state is SUCCESS/FAILED
• ocp_database_query_duration_seconds{operation} — histogram, operation parsed from the SQL statement's first keyword (select/insert/update/...)

Design

The instrumentation is wired in at the driver.Connector level (internal/database/instrumented.go), not by touching any of the ~50 methods in database.go — every one of them already funnels through sql.OpenDB(connector) for postgres/CockroachDB/mysql (sqlite's sql.Open path is left alone; it's the in-memory/dev default, not a production case).

The tricky part of this kind of wrapping is driver.Conn's several optional interfaces (QueryerContext, ExecerContext, Pinger, ConnPrepareContext, ConnBeginTx, SessionResetter, NamedValueChecker, Validator) — database/sql detects which ones a Conn implements to decide its execution path, and a wrapper that doesn't preserve the same set of capabilities silently changes behavior (e.g. falling back to a slower prepare-then-exec path) without erroring.

An earlier version of this PR tried to solve this by constructing a different wrapper type per detected capability combination. TestInstrumentedConn_ForwardsUnrelatedOptionalInterfaces caught a real bug in that approach (driver.Pinger support silently dropped), which led to redesigning around the pattern used by github.com/XSAM/otelsql (not taken as a dependency — it's OTel-native, this stays Prometheus-native to match the rest of internal/metrics): the wrapper unconditionally implements every optional interface, and each method checks at call time whether the wrapped conn actually supports it, returning driver.ErrSkip when it doesn't — the sentinel database/sql defines for exactly this case, so it falls back exactly as it would without any wrapping. This avoids the combinatorial-type problem entirely: there's one wrapper type, and no capability can be silently dropped.

PrepareContext/BeginTx fallback logic (used when the wrapped conn doesn't implement the optional context-aware version) is adapted from otelsql, which itself credits it to database/sql's own internal ctxutil.go.

Wiring

Database.WithMetrics(*metrics.Metrics) mirrors the existing WithLogger/WithAuthorizer pattern. Service.WithMetrics now also propagates to s.database, mirroring how it already does for WithLogger.

Also adds DatabaseMetrics to the public MetricsConfig schema (internal/config/config.go, regenerated schema.json/config/schema.json) for parity with GitSyncMetrics/WorkerMetrics/HTTPMetrics, so operators can fine-tune or disable it the same way.

Test plan

• TestInstrumentedConn_FallsBackWhenUnsupported — wrapping a conn with neither QueryerContext nor ExecerContext returns driver.ErrSkip, not a panic or a silent no-op.
• TestInstrumentedConn_ForwardsUnrelatedOptionalInterfaces — driver.Pinger support is preserved through the wrapper (the regression test for the bug the earlier design had).
• TestInstrumentedConn_RecordsMetrics — a successful query and a failed exec both show up correctly in ocp_database_query_count_total via a real prometheus.Registry.
• TestInstrumentedConn_BeginTxFallback — the ctxutil.go-derived fallback path propagates errors correctly and rejects unsupported non-default isolation levels.
• go test ./internal/database/... ./internal/metrics/... ./internal/config/... — all pass except the pre-existing postgres/mysql/cockroachdb testcontainers-based subtests, which fail in my sandbox with "rootless Docker not found" — confirmed this also fails identically on unmodified main, so it's an environment limitation, not something this PR broke. Would appreciate a maintainer or CI run to confirm this actually instruments real CockroachDB/postgres/mysql connections end-to-end, since that's the one thing my sandbox couldn't verify directly.

[patrick-east]

Code changes LGTM, just a couple things I'd recommend cleaning up in the comments

[patrick-east]

We probably don't need commentary from previous iterations included in the comments. It seems like the only take away that is important here is that instrumentedConn must implement all of the driver.Conn interfaces.

[yi-chen-roger]

updated

[patrick-east]

Similar here, we don't need comments about previous issues

[yi-chen-roger]

updated

[ashutosh-narkar]

The changes look good. We should add some tests to exercise the metrics package.

[ashutosh-narkar]

Not sure how this test fits into what we're implementing here. Please remove if not needed.

[yi-chen-roger]

DatabaseOperation is new logic added by this PR (parses the SQL keyword for the operation label) — this test covers its parsing edge cases (unrecognized/empty/multi-keyword input) directly, separate from the end-to-end recording test.

[ashutosh-narkar]

Thanks for adding the tests @yi-chen-roger. The changes lgtm. I'll let @patrick-east give the final sign-off.