fix(valkey-operator): align ServiceMonitor with metrics.secure (skip-verify opt-in)

[daanvinken]

Summary

Fixes ServiceMonitor scrapes when metrics.secure is true, and makes TLS skip-verify opt-in rather than a silent default.

Bug

With metrics.secure: true, the metrics Service exposes port https, but the ServiceMonitor always targeted port http, so scrapes never hit the right endpoint.

What we change

1. Port / scheme always follow metrics.secure (https + default scheme: https when secure and serviceMonitor.scheme is unset).
2. metrics.serviceMonitor.insecureSkipVerify (default false). Only when this is true (and tlsConfig is empty) do we inject tlsConfig.insecureSkipVerify: true for the operator's self-signed metrics cert. Explicit serviceMonitor.tlsConfig always wins.

Design question for reviewers

Should skip-verify stay disabled by default?

Default	Behavior
false (this PR)	Secure scrapes need either insecureSkipVerify: true or a real tlsConfig (CA). No silent MITM-friendly default.
true	Secure + ServiceMonitor works out of the box with self-signed certs, but enabling secure metrics also disables cert verification unless the user overrides.

We prefer default false: "secure" should not imply "do not verify." The operator cert is self-signed, so skip-verify is still available as an explicit one-liner for the common case.

metrics:
  secure: true
  serviceMonitor:
    enabled: true
    insecureSkipVerify: true  # opt-in for self-signed metrics cert

If maintainers prefer out-of-the-box scrapes, flipping the default to true is a one-line change; the important part is that it is named and documented, not hard-coded only in the template.

Docs

README (Metrics / ServiceMonitor), values comments, and CHANGELOG 0.4.1 describe port/scheme alignment and the opt-in flag.

Not in this PR

Bearer token for secure metrics auth (WithAuthenticationAndAuthorization) is a separate follow-up so scrapes do not get 401 after TLS is fixed.

Testing

# insecure: port http, no scheme/tls
helm template t valkey-operator \
  --set metrics.serviceMonitor.enabled=true

# secure, default: port https, scheme https, no tlsConfig
helm template t valkey-operator \
  --set metrics.secure=true \
  --set metrics.serviceMonitor.enabled=true

# secure + opt-in skip-verify
helm template t valkey-operator \
  --set metrics.secure=true \
  --set metrics.serviceMonitor.enabled=true \
  --set metrics.serviceMonitor.insecureSkipVerify=true

[Bloodraven21]

@daanvinken please rebase the branch

[daanvinken]

Check

[rlunar]

@greptileai

[greptile-apps]

Greptile Summary

This change makes the Valkey Operator ServiceMonitor select the secure metrics port, default secure scrapes to HTTPS, and support explicit TLS settings or opt-in certificate verification bypass. Secure metrics scraping is still unavailable through the generated ServiceMonitor because it does not send the token required by the protected endpoint. Rendering checks also confirmed that the newly added secure transport branches have no focused regression coverage.

Confidence Score: 3/5

Not safe to merge for deployments that enable secure operator metrics, because Prometheus cannot authenticate to the protected metrics endpoint.

Two confirmed issues affect the secure ServiceMonitor path: missing scrape credentials prevent monitoring, and the transport branches that introduced this behavior are not covered by focused tests.

Files Needing Attention: valkey-operator/templates/servicemonitor.yaml needs endpoint authorization support; valkey-operator/tests/servicemonitor_test.yaml needs secure transport rendering cases.

T-Rex Logs

What T-Rex did

• Generated a P1 finding proof for the secure metrics authentication-path test, with supporting artifacts including the test source and two HTTPS request logs.
• Generated a P1 finding proof covering ServiceMonitor coverage and live renders, including the harness source, unit-suite results, live renders, and the chart lint result.
• Validated the HTTPS /metrics contract by showing a 401 response without Authorization and a 200 response with a Bearer token, and confirmed the rendered endpoint fields port=https, scheme=https, and honorLabels=false.
• Compared pre- and post-capture coverage; after capture, all four value inputs are absent and three secure outputs appear, and the chart lint passed.

_{Ran code and verified through T-Rex}

Comments Outside Diff (2)

1. General comment

   Secure ServiceMonitor cannot authenticate to protected operator metrics

   • Bug
     • The secure branch at valkey-operator/templates/servicemonitor.yaml lines 29-30 sets scheme: https, but the endpoint has no bearer-token or authorization configuration. The chart separately binds a scraper ServiceAccount to metrics-reader, and its documentation states a scraper must present that ServiceAccount token. Prometheus therefore sends an unauthenticated HTTPS scrape and is rejected by the protected metrics endpoint.
   • Cause
     • The ServiceMonitor template configures transport security only; it does not reference a Secret-backed token through authorization (or a supported equivalent) for the configured scraper identity.
   • Fix
     • Add explicit, documented ServiceMonitor authentication values (for example an authorization.credentials SecretKeySelector), render them on the endpoint when secure metrics are enabled, and require/validate them or clearly disable the secure ServiceMonitor configuration until supplied.

   _{Ran code and verified through T-Rex}

2. General comment

   ServiceMonitor secure rendering branches have no unit coverage

   • Bug
     • The ServiceMonitor suite asserts only the insecure default endpoint port (http) among the endpoint security-selection paths. It never sets metrics.secure, a ServiceMonitor scheme, tlsConfig, or insecureSkipVerify, so it cannot detect regressions in secure port selection, default HTTPS scheme, explicit TLS configuration precedence, or self-signed certificate skip verification.
   • Cause
     • tests/servicemonitor_test.yaml lacks secure-value test cases and assertions for the conditional branches at templates/servicemonitor.yaml lines 19 and 27-38.
   • Fix
     • Add focused helm-unittest cases that enable the ServiceMonitor and: (1) set metrics.secure: true and assert port: https plus default scheme: https; (2) set an explicit scheme and assert it overrides the default; (3) set both tlsConfig and insecureSkipVerify: true and assert only the explicit TLS configuration is rendered; and (4) set secure metrics plus insecureSkipVerify: true without tlsConfig and assert tlsConfig.insecureSkipVerify: true.

   _{Ran code and verified through T-Rex}

_{Reviews (1): Last reviewed commit: "chore(valkey-operator): bump chart to 0...." | Re-trigger Greptile}

[greptile-apps]

Secure ServiceMonitor lacks scrape authentication

When metrics.secure is enabled, this endpoint selects HTTPS but does not configure authorization or a bearer-token secret. The protected operator metrics endpoint requires the metrics-reader service-account token, so Prometheus sends an unauthenticated scrape and receives 401 even when the intended scraper identity has the required RBAC binding. Add a documented Secret-backed authorization configuration to the ServiceMonitor endpoint for secure metrics.

Artifacts

Secure metrics authentication-path test source

• Authored Python test source that checks the chart's secure endpoint and exercises an HTTPS metrics endpoint requiring a Bearer token; it demonstrates the required authentication condition.

HTTPS metrics request without ServiceMonitor authentication

• Executed HTTPS `/metrics` request after checking the chart's secure endpoint fields; no Authorization header was sent and the auth-enforcing endpoint returned 401, confirming rejection.

HTTPS metrics request with bound service account Bearer token

• Executed the same HTTPS `/metrics` request with the expected Bearer token; the endpoint returned 200 and metrics, confirming that token authentication is required.

_{Ran code and verified through T-Rex}

[greptile-apps]

Secure ServiceMonitor branches lack regression coverage

The focused ServiceMonitor suite only asserts the default insecure http endpoint. It never enables metrics.secure or exercises the explicit scheme, tlsConfig, and insecureSkipVerify paths, so regressions in secure port selection, HTTPS defaults, TLS-config precedence, or self-signed certificate handling pass the current tests. Add focused Helm unit tests for each secure rendering path.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Artifacts

Secure metrics authentication-path test source

• Authored Python test source that checks the chart's secure endpoint and exercises an HTTPS metrics endpoint requiring a Bearer token; it demonstrates the required authentication condition.

HTTPS metrics request without ServiceMonitor authentication

• Executed HTTPS `/metrics` request after checking the chart's secure endpoint fields; no Authorization header was sent and the auth-enforcing endpoint returned 401, confirming rejection.

HTTPS metrics request with bound service account Bearer token

• Executed the same HTTPS `/metrics` request with the expected Bearer token; the endpoint returned 200 and metrics, confirming that token authentication is required.

ServiceMonitor coverage and rendering harness source

• Authored shell harness searches the focused test suite for the claimed inputs and renders the three secure ServiceMonitor cases, providing a reproducible narrow check; it establishes how the observed outputs were produced.

Current ServiceMonitor unit suite passes without secure branch cases

• Executed `helm unittest valkey-operator -f tests/servicemonitor_test.yaml` with the helm-unittest plugin and recorded 10 passing tests; it shows the existing suite passes before any added coverage.

Secure ServiceMonitor branch coverage check and live renders

• Executed the authored harness with Helm and captured absent test-value inputs plus live secure-default, explicit-TLS-precedence, and insecure-skip-verify renders; it confirms the missing coverage and distinct untested outcomes.

Valkey operator chart lint result

• Executed Helm lint against the valkey-operator chart and captured one chart linted with zero failures; it confirms the rendered template is valid.

_{Ran code and verified through T-Rex}