feat: long-haul (canary) test driver for the operator (#220)

[WentingWu666666]

Closes #220.

What this PR adds

A standalone long-haul (canary) test driver for the DocumentDB Kubernetes Operator, plus the GitHub Actions plumbing to keep it running on a long-lived AKS cluster.

The driver lives under test/longhaul/ and is its own Go module. It is not a Ginkgo suite — it is a long-running binary (cmd/longhaul) deployed as a Deployment into a namespace alongside a real DocumentDB CR. It exercises the operator with a continuous, realistic workload while a weighted-random scheduler injects disruptive operations, and it journals every event so failures can be reproduced.

Pieces that ship in this PR

• Continuous data plane (workload/): writer + verifier goroutines, sequence / checksum gap detection, lag-aware reads — the durability oracle for FM1/FM3/FM7.
• Operation scheduler (operations/): weighted random selection, per-category cooldowns, steady-state gating, and an outage-policy verdict per operation. Operations implemented today: ScaleUp, ScaleDown, UpgradeDocumentDB (auto-skips when instancesPerNode < 2 because rolling restart on a single instance is unavoidable downtime).
• Real cluster client (monitor/k8sclient.go): reads the DocumentDB CR, watches pods, pulls metrics-server samples — no fakes.
• Health monitor + leak detector (monitor/health.go, monitor/leakdetect.go): steady-state detection plus linear-regression slope on memory/CPU samples.
• Append-only journal (journal/): every disruption window, every workload event, every verdict. The disruption-window oracle is what makes verdicts reproducible.
• Reporter + alerter (report/): writes the longhaul-report ConfigMap so a human can kubectl get cm longhaul-report -o yaml for a single source of truth on pass/fail. Alerts surface in the monitor workflow.
• In-cluster packaging (Dockerfile, deploy/longhaul.yaml): runs as a Deployment (not a Job) — if the process exits, it restarts; the ConfigMap is the durable verdict, not the pod's exit code.
• Two GHA workflows chained via workflow_run:
  • LONGHAUL - Build Test Driver Image — builds and pushes to GHCR.
  • LONGHAUL - Deploy Test Driver to AKS — uses a long-lived ServiceAccount-token kubeconfig (LONGHAUL_KUBECONFIG repo secret) to roll the Deployment.
• LONGHAUL - Monitor — hourly cron that pulls the report ConfigMap and posts a workflow summary; alerts on stale or failing reports.
• Tier-1 unit tests across config, journal, monitor, report, operations — no cluster needed, run with go test ./... from test/longhaul/.
• Docs — docs/designs/long-haul-test-design.md (problem statement, eight failure modes, design principles, architecture, operations catalog with status, env-var table, phase tracker) and test/longhaul/README.md (how to run, what every env var does, why Deployment over Job).

Why this is structured the way it is

• Long-haul is what happens when the failure modes you need to detect require more time and operations than fit in any bounded test run. Memory leaks, CR-history drift, upgrade-under-state failures — these don't speed up. The eight failure modes in the design doc are why every component exists.
• No drain before disruption. Workload runs through scale, upgrade, and (future) failover ops — that's exactly the bug class we're trying to find.
• Deployment, not Job. A Job that finishes is a Job no one looks at. The longhaul-report ConfigMap is the durable verdict, and the monitor cron makes sure a human sees stale runs.
• Single cluster today, two clusters later. A baseline DocumentDB cluster (control group) is in the design but not in this PR — the single-cluster driver already exercises the primary failure modes.

Live canary

I've stood up a long-running cluster — the driver Deployment lives there and gets re-rolled by the deploy workflow on every successful image build:

• AKS cluster: longhaul-aks (longhaul-rg, sub 81901d5e…)
• Everything (DocumentDB CR + driver Deployment + longhaul-report ConfigMap) runs in documentdb-test-ns.
• Useful pokes:
  • kubectl logs deploy/longhaul-test -n documentdb-test-ns
  • kubectl get cm longhaul-report -n documentdb-test-ns -o yaml

What's intentionally out of scope

Tracked in the design doc's phase table:

• Two-cluster topology (Phase 3): primary + baseline, so signals become attributable.
• Backup / restore operations (Phase 2a) and chaos primitives beyond scale (Phase 2d): drain node, kill primary pod, operator restart.
• Operator version upgrade operation: we exercise DocumentDB version upgrades today; operator upgrades will land once the release workflow is wired to bump the canary's target.
• OutagePolicy.AllowedDowntime enforcement: the field is reserved on the struct; current verdicts use AllowedWriteFailures + MustRecoverWithin. Called out explicitly in the design doc.

How to review

If you only want to skim, three files give you the shape of the PR:

1. docs/designs/long-haul-test-design.md — the why.
2. test/longhaul/cmd/longhaul/main.go — the wiring.
3. test/longhaul/operations/scheduler.go — the heart of the disruption loop.

If you want to run it locally:

cd test/longhaul
LONGHAUL_CLUSTER_NAME=documentdb-sample LONGHAUL_NAMESPACE=documentdb-test-ns \
  LONGHAUL_MAX_DURATION=10m \
  go run ./cmd/longhaul

Happy to walk through any area on a call.

[Copilot]

Pull request overview

Adds a long-haul (run-until-failure canary) testing framework to the DocumentDB Kubernetes Operator repo, including a design document and an initial Phase 1a skeleton that is CI-safe by default.

Changes:

• Introduces a long-haul Ginkgo suite with a BeforeSuite enablement gate (LONGHAUL_ENABLED) and a placeholder canary spec.
• Adds a test/longhaul/config package for env-based configuration loading/validation with unit tests.
• Documents the long-haul test architecture and usage via a new design doc and package README.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
operator/src/test/longhaul/suite_test.go	Adds the Ginkgo suite entry point for long-haul tests.
operator/src/test/longhaul/longhaul_test.go	Adds gated BeforeSuite config loading/validation and a placeholder canary spec.
operator/src/test/longhaul/config/suite_test.go	Adds the Ginkgo suite entry point for config unit tests.
operator/src/test/longhaul/config/config_test.go	Adds unit tests for config defaults, env parsing, validation, and enablement gating.
operator/src/test/longhaul/config/config.go	Implements long-haul test configuration (env vars, defaults, validation, enablement).
operator/src/test/longhaul/README.md	Adds usage docs, configuration reference, and CI-safety notes for long-haul tests.
docs/designs/long-haul-test-design.md	Adds the long-haul canary design document and phased implementation plan.

[xgerman]

Review

Verdict: request-changes — lightly. No blocking issues for the skeleton itself; the skip gate is airtight and the config tests are appropriate. But there are four structural items the Phase 1a framing bakes in that will be expensive to unwind in Phase 1b/1c if not settled now.

Verified first: test-unit.yml uses operator/src as working-dir and runs go list ./..., which enumerates the new packages. With LONGHAUL_ENABLED unset the BeforeSuite calls Skip() and the suite registers as a single "skipped" line. Zero CI overhead. ✅

---

🟠 Major

M1. Module placement will pollute operator/src/go.mod — operator/src/test/longhaul/ (entire tree)

The existing test/e2e/ is a separate Go module at test/e2e/go.mod — a deliberate isolation so mongo-driver/v2, CNPG test utilities, and other test-only deps never enter the operator's dependency graph. This PR puts longhaul inside the operator module.

For Phase 1a (stdlib + Ginkgo/Gomega, already in the operator go.mod) this is harmless. For the phases this PR's design describes:

• Phase 1b adds go.mongodb.org/mongo-driver (design line 75)
• Phase 1c adds a file-backed journal
• Phase 1d adds OTel / Prometheus client libs
• Phase 2f adds k8s Job manifests + likely controller-runtime client

…all would land in operator/src/go.mod and be forced onto every operator build. Resolve before the first non-stdlib dep is introduced. Preferred: move to test/longhaul/ with its own go.mod (mirrors e2e). Alternative: nest a go.mod under operator/src/test/longhaul/ and document the convention in CONTRIBUTING.md.

M2. Design does not engage with the existing e2e suite — docs/designs/long-haul-test-design.md throughout

Neither the PR description nor the design doc reference docs/designs/e2e-test-suite.md or test/e2e/. That suite already ships primitives long-haul will need:

• Ginkgo v2 harness with SynchronizedBeforeSuite for cluster bootstrapping (test/e2e/pkg/testenv)
• Label taxonomy (test/e2e/labels.go)
• Run-ID tagging for artifact separation (test/e2e/runid.go)
• envsubst-based manifest rendering
• A mongo-driver v2 wire-protocol client (same driver the design proposes)
• .github/actions/setup-test-environment composite action for cluster provisioning

The "Directory Structure" section (lines 213–244) proposes reinventing monitor/, config/, suite bootstrap, and mongo client from scratch. The design should commit to either reuse — most naturally by making longhaul a sibling module that depends on test/e2e/pkg/ — or explicitly enumerate what it forks and why. Otherwise, six months from now we'll have two drifting implementations of "connect to a DocumentDB CR and open a mongo client" with different flag names, different retry policies, and different label conventions.

M3. Four design gaps that will cost rework, not captured in §Open Questions — long-haul-test-design.md:443-445

1. Event journal durability across process/pod restarts. Line 239 says "In-memory + file-backed" but never says where the file lives. If the canary runs as a Kubernetes Job (line 305), the pod's ephemeral filesystem disappears on restart — and Phase 2e explicitly plans for operator restart / pod eviction chaos. The journal is the only post-mortem artifact and it will be destroyed by the very events it is supposed to record. Commit to a PVC mount or an external sink (object storage, Loki) in the design.
2. Workload resumption idempotency. Writers use a unique index on (writer_id, seq) (line 80). If a writer or the Job pod restarts, where does seq restart from? Reset to 0 → writes collide. Bootstrap from max(seq) per writer_id → the oracle must tolerate a gap between crash and resume without flagging data loss. Neither path is described.
3. Teardown on abort. "On failure: collect artifacts, preserve cluster state" (line 188) handles the fatal path. But Ctrl-C, CI cancellation, or Job preemption mid-run leaves writer data in the target DB, possibly an in-progress restore cluster (Phase 2a), possibly a half-scaled topology (Phase 1e). No cleanup hooks, resumable state files, or leftover-run detector.
4. Latency-regression baseline policy. Failure tiers list "latency regression" as a warning (line 197), but no baseline/threshold policy is defined. List this as an Open Question at minimum so Phase 1b doesn't invent a threshold policy in a PR review thread.

These shape the Phase 1b/1c interfaces and should be addressed (or explicitly deferred in §Open Questions) before the design doc merges.

M4. MaxDuration=30m default contradicts "run until failure" — config.go:594, long-haul-test-design.md:37,188

The design repeatedly commits to the Strimzi run-until-failure canary model. The Phase 1a default is 30m; the env doc (line 259) says 0s opts into infinite. Fine for local development — exactly wrong for the Phase 2f canary Job. Pick one: either (a) document that the canary Job manifest will set LONGHAUL_MAX_DURATION=0s explicitly and mention this in the README "CI Safety" section, or (b) flip the default to 0 and require local-dev to pass LONGHAUL_MAX_DURATION=10m.

---

🟡 Minor (cheap in-PR fixes)

• m1 IsEnabled() — config.go:637-640 — doesn't trim whitespace. LONGHAUL_ENABLED=" true " → false (silent skip). Add strings.TrimSpace. Silent-failure mode; worth fixing in-PR.
• m2 Validate() — config.go:625-633 — accepts negative MaxDuration. time.ParseDuration("-5m") succeeds. Add if c.MaxDuration < 0 { return fmt.Errorf(...) }.
• m3 Error strings capitalized — config.go:627,630 — ST1005 / Go convention: lowercase. Matching tests use ContainSubstring("Namespace") / "ClusterName" and will need adjustment.
• m4 Path inconsistency — long-haul-test-design.md:340 says kubectl apply -f test/longhaul/deploy/job.yaml but the directory structure roots at operator/src/test/longhaul/. Fix path, or resolve M1 first.
• m5 README overstates CI safety — README.md:543-547 — phrase as "no CI workflow currently sets LONGHAUL_ENABLED; do not set it in any PR-gated workflow" rather than as a property of the code.
• m6 config is a very generic package name. If Phase 2 ever imports the operator's own config, it will shadow. Consider longhaulcfg or cfg. Not blocking.
• m7 Missing IsEnabled test cases — config_test.go:727-757 — given the silent-skip failure mode, lock it down: leading/trailing whitespace, mixed case ("True", "YES"), "0" / "no" / "false" → false.

🟢 Nit

• longhaul_test.go:799 — package-level mutable testConfig. Fine for a single-file suite; revisit if Phase 1b splits files.
• DefaultConfig() by value vs. Validate() pointer receiver — slight asymmetry. Leave it.
• README line 509's -timeout 0 guidance is correct and rarely documented — nice touch.

---

Positive feedback

• Skip gate is correctly placed in BeforeSuite, not BeforeEach — single skipped line instead of N per spec. Exactly right.
• LoadFromEnv cleanly separates parsing from validation, so Validate() can be called on a programmatically-built Config in Phase 1b unit tests without fighting env vars.
• The design doc's survey of Strimzi / CloudNative-PG / CockroachDB / Vitess and the "adopt / skip" table (lines 354–361) is unusually high-quality framing for an OSS design doc.
• Phased breakdown (Phase 1a → 3) is appropriately granular — each phase is demoable and reviewable.
• Test coverage on the config package is thorough for the surface area.

---

Questions for the author

1. (M1) Any reason longhaul can't live at test/longhaul/ as a sibling module to test/e2e/? If there is one, please capture it in the design.
2. (M2) Have you looked at test/e2e/pkg/ for reusable primitives (typed clients, envsubst, runid, labels)? What's the plan for avoiding a second mongo-client implementation?
3. (M3.1) For the journal: PVC, external object store, or stdout-scraped-by-Loki?
4. Issue #220 text isn't quoted in the PR — are there SLO targets, acceptance criteria, or a required timeframe (e.g., 72h)? Restating them in the design's Problem Statement would make future reviews easier.
5. Should CONTRIBUTING.md or AGENTS.md get a one-liner pointing at operator/src/test/longhaul/README.md so contributors aren't left wondering "what is this tree?"

---

Reviewed with the GitHub Copilot code-review agent.

[WentingWu666666]

Response to @xgerman's Review

Thank you for the thorough review! I've addressed all items. Here's a summary:

---

Major Items

M1 (Module placement): Agreed extracted long-haul tests to test/longhaul/ at the repo root as a separate Go module (github.com/documentdb/documentdb-operator/test/longhaul). This keeps Ginkgo/Gomega and any future test-only deps completely out of the operator's runtime go.mod. Follows the three-directory layout described in the updated design doc.

M2 (E2E engagement): Designed a three-directory layout for test code reuse:

• test/utils/ shared utilities (future extraction from PR docs(design): unified Go/Ginkgo E2E test suite #346)
• test/e2e/ E2E test suite (PR docs(design): unified Go/Ginkgo E2E test suite #346)
• test/longhaul/ long-haul canary tests (this PR)

Each directory has its own go.mod with replace directives. The design doc now includes a package-to-use mapping table showing which shared utils each long-haul phase will reuse (mongo client, assertions, DocumentDB CR helpers, etc.). Extraction of shared packages from test/e2e/pkg/e2eutils/ to test/utils/ will be coordinated with @xgerman in a follow-up.

M3 (Design gaps): Upgraded from Open Questions to Provisional Design Decisions in the design doc:

1. Journal durability: PVC-backed file at /data/journal/, structured JSON lines, on-startup reconstruction from existing file.
2. Writer seq resumption: Bootstrap from max(seq) per writer_id in DB. Oracle tolerates crash-recovery gaps (logged, not flagged as data loss).
3. Teardown on abort: Signal handler for SIGTERM/SIGINT cancel contexts flush journal write final status to ConfigMap exit. Leftover-run detector on startup.
4. Latency baseline: Kept as Open Question establish P99 baseline during first 30min warmup, alert on >2 sustained regression.

These lock the approach for Phase 1b/1c interfaces while leaving implementation details to those PRs.

M4 (MaxDuration default): Added notes in both the design doc (Configuration section) and README (CI Safety section) that the persistent canary Job manifest explicitly sets LONGHAUL_MAX_DURATION=0s. The 30m default remains as a safety net for local development.

---

Minor Items All Fixed

• m1: Added strings.TrimSpace to IsEnabled()
• m2: Added negative MaxDuration validation in Validate()
• m3: Lowercased error strings (Go convention ST1005)
• m4: Fixed path: test/longhaul/deploy/job.yaml
• m5: Rephrased CI safety: "No CI workflow currently sets this variable do not add it to any PR-gated workflow"
• m6: Valid concern. Keeping config for Phase 1a (no shadowing risk yet). Will revisit if the operator's own config needs importing in later phases.
• m7: Added 8 new test cases: whitespace (" true ", " yes "), whitespace-only (" "), mixed case ("True", "YES"), explicit false values ("0", "no", "false"). Total: 23 specs (was 15).

---

Questions

1. (M1) Resolved extracted to separate Go module at test/longhaul/.
2. (M2) Resolved three-directory layout with shared utils plan documented.
3. (M3.1) PVC-backed file journal documented as provisional design decision.
4. (Issue Set up long haul test cluster #220) The design's Problem Statement already quotes the three requirements from Set up long haul test cluster #220. SLO targets remain as Open Question Adding Microsoft SECURITY.MD #2 they need team input.
5. (CONTRIBUTING.md/AGENTS.md) Good idea. Will add a one-liner in a follow-up PR to avoid scope creep here.

---

Test results: 23 config specs + 1 canary spec (skipped) = all passing. go vet clean.

[hossain-rayhan]

The plan looks good.
I have one concern about the test restart policy.

We should keep running parallel tests for different combination. Like, operator n version with gateway/extension n-1 version. Operator n-2 version with keeps working fine with latest non-schema changes.

At least, different operator version tests, not only the latest release restarts the tests.

[WentingWu666666]

Thanks for the feedback @hossain-rayhan! Great point about version compatibility.

Since the operator and database images follow independent version tracks, we agree cross-version testing matters e.g., upgrading the operator while the database is still on the previous version, and vice versa.

That said, we think this is better suited for the E2E test suite rather than long-haul. Cross-version compat bugs surface immediately on upgrade (deterministic, finite), while long-haul focuses on time-dependent issues memory leaks, connection pool exhaustion, replication drift that only appear after hours/days of sustained operation.

We'll make sure the E2E suite covers both operator-first and database-first upgrade paths with version N/N-1 combinations.

[WentingWu666666]

@xgerman thanks again for the detailed review all items from your latest review are now addressed. Quick disposition summary:

Critical

1. ScaleUp/Down now patch spec.instancesPerNode (not nodeCount).
2. Env vars renamed to LONGHAUL_MIN_INSTANCES / LONGHAUL_MAX_INSTANCES everywhere (code, README, workflow).
3. Unit tests added for all 6 packages (config, journal, monitor, operations, report, workload) commit 463c0ae.
4. Upgrade auto-skips when instancesPerNode < 2 (operations/upgrade.go Precondition).

Major
5. Verifier resumes from nextSeq instead of full re-scan every cycle.
6. Writer now treats mongo.IsDuplicateKeyError as success (idempotent retry).
7. test/longhaul/go.mod aligned to go 1.25.9 to match operator/e2e needed downgrading k8s.io deps from v0.36.0 v0.35.0 since v0.36 requires Go 1.26+ (commit e17fcf3).
8. OutagePolicy.AllowedDowntime now has a TODO + enforcement skeleton in journal/policy.go.

Suggestions / Nitpicks
9. waitForImage now logs polling errors instead of swallowing them.
10. Acknowledged scheduler poll cadence kept as-is.
11. Out of scope for this PR (cross-module API pin).
12. Removed unused ClusterHealth.WritesHealthy.
13. setup.yaml header now references deployment.yaml.
14. setup.yaml defaults to instancesPerNode: 2.
15. EmitAnnotation uses Round(time.Second).
16. Declined keeping timestamp-based payload to preserve write-time observability.
17. Suggestion only workflow secret format unchanged.

Verified locally: go build ./... and go test ./... from test/longhaul/ are green. Ready for re-review when you have a moment.

[xgerman]

Review: long-haul harness vs. existing test/e2e/pkg/e2eutils/

Verdict: the PR does not leverage the existing e2e helpers and ships meaningful duplication of the DocumentDB CR contract and Mongo connection plumbing.

Root cause

test/longhaul/go.mod is a standalone module with no replace github.com/documentdb/documentdb-operator => ../../operator/src directive and no dependency on the operator or e2e module. As a result, longhaul cannot import previewv1 or test/e2e/pkg/e2eutils/, and reimplements them via raw dynamic.Interface + unstructured. test/e2e/go.mod:110 has the replace directive that makes this work for the e2e suite.

Concrete duplications

1. test/longhaul/monitor/k8sclient.go duplicates test/e2e/pkg/e2eutils/documentdb/documentdb.go

Longhaul (raw dynamic)	E2E (typed)
K8sClusterClient.ScaleCluster (k8sclient.go:178)	documentdb.PatchInstances (documentdb.go:299)
K8sClusterClient.UpgradeDocumentDB (k8sclient.go:221)	documentdb.PatchSpec — generic mutator (documentdb.go:325)
K8sClusterClient.GetInstancesPerNode (k8sclient.go:155)	documentdb.Get + field access (documentdb.go:428)
K8sClusterClient.GetCurrentDocumentDBImageTag (k8sclient.go:200)	documentdb.Get + field access
K8sClusterClient.GetClusterHealth — status == "Cluster in healthy state" hardcoded at k8sclient.go:148	documentdb.ReadyStatus constant (documentdb.go:68)
Polling-for-healthy in HealthMonitor.check (health.go:93)	documentdb.WaitHealthy (documentdb.go:346)

The hardcoded readiness string is the most concerning item: if the operator ever changes its ReadyStatus, e2e picks it up automatically; longhaul silently breaks.

2. cmd/longhaul/main.go mongo connection duplicates test/e2e/pkg/e2eutils/mongo/

• main.go does raw mongo.Connect(options.Client().ApplyURI(cfg.MongoURI)) plus a manual ping.
• e2e already exports mongo.NewClient, mongo.BuildURI, and mongo.NewFromDocumentDB — the last reads the documentdb-credentials Secret, opens a port-forward when needed, applies TLS, and retries pings.
• Longhaul sidesteps the credentials Secret by requiring an externally-injected LONGHAUL_MONGO_URI env var via a separate longhaul-mongo-credentials Secret (deploy/setup.yaml). That's an operational workaround, not a code-reuse decision — mongo.BuildURI and the credentials-secret reader still apply even when the cluster is LB-exposed.

3. GVR hardcoding

monitor/k8sclient.go:78-82 hardcodes {Group: "documentdb.io", Version: "preview", Resource: "dbs"}. The typed client encodes the same fact via AddToScheme and would never drift. (Note the resource is dbs, not the more obvious documentdbs — exactly the kind of detail you only get wrong once when you bypass the scheme.)

4. Operator pod health monitoring

monitor/health.go ad-hoc counts pod readiness. test/e2e/pkg/e2eutils/operatorhealth/gate.go already encapsulates "snapshot operator pod identity + restart count, detect churn" — the exact primitive needed to distinguish "operator died" from "operator was upgraded mid-run".

Not duplicated (longhaul-specific, fine as-is)

workload/verifier.go, workload/writer.go, monitor/leakdetect.go, journal/, operations/scheduler.go, report/report.go are genuinely longhaul-only — keep them where they are.

Recommendations (ranked)

1. Add the operator replace directive and an e2eutils dependency to test/longhaul/go.mod, or extract e2eutils/{documentdb,mongo,operatorhealth,portforward} into a test/shared/ module that both test/e2e/ and test/longhaul/ consume. The shared-module option is cleaner long-term because it avoids importing Ginkgo into a non-Ginkgo binary.
2. Replace the CR operations in monitor/k8sclient.go with documentdb.Get / PatchInstances / PatchSpec / WaitHealthy, and reference documentdb.ReadyStatus instead of the literal string. operations/scale.go and operations/upgrade.go consume the monitor.ClusterClient interface, so this is an internal refactor and won't ripple.
3. Replace mongo.Connect in cmd/longhaul/main.go with mongo.NewFromDocumentDB (or mongo.NewClient + mongo.BuildURI when an external URI is genuinely required).
4. Lower priority — even if (1) is rejected as too invasive: at minimum expose ReadyStatus from a small shared constants package and import it. Do not duplicate the magic string.

The harness itself is good work; this is a maintenance-liability call, not a behavioural one. Worth resolving before merge so we don't end up with two source-of-truths for "how to talk to a DocumentDB CR".