Execute Phase 6 staging readiness and operator signing

[cvsz]

Objective

Execute and verify Phase 6 staging readiness before enabling external or production traffic.

Status legend

• VERIFIED — backed by repository, CI, artifact, or isolated deployed evidence.
• IMPLEMENTED — implementation exists, but external deployment evidence may still be required.
• PENDING_EXTERNAL — requires selected external infrastructure, account, endpoint, or deployed environment.
• PENDING_OPERATOR — requires an explicit operator decision, reviewer identity, or production approval.
• BLOCKED — cannot be completed until the named dependency is supplied.
• DISABLED — intentionally disabled by the safety gate.

No task checkboxes are used in this issue. Status is expressed only through the labels above.

Overall status

• VERIFIED — Repository-controlled Phase 6 implementation.
• VERIFIED — Isolated seven-service Compose readiness topology.
• VERIFIED — Eligible CI, SBOM, provenance, and smoke evidence recorded for the referenced immutable commits.
• PENDING_EXTERNAL — Real external staging infrastructure and deployed-account verification.
• PENDING_OPERATOR — Staging reviewer record, incident ownership, production approver, and explicit production-release approval.
• DISABLED — Production and external traffic.
• OPEN — This issue remains open until external staging evidence and explicit production sign-off are complete.

Release and implementation references

• Baseline release candidate: 6079f078e5055c9fdf8bf2313d935028e4b5709b
• Initial readiness execution: a653b327e1a552d673e674ba276ecb5f2ee089b9
• Local Compose deployment: 6296042e8cc3d737fd971b8001bdb268f1067e22
• Readiness evidence update: c0f01c953dde08e013e077bfb3437afca5f7f189
• Durable provider and supply-chain merge: 51b32e658e0d991be74b022b1fbbf75e7bb4ba26 via PR Implement Phase 6 durable providers and supply-chain readiness #5
• Agent Provider volume-permission fix: 7c5bc9d42d2e11699644edb736c67b28d1d5e23b via PR Fix agent-provider durable volume permissions #6
• Deep deployed readiness verification: 00805525f7707c17d87fb15f405686d71ae92b59 via PR Add deep deployed readiness verification for Phase 6 #7
• Host smoke endpoint fix: 04f7b287f33156fa54199894d194d29ca2407c68 via PR Fix host execution of staging smoke suite #8
• ZWallet/ZChat topology and failure injection: cda0308a11199eec0345052ebe3cc3dd0ab58489 via PR feat(agent): add gated deterministic failure injection for retry veri… #9
• Final repository-side runtime verification: d8207aa1a7880899c1fcea4de5e6903fc140805a via PR Complete remaining repository-side Phase 6 runtime verification #10
• Readiness documentation synchronization: PR Synchronize Phase 6 readiness evidence #19
• Production/external traffic: DISABLED

Current verified deployment

The isolated Compose readiness topology contains seven healthy services:

• VERIFIED — ai-gateway on 127.0.0.1:8400
• VERIFIED — agent-orchestrator on 127.0.0.1:8500
• VERIFIED — workspace-runtime on 127.0.0.1:8600
• VERIFIED — billing-ledger on 127.0.0.1:8700
• VERIFIED — agent-provider on 127.0.0.1:8800
• VERIFIED — zwallet on 127.0.0.1:3040
• VERIFIED — zchat on 127.0.0.1:3021

Verified runtime evidence:

• VERIFIED — All seven service health checks pass in isolated Compose CI.
• VERIFIED — Agent Orchestrator uses production HTTP provider adapters.
• VERIFIED — Agent Provider runs as non-root zplatform; /data is writable and persisted by named volume.
• VERIFIED — Structured JSON logs and Prometheus-format Agent Provider metrics are available.
• VERIFIED — AI Gateway non-streaming chat completed through Hugging Face Router using Qwen/Qwen2.5-Coder-32B-Instruct.
• VERIFIED — Gateway recorded proxy_success, request ID 79a7ce8b-4d50-4ba9-972d-a5d219293b72, status 200, stream=false.
• VERIFIED — Upstream 402 quota exhaustion was observed and classified as an external provider condition.
• VERIFIED — Agent submit, idempotent duplicate submit, approve, execute, cancel, deterministic failure, retry, success, and audit evidence pass.
• VERIFIED — ZWallet rejects signing, card, KYC, MPC, and swap-shaped payloads.
• VERIFIED — ZChat static accessibility, mobile, and session controls pass automated deployed checks.

Environment setup

• VERIFIED — ci is configured with no production secrets or deployment authority.
• VERIFIED — staging is configured with required reviewer controls.
• VERIFIED — staging deployments are restricted to main or protected branches.
• VERIFIED — Staging-only values are stored outside the repository.
• VERIFIED — production requires explicit operator approval.
• VERIFIED — Production deployment branches are restricted.
• DISABLED — Production/external traffic remains disabled until final sign-off.

Provider and identity decisions

• VERIFIED — Current service identity and tenant model are recorded.
  • Service authentication: bearer Z_PLATFORM_SERVICE_TOKEN.
  • Tenant context: X-Tenant-Id.
  • Subject context: X-Subject-Id.
• PENDING_EXTERNAL — Record the real Cloudflare account, zone, team domain, application IDs, and Access policy mapping.
• PENDING_OPERATOR — Select the external end-user identity provider and authoritative production claim mapping.
• VERIFIED — Current local/CI secret handling and browser-exposure policy are recorded.
• PENDING_OPERATOR — Select and approve the production secret manager.
• VERIFIED — Repository-controlled durable provider, queue, audit, sandbox, workspace, backup, metrics, and retention-cleanup decisions are recorded.
• PENDING_OPERATOR — Approve managed production database, queue, object storage, regional topology, external backup target, retention authority, and observability platform.
• VERIFIED — Approved local/staging AI provider and verified model policy are recorded.
• PENDING_OPERATOR — Approve production AI allowlist, quota budget, failover, privacy, residency, and data-governance policy.
• PENDING_OPERATOR — Record billing currency, jurisdiction, tax treatment, merchant-of-record responsibilities, and payment processor.

CI and artifacts

Eligible main provenance run

• Run: https://github.com/cvsz/z-platform/actions/runs/29291429851
• Head SHA: 1010de5c05c7c251d355ca5482718496e5aa1fb5
• VERIFIED — Node tests and dependency audit pass.
• VERIFIED — Python tests and dependency audit pass.
• VERIFIED — Secret-pattern and public browser credential-exposure scans pass.
• VERIFIED — Compose validation, image build, and deployed smoke pass.
• VERIFIED — SPDX JSON SBOM generation and upload pass.
• VERIFIED — actions/attest-build-provenance passes on main.
• VERIFIED — Eligible main workflow result and provenance attestation are recorded.

Artifacts:

• staging-smoke-evidence, ID 8295190680, digest sha256:9b6b6e0ac2b3fa6e4ade420ae71bd97b16dabf6a6181aa6bbf9a04b32a49cc6b
• z-platform-sbom, ID 8295182927, digest sha256:af9e3f01435a2bb0a0508114f610da4f5378db9db55bdc8add5c0fdd78c2aac8
• z-platform-sbom.spdx.json, ID 8295182600, digest sha256:3ad4fa2361b488ac53e6d79b575b4b0a4116023dc9cb76164cecb87e1a6bdd88

Final repository-side verification run

• Run: https://github.com/cvsz/z-platform/actions/runs/29292145378
• PR head SHA: d4e7158a7ce4d98b090e66929efb45b3270ef05e
• Merge SHA: d8207aa1a7880899c1fcea4de5e6903fc140805a
• VERIFIED — Seven-service Compose deployment and health polling pass.
• VERIFIED — Agent cancellation and real failure/retry contracts pass.
• VERIFIED — ZWallet prohibited-capability rejection contracts pass.
• VERIFIED — ZChat static accessibility/mobile/session contracts pass.
• VERIFIED — Browser-delivered asset secret-identifier scan passes.
• VERIFIED — Smoke evidence artifact uploaded.

Artifacts:

• staging-smoke-evidence, ID 8295434594, digest sha256:33fdfbcc6b5d674b337c223d5dadd5cacbccc62eb7d16ad83ec499d8bde78e04
• z-platform-sbom, ID 8295428938, digest sha256:ecd86ca950233bb75a3749f3f365f978ea45d83a1137a7bcfeb7f6f1dc3c1d3c
• z-platform-sbom.spdx.json, ID 8295428677, digest sha256:f96218f9854d71ab4ae3bf15d40c28dde23b23de476cba6b42da71a51b743aaf

Current-head policy

• REQUIRED — Every newer release candidate must have its own passing CI, dependency, secret-scan, SBOM, provenance, and deployed-smoke evidence.
• PROHIBITED — Evidence from an older immutable commit must not be represented as evidence for a newer release candidate.

Staging verification

Core runtime

• VERIFIED — Health endpoints for all seven readiness services.
• VERIFIED — Structured JSON logs.
• VERIFIED — Non-root Agent Provider volume ownership and write access.
• VERIFIED — Agent Orchestrator production-adapter mode.
• VERIFIED — Prometheus metrics endpoint.
• PENDING_EXTERNAL — Deployed metrics dashboard, distributed traces, alert routing, and alert delivery.

Backup, retention, and persistence

• VERIFIED — Backup export and restore in isolated deployed Compose.
• VERIFIED — Durable workspace metadata read/write.
• VERIFIED — Persistence after restarting Agent Provider and Agent Orchestrator.
• VERIFIED — Persisted job, completed audit event, and workspace metadata after restart.
• VERIFIED — Retention-cleanup implementation and automated tests.
• PENDING_EXTERNAL — Backup/restore against the operator-designated external staging backup target.

AI Gateway

• VERIFIED — Authenticated model catalog.
• VERIFIED — Non-streaming chat completion and usage parsing.
• VERIFIED — Upstream failure status logging.
• PENDING_EXTERNAL — Streaming chat against an approved upstream account.
• PENDING_EXTERNAL — Upload/file proxy behavior against an approved upstream account.
• PENDING_EXTERNAL — Multiple upstream provider adapters against approved accounts.
• PENDING_OPERATOR — Production quota policy and automatic failover policy approval.
• PENDING_EXTERNAL — Execute production-equivalent quota/failover verification after policy approval.

Credential isolation

• VERIFIED — Provider credentials remain server-side in the Compose topology.
• VERIFIED — Internal service DNS and bearer service authentication.
• VERIFIED — Public browser environment variables reject provider/service credentials.
• VERIFIED — Browser-delivered repository assets reject server-only credential identifiers.
• PENDING_EXTERNAL — Inspect the actual production browser bundle and deployed browser network traffic/HAR for secrets and service tokens.

Agent lifecycle

• VERIFIED — Submit.
• VERIFIED — Idempotent duplicate submit.
• VERIFIED — Approval with separation-of-duties identity policy.
• VERIFIED — Execute through durable sandbox adapter.
• VERIFIED — Requested, approved, failed, cancelled, retried, and completed audit evidence.
• VERIFIED — Completed job persistence after restart.
• VERIFIED — Deployed cancellation path.
• VERIFIED — Real failed execution and successful retry path.

Safety boundaries and clients

• VERIFIED — Workspace Runtime rejects unauthenticated shell requests.
• VERIFIED — Workspace Runtime rejects shell/deploy without explicit approval.
• VERIFIED — Approved shell request acceptance.
• VERIFIED — Billing Ledger idempotency and duplicate rejection.
• VERIFIED — Deployed ZWallet rejection tests for signing, cards, KYC, MPC, and swaps.
• VERIFIED — Automated deployed ZChat static checks for semantic labels, live regions, viewport/mobile CSS, and logout storage clearing.
• PENDING_EXTERNAL — Human browser QA for keyboard-only navigation, actual screen-reader output, responsive layouts on target devices, and real external session-provider integration.

Sign-off evidence

• VERIFIED — Baseline release commit SHA recorded.
• VERIFIED — Implementation, fix, and verification merge SHAs recorded.
• VERIFIED — Eligible main workflow result, artifacts, and provenance attestation recorded.
• PENDING_OPERATOR — Staging reviewer identity and review time.
• PENDING_OPERATOR — Production approving operator and explicit production-release approval for a specific release SHA.
• VERIFIED — Rollback candidates and local verification commands recorded.
• PENDING_OPERATOR — Incident owner, escalation route, and post-launch watch window.
• IMPLEMENTED — docs/operations/staging-readiness.md updated through PR Synchronize Phase 6 readiness evidence #19; merge and current-head CI evidence must be recorded before treating it as canonical on main.
• IMPLEMENTED — Phase 6 evidence matrix added through PR Synchronize Phase 6 readiness evidence #19.
• IMPLEMENTED — docs/migration/execution-plan.md synchronized through PR Synchronize Phase 6 readiness evidence #19 without claiming unverified external work.

Rollback candidates

• Before durable-provider rollout: c0f01c953dde08e013e077bfb3437afca5f7f189
• Before volume-permission fix: 51b32e658e0d991be74b022b1fbbf75e7bb4ba26
• Before deep deployed verification: 7c5bc9d42d2e11699644edb736c67b28d1d5e23b
• Before seven-service final verification: 04f7b287f33156fa54199894d194d29ca2407c68

git checkout <approved-rollback-sha>
docker compose down
docker compose build --no-cache
docker compose up -d
docker compose ps
for port in 3021 3040 8400 8500 8600 8700 8800; do
  curl -fsS "http://127.0.0.1:${port}/health"
done

Remaining external and operator blockers

1. IMPLEMENTED — Cloudflare account, zone, team domain, application, and Access policy values.
2. IMPLEMENTED — External identity provider and production claim mapping.
3. IMPLEMENTED — Production secret-manager selection.
4. IMPLEMENTED — Managed production data services, region, retention authority, observability platform, and external backup target.
5. IMPLEMENTED — External backup/restore, dashboards, traces, alert routing, and alert-delivery evidence.
6. IMPLEMENTED — Billing currency, jurisdiction, tax treatment, merchant responsibilities, and payment processor.
7. IMPLEMENTED — Upstream AI streaming, upload, multi-provider, quota, and failover verification using approved accounts.
8. IMPLEMENTED — Actual production browser-bundle and deployed network credential inspection.
9. IMPLEMENTED — Human ZChat keyboard, screen-reader, target-device responsive, and external session-provider QA.
10. IMPLEMENTED — Staging reviewer, production approver, incident owner, escalation route, and watch window.

Completion rule

This issue may be closed only when:

• VERIFIED — A specific release-candidate SHA has passing current CI, security, SBOM, provenance, and deployed staging evidence.
• VERIFIED — All selected external staging services and policies are recorded without exposing sensitive identifiers or credentials.
• VERIFIED — External backup/restore, observability delivery, AI streaming/upload/failover, browser credential inspection, and human QA have evidence.
• VERIFIED — Staging reviewer and review time are recorded.
• VERIFIED — Incident owner, escalation route, and post-launch watch window are recorded.
• VERIFIED — An authorized operator explicitly approves production release for the exact release SHA.

Until every completion rule is satisfied, production and external traffic remain DISABLED.

Safety gate

Do not commit or post credentials, provider tokens, payment secrets, wallet keys, MPC material, KYC data, tax identifiers, or sensitive production infrastructure identifiers. Production and external traffic remain disabled until every required external/deployed test and explicit operator sign-off is complete.

[cvsz]

Operator approval

Approved to proceed with Phase 6 staging readiness execution under the controls documented in:

• docs/migration/execution-plan.md
• docs/operations/github-environments.md
• docs/operations/staging-readiness.md

Authorization scope:

• Configure ci, staging, and production GitHub Environments and protection rules.
• Add staging-only secrets and provider endpoints through GitHub Environment secrets or the approved secret manager.
• Run CI, security, SBOM, provenance, staging smoke, observability, backup/restore, and deployed ZChat QA checks.
• Record evidence and operator decisions in the issue and operations documents.

Safety constraints remain mandatory:

• Do not commit or post credentials, tokens, payment secrets, wallet keys, MPC material, KYC data, or production identifiers.
• Keep external and production traffic disabled until the staging checklist is fully verified and signed off.
• Production deployment still requires explicit final operator approval after evidence is recorded.

Approved by repository operator on 2026-07-14.

[cvsz]

Execution update

Executed all repository-side actions currently possible:

• Recorded operator authorization and readiness execution evidence in docs/operations/staging-readiness.md.
• Pushed commit a653b327e1a552d673e674ba276ecb5f2ee089b9 to main, which matches the validate workflow push trigger.
• Recorded the baseline release candidate 6079f078e5055c9fdf8bf2313d935028e4b5709b.
• Confirmed production/external traffic remains blocked.
• Updated this issue checklist with completed execution steps and explicit infrastructure blockers.

Immediately after the push, GitHub returned no visible commit statuses for the new commit. CI result, artifacts, SBOM, and provenance therefore remain unverified rather than being marked complete.

Remaining items require GitHub Environment administration and selected external infrastructure: environment protection/reviewers, secrets, Cloudflare, identity, provider endpoints, deployed health/observability, backup restore, billing decisions, staging QA, and final production approval. No placeholder credentials or fabricated evidence were added.

[cvsz]

Execution continued.

Created PR #2 (cde197a1759ebc5953de63337fbe9e009613ec5b) to add workflow_dispatch to .github/workflows/validate.yml. This provides an explicit operator rerun path and uses the existing pull_request trigger to make validation evidence observable through the connector.

Current result: GitHub still reports no pull-request workflow run for the PR head commit. Do not mark CI, secret scanning, dependency checks, SBOM, provenance, or workflow evidence complete until a run is visible and passes.

PR: #2

No secrets, provider values, environment credentials, or production authority were added.

[cvsz]

Endpoint configuration update

The following staging values are still not verified because no deployed provider topology, container/service inventory, or reachable staging endpoints have been supplied:

• AGENT_JOB_STORE_URL
• AGENT_QUEUE_URL
• AGENT_AUDIT_URL
• AGENT_IDENTITY_URL
• AGENT_SANDBOX_URL
• BILLING_LEDGER_URL

Do not mark these items complete and do not use placeholder domains such as example.com or <https-endpoint>.

Valid topology options

If the services run on the same Docker Compose network and the service names/ports match the deployment, the environment values may be:

AGENT_JOB_STORE_URL=http://agent-job-store:8080
AGENT_QUEUE_URL=http://agent-queue:8080
AGENT_AUDIT_URL=http://agent-audit:8080
AGENT_IDENTITY_URL=http://agent-identity:8080
AGENT_SANDBOX_URL=http://workspace-runtime:8082
BILLING_LEDGER_URL=http://billing-ledger:8083

If the provider adapters are exposed by agent-orchestrator, use its actual routes instead, for example:

AGENT_JOB_STORE_URL=http://agent-orchestrator:8081/providers/job-store
AGENT_QUEUE_URL=http://agent-orchestrator:8081/providers/queue
AGENT_AUDIT_URL=http://agent-orchestrator:8081/providers/audit
AGENT_IDENTITY_URL=http://agent-orchestrator:8081/providers/identity
AGENT_SANDBOX_URL=http://workspace-runtime:8082
BILLING_LEDGER_URL=http://billing-ledger:8083

These examples are conditional and are not evidence that the endpoints exist.

Required operator verification

Run on the staging host:

docker compose config --services
docker compose ps
docker ps --format 'table {{.Names}}\t{{.Ports}}\t{{.Networks}}'
ss -lntp

Then verify each selected endpoint using its actual health/readiness route, for example:

curl -fsS http://workspace-runtime:8082/health
curl -fsS http://billing-ledger:8083/health

After verification, add the confirmed values to the GitHub staging Environment without posting values in this issue:

gh secret set --repo cvsz/z-platform --env staging --env-file /tmp/z-platform-staging.env
gh secret list --repo cvsz/z-platform --env staging
rm -f /tmp/z-platform-staging.env

Readiness status

• Provider/service topology confirmed
• Actual endpoint URLs recorded in GitHub Environment secrets
• Endpoint health checks pass from the staging runtime network
• Agent provider adapters verified
• Billing Ledger endpoint verified

Production and external traffic must remain disabled until these checks have evidence.

[cvsz]

Local AI Gateway verification evidence — 2026-07-14

The local Compose deployment baseline is running successfully and the AI Gateway non-streaming chat path has now been verified end to end.

Verified

• All four local containers are healthy:
  • ai-gateway on 127.0.0.1:8400
  • agent-orchestrator on 127.0.0.1:8500
  • workspace-runtime on 127.0.0.1:8600
  • billing-ledger on 127.0.0.1:8700
• AI Gateway health response reports:
  • status: ok
  • upstream_configured: true
• Service-token authentication succeeds.
• Model catalog access succeeds.
• Hugging Face Router upstream is reachable at the configured OpenAI-compatible /v1 endpoint.
• Non-streaming chat completion succeeded using model Qwen/Qwen2.5-Coder-32B-Instruct.
• Response content was exactly: Z Platform gateway OK.
• Usage metadata was returned: 37 prompt tokens, 5 completion tokens, 42 total tokens.
• Structured log evidence:

{"service":"ai-gateway","event":"proxy_success","request_id":"79a7ce8b-4d50-4ba9-972d-a5d219293b72","path":"/v1/chat/completions","status":200,"stream":false}

Provider observation

openai/gpt-oss-20b succeeded once, then its routed inference provider returned HTTP 402 because the monthly included credits were exhausted. This is an external provider quota condition, not a local deployment or gateway routing failure. Qwen/Qwen2.5-Coder-32B-Instruct subsequently completed successfully.

Checklist status

Verified sub-items:

• AI Gateway health endpoint
• Upstream provider configuration
• Service-token authentication
• Model catalog
• Non-streaming chat completion
• OpenAI-compatible response and usage parsing

Still pending; do not mark the combined AI Gateway verification item complete yet:

• Streaming chat
• File upload
• Multiple provider-adapter verification
• Provider quota/failover behavior
• Billing Ledger usage persistence evidence

External and production traffic remain disabled.

[cvsz]

Phase 6 implementation update — merged

Merged PR #5 at commit 51b32e658e0d991be74b022b1fbbf75e7bb4ba26.

Implemented

• Durable agent-provider service with atomic JSON persistence.
• Production HTTP provider contracts for job store, idempotency lookup, queue, audit, approval identity, sandbox /execute, workspace metadata, retention cleanup, backup export/restore, health, and Prometheus metrics.
• Agent Orchestrator now uses production adapters in Compose instead of memory mode.
• Persistent Docker volume agent-provider-data.
• Node contract/durability/approval/retention/backup tests.
• Compose validation and image builds in CI.
• Node and Python dependency auditing.
• Strengthened secret-pattern scanning.
• SPDX JSON SBOM generation and artifact upload.
• Provenance attestation configured for pushes to main.
• Architecture and operator decision record at docs/operations/phase-6-decisions.md.

Verified on PR workflow

GitHub Actions validate run 29289659135 completed successfully for:

• Node tests and dependency audit
• Python tests and dependency audit
• Secret-pattern scanning
• Compose validation and image build
• SBOM generation and artifact upload

Provenance was correctly skipped on the PR and is configured to run on the merged main push.

Still requires external/operator evidence

The following cannot be truthfully completed from repository code alone and remain production gates:

• Real Cloudflare account/zone/team-domain/application/policy identifiers
• Authoritative external OIDC provider and claim mapping
• Production secret-manager selection and workload identity
• Managed production database/queue/object-storage selections
• Metrics dashboards, distributed traces, alert routing, and staging restore evidence
• Billing currency, merchant jurisdiction, tax treatment, and payment processor
• Named staging reviewer, production approving operator, incident owner, and watch window
• Deployed ZChat browser/mobile accessibility evidence

Production and external traffic remain disabled until these operator-controlled values and verification evidence are recorded.