app-promotion-deploy: manual/auto apply concurrency collisions + wrong detect matrix

[Svaag]

Two related deploy-orchestration bugs surfaced while promoting noc-agent (the proactive-loop work). Sibling to #262.

Bug 1 — manual apply and auto app-promotion-deploy cancel each other

app-promotion-deploy.yml triggers on push to main (deploy paths) and calls apply.yml with dry_run: false, which waits at the production gate. Both that reusable call and any manual apply.yml workflow_dispatch land in the same live concurrency group:

# apply.yml
concurrency:
  group: ${{ inputs.dry_run && format('production-infra-dry-run-{0}', github.run_id) || 'production-infra-live-v2' }}
  cancel-in-progress: false

When an auto-deploy is sitting at the gate and an operator also dispatches a manual apply (or vice versa), GitHub cancels one of the pending runs in the group. Result: applies silently end as cancelled with no job ever starting, and it looks random.

Worse, unapproved auto-deploys accumulate at the gate and hold the lane. Evidence this session:

• Manual applies cancelled mid-gate: 27648731930, 27654535843.
• A stale auto-deploy from an earlier promotion merge (Promote noc-agent 84cb3c9 #261) sat waiting ~2h: 27648482775 — and because the called apply checks out that merge commit, approving it would have deployed the superseded (old) pin, not current main.
• Re-dispatching only worked when the lane happened to be clear (27649053353, 27655203336).

Proposed fix

• Make superseded promotion deploys auto-cancel (e.g. cancel-in-progress: true for the promotion-deploy lane, or have app-promotion-deploy cancel prior pending runs for the same target) so they don't pile up.
• Document/enforce a single deploy path: after a promotion merge, approve the auto-deploy gate rather than manually dispatching apply; if a manual apply is needed, ensure the live lane is clear first.
• Consider checking out current main rather than the push SHA so a queued promotion deploy can't ship a superseded pin.

Bug 2 — detect maps a noc_agent_version change to icinga2/mon, not noc

The #264 merge changed only ansible/inventory/host_vars/noc.yml (noc_agent_version: 84cb3c9 → 7941d45). The resulting auto-deploy (27654482813) detect job produced a matrix of just apply (icinga2, mon) — so the app pin bump would not have deployed noc-agent at all; it only re-rendered icinga on mon.

Expected: a noc_agent_version change in host_vars/noc.yml should map to playbook=noc, limit=noc (the app deploy), optionally also an icinga refresh — but it must include noc.

Proposed fix

Fix the detect path→playbook mapping (the add_once(playbook, limit) logic) so an app-version pin change in host_vars/<host>.yml maps to that host's app playbook.

Acceptance

• Merging a promotion PR that bumps noc_agent_version triggers an auto-deploy whose matrix includes (noc, noc).
• A manual apply and an auto-deploy no longer silently cancel each other; superseded promotion deploys are cancelled rather than left waiting.

Refs: app-promotion-deploy.yml, apply.yml; surfaced promoting noc-agent for the proactive NOC loop (#264). Sibling to #262.

🤖 Generated with Claude Code

[Svaag]

Bug 2 is broader: detect ignores configs/ deploy paths entirely

Confirmed while deploying #270 (expose NOC dashboard via Caddy). The #270 merge changed:

• configs/servify.network.zone (new noc record)
• configs/Caddyfile.j2 (new noc.servify.network block)
• ansible/inventory/host_vars/noc.yml + ansible/generated/noc/nftables.conf (firewall rule)

Its auto-deploy (27674526987) detect produced a matrix of just apply (icinga2, mon) — the same mis-map as Bug 2. None of the actual changes were deployed:

• configs/servify.network.zone → should map to knot, dns (the knot role distributes configs/*.zone)
• configs/Caddyfile.j2 → has no Ansible playbook at all (proxy-VM Caddy is a manual deploy per docs/deployment.md Phase 7); detect can't map it until a caddy role/playbook exists
• host_vars/noc.yml firewall change → should map to firewall, noc (and the app-pin case to noc, noc per Bug 2)

So #270 had to be deployed via targeted manual applies (firewall/noc, knot/dns) plus a manual Caddy deploy on proxy. The detect mapping needs: (a) configs/<zone>.zone → knot/dns; (b) host_vars/<host>.yml non-pin changes → firewall/<host>; (c) a real caddy playbook so Caddyfile changes are deployable + mappable.

Updated acceptance

• A PR touching configs/*.zone triggers an auto-deploy whose matrix includes (knot, dns).
• A PR touching configs/Caddyfile.j2 deploys Caddy (requires a caddy role/playbook first).
• A host_vars/<host>.yml firewall change maps to (firewall, <host>).

🤖 Generated with Claude Code

[hyrule-engineering-loop]

Reliability Governor Decision

• role: staff_sre_autonomous_operations
• decision: knowledge_gap
• next_loop: knowledge
• handoff_contract: knowledge_context_pack
• source: human
• intent: compliance / tier 4
• capability: none
• knowledge: missing / unknown
• reasons: Knowledge policy result is deny; Knowledge context returned no included_refs

{
  "affected_assets": [
    "AS215932/network-operations"
  ],
  "affected_customers": [],
  "affected_services": [],
  "allowed_paths": [],
  "authority_text_hash": "38a3499e8eb2f2403d0c0591517f5ed8033318d87cb33d44848dbcbe511b5dcf",
  "blast_radius": "compliance",
  "controlled_loops": [
    "engineering",
    "noc",
    "knowledge"
  ],
  "created_at": "2026-06-30T01:55:13.543496+00:00",
  "denial_reasons": [
    "Knowledge policy result is deny",
    "Knowledge context returned no included_refs"
  ],
  "expected_paths": [
    "compliance/"
  ],
  "forbidden_paths": [],
  "governor_name": "Reliability Governor",
  "governor_role": "staff_sre_autonomous_operations",
  "handoff_contract": "knowledge_context_pack",
  "intent_type": "compliance",
  "issue_id": "AS215932/network-operations#265",
  "issue_number": 265,
  "issue_text_hash": "7572bdb643a989275e9c28de7d0b78c07655bf893f33b8766ff92f1a1df567b9",
  "knowledge_authority_level": "unknown",
  "knowledge_context_pack_id": "ctx_13f977f271fb48fdcff2aa4e83d35733",
  "knowledge_export_version": "local-20260623120725",
  "knowledge_status": "missing",
  "labels_to_add": [
    "loop:knowledge-gap"
  ],
  "labels_to_remove": [
    "loop:intake",
    "loop:candidate",
    "loop:approved",
    "loop:needs-context",
    "loop:needs-human"
  ],
  "lhp": null,
  "matched_capability": null,
  "next_loop": "knowledge",
  "policy_rules": [
    "deny stale, contradictory, missing, or errored Knowledge context"
  ],
  "record_id": "e124ea533f79db679eb8",
  "repo": "AS215932/network-operations",
  "required_checks": [],
  "risk_tier": 4,
  "rollback_plan": "",
  "routing_decision": "knowledge_gap",
  "schema_version": "reliability-governor.cdr.v1",
  "source": "human",
  "storage_path": "/var/lib/engineering-loop/reliability-governor/as215932-network-operations-265-e124ea533f79db679eb8.json",
  "verification_method": "Use the issue-specified verification: tests/checks/evidence named in the request."
}