[AI-98] HITL issue-to-PR automation pipeline (Phases 0–7, code) — DRAFT#64
Draft
roberthunterjr wants to merge 29 commits into
Draft
[AI-98] HITL issue-to-PR automation pipeline (Phases 0–7, code) — DRAFT#64roberthunterjr wants to merge 29 commits into
roberthunterjr wants to merge 29 commits into
Conversation
Captures approved architecture for HITL bug-fix automation: Azure Durable Functions orchestration, Azure AI Agent Service (Claude), MCP tool server via Azure Functions, and GitHub Actions as the execution environment for lint/test validation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- package.json, host.json, biome.json, jest.config.js, local.settings.json.example - lib/webhook-validator.js: constant-time HMAC-SHA256 signature verification (6 tests) - src/functions/GitHubWebhookReceiver.js: filters agent-ready label events, starts Durable orchestration, returns 202 immediately (6 tests) - .gitignore: override root lib/ rule (Python artifact) to allow src/lib/ Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Full phase-by-phase implementation plan for the issue-to-pr pipeline, covering infrastructure (Bicep IaC), GitHub App setup, auth layer, MCP tool server (9 handlers), agent runner, Durable orchestrator, GitHub Actions workflows, and end-to-end validation. Incorporates GitHub App auth model from the Phase 1 implementation pack. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Item 4 (Foundry model): deployment name `claude-opus-4-8`, referenced by agent-runner.js via @azure/ai-projects; pin specific version, not alias - Item 5 (GHA auth): corrected Phase 6.3 from OIDC to existing pattern — azure/login@v3.0.0 with AZURE_CREDENTIALS service principal JSON stored at environment level, matching deploy-fiona-slack-container.yml pattern - Item 6 (Slack webhook): added slack-webhook-url to Phase 0.3 Key Vault secrets table and updated Bicep app setting to Key Vault reference - Open Items table updated to show resolved vs pending status Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolve agent-layer contradiction, Durable timeout risk, MCP auth gap,
idempotency, and branch naming; correct Azure resource names from az lookups.
- Phase 0.1: confirmed fiona-kv-bronze, fiona-db-dev-cosmos (edfi-fiona-rg);
new Foundry hub/project created in edfi-fiona-rg (single RG, no cross-RG)
- New Phase 3.0 agent-layer spike gates Foundry vs direct-Anthropic
- Phase 4 orchestrator restructured: owns GHA polling via df.createTimer;
agent dispatches then yields (stay on Consumption plan)
- MCP tool server: AAD bearer-token auth baseline, function-key fallback
- Deterministic Durable instanceId for idempotency; purge-and-rerun on terminal
- Orchestrator-computed branch name agent/issue-{n}-{slug}
- Minor fixes: secret count 3->4, github-client single-installation lookup,
local.settings example vars, GitHubWebhookReceiver.test.js in checklists
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike (doc-level, no provisioning) confirmed claude-opus-4-8 is GA in the Foundry catalog (Anthropic/GlobalStandard) in eastus2. Decision: Path B — Claude hosted on Microsoft Foundry via @anthropic-ai/foundry-sdk with a self-owned tool-use loop and managed-identity Entra ID auth. - Drop @azure/ai-projects / Azure AI Agent Service; agent-runner.js owns the loop - Foundry resource pinned to eastus2 (Claude region constraint); single RG - No remote MCP server: 9 GitHub handlers dispatched in-process (McpToolServer not built; MCP-endpoint AAD auth concern moot) - No anthropic-api-key secret (3 KV secrets); MI gets Cognitive Services User; new settings ANTHROPIC_FOUNDRY_BASE_URL + CLAUDE_DEPLOYMENT_NAME - Phase 3.0 reframed GATE -> RESOLVED; live tool round-trip folds into 3.1 - ai-foundry.bicep adopted (base on Azure-Samples/claude starter kit) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Provisions: Functions app (Consumption, Node 22, system-assigned MI), AI Foundry hub (eastus2, claude-opus-4-8 deployment), Cosmos DB agent-runs container, RBAC role assignments (KV Secrets User, Cosmos Data Contributor, Cognitive Services User). Also updates Phase 0.5 local.settings.json.example to match production app settings exactly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Review fixes: 1. Cosmos DB Built-in Data Contributor is a data-plane SQL role, not an Azure RBAC role. Replace the Microsoft.Authorization/roleAssignments grant with a Microsoft.DocumentDB sqlRoleAssignments resource on the existing fiona-db-dev-cosmos account, scoped to the account id. 2. Untrack infra/issue-to-pr/main.json (compiled ARM output) and add it to .gitignore. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements RS256 JWT generation using node:crypto, installation ID lookup, installation access token exchange, and a 60-second-before-expiry in-memory cache. All 5 required test cases pass (happy path, cache hit, cache refresh, malformed key, 404 installation not found). No new runtime dependencies. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds src/lib/mcp-handlers/github-request.js (authenticated GitHub API helper
returning {status, data}; throws descriptive errors on non-2xx), plus handlers
read-issue, list-directory, read-file, write-file with full tests. Also fixes
pre-existing lint issues in webhook-validator.test.js and GitHubWebhookReceiver.test.js.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…e-file tests
- github-request: attach numeric err.status to non-2xx errors so callers
can branch on err.status === 404 instead of string-matching error messages
- write-file: replace err.message.includes('status 404') with err.status === 404
- read-file: use Buffer.byteLength for byte-accurate 1 MB secondary guard
- Add test/mcp-handlers/write-file.test.js covering update, new file, and oversize
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…validation-status, create-draft-pr, add-issue-comment Implements the five remaining GitHub tool handlers with TDD test files. All 45 tests pass; biome lint clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ry-sdk Path B uses the Anthropic SDK against the Foundry endpoint; @azure/ai-projects (Agent Service) is no longer used. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implement the self-owned Claude-on-Foundry tool-use loop for the issue-to-PR pipeline: - agent-tools.js: nine Anthropic tool defs + context-injecting dispatchTool - run-store.js: Cosmos agent-runs create/update via DefaultAzureCredential - foundry-client.js: mockable single-turn Foundry model call wrapper - agent-runner.js: startAgentEdits + agentReactToResult; the loop yields on run_validation (returns dispatched without the tool_result) and resumes with the deferred validation conclusion Tests mock the Foundry SDK, tool dispatch, and Cosmos; 64 tests pass, lint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… init - agent-runner: pass disable_parallel_tool_use:true on every model call so run_validation is always the sole tool_use in its turn; add defensive guard that throws (-> failRun) if the invariant is somehow violated - run-store: replace read-then-replace updateRunRecord with Cosmos patch() using set operations to avoid lost-update race and thin-document resurrection - foundry-client: lazy-init DefaultAzureCredential and AnthropicFoundry client on first use (module-scoped cache) to avoid authenticating on import - tests: assert tool_choice shape on every model call; test defensive guard routes to failed; update run-store tests to assert patch operations shape Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Implements WorkflowOrchestrator.js (generator-based Durable orchestrator) and WorkflowActivities.js (5 activities: PostSlackNotification, StartAgentEdits, GetValidationStatus, AgentReactToResult, PostIssueComment). Threads agent state (messages/pendingToolUseId/context/dispatchedAt) between activities; orchestrator is deterministic (no Date.now/fetch in body). Adds `main` field to package.json for Azure Functions host discovery. 79/79 tests pass, biome lint clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…OLLS give-up test Strengthens the determinism scan (Math.random/process.env) with a positive assertion that the poll timer fires at currentUtcDateTime+30s, and adds the previously-untested MAX_POLLS give-up path. (Recovered from a stalled fix agent; corrected its off-by-one in the queued-response fixture.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…condition) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
User/reviewer guide of the auth layer, 9 GitHub tools, agent runner, and orchestrator — each feature mapped to its unit tests for usability review and merge confidence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- sanitizeInstanceId: replace non-[A-Za-z0-9_-] with '-' for deterministic instanceId
- slug: lowercase, collapse non-alnum runs to '-', trim edges, truncate to 30 chars
- Handler: getStatus check before startNew; no-op if non-terminal (Running/Pending/ContinuedAsNew/Suspended); purge+restart if terminal (Completed/Failed/Terminated/Canceled)
- Add branchName (agent/issue-{n}-{slug}) to orchestration input
- Pass explicit instanceId to startNew; log uses computed id
- Tests: 13 new cases (idempotency, terminal re-run, branch slug, messy title, truncation)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add three workflow files for the issue-to-pr pipeline: - agent-execution.yml: repository_dispatch handler for agent-triggered validation - on-pullrequest-issue-to-pr.yml: build/test/CodeQL PR gate - deploy-issue-to-pr-function.yml: IaC-first deploy to Azure on push to main Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…able Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Monitoring, orchestration inspection, idempotent retry via re-label, debugging (in-process tools / Foundry agent), and common failure modes — grounded in the Path B implementation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… remain Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Defense-in-depth from the branch security review (neither was an exploitable finding, both tighten the auth gate before deploy): - webhook-validator: reject when the secret is empty/undefined (an unresolved Key Vault reference can no longer be bypassed with an empty-key HMAC) - receiver: enforce AGENT_ALLOWED_REPOS (comma-separated owner/repo) when set; unset = accept all installed repos (backward compatible). Wired into functions.bicep (default Ed-Fi-Alliance-OSS/Fiona) + local.settings example. - 5 new tests (98 total); lint clean; bicep compiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI-98 — HITL Issue-to-PR Automation Pipeline
Automates TDD bug fixes: when an engineer labels a GitHub issue
agent-ready(the human-in-the-loop gate), an Azure Durable Functions app runs a Claude Opus 4.8 agent hosted on Microsoft Foundry that reproduces the bug with a failing test, fixes it, runs validation on GitHub Actions, and opens a draft PR.What's included (Phases 0–7, code & docs)
infra/issue-to-pr/— Consumption Functions app (Node 22, system-assigned MI), Foundry resource +claude-opus-4-8deployment (eastus2),agent-runsCosmos container, RBAC (KV Secrets User, Cosmos data-plane role, Cognitive Services User).github-client.js— GitHub App JWT → cached installation token.github-requesthelper.agent-runner.jsself-owned Claude tool-use loop that yields at validation so the orchestrator owns GHA polling;agent-tools.js,foundry-client.js,run-store.js.WorkflowOrchestrator.js+WorkflowActivities.js— deterministic Durable generator with adf.createTimerpolling loop,MAX_ROUNDS/MAX_POLLSguards.GitHubWebhookReceiver.js— HMAC validation, label filter, deterministic instance id (idempotent re-labels),branchNamecomputation.agent-execution.ymlvalidation runner, PR gate, IaC-first deploy), actions SHA-pinned to the allowed list.Architecture
agent-readylabel → receiver (HMAC + idempotent start) → Durable orchestrator → Slack notice → agent edits + dispatches validation, yields → orchestrator polls GHA on a durable timer → agent reacts (opens draft PR / re-validates / fails) → failure comment on the issue. Tool calls run in-process (Path B — no remote MCP server); Claude auth is managed-identity Entra ID (no API key). Decision record:docs/superpowers/specs/.Testing
az bicep buildcompiles. Each feature is mapped to its tests indocs/AI-98-feature-guide.md.Security review
A security review was run on this branch — no HIGH/MEDIUM findings at the reporting threshold. RS256 signing, timing-safe HMAC, parameterized Cosmos access, and post-validation parsing were verified sound. Two defense-in-depth items surfaced by the review have been applied (commit
db4f6b9): (1) the webhook validator now fails closed if the secret is empty/unresolved; (2) the receiver enforces anAGENT_ALLOWED_REPOSallowlist when set (defaults toEd-Fi-Alliance-OSS/Fionain Bicep). 98 tests pass.Remaining — operator-gated (NOT in this PR)
az deployment group create(with the GitHub App ID); set the GitHub App webhook URL post-deploy.This change set was substantially authored by AI (Claude), built via test-driven development with a per-task implement→review→fix loop. Per Foundation policy, a human must review before merge/deploy, and accountability for the final output rests with the reviewer. Assumptions/limitations are documented in
docs/AI-98-implementation-status.md.🤖 Generated with Claude Code