Configure Spec-Driven Development + spec/plan/tasks for test-coverage ratchet (#1292)#1297
Open
sstjean wants to merge 9 commits into
Open
Configure Spec-Driven Development + spec/plan/tasks for test-coverage ratchet (#1292)#1297sstjean wants to merge 9 commits into
sstjean wants to merge 9 commits into
Conversation
Initialize GitHub Spec Kit for Copilot: .specify templates/scripts/memory, slash-command prompts, and agent definitions. Add Mission Landing Zone constitution v1.0.0 (adapted from EP Cube Graph) with a ratcheting no-regression coverage policy in Principle IV.
Feature #1292: establish automated test-coverage measurement and a no-regression coverage ratchet. Adds spec.md (US1-US4 -> #1293-#1296), plan.md (Pester 5 + PSRule for Azure + version-controlled baseline gate), tasks.md (30 tasks), research/data-model/quickstart/contracts. US1 tests the src/artifacts PowerShell scripts with zero production edits (mock-and-invoke).
Standardize on Pester 6.0.0 for local + CI parity. v6 runs on Windows PowerShell 5.1 and PowerShell 7.4+ (covers the Windows-majority team), JaCoCo stays the default coverage format, and the config/authoring API is unchanged from v5. Fixes the loose -MinimumVersion 5.0 in the CI task (would pull 6.x unpinned) to -RequiredVersion 6.0.0 -SkipPublisherCheck. Records 5.7.1 as the documented fallback.
Add tests/Invoke-Tests.ps1 (Pester 6 + JaCoCo) and behavioral tests for the three src/artifacts scripts, invoked under mocks with ZERO edits to the production scripts. 8 tests pass; initial real line coverage = 76.81%. Adds .github/workflows/pester-coverage.yml (PR + push to main + dispatch), gitignores coverage.xml/testResults.xml. Closes the US1 slice of #1292.
Add tests/Compare-Coverage.ps1 (ratchet gate CLI per contract) + 10 contract tests, and tests/Get-CoveragePercent.ps1 (shared JaCoCo LINE parser used by both the harness and the gate so local, CI, and baseline all agree). Record tests/coverage-baseline.json at 78.1%. Wire the gate as a failing step in pester-coverage.yml and document the required-status-check requirement. Unifies the displayed coverage number on the JaCoCo LINE counter (78.1%).
Split US3 (#1294) out of the coverage feature so US1+US2+US4 ship together. PSRule for Azure needs template/Bicep expansion to apply rules (compiled mlz.json alone -> Outcome=None), and will be introduced reporting-first on the follow-up. Finding + rationale recorded in research.md Decision 2; issue #1294 commented.
super-linter runs on the whole codebase; the generated Spec Kit scaffolding (.specify scripts, .github agent/prompt files, copilot-instructions.md) and the SDD planning artifacts under specs/ are tool-managed and regenerated by /speckit.* commands, so they are excluded from linting via FILTER_REGEX_EXCLUDE. Product code (src/, tests/, docs/, real workflows) is still fully linted.
Add docs/testing.md covering how to run the Pester tests locally, how coverage is measured (JaCoCo LINE), the ratchet gate exit codes, how to raise the baseline, and the required-status-check step for making the gate non-bypassable. Link it from CONTRIBUTING.md. Marks the polish checks (T028-T030): 18/18 tests pass, 78.1% coverage, gate passes at baseline, zero edits to src/. Completes the shippable US1+US2+US4 slice of #1292; US3 (#1294) follows separately.
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Introduces a first-class PowerShell test harness with coverage measurement and a no-regression “coverage ratchet” gate, plus contributor documentation and Spec Kit workflow artifacts to standardize the SDD process.
Changes:
- Add Pester behavioral tests, a coverage runner, and a ratchet gate (with contract tests) for
src/artifacts/*.ps1. - Add CI workflow to run tests, publish coverage, and enforce the baseline.
- Add contributor docs and Spec Kit spec/plan/contract artifacts supporting the coverage initiative.
Reviewed changes
Copilot reviewed 70 out of 71 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/coverage-baseline.json | Adds initial committed coverage baseline. |
| tests/artifacts/Remove-VirtualMachine.Tests.ps1 | Adds behavioral Pester coverage for VM removal script. |
| tests/artifacts/New-KeyVaultKey.Tests.ps1 | Adds behavioral Pester coverage for Key Vault key script. |
| tests/artifacts/New-ADDSForest.Tests.ps1 | Adds stubbed behavioral coverage for Windows-only ADDS script. |
| tests/Invoke-Tests.ps1 | Adds local/CI entrypoint to run Pester + JaCoCo coverage. |
| tests/Get-CoveragePercent.ps1 | Adds shared helper to compute a single coverage percentage. |
| tests/Compare-Coverage.ps1 | Adds baseline compare + ratchet updater CLI. |
| tests/Compare-Coverage.Tests.ps1 | Adds contract tests for ratchet CLI behavior. |
| specs/001-test-coverage-ratchet/spec.md | Adds feature spec for coverage + ratchet initiative. |
| specs/001-test-coverage-ratchet/research.md | Records tool/approach decisions and deferrals. |
| specs/001-test-coverage-ratchet/quickstart.md | Adds local validation quickstart for coverage/ratchet. |
| specs/001-test-coverage-ratchet/plan.md | Adds implementation plan and target repo structure. |
| specs/001-test-coverage-ratchet/data-model.md | Defines baseline/measurement/gate data shapes. |
| specs/001-test-coverage-ratchet/contracts/ratchet-cli.md | Documents the Compare-Coverage CLI contract. |
| specs/001-test-coverage-ratchet/contracts/coverage-baseline.schema.json | Adds JSON schema for coverage baseline file. |
| specs/001-test-coverage-ratchet/checklists/requirements.md | Adds spec quality checklist for the feature. |
| docs/testing.md | Adds contributor docs for running tests/coverage/ratchet. |
| CONTRIBUTING.md | Links contributors to testing/coverage docs. |
| .specify/workflows/workflow-registry.json | Registers Spec Kit workflow metadata. |
| .specify/workflows/speckit/workflow.yml | Adds Spec Kit workflow steps and gates. |
| .specify/templates/tasks-template.md | Adds/updates Spec Kit tasks template. |
| .specify/templates/spec-template.md | Adds/updates Spec Kit spec template. |
| .specify/templates/plan-template.md | Adds/updates Spec Kit plan template. |
| .specify/templates/constitution-template.md | Adds/updates Spec Kit constitution template. |
| .specify/templates/checklist-template.md | Adds/updates Spec Kit checklist template. |
| .specify/scripts/bash/setup-tasks.sh | Adds tasks setup script for Spec Kit. |
| .specify/scripts/bash/setup-plan.sh | Adds plan setup script for Spec Kit. |
| .specify/scripts/bash/create-new-feature.sh | Adds feature directory/branch naming helper. |
| .specify/scripts/bash/check-prerequisites.sh | Adds prerequisite checker for Spec Kit flows. |
| .specify/memory/constitution.md | Adds repository constitution content. |
| .specify/integrations/speckit.manifest.json | Records installed speckit integration files/hashes. |
| .specify/integrations/copilot.manifest.json | Records installed copilot integration files/hashes. |
| .specify/integration.json | Sets active/default integration metadata. |
| .specify/init-options.json | Records Spec Kit init options (integration/numbering). |
| .specify/feature.json | Pins the active feature directory path. |
| .specify/extensions/agent-context/scripts/powershell/update-agent-context.ps1 | Adds PowerShell context updater for agent instructions. |
| .specify/extensions/agent-context/scripts/bash/update-agent-context.sh | Adds Bash context updater for agent instructions. |
| .specify/extensions/agent-context/extension.yml | Declares the agent-context extension and hooks. |
| .specify/extensions/agent-context/commands/speckit.agent-context.update.md | Documents agent-context update command. |
| .specify/extensions/agent-context/agent-context-config.yml | Configures which context file/markers to manage. |
| .specify/extensions/agent-context/README.md | Documents the agent-context extension behavior. |
| .specify/extensions/.registry | Registers installed extensions and commands. |
| .specify/extensions.yml | Configures enabled extensions and hook execution. |
| .gitignore | Ignores generated coverage/test result outputs. |
| .github/workflows/super-linter.yml | Excludes tool-managed/spec artifacts from linting. |
| .github/workflows/pester-coverage.yml | Adds CI workflow for Pester coverage + ratchet gate. |
| .github/prompts/speckit.taskstoissues.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.tasks.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.specify.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.plan.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.implement.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.converge.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.constitution.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.clarify.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.checklist.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.analyze.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/prompts/speckit.agent-context.update.prompt.md | Adds prompt front-matter for agent prompt file. |
| .github/copilot-instructions.md | Adds managed Spec Kit plan pointer section. |
| .github/agents/speckit.taskstoissues.agent.md | Adds agent definition for tasks-to-issues flow. |
| .github/agents/speckit.tasks.agent.md | Adds agent definition for tasks generation flow. |
| .github/agents/speckit.specify.agent.md | Adds agent definition for spec authoring flow. |
| .github/agents/speckit.plan.agent.md | Adds agent definition for plan authoring flow. |
| .github/agents/speckit.implement.agent.md | Adds agent definition for implementation flow. |
| .github/agents/speckit.converge.agent.md | Adds agent definition for convergence checks. |
| .github/agents/speckit.constitution.agent.md | Adds agent definition for constitution updates. |
| .github/agents/speckit.clarify.agent.md | Adds agent definition for clarification flow. |
| .github/agents/speckit.checklist.agent.md | Adds agent definition for checklist generation. |
| .github/agents/speckit.analyze.agent.md | Adds agent definition for cross-artifact analysis. |
| .github/agents/speckit.agent-context.update.agent.md | Adds agent definition for context updating. |
Comments suppressed due to low confidence (3)
tests/Get-CoveragePercent.ps1:1
- The function contract says it returns
$nullfor unparseable/invalid coverage data, but the[int]casts can throw (e.g., missing/empty/non-numericcovered/missedattributes) and bubble as a terminating error. Wrap the numeric parsing in atry/catch(or validate attributes) and return$nullon conversion failure to preserve the documented behavior.
tests/Invoke-Tests.ps1:1 - This harness documents a pinned Pester version (6.0.0) for local/CI parity, but the import only enforces
-MinimumVersion 5.0.0, which can allow a different major/minor locally and produce a different coverage number or behavior. Prefer importing with the pinned version (or explicitly failing when 6.0.0 isn’t available) to keep the gate’s number deterministic across environments.
tests/Compare-Coverage.ps1:1 - On Windows PowerShell 5.1,
Set-Contentdefaults to UTF-16LE, which can introduce noisy diffs (and potential tooling friction) when ratcheting the committed JSON baseline. Specify a consistent encoding (typically UTF-8) when writingtests/coverage-baseline.jsonso updates are stable across platforms.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…rred PSRule from quickstart - pester-coverage.yml: if-no-files-found error->warn so a missing coverage.xml from an earlier step failure no longer masks the real root cause - quickstart.md: remove PSRule install line, Step 5, psrule-bicep.yml table row, and SC-008 signal (US3 is deferred to a follow-up PR per docs/testing.md)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Configures Spec-Driven Development (GitHub Spec Kit) for the repository and lands the spec, plan, and tasks for the first feature: automated test-coverage measurement with a no-regression coverage ratchet.
This is a planning + scaffolding PR (draft). No product/CI behavior changes yet — implementation of the user stories will follow as commits on this branch.
Relates to Feature #1292 (does not close it — the implementation stories remain).
What's included
Spec Kit setup + constitution (
50eeb70).specify/templates, scripts, memory, and workflow.github/prompts/slash commands and.github/agents/agent definitions.specify/memory/constitution.md), adapted to a Bicep IaC / SCCA / zero-trust landing zone. Principle IV (Validation-Driven Infrastructure) uses a ratcheting "no regression on coverage %" policy toward an eventual 100% target.Feature 001 spec/plan/tasks (
877a007) underspecs/001-test-coverage-ratchet/spec.md— 4 user stories mapped to issues: US1 Measure PowerShell artifact test coverage with Pester in CI #1293 (P1, Pester coverage — MVP), US2 Enforce a no-regression coverage ratchet gate in CI #1295 (P2, ratchet gate), US3 Add rule-based validation testing for Bicep templates in CI #1294 (P3, PSRule Bicep validation), US4 Document the testing and coverage policy for contributors #1296 (P4, docs)plan.md— Pester 5 (+ JaCoCo) forsrc/artifacts/*.ps1, PSRule for Azure for Bicep, version-controlledtests/coverage-baseline.jsongate; existingsuper-linter/validate-build-bicepgates untouchedtasks.md— 30 dependency-ordered tasksresearch.md,data-model.md,quickstart.md,contracts/Notes for reviewers
src/mlz.jsonis not modified by this feature.Linked issues