Phase 5 – CI/CD Pipeline Design & Acceptance Guide

[jenniferoko]

Ready for Review -

Excel:

Phase5_CICD_Pipeline_Validation_Workbook.xlsx

Document:

Phase5_CICD_Pipeline_Design_and_Acceptance_Guide.docx

Phase 5 – CI/CD Pipeline Design & Acceptance Guide

Executive Overview

Phase 5 integrates the entire smoke test suite into GitHub Actions to provide continuous validation after every code change and nightly regression testing.

The engineer is responsible for implementing the complete CI/CD pipeline based on the approved workflow specification.

Jennifer is responsible for validating that the pipeline design aligns with the team's operational requirements before enabling automated execution.

---

Phase Objectives

At the completion of Phase 5:

• All smoke tests execute automatically through GitHub Actions.
• Every pipeline produces an HTML Playwright report.
• Reports are retained as workflow artifacts.
• Failed nightly runs generate Slack notifications.
• Secrets are securely stored in GitHub.
• Jennifer approves the first successful nightly execution before production use.

---

Jennifer's Deliverables

Before implementation is approved, Jennifer should review the following with the engineering team.

Pipeline Review

Confirm:

• Job names match team standards.
• Trigger conditions are correct.
• Schedule matches expected execution time.
• Runtime estimates are reasonable.
• HTML reports are uploaded.
• Failure notifications are sent to the correct Slack channel.
• Required GitHub secrets exist.
• Pipeline follows repository conventions.

Jennifer should also review the first nightly report before enabling ongoing automation.

---

Pipeline Architecture

Developer Push
│
▼
Critical Smoke
(~10 sec)

Pull Request
│
▼
Critical Smoke
(~10 sec)

Manual Trigger
│
▼
Single Service Smoke
(~5 min)

Nightly (03:00 ET)
│
▼
Complete Smoke Suite
(~20 min)

Upload Report
│
▼
Slack Notification (if failed)

---

GitHub Actions Workflow

.github/workflows/smoke.yml

Pipeline Name

ZCP API Smoke Tests

---

Job Overview

Job	Purpose	Trigger	Estimated Runtime
Critical	Authentication validation	Push / Pull Request	~10 seconds
Service Smoke	Individual service validation	Manual	~5 minutes
Nightly	Complete smoke suite	Scheduled	~20 minutes

---

Job 1 — Critical

Purpose

Provides immediate validation after every code change.

Runs only the critical authentication tests.

Tag

@critical

Triggers

• Push to main
• Pull Request

Expected runtime

10 seconds

Validation

• Checkout repository
• Install Node
• Install dependencies
• Execute

npx playwright test --grep @critical

---

Job 2 — Service Smoke

Purpose

Allows engineers to validate one service independently.

Example

@compute

or

@storage

or

@networking

Trigger

workflow_dispatch

Engineer supplies

tag

Example

@dns

Expected runtime

5 minutes

Validation

• Execute only selected tag
• Upload Playwright HTML report
• Retain artifact for 14 days

---

Job 3 — Nightly

Purpose

Runs the complete regression suite every night.

Tag

@smoke

Trigger

03:00 ET

Cron

0 7 * * *

Expected runtime

20 minutes

Timeout

30 minutes

Validation

• Execute every smoke test
• Upload HTML report
• Retain report for 30 days
• Notify Slack if failed

---

Pipeline Lifecycle

Push / PR
│
▼
Critical Job

Manual
│
▼
Service Smoke

Nightly
│
▼
Run All Smoke Tests

Generate HTML Report
│
▼
Upload Artifact
│
▼
Slack Notification (Failures Only)

---

# Required GitHub Secrets

| Secret | Purpose |
|----------|----------|
| `ZCP_SMOKE_TOKEN` | Smoke test API token |
| `ZCP_SMOKE_PROJECT` | Project slug |
| `SLACK_QA_WEBHOOK` | Slack Incoming Webhook |
| `ZCP_SMOKE_REGION` | Default cloud region |

---

# Environment Configuration

Example

```text
ZCP_API_BASE=https://api.zcp.zsoftly.ca/api

ZCP_SMOKE_TOKEN=xxxxxxxxxxxxxxxx

ZCP_SMOKE_PROJECT=smoke-tests

ZCP_SMOKE_REGION=yow

Environment variables should never be committed into source control.

Use

GitHub Secrets

instead.

---

API Validation

Each workflow must verify

• Successful authentication
• Correct environment
• Required secrets loaded
• Playwright starts successfully
• HTML report generated
• Exit code returned correctly

---

Artifact Validation

Verify

• HTML report uploaded
• Artifact name includes workflow run ID
• Report opens successfully
• Screenshots included when failures occur
• Trace files available (if enabled)

---

Slack Notification Validation

Notification should only occur when

failure()

returns true.

Verify

• Correct Slack channel
• Link opens GitHub Actions run
• Message clearly identifies failed workflow
• Only one notification per failed run

Example

ZCP nightly smoke FAILED

View Report

---

Security Validation

Verify

• Secrets are never logged
• API tokens are masked
• Webhook URL is hidden
• Reports contain no credentials

---

Performance Expectations

Job	Maximum Runtime
Critical	30 seconds
Service Smoke	10 minutes
Nightly	30 minutes

Investigate significant runtime increases.

---

Cleanup Validation

Each workflow should verify

• Temporary files removed
• Playwright processes terminated
• HTML report generated
• No orphaned test resources remain

---

Acceptance Criteria

Category	Acceptance Criteria	Verification
Workflow File	.github/workflows/smoke.yml committed	Code Review
Critical Job	Executes on every Push and PR	GitHub Actions
Service Smoke	Executes manually using tag input	GitHub Actions
Nightly Job	Executes on schedule	GitHub Actions
HTML Reports	Uploaded successfully	Artifact Review
Artifact Retention	14-day and 30-day retention configured	Workflow Review
Secrets	Configured in GitHub	Repository Settings
Slack Alerts	Failure notifications delivered	Slack Verification
Runtime	Jobs complete within expected duration	Pipeline Review
Regression	All 50 smoke tests pass	Nightly Run
Jennifer Approval	Nightly report reviewed	Sign-off

---

Engineer Checklist

Required Before Phase 5 Sign-off

• smoke.yml implemented
• Critical job executes successfully
• Service Smoke executes successfully
• Nightly job executes successfully
• HTML reports generated
• Artifacts uploaded
• GitHub Secrets configured
• Slack webhook configured
• Failure notifications verified
• Runtime within expected limits
• Full smoke suite (Tests 1–50) passes
• Jennifer reviews nightly report

---

Resource Verification

Resource	Verified
HTML Report Uploaded	- [ ]
Artifact Download Works	- [ ]
Slack Notification Sent	- [ ]
GitHub Secrets Configured	- [ ]
Nightly Schedule Verified	- [ ]

---

Engineer Sign-off

Field	Value
Engineer
Reviewer (Jennifer)
Date
Build / Release Version
Git Commit / PR
GitHub Actions Run

---

Final Result

• PASS – Pipeline Approved
• FAIL – Rework Required

---

Reviewer Comments

_________________________________________________________

_________________________________________________________

_________________________________________________________

_________________________________________________________

---

Exit Rule

Phase 5 is complete only when:

• All GitHub Actions jobs execute successfully.
• The complete smoke suite (Tests 1–50) passes.
• HTML reports are generated and uploaded as workflow artifacts.
• Slack failure notifications are delivered to the approved QA channel.
• GitHub Secrets are configured securely.
• Jennifer reviews the first successful nightly execution and signs off.

Only then may the CI/CD pipeline be enabled for ongoing automated validation.

[godsonten]

You covered the right operational concerns: trigger design, reports, artifacts, Slack alerts, secrets, runtime expectations, and Jennifer's sign-off on the first successful nightly run.

I would still revise a few items before QA sign-off.

The first issue is the cron schedule.

You wrote 0 7 * * * for 03:00 ET. That is correct only during Eastern Daylight Time (UTC-4). During Eastern Standard Time (UTC-5), 03:00 ET is 0 8 * * *.

GitHub cron runs in UTC and does not automatically follow ET daylight saving changes. You need to either accept seasonal drift or document how the schedule will be adjusted.

The second issue is trigger scope.

The epic says the critical job runs on push to main and every PR. That is fine, but confirm whether pushes to feature branches should also run @critical.

If not, keep it as:

branches:
  - main

If yes, broaden the trigger. This should match the team's workflow.

The third issue is the manual service-smoke tag input.

You should restrict or validate allowed tags. Right now someone can type any grep pattern. At minimum, document accepted values:

• @compute
• @storage
• @networking
• @kubernetes
• @dns
• @read
• @critical
• @smoke

Ideally use a choice input in GitHub Actions instead of free text.

Environment variables

You also need to include all required environment variables in every job.

Your secrets list includes ZCP_SMOKE_REGION, but the critical and service-smoke job descriptions do not clearly pass it.

The .env.example also includes ZCP_API_BASE, but the workflow spec does not pass it. If the code defaults ZCP_API_BASE, that is fine, but the behavior should be explicit.

Artifact handling

For artifacts, make sure reports upload with:

if: always()

for all jobs where a report is expected, including failures.

The critical job currently does not mention uploading a report. If the Phase 5 objective says every pipeline produces an HTML Playwright report, then the critical job must upload one too.

If you only want reports for service-smoke and nightly, revise the objective.

Slack validation

For Slack validation, be careful with testing failure notifications. You should require one controlled failure test or dry-run validation before enabling the workflow, then revert it.

Do not wait for a real nightly failure to prove Slack works.

Security

Your checklist is good, but add one more requirement: the Playwright HTML report must not include request headers or Bearer tokens.

API test reporters can accidentally expose headers, failed request payloads, or environment values if the engineer logs too much.

Cleanup validation

Cleanup validation also needs sharper wording. The workflow itself cannot guarantee no orphaned cloud resources unless the tests perform cleanup and a post-run verification step checks smoke-* resources.

Tie this back to the cleanup registry and resource verification from Phases 2–4.

Final QA recommendation

Phase 5 is close to ready. Before approval, revise the cron/DST handling, validate manual tag input, clarify environment variables per job, align the "every pipeline has a report" objective with the workflow, add controlled Slack notification validation, and explicitly protect secrets from logs and reports.

After those changes, I would approve Phase 5 for implementation.