Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 16 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,9 +205,9 @@ sibling. New page objects come with a fixture-registration snippet in the output
### `run` + `analyze`

`run` executes the scenarios in a real browser and retries on failure to tell a **flaky**
scenario (passes on re-run) from a **consistent** one. `analyze` reads the Cucumber report
and classifies each failure as `app-bug | test-bug | flaky | environment` with a root cause
and a concrete fix.
scenario (passes on re-run) from a **consistent** one. `analyze` reads the run's results and
classifies each failure as `app-bug | test-bug | flaky | environment` with a root cause and a
concrete fix.

```bash
npm test # all scenarios (parallel)
Expand All @@ -219,13 +219,21 @@ npm run ai:analyze # → reports/ai-analysis.md
## Running tests

```bash
npm test # all scenarios (parallel)
npm test # all scenarios (UI + API, parallel) → one Allure result set
npm run test:api # only the @api lane (browserless)
npm run test:smoke # only @smoke tagged
npm run test:ui # Playwright UI mode
HEADLESS=false npm test # watch the browser
npm run report # open the Cucumber HTML report
npm run report # generate + open the Allure report
```

> **Reporting:** [Allure](https://allurereport.org/) is the human-facing report for both lanes
> (UI + API) — run history, per-step detail, and traces/screenshots attached on failure. The test
> scripts clear `reports/allure-results` first, so `npm test` (which runs both projects) gives one
> combined report; `npm run report` renders it (needs Java for the Allure CLI). A Cucumber JSON
> (`reports/*-report.json`) is still emitted, but only as the machine feed the AI pipeline reads
> (`analyze` / `heal-selectors` / `heal-contract`) — not a report you open.

> **CI note:** `npm test` targets the live app (`https://getmobil.com`). Public sites can sit
> behind bot challenges that block data-centre IPs (e.g. CI runners), so the browser tests
> run locally where a normal browser/IP passes. CI gates on the offline checks (type-check,
Expand All @@ -240,7 +248,7 @@ lanes share the `features/` tree and are split by tag: `@api` scenarios run in t
Playwright project (no browser), everything else in `chromium`.

```bash
npm run test:api # @api scenarios only (browserless) → reports/api-report.{json,html}
npm run test:api # @api scenarios only (browserless) → Allure results + cucumber JSON feed
npm run mock:api # run the local mock API standalone (otherwise auto-booted)
```

Expand Down Expand Up @@ -368,8 +376,9 @@ giving a number to "how well does it work" instead of a vibe.
- [x] **agent** — autonomous orchestrator over the whole pipeline, with guardrails
- [x] **self-healing** — compile (`heal`) + runtime selector (`heal-selectors`) + runtime contract (`heal-contract`)
- [x] **API lane** — browserless `APIRequestContext` suite with `probe` (OpenAPI → endpoint map) and `heal-contract`
- [x] **`generate` API mode** — endpoint map → @api feature + steps + client/contract files (build API suites end to end)
- [x] CI/CD integration (GitHub Actions: type-check + redaction gate)
- [ ] `generate` API mode — endpoint map → API feature + client objects (build API suites end to end, not just heal them)
- [x] **Allure reporting** — combined UI + API report (history, steps, traces/screenshots); cucumber JSON kept as the AI feed
- [ ] human-approval web UI over the agent loop (surface run state + confirm/escalate gates)
- [ ] Jira integration (pull stories, write results back)
- [ ] MCP server (secure tool access layer)
Expand Down
26 changes: 26 additions & 0 deletions docs/api/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,15 @@
"404": { "description": "No product with that id", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/Error" } } } }
}
}
},
"/api/categories": {
"get": {
"operationId": "listCategories",
"summary": "List product categories with counts",
"responses": {
"200": { "description": "Categories listed", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/CategoriesResponse" } } } }
}
}
}
},
"components": {
Expand All @@ -74,6 +83,23 @@
"items": { "type": "array", "items": { "$ref": "#/components/schemas/Product" } }
}
},
"Category": {
"type": "object",
"required": ["slug", "name", "count"],
"properties": {
"slug": { "type": "string", "example": "telefon" },
"name": { "type": "string", "example": "Telefon" },
"count": { "type": "integer", "minimum": 0, "example": 2 }
}
},
"CategoriesResponse": {
"type": "object",
"required": ["total", "categories"],
"properties": {
"total": { "type": "integer", "minimum": 0 },
"categories": { "type": "array", "items": { "$ref": "#/components/schemas/Category" } }
}
},
"Error": {
"type": "object",
"required": ["error"],
Expand Down
22 changes: 22 additions & 0 deletions features/api/categories.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
@api
Feature: List product categories
As an API consumer I can list the product categories so that
I can browse products by category.

Background:
Given the API is up

Scenario: Happy path – categories list is returned with valid shape
When the categories list is fetched
Then the response status is 200
And the response body contains a categories array
And every category has slug, name and count fields
And every category count is a positive integer
And the total equals the number of categories returned

Scenario: Categories endpoint returns no unexpected error fields
When the categories list is fetched
Then the response status is 200
And the response body contains a categories array
And every category has slug, name and count fields
And the categories array is not empty
11 changes: 11 additions & 0 deletions mock/server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,17 @@ app.get('/api/products/:id', (req, res) => {
res.json(product);
});

app.get('/api/categories', (_req, res) => {
const counts = new Map<string, number>();
for (const p of CATALOG) counts.set(p.category, (counts.get(p.category) ?? 0) + 1);
const categories = [...counts.entries()].map(([slug, count]) => ({
slug,
name: slug.charAt(0).toLocaleUpperCase('tr') + slug.slice(1),
count
}));
res.json({ total: categories.length, categories });
});

app.listen(PORT, () => {
console.log(`Mock search API listening on http://localhost:${PORT}`);
});
82 changes: 82 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 5 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@
"description": "AI QA Agent - TypeScript + Playwright + Cucumber BDD framework with Claude-powered test generation and failure analysis",
"license": "MIT",
"scripts": {
"test": "bddgen && playwright test",
"test:api": "bddgen && REPORT_NAME=api-report playwright test --project=api",
"test": "rm -rf reports/allure-results && bddgen && playwright test",
"test:api": "rm -rf reports/allure-results && bddgen && REPORT_NAME=api-report playwright test --project=api",
"mock:api": "ts-node mock/server.ts",
"test:smoke": "bddgen --tags @smoke && playwright test",
"test:tag": "sh -c 'bddgen --tags \"@negative\" && playwright test'",
"test:ui": "bddgen && playwright test --ui",
"report": "open reports/cucumber-report.html",
"report": "allure generate reports/allure-results --clean -o reports/allure-report && allure open reports/allure-report",
"web": "ts-node src/web/server.ts",
"ai:agent": "ts-node src/cli/index.ts agent",
"ai:design": "ts-node src/cli/index.ts design",
Expand All @@ -29,6 +29,8 @@
"@playwright/test": "^1.60.0",
"@types/express": "^5.0.6",
"@types/node": "^25.9.3",
"allure-commandline": "^2.43.0",
"allure-playwright": "^3.10.2",
"playwright-bdd": "^9.0.0",
"ts-node": "^10.9.2",
"typescript": "^6.0.3"
Expand Down
10 changes: 8 additions & 2 deletions playwright.config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,14 @@ export default defineConfig({
// so `npm test` / ai:analyze keep reading reports/cucumber-report.json as before.
reporter: [
['list'],
cucumberReporter('json', { outputFile: `reports/${REPORT_NAME}.json` }),
cucumberReporter('html', { outputFile: `reports/${REPORT_NAME}.html` })
// Allure is the human-facing report for BOTH lanes (UI + API): history, per-step detail,
// traces/screenshots attached on failure. Results accumulate in reports/allure-results;
// render with `npm run report`.
['allure-playwright', { resultsDir: 'reports/allure-results', detail: true, suiteTitle: true }],
// Cucumber JSON is kept as the MACHINE feed for the AI pipeline only (not a report you open):
// failureAnalyzer.extractFailures reads it for `analyze` / `heal-selectors` / `heal-contract`.
// It is a single file overwritten each run (no staleness), unlike the accumulating Allure dir.
cucumberReporter('json', { outputFile: `reports/${REPORT_NAME}.json` })
],
use: {
baseURL: process.env.BASE_URL ?? 'https://getmobil.com',
Expand Down
20 changes: 20 additions & 0 deletions src/api/clients/CategoriesApi.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import { BaseApiClient } from '../BaseApiClient';
import { validateCategoriesResponse, type CategoriesResponse } from '../contracts/categories';

export class CategoriesApi extends BaseApiClient {
/**
* Lists all product categories.
* GET /api/categories
*/
async list(): Promise<{ status: number; body: CategoriesResponse }> {
const res = await this.get('/api/categories');
const body = (await res.json()) as CategoriesResponse;
if (res.status() === 200) {
const issues = validateCategoriesResponse(body);
if (issues.length) {
throw new Error('Contract violation on /api/categories: ' + issues.join('; '));
}
}
return { status: res.status(), body };
}
}
42 changes: 42 additions & 0 deletions src/api/contracts/categories.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
/**
* Contract for GET /api/categories (operationId: listCategories)
*/
export interface Category {
slug: string;
name: string;
count: number;
}

export interface CategoriesResponse {
categories: Category[];
/** Optional: some implementations include a top-level total */
total?: number;
}

export function validateCategoriesResponse(body: unknown): string[] {
const problems: string[] = [];
if (typeof body !== 'object' || body === null) {
return ['body is not an object'];
}
const b = body as Record<string, unknown>;
if (!Array.isArray(b['categories'])) {
problems.push('categories is not an array');
return problems;
}
(b['categories'] as unknown[]).forEach((item, i) => {
if (typeof item !== 'object' || item === null) {
problems.push(`categories[${i}] is not an object`);
return;
}
const cat = item as Record<string, unknown>;
if (typeof cat['slug'] !== 'string') problems.push(`categories[${i}].slug is not a string`);
if (typeof cat['name'] !== 'string') problems.push(`categories[${i}].name is not a string`);
if (typeof cat['count'] !== 'number') problems.push(`categories[${i}].count is not a number`);
else if (!Number.isInteger(cat['count']) || (cat['count'] as number) <= 0)
problems.push(`categories[${i}].count must be a positive integer, got ${cat['count']}`);
});
if ('total' in b && typeof b['total'] !== 'number') {
problems.push('total is present but not a number');
}
return problems;
}
3 changes: 3 additions & 0 deletions src/api/fixtures.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ import { test as base } from 'playwright-bdd';
import type { APIRequestContext } from '@playwright/test';
import { SearchApi } from './clients/SearchApi';
import { ProductApi } from './clients/ProductApi';
import { CategoriesApi } from './clients/CategoriesApi';

/**
* Fixtures for the browserless `api` project (see playwright.config.ts). The `apiClient` is an
Expand All @@ -27,6 +28,7 @@ type ApiFixtures = {
apiClient: APIRequestContext;
searchApi: SearchApi;
productApi: ProductApi;
categoriesApi: CategoriesApi;
apiState: { last?: ApiResponse };
};

Expand All @@ -38,5 +40,6 @@ export const test = base.extend<ApiFixtures>({
},
searchApi: async ({ apiClient }, use) => use(new SearchApi(apiClient)),
productApi: async ({ apiClient }, use) => use(new ProductApi(apiClient)),
categoriesApi: async ({ apiClient }, use) => use(new CategoriesApi(apiClient)),
apiState: async ({}, use) => use({})
});
Loading
Loading