Diagnose where Jira-to-GitLab delivery flow is slowing down, recommend one measurable intervention, and verify whether it improved engineering throughput.
DeliveryLens AI is an AWS-native enterprise delivery-intelligence platform. It connects Jira work items with GitLab merge requests and CI pipelines, calculates auditable flow metrics, identifies workflow bottlenecks, and presents approval-gated recommendations with an explicit ROI estimate.
The repository includes:
- A local runtime with reproducible bootstrap Jira and GitLab-style delivery events
- A responsive 3D command-center dashboard for local product demonstrations
- Live Jira Cloud REST and GitLab GraphQL connector clients
- GitLab and Jira-relay webhook verification
- A guarded Text2SQL endpoint with read-only Athena templates
- AWS Glue PySpark job and Databricks analytics notebook
- An offline DSPy optimization experiment for workflow diagnosis
- Terraform for a budget-conscious AWS serverless deployment
- Customer onboarding, workshop and ROI guidance
Bootstrap data lets a new workspace run before external services are connected. Live connectors make network calls only after credentials are configured.
Engineering-delivery data is fragmented:
- Jira knows the planned work, status and blockers.
- GitLab knows merge requests, review timing and pipeline results.
- CI records repeated failures and wasted execution time.
- Managers still struggle to explain why a release slowed down.
- AI recommendations often sound plausible without evidence or follow-up measurement.
DeliveryLens AI turns those disconnected events into a measurable workflow:
| Question | DeliveryLens answer |
|---|---|
| Which merge requests waited longest for review? | Ticket-level review pickup metrics and evidence |
| Are blocked Jira tickets escalating quickly enough? | Blocked-hour calculation and escalation recommendation |
| Are CI failures consuming meaningful time? | Pipeline failure rate and retry-minute estimate |
| Which work is not linked correctly? | Percentage of Jira tickets without a GitLab merge request |
| Did the intervention help? | Baseline vs. current-sprint comparison with explicit ROI assumptions |
This is not an incident-response system. It analyzes organizational delivery flow from ticket creation to deployment rather than production logs, Kubernetes state or service health.
| Area | Capability |
|---|---|
| Enterprise integrations | Jira Cloud REST client, GitLab GraphQL client, webhook verification |
| GraphQL | Versionless GitLab /api/graphql query for merge requests and head pipelines |
| Workflow diagnosis | Transparent baseline identifies slow reviews, blocked work, CI retry waste and missing links |
| Flow metrics | Cycle time, blocked time, review pickup time, review duration, pipeline failure rate, deployment frequency |
| ROI measurement | Before/after metric comparison with editable hourly-cost assumption and attribution guidance |
| Guarded Text2SQL | Natural-language question -> allowlisted Athena SQL template -> safe result |
| AWS serverless | CloudFront, private S3, API Gateway HTTP API, Lambda, SQS, DLQ, DynamoDB, Step Functions |
| Data engineering | Immutable S3 raw-event lake, Glue PySpark ETL, curated Parquet output, Athena workgroup |
| Databricks | Reproducible PySpark notebook for metric and ROI exploration |
| DSPy | Offline MIPROv2 optimization experiment against labelled workflow cases |
| Human approval | Local recommendation review API and Step Functions callback-token workflow |
| Customer enablement | Public onboarding guide, workshop, API reference, deployment guide and ROI methodology |
| CI portability | GitHub Actions as primary CI plus a GitLab CI mirror |
The local dashboard is designed as a recruiter-ready product walkthrough. It exposes:
- 3D command-center overview - animated delivery-flow visualization, live workspace status and a concise explanation of the product value.
- Flow metrics - cycle time, review pickup, blocked hours, CI waste, failures and unlinked work.
- Workflow diagnoses - each recommendation includes ticket-level evidence and a metric to watch.
- ROI estimate - baseline and current values are visible alongside the editable cost assumption.
- Guarded Text2SQL - a manager can ask a supported delivery question, use query presets and inspect the read-only SQL template before execution.
- Runtime readiness - the dashboard distinguishes deterministic runtime diagnosis, optional offline DSPy optimization and enterprise connector status.
- Human approval queue - recommendations can be approved or rejected locally.
- Ticket-level evidence explorer - the bootstrap flow rows remain visible for audit.
The static frontend is served by FastAPI locally and by S3 plus CloudFront in AWS.
flowchart LR
Browser --> CF[CloudFront]
CF --> S3FE[Private S3 frontend]
CF --> APIGW[API Gateway HTTP API]
APIGW --> API[FastAPI Lambda]
Jira[Jira Cloud REST + webhook relay] --> APIGW
GitLab[GitLab GraphQL + project webhooks] --> APIGW
API --> SQS[SQS ingestion queue]
SQS --> Worker[Worker Lambda]
SQS -. 3 failed attempts .-> DLQ[SQS DLQ]
Worker --> Raw[S3 raw event lake]
Worker --> DDB[DynamoDB event index]
Scheduler[EventBridge Scheduler] --> Glue[Glue PySpark ETL]
Raw --> Glue
Glue --> Gold[S3 curated Parquet]
Gold --> Athena[Athena read-only views]
API --> T2SQL[Guarded Text2SQL]
T2SQL --> Athena
API --> SFN[Step Functions approval workflow]
SFN --> Approval[SQS approval queue]
Read docs/architecture.md before extending the AWS path. It separates what is fully implemented from the production integration points that require organization-specific credentials and schema mapping.
- FastAPI loads reproducible delivery events from
data/sample_events.json. - The metric engine groups events by Jira ticket key.
- It calculates cycle time, review pickup time, review duration, blocked time and CI retry waste.
- The deterministic diagnosis baseline identifies evidence-backed bottlenecks.
- The ROI calculator compares
baselineandcurrentsprint metrics. - The frontend renders metrics, recommendations, connector status and query results.
- The user asks a supported delivery question.
- The compiler maps the question to an allowlisted intent.
- It returns a read-only Athena SQL template restricted to a curated view.
- The local runtime executes the equivalent Python aggregation against bootstrap events.
- Unsupported questions are rejected with HTTP
422.
The application never executes arbitrary model-generated SQL.
- A verified Jira-relay or GitLab webhook reaches API Gateway.
- The API Lambda verifies the configured secret.
- The accepted raw payload is queued in SQS.
- The worker Lambda stores the immutable payload in S3 and indexes it in DynamoDB.
- After schema normalization, Glue PySpark produces curated Parquet metric rows.
- Athena exposes read-only metric views through a workgroup with a
100 MBper-query scan limit.
| Layer | Technology |
|---|---|
| Frontend | Responsive static HTML, CSS and JavaScript dashboard with CSS-based 3D visualization |
| Local API | FastAPI + Pydantic |
| AWS API runtime | Lambda ZIP + Mangum + API Gateway HTTP API |
| Edge hosting | Private S3 origin + CloudFront origin access control |
| Async ingestion | SQS + DLQ + worker Lambda |
| Raw persistence | S3 event lake + DynamoDB event index |
| Workflow approval | AWS Step Functions Standard callback token + SQS approval queue |
| Batch analytics | AWS Glue PySpark -> S3 Parquet |
| SQL analytics | Athena curated views + enforced workgroup scan limit |
| Notebook analysis | Databricks PySpark source notebook |
| Prompt optimization | DSPy MIPROv2 offline experiment |
| Infrastructure | Terraform |
| CI | GitHub Actions and GitLab CI |
Requirements:
- Docker Desktop with Docker Compose
- PowerShell
From the repository root:
Copy-Item .env.example .env
docker compose up --build -dBefore starting Docker, adjust .env when needed:
# Change this when port 8000 is already occupied
DELIVERYLENS_PORT=8010
# Optional: required only for the offline DSPy optimizer
OPENAI_API_KEY=
DSPY_MODEL=openai/gpt-4.1-mini
Docker Compose reads .env, builds the FastAPI image and serves both the API and the
frontend. The default URLs are:
Dashboard: http://localhost:8000
OpenAPI: http://localhost:8000/docs
Health: http://localhost:8000/api/health
If port 8000 is already in use, set this in .env before starting Docker:
DELIVERYLENS_PORT=8010
Then open http://localhost:8010.
Useful Docker commands:
docker compose logs -f api
docker compose ps
docker compose downKeep .env private. It is ignored by Git. Avoid sharing the output of
docker compose config, because Docker expands environment-variable values in that
output.
Use this path when editing backend code and you want Uvicorn reload behavior:
.\scripts\run_local.ps1 -SetupChoose a different port when needed:
.\scripts\run_local.ps1 -Port 8010This helper creates .env when missing, installs development dependencies during setup
and launches FastAPI directly.
Set the local URL once. Use 8010 here if you configured a different Docker port:
$baseUrl = "http://localhost:8000"Health check:
Invoke-RestMethod "$baseUrl/api/health"Ask a guarded Text2SQL question:
$body = @{ question = "Which tickets waited longest for review?" } | ConvertTo-Json
Invoke-RestMethod "$baseUrl/api/query" `
-Method Post `
-ContentType "application/json" `
-Body $bodyInspect the GitLab GraphQL query without credentials:
Invoke-RestMethod "$baseUrl/api/integrations/gitlab/sample-query"Approve a bootstrap recommendation:
$body = @{ decision = "approved" } | ConvertTo-Json
Invoke-RestMethod "$baseUrl/api/recommendations/rec-review-slo/review" `
-Method Post `
-ContentType "application/json" `
-Body $bodyRead docs/public-api.md for the endpoint table.
Copy .env.example to .env and set the credentials for a sandbox project:
JIRA_BASE_URL=https://your-domain.atlassian.net
JIRA_EMAIL=you@example.com
JIRA_API_TOKEN=...
JIRA_WEBHOOK_SECRET=...
GITLAB_BASE_URL=https://gitlab.com
GITLAB_TOKEN=...
GITLAB_PROJECT_FULL_PATH=your-group/your-project
GITLAB_WEBHOOK_SECRET=...
The GitLab adapter calls the official GitLab GraphQL API. The Jira adapter calls Jira Cloud REST API v3 and the onboarding guide explains the signed webhook relay pattern. Read docs/integrations.md.
.\scripts\run_tests.ps1Or:
python -m unittest discover -s tests -v
python -m compileall -q backend analytics evals testsThe test suite verifies:
- Event-to-ticket aggregation
- Linked and unlinked Jira-to-GitLab work
- Pipeline failure and blocker metrics
- ROI calculation assumptions
- Allowlisted Text2SQL query templates
- Rejection of unsupported questions
- Jira relay HMAC verification
- GitLab webhook token verification
- FastAPI health, dashboard and query endpoints
The FastAPI dashboard does not require an LLM key. It uses a deterministic, auditable diagnosis baseline and allowlisted Text2SQL templates so the complete local demo remains available without external model calls.
DSPy is used offline to optimize a structured diagnosis program against labelled
workflow cases. Add the OpenAI key to .env:
OPENAI_API_KEY=sk-your-key-here
DSPY_MODEL=openai/gpt-4.1-mini
Do not commit .env; it is already ignored by Git.
python -m pip install -r requirements-dspy.txt
python evals\dspy\optimize_diagnosis.pyThe script loads .env from the repository root. The default model is
openai/gpt-4.1-mini; override it with DSPY_MODEL in .env when needed.
The dashboard reports whether the DSPy optimizer is configured, but never sends the API key to the browser.
The script writes:
evals/dspy/outputs/optimized_diagnosis.json
Expand data/dspy_training_examples.jsonl before enabling an optimized program in a
production environment. The included cases validate the optimization pipeline only.
Two analytics components are included:
| Component | Purpose |
|---|---|
analytics/glue/delivery_flow_etl.py |
AWS Glue PySpark batch job for curated Parquet metrics |
analytics/databricks/delivery_flow_analysis.py |
Databricks notebook source export for metric and ROI exploration |
To run the Databricks analysis, upload data/sample_events.json to a workspace volume,
adjust SOURCE_PATH, and execute the notebook. Databricks reproduces the Spark metric
transformations independently of the AWS Glue production runtime.
The Terraform stack provisions:
- Private S3 frontend bucket and CloudFront distribution
- API Gateway HTTP API and Lambda ZIP runtime
- SQS ingestion queue, DLQ and worker Lambda
- Encrypted, versioned S3 event lake
- DynamoDB event index
- Step Functions callback-token approval workflow
- Glue PySpark job and disabled-by-default daily scheduler
- Athena read-only workgroup with enforced scan limit
Run:
.\scripts\build_lambda.ps1
$env:TF_VAR_jira_webhook_secret = "replace-me"
$env:TF_VAR_gitlab_webhook_secret = "replace-me"
terraform -chdir=infra\terraform init
terraform -chdir=infra\terraform plan
terraform -chdir=infra\terraform apply
.\scripts\deploy_frontend.ps1
terraform -chdir=infra\terraform output -raw cloudfront_urlThe Glue daily schedule is disabled by default. Enable it only after normalized input exists and after reviewing Glue cost:
terraform -chdir=infra\terraform apply -var="enable_daily_glue_schedule=true"Read docs/aws-deployment.md for the full process. Configure an AWS Budget alert before deploying.
Public technical guidance is part of the product, not an afterthought:
| Document | Purpose |
|---|---|
| Customer onboarding | 30-minute sandbox validation setup |
| 15-minute workshop | Guided enablement session |
| Architecture | System design, boundaries and security decisions |
| Integration guide | Jira REST, GitLab GraphQL and webhook configuration |
| ROI methodology | Savings calculation and attribution limits |
| AWS deployment | Budget-conscious serverless deployment |
| Public API | Endpoint reference |
deliverylens-ai/
|-- .github/workflows/
| |-- ci.yml
| `-- deploy-aws.yml
|-- analytics/
| |-- athena/create_views.sql
| |-- databricks/delivery_flow_analysis.py
| `-- glue/delivery_flow_etl.py
|-- backend/
| |-- Dockerfile
| `-- app/
| |-- integrations/
| | |-- gitlab.py
| | `-- jira.py
| |-- aws_runtime.py
| |-- config.py
| |-- diagnosis.py
| |-- lambda_handler.py
| |-- main.py
| |-- metrics.py
| |-- security.py
| |-- store.py
| |-- text2sql.py
| `-- worker.py
|-- data/
| |-- dspy_training_examples.jsonl
| |-- recommendations.json
| `-- sample_events.json
|-- docs/
|-- evals/dspy/optimize_diagnosis.py
|-- frontend/
| |-- app.js
| |-- index.html
| `-- styles.css
|-- infra/terraform/
|-- scripts/
|-- tests/
|-- .env.example
|-- .gitlab-ci.yml
|-- docker-compose.yml
|-- pyproject.toml
|-- requirements-dev.txt
|-- requirements-dspy.txt
`-- requirements.txt
Before connecting an operational workspace:
- Deploy the CloudFront distribution and configure an AWS Budget alert.
- Connect one sandbox Jira project.
- Connect one sandbox GitLab project and receive a webhook.
- Replace bootstrap ROI values with operational data.
- Move integration secrets to AWS Secrets Manager.
- Run the Databricks notebook and compare its output with the Glue metric layer.
- Add at least 30 labelled DSPy evaluation cases before enabling optimized diagnosis.
- Configure CloudWatch alarms for Lambda failures, DLQ depth and Glue job failures.
- Complete the onboarding workshop with the first engineering team.
- GitLab GraphQL API
- Jira Cloud REST API v3 webhooks
- AWS Step Functions callback task tokens
- Athena workgroup query controls
- Databricks Free Edition
- DSPy optimizers
MIT