feat: implement LLM usage tracking with cost estimation and add web d…

README.md

@@ -191,6 +191,8 @@ A separate **coverage-service** microservice (`coverage-service/`) sits behind O
 3. Runs Python `diff-cover` to compute coverage *on the PR's changed lines only*.
 4. Asks an LLM (OpenAI / Anthropic) to generate unit tests for files below the threshold.
 5. Verifies the new tests pass with `node --test`, then recomputes coverage.
+6. Records LLM token usage and calculates estimated cost based on the model pricing.
+7. Displays PR runs, coverage deltas, token usage, and total spend in the Next.js **Coverage Dashboard** (`web/`).
 
 OpenReview then takes those generated files, **opens a stacked PR** against the original feature branch, and **posts a coverage-delta comment** on the original PR. See [Kenil27/band#4](https://github.com/Kenil27/band/pull/4) for an example.
 
@@ -264,9 +266,9 @@ cp .env.example .env                                # GITHUB_PAT, OPENAI_API_KEY
 cp service/.env.example service/.env                # add the same GITHUB_PAT
 cp coverage-service/.env.example coverage-service/.env  # add the same GITHUB_PAT + OPENAI_API_KEY
 
-# 3. Generate the Prisma client + push the schema to Postgres
+# 3. Generate the Prisma client + run database migrations
 pnpm coverage:db:generate
-pnpm coverage:db:push
+pnpm coverage:db:migrate
 
 # 4. Install the Python diff-cover binary the coverage worker shells out to
 pnpm coverage:setup:worker-deps
@@ -277,17 +279,19 @@ pnpm coverage:setup:worker-deps
 #       COVERAGE_SERVICE_URL=http://localhost:3010
 ```
 
-### Run the stack (4 terminals)
+### Run the stack (5 terminals)
 
 | # | Command | Binds | Role |
 |---|---|---|---|
 | 1 | `pnpm coverage:dev:api` | `:3010` | Coverage-service HTTP API (Nest) |
 | 2 | `pnpm coverage:dev:worker` | (none) | Coverage-service worker — does the actual clone + coverage + LLM work |
 | 3 | `pnpm --filter @openreview/service start:web` | `:3003` | OpenReview HTTP — webhooks + `/coverage-runs/trigger` |
 | 4 | `pnpm --filter @openreview/service start:worker` | (none) | OpenReview worker — opens the stacked PR + posts the comment |
+| 5 | `cd web && pnpm dev` | `:3000` | Next.js Coverage & Cost Monitoring Dashboard |
 
 > Ports are configurable. `:3010` is `API_PORT` in `coverage-service/.env`; `:3003` is `PORT` in `service/.env`.
 
+
 ### Verified end-to-end flow (curl path, no webhook needed)
 
 This is the exact sequence verified on [Kenil27/band#3 → #4](https://github.com/Kenil27/band/pull/4).

coverage-service/README.md

@@ -66,7 +66,7 @@ pnpm coverage:dev:worker
 ### 4. Register a repository
 
 ```bash
-curl -X POST http://localhost:3000/repositories \
+curl -X POST http://localhost:3010/repositories \
   -H "Content-Type: application/json" \
   -d '{
     "githubRepo": "owner/repo",
@@ -78,11 +78,50 @@ curl -X POST http://localhost:3000/repositories \
 
 ### 5. Configure GitHub webhook
 
-- **URL:** `http://<your-host>:3000/webhooks/github`
+- **URL:** `http://<your-host>:3010/webhooks/github`
 - **Content type:** `application/json`
 - **Secret:** same value as `WEBHOOK_SECRET` in `.env`
 - **Events:** Pull requests
 
+## Database Migrations
+
+Before starting the services, ensure all database migrations are applied to create the `LlmUsageRecord` and other tables:
+
+```bash
+pnpm coverage:db:migrate
+```
+
+If you modify the Prisma schema (`coverage-service/api/prisma/schema.prisma`), you can create a new migration locally using:
+
+```bash
+npx prisma migrate dev --name <migration_name> --schema=coverage-service/api/prisma/schema.prisma
+```
+
+## LLM Cost Monitoring Dashboard (Web UI)
+
+A Next.js dashboard is available in the `web/` directory to monitor coverage runs, LLM token counts, and estimated cost tracking.
+
+### Running the Dashboard
+
+```bash
+cd web
+pnpm install
+pnpm dev
+```
+
+Open [http://localhost:3000](http://localhost:3000) in your browser.
+
+### Configuration
+
+Ensure the following environment variables are aligned:
+
+*   **API (`coverage-service/.env`):**
+    *   `API_PORT=3010`
+    *   `DASHBOARD_SECRET`: A secret bearer token used to authenticate frontend dashboard requests.
+*   **Web UI (`web/.env.local`):**
+    *   `NEXT_PUBLIC_API_URL=http://localhost:3010` (points to the Coverage API port)
+    *   `DASHBOARD_SECRET`: Matches the API's secret token.
+
 ## API Endpoints
 
 | Method | Path | Description |
@@ -98,8 +137,11 @@ curl -X POST http://localhost:3000/repositories \
 | GET | `/pr-runs/:id/tests/:testId` | Download generated test |
 | POST | `/pr-runs/:id/retry` | Re-enqueue analysis |
 | GET | `/pr-runs/repository/:repositoryId` | List runs for a repo |
-| GET | `/test-generation-runs/:id` | Get test generation run status and JSON result (`generatedTest.fileName`, `generatedTest.content`) |
+| GET | `/test-generation-runs/:id` | Get test generation run status and JSON result |
+| **GET** | `/stats/global` | Exposes global LLM cost metrics (total spend, calls, tokens) |
+| **GET** | `/repositories/:id/cost-summary` | Exposes repository-specific cost details, breakdowns by model, and per-PR spending |
 
 ## Integration with OpenReview
 
 The main OpenReview review service (`@openreview/service`) can forward PR events to this service via its downstream dispatcher when `COVERAGE_SERVICE_URL` is configured (future integration).
+

coverage-service/api/prisma/migrations/20250624000000_add_llm_usage_record/migration.sql

@@ -0,0 +1,31 @@
+-- CreateTable
+CREATE TABLE "LlmUsageRecord" (
+    "id"                  TEXT NOT NULL,
+    "prRunId"             TEXT,
+    "testGenerationRunId" TEXT,
+    "provider"            TEXT NOT NULL,
+    "modelName"           TEXT NOT NULL,
+    "promptTokens"        INTEGER NOT NULL,
+    "completionTokens"    INTEGER NOT NULL,
+    "totalTokens"         INTEGER NOT NULL,
+    "estimatedCostUsd"    DOUBLE PRECISION,
+    "createdAt"           TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+
+    CONSTRAINT "LlmUsageRecord_pkey" PRIMARY KEY ("id")
+);
+
+-- CreateIndex
+CREATE INDEX "LlmUsageRecord_prRunId_idx" ON "LlmUsageRecord"("prRunId");
+
+-- CreateIndex
+CREATE INDEX "LlmUsageRecord_testGenerationRunId_idx" ON "LlmUsageRecord"("testGenerationRunId");
+
+-- AddForeignKey
+ALTER TABLE "LlmUsageRecord" ADD CONSTRAINT "LlmUsageRecord_prRunId_fkey"
+    FOREIGN KEY ("prRunId") REFERENCES "PullRequestRun"("id")
+    ON DELETE CASCADE ON UPDATE CASCADE;
+
+-- AddForeignKey
+ALTER TABLE "LlmUsageRecord" ADD CONSTRAINT "LlmUsageRecord_testGenerationRunId_fkey"
+    FOREIGN KEY ("testGenerationRunId") REFERENCES "TestGenerationRun"("id")
+    ON DELETE CASCADE ON UPDATE CASCADE;

coverage-service/api/prisma/schema.prisma

@@ -52,6 +52,7 @@ model PullRequestRun {
   coverageResult CoverageResult?
   generatedTests GeneratedTestArtifact[]
   executionLogs  ExecutionLog[]
+  llmUsage       LlmUsageRecord[]
 
   @@index([repositoryId, prNumber])
 }
@@ -95,6 +96,7 @@ model TestGenerationRun {
   completedAt  DateTime?
   generatedTest GeneratedTestArtifact?
   executionLogs TestGenerationLog[]
+  llmUsage      LlmUsageRecord[]
 
   @@index([repositoryId, prNumber])
 }
@@ -129,3 +131,25 @@ model ExecutionLog {
   message   String         @db.Text
   createdAt DateTime       @default(now())
 }
+
+model LlmUsageRecord {
+  id                  String             @id @default(cuid())
+  /// Set when the LLM call belongs to a full PR analysis run
+  prRunId             String?
+  prRun               PullRequestRun?    @relation(fields: [prRunId], references: [id], onDelete: Cascade)
+  /// Set when the LLM call belongs to a standalone test-generation run
+  testGenerationRunId String?
+  testGenerationRun   TestGenerationRun? @relation(fields: [testGenerationRunId], references: [id], onDelete: Cascade)
+
+  provider            String             // 'openai' | 'anthropic' | 'local'
+  modelName           String
+  promptTokens        Int
+  completionTokens    Int
+  totalTokens         Int
+  /// USD estimated at write-time from baked-in pricing table; null for unknown/local models
+  estimatedCostUsd    Float?
+  createdAt           DateTime           @default(now())
+
+  @@index([prRunId])
+  @@index([testGenerationRunId])
+}

coverage-service/api/scripts/with-env.mjs

@@ -25,7 +25,7 @@ if (args.length === 0) {
   process.exit(1);
 }
 
-const result = spawnSync('pnpm', ['exec', 'prisma', ...args], {
+const result = spawnSync('npx', ['prisma', ...args], {
   stdio: 'inherit',
   env: process.env,
   cwd: resolve(here, '..'),

coverage-service/api/src/app.module.ts

@@ -1,6 +1,7 @@
 import { Module } from '@nestjs/common';
 
 import { AnalysisModule } from './analysis/analysis.module';
+import { CostModule } from './cost/cost.module';
 import { GitHubModule } from './github/github.module';
 import { HealthController } from './health/health.controller';
 import { PrRunsModule } from './pr-runs/pr-runs.module';
@@ -20,7 +21,9 @@ import { WebhooksModule } from './webhooks/webhooks.module';
     TestGenerationModule,
     WebhooksModule,
     PrRunsModule,
+    CostModule,
   ],
   controllers: [HealthController],
 })
 export class AppModule {}
+

coverage-service/api/src/cost/cost.controller.ts

@@ -0,0 +1,18 @@
+import { Controller, Get, Param } from '@nestjs/common';
+
+import { CostService } from './cost.service';
+
+@Controller()
+export class CostController {
+  constructor(private readonly costService: CostService) {}
+
+  @Get('repositories/:repositoryId/cost-summary')
+  getRepositorySummary(@Param('repositoryId') repositoryId: string) {
+    return this.costService.getRepositorySummary(repositoryId);
+  }
+
+  @Get('stats/global')
+  getGlobalStats() {
+    return this.costService.getGlobalStats();
+  }
+}

coverage-service/api/src/cost/cost.module.ts

@@ -0,0 +1,14 @@
+import { Module } from '@nestjs/common';
+
+import { PrismaModule } from '../prisma/prisma.module';
+
+import { CostController } from './cost.controller';
+import { CostService } from './cost.service';
+
+@Module({
+  imports: [PrismaModule],
+  controllers: [CostController],
+  providers: [CostService],
+  exports: [CostService],
+})
+export class CostModule {}

coverage-service/api/src/cost/cost.service.ts

@@ -0,0 +1,146 @@
+import { Injectable } from '@nestjs/common';
+
+import { PrismaService } from '../prisma/prisma.service';
+
+export interface ModelCostBreakdown {
+  provider: string;
+  modelName: string;
+  calls: number;
+  promptTokens: number;
+  completionTokens: number;
+  totalTokens: number;
+  estimatedCostUsd: number;
+}
+
+export interface PrCostBreakdown {
+  prNumber: number;
+  prRunId: string;
+  calls: number;
+  totalTokens: number;
+  estimatedCostUsd: number;
+}
+
+export interface RepositoryCostSummary {
+  repositoryId: string;
+  githubRepo: string;
+  totalCalls: number;
+  totalTokens: number;
+  estimatedCostUsd: number;
+  byModel: ModelCostBreakdown[];
+  byPr: PrCostBreakdown[];
+}
+
+export interface GlobalStats {
+  totalRepositories: number;
+  totalPrRuns: number;
+  totalTestGenerationRuns: number;
+  totalLlmCalls: number;
+  totalTokens: number;
+  estimatedCostUsd: number;
+}
+
+@Injectable()
+export class CostService {
+  constructor(private readonly prisma: PrismaService) {}
+
+  async getRepositorySummary(repositoryId: string): Promise<RepositoryCostSummary> {
+    const repo = await this.prisma.repository.findUniqueOrThrow({
+      where: { id: repositoryId },
+    });
+
+    // Fetch all LLM usage records for this repository — via PR runs
+    const records = await this.prisma.llmUsageRecord.findMany({
+      where: {
+        OR: [
+          {
+            prRun: { repositoryId },
+          },
+          {
+            testGenerationRun: { repositoryId },
+          },
+        ],
+      },
+      include: {
+        prRun: { select: { prNumber: true, id: true } },
+        testGenerationRun: { select: { prNumber: true } },
+      },
+      orderBy: { createdAt: 'asc' },
+    });
+
+    // Aggregate by model
+    const modelMap = new Map<string, ModelCostBreakdown>();
+    for (const r of records) {
+      const key = `${r.provider}::${r.modelName}`;
+      const existing = modelMap.get(key) ?? {
+        provider: r.provider,
+        modelName: r.modelName,
+        calls: 0,
+        promptTokens: 0,
+        completionTokens: 0,
+        totalTokens: 0,
+        estimatedCostUsd: 0,
+      };
+      existing.calls += 1;
+      existing.promptTokens += r.promptTokens;
+      existing.completionTokens += r.completionTokens;
+      existing.totalTokens += r.totalTokens;
+      existing.estimatedCostUsd += r.estimatedCostUsd ?? 0;
+      modelMap.set(key, existing);
+    }
+
+    // Aggregate by PR
+    const prMap = new Map<string, PrCostBreakdown>();
+    for (const r of records) {
+      const prRunId = r.prRunId ?? r.testGenerationRun?.prNumber?.toString() ?? 'unknown';
+      const prNumber = r.prRun?.prNumber ?? r.testGenerationRun?.prNumber ?? 0;
+      const key = r.prRunId ?? `tg-pr-${prNumber}`;
+      const existing = prMap.get(key) ?? {
+        prNumber,
+        prRunId: r.prRunId ?? '',
+        calls: 0,
+        totalTokens: 0,
+        estimatedCostUsd: 0,
+      };
+      void prRunId;
+      existing.calls += 1;
+      existing.totalTokens += r.totalTokens;
+      existing.estimatedCostUsd += r.estimatedCostUsd ?? 0;
+      prMap.set(key, existing);
+    }
+
+    const totalTokens = records.reduce((s, r) => s + r.totalTokens, 0);
+    const estimatedCostUsd = records.reduce((s, r) => s + (r.estimatedCostUsd ?? 0), 0);
+
+    return {
+      repositoryId,
+      githubRepo: repo.githubRepo,
+      totalCalls: records.length,
+      totalTokens,
+      estimatedCostUsd,
+      byModel: [...modelMap.values()].sort((a, b) => b.estimatedCostUsd - a.estimatedCostUsd),
+      byPr: [...prMap.values()].sort((a, b) => b.prNumber - a.prNumber),
+    };
+  }
+
+  async getGlobalStats(): Promise<GlobalStats> {
+    const [totalRepositories, totalPrRuns, totalTestGenerationRuns, usageAgg] =
+      await Promise.all([
+        this.prisma.repository.count(),
+        this.prisma.pullRequestRun.count(),
+        this.prisma.testGenerationRun.count(),
+        this.prisma.llmUsageRecord.aggregate({
+          _count: { id: true },
+          _sum: { totalTokens: true, estimatedCostUsd: true },
+        }),
+      ]);
+
+    return {
+      totalRepositories,
+      totalPrRuns,
+      totalTestGenerationRuns,
+      totalLlmCalls: usageAgg._count.id,
+      totalTokens: usageAgg._sum.totalTokens ?? 0,
+      estimatedCostUsd: usageAgg._sum.estimatedCostUsd ?? 0,
+    };
+  }
+}