feat: cumulative data, web research, threading, run monitoring#4
Open
madhyamakist wants to merge 18 commits into
Open
feat: cumulative data, web research, threading, run monitoring#4madhyamakist wants to merge 18 commits into
madhyamakist wants to merge 18 commits into
Conversation
…+ code)
- Tenant-scoped link dedup via link_leads junction table
- Web-search-based 3A/3B/3C enrichment research redesign
- Company-size non-blocking qualification
- Non-blocking ThreadPoolExecutor runner (ZER0_RUNNER_MAX_WORKERS)
- CampaignRunRow tracking + GET /runs, /runs/{id}, /stats endpoints
- contacts.customer_id cross-campaign dedup on (customer_id, email)
- Alembic migration 0005 covers all schema changes
Spec files updated: spec/product/02-architecture.md,
spec/product/04-capabilities/01-discovery.md,
spec/product/04-capabilities/02-enrichment.md,
spec/product/04-capabilities/03-qualification.md,
spec/product/07-data-model.md, spec/product/09-api.md
Also adds spec/engineering/lessons.md (L-001 through L-006) and
mandatory spec manifest enforcement in ai-agents.md §2.
Co-Authored-By: GitHub Copilot <noreply@github.com>
Dashboard now calls GET /api/v1/tenants to list all tenants directly from the database. The localStorage knownTenants workaround is removed. Backend: added GET /tenants list endpoint to tenant.py. Frontend: - useTenants hook rewritten to call api.listTenants() - tenant-context stripped of knownTenantIds/addKnownTenant/removeKnownTenant - TopBar uses useTenants() for the switcher dropdown (shows names not UUIDs) - Dashboard page simplified: no add-by-UUID form, no 'Not found in DB' row - TenantCard: removed onRemove prop - constants.ts: removed KNOWN_TENANTS_KEY Spec: spec/product/11-ui-dashboard.md (Dashboard home → GET /api/v1/tenants) Co-Authored-By: GitHub Copilot <noreply@github.com>
CampaignRow.links referenced links.campaign_id but migration 0005 dropped the FK constraint from that column (links are now tenant-scoped, with the campaign association living in link_leads). SQLAlchemy raised NoForeignKeysError on every request as a result. Fix: remove CampaignRow.links relationship — it is not used anywhere; access links per campaign via link_leads junction instead. Also updated TestCampaignTrigger to patch runner_service.submit (the new non-blocking thread-pool path) instead of the removed run_campaign shim. 130/130 tests pass. GET /api/v1/tenants smoke-tested successfully. Co-Authored-By: GitHub Copilot <noreply@github.com>
Critical fixes:
- approvals.py: stage filter 'qualified' to 'approval'; field names
lead.name/company/url to lead.company_name/domain; approve transition
'approved' (invalid) to 'outreach' with contacts.approved_for_outreach=True
- nodes.py: import linkedin_search (was missing, NameError at runtime)
- runner.py: detached-instance fix (loop inside with create_db_session())
Major spec fixes:
- contacts.py: GET + GET/{id} + PATCH /contacts with lead_id filter
- runner_service.py: recover_orphaned_runs() on startup; submit guard
covers pending AND running states
- api/__init__.py: _lifespan calls recover_orphaned_runs at startup
- settings.py: model_validator warns on missing jwt_secret + cred_enc_key
- tenant.py: TODO comment documenting JWT auth debt
DB / ORM:
- migration 0006: contacts.seniority_level TEXT,
contacts.decision_maker_score NUMERIC, tenants.enabled BOOLEAN
per spec/product/07-data-model.md and spec/product/03-tenancy.md
- models.py: ContactRow + TenantRow updated to match migration 0006
UI:
- campaign page: stats bar, Activity tab, Runs tab with NodePipeline
- LeadTable: correct LEAD_STAGES enum values with count badges
- contacts/page.tsx: Suspense wrapper for useSearchParams (Next.js 15)
- customers/page.tsx: new page linked from Sidebar
- api.ts: RunData, CampaignStats, ContactData types; listRuns, getRun,
getCampaignStats, listContacts, getContact, patchContact endpoints
- new hooks: useRuns, useCampaignStats, useContacts, useCustomers
Tests: 130 passed
Spec refs: spec/product/07-data-model.md, spec/product/03-tenancy.md,
spec/product/09-api.md, spec/product/04-capabilities/08-approval.md,
spec/product/10-agent-graph.md, spec/engineering/secret-hygiene.md
Co-Authored-By: GitHub Copilot <noreply@github.com>
…d gated progress - Add Leads view as first-class navigation item (per-tenant, all campaigns) - Enrich lead pipeline table: source badges (web/linkedin/directory), research freshness icons, hover details - Expand lead detail drawer with comprehensive information: - Research timeline with freshness indicators - Signals list with detection timestamps - Rejection reason alerts for rejected leads - Per-criterion qualification scoring with weights and benchmarks - Enhanced profile with detected language and source - Complete events audit trail - Refactor run progress indicator to show only agent-driven stages (5-stage flow): discover → scrape → identify → research → qualify (Rejected leads finish at qualification; no additional agent stages shown) - Rationale: Rejected leads require no action; qualified leads need operator approval (not agent-driven)
- Delete spec/product/06-cli.md completely (CLI out of v1 scope)
- Delete src/zer0/cli/ implementation entirely
- Remove POST /leads/{id}/trigger-followup from spec and code
- Scope v1 to approval stage post-qualification; follow-up pushed to v1.1
- Blocking via PATCH /leads/{id} with { blocked: true } is canonical
- Remove CLI from spec/product/02-architecture.md (module table and flowchart) - Remove CLI references from spec/product/11-ui-dashboard.md - Remove triggerFollowUp client method from src/ui/src/lib/api.ts - Fix mermaid diagram: remove spurious POST /block node (PATCH is canonical)
- Add 10-column customers table: company, domain, industry, business type, headcount, signal count, freshness badge, first seen, last enriched, notes preview - Add freshness icons (🟢 < 7 days / 🟡 7-30 days / 🔴 > 30 days) showing enrichment status - Add relative time formatting (e.g. '2 days ago') with absolute timestamps on hover - Add customer detail drawer showing: profile, research summary, signals list, operator notes - Make customer rows clickable to open detail view - Update spec/product/11-ui-dashboard.md with comprehensive Customers view and Customer detail sections
- Capture Gemini token usage (input/output/total + cost_usd) after every
LLM call via usage_sink ContextVar; flush to campaign_runs on completion
- Add Gemini pricing table (llm/pricing.py) with per-million-token rates
- DB migration 0008: add page_excerpt to links, add 5 cost columns to
campaign_runs (input_tokens, output_tokens, total_tokens, llm_call_count,
estimated_cost_usd)
- Populate links.page_excerpt (first 500 chars) during scrape_links node
- Expose page_excerpt in links API; expose cost columns in runs API
- Add GET /links/{id} endpoint for single-link fetch
- UI: show token/cost row on each run card; show source link + collapsible
page excerpt on lead detail; show source links table on company detail
- Spec: fix duplicate data-model block, add run_status enum, cost columns,
page_excerpt, and source-links UI sections for spec/code consistency
Co-Authored-By: Claude <noreply@anthropic.com>
…rides
Run cards now show per-stage event counts (links found, leads ID'd,
researched, qualified/rejected, people found, messages sent) sourced from
a new GET /campaigns/{id}/runs/{run_id}/events/summary endpoint. Events are
tagged with run_id via the existing ContextVar so counts are run-scoped.
Also adds migration 0009 (events.run_id) and writes run_id into every
write_event() call automatically from the ContextVar set by the runner.
Fix: campaign edit page was hardcoding discovery_sources, qualification_
threshold, outreach_channels, follow_up_count, and follow_up_spacing_days
to defaults, losing any saved override. GET /campaigns/{id} now returns
the four override JSONB columns; the edit page reads them on load.
Co-Authored-By: Claude <noreply@anthropic.com>
Migration was written but not applied — events queries were 500ing. Also adds gemini-2.5-flash and its preview variant to the pricing table so cost estimates are non-zero for that model. Co-Authored-By: Claude <noreply@anthropic.com>
Companies: drawer now fetches full detail (GET /companies/{id}) on open,
so source_links are always populated — the list endpoint doesn't join them.
People: API now joins person → lead → link and returns source_link on
every PersonOut. People table gets a Source column; clicking any row
opens a drawer showing contact details, outreach status, and the source
link URL with a collapsible page excerpt.
Co-Authored-By: Claude <noreply@anthropic.com>
Blocked domains (Facebook, LinkedIn, Twitter, Instagram, Reddit, YouTube, TikTok, Pinterest, Snapchat) now return "" immediately without making an HTTP request — they always 4xx/999 and the attempts fill the log with noise. Failed links were also being retried on every run because scraped_at was only stamped on success. Now scraped_at is always stamped (success or failure) and the skip condition checks scraped_at instead of page_text, so permanently-blocked links are only attempted once per link. Co-Authored-By: Claude <noreply@anthropic.com>
- Add migration 0010: scrape_status, page_type, page_summary, page_detail columns to links table; backfills existing rows - Rewrite node_scrape_links: skip only if scrape_status='scraped', set status='blocked' for known-bad domains (no HTTP call), status='failed' on error (retried next run), call LLM to classify page type + summary + detail on success - Remove LinkedIn from blocked-domains list (sometimes accessible) - Make links list endpoint campaign_id optional for tenant-wide view - Expose new fields in LinkOut API response and LinkData TS type - Add tenant-wide Links page with status badges, type, summary, drawer showing detail + full excerpt; add Links to sidebar nav Co-Authored-By: Claude <noreply@anthropic.com>
complete() takes keyword-only arguments; was being called with a positional string. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Strips the repo back to the spec skeleton before the product vision is rewritten.
What changed
Next step
Redefine spec/product/01-vision.md with the new product direction.