feat: cumulative data, web research, threading, run monitoring

.github/copilot-instructions.md

@@ -2,6 +2,15 @@
 
 All rules are in [`spec/engineering/ai-agents.md`](../spec/engineering/ai-agents.md). Read that first.
 
+## Mandatory spec reads — every session
+
+Before responding to any coding task, you **must** have read every file listed in
+[`instructions/spec-files.instructions.md`](instructions/spec-files.instructions.md)
+in full during the current session — from disk, not from a summary or conversation history.
+
+The full manifest and the rule are in `spec/engineering/ai-agents.md` §2 ("Mandatory spec manifest").  
+A summarised spec is **not** a spec. If your context window holds only a paraphrase of a spec file, re-read it.
+
 ## Copilot-specific
 
 Scoped instructions in [`instructions/`](instructions/) auto-apply to matching paths via `applyTo` frontmatter — each is a thin pointer to a `spec/engineering/` file.

.github/instructions/spec-files.instructions.md

@@ -0,0 +1,49 @@
+---
+applyTo: "**"
+---
+
+## Spec files — mandatory reads before any code or spec work
+
+The spec is the source of truth for this repo. Before responding to any coding task,
+you **must** confirm that the following files have been read in full during the current
+session — from disk, not from a summary or paraphrase in conversation history.
+
+A summarised spec is not a spec. If any file below appears only as a bullet-point
+digest or paraphrase in your context window, call `read_file` to re-read it before
+proceeding.
+
+### Product spec
+
+```
+spec/product/01-vision.md
+spec/product/02-architecture.md
+spec/product/03-tenancy.md
+spec/product/04-capabilities/01-discovery.md
+spec/product/04-capabilities/02-enrichment.md
+spec/product/04-capabilities/03-qualification.md
+spec/product/04-capabilities/04-outreach.md
+spec/product/04-capabilities/05-follow-up.md
+spec/product/04-capabilities/06-reply-handling.md
+spec/product/04-capabilities/07-contact-discovery.md
+spec/product/04-capabilities/08-approval.md
+spec/product/05-config.md
+spec/product/06-cli.md
+spec/product/07-data-model.md
+spec/product/08-prompts.md
+spec/product/09-api.md
+spec/product/10-agent-graph.md
+spec/product/11-ui-dashboard.md
+```
+
+### Engineering rules
+
+```
+spec/engineering/ai-agents.md
+spec/engineering/spec-driven.md
+spec/engineering/secret-hygiene.md
+spec/engineering/tenant-isolation.md
+spec/engineering/code-style.md
+spec/engineering/commits.md
+```
+
+Full rule: `spec/engineering/ai-agents.md` §2 — "Mandatory spec manifest".

CLAUDE.md

@@ -7,3 +7,46 @@ All rules are in [`spec/engineering/ai-agents.md`](spec/engineering/ai-agents.md
 - Subagents and slash commands are defined in [`.claude/`](.claude/).
 - Use the `planner` subagent before any multi-file change.
 - Use the `drift-auditor` subagent to check spec/code drift.
+
+## Mandatory spec reads — session start
+
+Before reading any task or writing any code, call `Read` on every file in the spec manifest defined in
+`spec/engineering/ai-agents.md` §2 ("Mandatory spec manifest"). Summaries and conversation history
+are **not** a substitute — re-read from disk.
+
+Product spec files to read on every session start:
+
+```
+spec/product/01-vision.md
+spec/product/02-architecture.md
+spec/product/03-tenancy.md
+spec/product/04-capabilities/01-discovery.md
+spec/product/04-capabilities/02-enrichment.md
+spec/product/04-capabilities/03-qualification.md
+spec/product/04-capabilities/04-outreach.md
+spec/product/04-capabilities/05-follow-up.md
+spec/product/04-capabilities/06-reply-handling.md
+spec/product/04-capabilities/07-contact-discovery.md
+spec/product/04-capabilities/08-approval.md
+spec/product/05-config.md
+spec/product/06-cli.md
+spec/product/07-data-model.md
+spec/product/08-prompts.md
+spec/product/09-api.md
+spec/product/10-agent-graph.md
+spec/product/11-ui-dashboard.md
+```
+
+Engineering rules (read once per session):
+
+```
+spec/engineering/ai-agents.md
+spec/engineering/spec-driven.md
+spec/engineering/secret-hygiene.md
+spec/engineering/tenant-isolation.md
+spec/engineering/code-style.md
+spec/engineering/commits.md
+```
+
+A spec that was read in a prior session and then summarised does **not** count. If the current context
+window contains only a summary of a spec file, re-read the file.

alembic/versions/0005_cumulative_data_and_run_tracking.py

@@ -0,0 +1,133 @@
+"""Cumulative data model: tenant-scoped links, link_leads junction,
+contacts customer_id, campaign_runs table.
+
+Spec: spec/product/07-data-model.md
+
+Changes:
+- links.campaign_id: drop FK constraint + NOT NULL → nullable (provenance only).
+- links unique constraint: replace (tenant_id, campaign_id, url) with (tenant_id, url).
+- New table: link_leads — junction mapping links to leads per campaign.
+- contacts: add customer_id FK → customers.id (nullable); backfill from leads.
+- contacts: add unique constraint (customer_id, email) for cross-campaign dedup.
+- New table: campaign_runs — tracks non-blocking agent run lifecycle.
+
+Revision ID: 0005_cumulative_data_and_run_tracking
+Revises: 0004_customers_and_link_tracking
+"""
+
+from __future__ import annotations
+
+from typing import Sequence
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects.postgresql import TIMESTAMP, UUID
+
+revision: str = "0005_cumulative_data_and_run_tracking"
+down_revision: str | None = "0004_customers_and_link_tracking"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def upgrade() -> None:
+    # ------------------------------------------------------------------
+    # links — make campaign_id nullable (drop FK + NOT NULL)
+    # ------------------------------------------------------------------
+    # Drop old FK constraint (name follows Alembic convention)
+    op.drop_constraint("links_campaign_id_fkey", "links", type_="foreignkey")
+    # Drop old unique index on (tenant_id, campaign_id, url) if it exists
+    op.execute("DROP INDEX IF EXISTS idx_links_url")
+
+    op.alter_column("links", "campaign_id", nullable=True)
+
+    # New unique: one URL per tenant (global dedup)
+    op.create_unique_constraint("uq_links_tenant_url", "links", ["tenant_id", "url"])
+
+    # ------------------------------------------------------------------
+    # link_leads — junction table (link ↔ lead ↔ campaign)
+    # ------------------------------------------------------------------
+    op.create_table(
+        "link_leads",
+        sa.Column("id", UUID(as_uuid=False), primary_key=True),
+        sa.Column("tenant_id", UUID(as_uuid=False), sa.ForeignKey("tenants.id"), nullable=False),
+        sa.Column("link_id", UUID(as_uuid=False), sa.ForeignKey("links.id"), nullable=False),
+        sa.Column("lead_id", UUID(as_uuid=False), sa.ForeignKey("leads.id"), nullable=False),
+        sa.Column("campaign_id", UUID(as_uuid=False), sa.ForeignKey("campaigns.id"), nullable=False),
+        sa.Column("created_at", TIMESTAMP(timezone=True), nullable=False, server_default=sa.text("now()")),
+    )
+    op.create_unique_constraint("uq_link_leads", "link_leads", ["tenant_id", "link_id", "lead_id"])
+    op.create_index("idx_link_leads_link", "link_leads", ["tenant_id", "link_id"])
+    op.create_index("idx_link_leads_lead", "link_leads", ["tenant_id", "lead_id"])
+
+    # Backfill link_leads from existing leads rows that have a link_id
+    op.execute(
+        """
+        INSERT INTO link_leads (id, tenant_id, link_id, lead_id, campaign_id, created_at)
+        SELECT gen_random_uuid(), l.tenant_id, l.link_id, l.id, l.campaign_id, now()
+        FROM leads l
+        WHERE l.link_id IS NOT NULL
+        ON CONFLICT DO NOTHING
+        """
+    )
+
+    # ------------------------------------------------------------------
+    # contacts — add customer_id for cross-campaign dedup
+    # ------------------------------------------------------------------
+    op.add_column(
+        "contacts",
+        sa.Column("customer_id", UUID(as_uuid=False), sa.ForeignKey("customers.id"), nullable=True),
+    )
+    op.create_index("idx_contacts_customer", "contacts", ["tenant_id", "customer_id"])
+
+    # Backfill customer_id from the parent lead's customer_id
+    op.execute(
+        """
+        UPDATE contacts c
+        SET customer_id = l.customer_id
+        FROM leads l
+        WHERE c.lead_id = l.id
+          AND l.customer_id IS NOT NULL
+        """
+    )
+
+    # Cross-campaign unique guard: one contact row per (customer, email)
+    op.create_unique_constraint(
+        "uq_contacts_customer_email",
+        "contacts",
+        ["customer_id", "email"],
+    )
+
+    # ------------------------------------------------------------------
+    # campaign_runs — non-blocking agent run tracking
+    # ------------------------------------------------------------------
+    op.create_table(
+        "campaign_runs",
+        sa.Column("id", UUID(as_uuid=False), primary_key=True),
+        sa.Column("tenant_id", UUID(as_uuid=False), sa.ForeignKey("tenants.id"), nullable=False),
+        sa.Column("campaign_id", UUID(as_uuid=False), sa.ForeignKey("campaigns.id"), nullable=False),
+        sa.Column("status", sa.String(32), nullable=False, server_default="pending"),
+        sa.Column("current_node", sa.Text, nullable=True),
+        sa.Column("started_at", TIMESTAMP(timezone=True), nullable=True),
+        sa.Column("finished_at", TIMESTAMP(timezone=True), nullable=True),
+        sa.Column("error", sa.Text, nullable=True),
+        sa.Column("created_at", TIMESTAMP(timezone=True), nullable=False, server_default=sa.text("now()")),
+    )
+    op.create_index("idx_runs_campaign", "campaign_runs", ["tenant_id", "campaign_id"])
+    op.create_index(
+        "idx_runs_running",
+        "campaign_runs",
+        ["tenant_id", "campaign_id", "status"],
+        postgresql_where=sa.text("status = 'running'"),
+    )
+
+
+def downgrade() -> None:
+    op.drop_table("campaign_runs")
+    op.drop_constraint("uq_contacts_customer_email", "contacts", type_="unique")
+    op.drop_index("idx_contacts_customer", table_name="contacts")
+    op.drop_column("contacts", "customer_id")
+    op.drop_table("link_leads")
+    op.drop_constraint("uq_links_tenant_url", "links", type_="unique")
+    op.alter_column("links", "campaign_id", nullable=False)
+    op.create_foreign_key("links_campaign_id_fkey", "links", "campaigns", ["campaign_id"], ["id"])
+    op.create_index("idx_links_url", "links", ["tenant_id", "campaign_id", "url"], unique=True)

alembic/versions/0006_spec_alignment_columns.py

@@ -0,0 +1,35 @@
+"""Add spec-required DB columns: contacts.seniority_level, contacts.decision_maker_score,
+tenants.enabled.
+
+Spec refs:
+  - spec/product/07-data-model.md — contacts table: seniority_level TEXT, decision_maker_score NUMERIC
+  - spec/product/03-tenancy.md — tenants lifecycle: enabled BOOLEAN
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op
+
+revision: str = "0006_spec_alignment_columns"
+down_revision: str = "0005_cumulative_data_and_run_tracking"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # contacts: spec/product/07-data-model.md — seniority_level, decision_maker_score
+    op.add_column("contacts", sa.Column("seniority_level", sa.Text(), nullable=True))
+    op.add_column("contacts", sa.Column("decision_maker_score", sa.Numeric(), nullable=True))
+
+    # tenants: spec/product/03-tenancy.md — enabled flag for scheduler lifecycle
+    op.add_column(
+        "tenants",
+        sa.Column("enabled", sa.Boolean(), nullable=False, server_default=sa.text("true")),
+    )
+
+
+def downgrade() -> None:
+    op.drop_column("contacts", "seniority_level")
+    op.drop_column("contacts", "decision_maker_score")
+    op.drop_column("tenants", "enabled")

alembic/versions/0007_rename_customers_contacts_to_companies_people.py

@@ -0,0 +1,97 @@
+"""Rename customers/contacts to companies/people.
+
+Spec refs:
+  - spec/product/07-data-model.md
+  - spec/product/09-api.md
+
+This migration keeps existing data and renames the physical schema to match the
+runtime ORM and API terminology:
+- customers -> companies
+- contacts -> people
+- *_customer_id -> *_company_id
+- *_contact_id -> *_person_id
+"""
+
+from __future__ import annotations
+
+from typing import Sequence
+
+from alembic import op
+
+revision: str = "0007_rename_customers_contacts_to_companies_people"
+down_revision: str | None = "0006_spec_alignment_columns"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _rename_constraint(table: str, old_name: str, new_name: str) -> None:
+    op.execute(
+        f"""
+        DO $$
+        BEGIN
+            IF EXISTS (
+                SELECT 1
+                FROM pg_constraint
+                WHERE conname = '{old_name}'
+            ) THEN
+                EXECUTE 'ALTER TABLE {table} RENAME CONSTRAINT {old_name} TO {new_name}';
+            END IF;
+        END $$;
+        """
+    )
+
+
+def _rename_index(old_name: str, new_name: str) -> None:
+    op.execute(f"ALTER INDEX IF EXISTS {old_name} RENAME TO {new_name}")
+
+
+def upgrade() -> None:
+    op.rename_table("customers", "companies")
+    op.rename_table("contacts", "people")
+
+    op.alter_column("leads", "customer_id", new_column_name="company_id")
+    op.alter_column("people", "customer_id", new_column_name="company_id")
+    op.alter_column("messages", "contact_id", new_column_name="person_id")
+    op.alter_column("replies", "contact_id", new_column_name="person_id")
+    op.alter_column("events", "contact_id", new_column_name="person_id")
+
+    _rename_constraint("companies", "uq_customers_tenant_domain", "uq_companies_tenant_domain")
+    _rename_constraint("leads", "leads_customer_id_fkey", "leads_company_id_fkey")
+    _rename_constraint("people", "contacts_lead_id_fkey", "people_lead_id_fkey")
+    _rename_constraint("people", "contacts_customer_id_fkey", "people_company_id_fkey")
+    _rename_constraint("messages", "messages_contact_id_fkey", "messages_person_id_fkey")
+    _rename_constraint("replies", "replies_contact_id_fkey", "replies_person_id_fkey")
+    _rename_constraint("events", "events_contact_id_fkey", "events_person_id_fkey")
+    _rename_constraint("people", "uq_contacts_customer_email", "uq_people_company_email")
+
+    _rename_index("ix_leads_customer_id", "ix_leads_company_id")
+    _rename_index("idx_contacts_customer", "idx_people_company")
+    _rename_index("ix_messages_contact_id", "ix_messages_person_id")
+    _rename_index("ix_replies_contact_id", "ix_replies_person_id")
+    _rename_index("ix_events_contact_id", "ix_events_person_id")
+
+
+def downgrade() -> None:
+    _rename_index("ix_events_person_id", "ix_events_contact_id")
+    _rename_index("ix_replies_person_id", "ix_replies_contact_id")
+    _rename_index("ix_messages_person_id", "ix_messages_contact_id")
+    _rename_index("idx_people_company", "idx_contacts_customer")
+    _rename_index("ix_leads_company_id", "ix_leads_customer_id")
+
+    _rename_constraint("people", "uq_people_company_email", "uq_contacts_customer_email")
+    _rename_constraint("events", "events_person_id_fkey", "events_contact_id_fkey")
+    _rename_constraint("replies", "replies_person_id_fkey", "replies_contact_id_fkey")
+    _rename_constraint("messages", "messages_person_id_fkey", "messages_contact_id_fkey")
+    _rename_constraint("people", "people_company_id_fkey", "contacts_customer_id_fkey")
+    _rename_constraint("people", "people_lead_id_fkey", "contacts_lead_id_fkey")
+    _rename_constraint("leads", "leads_company_id_fkey", "leads_customer_id_fkey")
+    _rename_constraint("companies", "uq_companies_tenant_domain", "uq_customers_tenant_domain")
+
+    op.alter_column("events", "person_id", new_column_name="contact_id")
+    op.alter_column("replies", "person_id", new_column_name="contact_id")
+    op.alter_column("messages", "person_id", new_column_name="contact_id")
+    op.alter_column("people", "company_id", new_column_name="customer_id")
+    op.alter_column("leads", "company_id", new_column_name="customer_id")
+
+    op.rename_table("people", "contacts")
+    op.rename_table("companies", "customers")