Skip to content

Preserve complete semantic provenance and add append-only review auditing #102

Description

@jmcte

Problem

enqueue_candidate() currently stores serialized evidence_refs in the database provenance column, discards source_message_ids, forces review_status='candidate', and omits extractor version and creation metadata.

Review decisions update the queue row in place, do not record reviewer identity, and allow unrestricted transitions. The runbook claims queue operations are auditable, but no durable audit event exists.

Relevant code

  • src/mailplus_intelligence/queue.py:13-64,68-94
  • src/mailplus_intelligence/semantic_contract.py
  • src/mailplus_intelligence/migrations/003_cache_and_queue.sql:17-39
  • src/mailplus_intelligence/exporters.py
  • docs/semantic-output-schema.md
  • docs/promotion-review-workflow.md
  • docs/audit-event-format.md
  • docs/integration/live-export.md

Scope

  • Define one canonical immutable semantic artifact envelope.
  • Preserve source thread, source message IDs, locators, evidence references, provenance type, extractor/model/rule version, creation time, confidence, and initial review state.
  • Validate artifacts before enqueue and again before export.
  • Add append-only review decision records with reviewer identity, timestamp, notes, correction, and prior/new state.
  • Implement explicit legal state transitions and optimistic concurrency/version checking.
  • Preserve the original summary when corrected.
  • Add export/outbox state sufficient for idempotency and rollback.
  • Provide a migration for existing databases, including handling of legacy rows where provenance contains evidence JSON.

Non-goals

  • Do not implement a web review UI.
  • Do not create remote reminders/wiki records.
  • Do not store raw bodies or unbounded evidence text.

Acceptance criteria

  • Enqueue round-trips every canonical artifact field without reinterpretation.
  • review_needed is not silently converted to candidate.
  • Invalid artifact type, confidence, source, or status is rejected before persistence.
  • Review history is append-only and includes reviewer identity.
  • Illegal or stale concurrent transitions fail clearly.
  • Corrections preserve original content and provenance.
  • Export requires an eligible latest review decision and complete provenance.
  • Legacy queue rows migrate deterministically with documented limitations.
  • Tests cover deterministic and LLM provenance, correction, rejection, concurrency, migration, export, and rollback-needed.

Validation

PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=src python3.12 -m unittest tests.test_semantic_contract tests.test_queue tests.test_exporters -v
bash scripts/ci/run-fast-checks.sh

Parent

Part of #98.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:dataData, schema, storage, privacy, or migration surface.area:securitySecurity, auth, secret, permission, or policy surface.bugSomething isn't workingpriority:P1Codex Connector P1; blocks execution until Athena and Ares validate.ready-for-agentIssue is approved for worker-lane execution.risk:highHigh-risk change; validation required.status:ready-for-agentIssue is approved for worker-lane execution.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions