Skip to content

P2: Deepen prompt-ingest segmentation & distillation (ADR 0006 follow-ups) #53

Description

@macanderson

Follow-up to #44 / ADR 0006. The prompt-ingest provider (contextgraph-host/src/ingest.rs) ships a deliberately small, deterministic segmenter behind a single classify seam. The frame shapes are final and conformance-covered; these are additive quality improvements that never touch the wire.

Segmentation

  • Stack traces as a first-class kind. Today a stack trace folds into Log detection (the STACK_MARKERS heuristic). A dedicated StackTrace kind with its own distiller — keep the exception/panic line + top N frames + … (M more frames) — would beat the generic log distiller for the common "paste a traceback" case.
  • Richer tabular ingestion. looks_like_table / distill_table handle pipe and tab delimiters only. Add CSV (comma) and whitespace-aligned columns, and improve infer_column_type (currency, percent, nullable columns).
  • Classification precedence edge cases. Table detection runs before log, so a pipe-delimited log (ts | LEVEL | msg) classifies as Fact rather than Episode; a scheme-less example.com/path token becomes an anchor. Both stay honest (cost/invariants hold), so they're documented limits — worth refining the precedence and looks_like_path precision.

Distillation

  • Normalize common log timestamps to the F4 profile. leading_timestamp only sets valid_from/valid_to when the first token is already exactly YYYY-MM-DDTHH:MM:SSZ. Real logs (2026-07-20 18:00:01, syslog, bracketed) never match, so the temporal window rarely populates. Parse a few common shapes and normalize to F4 (guarded — never emit a non-F4 string, per Close enforcement gaps: audit max_frames, validate temporal fields as RFC 3339, probe as_of #10).
  • Duplicate-line collapse in logs. Collapse runs of identical log lines to … (×K) before selecting salient lines, so a spammy retry loop doesn't dominate the compact rendering.

Scope note: distillation quality is provider policy (ADR 0006), so none of this is normative — it only changes what the prompt-ingest provider chooses to inline.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions