AI enrichment pipeline with auto-enrich reports

.agents/LEARNINGS.md

@@ -2,7 +2,7 @@
 
 ## User Preferences
 
-- Keep Wildcard lightweight with minimal permissions; avoid heavy, screenpipe-style capture by default.
+- Keep Wildcard lightweight with minimal permissions; avoid heavy capture by default.
 - Deprioritize context-switching tips; AI-agent workflows can require frequent intentional switching.
 - After wildcard code updates, do not rely on `pm2 restart` (cache/stale behavior). Use:
   `pm2 delete wildcard` then `pm2 start ~/.pm2/ecosystem.config.js --only wildcard`.
@@ -26,6 +26,7 @@
 - For macOS app-title capture reliability and cleaner TCC attribution, running wildcard via launchd (`install-launchd`, `com.wildcard.wildcard`) works better than PM2.
 - In `AppUsageTracker.tick`, always catch/log sample flush errors to prevent unhandled interval-loop failures; also refresh `window_title` while app identity remains unchanged.
 - In `src/wildcard.ts`, `commander` command definitions plus typed option objects are easier to maintain than mutable `argv` helpers (`pop_flag`/`has_flag`) while preserving CLI behavior.
+- For hidden Commander compatibility aliases, use `.command(name, { hidden: true })`; `.hideHelp()` exists for `Option`, not `Command`.
 - For local CLI iteration, symlink `src/wildcard.ts` directly (`ln -sf "$(pwd)/src/wildcard.ts" ~/.bun/bin/wildcard`) so daily commands do not depend on a repo-root launcher wrapper.
 - Keeping `src/wildcard.ts` as thin commander wiring and moving handlers into `src/commands/*` makes future command changes safer and easier to review.
 - For long-running CLI command handlers, splitting orchestration (`watch.ts`) from runtime loop (`watch_runtime.ts`) and diagnostics (`status.ts`) keeps behavior stable while reducing file complexity.
@@ -36,16 +37,34 @@
 - In sync flows, extracting tiny helpers for cursor and error persistence reduces duplicate `setState`/`flush` blocks without changing runtime behavior.
 - For sync shutdown verification, use a delayed `/bulk-upload/v1` server and assert all three: upload started, shutdown latency stayed low, and `sync.*.cursor_rowid` did not advance.
 - In shutdown-abort sync tests, `watch` may still create extra `app_usage` rows while running; assert cursor/pending-event invariants rather than an exact `pending_app_usage` count.
-- When refreshing docs/tutorials, treat `src/wildcard.ts` and `ARCHITECTURE.md` as source of truth first; old `apps/*`, `packages/*`, and Screenpipe references can linger after architecture consolidation.
+- When refreshing docs/tutorials, treat `src/wildcard.ts` and `ARCHITECTURE.md` as source of truth first; old `apps/*` and `packages/*` references can linger after architecture consolidation.
+- For `docs/`, keep `docs/ARCHITECTURE.md` as the canonical system spec and treat the HTML pages as topic-specific visual explainers; trim duplication instead of letting each HTML page become a parallel architecture doc.
 - During snake_case -> kebab-case migrations, sweep architecture/docs tables and diagrams for stale underscore filenames to keep docs executable.
 - For report-style commands with human output on `stdout`, send progress/status lines to `stderr` to keep markdown/JSON pipe-safe.
 - For table-like CLI output with variable project/branch labels, compute the label column width from the current rows instead of hard-coding `padEnd(...)`; fixed widths break quickly on long branch names.
 - AI provider env var fallbacks should be provider-specific: only use `OPENAI_API_KEY` when provider is `openai`, not `ollama`.
-- For architecture-probe workflows, prefer creating `scratch/` scripts that favor read-only real module imports and print resolved module/config boundaries before performing any writes.
+- When reorganizing repo docs into `docs/`, keep a small root `SECURITY.md` pointer because GitHub expects that filename at the repository root for security-policy discovery.
+- For architecture-probe workflows, prefer creating hidden `.scratch/` scripts that favor read-only real module imports and keep all temp DB/files inside `.scratch/`.
+- For updated scratch refreshes, mirror the numbered-probe layout: `01-*` through `0N-*`, `99-compose.ts`, and `.scratch/Tutorial.md`.
+- `TimelineDb` does not expose a generic `queryAppUsage()` façade; use `queryAppUsageEndingBefore(...)` or `getAppUsageSummary(...)` for public app-usage reads.
 - `ConfigStore` + `wildcard` CLI outputs show that command behavior is mostly orchestrator-based and storage-driven; this helps in architecture mapping.
 - `SyncEngine.syncOnce()` behavior is visible with a temp local `/bulk-upload/v1` server, but empty app_usage streams can leave corresponding cursor keys unset until data arrives.
 - For time-bucketing utilities, anchor bucket boundaries to the requested range start instead of epoch multiples so local day/week labels do not drift in non-UTC timezones.
 - For prompt readability in summary/report flows, keep stable system instructions in feature-local markdown files (`src/summary/prompts/*.md`) and keep dynamic session/project payload assembly in code.
+- For server config precedence where `config` must beat env, keep server env resolution out of `ConfigStore.load()` and use `ConfigStore.loadSources()` plus a server-specific resolver instead of inferring from merged defaults.
+- For Bun package hygiene, `files: ["src"]` still ships `*.test.ts`; use explicit `src/**/*.ts` + `src/**/*.md` entries with a `!src/**/*.test.ts` negation to keep the published tarball runtime-only.
+- For managed `wildcard server` lifecycle, never trust the PID file alone; verify the live process command still matches `server start --child` before `stop` or `doctor`, or a reused PID can target an unrelated process.
+- When moving server entrypoints to the CLI namespace, update `ecosystem.config.cjs` in the same change so PM2 keeps using `src/wildcard.ts server start` and the `WILDCARD_SERVER_*` env names/default `timeline.db`.
+- For `server start --daemon`, the detached child must ignore `config.yaml` server fields and boot from the parent-resolved `WILDCARD_SERVER_*` env, or CLI flags can diverge from the actual bind host/port/db path.
+- PM2 should launch Bun directly with `interpreter: 'none'` and `script: ~/.bun/bin/bun`; PM2's Bun interpreter mode against `src/wildcard.ts` is unreliable here.
+- For ingest body limits, do not trust `Content-Length` alone; enforce `max_body_size_bytes` while streaming/chunking the request body so chunked uploads cannot bypass the cap.
+- For first-run UX and docs, avoid hard-coding one developer's workspace layout (like `~/Projects`); prefer portable examples such as `~/code` and let setup infer from common directories plus the current working directory.
+- When invoking `codex exec` in a fresh temp directory as a trust boundary, include `--skip-git-repo-check` or the subprocess will fail before doing any work.
+- In interactive setup flows, disabling an optional config block should explicitly delete it from the assembled config object; starting from `...existing` makes toggles sticky otherwise.
+- If `defaultConfig()` defines an optional config block, deleting that block from `config.yaml` does not disable it after `deepMerge`; add an explicit `enabled` flag or a load-time opt-out when true disablement matters.
+- When moving repo docs under `docs/`, keep lightweight root pointers for GitHub-discovered files such as `SECURITY.md` and `CONTRIBUTING.md`; otherwise repo metadata links regress even if the docs still exist.
+- When renaming CLI commands, update `README.md`, `SKILL.md`, `docs/`, relevant `tutorials/`, and any command-shape tests in the same change; hidden compatibility aliases do not keep docs/tests correct on their own.
+- In `cmdReport`, only auto-run enrichment when the report has an AI provider and enrichment is explicitly configured; otherwise heuristic/disabled-AI reports do unnecessary Codex work without affecting output.
 
 ## Patterns That Don't Work
 
@@ -59,12 +78,13 @@
   are not installed; use `pnpm dlx` or `bunx` launchers instead.
 - `@clack/prompts` `log.*` and `spinner()` write to `stdout`, so they pollute pipeable report output if used for progress.
 - With Commander, `--no-<flag>` sets `<flag>` to `false` (e.g. `opts.ai === false`), not `opts.no<Flag>`.
+- In this Commander version, `Command.hideHelp()` does not exist; trying to chain it on subcommands breaks startup and typecheck.
 
 ## Domain Notes
 
+- 2026-03-21: `AGENTS.md` still references the older `apps/*` + `packages/*` monorepo layout, but the active Wildcard runtime is consolidated under `src/`; for architecture and implementation questions, inspect `src/` and `docs/ARCHITECTURE.md` first.
 - Session thresholds tuned for file-edit tracking:
   `idle_threshold_s: 300`, `activity_window_s: 120`.
-- Screenpipe events arrive frequently (every few seconds), so session thresholds matter less when screenpipe is active.
 - Watcher can hold a DB lock: `report` may fail with `database is locked` while watcher is running; `show` uses read-only queries and is typically safe.
 - Frontmost-app metadata on file events is sparse; dedicated polled `app_usage` segments provide better app-time aggregation.
 - `frontmost_app` uses AppleScript AXTitle via `System Events`; permissions apply to the runtime context.
@@ -78,5 +98,5 @@
 
 - 2026-02-23: Ignore pattern updates should be mirrored in both `src/config/config-store.ts` (default config ignores) and `src/capture/ignore-patterns.ts` (defensive fallback) to keep watch filtering consistent when config is unavailable.
 - 2026-02-24: In `SKILL.md` report guidance, require a two-pass "timeline + project enrichment" workflow: run `show/report`, then inspect top projects' README/manifests and changed high-signal files to produce impact-oriented summaries with confidence and explicit unknowns.
-- 2026-03-03: For `scratch/` architecture probes, combine source parsing with real `wildcard --help`/`status --json` execution; parser-only command-dispatch checks can miss multiline `.action(...)` wiring.
+- 2026-03-03: For `.scratch/` architecture probes, combine source parsing with real `wildcard --help`/`status --json` execution; parser-only command-dispatch checks can miss multiline `.action(...)` wiring.
 - 2026-03-20: When refreshing Wildcard docs, verify `README.md`, `docs/ARCHITECTURE.md`, and `SKILL.md` against `src/wildcard.ts`, `src/config/config-store.ts`, and `src/server/bulk-upload-server.ts`; command flags, config defaults, and ingest routes drift independently.

.gitignore

@@ -16,6 +16,7 @@ report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
 
 # Runtime data
 pids
+.scratch/
 *.pid
 *.seed
 *.pid.lock

.npmignore

@@ -0,0 +1 @@
+**/*.test.ts

.specify/init-options.json

@@ -6,4 +6,4 @@
   "preset": null,
   "script": "sh",
   "speckit_version": "0.3.1"
-}
+}

.specify/memory/constitution.md

@@ -3,18 +3,23 @@
 ## Core Principles
 
 ### I. Local-First, Minimal Permissions
+
 Wildcard must stay local-first. Core capture, search, sessionization, and reporting flows must work without network dependencies, and new features must preserve the project's lightweight, minimal-permission posture by default.
 
 ### II. CLI-First, Pipe-Safe Interfaces
+
 Every user-facing workflow must be operable from the CLI. Human-readable output goes to `stdout`, progress and diagnostics go to `stderr`, and machine-readable modes must remain stable enough for automation.
 
 ### III. Small, Typed Changes
+
 Changes should follow the existing Bun + TypeScript patterns: strict typing, focused modules, 2-space indentation, single quotes, and small helpers over sprawling abstractions. Prefer extending current command/session/storage flows before introducing new layers.
 
 ### IV. Validation Before Commit
+
 Behavior changes must be backed by the smallest validation that proves them safe: focused tests for new logic, full `bun test` and `bun run check` before marking cross-cutting work complete, and CLI smoke checks for user-facing commands.
 
 ### V. Honest Specs and Docs
+
 Specs, task lists, README/tutorial content, and architecture docs must reflect the code that actually ships. Tasks may only be marked done after the corresponding implementation and validation have really happened.
 
 ## Additional Constraints

.specify/templates/checklist-template.md

@@ -6,16 +6,16 @@
 
 **Note**: This checklist is generated by the `/speckit.checklist` command based on feature context and requirements.
 
-<!-- 
+<!--
   ============================================================================
   IMPORTANT: The checklist items below are SAMPLE ITEMS for illustration only.
-  
+
   The /speckit.checklist command MUST replace these with actual items based on:
   - User's specific checklist request
   - Feature requirements from spec.md
   - Technical context from plan.md
   - Implementation details from tasks.md
-  
+
   DO NOT keep these sample items in the generated checklist file.
   ============================================================================
 -->

.specify/templates/constitution-template.md

@@ -1,50 +1,73 @@
 # [PROJECT_NAME] Constitution
+
 <!-- Example: Spec Constitution, TaskFlow Constitution, etc. -->
 
 ## Core Principles
 
 ### [PRINCIPLE_1_NAME]
+
 <!-- Example: I. Library-First -->
+
 [PRINCIPLE_1_DESCRIPTION]
+
 <!-- Example: Every feature starts as a standalone library; Libraries must be self-contained, independently testable, documented; Clear purpose required - no organizational-only libraries -->
 
 ### [PRINCIPLE_2_NAME]
+
 <!-- Example: II. CLI Interface -->
+
 [PRINCIPLE_2_DESCRIPTION]
+
 <!-- Example: Every library exposes functionality via CLI; Text in/out protocol: stdin/args → stdout, errors → stderr; Support JSON + human-readable formats -->
 
 ### [PRINCIPLE_3_NAME]
+
 <!-- Example: III. Test-First (NON-NEGOTIABLE) -->
+
 [PRINCIPLE_3_DESCRIPTION]
+
 <!-- Example: TDD mandatory: Tests written → User approved → Tests fail → Then implement; Red-Green-Refactor cycle strictly enforced -->
 
 ### [PRINCIPLE_4_NAME]
+
 <!-- Example: IV. Integration Testing -->
+
 [PRINCIPLE_4_DESCRIPTION]
+
 <!-- Example: Focus areas requiring integration tests: New library contract tests, Contract changes, Inter-service communication, Shared schemas -->
 
 ### [PRINCIPLE_5_NAME]
+
 <!-- Example: V. Observability, VI. Versioning & Breaking Changes, VII. Simplicity -->
+
 [PRINCIPLE_5_DESCRIPTION]
+
 <!-- Example: Text I/O ensures debuggability; Structured logging required; Or: MAJOR.MINOR.BUILD format; Or: Start simple, YAGNI principles -->
 
 ## [SECTION_2_NAME]
+
 <!-- Example: Additional Constraints, Security Requirements, Performance Standards, etc. -->
 
 [SECTION_2_CONTENT]
+
 <!-- Example: Technology stack requirements, compliance standards, deployment policies, etc. -->
 
 ## [SECTION_3_NAME]
+
 <!-- Example: Development Workflow, Review Process, Quality Gates, etc. -->
 
 [SECTION_3_CONTENT]
+
 <!-- Example: Code review requirements, testing gates, deployment approval process, etc. -->
 
 ## Governance
+
 <!-- Example: Constitution supersedes all other practices; Amendments require documentation, approval, migration plan -->
 
 [GOVERNANCE_RULES]
+
 <!-- Example: All PRs/reviews must verify compliance; Complexity must be justified; Use [GUIDANCE_FILE] for runtime development guidance -->
 
 **Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE]
+
 <!-- Example: Version: 2.1.1 | Ratified: 2025-06-13 | Last Amended: 2025-07-16 -->

.specify/templates/plan-template.md

@@ -29,7 +29,7 @@
 
 ## Constitution Check
 
-*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+_GATE: Must pass before Phase 0 research. Re-check after Phase 1 design._
 
 [Gates determined based on constitution file]
 
@@ -48,6 +48,7 @@ specs/[###-feature]/
 ```
 
 ### Source Code (repository root)
+
 <!--
   ACTION REQUIRED: Replace the placeholder tree below with the concrete layout
   for this feature. Delete unused options and expand the chosen structure with
@@ -98,7 +99,7 @@ directories captured above]
 
 > **Fill ONLY if Constitution Check has violations that must be justified**
 
-| Violation | Why Needed | Simpler Alternative Rejected Because |
-|-----------|------------|-------------------------------------|
-| [e.g., 4th project] | [current need] | [why 3 projects insufficient] |
-| [e.g., Repository pattern] | [specific problem] | [why direct DB access insufficient] |
+| Violation                  | Why Needed         | Simpler Alternative Rejected Because |
+| -------------------------- | ------------------ | ------------------------------------ |
+| [e.g., 4th project]        | [current need]     | [why 3 projects insufficient]        |
+| [e.g., Repository pattern] | [specific problem] | [why direct DB access insufficient]  |

.specify/templates/spec-template.md

@@ -5,13 +5,13 @@
 **Status**: Draft  
 **Input**: User description: "$ARGUMENTS"
 
-## User Scenarios & Testing *(mandatory)*
+## User Scenarios & Testing _(mandatory)_
 
 <!--
   IMPORTANT: User stories should be PRIORITIZED as user journeys ordered by importance.
   Each user story/journey must be INDEPENDENTLY TESTABLE - meaning if you implement just ONE of them,
   you should still have a viable MVP (Minimum Viable Product) that delivers value.
-  
+
   Assign priorities (P1, P2, P3, etc.) to each story, where P1 is the most critical.
   Think of each story as a standalone slice of functionality that can be:
   - Developed independently
@@ -75,7 +75,7 @@
 - What happens when [boundary condition]?
 - How does system handle [error scenario]?
 
-## Requirements *(mandatory)*
+## Requirements _(mandatory)_
 
 <!--
   ACTION REQUIRED: The content in this section represents placeholders.
@@ -85,22 +85,22 @@
 ### Functional Requirements
 
 - **FR-001**: System MUST [specific capability, e.g., "allow users to create accounts"]
-- **FR-002**: System MUST [specific capability, e.g., "validate email addresses"]  
+- **FR-002**: System MUST [specific capability, e.g., "validate email addresses"]
 - **FR-003**: Users MUST be able to [key interaction, e.g., "reset their password"]
 - **FR-004**: System MUST [data requirement, e.g., "persist user preferences"]
 - **FR-005**: System MUST [behavior, e.g., "log all security events"]
 
-*Example of marking unclear requirements:*
+_Example of marking unclear requirements:_
 
 - **FR-006**: System MUST authenticate users via [NEEDS CLARIFICATION: auth method not specified - email/password, SSO, OAuth?]
 - **FR-007**: System MUST retain user data for [NEEDS CLARIFICATION: retention period not specified]
 
-### Key Entities *(include if feature involves data)*
+### Key Entities _(include if feature involves data)_
 
 - **[Entity 1]**: [What it represents, key attributes without implementation]
 - **[Entity 2]**: [What it represents, relationships to other entities]
 
-## Success Criteria *(mandatory)*
+## Success Criteria _(mandatory)_
 
 <!--
   ACTION REQUIRED: Define measurable success criteria.