feat(#5161): analyze-transcript audit subcommand, telemetry detection, HTTP filtering

[rh-hemartin]

Summary

Closes #5161.

• Add audit subcommand combining summary + errors + tool breakdown in one pass
• Detect OTLP telemetry files early and exit with a clear warning instead of producing confusing output
• Add --method and --host filters to network --http for targeted HTTP request analysis
• Harden SKILL.md: allowed-tools patterns, no-shell-variables rule, question routing table, missing-transcript troubleshooting, full audit workflow docs

Test plan

• Run audit against a real agent transcript JSONL and verify it outputs summary, tool counts, and errors in one pass
• Pass a run-telemetry.jsonl file to summary and verify it exits with a telemetry warning
• Run network <sandbox-log> --http --method POST --host github.com and verify filtering works
• Invoke the skill via /analyze-transcript and confirm no permission prompts for standard commands

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

Add analyze-transcript audit, OTLP telemetry detection, and HTTP request filters

✨ Enhancement 📝 Documentation 🕐 20-40 Minutes

AI Description

• Add audit subcommand to emit summary, tool counts, and errors in one pass.
• Detect OTLP telemetry JSONL early and exit with a clear warning.
• Add --method/--host filters for network --http and document full workflow.

Diagram

graph TD
  U["User"] --> CLI["analyze-transcript.py CLI"] --> FT{"File type\ndetection"} --> OK["Transcript parsing"] --> AUD["audit: summary+tools+errors"]
  OK --> SUM["summary/tools/errors/search"]
  U --> NET["network --http"] --> FIL{"method/host\nfilters"} --> OUT["Filtered HTTP list"]
  FT --> WARN["Telemetry warning\n(exit 1)"]

  subgraph Legend
    direction LR
    _u["User"] ~~~ _p["Process"] ~~~ _d{"Decision"}
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Refactor to shared aggregator used by summary/tools/errors

• ➕ Avoids duplicating accounting logic across cmd_summary and cmd_audit (tokens, stop reasons, model/session extraction).
• ➕ Makes it easier to keep output consistent across commands over time.
• ➖ Larger refactor surface area; higher risk of subtle output regressions in existing commands.
• ➖ More code churn than needed to deliver analyze-transcript: audit subcommand, telemetry detection, HTTP filtering, hardened skill rules #5161’s workflow improvement quickly.

2. Make telemetry detection an explicit flag (e.g., `--telemetry`)

• ➕ No heuristics; behavior is fully user-controlled.
• ➕ Avoids false positives if a transcript line accidentally contains OTLP-like keys.
• ➖ Worse UX for the common failure mode: users already pass the wrong file and want guidance.
• ➖ Requires users to understand the difference before they get help.

3. Support richer HTTP filtering (regex / status / path filters)

• ➕ More powerful investigations for network-heavy runs.
• ➕ Could reduce reliance on post-processing with grep.
• ➖ More CLI surface and testing needed; not required for the targeted method/host ask in analyze-transcript: audit subcommand, telemetry detection, HTTP filtering, hardened skill rules #5161.

Recommendation: Keep the PR’s approach: the audit single-pass command materially improves the common workflow without forcing a broad refactor, and heuristic OTLP detection is a pragmatic UX win for the frequent wrong-file case. If audit grows further, consider extracting a shared “transcript stats/errors/tools” aggregator to reduce drift between summary and audit outputs.

Files changed (2) +283 / -22

Enhancement (1) +178 / -3

analyze-transcript.pyAdd 'audit' subcommand, OTLP telemetry guard, and HTTP request filters +178/-3

Add 'audit' subcommand, OTLP telemetry guard, and HTTP request filters

• Introduces a lightweight file sniffer to detect OTLP telemetry JSONL and abort transcript-oriented commands with a clear warning. Adds a new 'audit' subcommand that aggregates summary stats, tool call counts, and detected errors in a single transcript pass. Extends 'network --http' output with '--method' and '--host' filters and makes the printed header reflect active filters.

skills/analyze-transcript/analyze-transcript.py

Documentation (1) +105 / -19

SKILL.mdDocument audit workflow and tighten skill execution rules +105/-19

Document audit workflow and tighten skill execution rules

• Adds explicit allowed-tools patterns and operational rules (relative paths only, no shell variables, output limiting, constrained 'find'). Documents question routing, missing-transcript troubleshooting (including OTLP telemetry files), HTTP filtering examples, and a full audit workflow sequence.

skills/analyze-transcript/SKILL.md

[github-actions]

Site preview

Preview: https://b90b1217-site.fullsend-ai.workers.dev

Commit: 0be49b669eb3cef1f74c45bcc24e0d689d493ed1

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

1. allowed-tools quote typo ✗ Dismissed 🐞 Bug ☼ Reliability

Description

skills/analyze-transcript/SKILL.md includes allowed-tools patterns with an unmatched " (e.g.,
Bash(mkdir -p ".transcripts/*)), which is likely not the intended pattern and may fail to match
real quoted shell commands. This matters because the same doc states mismatched commands vs
allowed-tools patterns can trigger permission prompts and derail the workflow.

Code

skills/analyze-transcript/SKILL.md[R12-15]

+  - Bash(mkdir -p .transcripts/*)
+  - Bash(mkdir -p ".transcripts/*)
+  - Bash(find .transcripts/run-*)
+  - Bash(find ".transcripts/run-*)

Relevance

⭐⭐⭐ High

Team often fixes SKILL.md command/pattern correctness; analyze-transcript SKILL.md doc fixes were
accepted.

PR-#746

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The frontmatter contains patterns with a stray/unmatched quote, and the skill’s own rules explain
that mismatches against these patterns can cause permission prompts.

skills/analyze-transcript/SKILL.md[9-16]
skills/analyze-transcript/SKILL.md[35-40]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`allowed-tools` includes patterns with an unmatched double-quote, which is likely a typo and may prevent quoted variants of the commands from matching the allowlist.

### Issue Context
The skill explicitly warns that commands drifting from `allowed-tools` patterns can trigger permission prompts; a typo in the patterns increases the chance of that drift.

### Fix Focus Areas
- skills/analyze-transcript/SKILL.md[9-16]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Audit misses error mentions ✓ Resolved 🐞 Bug ≡ Correctness

Description

cmd_audit does not implement the assistant-text “mentions” heuristics that cmd_errors uses (e.g.
matching permission denied, fatal error), so audit can miss failures that errors would
report. This breaks the expectation set by the new documentation that audit is “summary + errors +
tools in one pass.”

Code

skills/analyze-transcript/analyze-transcript.py[R437-450]

+        for btype, block in extract_content_blocks(msg):
+            if btype == "tool_result":
+                if _is_error_result(block):
+                    text = get_tool_result_text(block)
+                    errors.append((i, truncate(text.strip(), max_w)))
+                else:
+                    text = get_tool_result_text(block)
+                    if _RESULT_ERROR_PATTERNS.search(text):
+                        errors.append((i, truncate(text.strip(), max_w)))
+            elif role == "user" and btype == "text":
+                text = block if isinstance(block, str) else block.get("text", "")
+                if "<error>" in text:
+                    errors.append((i, truncate(text.strip(), max_w)))
+

Relevance

⭐⭐ Medium

No prior evidence on audit matching errors’ mention heuristics; could be treated as scope change,
not bugfix.

PR-#746

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

cmd_errors explicitly adds assistant-text “mentions” based on common failure phrases, while
cmd_audit’s error collection loop lacks any analogous assistant-text scanning.

skills/analyze-transcript/analyze-transcript.py[324-366]
skills/analyze-transcript/analyze-transcript.py[385-450]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The new `audit` subcommand claims to combine `summary + errors + tools`, but it only collects tool_result errors and user `<error>` strings. It omits the assistant-text error/denial heuristics present in `cmd_errors`, so `audit` can miss relevant failures.

### Issue Context
`cmd_errors` already has the desired behavior; `cmd_audit` should reuse the same logic (ideally via a shared helper) so the “quick audit” path is not less complete than running `errors` separately.

### Fix Focus Areas
- skills/analyze-transcript/analyze-transcript.py[324-366]
- skills/analyze-transcript/analyze-transcript.py[385-497]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Audit double-parses blocks ✓ Resolved 🐞 Bug ➹ Performance

Description

cmd_audit calls extract_content_blocks(msg) twice per message (once for tool counting, again for
error scanning), doubling block-walk overhead for assistant messages. For large transcripts, this
adds avoidable CPU and allocations in a command explicitly intended to be an efficient “one pass”
audit.

Code

skills/analyze-transcript/analyze-transcript.py[R421-438]

+        if role == "assistant":
+            model = msg.get("model")
+            if model:
+                models.add(model)
+            usage = msg.get("usage", {})
+            total_input_tokens += usage.get("input_tokens", 0)
+            total_output_tokens += usage.get("output_tokens", 0)
+            total_cache_read += usage.get("cache_read_input_tokens", 0)
+            total_cache_create += usage.get("cache_creation_input_tokens", 0)
+            sr = msg.get("stop_reason")
+            if sr:
+                stop_reasons[sr] += 1
+            for btype, block in extract_content_blocks(msg):
+                if btype == "tool_use":
+                    tool_counts[block.get("name", "unknown")] += 1
+
+        for btype, block in extract_content_blocks(msg):
+            if btype == "tool_result":

Relevance

⭐⭐⭐ High

They’ve accepted removing redundant parsing/computation (single-parse refactors) for efficiency in
similar codepaths.

PR-#3820
PR-#816

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The code shows two separate loops over extract_content_blocks(msg) in cmd_audit, one nested
under role == "assistant" and one unconditional for error detection.

skills/analyze-transcript/analyze-transcript.py[421-449]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`cmd_audit` iterates content blocks twice for the same message. This is redundant work and can be avoided by extracting blocks once per message and reusing them for both tool_use counting and tool_result/user-text error detection.

### Issue Context
This is a local refactor: materialize `blocks = list(extract_content_blocks(msg))` once per message, then iterate `blocks` for both purposes.

### Fix Focus Areas
- skills/analyze-transcript/analyze-transcript.py[402-450]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[fullsend-ai-review]

Review

Findings

Medium

• [protected-path] skills/analyze-transcript/SKILL.md, skills/analyze-transcript/analyze-transcript.py — This PR modifies files under the skills/ protected path. The changes are well-scoped to issue analyze-transcript: audit subcommand, telemetry detection, HTTP filtering, hardened skill rules #5161 and the PR description explains the rationale clearly. Human approval is required for all protected-path changes regardless of review outcome.

Low

• [edge-case] skills/analyze-transcript/analyze-transcript.py — detect_file_type() returns None (no warning) when none of the first 5 lines contain valid JSON, or when all JSON lines have unrecognized schemas (neither OTLP markers nor transcript type fields). Non-transcript files (CSV, plain text, or unknown JSON formats) pass through silently, producing empty output with no diagnostic. The behavior is graceful degradation (no crash), but a warning for "no recognized content type" would improve the user experience when a wrong file is passed.

• [documentation-consistency] skills/analyze-transcript/SKILL.md — New top-level sections (Rules, Question routing, Full audit workflow) are not precedented in other examined skills (e2e-health, replay-session, filing-issues, topissues), which typically keep guidance within the Workflow section. This may represent an intentional evolution of the skill documentation pattern — the added structure is beneficial for a complex skill — but is worth noting for cross-skill consistency.

Previous run

Review

Verdict: Approve — clean feature addition, well-scoped to issue #5161.

This PR adds three features to the analyze-transcript skill: a combined audit subcommand, OTLP telemetry file detection, and HTTP request filtering for the network subcommand. It also hardens the SKILL.md with allowed-tools patterns, explicit rules (no shell variables, relative paths, plain find), a question routing table, and missing-transcript troubleshooting. All changes trace to the authorized scope in #5161.

The code follows established patterns in analyze-transcript.py — argparse subcommands, cmd_* function naming, Counter-based aggregation, and consistent output formatting. The telemetry detection guard (detect_file_type) is a practical improvement that prevents confusing output when users accidentally pass OTLP span files instead of Claude transcripts.

Observations (non-blocking)

1. detect_file_type() silent pass-through for non-JSON files — If none of the first 5 lines contain valid JSON (e.g., a binary file or plain text log), the function returns None (treating it as a valid transcript). The subsequent parsing silently produces empty output. A warning for "no valid JSON lines found" would improve the user experience, but the current behavior is graceful degradation, not a crash.

2. cmd_audit() code duplication — The audit subcommand reimplements summary-computation and error-detection logic from cmd_summary() and cmd_errors() rather than sharing helpers. This is a maintainability concern: future fixes to summary or error logic would need to be applied in two places. Extracting shared helpers (e.g., _compute_summary(), _detect_errors()) would reduce drift risk. This is follow-up work, not a blocker for this PR.

3. No tests — No test file exists for analyze-transcript.py. The script is stdlib-only Python in the skills directory, and the project doesn't appear to require tests for skill scripts. The new detect_file_type() function and HTTP filtering logic have edge cases that would benefit from unit tests as the script grows.

Checks performed

• ✅ Scope authorization: all changes trace to analyze-transcript: audit subcommand, telemetry detection, HTTP filtering, hardened skill rules #5161
• ✅ PR title convention: feat(#5161) is correct per COMMITS.md — new CLI subcommand and flags are user-facing features
• ✅ No breaking changes requiring ! suffix
• ✅ No security concerns: CLI tool with no auth/permission surface
• ✅ No prompt injection patterns in SKILL.md content
• ✅ No secrets or credentials in diff
• ✅ SKILL.md documents all new features (audit, --method, --host)
• ✅ allowed-tools patterns are appropriately scoped
• ✅ No cross-repo contract impact (internal skill, no exported APIs)

---

Protected paths detected — this PR modifies files under one or more
protected paths. The review agent cannot approve PRs that touch these paths.
A human reviewer must approve this PR.

Protected files in this PR:

• skills/analyze-transcript/SKILL.md
• skills/analyze-transcript/analyze-transcript.py

---

Labels: PR adds features to the analyze-transcript skill (new subcommand, telemetry detection, HTTP filtering)

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 12:41 PM UTC · Completed 12:53 PM UTC
Commit: 0884698 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 9:30 AM UTC · Completed 9:43 AM UTC
Commit: 0be49b6 · View workflow run →