From c49e23e7efada592666f0505ead818183e63a192 Mon Sep 17 00:00:00 2001 From: daymade Date: Sat, 7 Mar 2026 14:54:33 +0800 Subject: [PATCH 001/137] release: v1.38.0 with continue-claude-work and skill-creator enhancements ## New Skill: continue-claude-work (v1.1.0) - Recover actionable context from local `.claude` session artifacts - Compact-boundary-aware extraction (reads Claude's own compaction summaries) - Subagent workflow recovery (reports completed vs interrupted subagents) - Session end reason detection (clean exit, interrupted, error cascade, abandoned) - Size-adaptive strategy for small/large sessions - Noise filtering (skips 37-53% of session lines) - Self-session exclusion, stale index fallback, MEMORY.md integration - Bundled Python script (no external dependencies) - Security scan passed, argument-hint added ## Skill Updates - **skill-creator** (v1.5.0): Complete rewrite with evaluation framework - Added agents/ (analyzer, comparator, grader) - Added eval-viewer/ (generate_review.py, viewer.html) - Added scripts/ (run_eval, aggregate_benchmark, improve_description, run_loop) - Added references/schemas.md (eval/benchmark schemas) - Expanded SKILL.md with inline vs fork guidance, progressive disclosure patterns - Enhanced package_skill.py and quick_validate.py - **transcript-fixer** (v1.2.0): CLI improvements and test coverage - Enhanced argument_parser.py and commands.py - Added correction_service.py improvements - Added test_correction_service.py - **tunnel-doctor** (v1.4.0): Quick diagnostic script - Added scripts/quick_diagnose.py - Enhanced SKILL.md with 5-layer conflict model - **pdf-creator** (v1.1.0): Auto DYLD_LIBRARY_PATH + rendering fixes - Auto-detect and set DYLD_LIBRARY_PATH for weasyprint - Fixed list rendering and CSS improvements - **github-contributor** (v1.0.3): Enhanced project evaluation - Added evidence-loop, redaction, and merge-ready PR guidance ## Documentation - Updated marketplace.json (v1.38.0, 42 skills) - Updated CHANGELOG.md with v1.38.0 entry - Updated CLAUDE.md (skill count, marketplace version, #42 description) - Updated README.md (badges, skill section #42, use case, requirements) - Updated README.zh-CN.md (badges, skill section #42, use case, requirements) - Fixed absolute paths in continue-claude-work/references/file_structure.md ## Validation - All skills passed quick_validate.py - continue-claude-work passed security_scan.py - marketplace.json validated (valid JSON) - Cross-checked version consistency across all docs --- .claude-plugin/marketplace.json | 36 +- CHANGELOG.md | 24 + CLAUDE.md | 7 +- README.md | 49 +- README.zh-CN.md | 49 +- continue-claude-work.skill | Bin 0 -> 14898 bytes continue-claude-work/.security-scan-passed | 4 + continue-claude-work/SKILL.md | 141 ++ .../references/file_structure.md | 249 ++++ .../scripts/extract_resume_context.py | 837 +++++++++++ .../references/project_evaluation.md | 13 +- pdf-creator/scripts/md_to_pdf.py | 48 +- skill-creator/SKILL.md | 704 ++++++--- skill-creator/agents/analyzer.md | 274 ++++ skill-creator/agents/comparator.md | 202 +++ skill-creator/agents/grader.md | 223 +++ skill-creator/assets/eval_review.html | 146 ++ skill-creator/eval-viewer/generate_review.py | 471 ++++++ skill-creator/eval-viewer/viewer.html | 1325 +++++++++++++++++ skill-creator/references/schemas.md | 430 ++++++ skill-creator/scripts/__init__.py | 0 skill-creator/scripts/aggregate_benchmark.py | 401 +++++ skill-creator/scripts/generate_report.py | 326 ++++ skill-creator/scripts/improve_description.py | 248 +++ skill-creator/scripts/package_skill.py | 88 +- skill-creator/scripts/quick_validate.py | 80 +- skill-creator/scripts/run_eval.py | 310 ++++ skill-creator/scripts/run_loop.py | 332 +++++ skill-creator/scripts/utils.py | 47 + .../scripts/cli/argument_parser.py | 4 +- transcript-fixer/scripts/cli/commands.py | 44 +- .../scripts/core/correction_service.py | 8 + .../scripts/tests/test_correction_service.py | 25 + tunnel-doctor/SKILL.md | 55 + tunnel-doctor/scripts/quick_diagnose.py | 448 ++++++ 35 files changed, 7350 insertions(+), 298 deletions(-) create mode 100644 continue-claude-work.skill create mode 100644 continue-claude-work/.security-scan-passed create mode 100644 continue-claude-work/SKILL.md create mode 100644 continue-claude-work/references/file_structure.md create mode 100755 continue-claude-work/scripts/extract_resume_context.py create mode 100644 skill-creator/agents/analyzer.md create mode 100644 skill-creator/agents/comparator.md create mode 100644 skill-creator/agents/grader.md create mode 100644 skill-creator/assets/eval_review.html create mode 100644 skill-creator/eval-viewer/generate_review.py create mode 100644 skill-creator/eval-viewer/viewer.html create mode 100644 skill-creator/references/schemas.md create mode 100644 skill-creator/scripts/__init__.py create mode 100755 skill-creator/scripts/aggregate_benchmark.py create mode 100755 skill-creator/scripts/generate_report.py create mode 100755 skill-creator/scripts/improve_description.py create mode 100755 skill-creator/scripts/run_eval.py create mode 100755 skill-creator/scripts/run_loop.py create mode 100644 skill-creator/scripts/utils.py create mode 100755 tunnel-doctor/scripts/quick_diagnose.py diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 5c342d14..df931871 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -5,8 +5,8 @@ "email": "daymadev89@gmail.com" }, "metadata": { - "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, and macOS programmatic window screenshot capture workflows", - "version": "1.37.0", + "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, and macOS programmatic window screenshot capture workflows", + "version": "1.38.0", "homepage": "https://github.com/daymade/claude-code-skills" }, "plugins": [ @@ -15,7 +15,7 @@ "description": "Essential meta-skill for creating effective Claude Code skills with initialization scripts, validation, packaging, marketplace registration, and privacy best practices", "source": "./", "strict": false, - "version": "1.4.1", + "version": "1.5.0", "category": "developer-tools", "keywords": [ "skill-creation", @@ -292,7 +292,7 @@ "description": "Corrects speech-to-text (ASR/STT) transcription errors in meeting notes, lecture recordings, interviews, and voice memos through dictionary-based rules and AI corrections. Supports Chinese domain names, AI fallback to Claude Code, and iterative dictionary building. Use when users mention transcript correction, ASR errors, speech-to-text mistakes, homophone errors, or working with transcription files", "source": "./", "strict": false, - "version": "1.1.0", + "version": "1.2.0", "category": "productivity", "keywords": [ "transcription", @@ -417,7 +417,7 @@ "description": "Create PDF documents from markdown with proper Chinese font support using weasyprint. Use when converting markdown to PDF, generating formal documents (legal filings, trademark applications, reports), or when Chinese typography is required. Triggers include convert to PDF, generate PDF, markdown to PDF, or printable documents", "source": "./", "strict": false, - "version": "1.0.0", + "version": "1.1.0", "category": "document-conversion", "keywords": [ "pdf", @@ -608,7 +608,7 @@ "description": "Strategic guide for becoming an effective GitHub contributor. Covers opportunity discovery, project selection, high-quality PR creation, and reputation building. Use when looking to contribute to open-source projects, building GitHub presence, or learning contribution best practices", "source": "./", "strict": false, - "version": "1.0.2", + "version": "1.0.3", "category": "developer-tools", "keywords": [ "github", @@ -735,7 +735,7 @@ "description": "Diagnoses and fixes conflicts between Tailscale and proxy/VPN tools (Shadowrocket, Clash, Surge) on macOS. Covers five conflict layers: route hijacking, HTTP proxy env var interception, system proxy bypass, SSH ProxyCommand double tunneling, and VM/container proxy propagation. Includes an automated quick-diagnose script plus SOP for remote development via SSH tunnels with proxy-safe Makefile patterns. Use when Tailscale ping works but SSH/HTTP times out, when browser returns 503 but curl works, when local vanity domains fail behind proxy, when git push fails with failed to begin relaying via HTTP, when Docker pull times out behind TUN/VPN, when setting up Tailscale SSH to WSL instances, or when bootstrapping remote dev environments over Tailscale", "source": "./", "strict": false, - "version": "1.3.0", + "version": "1.4.0", "category": "developer-tools", "keywords": [ "tailscale", @@ -876,6 +876,28 @@ "skills": [ "./financial-data-collector" ] + }, + { + "name": "continue-claude-work", + "description": "Recover actionable context from local `.claude` session artifacts and continue interrupted work without running `claude --resume`. Extracts compact boundary summaries, subagent workflow state, session end reason, and workspace drift via bundled Python script. Use when a user provides a Claude session ID, asks to continue prior work from local history, or wants to inspect `.claude` files before resuming implementation", + "source": "./", + "strict": false, + "version": "1.1.0", + "category": "developer-tools", + "keywords": [ + "claude-code", + "session-resume", + "context-recovery", + "jsonl", + "compaction", + "subagent", + "history", + "workflow-continuation", + "local-artifacts" + ], + "skills": [ + "./continue-claude-work" + ] } ] } diff --git a/CHANGELOG.md b/CHANGELOG.md index 14c1c277..74235cca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - None +## [1.38.0] - 2026-03-07 + +### Added +- **New Skill**: continue-claude-work v1.1.0 - Recover local `.claude` session context and continue interrupted work without `claude --resume` + - Bundled Python script (`extract_resume_context.py`) for one-call context extraction + - Compact-boundary-aware extraction using `isCompactSummary` flag (highest-signal context from session compaction summaries) + - Subagent workflow recovery — parses `subagents/` directory to report completed vs interrupted agents with last outputs + - Session end reason detection — classifies clean exit, interrupted (ctrl-c), error cascade, or abandoned + - Size-adaptive reading strategy based on file size and compaction count + - Noise filtering — skips progress/queue-operation/api_error (37-53% of session lines) + - Self-session exclusion, stale index fallback, ghost session warnings + - MEMORY.md and session-memory integration, git workspace state fusion + +### Changed +- **skill-creator** v1.4.1 → v1.5.0: SKILL.md rewrite, added eval benchmarking system (run_eval, run_loop, aggregate_benchmark), agents (analyzer, comparator, grader), eval-viewer, and improve_description script +- **transcript-fixer** v1.1.0 → v1.2.0: `--domain` defaults to all domains, added `get_domain_stats()`, cross-domain listing, and zero-match hints +- **tunnel-doctor** v1.3.0 → v1.4.0: Added Step 2C-1 for local vanity domain proxy interception, bundled `quick_diagnose.py` automated diagnostic script +- **pdf-creator** v1.0.0 → v1.1.0: Replaced Python `markdown` library with pandoc for MD→HTML conversion, removed `_ensure_list_spacing` workaround +- **github-contributor** v1.0.2 → v1.0.3: Fixed gh CLI field name (`stargazersCount` → `stargazerCount`), added Prerequisites section +- Updated marketplace skills/plugins count from 41 to 42 +- Updated marketplace version from 1.37.0 to 1.38.0 +- Updated README.md and README.zh-CN.md badges, installation commands, skill listings, use cases, quick links, and requirements +- Updated CLAUDE.md counts, version reference, and Available Skills list (added #42) + ## [1.37.0] - 2026-03-02 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 93c52c7f..38adf594 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Repository Overview -This is a Claude Code skills marketplace containing 41 production-ready skills organized in a plugin marketplace structure. Each skill is a self-contained package that extends Claude's capabilities with specialized knowledge, workflows, and bundled resources. +This is a Claude Code skills marketplace containing 42 production-ready skills organized in a plugin marketplace structure. Each skill is a self-contained package that extends Claude's capabilities with specialized knowledge, workflows, and bundled resources. **Essential Skill**: `skill-creator` is the most important skill in this marketplace - it's a meta-skill that enables users to create their own skills. Always recommend it first for users interested in extending Claude Code. @@ -134,7 +134,7 @@ Skills for public distribution must NOT contain: ## Marketplace Configuration The marketplace is configured in `.claude-plugin/marketplace.json`: - - Contains 41 plugins, each mapping to one skill + - Contains 42 plugins, each mapping to one skill - Each plugin has: name, description, version, category, keywords, skills array - Marketplace metadata: name, owner, version, homepage @@ -144,7 +144,7 @@ The marketplace is configured in `.claude-plugin/marketplace.json`: 1. **Marketplace Version** (`.claude-plugin/marketplace.json` → `metadata.version`) - Tracks the marketplace catalog as a whole - - Current: v1.37.0 + - Current: v1.38.0 - Bump when: Adding/removing skills, major marketplace restructuring - Semantic versioning: MAJOR.MINOR.PATCH @@ -218,6 +218,7 @@ This applies when you change ANY file under a skill directory: 39. **financial-data-collector** - Collect real financial data for US public companies via yfinance with validation, NaN detection, and NO FALLBACK principle 40. **excel-automation** - Create formatted Excel files, parse complex xlsm models, and control Excel windows on macOS via AppleScript 41. **capture-screen** - Programmatically capture macOS application windows using Swift window ID discovery and screencapture workflows + 42. **continue-claude-work** - Recover local `.claude` session context via compact-boundary extraction, subagent workflow recovery, and session end reason detection, then continue interrupted work without `claude --resume` **Recommendation**: Always suggest `skill-creator` first for users interested in creating skills or extending Claude Code. diff --git a/README.md b/README.md index 015a5f52..0c0e2eb3 100644 --- a/README.md +++ b/README.md @@ -6,15 +6,15 @@ [![简体中文](https://img.shields.io/badge/语言-简体中文-red)](./README.zh-CN.md) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Skills](https://img.shields.io/badge/skills-41-blue.svg)](https://github.com/daymade/claude-code-skills) -[![Version](https://img.shields.io/badge/version-1.37.0-green.svg)](https://github.com/daymade/claude-code-skills) +[![Skills](https://img.shields.io/badge/skills-42-blue.svg)](https://github.com/daymade/claude-code-skills) +[![Version](https://img.shields.io/badge/version-1.38.0-green.svg)](https://github.com/daymade/claude-code-skills) [![Claude Code](https://img.shields.io/badge/Claude%20Code-2.0.13+-purple.svg)](https://claude.com/code) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/daymade/claude-code-skills/graphs/commit-activity) -Professional Claude Code skills marketplace featuring 41 production-ready skills for enhanced development workflows. +Professional Claude Code skills marketplace featuring 42 production-ready skills for enhanced development workflows. ## 📑 Table of Contents @@ -237,6 +237,9 @@ claude plugin install excel-automation@daymade-skills # Programmatic macOS screenshot capture workflows claude plugin install capture-screen@daymade-skills + +# Resume interrupted Claude work from local session artifacts +claude plugin install continue-claude-work@daymade-skills ``` Each skill can be installed independently - choose only what you need! @@ -1749,6 +1752,41 @@ claude plugin install capture-screen@daymade-skills --- +### 42. **continue-claude-work** - Resume Interrupted Claude Work + +Recover actionable context from local `~/.claude` session artifacts and continue implementation without reopening the old interactive session. Uses a bundled Python script for intelligent context extraction. + +**When to use:** +- A user provides a Claude session ID and wants the task continued +- You need to inspect local `.claude` JSONL files instead of running `claude --resume` +- A previous session was interrupted and the next concrete step must be reconstructed +- A multi-agent workflow was interrupted and you need to know which subagents completed + +**Key features:** +- Compact-boundary-aware extraction — reads Claude's own session compaction summaries as highest-signal context +- Subagent workflow recovery — reports completed vs. interrupted subagents with last outputs +- Session end reason detection — classifies clean exit, interrupted (ctrl-c), error cascade, or abandoned +- Size-adaptive strategy — different reading approaches for small (<500KB) vs. large (>5MB) sessions +- Noise filtering — skips progress/queue-operation/api_error messages (37-53% of session lines) +- Self-session exclusion, stale index fallback, MEMORY.md integration, git workspace state + +**Example usage:** +```bash +# Install the skill +claude plugin install continue-claude-work@daymade-skills + +# Then ask Claude to resume from local artifacts +"continue work from session 123e4567-e89b-12d3-a456-426614174000" +"don't resume, just read the .claude files and continue" +"check what I was working on in the last session and keep going" +``` + +📚 **Documentation**: See [continue-claude-work/SKILL.md](./continue-claude-work/SKILL.md). + +**Requirements**: Python 3.8+, `git` for workspace reconciliation. + +--- + ## 🎬 Interactive Demo Gallery Want to see all demos in one place with click-to-enlarge functionality? Check out our [interactive demo gallery](./demos/index.html) or browse the [demos directory](./demos/). @@ -1812,6 +1850,9 @@ Use **prompt-optimizer** to transform vague feature requests into precise EARS s ### For Session History & File Recovery Use **claude-code-history-files-finder** to recover deleted files from previous Claude Code sessions, search for specific implementations across conversation history, or track file evolution over time. Essential for recovering accidentally deleted code or finding that feature implementation you remember but can't locate. +### For Resuming Interrupted Claude Sessions +Use **continue-claude-work** to recover the last actionable request from local `~/.claude` artifacts and continue implementation without reopening the original session. Combine with **claude-code-history-files-finder** when you need broader cross-session search, statistics, or deleted-file recovery. + ### For Documentation Maintenance Use **docs-cleaner** to consolidate redundant documentation while preserving valuable content. Perfect for cleaning up documentation sprawl after rapid development phases or merging overlapping docs into authoritative sources. @@ -1899,6 +1940,7 @@ Each skill includes: - **product-analysis**: See `product-analysis/SKILL.md` for workflow and `product-analysis/references/synthesis_methodology.md` for cross-agent weighting and recommendation logic - **excel-automation**: See `excel-automation/SKILL.md` for create/parse/control workflows and `excel-automation/references/formatting-reference.md` for formatting standards - **capture-screen**: See `capture-screen/SKILL.md` for CGWindowID-based screenshot workflows on macOS +- **continue-claude-work**: See `continue-claude-work/SKILL.md` for local artifact recovery, drift checks, and resume workflow ## 🛠️ Requirements @@ -1924,6 +1966,7 @@ Each skill includes: - **Codex CLI** (optional, for product-analysis multi-model mode) - **uv + openpyxl** (for excel-automation): `uv run --with openpyxl ...` - **macOS** (for capture-screen and excel-automation AppleScript control workflows) +- **Python 3.8+** (for continue-claude-work): bundled script for session extraction (no external dependencies) ## ❓ FAQ diff --git a/README.zh-CN.md b/README.zh-CN.md index 5404eaaf..36e7b56e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -6,15 +6,15 @@ [![简体中文](https://img.shields.io/badge/语言-简体中文-red)](./README.zh-CN.md) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Skills](https://img.shields.io/badge/skills-41-blue.svg)](https://github.com/daymade/claude-code-skills) -[![Version](https://img.shields.io/badge/version-1.37.0-green.svg)](https://github.com/daymade/claude-code-skills) +[![Skills](https://img.shields.io/badge/skills-42-blue.svg)](https://github.com/daymade/claude-code-skills) +[![Version](https://img.shields.io/badge/version-1.38.0-green.svg)](https://github.com/daymade/claude-code-skills) [![Claude Code](https://img.shields.io/badge/Claude%20Code-2.0.13+-purple.svg)](https://claude.com/code) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/daymade/claude-code-skills/graphs/commit-activity) -专业的 Claude Code 技能市场，提供 41 个生产就绪的技能，用于增强开发工作流。 +专业的 Claude Code 技能市场，提供 42 个生产就绪的技能，用于增强开发工作流。 ## 📑 目录 @@ -240,6 +240,9 @@ claude plugin install excel-automation@daymade-skills # macOS 程序化窗口截图工作流 claude plugin install capture-screen@daymade-skills + +# 基于本地会话产物续做中断的 Claude 工作 +claude plugin install continue-claude-work@daymade-skills ``` 每个技能都可以独立安装 - 只选择你需要的！ @@ -1791,6 +1794,41 @@ claude plugin install capture-screen@daymade-skills --- +### 42. **continue-claude-work** - 续做中断的 Claude 工作 + +从本地 `~/.claude` 会话产物中恢复可执行上下文，并在不重新打开旧交互会话的前提下继续实现工作。内置 Python 脚本实现智能上下文提取。 + +**使用场景：** +- 用户提供 Claude 会话 ID，希望继续上次的任务 +- 需要直接检查本地 `.claude` JSONL 文件，而不是运行 `claude --resume` +- 上一次会话被中断，需要重建下一步具体动作 +- 多 agent 工作流被中断，需要了解哪些 subagent 已完成 + +**主要功能：** +- Compact-boundary 感知提取 — 读取 Claude 自身的会话压缩摘要作为最高信噪比上下文 +- Subagent 工作流恢复 — 报告已完成与被中断的 subagent 及其最后输出 +- 会话结束原因检测 — 区分正常退出、中断（ctrl-c）、错误级联、废弃会话 +- 大小自适应策略 — 对小型（<500KB）和大型（>5MB）会话采用不同读取方式 +- 噪声过滤 — 跳过 progress/queue-operation/api_error 消息（占会话行数的 37-53%） +- 自会话排除、过期索引回退、MEMORY.md 集成、git 工作区状态 + +**示例用法：** +```bash +# 安装技能 +claude plugin install continue-claude-work@daymade-skills + +# 然后让 Claude 基于本地产物续做 +"continue work from session 123e4567-e89b-12d3-a456-426614174000" +"不用真的 resume，去 .claude 里找上下文继续做" +"查看上次会话做了什么，然后继续" +``` + +📚 **文档**：参见 [continue-claude-work/SKILL.md](./continue-claude-work/SKILL.md)。 + +**要求**：Python 3.8+，用于工作区核对的 `git`。 + +--- + ## 🎬 交互式演示画廊 想要在一个地方查看所有演示并具有点击放大功能？访问我们的[交互式演示画廊](./demos/index.html)或浏览[演示目录](./demos/)。 @@ -1854,6 +1892,9 @@ claude plugin install capture-screen@daymade-skills ### 会话历史与文件恢复 使用 **claude-code-history-files-finder** 从之前的 Claude Code 会话中恢复已删除的文件、在对话历史中搜索特定实现，或跟踪文件随时间的演变。对于恢复意外删除的代码或查找你记得但找不到的功能实现至关重要。 +### 续做中断的 Claude 会话 +使用 **continue-claude-work** 从本地 `~/.claude` 产物中恢复最后一个可执行请求，并在不重新打开原始会话的情况下继续实现。若还需要跨会话搜索、统计分析或恢复已删除文件，可与 **claude-code-history-files-finder** 配合使用。 + ### 文档维护 使用 **docs-cleaner** 在保留有价值内容的同时整合冗余文档。非常适合在快速开发阶段后清理文档扩散或将重叠的文档合并为权威来源。 @@ -1941,6 +1982,7 @@ claude plugin install capture-screen@daymade-skills - **product-analysis**：参见 `product-analysis/SKILL.md` 了解工作流，参见 `product-analysis/references/synthesis_methodology.md` 了解跨代理加权与推荐逻辑 - **excel-automation**：参见 `excel-automation/SKILL.md` 了解创建/解析/控制工作流，参见 `excel-automation/references/formatting-reference.md` 了解格式规范 - **capture-screen**：参见 `capture-screen/SKILL.md` 了解基于 CGWindowID 的 macOS 截图流程 +- **continue-claude-work**：参见 `continue-claude-work/SKILL.md` 了解本地会话产物恢复、漂移检查与续做流程 ## 🛠️ 系统要求 @@ -1963,6 +2005,7 @@ claude plugin install capture-screen@daymade-skills - **Mole**（可选，用于 macos-cleaner 可视化清理）：从 https://github.com/tw93/Mole 下载 - **uv + openpyxl**（用于 excel-automation）：`uv run --with openpyxl ...` - **macOS**（用于 capture-screen 与 excel-automation 的 AppleScript 控制流程） +- **Python 3.8+**（用于 continue-claude-work）：内置脚本进行会话提取（无外部依赖） ## ❓ 常见问题 diff --git a/continue-claude-work.skill b/continue-claude-work.skill new file mode 100644 index 0000000000000000000000000000000000000000..66ac186317f18d4191b3d8e49d56043f61208978 GIT binary patch literal 14898 zcmajGQ;=psyS7=jZC96VTW{I6ZFkwWZChQoZQJUyHN9tIX77Xj{j)MwtjIjLBA>H$ z<#VTkG$tr&qO-6wiYtoqENmxtd8aut!K8~=JOuNdF3-Wq2VQGe;~CZcjh)t@@hC0Cl5zK$ zU6T^%kTzX=UGG}{jDc;OIu{|oOIKTur1s0@kxudD+PM}jy5T+mI4TGR^|g0re`&sp zcI+u-RLu^eu0233tRm-{hU0Mp(bjsiMk|6i<#Z>wfT^nFy`oVl1Xp{N+Q3$~vPDsZ z2sXu1RsT+3&vj>8i?goTi&Q+3CCwfa5SEqi#nHatdD+Fu)CoOvk+aKy=2Nzr`?ALF zQhH{-y0wM_Su;K2S+u6?w8O4zi?jHg8Q3!A1jg-My;DqDq}{8K+%y`uEFt>5lBt{e z)ysH!X4iZ>GHvIMWZ-Nb8DP_Ai6G15ff#22jsSI#@y?a*v2P+~R}5+oV|N>(XjkMV z@VFevZ&M~hY5AzWFB?ZHowcBo0iyVQ)B97TwOXYgHb)a&I=}NB`byh$b##7p?uMhMP-_pzA4rFHa-nT770QU!u6M?EAL#dT0+d2Q(ZAFe*3I@rz~g8pxCQi` zgt|6ht}jUg9)VYFNT=YHiBM(q$G>-`ncOfEs6&CHX0rvROmgYaHmKagMlJoP84l)w=!U>8@`BTSfnDYvhL=&-MB6)^o13JhbO z`lS-X1qsp_@G2fdNxcktZ4ao7PfBWOGYe8FcpOV_T~eEA!qe9sp5!Q9E&0oq%Gh3; z=*}fMutnzwU#oi;%Mqp;0k5JbJfI&y;RaVkKn6r{K+-@}!iUaYM$>;;-na=fHx{(r zy0_maDzlta?b_XDWRu9HyONVxWGWn=Tii zWD6Q6-IvOt$JjubOsNC5B%3kgQmck*YaOg98P7#&8o=?h6>Bm+S%I*S098&`4u`<) z7u`l?s&Q#c!fXUYRXJ}8s}cI=mjkOcYmO6g??(T6yRDt-iuBc~Ug=Sg8ZuT4+9+w{ z##w3y@|xA(BgOd-ZiZOQyNM8l5PEnT&~9-`q^?BS-ouskPT=Tt5Q=LwcocUk`3kWE zfm|G2E~F&MteoNf!QrsDyl-IPXb5gr>!SqiHh1FalBgN0Bfu{4%Yy;AQ^70*(gV&{ zz2HNvN5%GYexgD>0-i0EzPFXBZaPiqTB{vJf#cF2l+U2GE`?owV7%79=|>4YuBnMy z_V*E9X)M3|+-mij7(^MVkTE5PDCB}YQeZHpC@+Luw$c`V-Gir5aH(^J35?mt6<#Ds z&I|wCmU3+JS-d@52|i_P;QJNvo98{8=dJ&YVh#b3aw8qh7Cn$6Bp|SIgE>gGPFDOq zMUR#@K0a?0Z4m6Fr%)aH&1C;FVW(hKPf4#Mmv2JoixVX?hX`@uBEKdAd8KMCDQSnl z`lC=}5o_fwG9d~i`+BYTO&M$K)bZl|>=pY}5sQ5tfFx*zd&x@&!OH6@pd?VkM7}MN z-JQM))A<3+uK_0A{eH(xjrDu>-A4^HS^6NO#OqRn*UQaDH%MrYyT7y9YcqD-trnAo zl8x})<-1BHOq7x#)eH{u;$z5`60V{b?y}>TQ2gx;WMHImDUF(o}`2tI?2WP zQm`gkZ@NW5k^PtfFkds)$SsLn5tY#lmc`}{}qRX8X$MY<-IY-DprVl{h74paq1nlJ~D*G>4qAfW_Y z-EhI0+i~U(Gf^x9_;DtEiTs{rwlUh6l58Yw-9AOS^YBU5qq`if#F9fwG)i#BdfS+_ zbm>nnQUi7sGHvOr`*eIMQM3cDCuuY74KSI<4}=FUdA0n<{*_~XwtB|XZdsvjhXqCLL7o$jyBm#3YA6`;hUbKyPK*CTK@jJ2>9I`p*$Kn|d=oRV#Qf&??@P)aCHm z42ExBjv;g2yz$NGA+C;|dD6Cuh<fetCP37};^?%I+u1S}-@d|q9l%ok61ZA?g$0Ro}T4U81 z4l=h=8RSRlojbVopn$+nSTBvZUb32@{v^tPk|K`-Lhz4d1MuQ^5vJcuRO;PEpRm23pCUdoCb>3WT}?8a-rY*o=VTFGpodc{eQ(O z+DuSdASM&KA?o9_EEF?ix^?9YuRIkG8|+;V zAha*XdPOlg0dL8`t2DftWw88#)y(u>oLda=8~-?Nr=%i`rNj)$P-Oyn-Ol$pkFaOK zOE6&&?H{C&H*tGsK4#qN3`zzgchj{U0%N|;!d!m$r#p|{NAl^kB4gm%-=x`2zM+t;1BB;Q%jFgvZkJU@#TbvC?26P8c077b-hse0G z7|n%=EhWe5CEvTw$_}=&@RpXhC1_4pljQzP>M=7~&u>_9T56EAIJ%_&t9?Z0&k9z) zMQSm)Qns~h)5+@=O?&U3MKDV^U$;)NN55*`R3-!h*A$APPgd`Dq27a^?zy<2-904{@v>l^ zt+vlBqFlLi%W-*oP+xzB%m__?Ko$N!88CA;*RkUp6;Aml0WJSt>i;AI^v5UvM?gk$z7tVvr4<<;vMaa_r53K700iwf zJ<)`?=;J3+vs>pAf#%%QCVF=39n))WRkZ^cr#%zWwD%^XRpfVgZsvVc)@@nnKM7wV zWzu5fIVmq|s9I*tXb*{gmfEA`UYm5vSTresPvJ9aO#$#4^|ZJt;{`QeN*;)wBk)f# zMC?G~x(S>WmE!uRzetS0`Q%&1)kKEu5>VDF0-lcd@1y#GtOSgT;=l#X@n|?zw%_L# zpZto%%)p8%C#KzDPb|w|Yj@7F7l7KjUaVa4ZLVUdiZd-U^>8ULn;Htm`6mbBIt=O9 zSK)HKqzzu+N_|yt7%Ggaa=`e0HKO_u^%F}j%$kgs5we=P{^}xE+(>zF70M#*b-L8;P+D=lFe6J(CMjFU>W`Wv1@ zFmha}ctZg`VU9|}zuT$WpxJAu;t<6MYTAPs>lNVI}*zbed&Q*0k<`zn-j6dEFXW;7ww$B9ZKjHhc*)XS z`Wb%FeuLkp8S<5hcyT@?o|DRy`fModtKj@#F?i)%J&K*}s8GG&P_!{&@kSLP) zR#vH{)hbb=kv-*n0*yK*^8hJ8Y7^7m9H3p7qH+2YMXOFL8Y52ntlD6dDUyT=uu94SvQ+U41851Yhb4nk|k3>yuHxL@#4>7 zQ)Dsz`TP)A%i=lF6g0oqRpA)8 z9b8Atx{Gb!iL@azfZts6Iz`0KVipgnSL^w_c8@Q*dK z2cCzY=iC7`eOht<64N~K;>HVX0f20f>i}gC1HuR*ZLrjsBYX~+cT0?`3@Ho*h~dxt zO9cDHC&)8Z-~3ag#j3PK(d&if#^E4=uc7p0#=(|cXdCXD`R7a}W60k8FS04w*UEH` zlgIeFh3VvFt4U}Ey*!m8(8u~`HQ!1TlX6ot*!&LJQ}jU|+84lqMOGN7p@-GxW{JGvmFLtqltjpWLxHnJ@A!-tb!f zOdC%N3q8exnWzybos21rVDD3@maGP1wugZADV9eS={N_V);j~Z!$c`jT82nB%o>rI zk|g~Ts|G9_r8buFbWAYUQ5hFtrgt7ma3O8nUpSMJM9`wH ze-dLSmYSPD&s35n2{tDQNAo49*N3=LCvBB5nym~pf&c}R3^MT(79)Rf1Y-1CT^9XK zB+|@59!bWu_4lvMqUH*G)3$EXO9e<98c6~ky|FZcuOgdr?FFkNrI2~eJpy6fr8p!| znWA(poSVE{93-b}Bl94VO-hF5jms6ka~T=K1K8>O(P}W36e+v`JO4BYreHu+0=DY_ z>rMd|^=sto?e7%jfqm1%am4SPS4fCfkwNXmppT{9_j)DNqKPRK{sdSzej%n2?+XX% zfbqZ}Ii+Ys6g?{iM^A=|hTWe~xoC3aJ(UU%Nkpkw!xh}=8KR5hCj!wkme;F#54 zqABzgwA#Fw&IsB()w;9zK1mk0q-<}D6W&u^z2y@JO|&M^=JCBAhEDme)pT*NUQM!y zILcUv3HHFH5dAIN4E5ecw5SMd+_7#gCYcA?1IS2G+vGaj{6O7HahHgO_ZH`L(TFE1TScDQ$~t!%>izo{*- zl0{{@2bFo4=ig6(bC^K>=2oN;!JC}RuHk$F`A%9e%O;_>e4xnmb50$g-g z>{{|$=@|}ryjP$4iwNYkJ2!z#IiN@S_t)s{Z_?Y6Hw3h(>}OF`h@vYeH1jU1W`I|L zdYu`|4aI$=D@g9AR9fjoWh^lwEB5ENbtl*NcqaU$4@)gCpK(=Ga!rufx8T=1#DngI zJ7VT)+egm+(mTp|lzIOsc;OT#8~V=)IU<8;^x z*3tw?iNk&kzrfXiL44$SyLmN)H8Q=6E)FBjPVj)VsEu2<*9G@?Wan&&8g(d(-1bb+ zT;F^^wizGS;C6R}8rT)=3}7kyvFlR6!0VD3gbpb-Y=|91X>6^@1@NXR+uMEmjjEl3 zZS)Kyx$vDC^^~UmU%%IaSvV*|46ed-?H;Pi_U@@Jsfcnf{x- zve@==c@P1&J(uU`lT&$_qd{8dzd_&dP>Ied{Y-$aO|kI(JQXUQ&MRw9o<&PRsOwkW*D3yMvj zRSxKi%J%HbW^F&Panjrlr9#r?R=Pum$4=h9T1Ouuy3;T`$&saUW2TkqC={Pzj$*5}FpEoreoaxn7G@4D0I)e$@hlj@|qQApBdP%I(_WmW2O%$ ztZ3eLPaA~&EC{lwIMNo?j>|*iVQ|JT>WU6+^ZgAn|MTaNHP^-e9pQYj)a;MZs!@wd z$bN<#$Y&jDvO|^g5}BT@*=df8iGyV=bi80}wOmql#-%x1P-Zh)H?JN^&& ze=EW8zvhQIWr&O^ArMf_Zy+G{|D*)Y#!i+FF3t?59xhIX#xD9!rp~UmruzTYDEYTU z@8EgOXYIT>mazLoW47OcVwjYi%(rXXKwI`XXP5lynyOoYHyucfg4RF+3;{%L-QG13 zH^wl<(Ec}99g&fU$~yQmSeI?G$sRF~ zh?Da}3H|O5%Iy#9?7^fbNo27O9SYM-6ZM-?gk=K5P?Tp^CfR`)!VgtaV{ofc=1fwf zpTlkK7)6z%Qlv(*Y6FgX6-h?;s-q*{xIK%bX7M=dbnVb_PI@ii)!JbW6%$4>Q#O&o zC_&|ptk!5=v2T)~`|L;KL=$RS|0s&Q&@-d1QA%eETF?<0^hn9Y?0lJ)s!A_I&-6~` zFs6Iw_|~JkqjsX0GZNTKb668xB-QkH$XAsnNuZ`)MBY6ubp7t|1&W?f(Pr15+?faM zbPXNy!X}OBs%DB^^l{E!smRXw>0jf&Zh0Y!hsi2X3aPrjiBs|cf@BHxd*_!;tMv)F4vap6CEf_lRh6ETpb*I zUT2+s9xL;$h@oH}Ws2odV3X~T?o*QLg<`)ewGt*Cq`biwm z`JI-0S25m9E|dEMPqW@k@I08O&?kZ#5g~BNGlY2EA@mU7b$L8p9Zoi#9lv=y0^;7A zekZsHJCX%{r&kL(-2VOA+|BELbtuSe;GY9b=YxF`hOa7Zin_LerBsGU^TfqI82*MQJP2Fg;GVt_=CRHVB_ z#wp!_`;(06-pw49MsC5;lRj#InEzm;369(Mug#KdS0D?#Fd3GI^L=pW93|8xLpnFp zC0I|SrOB$P31)C)$~yUQDVbTpg9RT7${YwVhvCQi;}We_LPi^_wB&Bw*H{T#M9Q%b zok{-+TdJj2B7Ax{y#8^v-GoRuv(qb-9ZF{151HPk=<)i%JRdC6q{mWe}^b`L-kElgEICv ziWd|YdZ>%`;SJ`OG>p=*l@J##-B?6=aem?$Yr=3;aa>`XK_nP7?W@5TiRMgys#_pw zu{&aQt0)pirc|*uNEZg0V*c>#0X)|4po^C|)jh5J6>hp-I6Bt+hCcxN#Ho3YAZnQ; zaS-1l(>%TYB8VDcTmG3cRuWfWvteh0mm>txA{pSIA1Ed;NZg2^ChD7u#ocdiOnS=2_#`Dl+S2mqy=c03RV#OZ4OphpnS_^ zd1(^vpgt^%O#C%WXP|0Bgi|Rs5fhWlLUCY=p&;BG(%^Z9{v9<+*!^3NAtvSS*25AU zsa!@!-kPXR9?vj#dO3{n(l>S#&+!Mr--wf6ro2566)Doh0K&B&YVfG~@T$9LoP$MZ z$><;v0dQ?4T!{Mg^m%Z&kyDs23ZErzhLB%6h>DiUtQi;QV4S4N>~dbp)`ydW3QLM&F$eNP1Vc#D>I*aVExNW(zURT=5iOPwQJi9_6e2B^GT}D4FRuOX%4xD42UumJ8oeZF=NMq1o6{e7tS!2C~Q08sSsP{fDKfO*!77( z+}zsxeZhXt1}CzTWQsrT9(C%!_yIo0v~bT^fO`?P#@U$%<4`P*uJJX!XzY4>Jw8rS zvMps2SQL}Vk;eqB_DM&Q>vamE2E|1LN-u(M5xYxTRrQ7+EOO0E%;Q>mpZF$#WeUKC z1=SU!x4S?xkJAZ)UZF4OSH0G0ji%?AiN_FHBO8|st6F#x&mrCS_n>a{xKj-#pA6ht z;j!W{^o;KW!mpoCF0EOcN8EAGxzvhg_W3B%KpRTzinR&IaA^wX$T zq`^KpaL3h&nkOg&@yRn8qjB#Eh9Crawe2A>EvvqcCJ zL9BR$#}ocRO6*5-F4(Q@q>%<|HSLBzCS3;5|EZ}53PWqgSd>93NVk}Is`km_9^_L3 zs`rtXzPC^iEFRYy(&rx4O^D3pw3JYh>!M%x!oT1&c43PcaP#OaS&?vymd1roWH>aT zV!`uiMqxlzNI!xjwG)6>ENhEu>4z&<)}atnwz!1XEg&Csz$z!}uOpuGI#b9VFk0?w z)&0{IE#*gEyY@#Mnq^(n5f)?4fwEKqZwM#IDC2uh3WvGk0S`k=fij6~qLQ1Zt!i8# z*0493eBDSaoEMAMIt5LBc>L~;{c@%9NS4`l(S;lcKM%}hxCw^SslD>(1;f}~UdNwT zmURO@M(b4ZAqt#-)zc?DlTi~<7@P!Ee!r#9#$X%x4{Dq!Ol+POi0tc*;D#JRKWn_D z8`iU&I4=)yxLCM`s3z9gLBNCLCxp33yJ9Gj1j2Op@G-DL|KTDWrFHO(B17)bBz1Sc zE|d#;?OzW@W4>yNgS$IN7dYV%!MwgS0bW7_W+#&IXTs5+6NeHe+-?w4)i7^D#9_yu zU2KW|pmO-ApvC}wo|D;iUfj;-R!MVSB{wT7^WfjaD;uzCA zNznn1;>i)N>e!;c-#&k3i2k`l-5){h(X&Tl1=S+M7U?%XWsG2zILZ_ zuyi~?9CSmiTG)MCPTS#lrI%R39~pS#dCR4M3Wf=uT=Uua-{W%1ZKT}R2>@xnFr}Yg zBlfDA^C6*k@-6gGI=&U=53QCjqgTNHDPuV@-vGS>etgJOk&-k`+Ao*sye$>HT}j!N zy^ypGd~PK7eNou-f-Nn%#92aG=Bd}ZT*k05R_~;zM8Q9bw!@jj^Pz|M^}!p(QF(jJ z3&|ZIQ$A8NAPB7noAvHWg)MlF`99N3ina@6=cS|_T&hZNEAmApGQ3xoL??&w)Ysg$ z37L1Btz;ay-pDT5s-sFkHAh${Mo9fz!eT?)oHhYlL#6Xt7CHv1yNI@ZnFkZPM^?urd5E3t;LNQ1c~{jjn=QrgLKq znBW3sJU8Euqb&J0!&3{a7hkWHkDK}1McbbN(B^2lG25juF@#%T? zi#F6Kf41m3Z>q@!z4vM|4oBJeISx( zMUlB;c#=__upcKviv?>H&E?x)Z=}YeAZy1od60^E6wyDdI2Yh*v!FTw@e(=4+IFT< z{lx+~vLqo-72#wD2|jCF7Q-c%X(jaT1;vi)<>jIsx+)+TRdZTKJ?mn`jD{MF^~BiT zyU~^fh7ZJ-`(C$m8$NwnR*k~AUliV#Ile?~o!mQV#Fup7h?$cD7xrXdkJfUsBIw3w z=%!Qd2$K#dLKzH=!{WBN%?&vkvDKw)+Nz~4*8TY#J5`Y8pm5@ykTKJI^zKxXU!|G=dcB8cMn6_Jq|Asu`NZ(r5^1>=|{j!cnHBYR|6e zI|Z-Ph}>dwy=KP3Wf0R9tOXO|dirFYi4U`V1}x{5k)gfom5cJ!4_i~V0TS#oZpP1sO1y~Sa%Qxu@kaOPoCX?P zV{2+{zH?(?KTU=Oz!ylJ^OE-ofL7pfKo_?;HF0ZEoej^zA4#8_r&sTk-8Lft_In6j-;Z@N*!)sTCe{*q=*rZgqKLiK zwi}JS4_+O_G7i@=;JrAyb^{XeLA_3;GL6sM|$; zDii)SSEafHIqbUfzTV3Fg_q{nMn$44cMB#%P=b=sEuvVDmJwTd@#34bs%3~oS240i@p(z+aJ2Kwj00ZwWt%7T zGidvF%9893^;U%5?}urAMsdL#zjMpbo$0<*rLZ7oQisEm2YzyCW#^m!<%~R;I4L_h&^8E zBoT`VcKw1GcR6ADY)p;jd}-wJsovO-U@(FN{x>ptUd6qTGV(zTY9%#!A#n(Hpf+&{ zxb7FTNx#gk34pw%trBlwN?qmg#5>{$yK+Lk*kr@vt=vdiDJDkf%Jb~9P4)1fID&I0 zwG~_%%;JxYIf`r7vqh%8BG#6Xeq=r@jHX1L~A>`gMnE-*!J6|7dM8VtM;c!ftO8)lRSMh`!v7jUGIwh z2Gd!0cR$&Xm1*5?;<*hlwpNel{uGfl`_l|Rt>*sXs#YJ$ZUsUDk(jSgHMMa04nCrL z9~-CG>LBQon3zpO#7~E&F~f=N>@Sn&s#eIi)2iVo!WP>xffwei*LL?1I^R!KI@alT zm}8N#(`U*bvovpIo{l3mCWG_gzCW0DPXT0?{&bfuFXFBF3Ggviw>j%VA}>Ptg&UuZ z(z&+vjvO;r0W_#*Jqqds$#0c?o%Fs>Yi` z$__HaT>T6*^jsm+fW)SgOk=I*zX84tMp;>}*azcuteR;jk7{-5GxwXh!X)L(mC=Xf zxsZ{?+TxISohFV;d54?hRC%Upb{zgIWpQ*wqdCc>LH(J6G=2ver4{7fIK01Kg%u^A zeO?`i`0KhI*z;28R+zv@M<65C*3(S2{KF$}`lGy))OQb`E%}P;J5J^_9L?o%CRaj7 zh8ZNck`6I0CYWwSwjmJ-7{@=7eZD{6lv|$%*5@|;DXu0>U~eI=Fq>n-kRh}VD(f1{Q;b2F9ba)&kKn|wH-X97}BGy@@GdU(Gn>M`K^)o;Na&fjajJzo84dYB?2rJxH%%vcQe|B z59-lSdb$Ds&ZgPwr^c4$>L=ahcv5JMy7O~{22B2HSj+p57s9XD#VI7h2CgnIsAHd1 z=Grq!PiOSeBMLTNq7@Gn(GhNn|6cB*@FqNGIdL^}X#|VY@AV*Nj2&VY@7%tBF;RX> zxbq@d?x&8gP>sphkN0hk;pvzbNMae;_mpNA$ziYiRz)CQ&CY4!U?_SqpY2nO)#LN_ zeZCk-UT9p14S*^hhtvwo6TxkO` zXw(wsQ+SDLU#2gut04{Woqfcdx6F+-b|ouqlj4coD|&D8&Yw)AS3m!aavrt89cxyf z1)kkO#tr#*iHz-2=B0tDToLe$cIEd=Xw&XS`qt%PJIkN&(u(85 z!(KeUV1g#;??cS8(+7yJmAj3H*Mm03R9@Ro_nir$Y1EchC5_y(|Q~AO{WjyqV=N=7l3A^*pjzKZ${TJ;Dn5KrxBYi zS@#(F2hoEc5T1Y*xKW?*JOneZG5|eCV?3}ceWi&**!Cxikp7+Z+PPh~?B8An`R?jJL z$FzYgX=e*8JpiV{OReo~E5!v_dc3KC-5pf5sXUg|mK(Y_$UJYbd3Kd88+}T#4oIS!-9roNL+caZNU#EScK)Qa8 zD+9%Z104~%JM5zAR~BDXC#!mzKAH2j_SMq+kTTWA1oe??x-fO4hv6nxE`%=o&vL6- zr`H2;Cjj9K6;?&LgYTg$V-q#9sx|?FPhOk(ufLBVqviaTR&(4}?!B-131M3MzI&y6 zcUgnf9Q>A_C`@jy>7J(MIv~H?a5SKwgTSjm5<|&z=r!LgCJl(4x_iPs!+$*abQ2>~ zoLMsbS|#!a=x$;|nUwFZYq@%wdZqIRX;}`Q;{gL49^d)T{V0Aoa$eU~-cNBSVJP(V zVc4ZRRX#~PaF@6OR+|z*P&qkY>ZdD5ebMTv8 zZyMuo+)@`pz~7xTXub1>IiSrZ)sQ3?96;iX%Z^*7M=pFWTDpvlKG1tM>r!1sr)L@( z%-P{7*VP%C%_>6Z#L`gq4XNsm94gy0y{EA1FFJQT5zq*FAM|zi10}fKX_2a1@^M$D zi{=c@@kZhm&^9rdBwI4UGcdM>7X(n}{nT`X))G`zDD>8HOuB?SCDY1b%1%rMzx;qo zuOQ!-+VKf>Pbk$uH;F+XP%)d8yWu+EvMjqY)yq61(C~KUxUX)XX)_q)gWNw?*|;kQ z`;x2Eb(e4Tpeihpgi9caJ68>vn4r(n(k-NTd&8U<(9XOpg!g8>N$XErHYI-5_#*zE zQ$XT&4vkHYR{m6tB8i|@h{}&mxp`6f%XKS^!T!h$ftM4#n2c#&J3Yw>*s#(rJJUa%-+=VAZV4#yS@L;Kocc+yFGOVU-bHG;P{Qq+jb zJyXncTb8~;_RxEB%Ms93`Ptl6ghP0HI0$j1?&&N0jqc%P+uJzz$&~0oWAi)IyZ>ZJ z*vRf^_bv)zq(HF;bo>j%wooeuy+8afrT3@RNEEiHTAHV-zSR8VLr61?yW_dKjS6%S z^E*!JmgfBAWoyf>=8fh!T3|{`xxp`PilJ;=OCh+kZ&_Y_;Bq8$m%(SGW45!87qPo` z=p0a_*YMdq>UdxZxvTGG-rg%P#APFb6`B}H;XsadRS%SAU!V2mEq*gfi@T0`-OB~8 za>z!nof$h=$X~&qK2QKUh2-i|DCEZdQLRN^4hrPg< z0b%*%J@qYJM!87PWifxJf>Na7_>OBT=A>q|ONrx#$HWYNWqpkjlJvJZ;g*fAwm>;@}@&OKO7sBKjXu;W``LlGP= z3Jc1FAQIzlx>8gQUZk5Wm`ydCuJ39t&RPaV5eX$(+HmL}?Tb?QR%Sn%N5;)~_5aAU zA8ivmdt)aTz(ghau!&sXH3s(TM{lL_xmYM%>g0{*I+tn!*Bv82v!ZINg{?@#W?TV- zSzW(<)zCQ?(AUK1D$eDVE}Y3vC!7l00Za_(d6Tkq)u!_Gn#`W0Ck0ZazS;l{E{2+9 z9=|r8V$PXe@ZX$$hU56KrBik3ci0A(dJ=iy-D@u7jC%&!&}Nc+6dMeRZ{0ojNd!#v zp!|$=L+$0d1tnbB%(}Y?n&Ee{^`Dn_9{JR$39g8;7-v*_(L^eMQ79^>`DtAnXrn@` z6BvV5n8*{)@T)GoT#@;Hg+AC69zs1_R++I^@((&sBNf427k3`@S5my|Qm)u&Y)X&N z;5@ySsr=m>{Sn++RQ|dQROaBu8$eBuJsr!@dkH9(et!cAlQFV|5Kfy&w3-%7@&+gC zm>A68G-7kBY^2H0rltr&X4ta9sr>bnJ-n)KBro}8OCclqFP5fD zv3^8`(J>G*Gge@DkP54JEZ#;Vft*toLo%f=k-#lOle7-`1FIkn41xywpYJXH>)!w0 zPjSG1eg2QP82=~ge@i|8KOi8W;sAKK|A6{mGtmE&`@aQ_|C@X9pSb_0`0@Y5|8L=i k{}+E9@P+jsw*CJbWl)d?hxjiJ*uP;03 + +# Search by topic +python3 scripts/extract_resume_context.py --query "auth feature" + +# List recent sessions +python3 scripts/extract_resume_context.py --list +``` + +The script outputs a structured Markdown **briefing** containing: +- **Session metadata** from `sessions-index.json` +- **Compact summary** — Claude's own distilled summary from the last compaction boundary (highest-signal context) +- **Last user requests** — the most recent explicit asks +- **Last assistant responses** — what was claimed done +- **Errors encountered** — tool failures and error outputs +- **Unresolved tool calls** — indicates interrupted session +- **Subagent workflow state** — which subagents completed, which were interrupted, their last outputs +- **Session end reason** — clean exit, interrupted (ctrl-c), error cascade, or abandoned +- **Files touched** — files created/edited/read during the session +- **MEMORY.md** — persistent cross-session notes +- **Git state** — current status, branch, recent log + +The script automatically skips the currently active session (modified < 60s ago) to avoid self-extraction. + +### Step 2: Branch by Session End Reason + +The briefing includes a **Session end reason**. Use it to choose the right continuation strategy: + +| End Reason | Strategy | +|-----------|----------| +| **Clean exit** | Session completed normally. Read the last user request that was addressed. Continue from pending work if any. | +| **Interrupted** | Tool calls were dispatched but never got results (likely ctrl-c or timeout). Retry the interrupted tool calls or assess whether they are still needed. | +| **Error cascade** | Multiple API errors caused the session to fail. Do not retry blindly — diagnose the root cause first. | +| **Abandoned** | User sent a message but got no response. Treat the last user message as the current request. | + +If the briefing has a **Subagent Workflow** section with interrupted subagents, check what each was doing and whether to retry or skip. + +### Step 3: Reconcile and Continue + +Before making changes: +1. Confirm the current directory matches the session's project. +2. If the git branch has changed from the session's branch, note this and decide whether to switch. +3. Inspect files related to pending work — verify old claims still hold. +4. Do not assume old claims are valid without checking. + +Then: +- Implement the next concrete step aligned with the latest user request. +- Run deterministic verification (tests, type-checks, build). +- If blocked, state the exact blocker and propose one next action. + +### Step 4: Report + +Respond concisely: +- **Context recovered**: which session, key findings from the briefing +- **Work executed**: files changed, commands run, test results +- **Remaining**: pending tasks, if any + +## How the Script Works + +### Compact-Boundary-Aware Extraction + +The script finds the **last** compact boundary in the session JSONL and extracts its summary. This is the single highest-signal piece of context in any long session -- Claude's own distilled understanding of the entire conversation up to that point. For details on compaction format and JSONL schemas, see `references/file_structure.md`. + +### Size-Adaptive Strategy + +| Session size | Strategy | +|-------------|----------| +| Has compactions | Read last compact summary + all post-compact messages | +| < 500 KB, no compactions | Read last 60% of messages | +| 500 KB - 5 MB | Read last 30% of messages | +| > 5 MB | Read last 15% of messages | + +### Subagent Context Extraction + +When a session has subagent directories (`/subagents/`), the script parses each subagent's JSONL to extract agent type, completion status, and last text output. This enables recovery of multi-agent workflows -- e.g., if a 32-subagent evaluation pipeline was interrupted, the briefing shows which agents completed and which need retry. + +### Session End Reason Detection + +The script classifies how the session ended: +- **completed** -- assistant had the last word (clean exit) +- **interrupted** -- unresolved tool calls (ctrl-c or timeout) +- **error_cascade** -- 3+ API errors +- **abandoned** -- user sent a message with no response + +### Noise Filtering + +These message types are skipped (37-53% of lines in real sessions): +- `progress`, `queue-operation`, `file-history-snapshot` -- operational noise +- `api_error`, `turn_duration`, `stop_hook_summary` -- system subtypes +- ``, `` -- filtered from user text extraction + +## Guardrails + +- Do not run `claude --resume` or `claude --continue` — this skill provides context recovery within the current session. +- Do not treat compact summaries as complete truth — they are lossy. Always verify claims against current workspace. +- Do not overwrite unrelated working-tree changes. +- Do not load the full session file into context — always use the script. + +## Limitations + +- Cannot recover sessions whose `.jsonl` files have been deleted from `~/.claude/projects/`. +- Cannot access sessions from other machines (files are local only). +- Edit tool operations show deltas, not full file content — use `claude-code-history-files-finder` for full file recovery. +- Compact summaries are lossy — early conversation details may be missing. +- `sessions-index.json` can be stale (entries pointing to deleted files). The script falls back to filesystem-based discovery. + +## Example Trigger Phrases + +- "continue work from session `abc123-...`" +- "don't resume, just read the .claude files and continue" +- "check what I was working on in the last session and keep going" +- "search my sessions for the PR review work" diff --git a/continue-claude-work/references/file_structure.md b/continue-claude-work/references/file_structure.md new file mode 100644 index 00000000..c4772407 --- /dev/null +++ b/continue-claude-work/references/file_structure.md @@ -0,0 +1,249 @@ +# Claude Code Local File Structure + +Ground-truth reference for `~/.claude/` directory layout and JSONL session format. + +## Directory Layout + +``` +~/.claude/ + projects/ # Per-project session storage (primary) + / + sessions-index.json # Master index of all sessions + .jsonl # Session transcript + / # Session subdirectory (optional) + subagents/ + agent-.meta.json # Agent metadata + agent-.jsonl # Agent transcript + tool-results/ + toolu_.txt # Large tool outputs + memory/ # Persistent memory files (MEMORY.md, etc.) + history.jsonl # Global prompt history (no session IDs) + tasks/ # Task tracking (per-session lock/highwatermark) + plans/ # Plan documents (random-name.md) + debug/ # Per-session debug logs (.txt) + transcripts/ # Global tool operation logs (ses_.jsonl) + file-history/ # File modification backups + todos/ # Todo items +``` + +## Path Normalization + +Project paths are encoded by replacing `/` with `-`: + +| Original | Normalized | +|----------|-----------| +| `/path/to/project` | `-path-to-project` | +| `/another/workspace/app` | `-another-workspace-app` | + +## sessions-index.json Schema + +```json +{ + "version": 1, + "entries": [ + { + "sessionId": "20089b2a-e3dd-48b8-809c-0647128bf3b8", + "fullPath": "~/.claude/projects/-path-to-project/20089b2a-....jsonl", + "fileMtime": 1741327503477, + "firstPrompt": "fix the login bug", + "summary": "Fixed authentication redirect...", + "messageCount": 42, + "created": "2026-03-07T03:25:03.477Z", + "modified": "2026-03-07T12:21:43.806Z", + "gitBranch": "main", + "projectPath": "/path/to/project", + "isSidechain": false + } + ], + "originalPath": "/path/to/project" +} +``` + +Key fields for session identification: +- `sessionId` — UUID v4 format +- `firstPrompt` — first user message (best for topic matching) +- `summary` — auto-generated summary of the session +- `modified` — last activity timestamp (ISO 8601) +- `gitBranch` — git branch at session time +- `isSidechain` — `false` for main conversations + +## Compaction in Session Files + +Claude Code uses [server-side compaction](https://platform.claude.com/docs/en/build-with-claude/compaction). When context fills up, two consecutive lines appear: + +### Line 1: compact_boundary marker + +```json +{ + "type": "system", + "subtype": "compact_boundary", + "parentUuid": null, + "logicalParentUuid": "prev-uuid", + "compactMetadata": { + "trigger": "input_tokens", + "preTokens": 180000 + } +} +``` + +### Line 2: Compact summary (special user message) + +```json +{ + "type": "user", + "isCompactSummary": true, + "isVisibleInTranscriptOnly": true, + "message": { + "role": "user", + "content": "This session is being continued from a previous conversation that ran out of context. The summary below covers the earlier portion of the conversation.\n\nAnalysis:\n1. **Initial Request**: User asked to...\n2. **Progress**: Completed X, Y, Z...\n3. **Current state**: Working on..." + } +} +``` + +Key properties: +- **`isCompactSummary: true`** — most reliable way to identify compact summaries +- **`isVisibleInTranscriptOnly: true`** — not sent to the API, only stored in the transcript +- Summary is always a plain string in `.message.content` (not an array) +- Typically 12K-31K characters (high-density information) +- A long session may have multiple compact boundaries (4+ is common for 10MB+ sessions) +- The **last** compact boundary's summary reflects the most recent state +- Messages after the last boundary are the "hot zone" — they were in Claude's live context + +### Default compaction prompt (from API docs) + +> "You have written a partial transcript for the initial task above. Please write a summary of the transcript. The purpose of this summary is to provide continuity so you can continue to make progress towards solving the task in a future context, where the raw history above may not be accessible and will be replaced with this summary." + +## Session JSONL Message Types + +Each `.jsonl` file has one JSON object per line. Common types: + +### file-history-snapshot (always first line) + +```json +{ + "type": "file-history-snapshot", + "messageId": "uuid", + "snapshot": { "trackedFileBackups": {}, "timestamp": "..." }, + "isSnapshotUpdate": false +} +``` + +### User message + +```json +{ + "parentUuid": "prev-uuid or null", + "isSidechain": false, + "cwd": "/path/to/project", + "sessionId": "session-uuid", + "version": "2.1.71", + "gitBranch": "main", + "type": "user", + "message": { + "role": "user", + "content": "fix the login bug" + }, + "uuid": "msg-uuid", + "timestamp": "2026-03-07T03:25:03.477Z" +} +``` + +**Important**: `.message.content` can be: +- A **string** for plain text user messages +- An **array** of content blocks for tool results and multi-part messages: + ```json + "content": [ + { "type": "tool_result", "tool_use_id": "toolu_...", "content": "..." }, + { "type": "text", "text": "now do X" } + ] + ``` + +### Assistant message + +```json +{ + "type": "assistant", + "message": { + "role": "assistant", + "model": "claude-opus-4-6", + "content": [ + { "type": "thinking", "thinking": "internal reasoning..." }, + { "type": "text", "text": "visible response text" }, + { "type": "tool_use", "id": "toolu_...", "name": "Bash", "input": { "command": "..." } } + ] + } +} +``` + +Content block types in assistant messages: +- `thinking` — internal reasoning (skip when extracting actionable context) +- `text` — visible response to user (extract this) +- `tool_use` — tool invocations (useful for understanding what was done) + +### Tool result (user message with tool output) + +```json +{ + "type": "user", + "message": { + "role": "user", + "content": [ + { "type": "tool_result", "tool_use_id": "toolu_...", "content": "command output..." } + ] + } +} +``` + +## history.jsonl Schema + +Global prompt log. Does NOT contain session IDs — only useful for finding when a prompt was issued and in which project: + +```json +{ + "display": "/init ", + "pastedContents": {}, + "timestamp": 1758996122446, + "project": "/path/to/project" +} +``` + +## Extraction Patterns + +### List recent sessions for current project + +```bash +cat ~/.claude/projects//sessions-index.json \ + | jq '.entries | sort_by(.modified) | reverse | .[:5] | .[] | {sessionId, firstPrompt, summary, messageCount, modified, gitBranch}' +``` + +### Extract user text from session tail + +Handles both string and array content formats: + +```bash +tail -n 200 .jsonl \ + | jq -r 'select(.type == "user" and .message.role == "user") + | .message.content + | if type == "string" then . + elif type == "array" then map(select(.type == "text") | .text) | join("\n") + else empty end' \ + | tail -n 80 +``` + +### Extract assistant text (excluding thinking/tool_use) + +```bash +tail -n 200 .jsonl \ + | jq -r 'select(.message.role == "assistant") + | .message.content + | if type == "array" then map(select(.type == "text") | .text) | join("\n") + else empty end' \ + | tail -n 120 +``` + +### Find sessions by keyword in firstPrompt + +```bash +cat ~/.claude/projects//sessions-index.json \ + | jq -r '.entries[] | select(.firstPrompt | test("keyword"; "i")) | "\(.modified) \(.sessionId) \(.firstPrompt)"' +``` diff --git a/continue-claude-work/scripts/extract_resume_context.py b/continue-claude-work/scripts/extract_resume_context.py new file mode 100755 index 00000000..7c602247 --- /dev/null +++ b/continue-claude-work/scripts/extract_resume_context.py @@ -0,0 +1,837 @@ +#!/usr/bin/env python3 +""" +Extract actionable resume context from Claude Code session files. + +Produces a structured Markdown briefing by fusing: +- Session index metadata (sessions-index.json) +- Compact boundary summaries (highest-signal context) +- Post-compact user/assistant messages (the "hot zone") +- Subagent workflow state (multi-agent recovery) +- Session end reason detection +- Git workspace state +- MEMORY.md persistent context +- Interrupted tool-call detection + +Usage: + # Extract context from latest session for current project + python3 extract_resume_context.py + + # Extract context from a specific session + python3 extract_resume_context.py --session + + # Search sessions by topic + python3 extract_resume_context.py --query "auth feature" + + # List recent sessions + python3 extract_resume_context.py --list + + # Specify project path explicitly + python3 extract_resume_context.py --project /path/to/project +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import time +from pathlib import Path + +CLAUDE_DIR = Path.home() / ".claude" +PROJECTS_DIR = CLAUDE_DIR / "projects" + +# Message types that are noise — skip when extracting context +NOISE_TYPES = {"progress", "queue-operation", "file-history-snapshot", "last-prompt"} +# System message subtypes that are noise +NOISE_SUBTYPES = {"api_error", "turn_duration", "stop_hook_summary"} + +# Patterns that indicate system/internal content, not real user requests +NOISE_USER_PATTERNS = [ + "This session is being continued", + "", + "", +] + + +def normalize_path(project_path: str) -> str: + """Convert absolute path to Claude's normalized directory name.""" + return project_path.replace("/", "-") + + +def find_project_dir(project_path: str) -> Path | None: + """Find the Claude projects directory for a given project path.""" + abs_path = os.path.abspath(project_path) + + # If the path is already inside ~/.claude/projects/, use it directly + projects_str = str(PROJECTS_DIR) + "/" + if abs_path.startswith(projects_str): + candidate = Path(abs_path) + if candidate.is_dir(): + return candidate + rel = abs_path[len(projects_str):] + top_dir = PROJECTS_DIR / rel.split("/")[0] + if top_dir.is_dir(): + return top_dir + + normalized = normalize_path(abs_path) + candidate = PROJECTS_DIR / normalized + if candidate.is_dir(): + return candidate + # Fallback: search for partial match + for d in PROJECTS_DIR.iterdir(): + if d.is_dir() and normalized in d.name: + return d + return None + + +def load_sessions_index(project_dir: Path) -> list[dict]: + """Load and parse sessions-index.json, sorted by modified desc.""" + index_file = project_dir / "sessions-index.json" + if not index_file.exists(): + return [] + with open(index_file, encoding="utf-8") as f: + data = json.load(f) + entries = data.get("entries", []) + entries.sort(key=lambda e: e.get("modified", ""), reverse=True) + return entries + + +def search_sessions(entries: list[dict], query: str) -> list[dict]: + """Search sessions by keyword in firstPrompt and summary.""" + query_lower = query.lower() + results = [] + for entry in entries: + first_prompt = (entry.get("firstPrompt") or "").lower() + summary = (entry.get("summary") or "").lower() + if query_lower in first_prompt or query_lower in summary: + results.append(entry) + return results + + +def format_session_entry(entry: dict, file_exists: bool = True) -> str: + """Format a session index entry for display.""" + sid = entry.get("sessionId", "?") + modified = entry.get("modified", "?") + msgs = entry.get("messageCount", "?") + branch = entry.get("gitBranch", "?") + prompt = (entry.get("firstPrompt") or "")[:80] + ghost = "" if file_exists else " [file missing]" + return f" {sid} [{branch}] {msgs} msgs {modified}{ghost}\n {prompt}" + + +# ── Session file parsing ──────────────────────────────────────────── + + +def parse_session_structure(session_file: Path) -> dict: + """Parse a session JSONL file and return structured data.""" + file_size = session_file.stat().st_size + total_lines = 0 + + # First pass: find compact boundaries and count lines + compact_boundaries = [] # (line_num, summary_text) + with open(session_file, encoding="utf-8") as f: + prev_boundary_line = None + for i, raw_line in enumerate(f): + total_lines += 1 + + # Detect compact summary via isCompactSummary flag (most reliable) + if '"isCompactSummary"' in raw_line: + try: + obj = json.loads(raw_line) + if obj.get("isCompactSummary"): + content = obj.get("message", {}).get("content", "") + if isinstance(content, str): + boundary_line = prev_boundary_line if prev_boundary_line is not None else max(0, i - 1) + compact_boundaries.append((boundary_line, content)) + prev_boundary_line = None + continue + except json.JSONDecodeError: + pass + + # Detect compact_boundary marker + if '"compact_boundary"' in raw_line and '"subtype"' in raw_line: + try: + obj = json.loads(raw_line) + if obj.get("subtype") == "compact_boundary": + prev_boundary_line = i + continue + except json.JSONDecodeError: + pass + + # Fallback: if prev line was boundary and this is a user message with long string content + if prev_boundary_line is not None: + try: + obj = json.loads(raw_line) + content = obj.get("message", {}).get("content", "") + if isinstance(content, str) and len(content) > 100: + compact_boundaries.append((prev_boundary_line, content)) + except (json.JSONDecodeError, AttributeError): + compact_boundaries.append((prev_boundary_line, "")) + prev_boundary_line = None + + # Determine hot zone: everything after last compact boundary + its summary + if compact_boundaries: + last_boundary_line = compact_boundaries[-1][0] + hot_zone_start = last_boundary_line + 2 # skip boundary + summary + else: + # No compact boundaries: use size-adaptive strategy + if file_size < 500_000: # <500KB: read last 60% + hot_zone_start = max(0, int(total_lines * 0.4)) + elif file_size < 5_000_000: # <5MB: read last 30% + hot_zone_start = max(0, int(total_lines * 0.7)) + else: # >5MB: read last 15% + hot_zone_start = max(0, int(total_lines * 0.85)) + + # Second pass: extract hot zone messages + messages = [] + unresolved_tool_calls = {} # tool_use_id -> tool_use_info + errors = [] + files_touched = set() + last_message_role = None + error_count = 0 + + with open(session_file, encoding="utf-8") as f: + for i, raw_line in enumerate(f): + if i < hot_zone_start: + continue + try: + obj = json.loads(raw_line) + except json.JSONDecodeError: + continue + + msg_type = obj.get("type", "") + if msg_type in NOISE_TYPES: + continue + if msg_type == "system": + subtype = obj.get("subtype", "") + if subtype in NOISE_SUBTYPES: + if subtype == "api_error": + error_count += 1 + continue + if subtype == "compact_boundary": + continue + + # Track tool calls and results + msg = obj.get("message", {}) + role = msg.get("role", "") + content = msg.get("content", "") + + # Extract tool_use from assistant messages + if role == "assistant" and isinstance(content, list): + for block in content: + if not isinstance(block, dict) or block.get("type") != "tool_use": + continue + tool_id = block.get("id", "") + tool_name = block.get("name", "?") + inp = block.get("input", {}) + unresolved_tool_calls[tool_id] = { + "name": tool_name, + "input_preview": str(inp)[:200], + } + # Track file operations + if tool_name in ("Write", "Edit", "Read"): + fp = inp.get("file_path", "") + if fp: + files_touched.add(fp) + elif tool_name == "Bash": + cmd = inp.get("command", "") + for match in re.findall(r'(? str: + """Detect why the session ended.""" + if unresolved: + return "interrupted" # Tool calls dispatched but no results — likely ctrl-c + if error_count >= 3: + return "error_cascade" # Multiple API errors suggest systemic failure + if last_role == "assistant": + return "completed" # Assistant had the last word — clean end + if last_role == "user": + return "abandoned" # User sent a message but got no response + return "unknown" + + +def _is_noise_user_text(text: str) -> bool: + """Check if user text is system noise rather than a real request.""" + for pattern in NOISE_USER_PATTERNS: + if text.startswith(pattern) or pattern in text[:200]: + return True + return False + + +def extract_user_text(messages: list[dict], limit: int = 5) -> list[str]: + """Extract the last N user text messages (not tool results or system noise).""" + user_texts = [] + for msg_obj in reversed(messages): + if msg_obj.get("isCompactSummary"): + continue + msg = msg_obj.get("message", {}) + if msg.get("role") != "user": + continue + content = msg.get("content", "") + if isinstance(content, str) and content.strip(): + if _is_noise_user_text(content): + continue + user_texts.append(content.strip()) + elif isinstance(content, list): + texts = [ + b.get("text", "") + for b in content + if isinstance(b, dict) and b.get("type") == "text" + ] + combined = "\n".join(t for t in texts if t.strip()) + if combined and not _is_noise_user_text(combined): + user_texts.append(combined) + if len(user_texts) >= limit: + break + user_texts.reverse() + return user_texts + + +def extract_assistant_text(messages: list[dict], limit: int = 3) -> list[str]: + """Extract the last N assistant text responses (no thinking/tool_use).""" + assistant_texts = [] + for msg_obj in reversed(messages): + msg = msg_obj.get("message", {}) + if msg.get("role") != "assistant": + continue + content = msg.get("content", "") + if isinstance(content, list): + texts = [ + b.get("text", "") + for b in content + if isinstance(b, dict) and b.get("type") == "text" + ] + combined = "\n".join(t for t in texts if t.strip()) + if combined: + assistant_texts.append(combined[:2000]) + if len(assistant_texts) >= limit: + break + assistant_texts.reverse() + return assistant_texts + + +# ── Subagent extraction ────────────────────────────────────────────── + + +def extract_subagent_context(session_file: Path) -> list[dict]: + """Extract subagent summaries from session subdirectories. + + Returns list of {name, type, status, last_text, is_interrupted}. + """ + session_dir = session_file.parent / session_file.stem + subagents_dir = session_dir / "subagents" + if not subagents_dir.is_dir(): + return [] + + # Group by agent ID: find meta.json and .jsonl pairs + agent_ids = set() + for f in subagents_dir.iterdir(): + if f.suffix == ".jsonl": + agent_ids.add(f.stem) + + results = [] + for agent_id in sorted(agent_ids): + jsonl_file = subagents_dir / f"{agent_id}.jsonl" + meta_file = subagents_dir / f"{agent_id}.meta.json" + + # Parse agent type from ID (format: agent-a- or agent-a) + agent_type = "unknown" + if meta_file.exists(): + try: + with open(meta_file, encoding="utf-8") as f: + meta = json.load(f) + agent_type = meta.get("type", meta.get("subagent_type", "unknown")) + except (json.JSONDecodeError, OSError): + pass + + if agent_type == "unknown": + # Infer from ID pattern: agent-a- + match = re.match(r'agent-a(compact|prompt_suggestion|[a-z_]+)-', agent_id) + if match: + agent_type = match.group(1) + + # Skip compact and prompt_suggestion agents (internal, not user work) + if agent_type in ("compact", "prompt_suggestion"): + continue + + # Read last few lines for final output + last_text = "" + is_interrupted = False + line_count = 0 + + if jsonl_file.exists(): + try: + lines = jsonl_file.read_text(encoding="utf-8").strip().split("\n") + line_count = len(lines) + has_tool_use_pending = False + # Check last 10 lines for final assistant text + for raw_line in reversed(lines[-10:]): + try: + obj = json.loads(raw_line) + msg = obj.get("message", {}) + role = msg.get("role", "") + content = msg.get("content", "") + + if role == "assistant" and isinstance(content, list): + for block in content: + if not isinstance(block, dict): + continue + block_type = block.get("type") + if block_type == "tool_use": + has_tool_use_pending = True + elif block_type == "text": + text = block.get("text", "") + if text.strip() and not last_text: + last_text = text.strip()[:500] + + if role == "user" and isinstance(content, list): + for block in content: + if isinstance(block, dict) and block.get("type") == "tool_result": + has_tool_use_pending = False + except json.JSONDecodeError: + continue + + is_interrupted = has_tool_use_pending + except OSError: + pass + + results.append({ + "id": agent_id, + "type": agent_type, + "last_text": last_text, + "is_interrupted": is_interrupted, + "lines": line_count, + }) + + return results + + +# ── Context sources ────────────────────────────────────────────────── + + +def get_git_state(project_path: str) -> str: + """Get current git status and recent log.""" + parts = [] + try: + branch = subprocess.run( + ["git", "branch", "--show-current"], + capture_output=True, text=True, cwd=project_path, timeout=5, + ) + if branch.stdout.strip(): + parts.append(f"**Current branch**: `{branch.stdout.strip()}`") + except (subprocess.TimeoutExpired, FileNotFoundError): + pass + + try: + status = subprocess.run( + ["git", "status", "--short"], + capture_output=True, text=True, cwd=project_path, timeout=10, + ) + if status.stdout.strip(): + parts.append(f"### git status\n```\n{status.stdout.strip()}\n```") + else: + parts.append("### git status\nClean working tree.") + except (subprocess.TimeoutExpired, FileNotFoundError): + parts.append("### git status\n(unavailable)") + + try: + log = subprocess.run( + ["git", "log", "--oneline", "-5"], + capture_output=True, text=True, cwd=project_path, timeout=10, + ) + if log.stdout.strip(): + parts.append(f"### git log (last 5)\n```\n{log.stdout.strip()}\n```") + except (subprocess.TimeoutExpired, FileNotFoundError): + pass + + return "\n\n".join(parts) + + +def get_memory_md(project_dir: Path) -> str | None: + """Read MEMORY.md if it exists in the project's memory directory.""" + memory_dir = project_dir / "memory" + memory_file = memory_dir / "MEMORY.md" + if memory_file.exists(): + content = memory_file.read_text(encoding="utf-8").strip() + if content: + return content[:3000] + return None + + +def get_session_memory(session_file: Path) -> str | None: + """Read session-memory/summary.md if it exists (newer CC versions).""" + session_dir = session_file.parent / session_file.stem + summary = session_dir / "session-memory" / "summary.md" + if summary.exists(): + content = summary.read_text(encoding="utf-8").strip() + if content: + return content[:3000] + return None + + +# ── Output formatting ──────────────────────────────────────────────── + + +END_REASON_LABELS = { + "completed": "Clean exit (assistant completed response)", + "interrupted": "Interrupted (unresolved tool calls — likely ctrl-c or timeout)", + "error_cascade": "Error cascade (multiple API errors)", + "abandoned": "Abandoned (user message with no response)", + "unknown": "Unknown", +} + + +def build_briefing( + session_entry: dict | None, + parsed: dict, + project_path: str, + project_dir: Path, + session_file: Path, +) -> str: + """Build the structured Markdown briefing.""" + sections = [] + + # Header + sections.append("# Resume Context Briefing\n") + + # Session metadata + if session_entry: + sid = session_entry.get("sessionId", "?") + modified = session_entry.get("modified", "?") + branch = session_entry.get("gitBranch", "?") + msg_count = session_entry.get("messageCount", "?") + first_prompt = session_entry.get("firstPrompt", "") + summary = session_entry.get("summary", "") + + sections.append("## Session Info\n") + sections.append(f"- **ID**: `{sid}`") + sections.append(f"- **Last active**: {modified}") + sections.append(f"- **Branch**: `{branch}`") + sections.append(f"- **Messages**: {msg_count}") + sections.append(f"- **First prompt**: {first_prompt}") + if summary: + sections.append(f"- **Summary**: {summary[:300]}") + + # File stats + end reason + file_mb = parsed["file_size"] / 1_000_000 + end_label = END_REASON_LABELS.get(parsed["end_reason"], parsed["end_reason"]) + sections.append(f"\n**Session file**: {file_mb:.1f} MB, {parsed['total_lines']} lines, " + f"{len(parsed['compact_boundaries'])} compaction(s)") + sections.append(f"**Session end reason**: {end_label}") + if parsed["error_count"] > 0: + sections.append(f"**API errors**: {parsed['error_count']}") + + # Session memory (newer CC versions generate this automatically) + session_mem = get_session_memory(session_file) + if session_mem: + sections.append("\n## Session Memory (auto-generated by Claude Code)\n") + sections.append(session_mem) + + # Compact summary (highest-signal context) + if parsed["compact_boundaries"]: + last_summary = parsed["compact_boundaries"][-1][1] + if last_summary: + # Allow up to 8K chars — this is the highest-signal content + display = last_summary[:8000] + if len(last_summary) > 8000: + display += f"\n\n... (truncated, full summary: {len(last_summary)} chars)" + sections.append("\n## Compact Summary (auto-generated by previous session)\n") + sections.append(display) + + # Last user requests + user_texts = extract_user_text(parsed["messages"]) + if user_texts: + sections.append("\n## Last User Requests\n") + for i, text in enumerate(user_texts, 1): + display = text[:500] + if len(text) > 500: + display += "..." + sections.append(f"### Request {i}\n{display}\n") + + # Last assistant responses + assistant_texts = extract_assistant_text(parsed["messages"]) + if assistant_texts: + sections.append("\n## Last Assistant Responses\n") + for i, text in enumerate(assistant_texts, 1): + display = text[:1000] + if len(text) > 1000: + display += "..." + sections.append(f"### Response {i}\n{display}\n") + + # Errors encountered + if parsed["errors"]: + sections.append("\n## Errors Encountered\n") + seen = set() + for err in parsed["errors"]: + short = err[:200] + if short not in seen: + seen.add(short) + sections.append(f"```\n{err}\n```\n") + + # Unresolved tool calls (interrupted session) + if parsed["unresolved_tool_calls"]: + sections.append("\n## Unresolved Tool Calls (session was interrupted)\n") + for tool_id, info in parsed["unresolved_tool_calls"].items(): + sections.append(f"- **{info['name']}**: `{tool_id}`") + sections.append(f" Input: {info['input_preview']}") + + # Subagent context (the "nobody has done this" feature) + subagents = extract_subagent_context(session_file) + if subagents: + interrupted = [s for s in subagents if s["is_interrupted"]] + completed = [s for s in subagents if not s["is_interrupted"]] + sections.append(f"\n## Subagent Workflow ({len(completed)} completed, {len(interrupted)} interrupted)\n") + if interrupted: + sections.append("### Interrupted Subagents\n") + for sa in interrupted: + sections.append(f"- **{sa['type']}** (`{sa['id']}`, {sa['lines']} lines)") + if sa["last_text"]: + sections.append(f" Last output: {sa['last_text'][:300]}") + if completed: + sections.append("\n### Completed Subagents\n") + for sa in completed: + sections.append(f"- **{sa['type']}** (`{sa['id']}`, {sa['lines']} lines)") + if sa["last_text"]: + sections.append(f" Last output: {sa['last_text'][:200]}") + + # Files touched in session + if parsed["files_touched"]: + sections.append("\n## Files Touched in Session\n") + for fp in sorted(parsed["files_touched"])[:30]: + sections.append(f"- `{fp}`") + + # MEMORY.md + memory = get_memory_md(project_dir) + if memory: + sections.append("\n## Persistent Memory (MEMORY.md)\n") + sections.append(memory) + + # Git state + sections.append("\n## Current Workspace State\n") + sections.append(get_git_state(project_path)) + + return "\n".join(sections) + + +# ── CLI ────────────────────────────────────────────────────────────── + + +def _check_session_files(entries: list[dict], project_dir: Path) -> dict[str, bool]: + """Check which index entries have actual files on disk.""" + status = {} + for entry in entries: + sid = entry.get("sessionId", "") + session_file = project_dir / f"{sid}.jsonl" + if session_file.exists(): + status[sid] = True + else: + full_path = entry.get("fullPath", "") + status[sid] = bool(full_path and Path(full_path).exists()) + return status + + +def main(): + parser = argparse.ArgumentParser( + description="Extract actionable resume context from Claude Code sessions.", + ) + parser.add_argument( + "--project", "-p", + default=os.getcwd(), + help="Project path (default: current directory)", + ) + parser.add_argument( + "--session", "-s", + default=None, + help="Session ID to extract context from", + ) + parser.add_argument( + "--query", "-q", + default=None, + help="Search sessions by keyword in firstPrompt/summary", + ) + parser.add_argument( + "--list", "-l", + action="store_true", + help="List recent sessions", + ) + parser.add_argument( + "--limit", "-n", + type=int, + default=10, + help="Number of sessions to list (default: 10)", + ) + parser.add_argument( + "--exclude-current", + default=None, + help="Session ID to exclude (typically the currently active session)", + ) + args = parser.parse_args() + + project_path = os.path.abspath(args.project) + project_dir = find_project_dir(project_path) + + if not project_dir: + print(f"Error: no Claude session data found for {project_path}", file=sys.stderr) + print(f"Looked in: {PROJECTS_DIR / normalize_path(project_path)}", file=sys.stderr) + sys.exit(1) + + entries = load_sessions_index(project_dir) + + # ── List mode ── + if args.list: + # Show both index entries and actual files, with file-exists status + file_status = _check_session_files(entries, project_dir) + existing = sum(1 for v in file_status.values() if v) + missing = sum(1 for v in file_status.values() if not v) + + if not entries and not list(project_dir.glob("*.jsonl")): + print("No sessions found.") + sys.exit(0) + + print(f"Sessions for {project_path}:\n") + if missing > 0: + print(f" (index has {len(entries)} entries, {existing} with files, {missing} with missing files)\n") + + for entry in entries[:args.limit]: + sid = entry.get("sessionId", "") + exists = file_status.get(sid, False) + print(format_session_entry(entry, file_exists=exists)) + print() + + # Also show files NOT in index + indexed_ids = {e.get("sessionId") for e in entries} + orphan_files = [ + f for f in sorted(project_dir.glob("*.jsonl"), key=lambda f: f.stat().st_mtime, reverse=True) + if f.stem not in indexed_ids + ] + if orphan_files: + print(f" Files not in index ({len(orphan_files)}):") + for f in orphan_files[:5]: + size_kb = f.stat().st_size / 1000 + print(f" {f.stem} ({size_kb:.0f} KB)") + print() + + sys.exit(0) + + # ── Query mode ── + if args.query: + results = search_sessions(entries, args.query) + if not results: + print(f"No sessions matching '{args.query}'.", file=sys.stderr) + sys.exit(1) + print(f"Sessions matching '{args.query}' ({len(results)} found):\n") + for entry in results[: args.limit]: + print(format_session_entry(entry)) + print() + if len(results) == 1: + args.session = results[0]["sessionId"] + else: + sys.exit(0) + + # ── Extract mode ── + session_id = args.session + session_entry = None + + if session_id: + for entry in entries: + if entry.get("sessionId") == session_id: + session_entry = entry + break + session_file = project_dir / f"{session_id}.jsonl" + if not session_file.exists(): + if session_entry: + full_path = session_entry.get("fullPath", "") + if full_path and Path(full_path).exists(): + session_file = Path(full_path) + if not session_file.exists(): + for jsonl in project_dir.glob("*.jsonl"): + if session_id in jsonl.name: + session_file = jsonl + break + else: + print(f"Error: session file not found for {session_id}", file=sys.stderr) + sys.exit(1) + else: + # Use latest session — prefer actual files, skip current session + jsonl_files = sorted( + project_dir.glob("*.jsonl"), + key=lambda f: f.stat().st_mtime, + reverse=True, + ) + if args.exclude_current: + jsonl_files = [f for f in jsonl_files if f.stem != args.exclude_current] + if not jsonl_files: + print("Error: no session files found.", file=sys.stderr) + sys.exit(1) + + # Skip the most recent file if it's likely the current active session + # (modified within the last 60 seconds and no explicit session requested) + if len(jsonl_files) > 1: + newest = jsonl_files[0] + age_seconds = time.time() - newest.stat().st_mtime + if age_seconds < 60: + print(f"Skipping active session {newest.stem} (modified {age_seconds:.0f}s ago)", + file=sys.stderr) + jsonl_files = jsonl_files[1:] + + session_file = jsonl_files[0] + session_id = session_file.stem + for entry in entries: + if entry.get("sessionId") == session_id: + session_entry = entry + break + + # Parse and build briefing + print(f"Parsing session {session_id} ({session_file.stat().st_size / 1_000_000:.1f} MB)...", + file=sys.stderr) + + parsed = parse_session_structure(session_file) + briefing = build_briefing(session_entry, parsed, project_path, project_dir, session_file) + + print(briefing) + + +if __name__ == "__main__": + main() diff --git a/github-contributor/references/project_evaluation.md b/github-contributor/references/project_evaluation.md index 8916de75..94d9d5ff 100644 --- a/github-contributor/references/project_evaluation.md +++ b/github-contributor/references/project_evaluation.md @@ -2,17 +2,24 @@ How to evaluate open-source projects before contributing. +## Prerequisites + +- Install GitHub CLI and verify availability: `gh --version` +- Authenticate before running commands: `gh auth status || gh auth login` + ## Quick Health Check ```bash # Check recent activity -gh repo view owner/repo --json updatedAt,stargazersCount,openIssues +gh repo view owner/repo \ + --json updatedAt,stargazerCount,issues \ + --jq '{updatedAt, stargazers: .stargazerCount, openIssues: .issues.totalCount}' # Check PR response time gh pr list --repo owner/repo --state merged --limit 10 # Check issue activity -gh issue list --repo owner/repo --limit 20 +gh issue list --repo owner/repo --state=open --limit 20 ``` ## Evaluation Criteria @@ -131,7 +138,7 @@ gh search repos "topic:cli" --sort=stars gh search repos "language:python" --sort=stars # Find with good first issues -gh search issues "good first issue" --language=rust +gh search issues "good first issue" --language=rust --state=open ``` ### By Need diff --git a/pdf-creator/scripts/md_to_pdf.py b/pdf-creator/scripts/md_to_pdf.py index cf79ea4e..1e351e78 100644 --- a/pdf-creator/scripts/md_to_pdf.py +++ b/pdf-creator/scripts/md_to_pdf.py @@ -2,7 +2,7 @@ """ Markdown to PDF converter with Chinese font support. -Converts markdown files to PDF using weasyprint, with proper Chinese typography. +Converts markdown files to PDF using pandoc (markdown→HTML) + weasyprint (HTML→PDF). Designed for formal documents (trademark filings, legal documents, reports). Usage: @@ -10,7 +10,8 @@ python md_to_pdf.py input.md # outputs input.pdf Requirements: - pip install weasyprint markdown + pip install weasyprint + pandoc (system install, e.g. brew install pandoc) macOS environment setup (if needed): export DYLD_LIBRARY_PATH="/opt/homebrew/lib:$DYLD_LIBRARY_PATH" @@ -18,7 +19,8 @@ import os import platform -import re +import shutil +import subprocess import sys from pathlib import Path @@ -30,7 +32,6 @@ if _homebrew_lib not in _cur: os.environ['DYLD_LIBRARY_PATH'] = f"{_homebrew_lib}:{_cur}" if _cur else _homebrew_lib -import markdown from weasyprint import CSS, HTML @@ -145,22 +146,21 @@ """ -def _ensure_list_spacing(text: str) -> str: - """Ensure blank lines before list items for proper markdown parsing. +def _md_to_html(md_file: str) -> str: + """Convert markdown to HTML using pandoc.""" + if not shutil.which('pandoc'): + print("Error: pandoc not found. Install with: brew install pandoc", file=sys.stderr) + sys.exit(1) - The Python markdown library requires a blank line before a list when it - follows a paragraph. Without it, list items render as plain text. - """ - lines = text.split('\n') - result = [] - list_re = re.compile(r'^(\s*)([-*+]|\d+\.)\s') - for i, line in enumerate(lines): - if i > 0 and list_re.match(line): - prev = lines[i - 1] - if prev.strip() and not list_re.match(prev): - result.append('') - result.append(line) - return '\n'.join(result) + result = subprocess.run( + ['pandoc', md_file, '-f', 'markdown', '-t', 'html'], + capture_output=True, text=True, + ) + if result.returncode != 0: + print(f"Error: pandoc failed: {result.stderr}", file=sys.stderr) + sys.exit(1) + + return result.stdout def markdown_to_pdf(md_file: str, pdf_file: str | None = None) -> str: @@ -179,14 +179,8 @@ def markdown_to_pdf(md_file: str, pdf_file: str | None = None) -> str: if pdf_file is None: pdf_file = str(md_path.with_suffix('.pdf')) - # Read and preprocess markdown content - md_content = _ensure_list_spacing(md_path.read_text(encoding='utf-8')) - - # Convert to HTML - html_content = markdown.markdown( - md_content, - extensions=['tables', 'fenced_code', 'codehilite', 'toc'] - ) + # Convert to HTML via pandoc + html_content = _md_to_html(md_file) # Create full HTML document full_html = f""" diff --git a/skill-creator/SKILL.md b/skill-creator/SKILL.md index b7da6f16..b224f242 100644 --- a/skill-creator/SKILL.md +++ b/skill-creator/SKILL.md @@ -1,49 +1,90 @@ --- name: skill-creator -description: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations. +description: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, update or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. license: Complete terms in LICENSE.txt --- # Skill Creator -This skill provides guidance for creating effective skills. +A skill for creating new skills and iteratively improving them. -## About Skills +At a high level, the process of creating a skill goes like this: -Skills are modular, self-contained packages that extend Claude's capabilities by providing -specialized knowledge, workflows, and tools. Think of them as "onboarding guides" for specific -domains or tasks—they transform Claude from a general-purpose agent into a specialized agent -equipped with procedural knowledge that no model can fully possess. +- Decide what you want the skill to do and roughly how it should do it +- Write a draft of the skill +- Create a few test prompts and run claude-with-access-to-the-skill on them +- Help the user evaluate the results both qualitatively and quantitatively + - While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist) + - Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics +- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks) +- Repeat until you're satisfied +- Expand the test set and try again at larger scale -### What Skills Provide +Your job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like "I want to make a skill for X". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat. -1. Specialized workflows - Multi-step procedures for specific domains -2. Tool integrations - Instructions for working with specific file formats or APIs -3. Domain expertise - Company-specific knowledge, schemas, business logic -4. Bundled resources - Scripts, references, and assets for complex and repetitive tasks +On the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop. -### Anatomy of a Skill +Of course, you should always be flexible and if the user is like "I don't need to run a bunch of evaluations, just vibe with me", you can do that instead. -Every skill consists of a required SKILL.md file and optional bundled resources: +Then after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill. + +Cool? Cool. + +## Communicating with the user + +The skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google "how to install npm". On the other hand, the bulk of users are probably fairly computer-literate. + +So please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea: + +- "evaluation" and "benchmark" are borderline, but OK +- for "JSON" and "assertion" you want to see serious cues from the user that they know what those things are before using them without explaining them + +It's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it. + +--- + +## Creating a skill + +### Capture Intent + +Start by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say "turn this into a skill"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step. + +1. What should this skill enable Claude to do? +2. When should this skill trigger? (what user phrases/contexts) +3. What's the expected output format? +4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide. + +### Interview and Research + +Proactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out. + +Check available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user. + +### Write the SKILL.md + +Based on the user interview, fill in these components: + +- **name**: Skill identifier +- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All "when to use" info goes here, not in the body. Note: currently Claude has a tendency to "undertrigger" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit "pushy". So for instance, instead of "How to build a simple fast dashboard to display internal Anthropic data.", you might write "How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'" +- **compatibility**: Required tools, dependencies (optional, rarely needed) +- **the rest of the skill :)** + +### Skill Writing Guide + +#### Anatomy of a Skill ``` skill-name/ ├── SKILL.md (required) -│ ├── YAML frontmatter metadata (required) -│ │ ├── name: (required) -│ │ └── description: (required) -│ └── Markdown instructions (required) +│ ├── YAML frontmatter (name, description required) +│ └── Markdown instructions └── Bundled Resources (optional) - ├── scripts/ - Executable code (Python/Bash/etc.) - ├── references/ - Documentation intended to be loaded into context as needed - └── assets/ - Files used in output (templates, icons, fonts, etc.) + ├── scripts/ - Executable code for deterministic/repetitive tasks + ├── references/ - Docs loaded into context as needed + └── assets/ - Files used in output (templates, icons, fonts) ``` -#### SKILL.md (required) - -**Metadata Quality:** The `name` and `description` in YAML frontmatter determine when Claude will use the skill. Be specific about what the skill does and when to use it. Use the third-person (e.g. "This skill should be used when..." instead of "Use this skill when..."). - -##### YAML Frontmatter Reference +#### YAML Frontmatter Reference All frontmatter fields except `description` are optional. Configure skill behavior using these fields between `---` markers: @@ -180,7 +221,62 @@ argument-hint: [scope] [--with-codex] [--docker] [--verbose] | `disable-model-invocation: true` | Yes | No | No | | `context: fork` + `disable-model-invocation: true` | Yes | No | Yes (when explicitly delegated) | -#### Bundled Resources (optional) +#### Progressive Disclosure + +Skills use a three-level loading system: +1. **Metadata** (name + description) - Always in context (~100 words) +2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal) +3. **Bundled resources** - As needed (unlimited, scripts can execute without loading) + +These word counts are approximate and you can feel free to go longer if needed. + +**Key patterns:** +- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up. +- Reference files clearly from SKILL.md with guidance on when to read them +- For large reference files (>300 lines), include a table of contents + +**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant: +``` +cloud-deploy/ +├── SKILL.md (workflow + selection) +└── references/ + ├── aws.md + ├── gcp.md + └── azure.md +``` +Claude reads only the relevant reference file. + +#### Principle of Lack of Surprise + +This goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a "roleplay as an XYZ" are OK though. + +#### Writing Patterns + +Prefer using the imperative form in instructions. + +**Defining output formats** - You can do it like this: +```markdown +## Report structure +ALWAYS use this exact template: +# [Title] +## Executive summary +## Key findings +## Recommendations +``` + +**Examples pattern** - It's useful to include examples. You can format them like this (but if "Input" and "Output" are in the examples you might want to deviate a little): +```markdown +## Commit message format +**Example 1:** +Input: Added user authentication with JWT tokens +Output: feat(auth): implement JWT-based authentication +``` + +### Writing Style + +Try to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it. + +#### Bundled Resources ##### Scripts (`scripts/`) @@ -196,58 +292,322 @@ Executable code (Python/Bash/etc.) for tasks that require deterministic reliabil Documentation and reference material intended to be loaded as needed into context to inform Claude's process and thinking. - **When to include**: For documentation that Claude should reference while working -- **Examples**: `references/finance.md` for financial schemas, `references/mnda.md` for company NDA template, `references/policies.md` for company policies, `references/api_docs.md` for API specifications +- **Examples**: `references/finance.md` for financial schemas, `references/mnda.md` for company NDA template - **Use cases**: Database schemas, API documentation, domain knowledge, company policies, detailed workflow guides - **Benefits**: Keeps SKILL.md lean, loaded only when Claude determines it's needed - **Best practice**: If files are large (>10k words), include grep search patterns in SKILL.md -- **Avoid duplication**: Information should live in either SKILL.md or references files, not both. Prefer references files for detailed information unless it's truly core to the skill—this keeps SKILL.md lean while making information discoverable without hogging the context window. Keep only essential procedural instructions and workflow guidance in SKILL.md; move detailed reference material, schemas, and examples to references files. +- **Avoid duplication**: Information should live in either SKILL.md or references files, not both ##### Assets (`assets/`) Files not intended to be loaded into context, but rather used within the output Claude produces. - **When to include**: When the skill needs files that will be used in the final output -- **Examples**: `assets/logo.png` for brand assets, `assets/slides.pptx` for PowerPoint templates, `assets/frontend-template/` for HTML/React boilerplate, `assets/font.ttf` for typography -- **Use cases**: Templates, images, icons, boilerplate code, fonts, sample documents that get copied or modified -- **Benefits**: Separates output resources from documentation, enables Claude to use files without loading them into context +- **Examples**: `assets/logo.png` for brand assets, `assets/slides.pptx` for PowerPoint templates +- **Use cases**: Templates, images, icons, boilerplate code, fonts, sample documents ##### Privacy and Path References **CRITICAL**: Skills intended for public distribution must not contain user-specific or company-specific information: -- **Forbidden**: Absolute paths to user directories (`/home/username/`, `/Users/username/`, `/mnt/c/Users/username/`) -- **Forbidden**: Personal usernames, company names, department names, product names -- **Forbidden**: OneDrive paths, cloud storage paths, or any environment-specific absolute paths -- **Forbidden**: Hardcoded skill installation paths like `~/.claude/skills/` or `/Users/username/Workspace/claude-code-skills/` +- **Forbidden**: Absolute paths to user directories (`/home/username/`, `/Users/username/`) +- **Forbidden**: Personal usernames, company names, product names +- **Forbidden**: Hardcoded skill installation paths like `~/.claude/skills/` - **Allowed**: Relative paths within the skill bundle (`scripts/example.py`, `references/guide.md`) - **Allowed**: Standard placeholders (`~/workspace/project`, `username`, `your-company`) -- **Best practice**: Reference bundled scripts using simple relative paths like `scripts/script_name.py` - Claude will resolve the actual location ##### Versioning **CRITICAL**: Skills should NOT contain version history or version numbers in SKILL.md: -- **Forbidden**: Version sections (`## Version`, `## Changelog`, `## Release History`) in SKILL.md -- **Forbidden**: Version numbers in SKILL.md body content +- **Forbidden**: Version sections (`## Version`, `## Changelog`) in SKILL.md - **Correct location**: Skill versions are tracked in marketplace.json under `plugins[].version` -- **Rationale**: Marketplace infrastructure manages versioning; SKILL.md should be timeless content focused on functionality -- **Example**: Instead of documenting v1.0.0 → v1.1.0 changes in SKILL.md, update the version in marketplace.json only +- **Rationale**: Marketplace infrastructure manages versioning; SKILL.md should be timeless content -### Progressive Disclosure Design Principle +#### Reference File Naming -Skills use a three-level loading system to manage context efficiently: +Filenames must be self-explanatory without reading contents. -1. **Metadata (name + description)** - Always in context (~100 words) -2. **SKILL.md body** - When skill triggers (<5k words) -3. **Bundled resources** - As needed by Claude (Unlimited*) +**Pattern**: `_.md` -*Unlimited because scripts can be executed without reading into context window. +**Examples**: +- Bad: `commands.md`, `cli_usage.md`, `reference.md` +- Good: `script_parameters.md`, `api_endpoints.md`, `database_schema.md` + +**Test**: Can someone understand the file's contents from the name alone? ### Skill Creation Best Practice Anthropic has wrote skill authoring best practices, you SHOULD retrieve it before you create or update any skills, the link is https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices.md -## ⚠️ CRITICAL: Edit Skills at Source Location +### Test Cases + +After writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] "Here are a few test cases I'd like to try. Do these look right, or do you want to add more?" Then run them. + +Save test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress. + +```json +{ + "skill_name": "example-skill", + "evals": [ + { + "id": 1, + "prompt": "User's task prompt", + "expected_output": "Description of expected result", + "files": [] + } + ] +} +``` + +See `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later). + +## Running and evaluating test cases + +This section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill. + +Put results in `-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go. + +### Step 1: Spawn all runs (with-skill AND baseline) in the same turn + +For each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time. + +**With-skill run:** + +``` +Execute this task: +- Skill path: +- Task: +- Input files: +- Save outputs to: /iteration-/eval-/with_skill/outputs/ +- Outputs to save: +``` + +**Baseline run** (same prompt, but the baseline depends on context): +- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`. +- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r /skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`. + +Write an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just "eval-0". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations. + +```json +{ + "eval_id": 0, + "eval_name": "descriptive-name-here", + "prompt": "The user's task prompt", + "assertions": [] +} +``` + +### Step 2: While runs are in progress, draft assertions + +Don't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check. + +Good assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment. + +Update the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark. + +### Step 3: As runs complete, capture timing data + +When each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory: + +```json +{ + "total_tokens": 84852, + "duration_ms": 23332, + "total_duration_seconds": 23.3 +} +``` + +This is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them. + +### Step 4: Grade, aggregate, and launch the viewer + +Once all runs are done: + +1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations. + +2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory: + ```bash + python -m scripts.aggregate_benchmark /iteration-N --skill-name + ``` + This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean +/- stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects. +Put each with_skill version before its baseline counterpart. + +3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the "Analyzing Benchmark Results" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs. + +4. **Launch the viewer** with both qualitative outputs and quantitative data: + ```bash + nohup python /eval-viewer/generate_review.py \ + /iteration-N \ + --skill-name "my-skill" \ + --benchmark /iteration-N/benchmark.json \ + > /dev/null 2>&1 & + VIEWER_PID=$! + ``` + For iteration 2+, also pass `--previous-workspace /iteration-`. + + **Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static ` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks "Submit All Reviews". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up. + +Note: please use generate_review.py to create the viewer; there's no need to write custom HTML. + +5. **Tell the user** something like: "I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know." + +### What the user sees in the viewer + +The "Outputs" tab shows one test case at a time: +- **Prompt**: the task that was given +- **Output**: the files the skill produced, rendered inline where possible +- **Previous Output** (iteration 2+): collapsed section showing last iteration's output +- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail +- **Feedback**: a textbox that auto-saves as they type +- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox + +The "Benchmark" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations. + +Navigation is via prev/next buttons or arrow keys. When done, they click "Submit All Reviews" which saves all feedback to `feedback.json`. + +### Step 5: Read the feedback + +When the user tells you they're done, read `feedback.json`: + +```json +{ + "reviews": [ + {"run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "..."}, + {"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."}, + {"run_id": "eval-2-with_skill", "feedback": "perfect, love this", "timestamp": "..."} + ], + "status": "complete" +} +``` + +Empty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints. + +Kill the viewer server when you're done with it: + +```bash +kill $VIEWER_PID 2>/dev/null +``` + +--- + +## Improving the skill + +This is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback. + +### How to think about improvements + +1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great. + +2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens. + +3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are *smart*. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach. + +4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel. + +This task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need. + +### The iteration loop + +After improving the skill: + +1. Apply your improvements to the skill +2. Rerun all test cases into a new `iteration-/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration. +3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration +4. Wait for the user to review and tell you they're done +5. Read the new feedback, improve again, repeat + +Keep going until: +- The user says they're happy +- The feedback is all empty (everything looks good) +- You're not making meaningful progress + +--- + +## Advanced: Blind comparison + +For situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks "is the new version actually better?"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won. + +This is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient. + +--- + +## Description Optimization + +The description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy. + +### Step 1: Generate trigger eval queries + +Create 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON: + +```json +[ + {"query": "the user prompt", "should_trigger": true}, + {"query": "another prompt", "should_trigger": false} +] +``` + +The queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them). + +Bad: `"Format this data"`, `"Extract text from PDF"`, `"Create a chart"` + +Good: `"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think"` + +For the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win. + +For the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate. + +The key thing to avoid: don't make should-not-trigger queries obviously irrelevant. "Write a fibonacci function" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky. + +### Step 2: Review with user + +Present the eval set to the user for review using the HTML template: + +1. Read the template from `assets/eval_review.html` +2. Replace the placeholders: + - `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment) + - `__SKILL_NAME_PLACEHOLDER__` → the skill's name + - `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description +3. Write to a temp file (e.g., `/tmp/eval_review_.html`) and open it: `open /tmp/eval_review_.html` +4. The user can edit queries, toggle should-trigger, add/remove entries, then click "Export Eval Set" +5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`) + +This step matters — bad eval queries lead to bad descriptions. + +### Step 3: Run the optimization loop + +Tell the user: "This will take some time — I'll run the optimization loop in the background and check on it periodically." + +Save the eval set to the workspace, then run in the background: + +```bash +python -m scripts.run_loop \ + --eval-set \ + --skill-path \ + --model \ + --max-iterations 5 \ + --verbose +``` + +Use the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences. + +While it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like. + +This handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude with extended thinking to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting. + +### How skill triggering works + +Understanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like "read this PDF" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches. + +This means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like "read file X" are poor test cases — they won't trigger skills regardless of description quality. + +### Step 4: Apply the result + +Take `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores. + +--- + +## CRITICAL: Edit Skills at Source Location **NEVER edit skills in `~/.claude/plugins/cache/`** — that's a read-only cache directory. All changes there are: - Lost when cache refreshes @@ -265,13 +625,15 @@ Anthropic has wrote skill authoring best practices, you SHOULD retrieve it befor **Before any edit**, confirm the file path does NOT contain `/cache/` or `/plugins/cache/`. -## Skill Creation Process +--- + +## Skill Creation Process (Step-by-Step) -To create a skill, follow the "Skill Creation Process" in order, skipping steps only if there is a clear reason why they are not applicable. +When creating or updating a skill, follow these steps in order. Skip steps only when clearly not applicable. ### Step 1: Understanding the Skill with Concrete Examples -Skip this step only when the skill's usage patterns are already clearly understood. It remains valuable even when working with an existing skill. +Skip this step only when the skill's usage patterns are already clearly understood. To create an effective skill, clearly understand concrete examples of how the skill will be used. This understanding can come from either direct user examples or generated examples that are validated with user feedback. @@ -279,151 +641,53 @@ For example, when building an image-editor skill, relevant questions include: - "What functionality should the image-editor skill support? Editing, rotating, anything else?" - "Can you give some examples of how this skill would be used?" -- "I can imagine users asking for things like 'Remove the red-eye from this image' or 'Rotate this image'. Are there other ways you imagine this skill being used?" - "What would a user say that should trigger this skill?" -To avoid overwhelming users, avoid asking too many questions in a single message. Start with the most important questions and follow up as needed for better effectiveness. - -Conclude this step when there is a clear sense of the functionality the skill should support. +To avoid overwhelming users, avoid asking too many questions in a single message. ### Step 2: Planning the Reusable Skill Contents -To turn concrete examples into an effective skill, analyze each example by: +Analyze each example by: 1. Considering how to execute on the example from scratch 2. Determining the appropriate level of freedom for Claude 3. Identifying what scripts, references, and assets would be helpful when executing these workflows repeatedly **Match specificity to task risk:** -- **High freedom (text instructions)**: Multiple valid approaches exist; context determines best path (e.g., code reviews, troubleshooting, content analysis) -- **Medium freedom (pseudocode with parameters)**: Preferred patterns exist with acceptable variation (e.g., API integration patterns, data processing workflows) -- **Low freedom (exact scripts)**: Operations are fragile, consistency critical, sequence matters (e.g., PDF rotation, database migrations, form validation) - -Example: When building a `pdf-editor` skill to handle queries like "Help me rotate this PDF," the analysis shows: - -1. Rotating a PDF requires re-writing the same code each time -2. A `scripts/rotate_pdf.py` script would be helpful to store in the skill - -Example: When designing a `frontend-webapp-builder` skill for queries like "Build me a todo app" or "Build me a dashboard to track my steps," the analysis shows: - -1. Writing a frontend webapp requires the same boilerplate HTML/React each time -2. An `assets/hello-world/` template containing the boilerplate HTML/React project files would be helpful to store in the skill - -Example: When building a `big-query` skill to handle queries like "How many users have logged in today?" the analysis shows: - -1. Querying BigQuery requires re-discovering the table schemas and relationships each time -2. A `references/schema.md` file documenting the table schemas would be helpful to store in the skill - -To establish the skill's contents, analyze each concrete example to create a list of the reusable resources to include: scripts, references, and assets. +- **High freedom (text instructions)**: Multiple valid approaches exist +- **Medium freedom (pseudocode with parameters)**: Preferred patterns exist with acceptable variation +- **Low freedom (exact scripts)**: Operations are fragile, consistency critical ### Step 3: Initializing the Skill -At this point, it is time to actually create the skill. - -Skip this step only if the skill being developed already exists, and iteration or packaging is needed. In this case, continue to the next step. +Skip this step if the skill already exists. -When creating a new skill from scratch, always run the `init_skill.py` script. The script conveniently generates a new template skill directory that automatically includes everything a skill requires, making the skill creation process much more efficient and reliable. - -Usage: +When creating a new skill from scratch, always run the `init_skill.py` script: ```bash scripts/init_skill.py --path ``` -The script: - -- Creates the skill directory at the specified path -- Generates a SKILL.md template with proper frontmatter and TODO placeholders -- Creates example resource directories: `scripts/`, `references/`, and `assets/` -- Adds example files in each directory that can be customized or deleted - -After initialization, customize or remove the generated SKILL.md and example files as needed. +The script creates a template skill directory with proper frontmatter, resource directories, and example files. ### Step 4: Edit the Skill -When editing the (newly-generated or existing) skill, remember that the skill is being created for another instance of Claude to use. Focus on including information that would be beneficial and non-obvious to Claude. Consider what procedural knowledge, domain-specific details, or reusable assets would help another Claude instance execute these tasks more effectively. - -#### Start with Reusable Skill Contents - -To begin implementation, start with the reusable resources identified above: `scripts/`, `references/`, and `assets/` files. Note that this step may require user input. For example, when implementing a `brand-guidelines` skill, the user may need to provide brand assets or templates to store in `assets/`, or documentation to store in `references/`. - -Also, delete any example files and directories not needed for the skill. The initialization script creates example files in `scripts/`, `references/`, and `assets/` to demonstrate structure, but most skills won't need all of them. - -**When updating an existing skill**: Scan all existing reference files to check if they need corresponding updates. New features often require updates to architecture, workflow, or other existing documentation to maintain consistency. +When editing, remember that the skill is being created for another instance of Claude to use. Focus on information that would be beneficial and non-obvious to Claude. -#### Reference File Naming - -Filenames must be self-explanatory without reading contents. - -**Pattern**: `_.md` - -**Examples**: -- ❌ `commands.md`, `cli_usage.md`, `reference.md` -- ✅ `script_parameters.md`, `api_endpoints.md`, `database_schema.md` - -**Test**: Can someone understand the file's contents from the name alone? - -#### Update SKILL.md - -**Writing Style:** Write the entire skill using **imperative/infinitive form** (verb-first instructions), not second person. Use objective, instructional language (e.g., "To accomplish X, do Y" rather than "You should do X" or "If you need to do X"). This maintains consistency and clarity for AI consumption. - -To complete SKILL.md, answer the following questions: - -1. What is the purpose of the skill, in a few sentences? -2. When should the skill be used? -3. In practice, how should Claude use the skill? All reusable skill contents developed above should be referenced so that Claude knows how to use them. +**When updating an existing skill**: Scan all existing reference files to check if they need corresponding updates. ### Step 5: Sanitization Review (Optional) **Ask the user before executing this step:** "This skill appears to be extracted from a business project. Would you like me to perform a sanitization review to remove business-specific content before public distribution?" -Skip this step if: -- The skill was created from scratch for public use -- The user explicitly declines sanitization -- The skill is intended for internal/private use only - -**When to perform sanitization:** -- Skill was extracted from a business/internal project -- Skill contains domain-specific examples from real systems -- Skill will be distributed publicly or to other teams +Skip if: skill was created from scratch for public use, user declines, or skill is for internal use. **Sanitization process:** 1. **Load the checklist**: Read [references/sanitization_checklist.md](references/sanitization_checklist.md) for detailed guidance - -2. **Run automated scans** to identify potential sensitive content: - ```bash - # Product/project names, person names, paths - grep -rniE "portal|underwriting|mercury|glean|/Users/|/home/" skill-folder/ - - # Chinese characters (if skill should be English-only) - grep -rn '[一-龥]' skill-folder/ - ``` - -3. **Review and replace** each category: - - Product/project names → generic terms - - Person names → "Alice", "Bob", role-based references - - Entity names → generic entities (ORDER, USER, PRODUCT) - - Folder structures → generic paths - - Internal jargon → industry-standard terms - - Language-specific content → translate or remove - -4. **Verify completeness**: - - Re-run all grep patterns (should return no matches) - - Read through skill to ensure coherence - - Confirm skill still functions correctly - -**Common replacements:** - -| Business-Specific | Generic Replacement | -|-------------------|---------------------| -| "Mercury Prepared" | "the project" | -| "Reviewer Portal" | "the application" | -| "Oliver will handle..." | "Alice will handle..." | -| `REVIEW_RESULT` | `ORDER` | -| `risk_level` | `status` | -| "ultrathink" | "deep review" | -| "后面再说" | "defer to later" | +2. **Run automated scans** to identify potential sensitive content +3. **Review and replace** each category (product names, person names, entity names, paths, jargon) +4. **Verify completeness**: Re-run patterns, read through skill, confirm functionality ### Step 6: Security Review @@ -458,22 +722,15 @@ brew install gitleaks - `3` - gitleaks not installed - `4` - Scan error -**Remediation for detected secrets:** - -1. Remove hardcoded secrets from all files -2. Use environment variables: `os.environ.get("API_KEY")` -3. Rotate credentials if previously committed to git -4. Re-run scan to verify fixes before packaging - ### Step 7: Packaging a Skill -Once the skill is ready, it should be packaged into a distributable zip file that gets shared with the user. The packaging process automatically validates the skill first to ensure it meets all requirements: +Once the skill is ready, package it into a distributable file: ```bash scripts/package_skill.py ``` -Optional output directory specification: +Optional output directory: ```bash scripts/package_skill.py ./dist @@ -481,17 +738,11 @@ scripts/package_skill.py ./dist The packaging script will: -1. **Validate** the skill automatically, checking: - - YAML frontmatter format and required fields - - Skill naming conventions and directory structure - - Description completeness and quality - - **Path reference integrity** - all `scripts/`, `references/`, and `assets/` paths mentioned in SKILL.md must exist - -2. **Package** the skill if validation passes, creating a zip file named after the skill (e.g., `my-skill.zip`) that includes all files and maintains the proper directory structure for distribution. +1. **Validate** the skill automatically (YAML frontmatter, naming conventions, path reference integrity) +2. **Verify security scan** (content hash must match last scan) +3. **Package** the skill into a distributable archive -**Common validation failure:** If SKILL.md references `scripts/my_script.py` but the file doesn't exist, validation will fail with "Missing referenced files: scripts/my_script.py". Ensure all bundled resources exist before packaging. - -If validation fails, the script will report the errors and exit without creating a package. Fix any validation errors and run the packaging command again. +If validation fails, the script reports errors and exits without creating a package. ### Step 8: Update Marketplace @@ -512,21 +763,86 @@ After packaging, update the marketplace registry to include the new or updated s } ``` -**For updated skills**, bump the version in `plugins[].version` following semver: -- Patch (1.0.x): Bug fixes, typo corrections -- Minor (1.x.0): New features, additional references -- Major (x.0.0): Breaking changes, restructured workflows - -**Also update** `metadata.version` and `metadata.description` if the overall plugin collection changed significantly. +**For updated skills**, bump the version in `plugins[].version` following semver. ### Step 9: Iterate After testing the skill, users may request improvements. Often this happens right after using the skill, with fresh context of how the skill performed. -**Iteration workflow:** -1. Use the skill on real tasks -2. Notice struggles or inefficiencies -3. Identify how SKILL.md or bundled resources should be updated -4. Implement changes and test again - **Refinement filter:** Only add what solves observed problems. If best practices already cover it, don't duplicate. + +--- + +### Package and Present (only if `present_files` tool is available) + +Check whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user: + +```bash +python -m scripts.package_skill +``` + +After packaging, direct the user to the resulting `.skill` file path so they can install it. + +--- + +## Claude.ai-specific instructions + +In Claude.ai, the core workflow is the same (draft -> test -> review -> improve -> repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt: + +**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested. + +**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: "How does this look? Anything you'd change?" + +**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user. + +**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one. + +**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai. + +**Blind comparison**: Requires subagents. Skip it. + +**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file. + +--- + +## Cowork-Specific Instructions + +If you're in Cowork, the main things to know are: + +- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.) +- You don't have a browser or display, so when generating the eval viewer, use `--static ` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser. +- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER *BEFORE* evaluating inputs yourself. You want to get them in front of the human ASAP! +- Feedback works differently: since there's no running server, the viewer's "Submit All Reviews" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first). +- Packaging works — `package_skill.py` just needs Python and a filesystem. +- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape. + +--- + +## Reference files + +The agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent. + +- `agents/grader.md` — How to evaluate assertions against outputs +- `agents/comparator.md` — How to do blind A/B comparison between two outputs +- `agents/analyzer.md` — How to analyze why one version beat another + +The references/ directory has additional documentation: +- `references/schemas.md` — JSON structures for evals.json, grading.json, benchmark.json, etc. +- `references/sanitization_checklist.md` — Checklist for sanitizing business-specific content before public distribution + +--- + +Repeating one more time the core loop here for emphasis: + +- Figure out what the skill is about +- Draft or edit the skill +- Run claude-with-access-to-the-skill on test prompts +- With the user, evaluate the outputs: + - Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them + - Run quantitative evals +- Repeat until you and the user are satisfied +- Package the final skill and return it to the user. + +Please add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put "Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases" in your TodoList to make sure it happens. + +Good luck! diff --git a/skill-creator/agents/analyzer.md b/skill-creator/agents/analyzer.md new file mode 100644 index 00000000..14e41d60 --- /dev/null +++ b/skill-creator/agents/analyzer.md @@ -0,0 +1,274 @@ +# Post-hoc Analyzer Agent + +Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions. + +## Role + +After the blind comparator determines a winner, the Post-hoc Analyzer "unblids" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved? + +## Inputs + +You receive these parameters in your prompt: + +- **winner**: "A" or "B" (from blind comparison) +- **winner_skill_path**: Path to the skill that produced the winning output +- **winner_transcript_path**: Path to the execution transcript for the winner +- **loser_skill_path**: Path to the skill that produced the losing output +- **loser_transcript_path**: Path to the execution transcript for the loser +- **comparison_result_path**: Path to the blind comparator's output JSON +- **output_path**: Where to save the analysis results + +## Process + +### Step 1: Read Comparison Result + +1. Read the blind comparator's output at comparison_result_path +2. Note the winning side (A or B), the reasoning, and any scores +3. Understand what the comparator valued in the winning output + +### Step 2: Read Both Skills + +1. Read the winner skill's SKILL.md and key referenced files +2. Read the loser skill's SKILL.md and key referenced files +3. Identify structural differences: + - Instructions clarity and specificity + - Script/tool usage patterns + - Example coverage + - Edge case handling + +### Step 3: Read Both Transcripts + +1. Read the winner's transcript +2. Read the loser's transcript +3. Compare execution patterns: + - How closely did each follow their skill's instructions? + - What tools were used differently? + - Where did the loser diverge from optimal behavior? + - Did either encounter errors or make recovery attempts? + +### Step 4: Analyze Instruction Following + +For each transcript, evaluate: +- Did the agent follow the skill's explicit instructions? +- Did the agent use the skill's provided tools/scripts? +- Were there missed opportunities to leverage skill content? +- Did the agent add unnecessary steps not in the skill? + +Score instruction following 1-10 and note specific issues. + +### Step 5: Identify Winner Strengths + +Determine what made the winner better: +- Clearer instructions that led to better behavior? +- Better scripts/tools that produced better output? +- More comprehensive examples that guided edge cases? +- Better error handling guidance? + +Be specific. Quote from skills/transcripts where relevant. + +### Step 6: Identify Loser Weaknesses + +Determine what held the loser back: +- Ambiguous instructions that led to suboptimal choices? +- Missing tools/scripts that forced workarounds? +- Gaps in edge case coverage? +- Poor error handling that caused failures? + +### Step 7: Generate Improvement Suggestions + +Based on the analysis, produce actionable suggestions for improving the loser skill: +- Specific instruction changes to make +- Tools/scripts to add or modify +- Examples to include +- Edge cases to address + +Prioritize by impact. Focus on changes that would have changed the outcome. + +### Step 8: Write Analysis Results + +Save structured analysis to `{output_path}`. + +## Output Format + +Write a JSON file with this structure: + +```json +{ + "comparison_summary": { + "winner": "A", + "winner_skill": "path/to/winner/skill", + "loser_skill": "path/to/loser/skill", + "comparator_reasoning": "Brief summary of why comparator chose winner" + }, + "winner_strengths": [ + "Clear step-by-step instructions for handling multi-page documents", + "Included validation script that caught formatting errors", + "Explicit guidance on fallback behavior when OCR fails" + ], + "loser_weaknesses": [ + "Vague instruction 'process the document appropriately' led to inconsistent behavior", + "No script for validation, agent had to improvise and made errors", + "No guidance on OCR failure, agent gave up instead of trying alternatives" + ], + "instruction_following": { + "winner": { + "score": 9, + "issues": [ + "Minor: skipped optional logging step" + ] + }, + "loser": { + "score": 6, + "issues": [ + "Did not use the skill's formatting template", + "Invented own approach instead of following step 3", + "Missed the 'always validate output' instruction" + ] + } + }, + "improvement_suggestions": [ + { + "priority": "high", + "category": "instructions", + "suggestion": "Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template", + "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior" + }, + { + "priority": "high", + "category": "tools", + "suggestion": "Add validate_output.py script similar to winner skill's validation approach", + "expected_impact": "Would catch formatting errors before final output" + }, + { + "priority": "medium", + "category": "error_handling", + "suggestion": "Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'", + "expected_impact": "Would prevent early failure on difficult documents" + } + ], + "transcript_insights": { + "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output", + "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors" + } +} +``` + +## Guidelines + +- **Be specific**: Quote from skills and transcripts, don't just say "instructions were unclear" +- **Be actionable**: Suggestions should be concrete changes, not vague advice +- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent +- **Prioritize by impact**: Which changes would most likely have changed the outcome? +- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental? +- **Stay objective**: Analyze what happened, don't editorialize +- **Think about generalization**: Would this improvement help on other evals too? + +## Categories for Suggestions + +Use these categories to organize improvement suggestions: + +| Category | Description | +|----------|-------------| +| `instructions` | Changes to the skill's prose instructions | +| `tools` | Scripts, templates, or utilities to add/modify | +| `examples` | Example inputs/outputs to include | +| `error_handling` | Guidance for handling failures | +| `structure` | Reorganization of skill content | +| `references` | External docs or resources to add | + +## Priority Levels + +- **high**: Would likely change the outcome of this comparison +- **medium**: Would improve quality but may not change win/loss +- **low**: Nice to have, marginal improvement + +--- + +# Analyzing Benchmark Results + +When analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements. + +## Role + +Review all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone. + +## Inputs + +You receive these parameters in your prompt: + +- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results +- **skill_path**: Path to the skill being benchmarked +- **output_path**: Where to save the notes (as JSON array of strings) + +## Process + +### Step 1: Read Benchmark Data + +1. Read the benchmark.json containing all run results +2. Note the configurations tested (with_skill, without_skill) +3. Understand the run_summary aggregates already calculated + +### Step 2: Analyze Per-Assertion Patterns + +For each expectation across all runs: +- Does it **always pass** in both configurations? (may not differentiate skill value) +- Does it **always fail** in both configurations? (may be broken or beyond capability) +- Does it **always pass with skill but fail without**? (skill clearly adds value here) +- Does it **always fail with skill but pass without**? (skill may be hurting) +- Is it **highly variable**? (flaky expectation or non-deterministic behavior) + +### Step 3: Analyze Cross-Eval Patterns + +Look for patterns across evals: +- Are certain eval types consistently harder/easier? +- Do some evals show high variance while others are stable? +- Are there surprising results that contradict expectations? + +### Step 4: Analyze Metrics Patterns + +Look at time_seconds, tokens, tool_calls: +- Does the skill significantly increase execution time? +- Is there high variance in resource usage? +- Are there outlier runs that skew the aggregates? + +### Step 5: Generate Notes + +Write freeform observations as a list of strings. Each note should: +- State a specific observation +- Be grounded in the data (not speculation) +- Help the user understand something the aggregate metrics don't show + +Examples: +- "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value" +- "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky" +- "Without-skill runs consistently fail on table extraction expectations (0% pass rate)" +- "Skill adds 13s average execution time but improves pass rate by 50%" +- "Token usage is 80% higher with skill, primarily due to script output parsing" +- "All 3 without-skill runs for eval 1 produced empty output" + +### Step 6: Write Notes + +Save notes to `{output_path}` as a JSON array of strings: + +```json +[ + "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value", + "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure", + "Without-skill runs consistently fail on table extraction expectations", + "Skill adds 13s average execution time but improves pass rate by 50%" +] +``` + +## Guidelines + +**DO:** +- Report what you observe in the data +- Be specific about which evals, expectations, or runs you're referring to +- Note patterns that aggregate metrics would hide +- Provide context that helps interpret the numbers + +**DO NOT:** +- Suggest improvements to the skill (that's for the improvement step, not benchmarking) +- Make subjective quality judgments ("the output was good/bad") +- Speculate about causes without evidence +- Repeat information already in the run_summary aggregates diff --git a/skill-creator/agents/comparator.md b/skill-creator/agents/comparator.md new file mode 100644 index 00000000..80e00eb4 --- /dev/null +++ b/skill-creator/agents/comparator.md @@ -0,0 +1,202 @@ +# Blind Comparator Agent + +Compare two outputs WITHOUT knowing which skill produced them. + +## Role + +The Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach. + +Your judgment is based purely on output quality and task completion. + +## Inputs + +You receive these parameters in your prompt: + +- **output_a_path**: Path to the first output file or directory +- **output_b_path**: Path to the second output file or directory +- **eval_prompt**: The original task/prompt that was executed +- **expectations**: List of expectations to check (optional - may be empty) + +## Process + +### Step 1: Read Both Outputs + +1. Examine output A (file or directory) +2. Examine output B (file or directory) +3. Note the type, structure, and content of each +4. If outputs are directories, examine all relevant files inside + +### Step 2: Understand the Task + +1. Read the eval_prompt carefully +2. Identify what the task requires: + - What should be produced? + - What qualities matter (accuracy, completeness, format)? + - What would distinguish a good output from a poor one? + +### Step 3: Generate Evaluation Rubric + +Based on the task, generate a rubric with two dimensions: + +**Content Rubric** (what the output contains): +| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) | +|-----------|----------|----------------|---------------| +| Correctness | Major errors | Minor errors | Fully correct | +| Completeness | Missing key elements | Mostly complete | All elements present | +| Accuracy | Significant inaccuracies | Minor inaccuracies | Accurate throughout | + +**Structure Rubric** (how the output is organized): +| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) | +|-----------|----------|----------------|---------------| +| Organization | Disorganized | Reasonably organized | Clear, logical structure | +| Formatting | Inconsistent/broken | Mostly consistent | Professional, polished | +| Usability | Difficult to use | Usable with effort | Easy to use | + +Adapt criteria to the specific task. For example: +- PDF form → "Field alignment", "Text readability", "Data placement" +- Document → "Section structure", "Heading hierarchy", "Paragraph flow" +- Data output → "Schema correctness", "Data types", "Completeness" + +### Step 4: Evaluate Each Output Against the Rubric + +For each output (A and B): + +1. **Score each criterion** on the rubric (1-5 scale) +2. **Calculate dimension totals**: Content score, Structure score +3. **Calculate overall score**: Average of dimension scores, scaled to 1-10 + +### Step 5: Check Assertions (if provided) + +If expectations are provided: + +1. Check each expectation against output A +2. Check each expectation against output B +3. Count pass rates for each output +4. Use expectation scores as secondary evidence (not the primary decision factor) + +### Step 6: Determine the Winner + +Compare A and B based on (in priority order): + +1. **Primary**: Overall rubric score (content + structure) +2. **Secondary**: Assertion pass rates (if applicable) +3. **Tiebreaker**: If truly equal, declare a TIE + +Be decisive - ties should be rare. One output is usually better, even if marginally. + +### Step 7: Write Comparison Results + +Save results to a JSON file at the path specified (or `comparison.json` if not specified). + +## Output Format + +Write a JSON file with this structure: + +```json +{ + "winner": "A", + "reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.", + "rubric": { + "A": { + "content": { + "correctness": 5, + "completeness": 5, + "accuracy": 4 + }, + "structure": { + "organization": 4, + "formatting": 5, + "usability": 4 + }, + "content_score": 4.7, + "structure_score": 4.3, + "overall_score": 9.0 + }, + "B": { + "content": { + "correctness": 3, + "completeness": 2, + "accuracy": 3 + }, + "structure": { + "organization": 3, + "formatting": 2, + "usability": 3 + }, + "content_score": 2.7, + "structure_score": 2.7, + "overall_score": 5.4 + } + }, + "output_quality": { + "A": { + "score": 9, + "strengths": ["Complete solution", "Well-formatted", "All fields present"], + "weaknesses": ["Minor style inconsistency in header"] + }, + "B": { + "score": 5, + "strengths": ["Readable output", "Correct basic structure"], + "weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"] + } + }, + "expectation_results": { + "A": { + "passed": 4, + "total": 5, + "pass_rate": 0.80, + "details": [ + {"text": "Output includes name", "passed": true}, + {"text": "Output includes date", "passed": true}, + {"text": "Format is PDF", "passed": true}, + {"text": "Contains signature", "passed": false}, + {"text": "Readable text", "passed": true} + ] + }, + "B": { + "passed": 3, + "total": 5, + "pass_rate": 0.60, + "details": [ + {"text": "Output includes name", "passed": true}, + {"text": "Output includes date", "passed": false}, + {"text": "Format is PDF", "passed": true}, + {"text": "Contains signature", "passed": false}, + {"text": "Readable text", "passed": true} + ] + } + } +} +``` + +If no expectations were provided, omit the `expectation_results` field entirely. + +## Field Descriptions + +- **winner**: "A", "B", or "TIE" +- **reasoning**: Clear explanation of why the winner was chosen (or why it's a tie) +- **rubric**: Structured rubric evaluation for each output + - **content**: Scores for content criteria (correctness, completeness, accuracy) + - **structure**: Scores for structure criteria (organization, formatting, usability) + - **content_score**: Average of content criteria (1-5) + - **structure_score**: Average of structure criteria (1-5) + - **overall_score**: Combined score scaled to 1-10 +- **output_quality**: Summary quality assessment + - **score**: 1-10 rating (should match rubric overall_score) + - **strengths**: List of positive aspects + - **weaknesses**: List of issues or shortcomings +- **expectation_results**: (Only if expectations provided) + - **passed**: Number of expectations that passed + - **total**: Total number of expectations + - **pass_rate**: Fraction passed (0.0 to 1.0) + - **details**: Individual expectation results + +## Guidelines + +- **Stay blind**: DO NOT try to infer which skill produced which output. Judge purely on output quality. +- **Be specific**: Cite specific examples when explaining strengths and weaknesses. +- **Be decisive**: Choose a winner unless outputs are genuinely equivalent. +- **Output quality first**: Assertion scores are secondary to overall task completion. +- **Be objective**: Don't favor outputs based on style preferences; focus on correctness and completeness. +- **Explain your reasoning**: The reasoning field should make it clear why you chose the winner. +- **Handle edge cases**: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better. diff --git a/skill-creator/agents/grader.md b/skill-creator/agents/grader.md new file mode 100644 index 00000000..558ab05c --- /dev/null +++ b/skill-creator/agents/grader.md @@ -0,0 +1,223 @@ +# Grader Agent + +Evaluate expectations against an execution transcript and outputs. + +## Role + +The Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment. + +You have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so. + +## Inputs + +You receive these parameters in your prompt: + +- **expectations**: List of expectations to evaluate (strings) +- **transcript_path**: Path to the execution transcript (markdown file) +- **outputs_dir**: Directory containing output files from execution + +## Process + +### Step 1: Read the Transcript + +1. Read the transcript file completely +2. Note the eval prompt, execution steps, and final result +3. Identify any issues or errors documented + +### Step 2: Examine Output Files + +1. List files in outputs_dir +2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced. +3. Note contents, structure, and quality + +### Step 3: Evaluate Each Assertion + +For each expectation: + +1. **Search for evidence** in the transcript and outputs +2. **Determine verdict**: + - **PASS**: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance + - **FAIL**: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content) +3. **Cite the evidence**: Quote the specific text or describe what you found + +### Step 4: Extract and Verify Claims + +Beyond the predefined expectations, extract implicit claims from the outputs and verify them: + +1. **Extract claims** from the transcript and outputs: + - Factual statements ("The form has 12 fields") + - Process claims ("Used pypdf to fill the form") + - Quality claims ("All fields were filled correctly") + +2. **Verify each claim**: + - **Factual claims**: Can be checked against the outputs or external sources + - **Process claims**: Can be verified from the transcript + - **Quality claims**: Evaluate whether the claim is justified + +3. **Flag unverifiable claims**: Note claims that cannot be verified with available information + +This catches issues that predefined expectations might miss. + +### Step 5: Read User Notes + +If `{outputs_dir}/user_notes.md` exists: +1. Read it and note any uncertainties or issues flagged by the executor +2. Include relevant concerns in the grading output +3. These may reveal problems even when expectations pass + +### Step 6: Critique the Evals + +After grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap. + +Good suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion *discriminating*: it passes when the skill genuinely succeeds and fails when it doesn't. + +Suggestions worth raising: +- An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content) +- An important outcome you observed — good or bad — that no assertion covers at all +- An assertion that can't actually be verified from the available outputs + +Keep the bar high. The goal is to flag things the eval author would say "good catch" about, not to nitpick every assertion. + +### Step 7: Write Grading Results + +Save results to `{outputs_dir}/../grading.json` (sibling to outputs_dir). + +## Grading Criteria + +**PASS when**: +- The transcript or outputs clearly demonstrate the expectation is true +- Specific evidence can be cited +- The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename) + +**FAIL when**: +- No evidence found for the expectation +- Evidence contradicts the expectation +- The expectation cannot be verified from available information +- The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete +- The output appears to meet the assertion by coincidence rather than by actually doing the work + +**When uncertain**: The burden of proof to pass is on the expectation. + +### Step 8: Read Executor Metrics and Timing + +1. If `{outputs_dir}/metrics.json` exists, read it and include in grading output +2. If `{outputs_dir}/../timing.json` exists, read it and include timing data + +## Output Format + +Write a JSON file with this structure: + +```json +{ + "expectations": [ + { + "text": "The output includes the name 'John Smith'", + "passed": true, + "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'" + }, + { + "text": "The spreadsheet has a SUM formula in cell B10", + "passed": false, + "evidence": "No spreadsheet was created. The output was a text file." + }, + { + "text": "The assistant used the skill's OCR script", + "passed": true, + "evidence": "Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'" + } + ], + "summary": { + "passed": 2, + "failed": 1, + "total": 3, + "pass_rate": 0.67 + }, + "execution_metrics": { + "tool_calls": { + "Read": 5, + "Write": 2, + "Bash": 8 + }, + "total_tool_calls": 15, + "total_steps": 6, + "errors_encountered": 0, + "output_chars": 12450, + "transcript_chars": 3200 + }, + "timing": { + "executor_duration_seconds": 165.0, + "grader_duration_seconds": 26.0, + "total_duration_seconds": 191.0 + }, + "claims": [ + { + "claim": "The form has 12 fillable fields", + "type": "factual", + "verified": true, + "evidence": "Counted 12 fields in field_info.json" + }, + { + "claim": "All required fields were populated", + "type": "quality", + "verified": false, + "evidence": "Reference section was left blank despite data being available" + } + ], + "user_notes_summary": { + "uncertainties": ["Used 2023 data, may be stale"], + "needs_review": [], + "workarounds": ["Fell back to text overlay for non-fillable fields"] + }, + "eval_feedback": { + "suggestions": [ + { + "assertion": "The output includes the name 'John Smith'", + "reason": "A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input" + }, + { + "reason": "No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught" + } + ], + "overall": "Assertions check presence but not correctness. Consider adding content verification." + } +} +``` + +## Field Descriptions + +- **expectations**: Array of graded expectations + - **text**: The original expectation text + - **passed**: Boolean - true if expectation passes + - **evidence**: Specific quote or description supporting the verdict +- **summary**: Aggregate statistics + - **passed**: Count of passed expectations + - **failed**: Count of failed expectations + - **total**: Total expectations evaluated + - **pass_rate**: Fraction passed (0.0 to 1.0) +- **execution_metrics**: Copied from executor's metrics.json (if available) + - **output_chars**: Total character count of output files (proxy for tokens) + - **transcript_chars**: Character count of transcript +- **timing**: Wall clock timing from timing.json (if available) + - **executor_duration_seconds**: Time spent in executor subagent + - **total_duration_seconds**: Total elapsed time for the run +- **claims**: Extracted and verified claims from the output + - **claim**: The statement being verified + - **type**: "factual", "process", or "quality" + - **verified**: Boolean - whether the claim holds + - **evidence**: Supporting or contradicting evidence +- **user_notes_summary**: Issues flagged by the executor + - **uncertainties**: Things the executor wasn't sure about + - **needs_review**: Items requiring human attention + - **workarounds**: Places where the skill didn't work as expected +- **eval_feedback**: Improvement suggestions for the evals (only when warranted) + - **suggestions**: List of concrete suggestions, each with a `reason` and optionally an `assertion` it relates to + - **overall**: Brief assessment — can be "No suggestions, evals look solid" if nothing to flag + +## Guidelines + +- **Be objective**: Base verdicts on evidence, not assumptions +- **Be specific**: Quote the exact text that supports your verdict +- **Be thorough**: Check both transcript and output files +- **Be consistent**: Apply the same standard to each expectation +- **Explain failures**: Make it clear why evidence was insufficient +- **No partial credit**: Each expectation is pass or fail, not partial diff --git a/skill-creator/assets/eval_review.html b/skill-creator/assets/eval_review.html new file mode 100644 index 00000000..938ff32a --- /dev/null +++ b/skill-creator/assets/eval_review.html @@ -0,0 +1,146 @@ + + + + + + Eval Set Review - __SKILL_NAME_PLACEHOLDER__ + + + + + + +

Eval Set Review: __SKILL_NAME_PLACEHOLDER__

+

Current description: __SKILL_DESCRIPTION_PLACEHOLDER__

+ +
+ + +
+ + + + + + + + + + +
QueryShould TriggerActions
+ +

+ + + + diff --git a/skill-creator/eval-viewer/generate_review.py b/skill-creator/eval-viewer/generate_review.py new file mode 100644 index 00000000..7fa59786 --- /dev/null +++ b/skill-creator/eval-viewer/generate_review.py @@ -0,0 +1,471 @@ +#!/usr/bin/env python3 +"""Generate and serve a review page for eval results. + +Reads the workspace directory, discovers runs (directories with outputs/), +embeds all output data into a self-contained HTML page, and serves it via +a tiny HTTP server. Feedback auto-saves to feedback.json in the workspace. + +Usage: + python generate_review.py [--port PORT] [--skill-name NAME] + python generate_review.py --previous-feedback /path/to/old/feedback.json + +No dependencies beyond the Python stdlib are required. +""" + +import argparse +import base64 +import json +import mimetypes +import os +import re +import signal +import subprocess +import sys +import time +import webbrowser +from functools import partial +from http.server import HTTPServer, BaseHTTPRequestHandler +from pathlib import Path + +# Files to exclude from output listings +METADATA_FILES = {"transcript.md", "user_notes.md", "metrics.json"} + +# Extensions we render as inline text +TEXT_EXTENSIONS = { + ".txt", ".md", ".json", ".csv", ".py", ".js", ".ts", ".tsx", ".jsx", + ".yaml", ".yml", ".xml", ".html", ".css", ".sh", ".rb", ".go", ".rs", + ".java", ".c", ".cpp", ".h", ".hpp", ".sql", ".r", ".toml", +} + +# Extensions we render as inline images +IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp"} + +# MIME type overrides for common types +MIME_OVERRIDES = { + ".svg": "image/svg+xml", + ".xlsx": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + ".pptx": "application/vnd.openxmlformats-officedocument.presentationml.presentation", +} + + +def get_mime_type(path: Path) -> str: + ext = path.suffix.lower() + if ext in MIME_OVERRIDES: + return MIME_OVERRIDES[ext] + mime, _ = mimetypes.guess_type(str(path)) + return mime or "application/octet-stream" + + +def find_runs(workspace: Path) -> list[dict]: + """Recursively find directories that contain an outputs/ subdirectory.""" + runs: list[dict] = [] + _find_runs_recursive(workspace, workspace, runs) + runs.sort(key=lambda r: (r.get("eval_id", float("inf")), r["id"])) + return runs + + +def _find_runs_recursive(root: Path, current: Path, runs: list[dict]) -> None: + if not current.is_dir(): + return + + outputs_dir = current / "outputs" + if outputs_dir.is_dir(): + run = build_run(root, current) + if run: + runs.append(run) + return + + skip = {"node_modules", ".git", "__pycache__", "skill", "inputs"} + for child in sorted(current.iterdir()): + if child.is_dir() and child.name not in skip: + _find_runs_recursive(root, child, runs) + + +def build_run(root: Path, run_dir: Path) -> dict | None: + """Build a run dict with prompt, outputs, and grading data.""" + prompt = "" + eval_id = None + + # Try eval_metadata.json + for candidate in [run_dir / "eval_metadata.json", run_dir.parent / "eval_metadata.json"]: + if candidate.exists(): + try: + metadata = json.loads(candidate.read_text()) + prompt = metadata.get("prompt", "") + eval_id = metadata.get("eval_id") + except (json.JSONDecodeError, OSError): + pass + if prompt: + break + + # Fall back to transcript.md + if not prompt: + for candidate in [run_dir / "transcript.md", run_dir / "outputs" / "transcript.md"]: + if candidate.exists(): + try: + text = candidate.read_text() + match = re.search(r"## Eval Prompt\n\n([\s\S]*?)(?=\n##|$)", text) + if match: + prompt = match.group(1).strip() + except OSError: + pass + if prompt: + break + + if not prompt: + prompt = "(No prompt found)" + + run_id = str(run_dir.relative_to(root)).replace("/", "-").replace("\\", "-") + + # Collect output files + outputs_dir = run_dir / "outputs" + output_files: list[dict] = [] + if outputs_dir.is_dir(): + for f in sorted(outputs_dir.iterdir()): + if f.is_file() and f.name not in METADATA_FILES: + output_files.append(embed_file(f)) + + # Load grading if present + grading = None + for candidate in [run_dir / "grading.json", run_dir.parent / "grading.json"]: + if candidate.exists(): + try: + grading = json.loads(candidate.read_text()) + except (json.JSONDecodeError, OSError): + pass + if grading: + break + + return { + "id": run_id, + "prompt": prompt, + "eval_id": eval_id, + "outputs": output_files, + "grading": grading, + } + + +def embed_file(path: Path) -> dict: + """Read a file and return an embedded representation.""" + ext = path.suffix.lower() + mime = get_mime_type(path) + + if ext in TEXT_EXTENSIONS: + try: + content = path.read_text(errors="replace") + except OSError: + content = "(Error reading file)" + return { + "name": path.name, + "type": "text", + "content": content, + } + elif ext in IMAGE_EXTENSIONS: + try: + raw = path.read_bytes() + b64 = base64.b64encode(raw).decode("ascii") + except OSError: + return {"name": path.name, "type": "error", "content": "(Error reading file)"} + return { + "name": path.name, + "type": "image", + "mime": mime, + "data_uri": f"data:{mime};base64,{b64}", + } + elif ext == ".pdf": + try: + raw = path.read_bytes() + b64 = base64.b64encode(raw).decode("ascii") + except OSError: + return {"name": path.name, "type": "error", "content": "(Error reading file)"} + return { + "name": path.name, + "type": "pdf", + "data_uri": f"data:{mime};base64,{b64}", + } + elif ext == ".xlsx": + try: + raw = path.read_bytes() + b64 = base64.b64encode(raw).decode("ascii") + except OSError: + return {"name": path.name, "type": "error", "content": "(Error reading file)"} + return { + "name": path.name, + "type": "xlsx", + "data_b64": b64, + } + else: + # Binary / unknown — base64 download link + try: + raw = path.read_bytes() + b64 = base64.b64encode(raw).decode("ascii") + except OSError: + return {"name": path.name, "type": "error", "content": "(Error reading file)"} + return { + "name": path.name, + "type": "binary", + "mime": mime, + "data_uri": f"data:{mime};base64,{b64}", + } + + +def load_previous_iteration(workspace: Path) -> dict[str, dict]: + """Load previous iteration's feedback and outputs. + + Returns a map of run_id -> {"feedback": str, "outputs": list[dict]}. + """ + result: dict[str, dict] = {} + + # Load feedback + feedback_map: dict[str, str] = {} + feedback_path = workspace / "feedback.json" + if feedback_path.exists(): + try: + data = json.loads(feedback_path.read_text()) + feedback_map = { + r["run_id"]: r["feedback"] + for r in data.get("reviews", []) + if r.get("feedback", "").strip() + } + except (json.JSONDecodeError, OSError, KeyError): + pass + + # Load runs (to get outputs) + prev_runs = find_runs(workspace) + for run in prev_runs: + result[run["id"]] = { + "feedback": feedback_map.get(run["id"], ""), + "outputs": run.get("outputs", []), + } + + # Also add feedback for run_ids that had feedback but no matching run + for run_id, fb in feedback_map.items(): + if run_id not in result: + result[run_id] = {"feedback": fb, "outputs": []} + + return result + + +def generate_html( + runs: list[dict], + skill_name: str, + previous: dict[str, dict] | None = None, + benchmark: dict | None = None, +) -> str: + """Generate the complete standalone HTML page with embedded data.""" + template_path = Path(__file__).parent / "viewer.html" + template = template_path.read_text() + + # Build previous_feedback and previous_outputs maps for the template + previous_feedback: dict[str, str] = {} + previous_outputs: dict[str, list[dict]] = {} + if previous: + for run_id, data in previous.items(): + if data.get("feedback"): + previous_feedback[run_id] = data["feedback"] + if data.get("outputs"): + previous_outputs[run_id] = data["outputs"] + + embedded = { + "skill_name": skill_name, + "runs": runs, + "previous_feedback": previous_feedback, + "previous_outputs": previous_outputs, + } + if benchmark: + embedded["benchmark"] = benchmark + + data_json = json.dumps(embedded) + + return template.replace("/*__EMBEDDED_DATA__*/", f"const EMBEDDED_DATA = {data_json};") + + +# --------------------------------------------------------------------------- +# HTTP server (stdlib only, zero dependencies) +# --------------------------------------------------------------------------- + +def _kill_port(port: int) -> None: + """Kill any process listening on the given port.""" + try: + result = subprocess.run( + ["lsof", "-ti", f":{port}"], + capture_output=True, text=True, timeout=5, + ) + for pid_str in result.stdout.strip().split("\n"): + if pid_str.strip(): + try: + os.kill(int(pid_str.strip()), signal.SIGTERM) + except (ProcessLookupError, ValueError): + pass + if result.stdout.strip(): + time.sleep(0.5) + except subprocess.TimeoutExpired: + pass + except FileNotFoundError: + print("Note: lsof not found, cannot check if port is in use", file=sys.stderr) + +class ReviewHandler(BaseHTTPRequestHandler): + """Serves the review HTML and handles feedback saves. + + Regenerates the HTML on each page load so that refreshing the browser + picks up new eval outputs without restarting the server. + """ + + def __init__( + self, + workspace: Path, + skill_name: str, + feedback_path: Path, + previous: dict[str, dict], + benchmark_path: Path | None, + *args, + **kwargs, + ): + self.workspace = workspace + self.skill_name = skill_name + self.feedback_path = feedback_path + self.previous = previous + self.benchmark_path = benchmark_path + super().__init__(*args, **kwargs) + + def do_GET(self) -> None: + if self.path == "/" or self.path == "/index.html": + # Regenerate HTML on each request (re-scans workspace for new outputs) + runs = find_runs(self.workspace) + benchmark = None + if self.benchmark_path and self.benchmark_path.exists(): + try: + benchmark = json.loads(self.benchmark_path.read_text()) + except (json.JSONDecodeError, OSError): + pass + html = generate_html(runs, self.skill_name, self.previous, benchmark) + content = html.encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(content))) + self.end_headers() + self.wfile.write(content) + elif self.path == "/api/feedback": + data = b"{}" + if self.feedback_path.exists(): + data = self.feedback_path.read_bytes() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + else: + self.send_error(404) + + def do_POST(self) -> None: + if self.path == "/api/feedback": + length = int(self.headers.get("Content-Length", 0)) + body = self.rfile.read(length) + try: + data = json.loads(body) + if not isinstance(data, dict) or "reviews" not in data: + raise ValueError("Expected JSON object with 'reviews' key") + self.feedback_path.write_text(json.dumps(data, indent=2) + "\n") + resp = b'{"ok":true}' + self.send_response(200) + except (json.JSONDecodeError, OSError, ValueError) as e: + resp = json.dumps({"error": str(e)}).encode() + self.send_response(500) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(resp))) + self.end_headers() + self.wfile.write(resp) + else: + self.send_error(404) + + def log_message(self, format: str, *args: object) -> None: + # Suppress request logging to keep terminal clean + pass + + +def main() -> None: + parser = argparse.ArgumentParser(description="Generate and serve eval review") + parser.add_argument("workspace", type=Path, help="Path to workspace directory") + parser.add_argument("--port", "-p", type=int, default=3117, help="Server port (default: 3117)") + parser.add_argument("--skill-name", "-n", type=str, default=None, help="Skill name for header") + parser.add_argument( + "--previous-workspace", type=Path, default=None, + help="Path to previous iteration's workspace (shows old outputs and feedback as context)", + ) + parser.add_argument( + "--benchmark", type=Path, default=None, + help="Path to benchmark.json to show in the Benchmark tab", + ) + parser.add_argument( + "--static", "-s", type=Path, default=None, + help="Write standalone HTML to this path instead of starting a server", + ) + args = parser.parse_args() + + workspace = args.workspace.resolve() + if not workspace.is_dir(): + print(f"Error: {workspace} is not a directory", file=sys.stderr) + sys.exit(1) + + runs = find_runs(workspace) + if not runs: + print(f"No runs found in {workspace}", file=sys.stderr) + sys.exit(1) + + skill_name = args.skill_name or workspace.name.replace("-workspace", "") + feedback_path = workspace / "feedback.json" + + previous: dict[str, dict] = {} + if args.previous_workspace: + previous = load_previous_iteration(args.previous_workspace.resolve()) + + benchmark_path = args.benchmark.resolve() if args.benchmark else None + benchmark = None + if benchmark_path and benchmark_path.exists(): + try: + benchmark = json.loads(benchmark_path.read_text()) + except (json.JSONDecodeError, OSError): + pass + + if args.static: + html = generate_html(runs, skill_name, previous, benchmark) + args.static.parent.mkdir(parents=True, exist_ok=True) + args.static.write_text(html) + print(f"\n Static viewer written to: {args.static}\n") + sys.exit(0) + + # Kill any existing process on the target port + port = args.port + _kill_port(port) + handler = partial(ReviewHandler, workspace, skill_name, feedback_path, previous, benchmark_path) + try: + server = HTTPServer(("127.0.0.1", port), handler) + except OSError: + # Port still in use after kill attempt — find a free one + server = HTTPServer(("127.0.0.1", 0), handler) + port = server.server_address[1] + + url = f"http://localhost:{port}" + print(f"\n Eval Viewer") + print(f" ─────────────────────────────────") + print(f" URL: {url}") + print(f" Workspace: {workspace}") + print(f" Feedback: {feedback_path}") + if previous: + print(f" Previous: {args.previous_workspace} ({len(previous)} runs)") + if benchmark_path: + print(f" Benchmark: {benchmark_path}") + print(f"\n Press Ctrl+C to stop.\n") + + webbrowser.open(url) + + try: + server.serve_forever() + except KeyboardInterrupt: + print("\nStopped.") + server.server_close() + + +if __name__ == "__main__": + main() diff --git a/skill-creator/eval-viewer/viewer.html b/skill-creator/eval-viewer/viewer.html new file mode 100644 index 00000000..6d8e9634 --- /dev/null +++ b/skill-creator/eval-viewer/viewer.html @@ -0,0 +1,1325 @@ + + + + + + Eval Review + + + + + + + +
+
+
+

Eval Review:

+
Review each output and leave feedback below. Navigate with arrow keys or buttons. When done, copy feedback and paste into Claude Code.
+
+
+
+ + + + + +
+
+ +
+
Prompt
+
+
+
+
+ + +
+
Output
+
+
No output files found
+
+
+ + + + + + + + +
+
Your Feedback
+
+ + + +
+
+
+ + +
+ + +
+
+
No benchmark data available. Run a benchmark to see quantitative results here.
+
+
+
+ + +
+
+

Review Complete

+

Your feedback has been saved. Go back to your Claude Code session and tell Claude you're done reviewing.

+
+ +
+
+
+ + +
+ + + + diff --git a/skill-creator/references/schemas.md b/skill-creator/references/schemas.md new file mode 100644 index 00000000..b6eeaa2d --- /dev/null +++ b/skill-creator/references/schemas.md @@ -0,0 +1,430 @@ +# JSON Schemas + +This document defines the JSON schemas used by skill-creator. + +--- + +## evals.json + +Defines the evals for a skill. Located at `evals/evals.json` within the skill directory. + +```json +{ + "skill_name": "example-skill", + "evals": [ + { + "id": 1, + "prompt": "User's example prompt", + "expected_output": "Description of expected result", + "files": ["evals/files/sample1.pdf"], + "expectations": [ + "The output includes X", + "The skill used script Y" + ] + } + ] +} +``` + +**Fields:** +- `skill_name`: Name matching the skill's frontmatter +- `evals[].id`: Unique integer identifier +- `evals[].prompt`: The task to execute +- `evals[].expected_output`: Human-readable description of success +- `evals[].files`: Optional list of input file paths (relative to skill root) +- `evals[].expectations`: List of verifiable statements + +--- + +## history.json + +Tracks version progression in Improve mode. Located at workspace root. + +```json +{ + "started_at": "2026-01-15T10:30:00Z", + "skill_name": "pdf", + "current_best": "v2", + "iterations": [ + { + "version": "v0", + "parent": null, + "expectation_pass_rate": 0.65, + "grading_result": "baseline", + "is_current_best": false + }, + { + "version": "v1", + "parent": "v0", + "expectation_pass_rate": 0.75, + "grading_result": "won", + "is_current_best": false + }, + { + "version": "v2", + "parent": "v1", + "expectation_pass_rate": 0.85, + "grading_result": "won", + "is_current_best": true + } + ] +} +``` + +**Fields:** +- `started_at`: ISO timestamp of when improvement started +- `skill_name`: Name of the skill being improved +- `current_best`: Version identifier of the best performer +- `iterations[].version`: Version identifier (v0, v1, ...) +- `iterations[].parent`: Parent version this was derived from +- `iterations[].expectation_pass_rate`: Pass rate from grading +- `iterations[].grading_result`: "baseline", "won", "lost", or "tie" +- `iterations[].is_current_best`: Whether this is the current best version + +--- + +## grading.json + +Output from the grader agent. Located at `/grading.json`. + +```json +{ + "expectations": [ + { + "text": "The output includes the name 'John Smith'", + "passed": true, + "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'" + }, + { + "text": "The spreadsheet has a SUM formula in cell B10", + "passed": false, + "evidence": "No spreadsheet was created. The output was a text file." + } + ], + "summary": { + "passed": 2, + "failed": 1, + "total": 3, + "pass_rate": 0.67 + }, + "execution_metrics": { + "tool_calls": { + "Read": 5, + "Write": 2, + "Bash": 8 + }, + "total_tool_calls": 15, + "total_steps": 6, + "errors_encountered": 0, + "output_chars": 12450, + "transcript_chars": 3200 + }, + "timing": { + "executor_duration_seconds": 165.0, + "grader_duration_seconds": 26.0, + "total_duration_seconds": 191.0 + }, + "claims": [ + { + "claim": "The form has 12 fillable fields", + "type": "factual", + "verified": true, + "evidence": "Counted 12 fields in field_info.json" + } + ], + "user_notes_summary": { + "uncertainties": ["Used 2023 data, may be stale"], + "needs_review": [], + "workarounds": ["Fell back to text overlay for non-fillable fields"] + }, + "eval_feedback": { + "suggestions": [ + { + "assertion": "The output includes the name 'John Smith'", + "reason": "A hallucinated document that mentions the name would also pass" + } + ], + "overall": "Assertions check presence but not correctness." + } +} +``` + +**Fields:** +- `expectations[]`: Graded expectations with evidence +- `summary`: Aggregate pass/fail counts +- `execution_metrics`: Tool usage and output size (from executor's metrics.json) +- `timing`: Wall clock timing (from timing.json) +- `claims`: Extracted and verified claims from the output +- `user_notes_summary`: Issues flagged by the executor +- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising + +--- + +## metrics.json + +Output from the executor agent. Located at `/outputs/metrics.json`. + +```json +{ + "tool_calls": { + "Read": 5, + "Write": 2, + "Bash": 8, + "Edit": 1, + "Glob": 2, + "Grep": 0 + }, + "total_tool_calls": 18, + "total_steps": 6, + "files_created": ["filled_form.pdf", "field_values.json"], + "errors_encountered": 0, + "output_chars": 12450, + "transcript_chars": 3200 +} +``` + +**Fields:** +- `tool_calls`: Count per tool type +- `total_tool_calls`: Sum of all tool calls +- `total_steps`: Number of major execution steps +- `files_created`: List of output files created +- `errors_encountered`: Number of errors during execution +- `output_chars`: Total character count of output files +- `transcript_chars`: Character count of transcript + +--- + +## timing.json + +Wall clock timing for a run. Located at `/timing.json`. + +**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact. + +```json +{ + "total_tokens": 84852, + "duration_ms": 23332, + "total_duration_seconds": 23.3, + "executor_start": "2026-01-15T10:30:00Z", + "executor_end": "2026-01-15T10:32:45Z", + "executor_duration_seconds": 165.0, + "grader_start": "2026-01-15T10:32:46Z", + "grader_end": "2026-01-15T10:33:12Z", + "grader_duration_seconds": 26.0 +} +``` + +--- + +## benchmark.json + +Output from Benchmark mode. Located at `benchmarks//benchmark.json`. + +```json +{ + "metadata": { + "skill_name": "pdf", + "skill_path": "/path/to/pdf", + "executor_model": "claude-sonnet-4-20250514", + "analyzer_model": "most-capable-model", + "timestamp": "2026-01-15T10:30:00Z", + "evals_run": [1, 2, 3], + "runs_per_configuration": 3 + }, + + "runs": [ + { + "eval_id": 1, + "eval_name": "Ocean", + "configuration": "with_skill", + "run_number": 1, + "result": { + "pass_rate": 0.85, + "passed": 6, + "failed": 1, + "total": 7, + "time_seconds": 42.5, + "tokens": 3800, + "tool_calls": 18, + "errors": 0 + }, + "expectations": [ + {"text": "...", "passed": true, "evidence": "..."} + ], + "notes": [ + "Used 2023 data, may be stale", + "Fell back to text overlay for non-fillable fields" + ] + } + ], + + "run_summary": { + "with_skill": { + "pass_rate": {"mean": 0.85, "stddev": 0.05, "min": 0.80, "max": 0.90}, + "time_seconds": {"mean": 45.0, "stddev": 12.0, "min": 32.0, "max": 58.0}, + "tokens": {"mean": 3800, "stddev": 400, "min": 3200, "max": 4100} + }, + "without_skill": { + "pass_rate": {"mean": 0.35, "stddev": 0.08, "min": 0.28, "max": 0.45}, + "time_seconds": {"mean": 32.0, "stddev": 8.0, "min": 24.0, "max": 42.0}, + "tokens": {"mean": 2100, "stddev": 300, "min": 1800, "max": 2500} + }, + "delta": { + "pass_rate": "+0.50", + "time_seconds": "+13.0", + "tokens": "+1700" + } + }, + + "notes": [ + "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value", + "Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent", + "Without-skill runs consistently fail on table extraction expectations", + "Skill adds 13s average execution time but improves pass rate by 50%" + ] +} +``` + +**Fields:** +- `metadata`: Information about the benchmark run + - `skill_name`: Name of the skill + - `timestamp`: When the benchmark was run + - `evals_run`: List of eval names or IDs + - `runs_per_configuration`: Number of runs per config (e.g. 3) +- `runs[]`: Individual run results + - `eval_id`: Numeric eval identifier + - `eval_name`: Human-readable eval name (used as section header in the viewer) + - `configuration`: Must be `"with_skill"` or `"without_skill"` (the viewer uses this exact string for grouping and color coding) + - `run_number`: Integer run number (1, 2, 3...) + - `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors` +- `run_summary`: Statistical aggregates per configuration + - `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields + - `delta`: Difference strings like `"+0.50"`, `"+13.0"`, `"+1700"` +- `notes`: Freeform observations from the analyzer + +**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually. + +--- + +## comparison.json + +Output from blind comparator. Located at `/comparison-N.json`. + +```json +{ + "winner": "A", + "reasoning": "Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.", + "rubric": { + "A": { + "content": { + "correctness": 5, + "completeness": 5, + "accuracy": 4 + }, + "structure": { + "organization": 4, + "formatting": 5, + "usability": 4 + }, + "content_score": 4.7, + "structure_score": 4.3, + "overall_score": 9.0 + }, + "B": { + "content": { + "correctness": 3, + "completeness": 2, + "accuracy": 3 + }, + "structure": { + "organization": 3, + "formatting": 2, + "usability": 3 + }, + "content_score": 2.7, + "structure_score": 2.7, + "overall_score": 5.4 + } + }, + "output_quality": { + "A": { + "score": 9, + "strengths": ["Complete solution", "Well-formatted", "All fields present"], + "weaknesses": ["Minor style inconsistency in header"] + }, + "B": { + "score": 5, + "strengths": ["Readable output", "Correct basic structure"], + "weaknesses": ["Missing date field", "Formatting inconsistencies", "Partial data extraction"] + } + }, + "expectation_results": { + "A": { + "passed": 4, + "total": 5, + "pass_rate": 0.80, + "details": [ + {"text": "Output includes name", "passed": true} + ] + }, + "B": { + "passed": 3, + "total": 5, + "pass_rate": 0.60, + "details": [ + {"text": "Output includes name", "passed": true} + ] + } + } +} +``` + +--- + +## analysis.json + +Output from post-hoc analyzer. Located at `/analysis.json`. + +```json +{ + "comparison_summary": { + "winner": "A", + "winner_skill": "path/to/winner/skill", + "loser_skill": "path/to/loser/skill", + "comparator_reasoning": "Brief summary of why comparator chose winner" + }, + "winner_strengths": [ + "Clear step-by-step instructions for handling multi-page documents", + "Included validation script that caught formatting errors" + ], + "loser_weaknesses": [ + "Vague instruction 'process the document appropriately' led to inconsistent behavior", + "No script for validation, agent had to improvise" + ], + "instruction_following": { + "winner": { + "score": 9, + "issues": ["Minor: skipped optional logging step"] + }, + "loser": { + "score": 6, + "issues": [ + "Did not use the skill's formatting template", + "Invented own approach instead of following step 3" + ] + } + }, + "improvement_suggestions": [ + { + "priority": "high", + "category": "instructions", + "suggestion": "Replace 'process the document appropriately' with explicit steps", + "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior" + } + ], + "transcript_insights": { + "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script", + "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods" + } +} +``` diff --git a/skill-creator/scripts/__init__.py b/skill-creator/scripts/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/skill-creator/scripts/aggregate_benchmark.py b/skill-creator/scripts/aggregate_benchmark.py new file mode 100755 index 00000000..3e66e8c1 --- /dev/null +++ b/skill-creator/scripts/aggregate_benchmark.py @@ -0,0 +1,401 @@ +#!/usr/bin/env python3 +""" +Aggregate individual run results into benchmark summary statistics. + +Reads grading.json files from run directories and produces: +- run_summary with mean, stddev, min, max for each metric +- delta between with_skill and without_skill configurations + +Usage: + python aggregate_benchmark.py + +Example: + python aggregate_benchmark.py benchmarks/2026-01-15T10-30-00/ + +The script supports two directory layouts: + + Workspace layout (from skill-creator iterations): + / + └── eval-N/ + ├── with_skill/ + │ ├── run-1/grading.json + │ └── run-2/grading.json + └── without_skill/ + ├── run-1/grading.json + └── run-2/grading.json + + Legacy layout (with runs/ subdirectory): + / + └── runs/ + └── eval-N/ + ├── with_skill/ + │ └── run-1/grading.json + └── without_skill/ + └── run-1/grading.json +""" + +import argparse +import json +import math +import sys +from datetime import datetime, timezone +from pathlib import Path + + +def calculate_stats(values: list[float]) -> dict: + """Calculate mean, stddev, min, max for a list of values.""" + if not values: + return {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0} + + n = len(values) + mean = sum(values) / n + + if n > 1: + variance = sum((x - mean) ** 2 for x in values) / (n - 1) + stddev = math.sqrt(variance) + else: + stddev = 0.0 + + return { + "mean": round(mean, 4), + "stddev": round(stddev, 4), + "min": round(min(values), 4), + "max": round(max(values), 4) + } + + +def load_run_results(benchmark_dir: Path) -> dict: + """ + Load all run results from a benchmark directory. + + Returns dict keyed by config name (e.g. "with_skill"/"without_skill", + or "new_skill"/"old_skill"), each containing a list of run results. + """ + # Support both layouts: eval dirs directly under benchmark_dir, or under runs/ + runs_dir = benchmark_dir / "runs" + if runs_dir.exists(): + search_dir = runs_dir + elif list(benchmark_dir.glob("eval-*")): + search_dir = benchmark_dir + else: + print(f"No eval directories found in {benchmark_dir} or {benchmark_dir / 'runs'}") + return {} + + results: dict[str, list] = {} + + for eval_idx, eval_dir in enumerate(sorted(search_dir.glob("eval-*"))): + metadata_path = eval_dir / "eval_metadata.json" + if metadata_path.exists(): + try: + with open(metadata_path) as mf: + eval_id = json.load(mf).get("eval_id", eval_idx) + except (json.JSONDecodeError, OSError): + eval_id = eval_idx + else: + try: + eval_id = int(eval_dir.name.split("-")[1]) + except ValueError: + eval_id = eval_idx + + # Discover config directories dynamically rather than hardcoding names + for config_dir in sorted(eval_dir.iterdir()): + if not config_dir.is_dir(): + continue + # Skip non-config directories (inputs, outputs, etc.) + if not list(config_dir.glob("run-*")): + continue + config = config_dir.name + if config not in results: + results[config] = [] + + for run_dir in sorted(config_dir.glob("run-*")): + run_number = int(run_dir.name.split("-")[1]) + grading_file = run_dir / "grading.json" + + if not grading_file.exists(): + print(f"Warning: grading.json not found in {run_dir}") + continue + + try: + with open(grading_file) as f: + grading = json.load(f) + except json.JSONDecodeError as e: + print(f"Warning: Invalid JSON in {grading_file}: {e}") + continue + + # Extract metrics + result = { + "eval_id": eval_id, + "run_number": run_number, + "pass_rate": grading.get("summary", {}).get("pass_rate", 0.0), + "passed": grading.get("summary", {}).get("passed", 0), + "failed": grading.get("summary", {}).get("failed", 0), + "total": grading.get("summary", {}).get("total", 0), + } + + # Extract timing — check grading.json first, then sibling timing.json + timing = grading.get("timing", {}) + result["time_seconds"] = timing.get("total_duration_seconds", 0.0) + timing_file = run_dir / "timing.json" + if result["time_seconds"] == 0.0 and timing_file.exists(): + try: + with open(timing_file) as tf: + timing_data = json.load(tf) + result["time_seconds"] = timing_data.get("total_duration_seconds", 0.0) + result["tokens"] = timing_data.get("total_tokens", 0) + except json.JSONDecodeError: + pass + + # Extract metrics if available + metrics = grading.get("execution_metrics", {}) + result["tool_calls"] = metrics.get("total_tool_calls", 0) + if not result.get("tokens"): + result["tokens"] = metrics.get("output_chars", 0) + result["errors"] = metrics.get("errors_encountered", 0) + + # Extract expectations — viewer requires fields: text, passed, evidence + raw_expectations = grading.get("expectations", []) + for exp in raw_expectations: + if "text" not in exp or "passed" not in exp: + print(f"Warning: expectation in {grading_file} missing required fields (text, passed, evidence): {exp}") + result["expectations"] = raw_expectations + + # Extract notes from user_notes_summary + notes_summary = grading.get("user_notes_summary", {}) + notes = [] + notes.extend(notes_summary.get("uncertainties", [])) + notes.extend(notes_summary.get("needs_review", [])) + notes.extend(notes_summary.get("workarounds", [])) + result["notes"] = notes + + results[config].append(result) + + return results + + +def aggregate_results(results: dict) -> dict: + """ + Aggregate run results into summary statistics. + + Returns run_summary with stats for each configuration and delta. + """ + run_summary = {} + configs = list(results.keys()) + + for config in configs: + runs = results.get(config, []) + + if not runs: + run_summary[config] = { + "pass_rate": {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0}, + "time_seconds": {"mean": 0.0, "stddev": 0.0, "min": 0.0, "max": 0.0}, + "tokens": {"mean": 0, "stddev": 0, "min": 0, "max": 0} + } + continue + + pass_rates = [r["pass_rate"] for r in runs] + times = [r["time_seconds"] for r in runs] + tokens = [r.get("tokens", 0) for r in runs] + + run_summary[config] = { + "pass_rate": calculate_stats(pass_rates), + "time_seconds": calculate_stats(times), + "tokens": calculate_stats(tokens) + } + + # Calculate delta between the first two configs (if two exist) + if len(configs) >= 2: + primary = run_summary.get(configs[0], {}) + baseline = run_summary.get(configs[1], {}) + else: + primary = run_summary.get(configs[0], {}) if configs else {} + baseline = {} + + delta_pass_rate = primary.get("pass_rate", {}).get("mean", 0) - baseline.get("pass_rate", {}).get("mean", 0) + delta_time = primary.get("time_seconds", {}).get("mean", 0) - baseline.get("time_seconds", {}).get("mean", 0) + delta_tokens = primary.get("tokens", {}).get("mean", 0) - baseline.get("tokens", {}).get("mean", 0) + + run_summary["delta"] = { + "pass_rate": f"{delta_pass_rate:+.2f}", + "time_seconds": f"{delta_time:+.1f}", + "tokens": f"{delta_tokens:+.0f}" + } + + return run_summary + + +def generate_benchmark(benchmark_dir: Path, skill_name: str = "", skill_path: str = "") -> dict: + """ + Generate complete benchmark.json from run results. + """ + results = load_run_results(benchmark_dir) + run_summary = aggregate_results(results) + + # Build runs array for benchmark.json + runs = [] + for config in results: + for result in results[config]: + runs.append({ + "eval_id": result["eval_id"], + "configuration": config, + "run_number": result["run_number"], + "result": { + "pass_rate": result["pass_rate"], + "passed": result["passed"], + "failed": result["failed"], + "total": result["total"], + "time_seconds": result["time_seconds"], + "tokens": result.get("tokens", 0), + "tool_calls": result.get("tool_calls", 0), + "errors": result.get("errors", 0) + }, + "expectations": result["expectations"], + "notes": result["notes"] + }) + + # Determine eval IDs from results + eval_ids = sorted(set( + r["eval_id"] + for config in results.values() + for r in config + )) + + benchmark = { + "metadata": { + "skill_name": skill_name or "", + "skill_path": skill_path or "", + "executor_model": "", + "analyzer_model": "", + "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), + "evals_run": eval_ids, + "runs_per_configuration": 3 + }, + "runs": runs, + "run_summary": run_summary, + "notes": [] # To be filled by analyzer + } + + return benchmark + + +def generate_markdown(benchmark: dict) -> str: + """Generate human-readable benchmark.md from benchmark data.""" + metadata = benchmark["metadata"] + run_summary = benchmark["run_summary"] + + # Determine config names (excluding "delta") + configs = [k for k in run_summary if k != "delta"] + config_a = configs[0] if len(configs) >= 1 else "config_a" + config_b = configs[1] if len(configs) >= 2 else "config_b" + label_a = config_a.replace("_", " ").title() + label_b = config_b.replace("_", " ").title() + + lines = [ + f"# Skill Benchmark: {metadata['skill_name']}", + "", + f"**Model**: {metadata['executor_model']}", + f"**Date**: {metadata['timestamp']}", + f"**Evals**: {', '.join(map(str, metadata['evals_run']))} ({metadata['runs_per_configuration']} runs each per configuration)", + "", + "## Summary", + "", + f"| Metric | {label_a} | {label_b} | Delta |", + "|--------|------------|---------------|-------|", + ] + + a_summary = run_summary.get(config_a, {}) + b_summary = run_summary.get(config_b, {}) + delta = run_summary.get("delta", {}) + + # Format pass rate + a_pr = a_summary.get("pass_rate", {}) + b_pr = b_summary.get("pass_rate", {}) + lines.append(f"| Pass Rate | {a_pr.get('mean', 0)*100:.0f}% ± {a_pr.get('stddev', 0)*100:.0f}% | {b_pr.get('mean', 0)*100:.0f}% ± {b_pr.get('stddev', 0)*100:.0f}% | {delta.get('pass_rate', '—')} |") + + # Format time + a_time = a_summary.get("time_seconds", {}) + b_time = b_summary.get("time_seconds", {}) + lines.append(f"| Time | {a_time.get('mean', 0):.1f}s ± {a_time.get('stddev', 0):.1f}s | {b_time.get('mean', 0):.1f}s ± {b_time.get('stddev', 0):.1f}s | {delta.get('time_seconds', '—')}s |") + + # Format tokens + a_tokens = a_summary.get("tokens", {}) + b_tokens = b_summary.get("tokens", {}) + lines.append(f"| Tokens | {a_tokens.get('mean', 0):.0f} ± {a_tokens.get('stddev', 0):.0f} | {b_tokens.get('mean', 0):.0f} ± {b_tokens.get('stddev', 0):.0f} | {delta.get('tokens', '—')} |") + + # Notes section + if benchmark.get("notes"): + lines.extend([ + "", + "## Notes", + "" + ]) + for note in benchmark["notes"]: + lines.append(f"- {note}") + + return "\n".join(lines) + + +def main(): + parser = argparse.ArgumentParser( + description="Aggregate benchmark run results into summary statistics" + ) + parser.add_argument( + "benchmark_dir", + type=Path, + help="Path to the benchmark directory" + ) + parser.add_argument( + "--skill-name", + default="", + help="Name of the skill being benchmarked" + ) + parser.add_argument( + "--skill-path", + default="", + help="Path to the skill being benchmarked" + ) + parser.add_argument( + "--output", "-o", + type=Path, + help="Output path for benchmark.json (default: /benchmark.json)" + ) + + args = parser.parse_args() + + if not args.benchmark_dir.exists(): + print(f"Directory not found: {args.benchmark_dir}") + sys.exit(1) + + # Generate benchmark + benchmark = generate_benchmark(args.benchmark_dir, args.skill_name, args.skill_path) + + # Determine output paths + output_json = args.output or (args.benchmark_dir / "benchmark.json") + output_md = output_json.with_suffix(".md") + + # Write benchmark.json + with open(output_json, "w") as f: + json.dump(benchmark, f, indent=2) + print(f"Generated: {output_json}") + + # Write benchmark.md + markdown = generate_markdown(benchmark) + with open(output_md, "w") as f: + f.write(markdown) + print(f"Generated: {output_md}") + + # Print summary + run_summary = benchmark["run_summary"] + configs = [k for k in run_summary if k != "delta"] + delta = run_summary.get("delta", {}) + + print(f"\nSummary:") + for config in configs: + pr = run_summary[config]["pass_rate"]["mean"] + label = config.replace("_", " ").title() + print(f" {label}: {pr*100:.1f}% pass rate") + print(f" Delta: {delta.get('pass_rate', '—')}") + + +if __name__ == "__main__": + main() diff --git a/skill-creator/scripts/generate_report.py b/skill-creator/scripts/generate_report.py new file mode 100755 index 00000000..959e30a0 --- /dev/null +++ b/skill-creator/scripts/generate_report.py @@ -0,0 +1,326 @@ +#!/usr/bin/env python3 +"""Generate an HTML report from run_loop.py output. + +Takes the JSON output from run_loop.py and generates a visual HTML report +showing each description attempt with check/x for each test case. +Distinguishes between train and test queries. +""" + +import argparse +import html +import json +import sys +from pathlib import Path + + +def generate_html(data: dict, auto_refresh: bool = False, skill_name: str = "") -> str: + """Generate HTML report from loop output data. If auto_refresh is True, adds a meta refresh tag.""" + history = data.get("history", []) + holdout = data.get("holdout", 0) + title_prefix = html.escape(skill_name + " \u2014 ") if skill_name else "" + + # Get all unique queries from train and test sets, with should_trigger info + train_queries: list[dict] = [] + test_queries: list[dict] = [] + if history: + for r in history[0].get("train_results", history[0].get("results", [])): + train_queries.append({"query": r["query"], "should_trigger": r.get("should_trigger", True)}) + if history[0].get("test_results"): + for r in history[0].get("test_results", []): + test_queries.append({"query": r["query"], "should_trigger": r.get("should_trigger", True)}) + + refresh_tag = ' \n' if auto_refresh else "" + + html_parts = [""" + + + +""" + refresh_tag + """ """ + title_prefix + """Skill Description Optimization + + + + + + +

""" + title_prefix + """Skill Description Optimization

+
+ Optimizing your skill's description. This page updates automatically as Claude tests different versions of your skill's description. Each row is an iteration — a new description attempt. The columns show test queries: green checkmarks mean the skill triggered correctly (or correctly didn't trigger), red crosses mean it got it wrong. The "Train" score shows performance on queries used to improve the description; the "Test" score shows performance on held-out queries the optimizer hasn't seen. When it's done, Claude will apply the best-performing description to your skill. +
+"""] + + # Summary section + best_test_score = data.get('best_test_score') + best_train_score = data.get('best_train_score') + html_parts.append(f""" +
+

Original: {html.escape(data.get('original_description', 'N/A'))}

+

Best: {html.escape(data.get('best_description', 'N/A'))}

+

Best Score: {data.get('best_score', 'N/A')} {'(test)' if best_test_score else '(train)'}

+

Iterations: {data.get('iterations_run', 0)} | Train: {data.get('train_size', '?')} | Test: {data.get('test_size', '?')}

+
+""") + + # Legend + html_parts.append(""" +
+ Query columns: + Should trigger + Should NOT trigger + Train + Test +
+""") + + # Table header + html_parts.append(""" +
+ + + + + + + +""") + + # Add column headers for train queries + for qinfo in train_queries: + polarity = "positive-col" if qinfo["should_trigger"] else "negative-col" + html_parts.append(f' \n') + + # Add column headers for test queries (different color) + for qinfo in test_queries: + polarity = "positive-col" if qinfo["should_trigger"] else "negative-col" + html_parts.append(f' \n') + + html_parts.append(""" + + +""") + + # Find best iteration for highlighting + if test_queries: + best_iter = max(history, key=lambda h: h.get("test_passed") or 0).get("iteration") + else: + best_iter = max(history, key=lambda h: h.get("train_passed", h.get("passed", 0))).get("iteration") + + # Add rows for each iteration + for h in history: + iteration = h.get("iteration", "?") + train_passed = h.get("train_passed", h.get("passed", 0)) + train_total = h.get("train_total", h.get("total", 0)) + test_passed = h.get("test_passed") + test_total = h.get("test_total") + description = h.get("description", "") + train_results = h.get("train_results", h.get("results", [])) + test_results = h.get("test_results", []) + + # Create lookups for results by query + train_by_query = {r["query"]: r for r in train_results} + test_by_query = {r["query"]: r for r in test_results} if test_results else {} + + # Compute aggregate correct/total runs across all retries + def aggregate_runs(results: list[dict]) -> tuple[int, int]: + correct = 0 + total = 0 + for r in results: + runs = r.get("runs", 0) + triggers = r.get("triggers", 0) + total += runs + if r.get("should_trigger", True): + correct += triggers + else: + correct += runs - triggers + return correct, total + + train_correct, train_runs = aggregate_runs(train_results) + test_correct, test_runs = aggregate_runs(test_results) + + # Determine score classes + def score_class(correct: int, total: int) -> str: + if total > 0: + ratio = correct / total + if ratio >= 0.8: + return "score-good" + elif ratio >= 0.5: + return "score-ok" + return "score-bad" + + train_class = score_class(train_correct, train_runs) + test_class = score_class(test_correct, test_runs) + + row_class = "best-row" if iteration == best_iter else "" + + html_parts.append(f""" + + + + +""") + + # Add result for each train query + for qinfo in train_queries: + r = train_by_query.get(qinfo["query"], {}) + did_pass = r.get("pass", False) + triggers = r.get("triggers", 0) + runs = r.get("runs", 0) + + icon = "✓" if did_pass else "✗" + css_class = "pass" if did_pass else "fail" + + html_parts.append(f' \n') + + # Add result for each test query (with different background) + for qinfo in test_queries: + r = test_by_query.get(qinfo["query"], {}) + did_pass = r.get("pass", False) + triggers = r.get("triggers", 0) + runs = r.get("runs", 0) + + icon = "✓" if did_pass else "✗" + css_class = "pass" if did_pass else "fail" + + html_parts.append(f' \n') + + html_parts.append(" \n") + + html_parts.append(""" +
IterTrainTestDescription{html.escape(qinfo["query"])}{html.escape(qinfo["query"])}
{iteration}{train_correct}/{train_runs}{test_correct}/{test_runs}{html.escape(description)}{icon}{triggers}/{runs}{icon}{triggers}/{runs}
+
+""") + + html_parts.append(""" + + +""") + + return "".join(html_parts) + + +def main(): + parser = argparse.ArgumentParser(description="Generate HTML report from run_loop output") + parser.add_argument("input", help="Path to JSON output from run_loop.py (or - for stdin)") + parser.add_argument("-o", "--output", default=None, help="Output HTML file (default: stdout)") + parser.add_argument("--skill-name", default="", help="Skill name to include in the report title") + args = parser.parse_args() + + if args.input == "-": + data = json.load(sys.stdin) + else: + data = json.loads(Path(args.input).read_text()) + + html_output = generate_html(data, skill_name=args.skill_name) + + if args.output: + Path(args.output).write_text(html_output) + print(f"Report written to {args.output}", file=sys.stderr) + else: + print(html_output) + + +if __name__ == "__main__": + main() diff --git a/skill-creator/scripts/improve_description.py b/skill-creator/scripts/improve_description.py new file mode 100755 index 00000000..a270777b --- /dev/null +++ b/skill-creator/scripts/improve_description.py @@ -0,0 +1,248 @@ +#!/usr/bin/env python3 +"""Improve a skill description based on eval results. + +Takes eval results (from run_eval.py) and generates an improved description +using Claude with extended thinking. +""" + +import argparse +import json +import re +import sys +from pathlib import Path + +import anthropic + +from scripts.utils import parse_skill_md + + +def improve_description( + client: anthropic.Anthropic, + skill_name: str, + skill_content: str, + current_description: str, + eval_results: dict, + history: list[dict], + model: str, + test_results: dict | None = None, + log_dir: Path | None = None, + iteration: int | None = None, +) -> str: + """Call Claude to improve the description based on eval results.""" + failed_triggers = [ + r for r in eval_results["results"] + if r["should_trigger"] and not r["pass"] + ] + false_triggers = [ + r for r in eval_results["results"] + if not r["should_trigger"] and not r["pass"] + ] + + # Build scores summary + train_score = f"{eval_results['summary']['passed']}/{eval_results['summary']['total']}" + if test_results: + test_score = f"{test_results['summary']['passed']}/{test_results['summary']['total']}" + scores_summary = f"Train: {train_score}, Test: {test_score}" + else: + scores_summary = f"Train: {train_score}" + + prompt = f"""You are optimizing a skill description for a Claude Code skill called "{skill_name}". A "skill" is sort of like a prompt, but with progressive disclosure -- there's a title and description that Claude sees when deciding whether to use the skill, and then if it does use the skill, it reads the .md file which has lots more details and potentially links to other resources in the skill folder like helper files and scripts and additional documentation or examples. + +The description appears in Claude's "available_skills" list. When a user sends a query, Claude decides whether to invoke the skill based solely on the title and on this description. Your goal is to write a description that triggers for relevant queries, and doesn't trigger for irrelevant ones. + +Here's the current description: + +"{current_description}" + + +Current scores ({scores_summary}): + +""" + if failed_triggers: + prompt += "FAILED TO TRIGGER (should have triggered but didn't):\n" + for r in failed_triggers: + prompt += f' - "{r["query"]}" (triggered {r["triggers"]}/{r["runs"]} times)\n' + prompt += "\n" + + if false_triggers: + prompt += "FALSE TRIGGERS (triggered but shouldn't have):\n" + for r in false_triggers: + prompt += f' - "{r["query"]}" (triggered {r["triggers"]}/{r["runs"]} times)\n' + prompt += "\n" + + if history: + prompt += "PREVIOUS ATTEMPTS (do NOT repeat these — try something structurally different):\n\n" + for h in history: + train_s = f"{h.get('train_passed', h.get('passed', 0))}/{h.get('train_total', h.get('total', 0))}" + test_s = f"{h.get('test_passed', '?')}/{h.get('test_total', '?')}" if h.get('test_passed') is not None else None + score_str = f"train={train_s}" + (f", test={test_s}" if test_s else "") + prompt += f'\n' + prompt += f'Description: "{h["description"]}"\n' + if "results" in h: + prompt += "Train results:\n" + for r in h["results"]: + status = "PASS" if r["pass"] else "FAIL" + prompt += f' [{status}] "{r["query"][:80]}" (triggered {r["triggers"]}/{r["runs"]})\n' + if h.get("note"): + prompt += f'Note: {h["note"]}\n' + prompt += "\n\n" + + prompt += f""" + +Skill content (for context on what the skill does): + +{skill_content} + + +Based on the failures, write a new and improved description that is more likely to trigger correctly. When I say "based on the failures", it's a bit of a tricky line to walk because we don't want to overfit to the specific cases you're seeing. So what I DON'T want you to do is produce an ever-expanding list of specific queries that this skill should or shouldn't trigger for. Instead, try to generalize from the failures to broader categories of user intent and situations where this skill would be useful or not useful. The reason for this is twofold: + +1. Avoid overfitting +2. The list might get loooong and it's injected into ALL queries and there might be a lot of skills, so we don't want to blow too much space on any given description. + +Concretely, your description should not be more than about 100-200 words, even if that comes at the cost of accuracy. + +Here are some tips that we've found to work well in writing these descriptions: +- The skill should be phrased in the imperative -- "Use this skill for" rather than "this skill does" +- The skill description should focus on the user's intent, what they are trying to achieve, vs. the implementation details of how the skill works. +- The description competes with other skills for Claude's attention — make it distinctive and immediately recognizable. +- If you're getting lots of failures after repeated attempts, change things up. Try different sentence structures or wordings. + +I'd encourage you to be creative and mix up the style in different iterations since you'll have multiple opportunities to try different approaches and we'll just grab the highest-scoring one at the end. + +Please respond with only the new description text in tags, nothing else.""" + + response = client.messages.create( + model=model, + max_tokens=16000, + thinking={ + "type": "enabled", + "budget_tokens": 10000, + }, + messages=[{"role": "user", "content": prompt}], + ) + + # Extract thinking and text from response + thinking_text = "" + text = "" + for block in response.content: + if block.type == "thinking": + thinking_text = block.thinking + elif block.type == "text": + text = block.text + + # Parse out the tags + match = re.search(r"(.*?)", text, re.DOTALL) + description = match.group(1).strip().strip('"') if match else text.strip().strip('"') + + # Log the transcript + transcript: dict = { + "iteration": iteration, + "prompt": prompt, + "thinking": thinking_text, + "response": text, + "parsed_description": description, + "char_count": len(description), + "over_limit": len(description) > 1024, + } + + # If over 1024 chars, ask the model to shorten it + if len(description) > 1024: + shorten_prompt = f"Your description is {len(description)} characters, which exceeds the hard 1024 character limit. Please rewrite it to be under 1024 characters while preserving the most important trigger words and intent coverage. Respond with only the new description in tags." + shorten_response = client.messages.create( + model=model, + max_tokens=16000, + thinking={ + "type": "enabled", + "budget_tokens": 10000, + }, + messages=[ + {"role": "user", "content": prompt}, + {"role": "assistant", "content": text}, + {"role": "user", "content": shorten_prompt}, + ], + ) + + shorten_thinking = "" + shorten_text = "" + for block in shorten_response.content: + if block.type == "thinking": + shorten_thinking = block.thinking + elif block.type == "text": + shorten_text = block.text + + match = re.search(r"(.*?)", shorten_text, re.DOTALL) + shortened = match.group(1).strip().strip('"') if match else shorten_text.strip().strip('"') + + transcript["rewrite_prompt"] = shorten_prompt + transcript["rewrite_thinking"] = shorten_thinking + transcript["rewrite_response"] = shorten_text + transcript["rewrite_description"] = shortened + transcript["rewrite_char_count"] = len(shortened) + description = shortened + + transcript["final_description"] = description + + if log_dir: + log_dir.mkdir(parents=True, exist_ok=True) + log_file = log_dir / f"improve_iter_{iteration or 'unknown'}.json" + log_file.write_text(json.dumps(transcript, indent=2)) + + return description + + +def main(): + parser = argparse.ArgumentParser(description="Improve a skill description based on eval results") + parser.add_argument("--eval-results", required=True, help="Path to eval results JSON (from run_eval.py)") + parser.add_argument("--skill-path", required=True, help="Path to skill directory") + parser.add_argument("--history", default=None, help="Path to history JSON (previous attempts)") + parser.add_argument("--model", required=True, help="Model for improvement") + parser.add_argument("--verbose", action="store_true", help="Print thinking to stderr") + args = parser.parse_args() + + skill_path = Path(args.skill_path) + if not (skill_path / "SKILL.md").exists(): + print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr) + sys.exit(1) + + eval_results = json.loads(Path(args.eval_results).read_text()) + history = [] + if args.history: + history = json.loads(Path(args.history).read_text()) + + name, _, content = parse_skill_md(skill_path) + current_description = eval_results["description"] + + if args.verbose: + print(f"Current: {current_description}", file=sys.stderr) + print(f"Score: {eval_results['summary']['passed']}/{eval_results['summary']['total']}", file=sys.stderr) + + client = anthropic.Anthropic() + new_description = improve_description( + client=client, + skill_name=name, + skill_content=content, + current_description=current_description, + eval_results=eval_results, + history=history, + model=args.model, + ) + + if args.verbose: + print(f"Improved: {new_description}", file=sys.stderr) + + # Output as JSON with both the new description and updated history + output = { + "description": new_description, + "history": history + [{ + "description": current_description, + "passed": eval_results["summary"]["passed"], + "failed": eval_results["summary"]["failed"], + "total": eval_results["summary"]["total"], + "results": eval_results["results"], + }], + } + print(json.dumps(output, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/skill-creator/scripts/package_skill.py b/skill-creator/scripts/package_skill.py index eb665363..ceab59b3 100755 --- a/skill-creator/scripts/package_skill.py +++ b/skill-creator/scripts/package_skill.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 """ -Skill Packager - Creates a distributable zip file of a skill folder +Skill Packager - Creates a distributable .skill file of a skill folder Usage: python utils/package_skill.py [output-directory] @@ -10,12 +10,35 @@ python utils/package_skill.py skills/public/my-skill ./dist """ +import fnmatch +import re import sys import zipfile -import re from pathlib import Path -from quick_validate import validate_skill -from security_scan import calculate_skill_hash +from scripts.quick_validate import validate_skill +from scripts.security_scan import calculate_skill_hash + +# Patterns to exclude when packaging skills. +EXCLUDE_DIRS = {"__pycache__", "node_modules"} +EXCLUDE_GLOBS = {"*.pyc"} +EXCLUDE_FILES = {".DS_Store"} +# Directories excluded only at the skill root (not when nested deeper). +ROOT_EXCLUDE_DIRS = {"evals"} + + +def should_exclude(rel_path: Path) -> bool: + """Check if a path should be excluded from packaging.""" + parts = rel_path.parts + if any(part in EXCLUDE_DIRS for part in parts): + return True + # rel_path is relative to skill_path.parent, so parts[0] is the skill + # folder name and parts[1] (if present) is the first subdir. + if len(parts) > 1 and parts[1] in ROOT_EXCLUDE_DIRS: + return True + name = rel_path.name + if name in EXCLUDE_FILES: + return True + return any(fnmatch.fnmatch(name, pat) for pat in EXCLUDE_GLOBS) def validate_security_marker(skill_path: Path) -> tuple[bool, str]: @@ -58,54 +81,54 @@ def validate_security_marker(skill_path: Path) -> tuple[bool, str]: def package_skill(skill_path, output_dir=None): """ - Package a skill folder into a zip file. + Package a skill folder into a .skill file. Args: skill_path: Path to the skill folder - output_dir: Optional output directory for the zip file (defaults to current directory) + output_dir: Optional output directory for the .skill file (defaults to current directory) Returns: - Path to the created zip file, or None if error + Path to the created .skill file, or None if error """ skill_path = Path(skill_path).resolve() # Validate skill folder exists if not skill_path.exists(): - print(f"❌ Error: Skill folder not found: {skill_path}") + print(f"Error: Skill folder not found: {skill_path}") return None if not skill_path.is_dir(): - print(f"❌ Error: Path is not a directory: {skill_path}") + print(f"Error: Path is not a directory: {skill_path}") return None # Validate SKILL.md exists skill_md = skill_path / "SKILL.md" if not skill_md.exists(): - print(f"❌ Error: SKILL.md not found in {skill_path}") + print(f"Error: SKILL.md not found in {skill_path}") return None # Step 1: Validate skill structure and metadata - print("🔍 Step 1: Validating skill structure...") + print("Step 1: Validating skill structure...") valid, message = validate_skill(skill_path) if not valid: - print(f"❌ FAILED: {message}") + print(f"FAILED: {message}") print(" Fix validation errors before packaging.") return None - print(f"✅ PASSED: {message}\n") + print(f"PASSED: {message}\n") # Step 2: Validate security scan (HARD REQUIREMENT) - print("🔍 Step 2: Validating security scan...") + print("Step 2: Validating security scan...") is_valid, message = validate_security_marker(skill_path) if not is_valid: - print(f"❌ BLOCKED: {message}") + print(f"BLOCKED: {message}") print(f" You MUST run: python scripts/security_scan.py {skill_path.name}") print(" Security review is MANDATORY before packaging.") return None - print(f"✅ PASSED: {message}\n") + print(f"PASSED: {message}\n") # Step 3: Package the skill - print("📦 Step 3: Creating package...") + print("Step 3: Creating package...") # Determine output location skill_name = skill_path.name @@ -115,24 +138,27 @@ def package_skill(skill_path, output_dir=None): else: output_path = Path.cwd() - zip_filename = output_path / f"{skill_name}.zip" + skill_filename = output_path / f"{skill_name}.skill" - # Create the zip file + # Create the .skill file (zip format) try: - with zipfile.ZipFile(zip_filename, 'w', zipfile.ZIP_DEFLATED) as zipf: - # Walk through the skill directory + with zipfile.ZipFile(skill_filename, 'w', zipfile.ZIP_DEFLATED) as zipf: + # Walk through the skill directory, excluding build artifacts for file_path in skill_path.rglob('*'): - if file_path.is_file(): - # Calculate the relative path within the zip - arcname = file_path.relative_to(skill_path.parent) - zipf.write(file_path, arcname) - print(f" Added: {arcname}") - - print(f"\n✅ Successfully packaged skill to: {zip_filename}") - return zip_filename + if not file_path.is_file(): + continue + arcname = file_path.relative_to(skill_path.parent) + if should_exclude(arcname): + print(f" Skipped: {arcname}") + continue + zipf.write(file_path, arcname) + print(f" Added: {arcname}") + + print(f"\nSuccessfully packaged skill to: {skill_filename}") + return skill_filename except Exception as e: - print(f"❌ Error creating zip file: {e}") + print(f"Error creating .skill file: {e}") return None @@ -147,7 +173,7 @@ def main(): skill_path = sys.argv[1] output_dir = sys.argv[2] if len(sys.argv) > 2 else None - print(f"📦 Packaging skill: {skill_path}") + print(f"Packaging skill: {skill_path}") if output_dir: print(f" Output directory: {output_dir}") print() diff --git a/skill-creator/scripts/quick_validate.py b/skill-creator/scripts/quick_validate.py index 2d022000..78eac04b 100755 --- a/skill-creator/scripts/quick_validate.py +++ b/skill-creator/scripts/quick_validate.py @@ -6,6 +6,7 @@ import sys import os import re +import yaml from pathlib import Path @@ -57,7 +58,7 @@ def find_path_references(content: str) -> list[str]: if any(x in line_lower for x in [ 'example:', 'examples:', 'e.g.', 'for example', '- **example', '- example:', 'such as', - 'pattern:', 'usage:', '❌', '✅', + 'pattern:', 'usage:', '\u274c', '\u2705', '- **allowed', '- **best practice', 'would be helpful', 'like `scripts/', 'like `references/', 'like `assets/', ]): @@ -92,6 +93,14 @@ def validate_path_references(skill_path: Path, content: str) -> tuple[bool, list return len(missing) == 0, missing +# Define allowed properties (union of official and our extensions) +ALLOWED_PROPERTIES = { + 'name', 'description', 'license', 'allowed-tools', 'metadata', + 'compatibility', 'context', 'agent', 'disable-model-invocation', + 'user-invocable', 'model', 'argument-hint', 'hooks', +} + + def validate_skill(skill_path): """Basic validation of a skill""" skill_path = Path(skill_path) @@ -111,10 +120,10 @@ def validate_skill(skill_path): if not match: return False, "Invalid frontmatter format" - frontmatter = match.group(1) + frontmatter_text = match.group(1) # Check for invalid indentation characters in frontmatter - invalid_indent = find_invalid_frontmatter_indentation(frontmatter) + invalid_indent = find_invalid_frontmatter_indentation(frontmatter_text) if invalid_indent: samples = ", ".join( f"line {line_no} ({describe_whitespace(ch)})" @@ -126,29 +135,62 @@ def validate_skill(skill_path): f"Found: {samples}{more}" ) + # Parse YAML frontmatter + try: + frontmatter = yaml.safe_load(frontmatter_text) + if not isinstance(frontmatter, dict): + return False, "Frontmatter must be a YAML dictionary" + except yaml.YAMLError as e: + return False, f"Invalid YAML in frontmatter: {e}" + + # Check for unexpected properties + unexpected_keys = set(frontmatter.keys()) - ALLOWED_PROPERTIES + if unexpected_keys: + return False, ( + f"Unexpected key(s) in SKILL.md frontmatter: {', '.join(sorted(unexpected_keys))}. " + f"Allowed properties are: {', '.join(sorted(ALLOWED_PROPERTIES))}" + ) + # Check required fields - if 'name:' not in frontmatter: - return False, "Missing 'name' in frontmatter" - if 'description:' not in frontmatter: + if 'description' not in frontmatter: return False, "Missing 'description' in frontmatter" - # Extract name for validation - name_match = re.search(r'name:\s*(.+)', frontmatter) - if name_match: - name = name_match.group(1).strip() - # Check naming convention (hyphen-case: lowercase with hyphens) - if not re.match(r'^[a-z0-9-]+$', name): - return False, f"Name '{name}' should be hyphen-case (lowercase letters, digits, and hyphens only)" - if name.startswith('-') or name.endswith('-') or '--' in name: - return False, f"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens" + # Extract name for validation (optional per official spec, but validate if present) + name = frontmatter.get('name', '') + if isinstance(name, str): + name = name.strip() + if name: + # Check naming convention (kebab-case: lowercase with hyphens) + if not re.match(r'^[a-z0-9-]+$', name): + return False, f"Name '{name}' should be kebab-case (lowercase letters, digits, and hyphens only)" + if name.startswith('-') or name.endswith('-') or '--' in name: + return False, f"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens" + # Check name length (max 64 characters per spec) + if len(name) > 64: + return False, f"Name is too long ({len(name)} characters). Maximum is 64 characters." + elif name is not None: + return False, f"Name must be a string, got {type(name).__name__}" # Extract and validate description - desc_match = re.search(r'description:\s*(.+)', frontmatter) - if desc_match: - description = desc_match.group(1).strip() + description = frontmatter.get('description', '') + if not isinstance(description, str): + return False, f"Description must be a string, got {type(description).__name__}" + description = description.strip() + if description: # Check for angle brackets if '<' in description or '>' in description: return False, "Description cannot contain angle brackets (< or >)" + # Check description length (max 1024 characters per spec) + if len(description) > 1024: + return False, f"Description is too long ({len(description)} characters). Maximum is 1024 characters." + + # Validate compatibility field if present (optional) + compatibility = frontmatter.get('compatibility', '') + if compatibility: + if not isinstance(compatibility, str): + return False, f"Compatibility must be a string, got {type(compatibility).__name__}" + if len(compatibility) > 500: + return False, f"Compatibility is too long ({len(compatibility)} characters). Maximum is 500 characters." # Validate path references exist paths_valid, missing_paths = validate_path_references(skill_path, content) @@ -161,7 +203,7 @@ def validate_skill(skill_path): if len(sys.argv) != 2: print("Usage: python quick_validate.py ") sys.exit(1) - + valid, message = validate_skill(sys.argv[1]) print(message) sys.exit(0 if valid else 1) diff --git a/skill-creator/scripts/run_eval.py b/skill-creator/scripts/run_eval.py new file mode 100755 index 00000000..e58c70be --- /dev/null +++ b/skill-creator/scripts/run_eval.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +"""Run trigger evaluation for a skill description. + +Tests whether a skill's description causes Claude to trigger (read the skill) +for a set of queries. Outputs results as JSON. +""" + +import argparse +import json +import os +import select +import subprocess +import sys +import time +import uuid +from concurrent.futures import ProcessPoolExecutor, as_completed +from pathlib import Path + +from scripts.utils import parse_skill_md + + +def find_project_root() -> Path: + """Find the project root by walking up from cwd looking for .claude/. + + Mimics how Claude Code discovers its project root, so the command file + we create ends up where claude -p will look for it. + """ + current = Path.cwd() + for parent in [current, *current.parents]: + if (parent / ".claude").is_dir(): + return parent + return current + + +def run_single_query( + query: str, + skill_name: str, + skill_description: str, + timeout: int, + project_root: str, + model: str | None = None, +) -> bool: + """Run a single query and return whether the skill was triggered. + + Creates a command file in .claude/commands/ so it appears in Claude's + available_skills list, then runs `claude -p` with the raw query. + Uses --include-partial-messages to detect triggering early from + stream events (content_block_start) rather than waiting for the + full assistant message, which only arrives after tool execution. + """ + unique_id = uuid.uuid4().hex[:8] + clean_name = f"{skill_name}-skill-{unique_id}" + project_commands_dir = Path(project_root) / ".claude" / "commands" + command_file = project_commands_dir / f"{clean_name}.md" + + try: + project_commands_dir.mkdir(parents=True, exist_ok=True) + # Use YAML block scalar to avoid breaking on quotes in description + indented_desc = "\n ".join(skill_description.split("\n")) + command_content = ( + f"---\n" + f"description: |\n" + f" {indented_desc}\n" + f"---\n\n" + f"# {skill_name}\n\n" + f"This skill handles: {skill_description}\n" + ) + command_file.write_text(command_content) + + cmd = [ + "claude", + "-p", query, + "--output-format", "stream-json", + "--verbose", + "--include-partial-messages", + ] + if model: + cmd.extend(["--model", model]) + + # Remove CLAUDECODE env var to allow nesting claude -p inside a + # Claude Code session. The guard is for interactive terminal conflicts; + # programmatic subprocess usage is safe. + env = {k: v for k, v in os.environ.items() if k != "CLAUDECODE"} + + process = subprocess.Popen( + cmd, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + cwd=project_root, + env=env, + ) + + triggered = False + start_time = time.time() + buffer = "" + # Track state for stream event detection + pending_tool_name = None + accumulated_json = "" + + try: + while time.time() - start_time < timeout: + if process.poll() is not None: + remaining = process.stdout.read() + if remaining: + buffer += remaining.decode("utf-8", errors="replace") + break + + ready, _, _ = select.select([process.stdout], [], [], 1.0) + if not ready: + continue + + chunk = os.read(process.stdout.fileno(), 8192) + if not chunk: + break + buffer += chunk.decode("utf-8", errors="replace") + + while "\n" in buffer: + line, buffer = buffer.split("\n", 1) + line = line.strip() + if not line: + continue + + try: + event = json.loads(line) + except json.JSONDecodeError: + continue + + # Early detection via stream events + if event.get("type") == "stream_event": + se = event.get("event", {}) + se_type = se.get("type", "") + + if se_type == "content_block_start": + cb = se.get("content_block", {}) + if cb.get("type") == "tool_use": + tool_name = cb.get("name", "") + if tool_name in ("Skill", "Read"): + pending_tool_name = tool_name + accumulated_json = "" + else: + return False + + elif se_type == "content_block_delta" and pending_tool_name: + delta = se.get("delta", {}) + if delta.get("type") == "input_json_delta": + accumulated_json += delta.get("partial_json", "") + if clean_name in accumulated_json: + return True + + elif se_type in ("content_block_stop", "message_stop"): + if pending_tool_name: + return clean_name in accumulated_json + if se_type == "message_stop": + return False + + # Fallback: full assistant message + elif event.get("type") == "assistant": + message = event.get("message", {}) + for content_item in message.get("content", []): + if content_item.get("type") != "tool_use": + continue + tool_name = content_item.get("name", "") + tool_input = content_item.get("input", {}) + if tool_name == "Skill" and clean_name in tool_input.get("skill", ""): + triggered = True + elif tool_name == "Read" and clean_name in tool_input.get("file_path", ""): + triggered = True + return triggered + + elif event.get("type") == "result": + return triggered + finally: + # Clean up process on any exit path (return, exception, timeout) + if process.poll() is None: + process.kill() + process.wait() + + return triggered + finally: + if command_file.exists(): + command_file.unlink() + + +def run_eval( + eval_set: list[dict], + skill_name: str, + description: str, + num_workers: int, + timeout: int, + project_root: Path, + runs_per_query: int = 1, + trigger_threshold: float = 0.5, + model: str | None = None, +) -> dict: + """Run the full eval set and return results.""" + results = [] + + with ProcessPoolExecutor(max_workers=num_workers) as executor: + future_to_info = {} + for item in eval_set: + for run_idx in range(runs_per_query): + future = executor.submit( + run_single_query, + item["query"], + skill_name, + description, + timeout, + str(project_root), + model, + ) + future_to_info[future] = (item, run_idx) + + query_triggers: dict[str, list[bool]] = {} + query_items: dict[str, dict] = {} + for future in as_completed(future_to_info): + item, _ = future_to_info[future] + query = item["query"] + query_items[query] = item + if query not in query_triggers: + query_triggers[query] = [] + try: + query_triggers[query].append(future.result()) + except Exception as e: + print(f"Warning: query failed: {e}", file=sys.stderr) + query_triggers[query].append(False) + + for query, triggers in query_triggers.items(): + item = query_items[query] + trigger_rate = sum(triggers) / len(triggers) + should_trigger = item["should_trigger"] + if should_trigger: + did_pass = trigger_rate >= trigger_threshold + else: + did_pass = trigger_rate < trigger_threshold + results.append({ + "query": query, + "should_trigger": should_trigger, + "trigger_rate": trigger_rate, + "triggers": sum(triggers), + "runs": len(triggers), + "pass": did_pass, + }) + + passed = sum(1 for r in results if r["pass"]) + total = len(results) + + return { + "skill_name": skill_name, + "description": description, + "results": results, + "summary": { + "total": total, + "passed": passed, + "failed": total - passed, + }, + } + + +def main(): + parser = argparse.ArgumentParser(description="Run trigger evaluation for a skill description") + parser.add_argument("--eval-set", required=True, help="Path to eval set JSON file") + parser.add_argument("--skill-path", required=True, help="Path to skill directory") + parser.add_argument("--description", default=None, help="Override description to test") + parser.add_argument("--num-workers", type=int, default=10, help="Number of parallel workers") + parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds") + parser.add_argument("--runs-per-query", type=int, default=3, help="Number of runs per query") + parser.add_argument("--trigger-threshold", type=float, default=0.5, help="Trigger rate threshold") + parser.add_argument("--model", default=None, help="Model to use for claude -p (default: user's configured model)") + parser.add_argument("--verbose", action="store_true", help="Print progress to stderr") + args = parser.parse_args() + + eval_set = json.loads(Path(args.eval_set).read_text()) + skill_path = Path(args.skill_path) + + if not (skill_path / "SKILL.md").exists(): + print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr) + sys.exit(1) + + name, original_description, content = parse_skill_md(skill_path) + description = args.description or original_description + project_root = find_project_root() + + if args.verbose: + print(f"Evaluating: {description}", file=sys.stderr) + + output = run_eval( + eval_set=eval_set, + skill_name=name, + description=description, + num_workers=args.num_workers, + timeout=args.timeout, + project_root=project_root, + runs_per_query=args.runs_per_query, + trigger_threshold=args.trigger_threshold, + model=args.model, + ) + + if args.verbose: + summary = output["summary"] + print(f"Results: {summary['passed']}/{summary['total']} passed", file=sys.stderr) + for r in output["results"]: + status = "PASS" if r["pass"] else "FAIL" + rate_str = f"{r['triggers']}/{r['runs']}" + print(f" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:70]}", file=sys.stderr) + + print(json.dumps(output, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/skill-creator/scripts/run_loop.py b/skill-creator/scripts/run_loop.py new file mode 100755 index 00000000..36f9b4e0 --- /dev/null +++ b/skill-creator/scripts/run_loop.py @@ -0,0 +1,332 @@ +#!/usr/bin/env python3 +"""Run the eval + improve loop until all pass or max iterations reached. + +Combines run_eval.py and improve_description.py in a loop, tracking history +and returning the best description found. Supports train/test split to prevent +overfitting. +""" + +import argparse +import json +import random +import sys +import tempfile +import time +import webbrowser +from pathlib import Path + +import anthropic + +from scripts.generate_report import generate_html +from scripts.improve_description import improve_description +from scripts.run_eval import find_project_root, run_eval +from scripts.utils import parse_skill_md + + +def split_eval_set(eval_set: list[dict], holdout: float, seed: int = 42) -> tuple[list[dict], list[dict]]: + """Split eval set into train and test sets, stratified by should_trigger.""" + random.seed(seed) + + # Separate by should_trigger + trigger = [e for e in eval_set if e["should_trigger"]] + no_trigger = [e for e in eval_set if not e["should_trigger"]] + + # Shuffle each group + random.shuffle(trigger) + random.shuffle(no_trigger) + + # Calculate split points + n_trigger_test = max(1, int(len(trigger) * holdout)) + n_no_trigger_test = max(1, int(len(no_trigger) * holdout)) + + # Split + test_set = trigger[:n_trigger_test] + no_trigger[:n_no_trigger_test] + train_set = trigger[n_trigger_test:] + no_trigger[n_no_trigger_test:] + + return train_set, test_set + + +def run_loop( + eval_set: list[dict], + skill_path: Path, + description_override: str | None, + num_workers: int, + timeout: int, + max_iterations: int, + runs_per_query: int, + trigger_threshold: float, + holdout: float, + model: str, + verbose: bool, + live_report_path: Path | None = None, + log_dir: Path | None = None, +) -> dict: + """Run the eval + improvement loop.""" + project_root = find_project_root() + name, original_description, content = parse_skill_md(skill_path) + current_description = description_override or original_description + + # Split into train/test if holdout > 0 + if holdout > 0: + train_set, test_set = split_eval_set(eval_set, holdout) + if verbose: + print(f"Split: {len(train_set)} train, {len(test_set)} test (holdout={holdout})", file=sys.stderr) + else: + train_set = eval_set + test_set = [] + + client = anthropic.Anthropic() + history = [] + exit_reason = "unknown" + + for iteration in range(1, max_iterations + 1): + if verbose: + print(f"\n{'='*60}", file=sys.stderr) + print(f"Iteration {iteration}/{max_iterations}", file=sys.stderr) + print(f"Description: {current_description}", file=sys.stderr) + print(f"{'='*60}", file=sys.stderr) + + # Evaluate train + test together in one batch for parallelism + all_queries = train_set + test_set + t0 = time.time() + all_results = run_eval( + eval_set=all_queries, + skill_name=name, + description=current_description, + num_workers=num_workers, + timeout=timeout, + project_root=project_root, + runs_per_query=runs_per_query, + trigger_threshold=trigger_threshold, + model=model, + ) + eval_elapsed = time.time() - t0 + + # Split results back into train/test by matching queries + train_queries_set = {q["query"] for q in train_set} + train_result_list = [r for r in all_results["results"] if r["query"] in train_queries_set] + test_result_list = [r for r in all_results["results"] if r["query"] not in train_queries_set] + + train_passed = sum(1 for r in train_result_list if r["pass"]) + train_total = len(train_result_list) + train_summary = {"passed": train_passed, "failed": train_total - train_passed, "total": train_total} + train_results = {"results": train_result_list, "summary": train_summary} + + if test_set: + test_passed = sum(1 for r in test_result_list if r["pass"]) + test_total = len(test_result_list) + test_summary = {"passed": test_passed, "failed": test_total - test_passed, "total": test_total} + test_results = {"results": test_result_list, "summary": test_summary} + else: + test_results = None + test_summary = None + + history.append({ + "iteration": iteration, + "description": current_description, + "train_passed": train_summary["passed"], + "train_failed": train_summary["failed"], + "train_total": train_summary["total"], + "train_results": train_results["results"], + "test_passed": test_summary["passed"] if test_summary else None, + "test_failed": test_summary["failed"] if test_summary else None, + "test_total": test_summary["total"] if test_summary else None, + "test_results": test_results["results"] if test_results else None, + # For backward compat with report generator + "passed": train_summary["passed"], + "failed": train_summary["failed"], + "total": train_summary["total"], + "results": train_results["results"], + }) + + # Write live report if path provided + if live_report_path: + partial_output = { + "original_description": original_description, + "best_description": current_description, + "best_score": "in progress", + "iterations_run": len(history), + "holdout": holdout, + "train_size": len(train_set), + "test_size": len(test_set), + "history": history, + } + live_report_path.write_text(generate_html(partial_output, auto_refresh=True, skill_name=name)) + + if verbose: + def print_eval_stats(label, results, elapsed): + pos = [r for r in results if r["should_trigger"]] + neg = [r for r in results if not r["should_trigger"]] + tp = sum(r["triggers"] for r in pos) + pos_runs = sum(r["runs"] for r in pos) + fn = pos_runs - tp + fp = sum(r["triggers"] for r in neg) + neg_runs = sum(r["runs"] for r in neg) + tn = neg_runs - fp + total = tp + tn + fp + fn + precision = tp / (tp + fp) if (tp + fp) > 0 else 1.0 + recall = tp / (tp + fn) if (tp + fn) > 0 else 1.0 + accuracy = (tp + tn) / total if total > 0 else 0.0 + print(f"{label}: {tp+tn}/{total} correct, precision={precision:.0%} recall={recall:.0%} accuracy={accuracy:.0%} ({elapsed:.1f}s)", file=sys.stderr) + for r in results: + status = "PASS" if r["pass"] else "FAIL" + rate_str = f"{r['triggers']}/{r['runs']}" + print(f" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:60]}", file=sys.stderr) + + print_eval_stats("Train", train_results["results"], eval_elapsed) + if test_summary: + print_eval_stats("Test ", test_results["results"], 0) + + if train_summary["failed"] == 0: + exit_reason = f"all_passed (iteration {iteration})" + if verbose: + print(f"\nAll train queries passed on iteration {iteration}!", file=sys.stderr) + break + + if iteration == max_iterations: + exit_reason = f"max_iterations ({max_iterations})" + if verbose: + print(f"\nMax iterations reached ({max_iterations}).", file=sys.stderr) + break + + # Improve the description based on train results + if verbose: + print(f"\nImproving description...", file=sys.stderr) + + t0 = time.time() + # Strip test scores from history so improvement model can't see them + blinded_history = [ + {k: v for k, v in h.items() if not k.startswith("test_")} + for h in history + ] + new_description = improve_description( + client=client, + skill_name=name, + skill_content=content, + current_description=current_description, + eval_results=train_results, + history=blinded_history, + model=model, + log_dir=log_dir, + iteration=iteration, + ) + improve_elapsed = time.time() - t0 + + if verbose: + print(f"Proposed ({improve_elapsed:.1f}s): {new_description}", file=sys.stderr) + + current_description = new_description + + # Find the best iteration by TEST score (or train if no test set) + if test_set: + best = max(history, key=lambda h: h["test_passed"] or 0) + best_score = f"{best['test_passed']}/{best['test_total']}" + else: + best = max(history, key=lambda h: h["train_passed"]) + best_score = f"{best['train_passed']}/{best['train_total']}" + + if verbose: + print(f"\nExit reason: {exit_reason}", file=sys.stderr) + print(f"Best score: {best_score} (iteration {best['iteration']})", file=sys.stderr) + + return { + "exit_reason": exit_reason, + "original_description": original_description, + "best_description": best["description"], + "best_score": best_score, + "best_train_score": f"{best['train_passed']}/{best['train_total']}", + "best_test_score": f"{best['test_passed']}/{best['test_total']}" if test_set else None, + "final_description": current_description, + "iterations_run": len(history), + "holdout": holdout, + "train_size": len(train_set), + "test_size": len(test_set), + "history": history, + } + + +def main(): + parser = argparse.ArgumentParser(description="Run eval + improve loop") + parser.add_argument("--eval-set", required=True, help="Path to eval set JSON file") + parser.add_argument("--skill-path", required=True, help="Path to skill directory") + parser.add_argument("--description", default=None, help="Override starting description") + parser.add_argument("--num-workers", type=int, default=10, help="Number of parallel workers") + parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds") + parser.add_argument("--max-iterations", type=int, default=5, help="Max improvement iterations") + parser.add_argument("--runs-per-query", type=int, default=3, help="Number of runs per query") + parser.add_argument("--trigger-threshold", type=float, default=0.5, help="Trigger rate threshold") + parser.add_argument("--holdout", type=float, default=0.4, help="Fraction of eval set to hold out for testing (0 to disable)") + parser.add_argument("--model", required=True, help="Model for improvement") + parser.add_argument("--verbose", action="store_true", help="Print progress to stderr") + parser.add_argument("--report", default="auto", help="Generate HTML report at this path (default: 'auto' for temp file, 'none' to disable)") + parser.add_argument("--results-dir", default=None, help="Save all outputs (results.json, report.html, log.txt) to a timestamped subdirectory here") + args = parser.parse_args() + + eval_set = json.loads(Path(args.eval_set).read_text()) + skill_path = Path(args.skill_path) + + if not (skill_path / "SKILL.md").exists(): + print(f"Error: No SKILL.md found at {skill_path}", file=sys.stderr) + sys.exit(1) + + name, _, _ = parse_skill_md(skill_path) + + # Set up live report path + if args.report != "none": + if args.report == "auto": + timestamp = time.strftime("%Y%m%d_%H%M%S") + live_report_path = Path(tempfile.gettempdir()) / f"skill_description_report_{skill_path.name}_{timestamp}.html" + else: + live_report_path = Path(args.report) + # Open the report immediately so the user can watch + live_report_path.write_text("

Starting optimization loop...

") + webbrowser.open(str(live_report_path)) + else: + live_report_path = None + + # Determine output directory (create before run_loop so logs can be written) + if args.results_dir: + timestamp = time.strftime("%Y-%m-%d_%H%M%S") + results_dir = Path(args.results_dir) / timestamp + results_dir.mkdir(parents=True, exist_ok=True) + else: + results_dir = None + + log_dir = results_dir / "logs" if results_dir else None + + output = run_loop( + eval_set=eval_set, + skill_path=skill_path, + description_override=args.description, + num_workers=args.num_workers, + timeout=args.timeout, + max_iterations=args.max_iterations, + runs_per_query=args.runs_per_query, + trigger_threshold=args.trigger_threshold, + holdout=args.holdout, + model=args.model, + verbose=args.verbose, + live_report_path=live_report_path, + log_dir=log_dir, + ) + + # Save JSON output + json_output = json.dumps(output, indent=2) + print(json_output) + if results_dir: + (results_dir / "results.json").write_text(json_output) + + # Write final HTML report (without auto-refresh) + if live_report_path: + live_report_path.write_text(generate_html(output, auto_refresh=False, skill_name=name)) + print(f"\nReport: {live_report_path}", file=sys.stderr) + + if results_dir and live_report_path: + (results_dir / "report.html").write_text(generate_html(output, auto_refresh=False, skill_name=name)) + + if results_dir: + print(f"Results saved to: {results_dir}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/skill-creator/scripts/utils.py b/skill-creator/scripts/utils.py new file mode 100644 index 00000000..51b6a07d --- /dev/null +++ b/skill-creator/scripts/utils.py @@ -0,0 +1,47 @@ +"""Shared utilities for skill-creator scripts.""" + +from pathlib import Path + + + +def parse_skill_md(skill_path: Path) -> tuple[str, str, str]: + """Parse a SKILL.md file, returning (name, description, full_content).""" + content = (skill_path / "SKILL.md").read_text() + lines = content.split("\n") + + if lines[0].strip() != "---": + raise ValueError("SKILL.md missing frontmatter (no opening ---)") + + end_idx = None + for i, line in enumerate(lines[1:], start=1): + if line.strip() == "---": + end_idx = i + break + + if end_idx is None: + raise ValueError("SKILL.md missing frontmatter (no closing ---)") + + name = "" + description = "" + frontmatter_lines = lines[1:end_idx] + i = 0 + while i < len(frontmatter_lines): + line = frontmatter_lines[i] + if line.startswith("name:"): + name = line[len("name:"):].strip().strip('"').strip("'") + elif line.startswith("description:"): + value = line[len("description:"):].strip() + # Handle YAML multiline indicators (>, |, >-, |-) + if value in (">", "|", ">-", "|-"): + continuation_lines: list[str] = [] + i += 1 + while i < len(frontmatter_lines) and (frontmatter_lines[i].startswith(" ") or frontmatter_lines[i].startswith("\t")): + continuation_lines.append(frontmatter_lines[i].strip()) + i += 1 + description = " ".join(continuation_lines) + continue + else: + description = value.strip('"').strip("'") + i += 1 + + return name, description, content diff --git a/transcript-fixer/scripts/cli/argument_parser.py b/transcript-fixer/scripts/cli/argument_parser.py index 086d67f9..a697e77e 100644 --- a/transcript-fixer/scripts/cli/argument_parser.py +++ b/transcript-fixer/scripts/cli/argument_parser.py @@ -62,8 +62,8 @@ def create_argument_parser() -> argparse.ArgumentParser: ) parser.add_argument( "--domain", "-d", - default="general", - help="Correction domain" + default=None, + help="Correction domain (default: all domains)" ) # Learning commands diff --git a/transcript-fixer/scripts/cli/commands.py b/transcript-fixer/scripts/cli/commands.py index 4caa832c..5b836b45 100644 --- a/transcript-fixer/scripts/cli/commands.py +++ b/transcript-fixer/scripts/cli/commands.py @@ -58,11 +58,22 @@ def cmd_list_corrections(args: argparse.Namespace) -> None: service = _get_service() corrections = service.get_corrections(args.domain) - print(f"\n📋 Corrections (domain: {args.domain})") + if args.domain: + header = f"domain: {args.domain}, {len(corrections)} total" + else: + header = f"all domains, {len(corrections)} total" + + print(f"\n📋 Corrections ({header})") print("=" * 60) - for wrong, correct in sorted(corrections.items()): - print(f" '{wrong}' → '{correct}'") - print(f"\nTotal: {len(corrections)} corrections\n") + + if args.domain: + for wrong, correct in sorted(corrections.items()): + print(f" '{wrong}' → '{correct}'") + else: + all_corrections = service.repository.get_all_corrections(active_only=True) + for c in all_corrections: + print(f" [{c.domain}] '{c.from_text}' → '{c.to_text}'") + print() def cmd_run_correction(args: argparse.Namespace) -> None: @@ -83,12 +94,23 @@ def cmd_run_correction(args: argparse.Namespace) -> None: # Load corrections and rules corrections = service.get_corrections(args.domain) context_rules = service.load_context_rules() + domain_stats = service.get_domain_stats() # Read input file print(f"📖 Reading: {input_path.name}") with open(input_path, 'r', encoding='utf-8') as f: original_text = f.read() - print(f" File size: {len(original_text):,} characters\n") + print(f" File size: {len(original_text):,} characters") + + # Show domain loading info + if args.domain: + print(f"📚 Loaded {len(corrections)} corrections (domain: {args.domain})") + elif domain_stats: + parts = ", ".join(f"{d}: {n}" for d, n in sorted(domain_stats.items())) + print(f"📚 Loaded {len(corrections)} corrections ({parts})") + else: + print(f"📚 No corrections in database") + print() # Stage 1: Dictionary corrections stage1_changes = [] @@ -109,7 +131,17 @@ def cmd_run_correction(args: argparse.Namespace) -> None: stage1_file = output_dir / f"{input_path.stem}_stage1.md" with open(stage1_file, 'w', encoding='utf-8') as f: f.write(stage1_text) - print(f"💾 Saved: {stage1_file.name}\n") + print(f"💾 Saved: {stage1_file.name}") + + # Hint when 0 corrections and other domains have rules + if summary['total_changes'] == 0 and args.domain and domain_stats: + other = {d: n for d, n in domain_stats.items() if d != args.domain} + if other: + parts = ", ".join(f"{d} ({n})" for d, n in sorted(other.items())) + total = sum(other.values()) + print(f"hint: no rules in domain '{args.domain}'. Available: {parts}") + print(f"hint: run without --domain to use all {total} rules") + print() # Stage 2: AI corrections stage2_changes = [] diff --git a/transcript-fixer/scripts/core/correction_service.py b/transcript-fixer/scripts/core/correction_service.py index 4d53c50c..d8417a37 100644 --- a/transcript-fixer/scripts/core/correction_service.py +++ b/transcript-fixer/scripts/core/correction_service.py @@ -382,6 +382,14 @@ def export_corrections(self, domain: str = "general") -> Dict[str, str]: # ==================== Statistics and Reporting ==================== + def get_domain_stats(self) -> Dict[str, int]: + """Get count of active corrections per domain.""" + all_corrections = self.repository.get_all_corrections(active_only=True) + stats: Dict[str, int] = {} + for c in all_corrections: + stats[c.domain] = stats.get(c.domain, 0) + 1 + return stats + def get_statistics(self, domain: Optional[str] = None) -> Dict[str, any]: """ Get correction statistics. diff --git a/transcript-fixer/scripts/tests/test_correction_service.py b/transcript-fixer/scripts/tests/test_correction_service.py index d81963db..a5521e54 100644 --- a/transcript-fixer/scripts/tests/test_correction_service.py +++ b/transcript-fixer/scripts/tests/test_correction_service.py @@ -160,6 +160,31 @@ def test_remove_nonexistent_correction(self): success = self.service.remove_correction("nonexistent", "general") self.assertFalse(success) + # ==================== Domain Stats Tests ==================== + + def test_get_corrections_all_domains(self): + """domain=None loads all domains.""" + self.service.add_correction("a", "b", "general") + self.service.add_correction("c", "d", "finance") + all_corr = self.service.get_corrections(None) + self.assertEqual(len(all_corr), 2) + self.assertIn("a", all_corr) + self.assertIn("c", all_corr) + + def test_get_domain_stats(self): + """get_domain_stats returns per-domain counts.""" + self.service.add_correction("a", "b", "general") + self.service.add_correction("c", "d", "finance") + self.service.add_correction("e", "f", "finance") + stats = self.service.get_domain_stats() + self.assertEqual(stats["general"], 1) + self.assertEqual(stats["finance"], 2) + + def test_get_domain_stats_empty(self): + """get_domain_stats returns empty dict when no corrections.""" + stats = self.service.get_domain_stats() + self.assertEqual(stats, {}) + # ==================== Import/Export Tests ==================== def test_import_corrections(self): diff --git a/tunnel-doctor/SKILL.md b/tunnel-doctor/SKILL.md index 865fbd17..f0e231a3 100644 --- a/tunnel-doctor/SKILL.md +++ b/tunnel-doctor/SKILL.md @@ -27,6 +27,7 @@ Proxy/VPN tools on macOS create conflicts at five independent layers. Layers 1-3 Determine which scenario applies: - **Browser returns HTTP 503, but `curl` and SSH both work** → System proxy bypass conflict (Step 2C) +- **`local.` fails in browser/default `curl`, but direct/no-proxy request works** → Local vanity domain proxy interception (Step 2C-1) - **Tailscale ping works, SSH works, but curl/HTTP times out** → HTTP proxy env var conflict (Step 2A) - **Tailscale ping works, SSH/TCP times out** → Route conflict (Step 2B) - **Remote dev server auth redirects to `localhost` → browser can't follow** → SSH tunnel needed (Step 2D) @@ -45,6 +46,25 @@ Determine which scenario applies: - If host `curl https://...` works but `docker pull` times out → Layer 5 (VM proxy propagation). - If DNS resolves to `198.18.x.x` virtual IPs → TUN DNS hijack (Step 2H). +### Fast Path: Run Automated Checks + +For common macOS conflicts (env proxy, system proxy exceptions, direct/proxy path split, local TLS trust), run: + +```bash +python3 scripts/quick_diagnose.py --host local.claude4.dev --url https://local.claude4.dev/health +``` + +Optional route ownership check for a Tailscale destination: + +```bash +python3 scripts/quick_diagnose.py --host --url http://:/health --tailscale-ip <100.x.x.x> +``` + +Interpretation: +- `direct=PASS` + `forced_proxy=FAIL` = host must bypass proxy (`skip-proxy` + `NO_PROXY`). +- `strict_tls=FAIL` + `direct=PASS` = path is reachable; trust issue only (install/trust local CA). +- `host in scutil exceptions: no` = browser/system clients still likely proxied. + ### Step 2A: Fix HTTP Proxy Environment Variables Check if proxy env vars are intercepting Tailscale HTTP traffic: @@ -150,6 +170,41 @@ skip-proxy = 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 100.64.0.0/10, localhost - `skip-proxy`: Bypasses the HTTP proxy layer only. Traffic still flows through the TUN interface and Tailscale utun handles it. Safe. - `tun-excluded-routes`: Removes the CIDR from the TUN routing entirely. This creates a competing `en0` route that overrides Tailscale. Breaks everything. +#### Step 2C-1: Fix Local Vanity Domain Interception (`local.`) + +**Symptom**: `https://local.` fails in browser or default `curl`, but succeeds with direct/no-proxy command: + +```bash +env -u http_proxy -u https_proxy curl -k -I https://local./health +# -> 200 +curl -I https://local./health +# -> proxy CONNECT then TLS reset/failure +``` + +**Root cause**: The domain is routed through system/shell proxy instead of local direct path. + +**Fix**: +1. Add domain to proxy app bypass list (`skip-proxy` for Shadowrocket). +2. Add domain to shell bypass list (`NO_PROXY`/`no_proxy`). +3. If local TLS uses internal CA, trust the local root certificate. + +```bash +# ~/.zshrc +export NO_PROXY=localhost,127.0.0.1,.ts.net,100.64.0.0/10,192.168.*,10.*,172.16.*,local.,www.local. +export no_proxy="$NO_PROXY" +``` + +**Verification**: + +```bash +python3 scripts/quick_diagnose.py --host local. --url https://local./health +``` + +Expected: +- `host in NO_PROXY: yes` +- `host in scutil exceptions: yes` +- `ambient=PASS` and `direct=PASS` + ### Step 2D: Fix Auth Redirect for Remote Dev (SSH Tunnel) **Symptom**: Dev server runs on a remote machine (e.g., Mac Mini via Tailscale). You access `http://:3010` in the browser. Login/signup works, but after auth, the app redirects to `http://localhost:3010/` which fails — `localhost` on your machine isn't running the dev server. diff --git a/tunnel-doctor/scripts/quick_diagnose.py b/tunnel-doctor/scripts/quick_diagnose.py new file mode 100755 index 00000000..346ca8ca --- /dev/null +++ b/tunnel-doctor/scripts/quick_diagnose.py @@ -0,0 +1,448 @@ +#!/usr/bin/env python3 +""" +Quick tunnel/proxy conflict diagnostics for macOS. + +This script detects the most common local and Tailscale networking conflicts: +1) Shell proxy env + NO_PROXY mismatch +2) System proxy exceptions mismatch +3) Proxy path failure vs direct path success +4) Local TLS trust issues +5) Route ownership conflicts for a Tailscale IP (optional) +""" + +import argparse +import json +import os +import re +import shlex +import socket +import subprocess +import sys +from typing import Dict, List, Optional, Tuple + + +def run(cmd: List[str], env: Optional[Dict[str, str]] = None) -> Tuple[int, str, str]: + proc = subprocess.run( + cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + env=env, + ) + return proc.returncode, proc.stdout.strip(), proc.stderr.strip() + + +def split_csv(value: str) -> List[str]: + if not value: + return [] + return [item.strip() for item in value.split(",") if item.strip()] + + +def match_proxy_pattern(host: str, pattern: str) -> bool: + p = pattern.strip().lower() + h = host.strip().lower() + if not p or not h: + return False + + # CIDR / wildcard IP patterns are not domain checks. + if "/" in p and not p.startswith("*."): + return False + if re.match(r"^\d+\.\d+\.\d+\.\*$", p): + return False + + if p == h: + return True + if p.startswith("*."): + suffix = p[1:] # keep leading dot + return h.endswith(suffix) + if p.startswith("."): + return h.endswith(p) + return False + + +def has_host_bypass(host: str, patterns: List[str]) -> bool: + return any(match_proxy_pattern(host, item) for item in patterns) + + +def parse_scutil_proxy() -> Dict[str, object]: + code, stdout, _stderr = run(["scutil", "--proxy"]) + if code != 0: + return {"raw": "", "exceptions": [], "http_enabled": False, "https_enabled": False} + + raw = stdout + exceptions: List[str] = [] + for line in raw.splitlines(): + m = re.search(r"\d+\s*:\s*(.+)$", line) + if m and "ExceptionsList" not in line: + exceptions.append(m.group(1).strip()) + + http_enabled = bool(re.search(r"HTTPEnable\s*:\s*1", raw)) + https_enabled = bool(re.search(r"HTTPSEnable\s*:\s*1", raw)) + + return { + "raw": raw, + "exceptions": exceptions, + "http_enabled": http_enabled, + "https_enabled": https_enabled, + } + + +def resolve_host(host: str) -> List[str]: + ips = [] + try: + infos = socket.getaddrinfo(host, None) + for item in infos: + ip = item[4][0] + if ip not in ips: + ips.append(ip) + except socket.gaierror: + pass + return ips + + +def curl_status(url: str, timeout: int, mode: str, proxy_url: Optional[str] = None) -> Dict[str, object]: + cmd = [ + "curl", + "-k", + "-sS", + "-o", + "/dev/null", + "-w", + "%{http_code}", + "--max-time", + str(timeout), + url, + ] + + env = os.environ.copy() + if mode == "direct": + for key in ( + "http_proxy", + "https_proxy", + "HTTP_PROXY", + "HTTPS_PROXY", + "all_proxy", + "ALL_PROXY", + ): + env.pop(key, None) + elif mode == "forced_proxy" and proxy_url: + cmd.extend(["--proxy", proxy_url]) + + code, stdout, stderr = run(cmd, env=env) + http_code = stdout if stdout else "000" + ok = code == 0 and http_code.isdigit() and http_code != "000" + + return { + "ok": ok, + "http_code": http_code, + "exit_code": code, + "stderr": stderr, + "command": " ".join(shlex.quote(x) for x in cmd), + } + + +def strict_tls_check(url: str, timeout: int) -> Dict[str, object]: + cmd = [ + "curl", + "-sS", + "-o", + "/dev/null", + "-w", + "%{http_code}", + "--max-time", + str(timeout), + url, + ] + env = os.environ.copy() + for key in ("http_proxy", "https_proxy", "HTTP_PROXY", "HTTPS_PROXY", "all_proxy", "ALL_PROXY"): + env.pop(key, None) + code, stdout, stderr = run(cmd, env=env) + cert_issue = "certificate" in stderr.lower() or "ssl" in stderr.lower() + return { + "ok": code == 0 and stdout != "000", + "http_code": stdout if stdout else "000", + "exit_code": code, + "stderr": stderr, + "cert_issue": cert_issue, + } + + +def route_check(tailscale_ip: str) -> Dict[str, object]: + code, stdout, stderr = run(["route", "-n", "get", tailscale_ip]) + if code != 0: + return {"ok": False, "interface": "", "gateway": "", "raw": stderr or stdout} + + interface = "" + gateway = "" + for line in stdout.splitlines(): + line = line.strip() + if line.startswith("interface:"): + interface = line.split(":", 1)[1].strip() + if line.startswith("gateway:"): + gateway = line.split(":", 1)[1].strip() + + return { + "ok": True, + "interface": interface, + "gateway": gateway, + "raw": stdout, + } + + +def pick_proxy_url() -> Optional[str]: + for key in ("http_proxy", "HTTP_PROXY", "https_proxy", "HTTPS_PROXY"): + value = os.environ.get(key) + if value: + return value + return None + + +def build_report( + host: str, + url: str, + timeout: int, + tailscale_ip: Optional[str], +) -> Dict[str, object]: + no_proxy = os.environ.get("NO_PROXY", "") + no_proxy_lc = os.environ.get("no_proxy", "") + no_proxy_entries = split_csv(no_proxy if no_proxy else no_proxy_lc) + + scutil_info = parse_scutil_proxy() + scutil_exceptions = scutil_info["exceptions"] + + proxy_url = pick_proxy_url() + direct = curl_status(url, timeout, mode="direct") + ambient = curl_status(url, timeout, mode="ambient") + forced_proxy = curl_status(url, timeout, mode="forced_proxy", proxy_url=proxy_url) if proxy_url else None + strict_tls = strict_tls_check(url, timeout) if url.startswith("https://") else None + + host_ips = resolve_host(host) + host_in_no_proxy = has_host_bypass(host, no_proxy_entries) + host_in_scutil_exceptions = has_host_bypass(host, scutil_exceptions) + + findings: List[Dict[str, str]] = [] + + if not host_ips: + findings.append( + { + "level": "error", + "title": "Host resolution failed", + "detail": f"{host} could not be resolved. Check DNS/hosts first.", + "fix": f"Add a hosts entry if this is local: 127.0.0.1 {host}", + } + ) + + if proxy_url and not host_in_no_proxy: + findings.append( + { + "level": "warn", + "title": "NO_PROXY missing target host", + "detail": f"Proxy is enabled ({proxy_url}) but NO_PROXY does not match {host}.", + "fix": ( + "Add host to NO_PROXY/no_proxy, e.g. " + f"NO_PROXY=...,{host}" + ), + } + ) + + if (scutil_info["http_enabled"] or scutil_info["https_enabled"]) and not host_in_scutil_exceptions: + findings.append( + { + "level": "warn", + "title": "System proxy exception missing target host", + "detail": f"scutil active exceptions do not include {host}.", + "fix": ( + "Add host to proxy app skip/bypass list (Shadowrocket/Clash/Surge), " + "then reload profile." + ), + } + ) + + if direct["ok"] and forced_proxy and not forced_proxy["ok"]: + findings.append( + { + "level": "error", + "title": "Proxy path is broken for target host", + "detail": ( + "Direct access works, but forced proxy tunnel fails. " + "Traffic must bypass proxy for this host." + ), + "fix": ( + f"Add {host} to both NO_PROXY and proxy app skip-proxy/DIRECT rules." + ), + } + ) + + if not ambient["ok"] and direct["ok"]: + findings.append( + { + "level": "error", + "title": "Ambient shell path fails while direct path works", + "detail": ( + "Current shell env/proxy settings break default access." + ), + "fix": ( + "Use NO_PROXY for this host, or temporarily unset proxy env for local verification." + ), + } + ) + + if strict_tls and not strict_tls["ok"] and direct["ok"] and strict_tls["cert_issue"]: + findings.append( + { + "level": "warn", + "title": "TLS trust issue detected", + "detail": "Network path is reachable, but strict TLS validation failed.", + "fix": "Trust local CA certificate (for local/internal TLS) or use a valid public cert.", + } + ) + + route_info = route_check(tailscale_ip) if tailscale_ip else None + if route_info and route_info["ok"]: + iface = str(route_info["interface"]) + if iface.startswith("en"): + findings.append( + { + "level": "error", + "title": "Possible route hijack for Tailscale destination", + "detail": f"route -n get {tailscale_ip} resolved to {iface}, not utun*.", + "fix": ( + "Check proxy TUN excluded-routes. Do not exclude 100.64.0.0/10 from TUN route table." + ), + } + ) + + summary = { + "host": host, + "url": url, + "host_ips": host_ips, + "proxy_url": proxy_url or "", + "env_no_proxy": no_proxy if no_proxy else no_proxy_lc, + "host_in_no_proxy": host_in_no_proxy, + "scutil_http_enabled": scutil_info["http_enabled"], + "scutil_https_enabled": scutil_info["https_enabled"], + "host_in_scutil_exceptions": host_in_scutil_exceptions, + "connectivity": { + "ambient": ambient, + "direct": direct, + "forced_proxy": forced_proxy, + "strict_tls": strict_tls, + }, + "tailscale_route": route_info, + "findings": findings, + } + return summary + + +def print_human(report: Dict[str, object]) -> int: + print("=== Tunnel Doctor Quick Diagnose ===") + print(f"Host: {report['host']}") + print(f"URL: {report['url']}") + ips = report["host_ips"] + print(f"Resolved IPs: {', '.join(ips) if ips else 'N/A'}") + print("") + + print("Proxy Context") + print(f"- proxy env: {report['proxy_url'] or '(not set)'}") + print(f"- host in NO_PROXY: {'yes' if report['host_in_no_proxy'] else 'no'}") + print( + "- system proxy enabled: " + f"HTTP={'yes' if report['scutil_http_enabled'] else 'no'} " + f"HTTPS={'yes' if report['scutil_https_enabled'] else 'no'}" + ) + print(f"- host in scutil exceptions: {'yes' if report['host_in_scutil_exceptions'] else 'no'}") + print("") + + conn = report["connectivity"] + print("Connectivity Checks") + for key in ("ambient", "direct", "forced_proxy", "strict_tls"): + value = conn.get(key) + if not value: + continue + ok = "PASS" if value.get("ok") else "FAIL" + print( + f"- {key:12s}: {ok} " + f"(http={value.get('http_code', '000')}, exit={value.get('exit_code', 'n/a')})" + ) + stderr = value.get("stderr") + if stderr: + print(f" stderr: {stderr}") + print("") + + route = report.get("tailscale_route") + if route: + if route.get("ok"): + print("Tailscale Route Check") + print(f"- interface: {route.get('interface') or 'N/A'}") + print(f"- gateway: {route.get('gateway') or 'N/A'}") + print("") + else: + print("Tailscale Route Check") + print(f"- failed: {route.get('raw', '')}") + print("") + + findings = report["findings"] + if not findings: + print("Result: no high-confidence conflict found.") + print("If browser still fails, verify proxy app profile mode/rule order and reload profile.") + return 0 + + print("Findings") + severity_order = {"error": 0, "warn": 1, "info": 2} + findings_sorted = sorted(findings, key=lambda x: severity_order.get(str(x.get("level")), 99)) + for idx, item in enumerate(findings_sorted, start=1): + print(f"{idx}. [{item['level'].upper()}] {item['title']}") + print(f" Detail: {item['detail']}") + print(f" Fix: {item['fix']}") + + return 1 if any(x["level"] == "error" for x in findings) else 0 + + +def main() -> int: + if sys.platform != "darwin": + print("This script is designed for macOS only.", file=sys.stderr) + return 2 + + parser = argparse.ArgumentParser( + description="Quick diagnostics for Tailscale + proxy conflicts on macOS." + ) + parser.add_argument("--host", default="local.claude4.dev", help="Target host to diagnose.") + parser.add_argument( + "--url", + default="", + help="Full URL to test. Default: https:///health", + ) + parser.add_argument( + "--timeout", + type=int, + default=8, + help="curl timeout seconds (default: 8).", + ) + parser.add_argument( + "--tailscale-ip", + default="", + help="Optional Tailscale IP for route ownership check (e.g. 100.101.102.103).", + ) + parser.add_argument( + "--json", + action="store_true", + help="Output JSON report.", + ) + args = parser.parse_args() + + url = args.url.strip() or f"https://{args.host}/health" + report = build_report( + host=args.host.strip(), + url=url, + timeout=max(args.timeout, 1), + tailscale_ip=args.tailscale_ip.strip() or None, + ) + + if args.json: + print(json.dumps(report, indent=2, ensure_ascii=False)) + return 0 + return print_human(report) + + +if __name__ == "__main__": + sys.exit(main()) From 6dc2805f0395c2b61e145f01a8375383bf574f59 Mon Sep 17 00:00:00 2001 From: daymade Date: Wed, 11 Mar 2026 13:58:39 +0800 Subject: [PATCH 002/137] fix(pdf-creator): restore list spacing preprocessor for pandoc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add _ensure_list_spacing() to handle lists without blank lines before them - Modify _md_to_html() to preprocess markdown content via stdin - Add automated test suite (scripts/tests/test_list_rendering.py) - Fix: Lists without preceding blank lines now render correctly - Original markdown files remain unmodified (preprocessing in memory only) Root cause: Pandoc requires blank lines before lists per CommonMark spec. Without preprocessing, lists following paragraphs render as plain text. Tested scenarios: ✅ Lists with blank lines (normal case) ✅ Lists without blank lines (critical fix) ✅ Ordered lists without blank lines ✅ Original file integrity preserved Co-Authored-By: Claude Opus 4.6 (1M context) --- pdf-creator/scripts/md_to_pdf.py | 36 +++- .../scripts/tests/test_list_rendering.py | 166 ++++++++++++++++++ 2 files changed, 199 insertions(+), 3 deletions(-) create mode 100755 pdf-creator/scripts/tests/test_list_rendering.py diff --git a/pdf-creator/scripts/md_to_pdf.py b/pdf-creator/scripts/md_to_pdf.py index 1e351e78..a1ce7575 100644 --- a/pdf-creator/scripts/md_to_pdf.py +++ b/pdf-creator/scripts/md_to_pdf.py @@ -19,6 +19,7 @@ import os import platform +import re import shutil import subprocess import sys @@ -146,15 +147,44 @@ """ +def _ensure_list_spacing(text: str) -> str: + """Ensure blank lines before list items for proper markdown parsing. + + Both Python markdown library and pandoc require a blank line before a list + when it follows a paragraph. Without it, list items render as plain text. + + This preprocessor adds blank lines before list items when needed, without + modifying the user's original markdown file. + """ + lines = text.split('\n') + result = [] + list_re = re.compile(r'^(\s*)([-*+]|\d+\.)\s') + for i, line in enumerate(lines): + if i > 0 and list_re.match(line): + prev = lines[i - 1] + if prev.strip() and not list_re.match(prev): + result.append('') + result.append(line) + return '\n'.join(result) + + def _md_to_html(md_file: str) -> str: - """Convert markdown to HTML using pandoc.""" + """Convert markdown to HTML using pandoc with list spacing preprocessing. + + Reads the markdown file, preprocesses it to ensure proper list spacing, + then passes the content to pandoc via stdin. The original file is not modified. + """ if not shutil.which('pandoc'): print("Error: pandoc not found. Install with: brew install pandoc", file=sys.stderr) sys.exit(1) + # Read and preprocess markdown to ensure list spacing + md_content = Path(md_file).read_text(encoding='utf-8') + md_content = _ensure_list_spacing(md_content) + result = subprocess.run( - ['pandoc', md_file, '-f', 'markdown', '-t', 'html'], - capture_output=True, text=True, + ['pandoc', '-f', 'markdown', '-t', 'html'], + input=md_content, capture_output=True, text=True, ) if result.returncode != 0: print(f"Error: pandoc failed: {result.stderr}", file=sys.stderr) diff --git a/pdf-creator/scripts/tests/test_list_rendering.py b/pdf-creator/scripts/tests/test_list_rendering.py new file mode 100755 index 00000000..a7944c97 --- /dev/null +++ b/pdf-creator/scripts/tests/test_list_rendering.py @@ -0,0 +1,166 @@ +#!/usr/bin/env python3 +""" +Test list rendering in PDF generation. + +Verifies that markdown lists are correctly rendered in PDFs, +even when they don't have blank lines before them. + +The original markdown files are NOT modified - preprocessing +happens in memory during conversion. +""" + +import subprocess +import sys +import tempfile +from pathlib import Path + +# Test markdown content with various list scenarios +TEST_MARKDOWN = """# 测试列表解析 + +## 场景1：列表前有空行（正常） + +这是一段文字。 + +- 列表项 1 +- 列表项 2 +- 列表项 3 + +## 场景2：列表前没有空行（关键测试） + +这是一段文字。 +- 列表项 1 +- 列表项 2 +- 列表项 3 + +## 场景3：有序列表前没有空行 + +这是一段文字。 +1. 第一项 +2. 第二项 +3. 第三项 + +## 场景4：有序列表前有空行（正常） + +这是一段文字。 + +1. 第一项 +2. 第二项 +3. 第三项 +""" + + +def run_test(): + """Run the list rendering test.""" + # Create temporary files + with tempfile.NamedTemporaryFile(mode='w', suffix='.md', delete=False, encoding='utf-8') as md_file: + md_file.write(TEST_MARKDOWN) + md_path = md_file.name + + pdf_path = md_path.replace('.md', '.pdf') + txt_path = md_path.replace('.md', '.txt') + + try: + # Generate PDF + script_dir = Path(__file__).parent.parent + md_to_pdf = script_dir / 'md_to_pdf.py' + + print(f"生成 PDF: {md_path} -> {pdf_path}") + result = subprocess.run( + ['uv', 'run', '--with', 'weasyprint', str(md_to_pdf), md_path, pdf_path], + capture_output=True, text=True, cwd=script_dir.parent + ) + + if result.returncode != 0: + print(f"❌ PDF 生成失败: {result.stderr}") + return False + + print(f"✅ PDF 已生成") + + # Extract text from PDF + result = subprocess.run( + ['pdftotext', pdf_path, txt_path], + capture_output=True, text=True + ) + + if result.returncode != 0: + print(f"❌ 文本提取失败: {result.stderr}") + return False + + # Read extracted text + with open(txt_path, 'r', encoding='utf-8') as f: + pdf_text = f.read() + + # Verify original file was not modified + with open(md_path, 'r', encoding='utf-8') as f: + original_content = f.read() + + if original_content != TEST_MARKDOWN: + print("❌ 原始文件被修改了！") + return False + + print("✅ 原始文件未被修改") + + # Verify list rendering + print("\n=== 列表渲染验证 ===") + + tests_passed = 0 + tests_total = 4 + + # Test 1: List with blank line before it + if '• 列表项 1' in pdf_text: + print("✅ 场景1: 列表前有空行 - 正确渲染") + tests_passed += 1 + else: + print("❌ 场景1: 列表前有空行 - 渲染失败") + + # Test 2: Critical test - list without blank line before it + scene2_start = pdf_text.find('场景2') + scene2_section = pdf_text[scene2_start:scene2_start+200] if scene2_start != -1 else "" + + if '• 列表项 1' in scene2_section and '- 列表项 1' not in scene2_section: + print("✅ 场景2: 列表前没有空行 - 正确渲染（关键测试）") + tests_passed += 1 + else: + print("❌ 场景2: 列表前没有空行 - 渲染失败") + print(f" 实际内容: {scene2_section}") + + # Test 3: Ordered list without blank line + scene3_start = pdf_text.find('场景3') + scene3_section = pdf_text[scene3_start:scene3_start+200] if scene3_start != -1 else "" + + if '1. 第一项' in scene3_section and '2. 第二项' in scene3_section: + print("✅ 场景3: 有序列表前没有空行 - 正确渲染") + tests_passed += 1 + else: + print("❌ 场景3: 有序列表前没有空行 - 渲染失败") + + # Test 4: Ordered list with blank line + if '1. 第一项' in pdf_text and '2. 第二项' in pdf_text: + print("✅ 场景4: 有序列表前有空行 - 正确渲染") + tests_passed += 1 + else: + print("❌ 场景4: 有序列表前有空行 - 渲染失败") + + print(f"\n=== 测试结果: {tests_passed}/{tests_total} 通过 ===") + + if tests_passed == tests_total: + print("\n✅ 所有测试通过！") + print(f"\n生成的文件:") + print(f" Markdown: {md_path}") + print(f" PDF: {pdf_path}") + print(f" Text: {txt_path}") + return True + else: + print(f"\n❌ {tests_total - tests_passed} 个测试失败") + return False + + except Exception as e: + print(f"❌ 测试失败: {e}") + import traceback + traceback.print_exc() + return False + + +if __name__ == '__main__': + success = run_test() + sys.exit(0 if success else 1) From 29f85d27c3abb5da5b244a090c939a6c5ddfb673 Mon Sep 17 00:00:00 2001 From: daymade Date: Wed, 11 Mar 2026 13:58:58 +0800 Subject: [PATCH 003/137] docs(github-contributor): add high-quality PR formula and investigation workflow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add 10-point high-quality PR formula based on real-world success cases - Add investigation phase workflow (post to issue before PR) - Add git history tracing techniques (git log, git blame) - Add evidence-loop pattern (reproduce → trace → link → post) - Add high-quality PR case study reference - Update PR checklist with investigation steps - Emphasize separation of concerns (detailed analysis in issue, fix summary in PR) Key principles: - Deep investigation before coding - Minimal, surgical fixes - Professional communication - No internal/irrelevant details in PR Co-Authored-By: Claude Opus 4.6 (1M context) --- github-contributor/SKILL.md | 288 ++++++++++++--- .../references/high_quality_pr_case_study.md | 328 ++++++++++++++++++ github-contributor/references/pr_checklist.md | 217 ++++++++---- 3 files changed, 705 insertions(+), 128 deletions(-) create mode 100644 github-contributor/references/high_quality_pr_case_study.md diff --git a/github-contributor/SKILL.md b/github-contributor/SKILL.md index 103c7732..70b7666a 100644 --- a/github-contributor/SKILL.md +++ b/github-contributor/SKILL.md @@ -118,6 +118,23 @@ gh search repos "topic:cli" --sort=stars --limit=20 ## PR Excellence +### The High-Quality PR Formula + +Based on real-world successful contributions to major open-source projects: + +``` +1. Deep investigation (post to issue, not PR) +2. Minimal, surgical fix (only change what's necessary) +3. Regression test (prevent future breakage) +4. CHANGELOG entry (if project uses it) +5. End-to-end validation (prove bug exists, prove fix works) +6. Clear PR structure (~50 lines, focused) +7. Professional communication +8. Separate concerns (detailed analysis in issue, fix summary in PR) +9. No internal/irrelevant details +10. Responsive to feedback +``` + ### Before Writing Code ``` @@ -127,6 +144,38 @@ Pre-PR Checklist: - [ ] Comment on issue to claim it - [ ] Understand project conventions - [ ] Set up development environment +- [ ] Trace through git history for context +- [ ] Identify root cause with evidence +``` + +### Investigation Phase (Post to Issue) + +**Do this BEFORE coding**: + +1. **Reproduce the bug** with exact commands and output +2. **Trace git history** to understand context + ```bash + git log --all --grep="keyword" --oneline + git blame file.ts | grep "relevant_line" + ``` +3. **Link related issues/PRs** that provide context +4. **Post detailed analysis to issue** (not PR) + - Timeline of related changes + - Root cause explanation + - Why previous approaches didn't work + +**Example structure**: +```markdown +## Investigation + +I traced this through the codebase history: + +1. [Date]: #[PR] introduced [feature] +2. [Date]: #[PR] added [workaround] because [reason] +3. [Date]: #[PR] changed [parameter] +4. Now: Safe to [fix] because [explanation] + +[Detailed evidence with code references] ``` ### Writing the PR @@ -140,76 +189,151 @@ docs(readme): update installation instructions for Windows refactor(validation): extract validation logic into separate module ``` -**Evidence loop**: Prove the change with a reproducible fail -> fix -> pass loop. +**Keep PR description focused** (~50 lines): +- Summary (1-2 sentences) +- Root cause (technical, with code refs) +- Changes (bullet list) +- Why it's safe +- Testing approach +- Related issues + +**Move detailed investigation to issue comments**, not PR. + +### Evidence Loop + +**Critical**: Prove the change with a reproducible fail → fix → pass loop. + +1. **Reproduce failure** with original version + ```bash + # Test with original version + npm install -g package@original-version + [command that triggers bug] + # Capture: error messages, exit codes, timestamps + ``` -- Reproduce the failure with one command. -- Apply the fix. -- Rerun the exact same command and capture the passing output. -- Compare baseline vs fixed vs reference behavior. -- Redact local paths, secrets, tokens, and internal hostnames before posting logs or screenshots. +2. **Apply fix** and test with patched version + ```bash + # Test with fixed version + npm install -g package@fixed-version + [same command] + # Capture: success output, normal exit codes + ``` -**Description**: Structured and thorough +3. **Document both** with timestamps, PIDs, exit codes, logs + +4. **Redact sensitive info**: + - Local absolute paths (`/Users/...`, `/home/...`) + - Secrets/tokens/API keys + - Internal URLs/hostnames + - Recheck every pasted block before submitting + +**Description**: Focused and reviewable (~50 lines) ````markdown ## Summary -[What this PR does in 1-2 sentences] +[1-2 sentences: what this fixes and why] -## Motivation -[Why this change is needed] +## Root Cause +[Technical explanation with code references] ## Changes -- [Change 1] -- [Change 2] +- [Actual code changes] +- [Tests added] +- [Docs updated] -## Evidence Loop -Command: -```bash -# Baseline (before fix) -[command] +## Why This Is Safe +[Explain why it won't break anything] -# Fixed (after fix, same command) -[command] -``` +## Testing -Raw output: +### Test 1: Reproduce Bug (Original Version) +Command: `[command]` +Result: ```text -[paste baseline output] +[failure output with timestamps, exit codes] ``` +### Test 2: Validate Fix (Patched Version) +Command: `[same command]` +Result: ```text -[paste fixed output] +[success output with timestamps, exit codes] ``` -## Comparison -| Case | Command / Scenario | Result | Evidence | -|------|--------------------|--------|----------| -| Baseline | `[same command]` | Fail | [raw output block] | -| Fixed | `[same command]` | Pass | [raw output block] | -| Reference | [spec, issue, or main branch behavior] | Expected | [link or note] | +## Related +- Fixes #[issue] +- Related: #[other issues/PRs] +```` -## Sources/Attribution -- [Issue, docs, or code references] +**What NOT to include in PR**: +- ❌ Detailed timeline analysis (put in issue) +- ❌ Historical context (put in issue) +- ❌ Internal tooling mentions +- ❌ Speculation or uncertainty +- ❌ Walls of text (>100 lines) -## Risks -- [Potential downside and impact] +### Code Changes Best Practices -## Rollback Plan -- Revert commit(s): [hash] -- Restore previous behavior with: [command] +**Minimal, surgical fixes**: +- ✅ Only change what's necessary to fix the bug +- ✅ Add regression test to prevent future breakage +- ✅ Update CHANGELOG if project uses it +- ❌ Don't refactor surrounding code +- ❌ Don't add "improvements" beyond the fix +- ❌ Don't change unrelated files -## Testing -[How you tested this] +**Example** (OpenClaw PR #39763): +``` +Files changed: 2 +- src/infra/process-respawn.ts (3 lines removed, 1 added) +- src/infra/process-respawn.test.ts (regression test added) -## Screenshots (if UI) -[Before/After images] -```` +Result: 278K star project, clean approval +``` + +### Separation of Concerns + +**Issue comments**: Detailed investigation +- Timeline analysis +- Historical context +- Related PRs/issues +- Root cause deep dive + +**PR description**: Focused on the fix +- Summary (1-2 sentences) +- Root cause (technical) +- Changes (bullet list) +- Testing validation +- ~50 lines total + +**Separate test comment**: End-to-end validation +- Test with original version (prove bug) +- Test with fixed version (prove fix) +- Full logs with timestamps ### After Submitting -- Respond to feedback promptly +- Monitor CI results +- Respond to feedback promptly (within 24 hours) - Make requested changes quickly - Be grateful for reviews -- Don't argue, discuss +- Don't argue, discuss professionally +- If you need to update PR: + - Add new commits (don't force push during review) + - Explain what changed in comment + - Re-request review when ready + +**Professional responses**: +``` +✅ "Good point! I've updated the implementation to..." +✅ "Thanks for catching that. Fixed in commit abc123." +✅ "I see what you mean. I chose this approach because... + Would you prefer if I changed it to...?" + +❌ "That's just your opinion." +❌ "It works on my machine." +❌ "This is how I always do it." +``` ## Building Reputation @@ -243,35 +367,70 @@ Level 4: Maintainer status ### Don't -- Submit drive-by PRs without context +- Submit drive-by PRs without investigation +- Include detailed timeline in PR (put in issue) +- Mention internal tooling or infrastructure - Argue with maintainers - Ignore code style guidelines - Make massive changes without discussion - Ghost after submitting +- Refactor code unrelated to the fix +- Add "improvements" beyond what was requested +- Force push during review (unless asked) ### Do +- Investigate thoroughly BEFORE coding +- Post detailed analysis to issue, not PR +- Keep PR focused and minimal (~50 lines) - Start with small, focused PRs - Follow project conventions exactly +- Add regression tests +- Update CHANGELOG if project uses it - Communicate proactively - Accept feedback gracefully - Build relationships over time +- Test with both original and fixed versions +- Redact sensitive info from logs ## Workflow Template ``` -Contribution Workflow: +High-Quality Contribution Workflow: + +Investigation Phase: - [ ] Find project with "good first issue" - [ ] Read contribution guidelines - [ ] Comment on issue to claim +- [ ] Reproduce bug with original version +- [ ] Trace git history for context +- [ ] Identify root cause with evidence +- [ ] Post detailed analysis to issue + +Implementation Phase: - [ ] Fork and set up locally -- [ ] Make focused changes -- [ ] Run evidence loop (reproduce fail -> apply fix -> rerun same command pass) -- [ ] Add baseline vs fixed vs reference comparison -- [ ] Redact paths/secrets/internal hosts in logs and screenshots -- [ ] Test thoroughly -- [ ] Write clear PR description -- [ ] Respond to review feedback +- [ ] Make minimal, focused changes +- [ ] Add regression test +- [ ] Update CHANGELOG (if applicable) +- [ ] Follow project conventions exactly + +Validation Phase: +- [ ] Test with original version (prove bug exists) +- [ ] Test with fixed version (prove fix works) +- [ ] Document both with timestamps/logs +- [ ] Redact paths/secrets/internal hosts + +Submission Phase: +- [ ] Write focused PR description (~50 lines) +- [ ] Link to detailed issue analysis +- [ ] Post end-to-end test results +- [ ] Ensure CI passes + +Review Phase: +- [ ] Respond to feedback within 24 hours +- [ ] Make requested changes quickly +- [ ] Don't force push during review +- [ ] Thank reviewers - [ ] Celebrate when merged! 🎉 ``` @@ -310,3 +469,28 @@ Types: `feat`, `fix`, `docs`, `style`, `refactor`, `test`, `chore` - `references/pr_checklist.md` - Complete PR quality checklist - `references/project_evaluation.md` - How to evaluate projects - `references/communication_templates.md` - Issue/PR templates +- `references/high_quality_pr_case_study.md` - Real-world successful PR walkthrough (OpenClaw #39763) + +## Success Indicators + +You know you have a high-quality PR when: + +- ✅ Maintainers understand the problem immediately +- ✅ Reviewers can verify the fix easily +- ✅ CI passes on first try +- ✅ No "can you explain..." questions +- ✅ Minimal back-and-forth +- ✅ Quick approval + +## Key Metrics for Quality PRs + +Based on successful contributions to major projects: + +- **Files changed**: 1-3 (focused scope) +- **Lines changed**: 10-50 (minimal fix) +- **PR description**: ~50 lines (concise) +- **Issue investigation**: 100-300 lines (thorough) +- **Time to first draft**: 2-3 days (proper investigation) +- **Time to ready**: 3-5 days (including validation) +- **Response time**: <24 hours (professional) + diff --git a/github-contributor/references/high_quality_pr_case_study.md b/github-contributor/references/high_quality_pr_case_study.md new file mode 100644 index 00000000..8942ab43 --- /dev/null +++ b/github-contributor/references/high_quality_pr_case_study.md @@ -0,0 +1,328 @@ +# High-Quality PR: Real-World Case Study + +Based on OpenClaw PR #39763 - A successful bug fix contribution to a 278K star TypeScript project. + +## What Made This PR High-Quality + +### 1. Complete Evidence Chain + +**Issue → Root Cause → Fix → Validation** + +``` +✅ Original bug report with symptoms +✅ Deep investigation with timeline analysis +✅ Root cause identified in source code +✅ Minimal, surgical fix +✅ End-to-end testing with before/after comparison +✅ Regression test added +``` + +### 2. Thorough Investigation Before Coding + +**Timeline Analysis** (Posted to issue, not PR): +- Traced bug through 3 years of related changes +- Identified when workaround was added (#29078) +- Explained why removing workaround is now safe +- Linked to all relevant historical PRs and issues + +**Key insight**: Detailed investigation goes in the issue, not the PR. Keep PR focused on the fix. + +### 3. Minimal, Focused Changes + +**Files changed**: 2 +- `src/infra/process-respawn.ts` - 3 lines removed, 1 line added +- `src/infra/process-respawn.test.ts` - Updated tests + regression test + +**What we didn't do**: +- ❌ Refactor surrounding code +- ❌ Add "improvements" beyond the fix +- ❌ Change unrelated files +- ❌ Add extensive comments + +### 4. Regression Test Added + +```typescript +test("launchd path never returns failed status", () => { + const result = detectSupervisor("launchd"); + expect(result.mode).not.toBe("failed"); + expect(result.mode).toBe("supervised"); +}); +``` + +**Why this matters**: Prevents the bug from being reintroduced. + +### 5. CHANGELOG Entry + +Following project conventions: +```markdown +## [Unreleased] + +### Fixed +- **darwin/launchd**: Remove `kickstart -k` self-restart to prevent race condition with launchd bootout (#39763) +``` + +**Key**: Check if project maintains CHANGELOG and follow their format exactly. + +### 6. Clear PR Structure + +**Title**: `fix(darwin): remove launchd kickstart race condition` +- Conventional commit format +- Scope indicates platform +- Clear what was fixed + +**Body** (~50 lines, trimmed from original 136): +```markdown +## Summary +[2 sentences: what + why] + +## Root Cause +[Technical explanation with code references] + +## Changes +- [Bullet list of actual changes] + +## Why This Is Safe +[Explain why the fix won't break anything] + +## Testing +[How it was validated] + +## Related +- Fixes #39760 +- Related: #27650, #29078 +``` + +### 7. Separation of Concerns + +**Issue comment**: Detailed timeline, investigation, evidence +**PR description**: Focused on the fix, testing, safety +**Separate test comment**: End-to-end validation results + +**Why**: Keeps PR reviewable. Detailed context available but not blocking review. + +### 8. End-to-End Testing + +**Test 1**: Reproduced bug with original version +``` +Result: Bootstrap failed: 5, SIGKILL, exit code -9 ✅ +``` + +**Test 2**: Validated fix with patched version +``` +Result: Clean restart, no errors, normal exit code ✅ +``` + +**Evidence**: Posted full logs with timestamps, PIDs, exit codes. + +### 9. What We Avoided + +**❌ Don't mention internal tooling**: +- We had a custom monitor script that auto-remediated the bug +- We initially mentioned it in PR comments +- **Removed it** because it's not part of OpenClaw - would confuse maintainers + +**❌ Don't over-explain in PR**: +- Moved detailed timeline analysis to issue +- Kept PR focused on fix validation + +**❌ Don't add noise**: +- No "I think this might work" comments +- No "please review" pings +- No unnecessary updates + +### 10. Professional Communication + +**In issue**: +```markdown +## Timeline Analysis + +I traced this through the codebase history: + +1. 2023-05: #27650 set ThrottleInterval to 60s +2. 2023-08: #29078 added kickstart workaround +3. Later: ThrottleInterval reduced to 1s +4. Now: Safe to remove kickstart + +[Detailed evidence with links] +``` + +**In PR**: +```markdown +## Testing Complete ✅ + +End-to-end testing confirms: +1. Bug reproduced with 2026.3.7 +2. Fix validated with PR branch +3. Ready for review + +Full logs: [link to issue comment] +``` + +## The High-Quality PR Formula + +``` +1. Deep investigation (post to issue) +2. Minimal, surgical fix +3. Regression test +4. CHANGELOG entry (if project uses it) +5. End-to-end validation +6. Clear PR structure +7. Professional communication +8. Separate concerns (issue vs PR) +9. No internal/irrelevant details +10. Responsive to feedback +``` + +## PR Lifecycle + +``` +Day 1: Investigation + ├─ Reproduce bug locally + ├─ Trace through codebase history + ├─ Identify root cause + └─ Post detailed analysis to issue + +Day 2: Implementation + ├─ Create minimal fix + ├─ Add regression test + ├─ Update CHANGELOG + └─ Test locally + +Day 3: Validation + ├─ Test with original version (reproduce bug) + ├─ Test with fixed version (validate fix) + ├─ Document test results + └─ Submit PR + +Day 4: Refinement + ├─ Trim PR description (move details to issue) + ├─ Add context about historical changes + ├─ Post end-to-end test results + └─ Mark ready for review + +Day 5+: Review cycle + ├─ Respond to feedback promptly + ├─ Make requested changes + └─ Wait for CI and approval +``` + +## Key Metrics + +**OpenClaw PR #39763**: +- Files changed: 2 +- Lines added: ~20 (including tests) +- Lines removed: 3 +- PR description: ~50 lines +- Issue investigation: ~200 lines +- Time to first draft: 3 days +- Time to ready: 4 days + +## What Maintainers Look For + +Based on this experience: + +1. **Does it fix the problem?** ✅ Bug reproduced and fixed +2. **Is it minimal?** ✅ Only changed what's necessary +3. **Will it break anything?** ✅ Explained why it's safe +4. **Can it regress?** ✅ Added regression test +5. **Is it documented?** ✅ CHANGELOG entry +6. **Is it tested?** ✅ End-to-end validation +7. **Is it reviewable?** ✅ Clear structure, focused scope + +## Anti-Patterns We Avoided + +1. ❌ **Drive-by PR**: "Here's a fix, hope it works" + - ✅ We did: Deep investigation, thorough testing + +2. ❌ **Kitchen sink PR**: "Fixed bug + refactored + added features" + - ✅ We did: Minimal, focused fix only + +3. ❌ **No evidence PR**: "Trust me, it works" + - ✅ We did: Reproduced bug, validated fix, posted logs + +4. ❌ **Wall of text PR**: 500-line description + - ✅ We did: Trimmed to ~50 lines, moved details to issue + +5. ❌ **Ghost PR**: Submit and disappear + - ✅ We did: Responsive, iterative refinement + +## Lessons Learned + +### Investigation Phase +- Trace through git history to understand context +- Link to all related issues and PRs +- Post detailed analysis to issue, not PR + +### Implementation Phase +- Make the smallest possible change +- Add regression test +- Follow project conventions exactly + +### Validation Phase +- Test with original version (prove bug exists) +- Test with fixed version (prove fix works) +- Document both with timestamps and logs + +### Communication Phase +- Keep PR focused and reviewable +- Move detailed context to issue +- Remove internal/irrelevant details +- Be professional and responsive + +## Template for Future PRs + +```markdown +## Summary +[1-2 sentences: what this fixes and why] + +## Root Cause +[Technical explanation with code references] + +## Changes +- [Actual code changes] +- [Tests added] +- [Docs updated] + +## Why This Is Safe +[Explain why it won't break anything] + +## Testing +[How you validated the fix] + +### Test 1: Reproduce Bug +Command: `[command]` +Result: [failure output] + +### Test 2: Validate Fix +Command: `[same command]` +Result: [success output] + +## Related +- Fixes #[issue] +- Related: #[other issues] +``` + +## Success Indicators + +You know you have a high-quality PR when: + +- ✅ Maintainers understand the problem immediately +- ✅ Reviewers can verify the fix easily +- ✅ CI passes on first try +- ✅ No "can you explain..." questions +- ✅ Minimal back-and-forth +- ✅ Quick approval + +## Final Checklist + +Before submitting: +- [ ] Bug reproduced with original version +- [ ] Fix validated with patched version +- [ ] Regression test added +- [ ] CHANGELOG updated (if applicable) +- [ ] PR description is focused (~50 lines) +- [ ] Detailed investigation in issue, not PR +- [ ] No internal tooling mentioned +- [ ] No local paths/secrets in logs +- [ ] All tests pass +- [ ] Follows project conventions diff --git a/github-contributor/references/pr_checklist.md b/github-contributor/references/pr_checklist.md index 319a1b5d..0e07aba7 100644 --- a/github-contributor/references/pr_checklist.md +++ b/github-contributor/references/pr_checklist.md @@ -2,13 +2,37 @@ Complete checklist for creating high-quality pull requests. -## Before Starting +# PR Quality Checklist + +Complete checklist for creating high-quality pull requests based on successful contributions to major open-source projects. + +## Investigation Phase (Before Coding) - [ ] Read CONTRIBUTING.md thoroughly - [ ] Check for existing PRs addressing same issue - [ ] Comment on issue to express interest -- [ ] Wait for maintainer acknowledgment (if required) -- [ ] Understand project's code style +- [ ] Reproduce bug with original version +- [ ] Trace git history for context (`git log --all --grep="keyword"`) +- [ ] Identify root cause with code references +- [ ] Post detailed investigation to issue (not PR) +- [ ] Link all related issues/PRs + +### Investigation Template (Post to Issue) + +```markdown +## Investigation + +I traced this through the codebase history: + +1. [Date]: #[PR] introduced [feature] +2. [Date]: #[PR] added [workaround] because [reason] +3. [Date]: #[PR] changed [parameter] +4. Now: Safe to [fix] because [explanation] + +[Detailed evidence with code references] +``` + +## Before Starting ## Environment Setup @@ -28,11 +52,27 @@ refactor/extract-validation-logic ## During Development -- [ ] Make small, focused commits +- [ ] Make minimal, focused changes (only what's necessary) +- [ ] Add regression test to prevent future breakage +- [ ] Update CHANGELOG if project uses it - [ ] Follow project's commit message format -- [ ] Add tests for new functionality -- [ ] Update documentation if needed - [ ] Run linter/formatter before committing +- [ ] Don't refactor unrelated code +- [ ] Don't add "improvements" beyond the fix + +### What to Change + +✅ **Do change**: +- Code directly related to the fix +- Tests for the fix +- CHANGELOG entry +- Relevant documentation + +❌ **Don't change**: +- Surrounding code (refactoring) +- Unrelated files +- Code style of existing code +- Add features beyond the fix ## Before Submitting @@ -46,10 +86,25 @@ refactor/extract-validation-logic ### Evidence Loop -- [ ] Reproduce the failure with one command and capture baseline output +- [ ] Test with original version and capture failure output - [ ] Apply the fix -- [ ] Rerun the exact same command and capture passing output -- [ ] Add baseline vs fixed vs reference comparison +- [ ] Test with fixed version and capture success output +- [ ] Document both tests with timestamps, exit codes, PIDs +- [ ] Compare baseline vs fixed behavior + +### Testing Commands + +```bash +# Test 1: Reproduce bug with original version +npm install -g package@original-version +[command that triggers bug] +# Capture: error messages, exit codes, timestamps + +# Test 2: Validate fix with patched version +npm install -g package@fixed-version +[same command] +# Capture: success output, normal exit codes +``` ### Redaction Gate @@ -67,15 +122,32 @@ refactor/extract-validation-logic ### PR Description -- [ ] Clear, descriptive title -- [ ] Summary of changes -- [ ] Motivation/context -- [ ] Testing approach -- [ ] Screenshots (for UI changes) +- [ ] Clear, descriptive title (conventional commit format) +- [ ] Focused description (~50 lines, not >100) +- [ ] Summary (1-2 sentences) +- [ ] Root cause (technical, with code refs) +- [ ] Changes (bullet list) +- [ ] Why it's safe +- [ ] Testing validation - [ ] Related issues linked -- [ ] Sources/Attribution section added -- [ ] Risks section added -- [ ] Rollback Plan section added +- [ ] No detailed timeline (move to issue) +- [ ] No internal tooling mentions +- [ ] No speculation or uncertainty + +### PR Description Length Guide + +✅ **Good**: ~50 lines +- Summary: 2 lines +- Root cause: 5 lines +- Changes: 5 lines +- Why safe: 5 lines +- Testing: 20 lines +- Related: 3 lines + +❌ **Too long**: >100 lines +- Move detailed investigation to issue +- Move timeline analysis to issue +- Keep PR focused on the fix ## PR Title Format @@ -96,82 +168,51 @@ chore(deps): update lodash to 4.17.21 ````markdown ## Summary -Brief description of what this PR does. - -## Motivation +[1-2 sentences: what this fixes and why] -Why is this change needed? Link to issue if applicable. +## Root Cause -Closes #123 +[Technical explanation with code references] ## Changes -- Change 1 -- Change 2 -- Change 3 +- [Actual code changes] +- [Tests added] +- [Docs updated] -## Evidence Loop +## Why This Is Safe -Command: -```bash -# Baseline (before fix) -[command] +[Explain why it won't break anything] -# Fixed (after fix, same command) -[command] -``` +## Testing -Raw output: +### Test 1: Reproduce Bug (Original Version) +Command: `[command]` +Result: ```text -[baseline output] +[failure output with timestamps, exit codes] ``` +### Test 2: Validate Fix (Patched Version) +Command: `[same command]` +Result: ```text -[fixed output] +[success output with timestamps, exit codes] ``` -## Comparison (Baseline vs Fixed vs Reference) - -| Case | Command / Scenario | Result | Evidence | -|------|--------------------|--------|----------| -| Baseline | `[same command]` | Fail | [raw output block] | -| Fixed | `[same command]` | Pass | [raw output block] | -| Reference | [spec, issue, or main behavior] | Expected | [link or note] | - -## Sources/Attribution - -- [Issue, docs, benchmark source, or code reference] - -## Risks +## Related -- [Risk and impact] - -## Rollback Plan - -- Revert commit(s): [hash] -- Restore previous behavior with: [command] - -## Testing - -How did you test this change? - -- [ ] Unit tests added/updated -- [ ] Manual testing performed -- [ ] Integration tests pass - -## Screenshots (if applicable) - -| Before | After | -|--------|-------| -| image | image | - -## Checklist - -- [ ] Tests pass -- [ ] Documentation updated -- [ ] Code follows project style +- Fixes #[issue] +- Related: #[other issues/PRs] ```` +**What NOT to include**: +- ❌ Detailed timeline analysis (put in issue) +- ❌ Historical context (put in issue) +- ❌ Internal tooling mentions +- ❌ Speculation or uncertainty +- ❌ Walls of text (>100 lines) + ## Comparison Table Template ```markdown @@ -185,10 +226,34 @@ How did you test this change? ## After Submitting - [ ] Monitor for CI results -- [ ] Respond to review comments promptly +- [ ] Respond to review comments within 24 hours - [ ] Make requested changes quickly - [ ] Thank reviewers for their time - [ ] Don't force push after review starts (unless asked) +- [ ] Add new commits during review (don't amend) +- [ ] Explain what changed in follow-up comments +- [ ] Re-request review when ready + +## Separation of Concerns + +### Issue Comments (Detailed Investigation) +- Timeline analysis +- Historical context +- Related PRs/issues +- Root cause deep dive +- 100-300 lines OK + +### PR Description (Focused on Fix) +- Summary (1-2 sentences) +- Root cause (technical) +- Changes (bullet list) +- Testing validation +- ~50 lines total + +### Separate Test Comment (End-to-End Validation) +- Test with original version +- Test with fixed version +- Full logs with timestamps ## Review Response Etiquette From 135a1873af9e176f0ca5dcdfb2bef7a5c528ee8b Mon Sep 17 00:00:00 2001 From: daymade Date: Wed, 11 Mar 2026 13:59:36 +0800 Subject: [PATCH 004/137] feat(transcript-fixer): add timestamp repair and section splitting scripts New scripts: - fix_transcript_timestamps.py: Repair malformed timestamps (HH:MM:SS format) - split_transcript_sections.py: Split transcript by keywords and rebase timestamps - Automated tests for both scripts Features: - Timestamp validation and repair (handle missing colons, invalid ranges) - Section splitting with custom names - Rebase timestamps to 00:00:00 for each section - Preserve speaker format and content integrity - In-place editing with backup Documentation updates: - Add usage examples to SKILL.md - Clarify dictionary iteration workflow (save stable patterns only) - Update workflow guides with new script references - Add script parameter documentation Use cases: - Fix ASR output with broken timestamps - Split long meetings into focused sections - Prepare sections for independent processing Co-Authored-By: Claude Opus 4.6 (1M context) --- transcript-fixer/SKILL.md | 22 +- .../references/iteration_workflow.md | 13 +- .../references/script_parameters.md | 55 ++++ transcript-fixer/references/workflow_guide.md | 25 ++ .../scripts/fix_transcript_timestamps.py | 282 ++++++++++++++++++ .../scripts/split_transcript_sections.py | 196 ++++++++++++ .../tests/test_fix_transcript_timestamps.py | 54 ++++ .../tests/test_split_transcript_sections.py | 45 +++ 8 files changed, 688 insertions(+), 4 deletions(-) create mode 100644 transcript-fixer/scripts/fix_transcript_timestamps.py create mode 100644 transcript-fixer/scripts/split_transcript_sections.py create mode 100644 transcript-fixer/scripts/tests/test_fix_transcript_timestamps.py create mode 100644 transcript-fixer/scripts/tests/test_split_transcript_sections.py diff --git a/transcript-fixer/SKILL.md b/transcript-fixer/SKILL.md index 4830456c..5785a47b 100644 --- a/transcript-fixer/SKILL.md +++ b/transcript-fixer/SKILL.md @@ -44,6 +44,20 @@ The enhanced wrapper automatically: - Moves output files to specified directory - Opens HTML visual diff in browser for immediate feedback +**Timestamp repair**: +```bash +uv run scripts/fix_transcript_timestamps.py meeting.txt --in-place +``` + +**Split transcript into sections and rebase each section to `00:00:00`**: +```bash +uv run scripts/split_transcript_sections.py meeting.txt \ + --first-section-name "课前聊天" \ + --section "正式上课::好，无缝切换嘛。对。那个曹总连上了吗？那个网页。" \ + --section "课后复盘::我们复盘一下。" \ + --rebase-to-zero +``` + **Alternative: Use Core Script Directly**: ```bash @@ -117,13 +131,15 @@ See `references/workflow_guide.md` for detailed workflows, `references/script_pa ## Critical Workflow: Dictionary Iteration -**MUST save corrections after each fix.** This is the skill's core value. +**Save stable, reusable ASR patterns after each fix.** This is the skill's core value. -After fixing errors manually, immediately save to dictionary: +After fixing errors manually, immediately save stable corrections to dictionary: ```bash uv run scripts/fix_transcription.py --add "错误词" "正确词" --domain general ``` +Do **not** save one-off deletions, ambiguous context-only rewrites, or section-specific cleanup to the dictionary. + See `references/iteration_workflow.md` for complete iteration guide with checklist. ## AI Fallback Strategy @@ -162,7 +178,9 @@ sqlite3 ~/.transcript-fixer/corrections.db "SELECT value FROM system_config WHER - `ensure_deps.py` - Initialize shared virtual environment (run once, optional) - `fix_transcript_enhanced.py` - Enhanced wrapper (recommended for interactive use) - `fix_transcription.py` - Core CLI (for automation) +- `fix_transcript_timestamps.py` - Normalize/repair speaker timestamps and optionally rebase to zero - `generate_word_diff.py` - Generate word-level diff HTML for reviewing corrections +- `split_transcript_sections.py` - Split a transcript by marker phrases and optionally rebase each section - `examples/bulk_import.py` - Bulk import example **References** (load as needed): diff --git a/transcript-fixer/references/iteration_workflow.md b/transcript-fixer/references/iteration_workflow.md index ba94a462..17383118 100644 --- a/transcript-fixer/references/iteration_workflow.md +++ b/transcript-fixer/references/iteration_workflow.md @@ -16,7 +16,7 @@ The core value of transcript-fixer is building a personalized correction diction └─────────────────────────────────────────────────┘ ``` -**Key principle**: Every correction you make should be saved to the dictionary. This transforms one-time work into permanent value. +**Key principle**: Every stable, reusable ASR correction you make should be saved to the dictionary. This transforms one-time work into permanent value without polluting the database. ## Workflow Checklist @@ -34,7 +34,7 @@ Correction Progress: ## Save Corrections Immediately -After fixing any transcript, save corrections: +After fixing any transcript, save stable corrections: ```bash # Single correction @@ -122,3 +122,12 @@ Patterns appearing ≥3 times at ≥80% confidence are suggested for review. 3. **Use domains**: Organize corrections by topic for better precision 4. **Verify**: Always run --list to confirm saves 5. **Review suggestions**: Periodically check --review-learned for auto-detected patterns + +## What NOT to Save to Dictionary + +Do **not** save these as reusable dictionary entries: + +- Full-sentence deletions +- One-off section headers or meeting-specific boilerplate +- Context-only disambiguations such as `cloud -> Claude` when `cloud` can also be legitimate +- File-local cleanup after section splitting or timestamp rebasing diff --git a/transcript-fixer/references/script_parameters.md b/transcript-fixer/references/script_parameters.md index a5537abc..6cbeab3d 100644 --- a/transcript-fixer/references/script_parameters.md +++ b/transcript-fixer/references/script_parameters.md @@ -9,6 +9,8 @@ Detailed command-line parameters and usage examples for transcript-fixer Python - [Correction Management](#correction-management) - [Correction Workflow](#correction-workflow) - [Learning Commands](#learning-commands) +- [fix_transcript_timestamps.py](#fix_transcript_timestampspy) - Normalize/repair speaker timestamps +- [split_transcript_sections.py](#split_transcript_sectionspy) - Split transcript into named sections - [diff_generator.py](#diffgeneratorpy) - Generate comparison reports - [Common Workflows](#common-workflows) - [Exit Codes](#exit-codes) @@ -74,6 +76,59 @@ python scripts/fix_transcription.py --input meeting.md --stage 3 --output ./corr - `2` - GLM_API_KEY environment variable not set (Stage 2 or 3 only) - `3` - API request failed +## fix_transcript_timestamps.py + +Normalize speaker timestamp lines such as `天生 00:21` or `Speaker 7 01:31:10`. + +### Syntax + +```bash +python scripts/fix_transcript_timestamps.py [--output FILE | --in-place | --check] +``` + +### Key Parameters + +- `--format {hhmmss,preserve}`: output timestamp style +- `--rebase-to-zero`: reset the first detected speaker timestamp to `00:00:00` +- `--rollover-backjump-seconds`: threshold for treating `59:58 -> 00:05` as a new hour +- `--jitter-seconds`: tolerated small backward jitter before flagging anomaly + +### Usage Examples + +```bash +# Normalize mixed MM:SS / HH:MM:SS +python scripts/fix_transcript_timestamps.py meeting.txt --in-place + +# Rebase a split transcript so it starts at 00:00:00 +python scripts/fix_transcript_timestamps.py workshop-class.txt --in-place --rebase-to-zero + +# Only inspect anomalies, do not write +python scripts/fix_transcript_timestamps.py meeting.txt --check +``` + +## split_transcript_sections.py + +Split a transcript into named sections using marker phrases. Useful for workshop transcripts that include setup chat, class, and debrief in one file. + +### Syntax + +```bash +python scripts/split_transcript_sections.py \ + --first-section-name \ + --section "Name::Marker" \ + --section "Name::Marker" +``` + +### Usage Example + +```bash +python scripts/split_transcript_sections.py workshop.txt \ + --first-section-name "课前聊天" \ + --section "正式上课::好，无缝切换嘛。对。那个曹总连上了吗？那个网页。" \ + --section "课后复盘::我们复盘一下。" \ + --rebase-to-zero +``` + ## generate_diff_report.py Multi-format diff report generator for comparing correction stages. diff --git a/transcript-fixer/references/workflow_guide.md b/transcript-fixer/references/workflow_guide.md index b6132b6d..5dbc7dad 100644 --- a/transcript-fixer/references/workflow_guide.md +++ b/transcript-fixer/references/workflow_guide.md @@ -17,6 +17,7 @@ Detailed step-by-step workflows for transcript correction and management. - [5. Stage-by-Stage Execution](#5-stage-by-stage-execution) - [6. Context-Aware Rules](#6-context-aware-rules) - [7. Diff Report Generation](#7-diff-report-generation) + - [8. Workshop Transcript Split + Timestamp Rebase](#8-workshop-transcript-split--timestamp-rebase) - [Batch Processing](#batch-processing) - [Process Multiple Files](#process-multiple-files) - [Parallel Processing](#parallel-processing) @@ -400,6 +401,30 @@ See `file_formats.md` for context_rules schema. See `script_parameters.md` for advanced diff options. +### 8. Workshop Transcript Split + Timestamp Rebase + +**Goal**: Split a long workshop transcript into sections such as setup chat, class, and debrief, then make each section start from `00:00:00`. + +**Steps**: + +1. **Correct transcript text first** (dictionary + AI/manual review) +2. **Pick marker phrases** for each section boundary +3. **Split and rebase**: + +```bash +uv run scripts/split_transcript_sections.py workshop.txt \ + --first-section-name "课前聊天" \ + --section "正式上课::好，无缝切换嘛。对。那个曹总连上了吗？那个网页。" \ + --section "课后复盘::我们复盘一下。" \ + --rebase-to-zero +``` + +4. **If you already split the files**, rebase a single file directly: + +```bash +uv run scripts/fix_transcript_timestamps.py class.txt --in-place --rebase-to-zero +``` + ## Batch Processing ### Process Multiple Files diff --git a/transcript-fixer/scripts/fix_transcript_timestamps.py b/transcript-fixer/scripts/fix_transcript_timestamps.py new file mode 100644 index 00000000..17754d29 --- /dev/null +++ b/transcript-fixer/scripts/fix_transcript_timestamps.py @@ -0,0 +1,282 @@ +#!/usr/bin/env python3 +"""Normalize and repair speaker timestamp lines in ASR transcripts. + +This script targets transcript lines shaped like: + + 天生 00:21 + Speaker 11 01:31:10 + +Common fixes: +- Normalize mixed `MM:SS` / `HH:MM:SS` into one format +- Repair hour rollovers for `MM:SS` timestamps after long recordings +- Optionally rebase a split transcript so it starts at 00:00:00 +- Report suspicious backward jumps instead of silently rewriting them +""" + +from __future__ import annotations + +import argparse +import re +import sys +from dataclasses import dataclass +from pathlib import Path + + +SPEAKER_TS_RE = re.compile(r"^(?P.+?)\s+(?P\d{1,2}:\d{2}(?::\d{2})?)\s*$") + + +@dataclass +class TimestampEvent: + line_no: int + original: str + fixed: str + reason: str + + +@dataclass +class RepairResult: + changed_lines: list[TimestampEvent] + anomalies: list[TimestampEvent] + repaired_text: str + matched_count: int + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Repair speaker timestamp lines in transcript files." + ) + parser.add_argument("input", help="Input transcript file") + parser.add_argument( + "--output", + help="Output path. Defaults to _timestamps_fixed when not using --check.", + ) + parser.add_argument( + "--in-place", + action="store_true", + help="Overwrite the input file", + ) + parser.add_argument( + "--check", + action="store_true", + help="Only analyze and report issues without writing a file", + ) + parser.add_argument( + "--format", + choices=("hhmmss", "preserve"), + default="hhmmss", + help="Timestamp output format. Default: hhmmss", + ) + parser.add_argument( + "--rollover-backjump-seconds", + type=int, + default=15 * 60, + help=( + "Backward jump threshold for treating MM:SS as a new hour rollover. " + "Default: 900" + ), + ) + parser.add_argument( + "--jitter-seconds", + type=int, + default=5, + help="Allowed small backward jitter before flagging an anomaly. Default: 5", + ) + parser.add_argument( + "--rebase-to-zero", + action="store_true", + help="Rebase the first detected speaker timestamp in the file to 00:00:00.", + ) + return parser.parse_args() + + +def to_seconds(parts: list[int]) -> int: + if len(parts) == 2: + minutes, seconds = parts + return minutes * 60 + seconds + hours, minutes, seconds = parts + return hours * 3600 + minutes * 60 + seconds + + +def format_timestamp(total_seconds: int, output_format: str) -> str: + hours = total_seconds // 3600 + minutes = (total_seconds % 3600) // 60 + seconds = total_seconds % 60 + + if output_format == "preserve" and hours == 0: + return f"{minutes:02d}:{seconds:02d}" + return f"{hours:02d}:{minutes:02d}:{seconds:02d}" + + +def repair_timestamps( + text: str, + *, + output_format: str, + rollover_backjump_seconds: int, + jitter_seconds: int, + rebase_to_zero: bool, +) -> RepairResult: + lines = text.splitlines(keepends=True) + repaired_lines: list[str] = [] + changed_lines: list[TimestampEvent] = [] + anomalies: list[TimestampEvent] = [] + + current_hour = 0 + last_abs_seconds: int | None = None + first_abs_seconds: int | None = None + matched_count = 0 + + for line_no, line in enumerate(lines, start=1): + stripped = line.rstrip("\r\n") + newline = line[len(stripped) :] + match = SPEAKER_TS_RE.match(stripped) + + if not match: + repaired_lines.append(line) + continue + + matched_count += 1 + prefix = match.group("prefix") + raw_ts = match.group("ts") + parts = [int(part) for part in raw_ts.split(":")] + + if len(parts) == 3: + abs_seconds = to_seconds(parts) + current_hour = parts[0] + + if ( + last_abs_seconds is not None + and abs_seconds + jitter_seconds < last_abs_seconds + ): + anomalies.append( + TimestampEvent( + line_no=line_no, + original=raw_ts, + fixed=format_timestamp(abs_seconds, output_format), + reason="explicit_hhmmss_backwards", + ) + ) + else: + minutes, seconds = parts + candidate = current_hour * 3600 + minutes * 60 + seconds + + if last_abs_seconds is not None: + while candidate + rollover_backjump_seconds < last_abs_seconds: + current_hour += 1 + candidate = current_hour * 3600 + minutes * 60 + seconds + + if candidate + jitter_seconds < last_abs_seconds: + anomalies.append( + TimestampEvent( + line_no=line_no, + original=raw_ts, + fixed=format_timestamp(candidate, output_format), + reason="small_backjump_after_rollover_check", + ) + ) + + abs_seconds = candidate + + if first_abs_seconds is None: + first_abs_seconds = abs_seconds + + display_seconds = ( + abs_seconds - first_abs_seconds + if rebase_to_zero and first_abs_seconds is not None + else abs_seconds + ) + + fixed_ts = format_timestamp(display_seconds, output_format) + repaired_lines.append(f"{prefix} {fixed_ts}{newline}") + + if fixed_ts != raw_ts: + changed_lines.append( + TimestampEvent( + line_no=line_no, + original=raw_ts, + fixed=fixed_ts, + reason="normalized_or_rollover_fixed", + ) + ) + + last_abs_seconds = abs_seconds + + return RepairResult( + changed_lines=changed_lines, + anomalies=anomalies, + repaired_text="".join(repaired_lines), + matched_count=matched_count, + ) + + +def default_output_path(input_path: Path) -> Path: + return input_path.with_name( + f"{input_path.stem}_timestamps_fixed{input_path.suffix}" + ) + + +def print_summary(result: RepairResult) -> None: + print(f"Matched speaker timestamp lines: {result.matched_count}") + print(f"Rewritten timestamp lines: {len(result.changed_lines)}") + print(f"Anomalies flagged: {len(result.anomalies)}") + + if result.changed_lines: + print("\nChanged lines:") + for event in result.changed_lines[:20]: + print( + f" L{event.line_no}: {event.original} -> {event.fixed} " + f"({event.reason})" + ) + if len(result.changed_lines) > 20: + print(f" ... {len(result.changed_lines) - 20} more") + + if result.anomalies: + print("\nAnomalies:") + for event in result.anomalies[:20]: + print( + f" L{event.line_no}: {event.original} -> {event.fixed} " + f"({event.reason})" + ) + if len(result.anomalies) > 20: + print(f" ... {len(result.anomalies) - 20} more") + + +def main() -> int: + args = parse_args() + input_path = Path(args.input).expanduser().resolve() + + if not input_path.exists(): + print(f"Input file not found: {input_path}", file=sys.stderr) + return 1 + + if args.check and args.in_place: + print("--check and --in-place cannot be used together", file=sys.stderr) + return 1 + + text = input_path.read_text(encoding="utf-8") + result = repair_timestamps( + text, + output_format=args.format, + rollover_backjump_seconds=args.rollover_backjump_seconds, + jitter_seconds=args.jitter_seconds, + rebase_to_zero=args.rebase_to_zero, + ) + + print_summary(result) + + if args.check: + return 0 if not result.anomalies else 2 + + if args.in_place: + output_path = input_path + elif args.output: + output_path = Path(args.output).expanduser().resolve() + else: + output_path = default_output_path(input_path) + + output_path.write_text(result.repaired_text, encoding="utf-8") + print(f"\nWrote repaired transcript: {output_path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/transcript-fixer/scripts/split_transcript_sections.py b/transcript-fixer/scripts/split_transcript_sections.py new file mode 100644 index 00000000..89733ba7 --- /dev/null +++ b/transcript-fixer/scripts/split_transcript_sections.py @@ -0,0 +1,196 @@ +#!/usr/bin/env python3 +"""Split a transcript into named sections and optionally rebase timestamps. + +Example: + uv run scripts/split_transcript_sections.py meeting.txt \ + --first-section-name "课前聊天" \ + --section "正式上课::好，无缝切换嘛。对。那个曹总连上了吗？那个网页。" \ + --section "课后复盘::我们复盘一下。" \ + --rebase-to-zero +""" + +from __future__ import annotations + +import argparse +import re +import sys +from dataclasses import dataclass +from pathlib import Path + +from fix_transcript_timestamps import repair_timestamps + + +INVALID_FILENAME_CHARS = re.compile(r'[\\/:*?"<>|]+') + + +@dataclass +class SectionSpec: + name: str + marker: str + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Split a transcript into named sections using marker phrases." + ) + parser.add_argument("input", help="Input transcript file") + parser.add_argument( + "--first-section-name", + default="part-1", + help="Name for the section before the first marker. Default: part-1", + ) + parser.add_argument( + "--section", + action="append", + required=True, + metavar="NAME::MARKER", + help=( + "Section boundary definition. Split starts at the first occurrence of MARKER " + "and names that segment NAME." + ), + ) + parser.add_argument( + "--output-dir", + help="Directory for output files. Defaults to the input file directory.", + ) + parser.add_argument( + "--rebase-to-zero", + action="store_true", + help="Rebase each output section so its first speaker timestamp becomes 00:00:00.", + ) + parser.add_argument( + "--format", + choices=("hhmmss", "preserve"), + default="hhmmss", + help="Timestamp output format when rebasing. Default: hhmmss", + ) + parser.add_argument( + "--rollover-backjump-seconds", + type=int, + default=15 * 60, + help="Backward jump threshold for MM:SS hour rollover repair. Default: 900", + ) + parser.add_argument( + "--jitter-seconds", + type=int, + default=5, + help="Allowed small backward jitter before flagging an anomaly. Default: 5", + ) + return parser.parse_args() + + +def parse_section_arg(value: str) -> SectionSpec: + if "::" not in value: + raise ValueError(f"Invalid --section value: {value!r}") + name, marker = value.split("::", 1) + name = name.strip() + marker = marker.strip() + if not name or not marker: + raise ValueError(f"Invalid --section value: {value!r}") + return SectionSpec(name=name, marker=marker) + + +def sanitize_filename_component(name: str) -> str: + return INVALID_FILENAME_CHARS.sub("-", name).strip().replace(" ", "-") + + +def split_text_by_markers( + text: str, + *, + first_section_name: str, + sections: list[SectionSpec], +) -> list[tuple[str, str]]: + boundaries: list[tuple[int, SectionSpec]] = [] + search_from = 0 + + for spec in sections: + idx = text.find(spec.marker, search_from) + if idx == -1: + raise ValueError(f"Marker not found for section {spec.name!r}: {spec.marker!r}") + boundaries.append((idx, spec)) + search_from = idx + 1 + + result: list[tuple[str, str]] = [] + prev_idx = 0 + prev_name = first_section_name + + for idx, spec in boundaries: + result.append((prev_name, text[prev_idx:idx].rstrip() + "\n")) + prev_idx = idx + prev_name = spec.name + + result.append((prev_name, text[prev_idx:].rstrip() + "\n")) + return result + + +def maybe_rebase( + text: str, + *, + rebase_to_zero: bool, + output_format: str, + rollover_backjump_seconds: int, + jitter_seconds: int, +) -> str: + if not rebase_to_zero: + return text + result = repair_timestamps( + text, + output_format=output_format, + rollover_backjump_seconds=rollover_backjump_seconds, + jitter_seconds=jitter_seconds, + rebase_to_zero=True, + ) + return result.repaired_text + + +def main() -> int: + args = parse_args() + input_path = Path(args.input).expanduser().resolve() + + if not input_path.exists(): + print(f"Input file not found: {input_path}", file=sys.stderr) + return 1 + + try: + sections = [parse_section_arg(value) for value in args.section] + except ValueError as exc: + print(str(exc), file=sys.stderr) + return 1 + + text = input_path.read_text(encoding="utf-8") + + try: + parts = split_text_by_markers( + text, + first_section_name=args.first_section_name, + sections=sections, + ) + except ValueError as exc: + print(str(exc), file=sys.stderr) + return 1 + + output_dir = ( + Path(args.output_dir).expanduser().resolve() + if args.output_dir + else input_path.parent + ) + output_dir.mkdir(parents=True, exist_ok=True) + + for idx, (name, content) in enumerate(parts, start=1): + content = maybe_rebase( + content, + rebase_to_zero=args.rebase_to_zero, + output_format=args.format, + rollover_backjump_seconds=args.rollover_backjump_seconds, + jitter_seconds=args.jitter_seconds, + ) + safe_name = sanitize_filename_component(name) + output_path = output_dir / f"{input_path.stem}-{idx:02d}-{safe_name}{input_path.suffix}" + output_path.write_text(content, encoding="utf-8") + print(f"{output_path}\t{len(content.splitlines())} lines") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/transcript-fixer/scripts/tests/test_fix_transcript_timestamps.py b/transcript-fixer/scripts/tests/test_fix_transcript_timestamps.py new file mode 100644 index 00000000..b7fd4579 --- /dev/null +++ b/transcript-fixer/scripts/tests/test_fix_transcript_timestamps.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""Tests for transcript timestamp normalization and rebasing.""" + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from fix_transcript_timestamps import repair_timestamps + + +class TestFixTranscriptTimestamps(unittest.TestCase): + def test_rollover_fix(self): + text = ( + "甲 58:50\n" + "内容 A\n" + "乙 59:58\n" + "内容 B\n" + "丙 00:05\n" + "内容 C\n" + ) + result = repair_timestamps( + text, + output_format="hhmmss", + rollover_backjump_seconds=15 * 60, + jitter_seconds=5, + rebase_to_zero=False, + ) + self.assertIn("甲 00:58:50", result.repaired_text) + self.assertIn("乙 00:59:58", result.repaired_text) + self.assertIn("丙 01:00:05", result.repaired_text) + self.assertEqual(len(result.anomalies), 0) + + def test_rebase_to_zero(self): + text = ( + "甲 01:31:10\n" + "内容 A\n" + "乙 01:31:12\n" + "内容 B\n" + ) + result = repair_timestamps( + text, + output_format="hhmmss", + rollover_backjump_seconds=15 * 60, + jitter_seconds=5, + rebase_to_zero=True, + ) + self.assertIn("甲 00:00:00", result.repaired_text) + self.assertIn("乙 00:00:02", result.repaired_text) + + +if __name__ == "__main__": + unittest.main() diff --git a/transcript-fixer/scripts/tests/test_split_transcript_sections.py b/transcript-fixer/scripts/tests/test_split_transcript_sections.py new file mode 100644 index 00000000..d896e2b6 --- /dev/null +++ b/transcript-fixer/scripts/tests/test_split_transcript_sections.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""Tests for transcript section splitting.""" + +import sys +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from split_transcript_sections import ( + SectionSpec, + sanitize_filename_component, + split_text_by_markers, +) + + +class TestSplitTranscriptSections(unittest.TestCase): + def test_split_text_by_markers(self): + text = ( + "预热内容\n" + "开始安装\n" + "装环境内容\n" + "我们复盘一下。\n" + "复盘内容\n" + ) + parts = split_text_by_markers( + text, + first_section_name="课前聊天", + sections=[ + SectionSpec(name="正式上课", marker="开始安装"), + SectionSpec(name="课后复盘", marker="我们复盘一下。"), + ], + ) + self.assertEqual(parts[0][0], "课前聊天") + self.assertEqual(parts[1][0], "正式上课") + self.assertEqual(parts[2][0], "课后复盘") + self.assertTrue(parts[1][1].startswith("开始安装")) + self.assertTrue(parts[2][1].startswith("我们复盘一下。")) + + def test_sanitize_filename_component(self): + self.assertEqual(sanitize_filename_component("课后/复盘"), "课后-复盘") + + +if __name__ == "__main__": + unittest.main() From 042c837db6b459053452a7e4a8c1f84d14604604 Mon Sep 17 00:00:00 2001 From: daymade Date: Wed, 11 Mar 2026 14:02:26 +0800 Subject: [PATCH 005/137] feat(claude-export-txt-better): add Claude Code export file fixer Add skill to fix broken line wrapping in Claude Code exported .txt files. Reconstructs tables, paragraphs, paths, and tool calls that were hard-wrapped at fixed column widths. Features: - State-machine parser with next-line look-ahead - Handles 10 content types (user prompts, Claude responses, tables, tool calls, etc.) - Pangu spacing for CJK/ASCII mixed text - 53 automated validation checks - Safety: never modifies original files, verifies marker counts Co-Authored-By: Claude Opus 4.6 (1M context) --- .gitignore | 1 + claude-export-txt-better/SKILL.md | 101 + claude-export-txt-better/evals/evals.json | 32 + .../scripts/fix-claude-export.py | 2065 +++++++++++++++++ .../scripts/validate-claude-export-fix.py | 313 +++ 5 files changed, 2512 insertions(+) create mode 100644 claude-export-txt-better/SKILL.md create mode 100644 claude-export-txt-better/evals/evals.json create mode 100644 claude-export-txt-better/scripts/fix-claude-export.py create mode 100644 claude-export-txt-better/scripts/validate-claude-export-fix.py diff --git a/.gitignore b/.gitignore index 27c1cf97..35689889 100644 --- a/.gitignore +++ b/.gitignore @@ -81,3 +81,4 @@ INSTALLATION.md # Private/commercial skills (moved to claude-code-skills-pro) seo-expert/ video-creator/ +/jsonl-viewer/ diff --git a/claude-export-txt-better/SKILL.md b/claude-export-txt-better/SKILL.md new file mode 100644 index 00000000..3caa42d2 --- /dev/null +++ b/claude-export-txt-better/SKILL.md @@ -0,0 +1,101 @@ +--- +name: fixing-claude-export-conversations +description: > + Fixes broken line wrapping in Claude Code exported conversation files (.txt), + reconstructing tables, paragraphs, paths, and tool calls that were hard-wrapped + at fixed column widths. Includes an automated validation suite (generic, file-agnostic checks). + Triggers when the user has a Claude Code export file with broken formatting, + mentions "fix export", "fix conversation", "exported conversation", "make export + readable", references a file matching YYYY-MM-DD-HHMMSS-*.txt, or has a .txt + file with broken tables, split paths, or mangled tool output from Claude Code. +--- + +# Fixing Claude Code Export Conversations + +Reconstruct broken line wrapping in Claude Code exported `.txt` files. + +## Quick Start + +```bash +# Fix and show stats +uv run /scripts/fix-claude-export.py --stats + +# Custom output +uv run /scripts/fix-claude-export.py -o fixed.txt + +# Validate the result (53 automated checks) +uv run /scripts/validate-claude-export-fix.py fixed.txt +``` + +Replace `` with the resolved path to this skill's directory. Find it with: +```bash +find ~/.claude -path "*/fixing-claude-export-conversations/scripts" -type d 2>/dev/null +``` + +## Workflow + +Copy this checklist and track progress: + +``` +- [ ] Step 1: Locate the exported .txt file +- [ ] Step 2: Run fix script with --stats +- [ ] Step 3: Run validation suite +- [ ] Step 4: Spot-check output (tables, CJK paragraphs, tool results) +- [ ] Step 5: Deliver fixed file to user +``` + +**Step 1: Locate the file.** Claude Code exports use the naming pattern `YYYY-MM-DD-HHMMSS-.txt`. + +**Step 2: Run the fix script.** +```bash +uv run /scripts/fix-claude-export.py -o --stats +``` +Review the stats output — typical results: 20-25% line reduction, 80+ table borders fixed, 160+ table cells fixed. + +**Step 3: Run the validation suite.** +```bash +uv run /scripts/validate-claude-export-fix.py +``` +All checks must pass. If any fail, investigate before delivering. Use `--verbose` for full details on passing checks too. + +**Step 4: Spot-check.** Open the output and verify: +- Tables have intact borders (box-drawing characters on single lines) +- CJK/English mixed text has pangu spacing (`Portal 都需要`, not `Portal都需要`) +- Tool result blocks (`⎿`) have complete content on joined lines +- Diff output within tool results has each line number on its own line + +**Step 5: Deliver** the fixed file to the user. + +## What Gets Fixed + +The script handles 10 content types using a state-machine with next-line look-ahead: + +- **User prompts** (`❯` prefix, dw=76 padding) — paragraph joins with pangu spacing +- **Claude responses** (`●` prefix) — narrative, bullet, and numbered list joins +- **Claude paragraphs** (2-space indent) — next-line look-ahead via `_is_continuation_fragment` +- **Tables** — border reconstruction, cell re-padding with pipe-count tracking +- **Tool calls** (`● Bash(` etc.) — path and argument reconstruction +- **Tool results** (`⎿` prefix) — continuation joins including deeper-indented fragments +- **Plan text** (5-space indent) — next-line look-ahead via `_is_plan_continuation_fragment` +- **Agent tree** (`├─`/`└─`) — preserved structure +- **Separators** (`────`, `---`) — never joined +- **Tree connectors** (standalone `│`) — preserved + +## Key Design Decisions + +**Next-line look-ahead** (not dw thresholds): Instead of asking "was this line wrapped?" (fragile threshold), the script asks "does the next line look like a continuation?" by examining its content patterns — lowercase start, CJK ideograph start, opening bracket, hyphen/slash/underscore continuation. + +**Pangu spacing**: Inserts spaces between ASCII alphanumeric characters and CJK ideographs at join boundaries. Also triggers for `%`, `#`, `+`, `:` adjacent to CJK. + +**Mid-token detection**: Joins without space when boundaries indicate identifiers (`BASE_` + `URL`), paths (`documents` + `/05-team`), or hyphenated names (`ready` + `-together`). Exception: `--` prefix gets a space (`run` + `--headed`). + +## Safety + +- Never modifies the original file +- Marker counts verified: `❯`, `●`, `✻`, `⎿`, `…` must match input/output +- Runaway join detection: warns if any line exceeds 500 display-width +- Strict UTF-8 encoding — no silent fallbacks + +## Dependencies + +Python 3.10+ via `uv run` — zero external packages (stdlib only: `unicodedata`, `argparse`, `re`, `pathlib`, `dataclasses`). \ No newline at end of file diff --git a/claude-export-txt-better/evals/evals.json b/claude-export-txt-better/evals/evals.json new file mode 100644 index 00000000..b0d7dc4e --- /dev/null +++ b/claude-export-txt-better/evals/evals.json @@ -0,0 +1,32 @@ +{ + "skill_name": "fixing-claude-export-conversations", + "evals": [ + { + "id": 1, + "prompt": "Fix this exported conversation file: — it was exported from Claude Code and has broken line wrapping everywhere. Output the fixed version next to the original.", + "expected_output": "A *-fixed.txt file with correct line wrapping, intact table borders, matching marker counts, and fewer total lines than the input.", + "files": [], + "assertions": [ + {"name": "output_file_exists", "description": "The fixed output file was created successfully"}, + {"name": "line_count_reduced", "description": "Output has fewer lines than input. Reasonable reduction is 10-25%."}, + {"name": "marker_counts_match", "description": "The count of ❯, ●, ✻, ⎿, and … characters in output matches input exactly"}, + {"name": "no_broken_table_borders", "description": "Every line starting with ┌/├/└ also contains the matching right border on the same line"}, + {"name": "no_runaway_joins", "description": "No output line exceeds 500 display-width characters"} + ] + }, + { + "id": 2, + "prompt": "我从 Claude Code 导出了一个对话记录 ，表格和段落的换行全部乱了。帮我修复一下，顺便看看统计数据——修了多少行？", + "expected_output": "Fixed file created with --stats output showing join counts per content type. Tables should be properly reconstructed with correct column padding.", + "files": [], + "assertions": [ + {"name": "output_file_exists", "description": "The fixed output file was created successfully"}, + {"name": "line_count_reduced", "description": "Output has fewer lines than input. Reasonable reduction is 10-25%."}, + {"name": "marker_counts_match", "description": "The count of ❯, ●, ✻, ⎿, and … characters in output matches input exactly"}, + {"name": "no_broken_table_borders", "description": "Every line starting with ┌/├/└ also contains the matching right border on the same line"}, + {"name": "no_runaway_joins", "description": "No output line exceeds 500 display-width characters"}, + {"name": "stats_output_present", "description": "The run log contains statistics output with join counts (e.g., 'Table rows merged', 'Tool results fixed')"} + ] + } + ] +} diff --git a/claude-export-txt-better/scripts/fix-claude-export.py b/claude-export-txt-better/scripts/fix-claude-export.py new file mode 100644 index 00000000..27c50d6e --- /dev/null +++ b/claude-export-txt-better/scripts/fix-claude-export.py @@ -0,0 +1,2065 @@ +#!/usr/bin/env python3 +"""Fix broken line wrapping in Claude Code exported conversation files. + +Claude Code exports hard-wrap lines at fixed column widths, breaking tables, +paragraphs, and paths. This script reconstructs the original logical lines +using a state-machine + lookahead merge approach. + +Usage: + uv run scripts/fix-claude-export.py + uv run scripts/fix-claude-export.py -o + uv run scripts/fix-claude-export.py --stats --dry-run +""" + +from __future__ import annotations + +import argparse +import re +import sys +import unicodedata +from dataclasses import dataclass +from pathlib import Path + +# --------------------------------------------------------------------------- +# Display-width helpers +# --------------------------------------------------------------------------- + +def display_width(s: str) -> int: + """Calculate display width accounting for CJK double-width characters.""" + w = 0 + for ch in s: + eaw = unicodedata.east_asian_width(ch) + w += 2 if eaw in ("W", "F") else 1 + return w + + +def is_wide_char(ch: str) -> bool: + """Return True if *ch* occupies two display columns (CJK full-width).""" + return unicodedata.east_asian_width(ch) in ("W", "F") + + +def _is_cjk_ideograph(ch: str) -> bool: + """Return True if *ch* is a CJK ideograph (not punctuation/symbol). + + CJK ideographs have Unicode category ``Lo`` (Letter, other) — e.g. + ``你``, ``好``, ``接``. CJK punctuation (``。，！？；：「」（）``) has + categories ``Ps``, ``Pe``, ``Po``, etc. and should NOT match. + + This distinction matters for pangu spacing: a space is inserted between + ASCII alphanumeric characters and CJK ideographs, but NOT between + CJK punctuation and anything. + """ + return is_wide_char(ch) and unicodedata.category(ch) == "Lo" + + +# --------------------------------------------------------------------------- +# Join helpers +# --------------------------------------------------------------------------- + +def smart_join(left: str, right_content: str) -> str: + """Join text with CJK-aware spacing (pangu style). + + Spacing rules at the join boundary: + + - **CJK ↔ CJK**: no space (both characters wide). + - **ASCII alnum ↔ CJK ideograph**: insert one space. This is "pangu + spacing" — the standard practice of separating Han characters from + Latin letters / digits in mixed CJK/English text. The original + content almost always has these spaces; they are lost when Claude + hard-wraps at the column boundary. + - **CJK punctuation ↔ anything**: no space. Punctuation like ``，`` + ``。`` ``）`` ``（`` clings to its neighbor. + - **ASCII ↔ ASCII**: insert one space (English word boundary). + """ + left_s = left.rstrip() + right_s = right_content.lstrip() + if not left_s or not right_s: + return left_s + right_s + last_ch = left_s[-1] + first_ch = right_s[0] + + last_wide = is_wide_char(last_ch) + first_wide = is_wide_char(first_ch) + + if last_wide and first_wide: + # Both CJK — no space. + return left_s + right_s + + if last_wide or first_wide: + # Mixed CJK/ASCII boundary — apply pangu spacing. + # In addition to alnum, certain symbols that attach to numbers + # or abbreviations (%, #, +, :) also trigger pangu spacing because + # they're part of the same "token" as the adjacent number/word. + _PANGU_SYMBOLS = "%#+:" + if last_wide: + # CJK → ASCII: space if CJK ideograph + ASCII alnum/symbol. + if _is_cjk_ideograph(last_ch) and (first_ch.isalnum() or first_ch in _PANGU_SYMBOLS): + return left_s + " " + right_s + return left_s + right_s + else: + # ASCII → CJK: space if ASCII alnum/symbol + CJK ideograph. + if (last_ch.isalnum() or last_ch in _PANGU_SYMBOLS) and _is_cjk_ideograph(first_ch): + return left_s + " " + right_s + return left_s + right_s + + # Both ASCII — mid-token detection before adding space. + # Path continuation: alnum + / (e.g., "documents" + "/05-team") + # Hyphen continuation: alnum + - (e.g., "ready" + "-together") + # Underscore continuation: _ + alnum (e.g., "BASE_" + "URL") + # Exception: -- prefix is a CLI flag (e.g., "run" + "--headed"). + if last_ch.isalnum() and first_ch in ("-", "/"): + if first_ch == "-" and right_s.startswith("--"): + pass # fall through to default (add space) + else: + return left_s + right_s + if last_ch in ("-", "/") and first_ch.isalnum(): + return left_s + right_s + # Underscore at identifier boundary (e.g., "E2E_PO_BASE_" + "URL") + if last_ch == "_" and first_ch.isalnum(): + return left_s + right_s + if last_ch.isalnum() and first_ch == "_": + return left_s + right_s + + # Default: word boundary → add space. + return left_s + " " + right_s + + +def raw_join(left: str, right: str) -> str: + """Strip trailing whitespace from *left*, leading spaces from *right*, concat.""" + return left.rstrip() + right.lstrip() + + +def _table_cell_content_join( + left: str, right: str, *, left_filled: bool = False +) -> str: + """Join two cell-content fragments from a multi-row table cell. + + Claude wraps long cell content across multiple physical rows at fixed + column widths. This function reassembles the original content by + detecting mid-word breaks (e.g. ``Contr`` + ``oller``) vs word + boundaries (e.g. ``Backend`` + ``Risk``). + + When *left_filled* is True, the left fragment filled its entire column + (≤1 trailing space before the ``│`` delimiter), meaning the split was + forced at the column boundary — almost certainly mid-word. This is the + strongest signal and overrides other heuristics. + + Heuristic priority: + 1. left_filled → raw join (column-boundary split) + 2. Continuation punctuation at boundary (``- _ . /``) → raw join + 3. Left ends with letter, right starts with lowercase → mid-word + 4. Left ends with digit, right starts with digit → mid-number + 5. Otherwise → smart_join (CJK-aware spacing) + """ + left_s = left.rstrip() + right_s = right.lstrip() + if not left_s or not right_s: + return left_s + right_s + + last_ch = left_s[-1] + first_ch = right_s[0] + + # Continuation punctuation at boundary — usually concatenate directly. + # Exception: double-hyphen at right boundary (``--flag``) is a CLI + # argument prefix, not a continuation of the left content. + if last_ch in ("-", "_", ".", "/") or first_ch in ("-", "_", ".", "/"): + if first_ch == "-" and right_s.startswith("--"): + pass # fall through to other checks + else: + return left_s + right_s + + if left_filled: + # Column-boundary split: content filled the cell width. + # The break is at a fixed position and likely mid-token, but we + # verify with character-class checks to avoid false positives + # like "Behavioral" + "Spec" (complete word at column edge). + if last_ch.isalpha() and first_ch.islower(): + return left_s + right_s + if last_ch.isupper() and first_ch.isupper(): + return left_s + right_s + if last_ch.isdigit() and first_ch.isdigit(): + return left_s + right_s + if last_ch.isalnum() and first_ch.isdigit(): + return left_s + right_s + if last_ch.isdigit() and first_ch.islower(): + # Hex hash continuation: c3df79 + b → c3df79b + return left_s + right_s + # Filled but no mid-word evidence → fall through to smart_join. + + # Everything else: word-boundary or ambiguous → CJK-aware spacing. + return smart_join(left_s, right_s) + + +def table_cell_join(left: str, right: str) -> str: + """Join table cell continuation, preserving column spacing. + + Unlike raw_join, this strips only the table indent (matching the left + side's indent) from the right side, keeping internal cell padding. + It also ensures a space between ``│`` and adjacent cell content. + """ + ls = left.rstrip() + if not ls: + return right.rstrip() + # Determine indent from left side (typically 5 spaces for plan tables) + indent = len(ls) - len(ls.lstrip()) + rs = right.rstrip() + # Strip exactly the indent, keep remaining whitespace (cell padding) + if len(rs) >= indent and rs[:indent].strip() == "": + rs = rs[indent:] + else: + rs = rs.lstrip() + if not rs: + return ls + # Ensure space between │ and non-│/non-space content + if ls[-1] == "│" and rs[0] not in " │": + return ls + " " + rs + if ls[-1] not in " │" and rs[0] == "│": + return ls + " " + rs + return ls + rs + + +def boundary_aware_join(left: str, right: str) -> str: + """Join with heuristic for mid-word vs word-boundary splits. + + If *left* had a trailing space before stripping, the hard wrap was at a + word boundary -- preserve one space. Otherwise the wrap split mid-word + -- concatenate directly (no space). + """ + # Check if left had trailing whitespace (word-boundary wrap). + left_had_trailing_space = left != left.rstrip() + left_s = left.rstrip() + right_s = right.lstrip() + if not left_s or not right_s: + return left_s + right_s + if left_had_trailing_space: + # Word-boundary wrap: preserve spacing via smart_join which + # handles pangu spacing (ASCII alnum ↔ CJK ideograph). + return smart_join(left_s, right_s) + # Mid-word wrap: no space. + return left_s + right_s + + +def _dw_aware_join(left: str, right: str, left_dw: int) -> str: + """Join with display-width-aware mid-word detection. + + Claude drops the trailing space at wrap points, making it impossible to + distinguish word-boundary from mid-word breaks by whitespace alone. + This function uses the display width of the *left* physical line to + resolve the ambiguity: + + - **dw < 74**: the line ended well below the wrap column (~76) — the + break was at a natural word boundary → smart_join. + - **dw >= 75**: the line was forced to break near the column limit. + Use character-class heuristics: alpha→lower = likely mid-word (raw + join); all other transitions = word boundary (smart_join). + """ + if left_dw < 75: + return smart_join(left, right) + + # Near wrap limit — check trailing space first. + left_had_trailing = left != left.rstrip() + left_s = left.rstrip() + right_s = right.lstrip() + if not left_s or not right_s: + return left_s + right_s + + if left_had_trailing: + return smart_join(left_s, right_s) + + # No trailing space, near wrap limit. + # Character-class heuristics for mid-word detection. + last_ch = left_s[-1] + first_ch = right_s[0] + # alpha→lower: mid-word (e.g., "Backgrou" + "nd") + if last_ch.isalpha() and first_ch.islower(): + return left_s + right_s + # Path/hyphenated-name continuations: slash or hyphen at boundary + # (e.g., ".claude/skills" + "/generating-..." or "ready" + "-together") + if last_ch.isalnum() and first_ch in ("-", "/"): + return left_s + right_s + if last_ch in ("-", "/") and first_ch.isalnum(): + return left_s + right_s + # digit→alpha (e.g., "md-e" + "2e-section" — hex/version fragments) + if last_ch.isalpha() and first_ch.isdigit(): + return left_s + right_s + + # All other cases: treat as word boundary. + return smart_join(left_s, right_s) + + +def _bullet_join(left: str, right: str) -> str: + """Join a bullet line with its wrapped continuation. + + Like smart_join, but with mid-word detection: when left ends with an + ASCII letter and right (after stripping) starts with a lowercase ASCII + letter, concatenate directly (the hard-wrap split a word mid-token, + e.g. ``RiskModelAss`` + ``ignment``). Otherwise delegate to smart_join + for CJK-aware spacing. + """ + left_s = left.rstrip() + right_s = right.lstrip() + if not left_s or not right_s: + return left_s + right_s + last_ch = left_s[-1] + first_ch = right_s[0] + # Mid-word split: left ends with a letter, right starts with lowercase. + if last_ch.isalpha() and first_ch.islower(): + return left_s + right_s + # Hyphenated names / paths split at boundary. + # e.g. "ready" + "-together-project" or "skills" + "/generating" + if last_ch.isalnum() and first_ch in ("-", "/"): + return left_s + right_s + if last_ch in ("-", "/") and first_ch.isalnum(): + return left_s + right_s + return smart_join(left_s, right_s) + + +# --------------------------------------------------------------------------- +# Line classification helpers +# --------------------------------------------------------------------------- + +# Markers that ALWAYS start a new logical line (never join TO these). +_USER_PROMPT_RE = re.compile(r"^❯ ") +_CLAUDE_ACTION_RE = re.compile(r"^● ") +_THINKING_RE = re.compile(r"^✻ ") +_HR_RE = re.compile(r"^ ---") +_BOX_HR_RE = re.compile(r"^ ────") +_AGENT_TREE_RE = re.compile(r"^ [├└]─") +_TOOL_RESULT_RE = re.compile(r"^ ⎿") +_BULLET_RE = re.compile(r"^ - ") +_NUMBERED_RE = re.compile(r"^ \d+\. ") + +# Indented bullets / numbered items inside plan blocks (5-space indent). +_PLAN_BULLET_RE = re.compile(r"^ - ") +_PLAN_NUMBERED_RE = re.compile(r"^ \d+\. ") + +# Tool call openers (● ToolName( ...). +_TOOL_CALL_RE = re.compile( + r"^● (?:Bash|Read|Write|Glob|Grep|Edit|Update|Searched|NotebookEdit)\(" +) + +# Table box-drawing characters. +_TABLE_CORNERS = set("┐┤┘") + + +def _is_truly_empty(line: str) -> bool: + """A truly empty line (zero length after stripping the newline).""" + return len(line) == 0 + + +def _is_structural_break(line: str) -> bool: + """Return True if *line* is a structural marker that must never be joined TO.""" + if _is_truly_empty(line): + return True + if _USER_PROMPT_RE.match(line): + return True + if _CLAUDE_ACTION_RE.match(line): + return True + if _THINKING_RE.match(line): + return True + if _HR_RE.match(line): + return True + if _BOX_HR_RE.match(line): + return True + if _AGENT_TREE_RE.match(line): + return True + if _TOOL_RESULT_RE.match(line): + return True + if _BULLET_RE.match(line): + return True + if _NUMBERED_RE.match(line): + return True + return False + + +# Regex for CJK labels like 模块:, 输出文件:, 状态:, 覆盖范围: +_CJK_LABEL_RE = re.compile(r"[\u4e00-\u9fff]{1,6}[:：]") +# Regex for English labelled list items: "Phase 1:", "Step 2:", "Layer 3:" +_LABELLED_ITEM_RE = re.compile(r"[A-Z]\w+ \d+[:.] ") + + +def _is_continuation_fragment(nl: str, acc: str) -> bool: + """Return True if *nl* looks like a wrapped continuation of *acc*. + + This is the core predicate for joining 2-space-indented paragraph text. + Instead of asking "was the current line wrapped?" (fragile dw threshold), + it asks "does the NEXT line look like a continuation fragment?" + + A continuation fragment has NO structural identity — it is not a new + bullet, numbered item, labelled field, or structural marker. It + typically starts with a lowercase letter, CJK ideograph, or is a short + uppercase fragment of a sentence that wrapped mid-phrase. + """ + # Must be 2-space indent (not deeper, not tool result). + if not nl.startswith(" "): + return False + if nl.startswith(" "): # 5+ space = plan/tool block + return False + if nl.startswith(" ⎿"): + return False + if _is_structural_break(nl): + return False + + stripped = nl.lstrip() + if not stripped: + return False + + # --- New-item patterns (NOT a continuation) --- + if stripped.startswith("- "): + return False + if re.match(r"\d+[.)] ", stripped): + return False + if _LABELLED_ITEM_RE.match(stripped): + return False + # CJK labels: 模块:, 输出文件:, 状态:, 覆盖范围: etc. + if _CJK_LABEL_RE.match(stripped): + return False + # Column layout (side-by-side comparison with internal spacing). + if " " in stripped: # 8+ internal spaces + return False + + nl_dw = display_width(nl.rstrip()) + # A "continuation" line that is itself full-width is likely independent. + if nl_dw >= 76: + return False + + first_ch = stripped[0] + + # --- Strong continuation signals --- + # Lowercase → mid-sentence continuation. + if first_ch.islower(): + return True + # CJK ideograph → continuing Chinese text. + if _is_cjk_ideograph(first_ch): + return True + # CJK/fullwidth punctuation → continues previous CJK content. + if is_wide_char(first_ch) and not _is_cjk_ideograph(first_ch): + return True + # Opening bracket → e.g. "(c67e5ded-..." UUID, list in parens. + if first_ch in ("(", "[", "{", "（", "「", "【"): + return True + # Hyphen/slash continuation: acc ends with alnum, next starts with -//. + # (e.g. "ready" + "-together-project", "skills" + "/generating") + if first_ch in ("-", "/"): + _acc_s = acc.rstrip() + if _acc_s and _acc_s[-1].isalnum(): + return True + + # --- Check if accumulated text signals continuation --- + acc_stripped = acc.rstrip() + if acc_stripped: + last_acc = acc_stripped[-1] + # Continuation operators at end of acc → next line must continue. + if last_acc in (",", "+", "→", "=", "&", "|", "、"): + return True + # Acc ends with sentence-terminal → next line is a new sentence. + if last_acc in "。.！!？?；;": + return False + + # --- Uppercase start: ambiguous --- + # Short fragment (< 55 dw) is likely a sentence fragment. + if nl_dw < 55: + return True + return False + + +def _is_plan_continuation_fragment(nl: str, acc: str) -> bool: + """Return True if *nl* looks like a wrapped continuation in 5-space plan text. + + Same design philosophy as _is_continuation_fragment: examine the NEXT + line's content to decide if it is a new item or a continuation fragment. + """ + if not nl.startswith(" "): + return False + if nl.startswith(" "): # 7+ space = deeper indent, separate item + return False + if _is_truly_empty(nl): + return False + if _is_structural_break(nl): + return False + if _is_plan_structural(nl): + return False + + stripped = nl.lstrip() + if not stripped: + return False + + # CJK labels (模块:, 输出文件:, 状态:) + if _CJK_LABEL_RE.match(stripped): + return False + # English labelled items with number (Phase 1:, Step 2:) + if _LABELLED_ITEM_RE.match(stripped): + return False + # English labels without number (Plan:, Context:, Summary:) + if re.match(r"[A-Z][a-zA-Z]+: ", stripped): + return False + # ASCII terminal labels (error:, hint:, remote:) + if re.match(r"[a-z]+: ", stripped): + return False + # Diff output line numbers (e.g., "600 - **Test Data**:") + # Pattern: 1-5 digits followed by 2+ spaces (diff line number format). + if re.match(r"\d{1,5}\s{2,}", stripped): + return False + # Column layout + if " " in stripped: + return False + + nl_dw = display_width(nl.rstrip()) + if nl_dw >= 76: + return False + + first_ch = stripped[0] + + # --- New-item patterns that start with lowercase --- + # ASCII labels like "error:", "hint:", "remote:" are terminal output + # lines, not continuations. They start a new message. + if re.match(r"[a-z]+: ", stripped): + return False + + # --- Strong continuation signals --- + if first_ch.islower(): + return True + if _is_cjk_ideograph(first_ch): + return True + if is_wide_char(first_ch) and not _is_cjk_ideograph(first_ch): + return True + if first_ch in ("(", "[", "{", "（", "「", "【"): + return True + # Hyphen/slash continuation (compound names, paths). + if first_ch in ("-", "/"): + _acc_s = acc.rstrip() + if _acc_s and _acc_s[-1].isalnum(): + return True + + # Non-alnum, non-CJK, non-bracket starts (!, #, >, *, etc.) + # are structural markers in terminal output, not continuations. + if not first_ch.isalnum(): + return False + + # --- Check accumulated text ending --- + acc_stripped = acc.rstrip() + if acc_stripped: + last_acc = acc_stripped[-1] + if last_acc in (",", "+", "→", "=", "&", "|", "、"): + return True + if last_acc in "。.！!？?；;": + return False + + # Uppercase start, ambiguous: short fragment = likely continuation. + if nl_dw < 55: + return True + return False + + +def _is_plan_structural(line: str) -> bool: + """Structural markers within 5-space-indented plan blocks.""" + if _PLAN_BULLET_RE.match(line): + return True + if _PLAN_NUMBERED_RE.match(line): + return True + stripped = line.lstrip() + if stripped.startswith("##"): + return True + # Box-drawing separators within plan blocks (5-space indent). + # The 2-space variant is caught by _BOX_HR_RE in _is_structural_break, + # but 5-space-indented ones slip through. + if stripped.startswith("────"): + return True + # Markdown HR within plan blocks (5-space indent). + if stripped == "---": + return True + # Tree connector (standalone │ used in ASCII dependency diagrams). + # Must not be confused with table data rows (which have 2+ │ chars). + if stripped == "│": + return True + # File-tree lines (├── or └── patterns) within plan blocks. + if stripped.startswith("├──") or stripped.startswith("└──"): + return True + # Expansion indicators ("… +N lines (ctrl+o to expand)"). + if stripped.startswith("…"): + return True + return False + + +def _has_continuation_signal(line: str) -> bool: + """Detect if *line* was almost certainly hard-wrapped mid-content. + + Lines ending with a trailing comma, a CJK character, or an unclosed + bracket are continuation signals — they indicate the content continues + on the next line regardless of how narrow the display width is. + """ + stripped = line.rstrip() + if not stripped: + return False + last_ch = stripped[-1] + if last_ch == ",": + return True + if is_wide_char(last_ch): + return True + if last_ch in ("(", "[", "{"): + return True + return False + + +def _has_unclosed_bracket(line: str) -> bool: + """Detect if *line* contains an opening bracket with no matching close. + + An unclosed bracket means the parenthetical/list continues on the next + physical line — a strong continuation signal regardless of display width. + This catches cases like ``Requirements（P0/P1 分层，每个 Feature`` where + Claude wraps at a word boundary well below the column limit because the + remaining CJK text would push past it. + """ + _PAIRS = (("(", ")"), ("[", "]"), ("{", "}"), + ("（", "）"), ("「", "」"), ("【", "】")) + for open_ch, close_ch in _PAIRS: + if open_ch in line and close_ch not in line: + return True + return False + + +def _looks_like_mid_word(left: str, right: str) -> bool: + """Heuristic: detect if a tool-call wrap split a token mid-character. + + Returns True when both sides appear to be fragments of one continuous + filesystem path or identifier. This is deliberately conservative -- + when in doubt, return False so a space gets inserted. + """ + if not left or not right: + return False + lc = left[-1] + rc = right[0] + # Mid-path: one side has '/' at the boundary. + if lc == "/" or rc == "/": + return True + # Mid-word with hyphen/underscore continuation + # (e.g., "/skills/gener" + "ating-e2e-test-suite"). + # Must distinguish from command-argument boundaries + # (e.g., "git add" + "test-cases/...") by checking the character + # preceding the left side's last token. + if lc.isalpha() and rc.isalpha() and lc.islower() and rc.islower(): + first_token = "" + for ch in right: + if ch.isalnum() or ch in "-_.": + first_token += ch + else: + break + if "-" in first_token or "_" in first_token: + # Find start of last token on the left side + last_token_start = len(left) + while last_token_start > 0 and ( + left[last_token_start - 1].isalnum() + or left[last_token_start - 1] in "-_." + ): + last_token_start -= 1 + # If preceded by '/', it might be mid-path -- but check if the + # token is a complete filename (has file extension like .md/.py). + if last_token_start > 0 and left[last_token_start - 1] == "/": + last_token = left[last_token_start:] + if re.search(r"\.\w{1,5}$", last_token): + return False # Complete filename, not a fragment + return True # Mid-path fragment (e.g., /skills/gener) + # If preceded by space or start-of-string, it's a word + # boundary (e.g., "git add" + "test-cases") -> insert space + return False + return False + + +# --------------------------------------------------------------------------- +# Table detection helpers +# --------------------------------------------------------------------------- + +def is_table_border_split(current: str, next_line: str) -> bool: + """Detect a table border that was split across two lines.""" + cs = current.rstrip() + if not cs or cs[-1] != "─": + return False + ns = next_line.lstrip() + if not ns: + return False + return ns[0] == "─" and ns.rstrip()[-1] in _TABLE_CORNERS + + +def _is_table_border_line(line: str) -> bool: + """Check if line is a table border (├─, └─, ┌─ patterns).""" + stripped = line.lstrip() + if not stripped: + return False + return stripped[0] in "├└┌" and "─" in stripped + + +def _count_expected_pipes(border_line: str) -> int: + """Count expected │ per data row from a ┌ or ├ border line. + + A border like ``┌──┬──┬──┐`` has 2 ``┬`` → 3 columns → 4 ``│`` per row. + """ + return border_line.count("┬") + 2 + + +def _parse_column_widths(border_line: str) -> list[int]: + """Extract column display widths from a ┌ or ├ border line. + + Splits by column separators (┬ or ┼) and counts ``─`` chars per segment. + Returns a list of column widths (inner content width, not including │). + """ + stripped = border_line.strip() + if len(stripped) < 2: + return [] + inner = stripped[1:-1] # Remove corner chars (┌/┐ or ├/┤) + segments = re.split("[┬┼]", inner) + return [len(seg) for seg in segments] + + +def _repad_table_row(row: str, col_widths: list[int]) -> str: + """Re-pad each cell in a table data row to match the column widths. + + Preserves existing left padding and content; only adds right-padding + so each cell reaches the correct display width from the border. + """ + parts = row.split("│") + # parts[0] = indent before first │; parts[-1] = after last │ (empty) + if len(parts) < 3: + return row + indent = parts[0] + cells = parts[1:-1] + if len(cells) != len(col_widths): + return row # Column count mismatch — leave unchanged + new_cells = [] + for cell, width in zip(cells, col_widths): + cell_dw = display_width(cell) + if cell_dw < width: + new_cells.append(cell + " " * (width - cell_dw)) + else: + new_cells.append(cell) + return indent + "│" + "│".join(new_cells) + "│" + + +# --------------------------------------------------------------------------- +# Statistics +# --------------------------------------------------------------------------- + +@dataclass +class Stats: + input_lines: int = 0 + output_lines: int = 0 + user_lines_joined: int = 0 + claude_lines_joined: int = 0 + table_borders_fixed: int = 0 + table_cells_fixed: int = 0 + tool_calls_fixed: int = 0 + tool_results_fixed: int = 0 + agent_tree_fixed: int = 0 + bullet_text_joined: int = 0 + plan_text_joined: int = 0 + table_multirow_merged: int = 0 + table_borders_realigned: int = 0 + box_rows_merged: int = 0 + + def summary(self) -> str: + lines = [ + "--- Statistics ---", + f" Input lines: {self.input_lines}", + f" Output lines: {self.output_lines}", + f" User lines joined: {self.user_lines_joined}", + f" Claude lines joined: {self.claude_lines_joined}", + f" Table borders fixed: {self.table_borders_fixed}", + f" Table cells fixed: {self.table_cells_fixed}", + f" Table rows merged: {self.table_multirow_merged}", + f" Borders realigned: {self.table_borders_realigned}", + f" Box rows merged: {self.box_rows_merged}", + f" Tool calls fixed: {self.tool_calls_fixed}", + f" Tool results fixed: {self.tool_results_fixed}", + f" Agent tree fixed: {self.agent_tree_fixed}", + f" Bullet text joined: {self.bullet_text_joined}", + f" Plan text joined: {self.plan_text_joined}", + ] + return "\n".join(lines) + + +# --------------------------------------------------------------------------- +# Main processing logic +# --------------------------------------------------------------------------- + + +def process(lines: list[str], stats: Stats) -> list[str]: + """Process all *lines* and return the list of fixed output lines.""" + stats.input_lines = len(lines) + output: list[str] = [] + i = 0 + n = len(lines) + + def peek(offset: int = 1) -> str | None: + idx = i + offset + return lines[idx] if idx < n else None + + while i < n: + line = lines[i] + + # --------------------------------------------------------------- + # 1) User prompt blocks (❯ at column 0, continuations at dw=76) + # --------------------------------------------------------------- + if _USER_PROMPT_RE.match(line): + i = _process_user_block(lines, i, n, output, stats) + continue + + # --------------------------------------------------------------- + # 2) Table border split → join, then enter table region + # --------------------------------------------------------------- + next_line = peek() + if next_line is not None and is_table_border_split(line, next_line): + acc = raw_join(line, next_line) + stats.table_borders_fixed += 1 + i += 2 + while i < n and is_table_border_split(acc, lines[i]): + acc = raw_join(acc, lines[i]) + stats.table_borders_fixed += 1 + i += 1 + output.append(acc) + # If this was a ┌ border, process the full table body + if "┌" in acc: + expected_pipes = _count_expected_pipes(acc) + col_widths = _parse_column_widths(acc) + i = _process_table_body( + lines, i, n, output, stats, expected_pipes, col_widths, + ) + continue + + # --------------------------------------------------------------- + # 2b) Non-split ┌ border → enter table body processor + # --------------------------------------------------------------- + stripped_for_border = line.lstrip() + if ( + stripped_for_border.startswith("┌") + and "─" in stripped_for_border + and "┐" in stripped_for_border + ): + expected_pipes = _count_expected_pipes(line) + col_widths = _parse_column_widths(line) + output.append(line) + i += 1 + i = _process_table_body( + lines, i, n, output, stats, expected_pipes, col_widths, + ) + continue + + # --------------------------------------------------------------- + # 3) Table cell row (│ in line, outside tracked table region) + # Fallback for tables whose ┌ border was not split and was + # already emitted before we entered this logic. + # --------------------------------------------------------------- + if "│" in line: + stripped = line.lstrip() + if stripped.startswith("│") or stripped.endswith("│"): + # Likely a table row. Check if PREVIOUS output line was a + # ┌ or ├ border to determine expected_pipes. + expected_pipes = 0 + fallback_col_widths: list[int] = [] + for prev in reversed(output): + ps = prev.strip() + if ps and (ps[0] in "┌├"): + expected_pipes = _count_expected_pipes(prev) + fallback_col_widths = _parse_column_widths(prev) + break + if ps and ps[0] not in "│": + break + if expected_pipes > 0: + acc = line.rstrip() + pipe_count = acc.count("│") + i += 1 + while pipe_count < expected_pipes and i < n: + nl = lines[i] + if _is_truly_empty(nl): + break + if _is_table_border_line(nl): + break + # Check for border split on this line + if i + 1 < n and is_table_border_split(nl, lines[i + 1]): + break + acc = table_cell_join(acc, nl) + pipe_count = acc.count("│") + stats.table_cells_fixed += 1 + i += 1 + if fallback_col_widths: + acc = _repad_table_row(acc, fallback_col_widths) + output.append(acc) + continue + + # --------------------------------------------------------------- + # 4) Tool call continuation (● Bash(, ● Read(, etc.) + # Continuations are 6-space indented. + # --------------------------------------------------------------- + if _TOOL_CALL_RE.match(line): + acc = line + i += 1 + while i < n: + nl = lines[i] + # 6-space continuation (tool call argument wrapping) + if nl.startswith(" ") and not _is_structural_break(nl): + # Strip the 6-char continuation indent, preserving + # any additional whitespace from the original. + right_part = nl[6:] + left_s = acc.rstrip() + if right_part and right_part[0] == " ": + # Extra space means the original had whitespace + # at this position -- preserve it. + acc = left_s + right_part + elif left_s and _looks_like_mid_word(left_s, right_part): + # Mid-word/mid-path split: concatenate directly. + acc = left_s + right_part + else: + # Argument boundary where the space was consumed + # by wrapping. Restore it. + acc = left_s + " " + right_part + stats.tool_calls_fixed += 1 + i += 1 + else: + break + output.append(acc) + continue + + # --------------------------------------------------------------- + # 5) Tool result continuation ( ⎿ ... at dw>=74) + # --------------------------------------------------------------- + if _TOOL_RESULT_RE.match(line): + last_raw_dw = display_width(line.rstrip()) + acc = line + i += 1 + # Phase 1: join 5-space continuations of the ⎿ line itself. + # Only join when the PREVIOUS raw line was near the wrap limit + # (dw >= 74). Lines well below the limit ended naturally — + # subsequent 5-space lines are separate output lines (e.g. + # git log entries), not wrapped continuations. + # Character-class check: alpha→lower = mid-word (raw join); + # all other transitions = word boundary (smart_join). + while last_raw_dw >= 74 and i < n: + nl = lines[i] + if nl.startswith(" ") and not _is_structural_break(nl) and not _is_plan_structural(nl): + prev_dw = last_raw_dw + last_raw_dw = display_width(nl.rstrip()) + acc = _dw_aware_join(acc, nl, prev_dw) + stats.tool_results_fixed += 1 + i += 1 + else: + break + # Phase 1 fallback: if ⎿ line ends with trailing space + # (word-boundary wrap just below the 74 threshold), join + # short continuation fragments. This catches lines like + # "Plan saved to: ... · /plan to " + "edit" (dw=72). + while i < n and acc != acc.rstrip(): + nl = lines[i] + if nl.startswith(" ") and not nl.startswith(" "): + if _is_plan_continuation_fragment(nl, acc): + # Trailing space = word-boundary wrap → smart_join + # (not _bullet_join, which would raw-join alpha→lower + # like "to" + "edit" → "toedit"). + acc = smart_join(acc, nl) + stats.tool_results_fixed += 1 + i += 1 + continue + break + output.append(acc) + + # Phase 2: handle remaining tool output lines at 6-space indent. + # After the ⎿ line, tool output continues at 6-space indent. + # Each such line may itself be wrapped, with 5-space continuations. + # We emit each output line individually, joining only its + # wrapped fragments. + while i < n: + nl = lines[i] + if _is_truly_empty(nl): + break + if _is_structural_break(nl): + break + # Tool output lines use 6-space indent (5-space lines that + # aren't continuations would be plan text, etc.) + if not nl.startswith(" ") or nl.startswith(" "): + break + acc2 = nl + last_raw_dw2 = display_width(nl.rstrip()) + i += 1 + # Join continuations of this output line. + # Phase 2a: high-dw continuations at 5-6 space indent. + while last_raw_dw2 >= 74 and i < n: + nl2 = lines[i] + if nl2.startswith(" ") and not _is_structural_break(nl2) and not _is_plan_structural(nl2): + prev_dw2 = last_raw_dw2 + last_raw_dw2 = display_width(nl2.rstrip()) + acc2 = _dw_aware_join(acc2, nl2, prev_dw2) + stats.tool_results_fixed += 1 + i += 1 + else: + break + # Phase 2b: deeper-indented continuations when the + # output line ends with a continuation signal: + # - comma (list continuation) + # - trailing space (word-boundary wrap) + # - underscore (identifier split, e.g. E2E_PO_BASE_ + URL) + while i < n: + _acc2_s = acc2.rstrip() + if not _acc2_s: + break + _last2 = _acc2_s[-1] + _has_trailing = acc2 != _acc2_s + if _last2 not in (",", "_") and not _has_trailing: + break # no continuation signal + nl2 = lines[i] + # Only accept DEEPER-indented continuations (7+ spaces). + # Same-indent lines (6 spaces) are sibling entries in the + # tool output — e.g. separate diff lines that happen to + # have trailing padding spaces. + if not nl2.startswith(" ") or _is_structural_break(nl2): + break + nl2_dw = display_width(nl2.rstrip()) + if nl2_dw >= 76: + break # full-width = independent line + acc2 = smart_join(acc2, nl2) + last_raw_dw2 = nl2_dw + stats.tool_results_fixed += 1 + i += 1 + output.append(acc2) + continue + + # --------------------------------------------------------------- + # 6) Agent tree continuation (├─ or └─ at 3-space indent) + # --------------------------------------------------------------- + if _AGENT_TREE_RE.match(line): + dw = display_width(line.rstrip()) + acc = line + i += 1 + if dw >= 70: + while i < n: + nl = lines[i] + if _is_structural_break(nl): + break + nl_stripped = nl.lstrip() + # Never join lines that are sub-results (contain ⎿) + # or new tree nodes (├─, └─, │) + if "⎿" in nl_stripped: + break + if nl_stripped.startswith("├─") or nl_stripped.startswith("└─"): + break + if nl_stripped.startswith("│"): + break + # Same-indent non-structural continuation + if nl.startswith(" "): + acc = raw_join(acc, nl) + stats.agent_tree_fixed += 1 + i += 1 + else: + break + output.append(acc) + continue + + # --------------------------------------------------------------- + # 7) Claude narrative (● text with dw>=55+, NOT tool call/short marker) + # Threshold lowered from 77 to 55 because CJK-heavy lines end + # content well before the 77-80 column wrap limit (CJK chars take + # 2 columns each, so word boundaries fall earlier). + # --------------------------------------------------------------- + if _CLAUDE_ACTION_RE.match(line) and not _TOOL_CALL_RE.match(line): + dw = display_width(line.rstrip()) + acc = line + i += 1 + if dw >= 55 or _has_continuation_signal(line): + while i < n: + nl = lines[i] + if _is_structural_break(nl): + break + nl_dw = display_width(nl.rstrip()) + # 2-space continuation, short, not structural + if nl.startswith(" ") and not nl.startswith(" ⎿") and nl_dw < 82: + nl_stripped = nl.lstrip() + if nl_stripped.startswith("- "): + break + if re.match(r"\d+\. ", nl_stripped): + break + acc = smart_join(acc, nl) + stats.claude_lines_joined += 1 + i += 1 + else: + break + output.append(acc) + continue + + # --------------------------------------------------------------- + # 7b) Bullet item in Claude response ( - text) + # When a bullet line has high dw or ends with a continuation + # signal (CJK char, comma, etc.), its wrapped continuation + # on the next line (2-space indent, NOT a bullet/numbered) + # must be joined. Uses _bullet_join to handle mid-word + # breaks (e.g. "RiskModelAss" + "ignment") correctly. + # --------------------------------------------------------------- + if _BULLET_RE.match(line): + dw = display_width(line.rstrip()) + acc = line + i += 1 + if dw >= 55 or _has_continuation_signal(line) or _has_unclosed_bracket(line): + while i < n: + nl = lines[i] + if not _is_continuation_fragment(nl, acc): + break + acc = _bullet_join(acc, nl) + stats.bullet_text_joined += 1 + i += 1 + # Peek-ahead for plan-context bullets (5+ space indent). + # Short plan bullets (dw < 55) whose text wraps to the base + # 5-space indent won't enter the join loop above. Check if + # the next line is a non-structural 5-space continuation. + elif line.startswith(" ") and i < n: + nl = lines[i] + if ( + nl.startswith(" ") + and not _is_structural_break(nl) + and not _is_plan_structural(nl) + ): + nl_dw = display_width(nl.rstrip()) + if nl_dw < 70: + acc = smart_join(acc, nl) + stats.bullet_text_joined += 1 + i += 1 + output.append(acc) + continue + + # --------------------------------------------------------------- + # 8) Numbered list item in Claude response ( N. text) + # --------------------------------------------------------------- + if _NUMBERED_RE.match(line): + dw = display_width(line.rstrip()) + acc = line + i += 1 + if dw >= 55 or _has_continuation_signal(line): + while i < n: + nl = lines[i] + if _is_structural_break(nl): + break + # Numbered list continuation is at 2-space indent + if nl.startswith(" ") and not nl.startswith(" ⎿"): + nl_stripped = nl.lstrip() + if nl_stripped.startswith("- "): + break + if re.match(r"\d+\. ", nl_stripped): + break + dw_nl = display_width(nl.rstrip()) + if dw_nl < 82: + acc = _bullet_join(acc, nl) + stats.claude_lines_joined += 1 + i += 1 + else: + break + else: + break + # Peek-ahead for plan-context numbered items (5+ space indent). + elif line.startswith(" ") and i < n: + nl = lines[i] + if ( + nl.startswith(" ") + and not _is_structural_break(nl) + and not _is_plan_structural(nl) + ): + nl_dw = display_width(nl.rstrip()) + if nl_dw < 70: + acc = _bullet_join(acc, nl) + stats.claude_lines_joined += 1 + i += 1 + output.append(acc) + continue + + # --------------------------------------------------------------- + # 8c) Claude paragraph text (2-space indent, standalone) + # Text paragraphs within Claude response blocks that aren't + # preceded by a ● marker — they appear after tables, bullet + # lists, or between structural elements. + # + # DESIGN: Instead of measuring the current line's dw to guess + # whether it was wrapped (fragile threshold), we examine the + # NEXT line via _is_continuation_fragment() to decide whether + # it is a new item or a continuation of the current paragraph. + # --------------------------------------------------------------- + if line.startswith(" ") and not line.startswith(" "): + acc = line + i += 1 + # Skip column-layout lines (side-by-side comparison format). + _content = line.strip() + if " " not in _content: # 8+ internal spaces = layout + while i < n: + nl = lines[i] + if _is_continuation_fragment(nl, acc): + acc = _bullet_join(acc, nl) + stats.claude_lines_joined += 1 + i += 1 + else: + break + output.append(acc) + continue + + # --------------------------------------------------------------- + # 9) Plan / indented text (5+ space indent) + # + # DESIGN: Like step 8c, uses next-line look-ahead via + # _is_plan_continuation_fragment() instead of dw thresholds. + # The join function (_dw_aware_join) still uses the last + # segment's dw to decide mid-word vs word-boundary joins. + # --------------------------------------------------------------- + if line.startswith(" "): + acc = line + last_seg_dw = display_width(line.rstrip()) + i += 1 + while i < n: + nl = lines[i] + if _is_plan_continuation_fragment(nl, acc): + prev_dw = last_seg_dw + last_seg_dw = display_width(nl.rstrip()) + acc = _dw_aware_join(acc, nl, prev_dw) + stats.plan_text_joined += 1 + i += 1 + else: + break + output.append(acc) + continue + + # --------------------------------------------------------------- + # 10) Default: emit as-is + # --------------------------------------------------------------- + output.append(line) + i += 1 + + # Post-processing: merge multi-row table cells. + # After the main loop, each physical data row is complete (correct pipe + # count, re-padded). But a single logical row may still span multiple + # physical rows when Claude wrapped cell content at column width. + # This pass collapses those into one row per logical cell. + output = _merge_multirow_table_cells(output, stats) + + # Post-processing: realign table borders. + # After merging, merged data rows may exceed original column widths. + # This pass recalculates border widths to match the widest data cells. + output = _realign_table_borders(output, stats) + + # Post-processing: merge wrapped text within single-column box items. + # Single-column boxes (exactly 2 │ per row) contain numbered/bulleted + # items that may wrap across multiple rows. This pass merges continuation + # lines back into their parent item. + output = _merge_singlecol_box_rows(output, stats) + + stats.output_lines = len(output) + return output + + +# --------------------------------------------------------------------------- +# Post-processing: multi-row table cell merge +# --------------------------------------------------------------------------- + +def _is_table_data_row(stripped: str) -> bool: + """Check if *stripped* (leading-whitespace-removed) is a multi-column table data row. + + A data row starts and ends with ``│`` and contains no ``─`` (which + would make it a border row). Requires at least 3 ``│`` (i.e. 2+ + columns) so that single-column boxes (which have only 2 ``│``) + are excluded — their rows are independent items, not wrapped cells. + """ + return ( + len(stripped) >= 5 + and stripped[0] == "│" + and stripped[-1] == "│" + and "─" not in stripped + and stripped.count("│") >= 3 + ) + + +def _merge_multirow_table_cells(lines: list[str], stats: Stats) -> list[str]: + """Collapse consecutive table data rows into single logical rows. + + Between border/separator rows (``┌├└``), Claude may emit multiple + physical data rows for one logical row when cell content exceeds the + column width. This function detects such groups and merges them, + joining cell content with ``_table_cell_content_join`` which handles + mid-word and CJK boundaries. + """ + result: list[str] = [] + i = 0 + n = len(lines) + + while i < n: + line = lines[i] + stripped = line.lstrip() + + if not _is_table_data_row(stripped): + result.append(line) + i += 1 + continue + + # Collect consecutive data rows (same table region). + group = [line] + j = i + 1 + while j < n: + next_stripped = lines[j].lstrip() + if _is_table_data_row(next_stripped): + group.append(lines[j]) + j += 1 + else: + break + + if len(group) == 1: + result.append(line) + i = j + continue + + # Multiple physical rows → merge into one logical row. + merged = _merge_row_group(group, stats) + result.append(merged) + i = j + + return result + + +def _merge_row_group(rows: list[str], stats: Stats) -> str: + """Merge a group of physical table data rows into one logical row.""" + first = rows[0] + indent = first[: len(first) - len(first.lstrip())] + + # Split each row into cell contents — keep both raw and stripped forms. + # The raw form preserves trailing spaces, which we use to detect whether + # the content filled the entire column (≤1 trailing space → mid-word split). + all_cells: list[list[str]] = [] + all_raw: list[list[str]] = [] + for row in rows: + parts = row.strip().split("│") + # parts[0] is empty (before first │), parts[-1] is empty (after last │) + raw_cells = parts[1:-1] + cells = [c.strip() for c in raw_cells] + all_cells.append(cells) + all_raw.append(raw_cells) + + num_cols = max(len(cells) for cells in all_cells) + + # Merge each column's fragments. + merged_cells: list[str] = [] + for col_idx in range(num_cols): + fragments: list[str] = [] + raw_fragments: list[str] = [] + for row_idx, row_cells in enumerate(all_cells): + if col_idx < len(row_cells) and row_cells[col_idx]: + fragments.append(row_cells[col_idx]) + raw_fragments.append(all_raw[row_idx][col_idx]) + + if not fragments: + merged_cells.append("") + elif len(fragments) == 1: + merged_cells.append(fragments[0]) + else: + acc = fragments[0] + for k in range(1, len(fragments)): + # Determine if the previous fragment filled its column. + prev_raw = raw_fragments[k - 1] + trailing_spaces = len(prev_raw) - len(prev_raw.rstrip()) + left_filled = trailing_spaces <= 1 + acc = _table_cell_content_join( + acc, fragments[k], left_filled=left_filled + ) + merged_cells.append(acc) + + # Reconstruct the row with 1-space padding per cell. + cell_parts = [f" {cell} " if cell else " " for cell in merged_cells] + merged = indent + "│" + "│".join(cell_parts) + "│" + + stats.table_multirow_merged += len(rows) - 1 + return merged + + +# --------------------------------------------------------------------------- +# Post-processing: merge wrapped text within single-column box items +# --------------------------------------------------------------------------- + +def _is_singlecol_data_row(line: str) -> bool: + """Check if *line* is a single-column box data row. + + A single-column data row starts and ends with ``│``, has exactly 2 + ``│`` characters, and contains no ``─`` (which would indicate a border). + """ + stripped = line.lstrip() + return ( + len(stripped) >= 3 + and stripped[0] == "│" + and stripped[-1] == "│" + and "─" not in stripped + and stripped.count("│") == 2 + ) + + +def _is_singlecol_border(line: str) -> bool: + """Check if *line* is a single-column box border (┌─┐, ├─┤, or └─┘).""" + stripped = line.lstrip() + if not stripped: + return False + return ( + stripped[0] in "┌├└" + and "─" in stripped + and stripped[-1] in "┐┤┘" + and "┬" not in stripped + and "┼" not in stripped + and "┴" not in stripped + ) + + +_ITEM_START_RE = re.compile(r"^\s*\d+\.\s") +_BULLET_START_RE = re.compile(r"^\s*[-*]\s") + + +def _merge_singlecol_box_rows(lines: list[str], stats: Stats) -> list[str]: + """Merge wrapped text within single-column box items. + + Single-column boxes (2 ``│`` per row) contain numbered/bulleted items + that may wrap across multiple rows. This function merges continuation + lines back into their parent item while keeping separate items on + separate rows. + + Title boxes (a single data row between borders) are left untouched. + """ + result: list[str] = [] + i = 0 + n = len(lines) + + while i < n: + line = lines[i] + + # Look for the start of a single-column box (┌ border with no ┬). + if not _is_singlecol_border(line) or not line.lstrip().startswith("┌"): + result.append(line) + i += 1 + continue + + # Found a ┌ border. Collect the entire box (border + data + └). + box_lines: list[str] = [line] + j = i + 1 + found_close = False + while j < n: + if _is_singlecol_data_row(lines[j]): + box_lines.append(lines[j]) + j += 1 + elif _is_singlecol_border(lines[j]): + box_lines.append(lines[j]) + if lines[j].lstrip().startswith("└"): + found_close = True + j += 1 + break + elif lines[j].lstrip().startswith("├"): + j += 1 + else: + j += 1 + break + else: + break # Non-box line — box ended unexpectedly + + if not found_close: + # Incomplete box — emit as-is. + for bl in box_lines: + result.append(bl) + i = j + continue + + # Extract data rows (skip borders). + data_indices = [ + idx for idx, bl in enumerate(box_lines) if _is_singlecol_data_row(bl) + ] + + # Title boxes: single data row between borders → skip merging. + if len(data_indices) <= 1: + for bl in box_lines: + result.append(bl) + i = j + continue + + # Group data rows into logical items and merge continuations. + data_rows = [box_lines[idx] for idx in data_indices] + merged_contents = _merge_box_items(data_rows, stats) + + # Determine whether box borders need to grow. + indent = line[: len(line) - len(line.lstrip())] + + # Get current box inner width from the ┌ border. + top_border = line.lstrip() + border_inner_dw = display_width(top_border) - 2 + + # Compute max content width after merge. + max_content_dw = 0 + for content in merged_contents: + # Content needs 2 spaces padding (1 left + 1 right minimum). + content_dw = display_width(content) + 2 + max_content_dw = max(max_content_dw, content_dw) + + new_inner_width = max(border_inner_dw, max_content_dw) + + # Rebuild the box: borders + merged data rows. + # Emit ┌ border (potentially wider). + result.append( + _rebuild_singlecol_border(top_border, new_inner_width, indent) + ) + + # Emit merged data rows. + for content in merged_contents: + padded = _pad_singlecol_content( + " " + content + " ", new_inner_width, + ) + result.append(indent + "│" + padded + "│") + + # Emit any ├ and └ borders (potentially wider). + for idx, bl in enumerate(box_lines): + if idx == 0: + continue # Already emitted ┌ + stripped_bl = bl.lstrip() + if stripped_bl and stripped_bl[0] in "├└" and "─" in stripped_bl: + result.append( + _rebuild_singlecol_border(stripped_bl, new_inner_width, indent) + ) + + i = j + + return result + + +def _merge_box_items( + data_rows: list[str], stats: Stats, +) -> list[str]: + """Merge continuation rows within each logical item of a single-column box. + + Returns a list of merged content strings (one per logical item). + """ + # Parse content from each row (strip │ and whitespace). + contents: list[str] = [] + for row in data_rows: + stripped = row.lstrip() + inner = stripped[1:-1] # Remove │ ... │ + contents.append(inner.strip()) + + # Group into logical items. + items: list[list[str]] = [] + for content in contents: + if _ITEM_START_RE.match(content) or _BULLET_START_RE.match(content): + items.append([content]) + elif not items: + items.append([content]) + else: + items[-1].append(content) + + # Merge fragments within each item. + merged: list[str] = [] + for fragments in items: + if len(fragments) == 1: + merged.append(fragments[0]) + else: + acc = fragments[0] + for frag in fragments[1:]: + acc = _table_cell_content_join(acc, frag) + stats.box_rows_merged += 1 + merged.append(acc) + + return merged + + +def _pad_singlecol_content(content: str, inner_width: int) -> str: + """Pad content to fill *inner_width* display columns inside a box.""" + content_dw = display_width(content) + if content_dw >= inner_width: + return content + return content + " " * (inner_width - content_dw) + + +def _rebuild_singlecol_border( + stripped: str, inner_width: int, indent: str, +) -> str: + """Rebuild a single-column box border to the given inner width. + + Preserves styled headers like ``┌─── Title ───┐`` by detecting + embedded text and re-centering it. + """ + left_corner = stripped[0] + right_corner = stripped[-1] + + # Check for embedded title text (e.g., ┌─── Pre-Suite ───┐). + inner = stripped[1:-1] + title_match = re.search(r"([^─]+)", inner) + if title_match: + title_text = title_match.group(1).strip() + if title_text: + # Styled header border: ─── Title ─── + title_with_spaces = f" {title_text} " + title_dw = display_width(title_with_spaces) + remaining = inner_width - title_dw + left_dashes = max(remaining // 2, 3) + right_dashes = max(remaining - left_dashes, 3) + return ( + indent + + left_corner + + "─" * left_dashes + + title_with_spaces + + "─" * right_dashes + + right_corner + ) + + # Plain border (all ─). + return indent + left_corner + "─" * inner_width + right_corner + + +# --------------------------------------------------------------------------- +# Post-processing: realign table borders after multi-row merge +# --------------------------------------------------------------------------- + +def _realign_table_borders(lines: list[str], stats: Stats) -> list[str]: + """Recalculate border widths to match (potentially wider) merged data rows. + + After ``_merge_multirow_table_cells`` collapses multi-row cells, the + merged content may exceed the original column widths encoded in the + border rows. This pass: + + 1. Identifies contiguous table *regions* (runs of border + data rows). + 2. For each region, computes the maximum display-width per column + across all data rows. + 3. Regenerates every border row with the correct ``─`` widths. + 4. Re-pads every data row so cells align with the new borders. + """ + result: list[str] = [] + i = 0 + n = len(lines) + + while i < n: + line = lines[i] + stripped = line.lstrip() + + # Detect start of a table region (┌ border). + if stripped.startswith("┌") and "─" in stripped and "┐" in stripped: + # Collect every line in this table region. + region_start = i + region: list[str] = [line] + i += 1 + while i < n: + s = lines[i].lstrip() + if _is_table_data_row(s): + region.append(lines[i]) + i += 1 + elif s and s[0] in "├└" and "─" in s: + region.append(lines[i]) + i += 1 + if s[0] == "└": + break # End of table + else: + break # Non-table line — region ended unexpectedly + + realigned = _realign_table_region(region, stats) + result.extend(realigned) + continue + + result.append(line) + i += 1 + + return result + + +def _realign_table_region(region: list[str], stats: Stats) -> list[str]: + """Realign borders and data rows within a single table region.""" + # Separate border and data rows; determine indent from first line. + first = region[0] + indent = first[: len(first) - len(first.lstrip())] + + # Collect data rows and their per-cell display widths. + data_rows: list[tuple[int, list[str]]] = [] # (index, cells) + border_indices: list[int] = [] + + for idx, row in enumerate(region): + stripped = row.lstrip() + if _is_table_data_row(stripped): + parts = stripped.split("│") + # parts[0] = '' (before first │), parts[-1] = '' (after last │) + cells = parts[1:-1] + data_rows.append((idx, cells)) + elif stripped and stripped[0] in "┌├└": + border_indices.append(idx) + + if not data_rows: + return region # No data rows — nothing to realign + + # Determine column count from data rows. + num_cols = max(len(cells) for _, cells in data_rows) + if num_cols == 0: + return region + + # Compute max display-width per column across all data rows. + # Each cell has 1-space padding on each side, so content width is what + # we see between the padding. But we measure the full cell (including + # padding) to get the column width that the border must span. + max_widths: list[int] = [0] * num_cols + for _, cells in data_rows: + for col_idx, cell in enumerate(cells): + if col_idx < num_cols: + cw = display_width(cell) + if cw > max_widths[col_idx]: + max_widths[col_idx] = cw + + # Ensure minimum width of 3 (1 space + 1 char + 1 space). + max_widths = [max(w, 3) for w in max_widths] + + # Check if any realignment is needed by comparing with current border. + current_widths = _parse_column_widths(region[border_indices[0]]) + if current_widths == max_widths: + return region # Already aligned + + # Rebuild the region. + rebuilt: list[str] = [] + for idx, row in enumerate(region): + stripped = row.lstrip() + if idx in border_indices: + rebuilt.append(_rebuild_border(stripped[0], stripped[-1], max_widths, indent)) + stats.table_borders_realigned += 1 + elif _is_table_data_row(stripped): + rebuilt.append(_repad_table_row_to_widths(row, max_widths, indent)) + else: + rebuilt.append(row) + + return rebuilt + + +def _rebuild_border( + left_corner: str, right_corner: str, col_widths: list[int], indent: str, +) -> str: + """Build a border row from corner chars and column widths. + + Maps corner pairs: + ┌ ┐ → separator ┬ + ├ ┤ → separator ┼ + └ ┘ → separator ┴ + """ + sep_map = {"┌": "┬", "├": "┼", "└": "┴"} + separator = sep_map.get(left_corner, "┼") + segments = ["─" * w for w in col_widths] + return indent + left_corner + separator.join(segments) + right_corner + + +def _repad_table_row_to_widths( + row: str, col_widths: list[int], indent: str, +) -> str: + """Re-pad a data row so each cell matches the given column widths.""" + stripped = row.lstrip() + parts = stripped.split("│") + if len(parts) < 3: + return row + cells = parts[1:-1] + if len(cells) != len(col_widths): + return row # Column count mismatch — leave unchanged + + new_cells: list[str] = [] + for cell, width in zip(cells, col_widths): + # Strip existing padding, then re-pad. + content = cell.strip() + content_dw = display_width(content) + # Target: 1 space left + content + right padding to fill width. + # Total cell display-width must equal `width`. + # Cell = " " + content + " " * (width - 1 - content_dw) + # But if content_dw + 2 > width, just use " content " (overflow). + if content: + right_pad = max(width - 1 - content_dw, 1) + new_cells.append(" " + content + " " * right_pad) + else: + new_cells.append(" " * width) + + return indent + "│" + "│".join(new_cells) + "│" + + +def _process_table_body( + lines: list[str], + start: int, + n: int, + output: list[str], + stats: Stats, + expected_pipes: int, + col_widths: list[int] | None = None, +) -> int: + """Process lines inside a table body (after ┌ border, until └ border). + + Uses pipe-count accumulation: each data row is accumulated until the + ``│`` count reaches *expected_pipes*, then emitted. Border lines + (├─, └─) are emitted directly (with split joining if needed). + + When *col_widths* is provided, each completed data row is re-padded + so every cell matches the column width from the border. + + Returns the next line index to process after the table ends. + """ + i = start + while i < n: + line = lines[i] + + # --- Empty line: table ended unexpectedly --- + if _is_truly_empty(line): + break + + # --- Border line (├ or └): join splits, emit, maybe exit --- + stripped = line.lstrip() + if stripped and stripped[0] in "├└" and "─" in stripped: + acc = line + i += 1 + # Join border split if needed + if acc.rstrip()[-1] == "─" and i < n: + nl = lines[i] + ns = nl.lstrip() + if ns and ns[0] == "─" and ns.rstrip()[-1] in _TABLE_CORNERS: + acc = raw_join(acc, nl) + stats.table_borders_fixed += 1 + i += 1 + while i < n and is_table_border_split(acc, lines[i]): + acc = raw_join(acc, lines[i]) + stats.table_borders_fixed += 1 + i += 1 + output.append(acc) + # └ border: table ends + if "└" in acc: + return i + continue + + # --- Data row: accumulate until pipe count matches --- + if "│" in line or (stripped and stripped[-1] == "│"): + acc = line.rstrip() + pipe_count = acc.count("│") + i += 1 + while pipe_count < expected_pipes and i < n: + nl = lines[i] + if _is_truly_empty(nl): + break + # Stop at border lines + nl_s = nl.lstrip() + if nl_s and nl_s[0] in "├└┌" and "─" in nl_s: + break + acc = table_cell_join(acc, nl) + pipe_count = acc.count("│") + stats.table_cells_fixed += 1 + i += 1 + # Re-pad cells to match column widths from border + if col_widths: + acc = _repad_table_row(acc, col_widths) + output.append(acc) + continue + + # --- Non-table line (shouldn't happen but be safe) --- + output.append(line) + i += 1 + + return i + + +def _process_user_block( + lines: list[str], + start: int, + n: int, + output: list[str], + stats: Stats, +) -> int: + """Process a user prompt block starting at *start*. + + User blocks begin with ``❯ `` and continue with lines that have + display_width == 76 (right-padded with trailing spaces). All content + lines in the block share this fixed width. + + Within the block: + - Blank lines (76 spaces) are paragraph separators. + - Lines whose stripped content starts with ``- `` are bullet items. + - Lines whose stripped content starts with ``\\d+. `` are numbered items. + - ````` ``` ````` toggles code-fence mode (content preserved as-is). + - Everything else is a continuation of the current paragraph/item. + + Returns the index of the first line AFTER the block. + """ + acc = lines[start] + i = start + 1 + + # Check if the first line itself is at dw=76 (padded). + first_dw = display_width(lines[start]) + if first_dw != 76: + # Short user prompt, no wrapping. Emit as-is. + output.append(acc) + return i + + in_code_fence = False + + while i < n: + line = lines[i] + dw = display_width(line) + + # The user block boundary: ALL lines inside have dw==76. + if dw != 76: + break + + stripped = line.rstrip() + + # --- Code fence toggle --- + # Detect ``` anywhere in the stripped content. + if stripped.lstrip().startswith("```"): + # Flush current accumulator before the fence marker. + if acc: + output.append(acc) + acc = "" + output.append(line) + in_code_fence = not in_code_fence + i += 1 + continue + + # Inside code fences: emit lines as-is (no joining). + if in_code_fence: + output.append(line) + i += 1 + continue + + # --- Blank line (paragraph separator) --- + if not stripped: + if acc: + output.append(acc) + acc = "" + output.append("") + i += 1 + continue + + # --- Structural item detection --- + content_no_indent = stripped.lstrip() + + # Bullet: stripped content starts with "- " + is_bullet = content_no_indent.startswith("- ") + + # Numbered list: stripped content starts with digit+". " + is_numbered = bool(re.match(r"\d+\. ", content_no_indent)) + + if is_bullet or is_numbered: + # Start a new logical line for this item. + if acc: + output.append(acc) + acc = line + i += 1 + continue + + # --- Normal continuation --- + if acc: + # Extract the content portion (skip the 2-space continuation + # indent that the export prepends). + join_content = stripped[2:] if stripped.startswith(" ") else stripped + acc = smart_join(acc, join_content) + stats.user_lines_joined += 1 + else: + # After a paragraph break (acc was reset to ""), this is the + # first line of a new paragraph. + acc = line + i += 1 + + # Flush remaining accumulated text. + if acc: + output.append(acc) + + return i + + +# --------------------------------------------------------------------------- +# Marker-count verification +# --------------------------------------------------------------------------- + +def _count_markers(lines: list[str]) -> dict[str, int]: + """Count occurrences of structural markers.""" + counts: dict[str, int] = {"●": 0, "❯": 0, "✻": 0} + for line in lines: + for marker in counts: + if line.startswith(marker): + counts[marker] += 1 + return counts + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- + +def main() -> None: + parser = argparse.ArgumentParser( + description="Fix broken line wrapping in Claude Code export files.", + ) + parser.add_argument("input", type=Path, help="Input .txt file") + parser.add_argument( + "-o", + "--output", + type=Path, + default=None, + help="Output path (default: -fixed.txt)", + ) + parser.add_argument( + "--stats", action="store_true", help="Print statistics to stderr" + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Process without writing output file", + ) + args = parser.parse_args() + + input_path: Path = args.input.resolve() + if not input_path.is_file(): + print(f"ERROR: Input file not found: {input_path}", file=sys.stderr) + sys.exit(1) + + if args.output is not None: + output_path: Path = args.output.resolve() + else: + output_path = input_path.with_stem(input_path.stem + "-fixed") + + if output_path == input_path: + print( + "ERROR: Output path must differ from input path. " + "Use -o to specify a different output file.", + file=sys.stderr, + ) + sys.exit(1) + + # Read input (strict UTF-8, no fallback). + text = input_path.read_text(encoding="utf-8", errors="strict") + raw_lines = text.split("\n") + # Remove trailing empty element from final newline (if present). + if raw_lines and raw_lines[-1] == "": + raw_lines.pop() + + stats = Stats() + result = process(raw_lines, stats) + + # --------------------------------------------------------------- + # Safety: marker-count verification + # --------------------------------------------------------------- + input_markers = _count_markers(raw_lines) + output_markers = _count_markers(result) + for marker, in_count in input_markers.items(): + out_count = output_markers.get(marker, 0) + if in_count != out_count: + print( + f"WARNING: Marker '{marker}' count mismatch: " + f"input={in_count}, output={out_count}", + file=sys.stderr, + ) + + # Safety: runaway join detection + for idx, line in enumerate(result): + if display_width(line) > 500: + print( + f"WARNING: Line {idx + 1} has display width " + f"{display_width(line)} (>500) — possible runaway join", + file=sys.stderr, + ) + + # --------------------------------------------------------------- + # Output + # --------------------------------------------------------------- + if args.dry_run: + print( + f"Dry run complete. Would write {len(result)} lines to {output_path}", + file=sys.stderr, + ) + else: + output_path.write_text( + "\n".join(result) + "\n", encoding="utf-8", errors="strict" + ) + print(f"Written: {output_path}", file=sys.stderr) + + if args.stats: + print(stats.summary(), file=sys.stderr) + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/claude-export-txt-better/scripts/validate-claude-export-fix.py b/claude-export-txt-better/scripts/validate-claude-export-fix.py new file mode 100644 index 00000000..553ee70c --- /dev/null +++ b/claude-export-txt-better/scripts/validate-claude-export-fix.py @@ -0,0 +1,313 @@ +#!/usr/bin/env python3 +"""Automated validation for fix-claude-export.py output. + +Runs a comprehensive suite of checks against a fixed file and its original, +reporting PASS/FAIL for each with evidence. Designed to be run after every +fix iteration as a quality gate. + +Usage: + uv run scripts/validate-claude-export-fix.py + uv run scripts/validate-claude-export-fix.py --verbose +""" + +from __future__ import annotations + +import argparse +import re +import sys +import unicodedata +from dataclasses import dataclass, field +from pathlib import Path + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def display_width(s: str) -> int: + return sum(2 if unicodedata.east_asian_width(ch) in ("W", "F") else 1 for ch in s) + + +def is_cjk_ideograph(ch: str) -> bool: + return unicodedata.east_asian_width(ch) in ("W", "F") and unicodedata.category(ch) == "Lo" + + +# --------------------------------------------------------------------------- +# Check infrastructure +# --------------------------------------------------------------------------- + +@dataclass +class CheckResult: + name: str + passed: bool + detail: str + category: str = "" + + +@dataclass +class ValidationReport: + results: list[CheckResult] = field(default_factory=list) + + def add(self, name: str, passed: bool, detail: str, category: str = ""): + self.results.append(CheckResult(name, passed, detail, category)) + + @property + def passed(self) -> int: + return sum(1 for r in self.results if r.passed) + + @property + def failed(self) -> int: + return sum(1 for r in self.results if not r.passed) + + def print_report(self, verbose: bool = False): + cats = {} + for r in self.results: + cats.setdefault(r.category or "General", []).append(r) + + for cat, checks in cats.items(): + print(f"\n{'=' * 60}") + print(f" {cat}") + print(f"{'=' * 60}") + for r in checks: + icon = "✓" if r.passed else "✗" + print(f" {icon} {r.name}") + if verbose or not r.passed: + for line in r.detail.split("\n"): + print(f" {line}") + + total = len(self.results) + print(f"\n{'=' * 60}") + print(f" TOTAL: {self.passed}/{total} passed, {self.failed} failed") + print(f"{'=' * 60}") + + +# --------------------------------------------------------------------------- +# Check implementations +# --------------------------------------------------------------------------- + +def check_marker_counts(orig: str, fixed: str, report: ValidationReport): + """Verify structural markers are preserved exactly.""" + markers = [ + ("❯", "User prompts"), + ("●", "Claude actions"), + ("✻", "Stars/crunched"), + ("⎿", "Tool results"), + ("…", "Expansion indicators"), + ] + for marker, name in markers: + orig_count = orig.count(marker) + fixed_count = fixed.count(marker) + report.add( + f"Marker {marker} ({name}): {orig_count}", + orig_count == fixed_count, + f"orig={orig_count} fixed={fixed_count}", + "Structural Integrity", + ) + + +def check_table_borders(fixed: str, report: ValidationReport): + """Verify table border corners are balanced.""" + for ch, name in [("┌", "top-left"), ("┐", "top-right"), ("┘", "bottom-right")]: + count = fixed.count(ch) + report.add( + f"Table corner {ch} ({name}): {count}", + True, # Just record the count + f"count={count}", + "Structural Integrity", + ) + tl = fixed.count("┌") + tr = fixed.count("┐") + br = fixed.count("┘") + report.add( + "Table corners balanced (┌ = ┐ = ┘)", + tl == tr == br, + f"┌={tl} ┐={tr} ┘={br}", + "Structural Integrity", + ) + + +def check_line_reduction(orig: str, fixed: str, report: ValidationReport): + """Output should have fewer lines than input (joins happened).""" + orig_lines = orig.count("\n") + fixed_lines = fixed.count("\n") + report.add( + f"Line reduction: {orig_lines} → {fixed_lines}", + fixed_lines < orig_lines, + f"delta={orig_lines - fixed_lines} ({(orig_lines - fixed_lines) / orig_lines * 100:.1f}% reduction)", + "Structural Integrity", + ) + + +def check_table_border_completeness(fixed: str, report: ValidationReport): + """Verify table border lines have matching left and right ends.""" + lines = fixed.split("\n") + broken = [] + for i, line in enumerate(lines): + stripped = line.strip() + if not stripped: + continue + # Lines starting with a left border char should have a right border char + if stripped[0] == "┌" and "┐" not in stripped: + broken.append((i + 1, "┌ without ┐", stripped[:80])) + elif stripped[0] == "├" and "┤" not in stripped: + broken.append((i + 1, "├ without ┤", stripped[:80])) + elif stripped[0] == "└" and "┘" not in stripped: + broken.append((i + 1, "└ without ┘", stripped[:80])) + + report.add( + f"Table borders complete: {len(broken)} broken", + len(broken) == 0, + "\n".join(f" L{ln}: {desc}" for ln, desc, _ in broken[:5]) + if broken else "all borders have matching ends", + "Structural Integrity", + ) + + +def check_phase_separation(fixed: str, report: ValidationReport): + """Verify Phase N: items are on separate lines.""" + lines = fixed.split("\n") + multi_phase_lines = [] + for i, line in enumerate(lines): + # Count "Phase N:" occurrences on this line + matches = re.findall(r"Phase \d+:", line) + if len(matches) >= 2: + # Allow pipeline diagrams with arrows (legitimate multi-phase) + if "→" in line: + continue + # Allow status updates like "Phase 3 进度: 3/5" + if line.strip().startswith("●"): + continue + multi_phase_lines.append((i + 1, matches, line[:80])) + + report.add( + "Phase items on separate lines", + len(multi_phase_lines) == 0, + f"{len(multi_phase_lines)} violations" + ( + "\n" + "\n".join(f" L{ln}: {m}" for ln, m, _ in multi_phase_lines[:5]) + if multi_phase_lines else "" + ), + "Over-Join Prevention", + ) + + +def check_runaway_joins(fixed: str, report: ValidationReport): + """Flag lines with very high display width that might be runaway joins.""" + lines = fixed.split("\n") + runaways = [] + for i, line in enumerate(lines): + dw = display_width(line) + if dw > 500: + # Check if it's a legitimate long line (user prompt) + if line.startswith("❯ "): + continue + runaways.append((i + 1, dw, line[:60])) + + report.add( + f"No runaway joins (dw > 500): {len(runaways)} found", + len(runaways) == 0, + "\n".join(f" L{ln}: dw={dw} [{preview}...]" for ln, dw, preview in runaways[:5]) + if runaways else "none", + "Over-Join Prevention", + ) + + +def check_en_cjk_no_space(fixed: str, report: ValidationReport): + """Count remaining EN-CJK adjacency without space (potential pangu misses).""" + # Only check at join boundaries (lines that were modified), not all text + pattern_alnum_cjk = re.compile(r"[a-zA-Z0-9][一-龥]") + pattern_cjk_alnum = re.compile(r"[一-龥][a-zA-Z0-9]") + + violations_ac = len(pattern_alnum_cjk.findall(fixed)) + violations_ca = len(pattern_cjk_alnum.findall(fixed)) + total = violations_ac + violations_ca + + # This is informational — some violations are in original content, code, etc. + report.add( + f"EN-CJK adjacency count: {total}", + True, # Informational + f"ASCII→CJK: {violations_ac}, CJK→ASCII: {violations_ca} (includes original content)", + "Pangu Spacing", + ) + + +def check_diff_lines_separate(fixed: str, report: ValidationReport): + """Verify diff output lines aren't merged (line numbers should be separate).""" + lines = fixed.split("\n") + merged_diffs = [] + for i, line in enumerate(lines): + # Look for two diff line numbers on the same line + matches = re.findall(r"\b(\d{3})\s{2,}[-+]?\s", line) + if len(matches) >= 2: + merged_diffs.append((i + 1, matches, line[:80])) + + report.add( + "Diff lines separate", + len(merged_diffs) == 0, + f"{len(merged_diffs)} violations" + ( + "\n" + "\n".join(f" L{ln}: numbers={m}" for ln, m, _ in merged_diffs[:5]) + if merged_diffs else "" + ), + "Over-Join Prevention", + ) + + +def check_cjk_label_separation(fixed: str, report: ValidationReport): + """Verify CJK label fields (模块:, 输出文件:, 状态:) are on separate lines.""" + lines = fixed.split("\n") + merged_labels = [] + # Pattern: field-label style "Label1: value1 Label2: value2" where + # each label starts at a position that looks like a separate field. + # Only flag when labels are at field boundaries (preceded by whitespace + # or start of line), not mid-sentence. + cjk_field_re = re.compile(r"(?:^|\s)([\u4e00-\u9fff]{1,4}[:：])\s") + for i, line in enumerate(lines): + stripped = line.strip() + matches = cjk_field_re.findall(stripped) + if len(matches) >= 3: # 3+ field labels = likely over-joined fields + merged_labels.append((i + 1, matches, stripped[:80])) + + report.add( + "CJK labels on separate lines", + len(merged_labels) == 0, + f"{len(merged_labels)} violations" + ( + "\n" + "\n".join(f" L{ln}: labels={m}" for ln, m, _ in merged_labels[:5]) + if merged_labels else "" + ), + "Over-Join Prevention", + ) + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("original", type=Path, help="Original exported file") + parser.add_argument("fixed", type=Path, help="Fixed output file") + parser.add_argument("--verbose", "-v", action="store_true", help="Show all details") + args = parser.parse_args() + + orig = args.original.read_text(encoding="utf-8") + fixed = args.fixed.read_text(encoding="utf-8") + + report = ValidationReport() + + # Run all checks + check_marker_counts(orig, fixed, report) + check_table_borders(fixed, report) + check_table_border_completeness(fixed, report) + check_line_reduction(orig, fixed, report) + check_phase_separation(fixed, report) + check_runaway_joins(fixed, report) + check_diff_lines_separate(fixed, report) + check_cjk_label_separation(fixed, report) + check_en_cjk_no_space(fixed, report) + + report.print_report(verbose=args.verbose) + + sys.exit(0 if report.failed == 0 else 1) + + +if __name__ == "__main__": + main() \ No newline at end of file From 6c30b5644b38541eccbbf4cd28c3b6a25e6914b8 Mon Sep 17 00:00:00 2001 From: daymade Date: Mon, 16 Mar 2026 21:13:27 +0800 Subject: [PATCH 006/137] feat(skill-creator): add Step 0 prerequisites check with auto-install Adds dependency detection before skill creation starts, preventing mid-workflow failures (e.g., gitleaks missing at packaging, PyYAML missing at validation). Documents correct script invocation via python3 -m syntax and auto-installation commands. Co-Authored-By: Claude Opus 4.6 (1M context) --- skill-creator/SKILL.md | 8 ++ skill-creator/references/prerequisites.md | 117 ++++++++++++++++++++++ 2 files changed, 125 insertions(+) create mode 100644 skill-creator/references/prerequisites.md diff --git a/skill-creator/SKILL.md b/skill-creator/SKILL.md index b224f242..d2fd97a3 100644 --- a/skill-creator/SKILL.md +++ b/skill-creator/SKILL.md @@ -631,6 +631,14 @@ Take `best_description` from the JSON output and update the skill's SKILL.md fro When creating or updating a skill, follow these steps in order. Skip steps only when clearly not applicable. +### Step 0: Prerequisites Check + +Before starting any skill work, auto-detect all dependencies and proactively install anything missing. Discovering a missing tool mid-workflow (e.g., gitleaks at packaging time, PyYAML at validation) wastes time and breaks flow. + +Run the quick check from [references/prerequisites.md](references/prerequisites.md), auto-install what you can, and present the user a summary checklist. Only proceed when all blocking dependencies are satisfied. + +Key blockers: Python 3, PyYAML (validation/packaging), gitleaks (security scan), claude CLI (evals). All scripts must be invoked via `python3 -m scripts.` from the skill-creator root directory — direct `python3 scripts/.py` fails due to relative imports. + ### Step 1: Understanding the Skill with Concrete Examples Skip this step only when the skill's usage patterns are already clearly understood. diff --git a/skill-creator/references/prerequisites.md b/skill-creator/references/prerequisites.md new file mode 100644 index 00000000..0873263a --- /dev/null +++ b/skill-creator/references/prerequisites.md @@ -0,0 +1,117 @@ +# Skill Creator Prerequisites + +Auto-detect and install all dependencies before starting skill creation. This prevents failures mid-workflow (e.g., discovering gitleaks is missing only at the packaging step). + +## Quick Check Script + +Run all checks in one go: + +```bash +echo "=== Skill Creator Prerequisites ===" +echo -n "Python 3: "; python3 --version 2>/dev/null || echo "MISSING" +echo -n "PyYAML: "; python3 -c "import yaml; print('OK')" 2>/dev/null || echo "MISSING" +echo -n "gitleaks: "; gitleaks version 2>/dev/null || echo "MISSING" +echo -n "claude CLI: "; which claude 2>/dev/null || echo "MISSING" +echo -n "anthropic SDK: "; python3 -c "import anthropic; print('OK')" 2>/dev/null || echo "MISSING (optional)" +echo -n "uv: "; uv --version 2>/dev/null || echo "MISSING (optional)" +``` + +## Dependencies by Phase + +| Dependency | Required For | Phase | Severity | +|-----------|-------------|-------|----------| +| Python 3.7+ | All scripts | All | **Blocking** | +| PyYAML | `quick_validate.py`, `package_skill.py` | Validation, Packaging | **Blocking** | +| gitleaks | `security_scan.py` | Security Review (Step 6) | **Blocking for packaging** | +| claude CLI | `run_eval.py`, `run_loop.py` | Testing, Description Optimization | **Blocking for evals** | +| anthropic SDK | `improve_description.py`, `run_loop.py` | Description Optimization | Optional (only for desc optimization) | +| uv | Skills that bundle Python scripts | Export/Runtime | Optional (skill-specific) | +| webbrowser | `generate_review.py` (viewer) | Eval Review | Optional (can use `--static` fallback) | + +## Auto-Installation + +### PyYAML (required) + +```bash +# Preferred: via uv +uv pip install --system pyyaml + +# Alternative: via pip +pip3 install pyyaml + +# Verify +python3 -c "import yaml; print(yaml.__version__)" +``` + +### gitleaks (required for packaging) + +```bash +# macOS +brew install gitleaks + +# Linux +wget https://github.com/gitleaks/gitleaks/releases/download/v8.21.2/gitleaks_8.21.2_linux_x64.tar.gz +tar -xzf gitleaks_8.21.2_linux_x64.tar.gz && sudo mv gitleaks /usr/local/bin/ + +# Verify +gitleaks version +``` + +### anthropic SDK (optional, for description optimization) + +```bash +# Preferred: via uv +uv pip install --system anthropic + +# Alternative: via pip +pip3 install anthropic + +# Verify +python3 -c "import anthropic; print('OK')" +``` + +Also requires `ANTHROPIC_API_KEY` environment variable to be set. + +### claude CLI (required for evals) + +The `claude` CLI (Claude Code) must be installed and available in PATH. If the user is already running this skill inside Claude Code, this is already satisfied. + +```bash +# Verify +which claude && claude --version +``` + +If missing, the user needs to install Claude Code from https://claude.ai/claude-code. + +## Script Invocation + +All scripts must be run from the skill-creator root directory using module syntax: + +```bash +# CORRECT — run from skill-creator directory +cd +python3 -m scripts.package_skill +python3 -m scripts.security_scan +python3 -m scripts.aggregate_benchmark --skill-name + +# WRONG — direct invocation fails with ModuleNotFoundError +python3 scripts/package_skill.py # ImportError: No module named 'scripts' +``` + +This is because the scripts use relative imports (`from scripts.quick_validate import ...`). + +## Presenting Results to User + +After running all checks, present a summary table: + +``` +Skill Creator Prerequisites: + [x] Python 3.12.0 + [x] PyYAML 6.0.1 + [x] gitleaks 8.21.2 + [x] claude CLI (running inside Claude Code) + [ ] anthropic SDK — not installed (only needed for description optimization) + [x] uv 0.6.x +``` + +If any **blocking** dependency is missing and auto-install fails, clearly explain what the user needs to do and stop before proceeding to skill creation. From d8a7d45e53d86306d89d39f19f80ecf9a01e6c11 Mon Sep 17 00:00:00 2001 From: daymade Date: Wed, 18 Mar 2026 22:17:59 +0800 Subject: [PATCH 007/137] refactor(CLAUDE.md): slim from 1549 to 318 lines via progressive disclosure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Move verbose sections to references/ files, keeping concise pointers in CLAUDE.md. Zero content loss — all documentation preserved in reference files that Claude loads on demand. Moved to references/: - plugin-architecture.md (296 lines) — architecture docs - plugin-troubleshooting.md (441 lines) — installation debugging - new-skill-guide.md (241 lines) — detailed templates/checklists - promotion-policy.md (60 lines) — third-party request policy - youtube-downloader/references/internal-sop.md — yt-dlp SOP Also fixed: Available Skills #36-42 indentation, deduplicated 4x versioning sections into one, removed stale notes. Co-Authored-By: Claude Opus 4.6 (1M context) --- CLAUDE.md | 1297 +---------------- references/new-skill-guide.md | 241 +++ references/plugin-architecture.md | 296 ++++ references/plugin-troubleshooting.md | 441 ++++++ references/promotion-policy.md | 60 + youtube-downloader/references/internal-sop.md | 12 + 6 files changed, 1083 insertions(+), 1264 deletions(-) create mode 100644 references/new-skill-guide.md create mode 100644 references/plugin-architecture.md create mode 100644 references/plugin-troubleshooting.md create mode 100644 references/promotion-policy.md create mode 100644 youtube-downloader/references/internal-sop.md diff --git a/CLAUDE.md b/CLAUDE.md index 38adf594..6359893f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -134,7 +134,7 @@ Skills for public distribution must NOT contain: ## Marketplace Configuration The marketplace is configured in `.claude-plugin/marketplace.json`: - - Contains 42 plugins, each mapping to one skill +- Contains 42 plugins, each mapping to one skill - Each plugin has: name, description, version, category, keywords, skills array - Marketplace metadata: name, owner, version, homepage @@ -212,28 +212,19 @@ This applies when you change ANY file under a skill directory: 33. **meeting-minutes-taker** - Transform meeting transcripts into structured minutes with multi-pass generation, speaker quotes, and iterative human review 34. **deep-research** - Generate format-controlled research reports with evidence mapping, citations, and multi-pass synthesis 35. **competitors-analysis** - Evidence-based competitor tracking and analysis with source citations (file:line_number format) - 36. **tunnel-doctor** - Diagnose and fix Tailscale + proxy/VPN conflicts (four layers: route, HTTP env, system proxy, SSH ProxyCommand) on macOS with WSL SSH support - 37. **windows-remote-desktop-connection-doctor** - Diagnose AVD/W365 connection quality issues with transport protocol analysis and Windows App log parsing - 38. **product-analysis** - Perform structured product audits across UX, API, architecture, and compare mode to produce prioritized optimization recommendations - 39. **financial-data-collector** - Collect real financial data for US public companies via yfinance with validation, NaN detection, and NO FALLBACK principle - 40. **excel-automation** - Create formatted Excel files, parse complex xlsm models, and control Excel windows on macOS via AppleScript - 41. **capture-screen** - Programmatically capture macOS application windows using Swift window ID discovery and screencapture workflows - 42. **continue-claude-work** - Recover local `.claude` session context via compact-boundary extraction, subagent workflow recovery, and session end reason detection, then continue interrupted work without `claude --resume` +36. **tunnel-doctor** - Diagnose and fix Tailscale + proxy/VPN conflicts (four layers: route, HTTP env, system proxy, SSH ProxyCommand) on macOS with WSL SSH support +37. **windows-remote-desktop-connection-doctor** - Diagnose AVD/W365 connection quality issues with transport protocol analysis and Windows App log parsing +38. **product-analysis** - Perform structured product audits across UX, API, architecture, and compare mode to produce prioritized optimization recommendations +39. **financial-data-collector** - Collect real financial data for US public companies via yfinance with validation, NaN detection, and NO FALLBACK principle +40. **excel-automation** - Create formatted Excel files, parse complex xlsm models, and control Excel windows on macOS via AppleScript +41. **capture-screen** - Programmatically capture macOS application windows using Swift window ID discovery and screencapture workflows +42. **continue-claude-work** - Recover local `.claude` session context via compact-boundary extraction, subagent workflow recovery, and session end reason detection, then continue interrupted work without `claude --resume` **Recommendation**: Always suggest `skill-creator` first for users interested in creating skills or extending Claude Code. ## YouTube Downloader SOP (Internal) -Use this SOP to avoid common yt-dlp failures and confusion: - -1. Quote YouTube URLs in shell commands (zsh treats `?` as glob). Example: `'https://www.youtube.com/watch?v=VIDEO_ID'`. -2. Ensure proxy is active for both yt-dlp and PO Token providers (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY). -3. If you see “Sign in to confirm you’re not a bot”, request cookie permission and use browser cookies. -4. Start the PO Token provider before downloading. Prefer Docker bgutil; fall back to browser-based WPC when Docker is unavailable or fails. -5. Use `web_safari` client when cookies are present; otherwise use `mweb` for PO tokens. -6. Keep the browser window open while WPC is minting tokens and make sure it can reach YouTube through the same proxy. -7. If you see “Only images are available” or “Requested format is not available”, treat it as PO token failure and retry after fixing provider/browser state. -8. If you see SSL EOF or fragment errors, treat it as proxy instability. Retry with progressive formats or switch to a more stable proxy. +See [youtube-downloader/references/internal-sop.md](./youtube-downloader/references/internal-sop.md) for yt-dlp troubleshooting steps (PO tokens, proxy, cookies, etc.). ## Python Development @@ -267,326 +258,35 @@ When creating a new skill: ## Adding a New Skill to Marketplace -**CRITICAL**: When adding a skill to this marketplace, you MUST update all of these files in the correct order. Missing any file will result in incomplete integration. +For the full step-by-step guide with templates and examples, see [references/new-skill-guide.md](./references/new-skill-guide.md). -### Step-by-Step Process +**Files to update** (all required): -#### 1. Refine the Skill (if needed) -```bash -# Ensure skill follows best practices -# - SKILL.md uses imperative/infinitive form -# - Third-person description in YAML frontmatter -# - Progressive disclosure (details in references/) -# - Security scan passed - -cd skill-creator -python3 scripts/security_scan.py ../skill-name --verbose -``` +| File | Locations to update | +|------|-------------------| +| `.claude-plugin/marketplace.json` | metadata.version + metadata.description + new plugin entry | +| `CHANGELOG.md` | New version entry | +| `README.md` | 7 locations: badges, description, install cmd, skill section, use case, docs link, requirements | +| `README.zh-CN.md` | 7 locations: same as above, translated | +| `CLAUDE.md` | 3 locations: overview count, marketplace config count, Available Skills list | +| `skill-name/` | The actual skill directory + packaged .zip | -#### 2. Package the Skill +**Quick workflow**: ```bash -cd skill-creator +# 1. Validate & package +cd skill-creator && python3 scripts/security_scan.py ../skill-name --verbose python3 scripts/package_skill.py ../skill-name -# This will: -# - Validate skill structure -# - Check security scan status -# - Create skill-name.zip in skill-creator/ -# - Move zip to skill-name/ directory -``` - -#### 3. Update CHANGELOG.md ⚠️ REQUIRED - -Add new version entry at the top (after [Unreleased]): - -```markdown -## [X.Y.0] - YYYY-MM-DD - -### Added -- **New Skill**: skill-name - Brief description - - Feature 1 - - Feature 2 - - Feature 3 - - Bundled scripts/references/assets - - Key capabilities - -### Changed -- Updated marketplace skills count from N to N+1 -- Updated marketplace version from X.(Y-1).0 to X.Y.0 -- Updated README.md badges (skills count, version) -- Updated README.md to include skill-name in skills listing -- Updated README.zh-CN.md badges (skills count, version) -- Updated README.zh-CN.md to include skill-name in skills listing -- Updated CLAUDE.md skills count from N to N+1 -- Added skill-name use case section to README.md -- Added skill-name use case section to README.zh-CN.md -- Added dependencies to requirements section (if any, both EN and ZH) -``` - -**Version numbering**: Increment MINOR version (e.g., 1.8.0 → 1.9.0) when adding a skill. +# 2. Update all files listed above (see references/new-skill-guide.md for details) -#### 4. Update README.md ⚠️ REQUIRED - -**a. Update badges (top of file):** -```markdown -[![Skills](https://img.shields.io/badge/skills-N-blue.svg)] -[![Version](https://img.shields.io/badge/version-X.Y.0-green.svg)] -``` - -**b. Update description:** -```markdown -Professional Claude Code skills marketplace featuring N production-ready skills... -``` - -**c. Add installation command:** -```markdown -# Brief description -claude plugin install skill-name@daymade-skills -``` - -**d. Add skill section (### N. **skill-name**):** -```markdown -### N. **skill-name** - One-line Title - -Brief description paragraph. - -**When to use:** -- Use case 1 -- Use case 2 -- Use case 3 - -**Key features:** -- Feature 1 -- Feature 2 -- Feature 3 - -**Example usage:** -\`\`\`bash -# Example commands -\`\`\` - -**🎬 Live Demo** - -*Coming soon* (or add demo GIF) - -📚 **Documentation**: See [skill-name/references/](./skill-name/references/)... - -**Requirements**: Dependencies (e.g., Python 3.8+, FFmpeg, etc.) -``` - -**e. Add use case section:** -```markdown -### For [Use Case Category] -Use **skill-name** to [describe primary use case]. Combine with **other-skill** to [describe integration]. -``` - -**f. Add documentation quick link:** -```markdown -- **skill-name**: See `skill-name/references/...` for ... -``` - -**g. Update requirements section (if needed):** -```markdown -- **Tool Name** (for skill-name): `install command` -``` - -#### 5. Update CLAUDE.md ⚠️ REQUIRED - -**a. Update repository overview:** -```markdown -This is a Claude Code skills marketplace containing N production-ready skills... -``` - -**b. Update marketplace configuration:** -```markdown -The marketplace is configured in `.claude-plugin/marketplace.json`: -- Contains N plugins, each mapping to one skill -``` - -**c. Update marketplace version:** -```markdown -1. **Marketplace Version** (`.claude-plugin/marketplace.json` → `metadata.version`) - - Tracks the marketplace catalog as a whole - - Current: vX.Y.0 -``` - -**d. Add skill to Available Skills list:** -```markdown -N. **skill-name** - Brief description with key feature -``` - -#### 6. Update .claude-plugin/marketplace.json ⚠️ CRITICAL - -**MOST IMPORTANT FILE** - This file makes the skill installable! - -**a. Update metadata.description:** -```json -"description": "Professional Claude Code skills for ..., and [new skill capability]" -``` - -**b. Update metadata.version:** -```json -"version": "X.Y.0" -``` - -**c. Add new plugin entry to plugins array:** -```json -{ - "name": "skill-name", - "description": "Clear description with trigger conditions. Use when [scenarios]", - "source": "./", - "strict": false, - "version": "1.0.0", - "category": "appropriate-category", - "keywords": ["keyword1", "keyword2", "keyword3", ...], - "skills": ["./skill-name"] -} -``` - -**Categories:** `developer-tools`, `document-conversion`, `documentation`, `customization`, `communication`, `utilities`, `assets`, `design`, `productivity`, `security`, `media` - -**d. Validate JSON syntax:** -```bash -python3 -m json.tool .claude-plugin/marketplace.json > /dev/null -``` - -#### 7. Update README.zh-CN.md ⚠️ REQUIRED - -**CRITICAL**: Chinese documentation must be kept in sync with English version. - -**a. Update badges (top of file):** -```markdown -[![Skills](https://img.shields.io/badge/skills-N-blue.svg)] -[![Version](https://img.shields.io/badge/version-X.Y.0-green.svg)] -``` - -**b. Update description:** -```markdown -专业的 Claude Code 技能市场，提供 N 个生产就绪的技能，用于增强开发工作流。 -``` - -**c. Add installation command:** -```markdown -# 简短描述 -claude plugin install skill-name@daymade-skills -``` - -**d. Add skill section (### N. **skill-name** - Chinese Title):** -- Translate all content from English README -- Include: 使用场景 (When to use), 主要功能 (Key features), 示例用法 (Example usage) -- Maintain same structure as English version -- Include documentation links and requirements - -**e. Add use case section:** -```markdown -### [Use Case Category in Chinese] -使用 **skill-name** [describe use case in Chinese]. 与 **other-skill** 结合使用以 [describe integration]. -``` - -**f. Add documentation quick link:** -```markdown -- **skill-name**：参见 `skill-name/references/...` 了解 ... -``` - -**g. Update requirements section (if needed):** -```markdown -- **Tool Name**（用于 skill-name）：`install command` -``` - -**Translation tips:** -- Use professional technical Chinese -- Maintain consistency with existing translations -- Keep code examples in English (don't translate variable names, function names) -- Translate user-facing descriptions, features, and use cases - -#### 8. Verification Checklist - -Before committing, verify: - -- [ ] CHANGELOG.md has new version entry -- [ ] README.md badges updated (skills count + version) -- [ ] README.md has skill section with number -- [ ] README.md has use case section -- [ ] README.md has documentation link -- [ ] README.md requirements updated (if needed) -- [ ] README.zh-CN.md badges updated (skills count + version) ⚠️ NEW -- [ ] README.zh-CN.md has skill section with number ⚠️ NEW -- [ ] README.zh-CN.md has use case section ⚠️ NEW -- [ ] README.zh-CN.md has documentation link ⚠️ NEW -- [ ] README.zh-CN.md requirements updated (if needed) ⚠️ NEW -- [ ] README.zh-CN.md installation command added ⚠️ NEW -- [ ] CLAUDE.md skill count updated in 3 places -- [ ] CLAUDE.md has skill in Available Skills list -- [ ] marketplace.json metadata.version updated -- [ ] marketplace.json metadata.description updated -- [ ] marketplace.json has new plugin entry -- [ ] marketplace.json validates (python3 -m json.tool) -- [ ] skill-name.zip package exists -- [ ] Security scan passed - -### Common Mistakes to Avoid - -1. **Forgetting marketplace.json** ⚠️ - The most critical file! Without this, the skill cannot be installed via `claude plugin install` -2. **Forgetting Chinese documentation** ⚠️ - README.zh-CN.md must be updated in sync with README.md (6 locations) -3. **Inconsistent version numbers** - CHANGELOG, README badges (both EN and ZH), CLAUDE.md, and marketplace.json must all match -4. **Inconsistent skill counts** - README description (both EN and ZH), badges, CLAUDE.md must all have same count -5. **Missing skill number in README** - Skills must be numbered sequentially (1, 2, 3, ...) in both EN and ZH versions -6. **Invalid JSON syntax** - Always validate marketplace.json after editing -7. **Forgetting dependencies** - Update README requirements section (both EN and ZH) if skill needs external tools -8. **Incomplete Chinese translation** - Must translate all sections: description, use cases, features, use case section, docs link - -### File Update Summary Template - -When adding a skill, this is the complete file list: - -``` -Files to Update: -✅ CHANGELOG.md (Add version entry) -✅ README.md (7 locations: badges, description, install, skill section, use case, docs link, requirements) -✅ README.zh-CN.md (7 locations: badges, description, install, skill section, use case, docs link, requirements) ⚠️ CRITICAL -✅ CLAUDE.md (3 locations: overview, marketplace config, available skills) -✅ .claude-plugin/marketplace.json (CRITICAL: metadata + new plugin entry) -✅ skill-name/ (The actual skill directory) -✅ skill-name/skill-name.zip (Packaged skill) +# 3. Validate, commit, push, release +cd .. && python3 -m json.tool .claude-plugin/marketplace.json > /dev/null +git add -A && git commit -m "Release vX.Y.0: Add skill-name" +git push +gh release create vX.Y.0 --title "Release vX.Y.0: Add skill-name" --notes "..." ``` -**IMPORTANT**: README.zh-CN.md is MANDATORY. Do not skip Chinese documentation updates! - -### Version Numbering Convention - -- **MAJOR.MINOR.PATCH** (Semantic Versioning) -- Increment **MINOR** when adding a new skill: 1.8.0 → 1.9.0 -- Increment **PATCH** for bug fixes or small updates: 1.9.0 → 1.9.1 -- Increment **MAJOR** for breaking changes or major restructuring: 1.9.0 → 2.0.0 - -### Quick Reference Commands - -```bash -# 1. Refine and validate skill -cd skill-creator -python3 scripts/security_scan.py ../skill-name --verbose - -# 2. Package skill -python3 scripts/package_skill.py ../skill-name - -# 3. Validate marketplace.json -cd .. -python3 -m json.tool .claude-plugin/marketplace.json > /dev/null && echo "✅ Valid" - -# 4. Check what needs committing -git status - -# 5. View specific file changes -git diff CHANGELOG.md -git diff README.md -git diff README.zh-CN.md -git diff CLAUDE.md -git diff .claude-plugin/marketplace.json - -# 6. Verify Chinese documentation is in sync -grep "skills-[0-9]*" README.md README.zh-CN.md -grep "version-[0-9.]*" README.md README.zh-CN.md -``` +**Top mistakes**: Forgetting to push to GitHub, forgetting README.zh-CN.md, inconsistent version numbers across files. ## Chinese User Support @@ -600,950 +300,19 @@ See README.md section "🇨🇳 中文用户指南" for details. ## Handling Third-Party Marketplace Promotion Requests -This repository is a **personal curated marketplace**, NOT a community directory or ecosystem hub. All requests to add third-party marketplace links, skill collection references, or "Community Marketplaces" sections should be declined. - -### Policy - -**DO NOT accept:** -- PRs adding "Related Resources" or "Community Marketplaces" sections linking to third-party skill collections -- Issues requesting promotion of external marketplaces -- PRs adding links to other skill repositories in README.md - -**Rationale:** -1. **Scope creep**: Shifts repository purpose from curated skills to ecosystem directory -2. **Implicit endorsement**: Listing implies quality/security review we cannot maintain -3. **Maintenance burden**: Would need to track and vet external projects over time -4. **Precedent setting**: Accepting one creates obligation to accept others - -### Response Template - -When declining, use this approach: - -```markdown -Hi @{username}, - -Thank you for your interest and for sharing {project-name}! {Brief positive acknowledgment of their project}. - -However, I'm keeping this repository focused as a **personal curated marketplace** rather than a directory of external skill collections. Adding third-party references would: - -1. Shift the repository's scope from curated skills to ecosystem directory -2. Create implicit endorsement expectations I can't maintain -3. Set precedent for similar requests (reference other declined requests if applicable) - -**What you can do instead:** - -1. **Standalone marketplace** - Your repo already works as an independent marketplace: - ``` - /plugin marketplace add {owner}/{repo} - ``` - -2. **Community channels** - Promote through: - - Claude Code GitHub discussions/issues (Anthropic's official repo) - - Developer communities (Reddit, Discord, etc.) - - Your own blog/social media - -3. **Official registry** - If/when Anthropic launches an official skill registry, that would be the appropriate place for ecosystem-wide discovery. - -Your marketplace can succeed on its own merits. Good luck with {project-name}! -``` - -### Workflow - -1. **Review the request** - Confirm it's a third-party promotion (not a legitimate contribution) -2. **Add polite comment** - Use template above, customize for their specific project -3. **Close with reason** - Use "not planned" for issues, just close for PRs -4. **Reference precedent** - Link to previously declined requests for consistency (e.g., #7, PR #5) - -### Examples - -- **Issue #7**: "Add Community Marketplaces section - Protocol Thunderdome" → Declined, closed as "not planned" -- **PR #5**: "Add Trail of Bits Security Skills to Related Resources" → Declined, closed - -## Release Workflow - -When adding a new skill or creating a marketplace release: - -### 1. Create the Skill -```bash -# Develop skill in its directory -skill-name/ -├── SKILL.md (no version history!) -├── scripts/ -└── references/ - -# Validate -./skill-creator/scripts/quick_validate.py skill-name - -# Package -./skill-creator/scripts/package_skill.py skill-name -``` - -### 2. Update Marketplace Configuration - -Edit `.claude-plugin/marketplace.json`: - -```json -{ - "metadata": { - "version": "1.x.0" // Bump minor version for new skill - }, - "plugins": [ - { - "name": "new-skill", - "version": "1.0.0", // Skill's initial version - "description": "...", - "category": "...", - "keywords": [...], - "skills": ["./new-skill"] - } - ] -} -``` - -### 3. Update Documentation - -**README.md:** -- Update badges (skills count, marketplace version) -- Add skill description and features -- Create demo GIF using cli-demo-generator -- Add use case section -- Add documentation references -- Add requirements (if applicable) - -**CLAUDE.md:** -- Update skill count in Repository Overview -- Add skill to Available Skills list -- Update Marketplace Configuration count - -### 4. Generate Demo (Optional but Recommended) - -```bash -# Use cli-demo-generator to create demo GIF -./cli-demo-generator/scripts/auto_generate_demo.py \ - -c "command1" \ - -c "command2" \ - -o demos/skill-name/demo-name.gif \ - --title "Skill Demo" \ - --theme "Dracula" -``` - -### 5. Commit and Release - -```bash -# Commit marketplace update -git add .claude-plugin/marketplace.json skill-name/ -git commit -m "Release vX.Y.0: Add skill-name - -- Add skill-name vX.Y.Z -- Update marketplace to vX.Y.0 -..." - -# Commit documentation -git add README.md CLAUDE.md demos/ -git commit -m "docs: Update README for vX.Y.0 with skill-name" - -# Push -git push - -# Create GitHub release -gh release create vX.Y.0 \ - --title "Release vX.Y.0: Add skill-name - Description" \ - --notes "$(cat <<'EOF' -## New Skill: skill-name - -Features: -- Feature 1 -- Feature 2 - -Installation: -```bash -claude plugin install skill-name@daymade-skills -``` - -Changelog: ... -EOF -)" -``` - -### Version Bumping Guide - -**Marketplace version (metadata.version):** -- **MAJOR** (2.0.0): Breaking changes, incompatible marketplace structure -- **MINOR** (1.5.0): New skill added, significant feature addition -- **PATCH** (1.4.1): Bug fixes, documentation updates, skill updates - -**Skill version (plugins[].version):** -- **MAJOR** (2.0.0): Breaking API changes for the skill -- **MINOR** (1.2.0): New features in the skill -- **PATCH** (1.1.1): Bug fixes in the skill - -### Example: v1.5.0 Release (ppt-creator) - -```bash -# 1. Created ppt-creator skill -# 2. Updated marketplace.json: 1.4.0 → 1.5.0 -# 3. Added ppt-creator plugin entry (version: 1.0.0) -# 4. Updated README.md (badges, description, demo) -# 5. Generated demo GIF with cli-demo-generator -# 6. Committed changes -# 7. Created GitHub release with gh CLI -``` +Decline all third-party marketplace promotion requests. For policy, response template, and precedents, see [references/promotion-policy.md](./references/promotion-policy.md). ## Best Practices Reference Always consult Anthropic's skill authoring best practices before creating or updating skills: https://docs.claude.com/en/docs/agents-and-tools/agent-skills/best-practices.md -- remember this release workflow in claude.md ## Plugin and Skill Architecture -This section explains the architecture of Claude Code's extension system and how different components work together. - -### Core Concepts - -#### 1. Skills - -**What**: Functional units that extend Claude's capabilities with specialized knowledge and workflows. - -**Structure**: -``` -skill-name/ -├── SKILL.md (required) # YAML frontmatter + Markdown instructions -├── scripts/ (optional) # Executable code (Python/Bash) -├── references/ (optional) # Documentation loaded as needed -└── assets/ (optional) # Templates and resources -``` - -**Loading mechanism** (Progressive Disclosure): -1. **Metadata** (~100 tokens): Always in context (name + description from YAML frontmatter) -2. **SKILL.md body** (<5k tokens): Loaded when Claude determines the skill applies -3. **Bundled resources**: Loaded only as needed by Claude - -**Location**: -- **Personal**: `~/.claude/skills/` (user-specific, not shared) -- **Project**: `.claude/skills/` (checked into git, shared with team) -- **Plugin cache**: `~/.claude/plugins/cache/{marketplace}/{plugin}/{version}/{skill}/` - -**Example**: When you ask "analyze my disk space", Claude loads the `macos-cleaner` skill's SKILL.md, then reads `references/cleanup_targets.md` as needed. - -#### 2. Plugins - -**What**: Distribution units that package one or more skills for installation via marketplaces. - -**Purpose**: Plugins enable: -- One-command installation (`claude plugin install skill-name@marketplace-name`) -- Version management -- Dependency tracking -- Marketplace distribution - -**Relationship to Skills**: -``` -Plugin (marketplace.json entry) -├── Skill 1 (./skill-name-1/) -├── Skill 2 (./skill-name-2/) -└── Skill 3 (./skill-name-3/) -``` - -**Configuration** (in `.claude-plugin/marketplace.json`): -```json -{ - "name": "my-plugin", - "description": "Use when...", - "version": "1.0.0", - "category": "utilities", - "keywords": ["keyword1", "keyword2"], - "skills": ["./skill-1", "./skill-2"] -} -``` - -**Example**: The `skill-creator` plugin contains one skill (`./skill-creator`), while a hypothetical `developer-tools` plugin might contain multiple skills like `./git-helper`, `./code-reviewer`, `./test-runner`. - -#### 3. Agents (Subagents) - -**What**: Specialized autonomous agents invoked via the `Task` tool for complex, multi-step operations. - -**Types**: -- **Bash**: Command execution specialist -- **general-purpose**: Research, search, multi-step tasks -- **Explore**: Fast codebase exploration -- **Plan**: Software architecture planning -- **skill-creator**: Meta-agent for creating skills -- **Custom**: Domain-specific agents (e.g., `test-runner`, `build-validator`) - -**When to use**: -- Tasks requiring multiple rounds of tool calls -- Open-ended exploration (finding files, searching code) -- Planning before implementation -- Autonomous execution without user intervention - -**Example**: -```python -# Instead of manually searching multiple times: -Task( - subagent_type="Explore", - description="Find error handling code", - prompt="Search the codebase for error handling patterns and list all files that handle HTTP errors" -) -``` - -#### 4. Commands - -**What**: Slash commands (e.g., `/commit`, `/review-pr`) that trigger skills. - -**Relationship**: Commands are shortcuts to invoke skills. -- `/commit` → invokes `commit` skill -- `/review-pr` → invokes `review-pr` skill - -**Configuration**: Defined in plugin's `commands/` directory or skill metadata. - -### Architecture Diagram - -``` -Marketplace (GitHub) - ↓ (git clone) -~/.claude/plugins/marketplaces/{marketplace-name}/ - ↓ (plugin install) -~/.claude/plugins/cache/{marketplace-name}/{plugin}/{version}/ - ├── skill-1/ - │ ├── SKILL.md - │ ├── scripts/ - │ └── references/ - └── skill-2/ - └── SKILL.md - ↓ (Claude loads) -Claude Code Context - ├── Metadata (always loaded) - ├── SKILL.md (loaded when relevant) - └── Resources (loaded as needed) -``` - -### Installation Flow - -#### Step 1: User initiates installation -```bash -claude plugin install macos-cleaner@daymade-skills -``` - -#### Step 2: CLI locates marketplace -```bash -# Check ~/.claude/plugins/marketplaces/daymade-skills/ -# If not exists, git clone from GitHub -``` - -#### Step 3: Read marketplace.json -```json -{ - "plugins": [ - { - "name": "macos-cleaner", - "version": "1.0.0", - "skills": ["./macos-cleaner"] - } - ] -} -``` - -#### Step 4: Download to cache -```bash -# Clone entire marketplace repo to: -~/.claude/plugins/cache/daymade-skills/macos-cleaner/1.0.0/ - -# Extract skill to: -~/.claude/plugins/cache/daymade-skills/macos-cleaner/1.0.0/macos-cleaner/ -``` - -#### Step 5: Record installation -```json -// ~/.claude/plugins/installed_plugins.json -{ - "plugins": { - "macos-cleaner@daymade-skills": [{ - "scope": "user", - "installPath": "~/.claude/plugins/cache/daymade-skills/macos-cleaner/1.0.0", - "version": "1.0.0", - "installedAt": "2026-01-11T08:03:46.593Z" - }] - } -} -``` - -#### Step 6: Claude Code loads skill -``` -When user asks: "My Mac is running out of space" - ↓ -Claude scans installed plugins metadata - ↓ -Finds "macos-cleaner" description matches - ↓ -Loads SKILL.md into context - ↓ -Executes workflow (analyze → report → confirm → cleanup) - ↓ -Loads references/scripts as needed -``` - -### Key Files and Locations - -#### Configuration Files - -| File | Location | Purpose | -|------|----------|---------| -| `marketplace.json` | `~/.claude/plugins/marketplaces/{name}/.claude-plugin/` | Defines available plugins | -| `installed_plugins.json` | `~/.claude/plugins/` | Tracks installed plugins | -| `known_marketplaces.json` | `~/.claude/plugins/` | Lists registered marketplaces | - -#### Directory Structure - -``` -~/.claude/ -├── skills/ # Personal skills (not from marketplace) -├── plugins/ -│ ├── marketplaces/ # Marketplace clones -│ │ ├── daymade-skills/ # Marketplace name -│ │ │ └── .claude-plugin/ -│ │ │ └── marketplace.json -│ │ └── anthropic-agent-skills/ -│ ├── cache/ # Installed plugins -│ │ └── daymade-skills/ -│ │ └── macos-cleaner/ -│ │ └── 1.0.0/ # Version -│ │ └── macos-cleaner/ # Skill directory -│ │ ├── SKILL.md -│ │ ├── scripts/ -│ │ └── references/ -│ ├── installed_plugins.json # Installation registry -│ └── known_marketplaces.json # Marketplace registry -``` - -### Data Flow - -#### Skill Activation -``` -User message - ↓ -Claude analyzes installed plugin metadata - ↓ -Matches description to user intent - ↓ -Loads SKILL.md (progressive disclosure) - ↓ -Executes instructions - ↓ -Loads bundled resources (scripts, references) as needed - ↓ -Generates response -``` - -#### Plugin Update -``` -Local changes to skill - ↓ -git add & commit - ↓ -git push to GitHub - ↓ -User runs: claude plugin marketplace update {marketplace-name} - ↓ -CLI pulls latest from GitHub - ↓ -Updates ~/.claude/plugins/marketplaces/{marketplace-name}/ - ↓ -User runs: claude plugin update {plugin-name@marketplace} - ↓ -Re-downloads to cache with new version number - ↓ -Updates installed_plugins.json -``` - -### Common Misconceptions - -#### ❌ Myth 1: "Updating local files immediately updates the plugin" -**Reality**: Plugins are distributed via GitHub. Local changes require `git push` before users can install updates. - -#### ❌ Myth 2: "Skills and plugins are the same thing" -**Reality**: Skills are functional units (SKILL.md + resources). Plugins are distribution packages (can contain multiple skills). - -#### ❌ Myth 3: "marketplace.json is just metadata" -**Reality**: marketplace.json is the **source of truth** for plugin discovery. Without correct configuration here, `claude plugin install` will fail with "Plugin not found". - -#### ❌ Myth 4: "Cache is just for performance" -**Reality**: Cache (`~/.claude/plugins/cache/`) is where installed plugins actually live. Deleting cache uninstalls all plugins. - -#### ❌ Myth 5: "Skills in ~/.claude/skills/ work the same as plugin skills" -**Reality**: -- `~/.claude/skills/` = Personal skills (manual management, no versioning) -- Plugin cache = Managed by CLI (versioned, updateable, shareable) - -### Best Practices - -#### For Skill Authors - -1. **Clear metadata**: Description should clearly state "Use when..." to help Claude match user intent -2. **Progressive disclosure**: Keep SKILL.md lean, move details to `references/` -3. **Test locally first**: Copy to `~/.claude/skills/` for testing before packaging -4. **Version properly**: Use semver (MAJOR.MINOR.PATCH) in marketplace.json -5. **Document bundled resources**: All scripts and references should be mentioned in SKILL.md - -#### For Marketplace Maintainers - -1. **Git workflow**: Always `git push` after updating marketplace.json -2. **Validate JSON**: Run `python -m json.tool marketplace.json` before committing -3. **Update cache**: Remind users to run `claude plugin marketplace update` after releases -4. **Version consistency**: Marketplace version ≠ plugin versions (they track independently) - -#### For Users - -1. **Update marketplaces**: Run `claude plugin marketplace update {name}` periodically -2. **Check installed plugins**: Inspect `~/.claude/plugins/installed_plugins.json` -3. **Clear cache on issues**: `rm -rf ~/.claude/plugins/cache/{marketplace-name}` then reinstall -4. **Understand scopes**: - - `--scope user`: Only you (default) - - `--scope project`: Shared with team via `.claude/plugins/` - - `--scope local`: Gitignored, local only +For full architecture documentation (core concepts, installation flow, data flow, common misconceptions, best practices), see [references/plugin-architecture.md](./references/plugin-architecture.md). ## Plugin and Skill Troubleshooting -This section provides systematic debugging steps for common plugin and skill installation issues. - -### Understanding the Architecture First - -**CRITICAL**: Before troubleshooting, understand that Claude Code's plugin system is **GitHub-based**, not local-file-based. - -``` -GitHub Repository (source of truth) - ↓ (git clone / git pull) -~/.claude/plugins/marketplaces/{marketplace-name}/ - ↓ (claude plugin install) -~/.claude/plugins/cache/{marketplace-name}/{plugin}/{version}/ - ↓ (Claude Code loads) -Active skill in Claude's context -``` - -**Key insight**: Local file changes are NOT visible to `claude plugin install` until pushed to GitHub. - -### Common Error 1: "Plugin not found in marketplace" - -**Error message**: -``` -Installing plugin "skill-name@marketplace-name"... -✘ Failed to install plugin: Plugin "skill-name" not found in marketplace "marketplace-name" -``` - -**Root causes** (in order of likelihood): - -#### Cause 1.1: Local changes not pushed to GitHub ⭐ **MOST COMMON** - -**Symptoms**: -- `git status` shows modified files or untracked directories -- marketplace.json updated locally but install fails -- All documentation updated but plugin not found - -**Diagnosis**: -```bash -# Check if you have uncommitted changes -git status - -# Check last commit vs remote -git log origin/main..HEAD - -# Verify GitHub has latest marketplace.json -# Open: https://github.com/{owner}/{repo}/blob/main/.claude-plugin/marketplace.json -``` - -**Solution**: -```bash -# 1. Commit all changes -git add -A -git commit -m "Release vX.Y.Z: Add {skill-name}" - -# 2. Push to GitHub -git push - -# 3. Clear local marketplace cache -rm -rf ~/.claude/plugins/cache/{marketplace-name} - -# 4. Update marketplace from GitHub -claude plugin marketplace update {marketplace-name} - -# 5. Retry installation -claude plugin install {skill-name}@{marketplace-name} -``` - -**Why this happens**: You updated marketplace.json locally, but Claude CLI pulls from GitHub, not your local filesystem. - -#### Cause 1.2: marketplace.json configuration error - -**Symptoms**: -- Git is up-to-date but install still fails -- Other plugins from same marketplace install fine - -**Diagnosis**: -```bash -# 1. Check marketplace.json syntax -python3 -m json.tool .claude-plugin/marketplace.json > /dev/null - -# 2. Verify plugin entry exists -cat .claude-plugin/marketplace.json | jq '.plugins[] | select(.name == "skill-name")' - -# 3. Check spelling matches exactly -# Plugin name in marketplace.json MUST match install command -``` - -**Common mistakes**: -```json -// ❌ Wrong: name mismatch -{ - "name": "macos_cleaner", // Underscore - "skills": ["./macos-cleaner"] // Dash -} - -// ✅ Correct: consistent naming -{ - "name": "macos-cleaner", - "skills": ["./macos-cleaner"] -} -``` - -**Solution**: Fix marketplace.json, then commit and push. - -#### Cause 1.3: Skill directory missing - -**Symptoms**: -- marketplace.json has entry -- Git is up-to-date -- Plugin installs but skills don't load - -**Diagnosis**: -```bash -# Check if skill directory exists -ls -la {skill-name}/ - -# Verify SKILL.md exists -ls -la {skill-name}/SKILL.md -``` - -**Solution**: Ensure skill directory and SKILL.md are committed to git. - -### Common Error 2: Marketplace cache is stale - -**Symptoms**: -- GitHub has latest changes -- Install finds plugin but gets old version -- Newly added plugins not visible - -**Diagnosis**: -```bash -# Check cache timestamp -ls -la ~/.claude/plugins/cache/{marketplace-name}/ - -# Compare with last git push -git log -1 --format="%ai" -``` - -**Solution**: -```bash -# Option 1: Update marketplace cache -claude plugin marketplace update {marketplace-name} - -# Option 2: Clear cache and re-fetch -rm -rf ~/.claude/plugins/cache/{marketplace-name} -claude plugin marketplace update {marketplace-name} -``` - -### Common Error 3: JSON syntax error - -**Error message**: -``` -Error parsing marketplace manifest -``` - -**Diagnosis**: -```bash -# Validate JSON syntax -python3 -m json.tool .claude-plugin/marketplace.json - -# Check for common issues: -# - Missing commas -# - Trailing commas (in arrays/objects) -# - Unescaped quotes in strings -# - Missing closing braces -``` - -**Solution**: Fix JSON syntax, validate, commit, push. - -### Systematic Debugging Process - -When facing ANY plugin/skill issue, follow this checklist: - -#### Step 1: Verify marketplace registration - -```bash -# List registered marketplaces -claude plugin marketplace list - -# Expected output should include your marketplace -``` - -If missing, register: -```bash -claude plugin marketplace add https://github.com/{owner}/{repo} -``` - -#### Step 2: Check Git status - -```bash -# Are there uncommitted changes? -git status - -# Is local ahead of remote? -git log origin/main..HEAD - -# Push if needed -git push -``` - -#### Step 3: Verify GitHub has latest - -```bash -# Open in browser or use gh CLI -gh browse .claude-plugin/marketplace.json - -# Or check with curl -curl https://raw.githubusercontent.com/{owner}/{repo}/main/.claude-plugin/marketplace.json | jq '.plugins[] | .name' -``` - -#### Step 4: Clear and update cache - -```bash -# Remove stale cache -rm -rf ~/.claude/plugins/cache/{marketplace-name} - -# Re-fetch from GitHub -claude plugin marketplace update {marketplace-name} -``` - -#### Step 5: Validate configuration - -```bash -# Check marketplace.json is valid JSON -python3 -m json.tool .claude-plugin/marketplace.json > /dev/null && echo "✅ Valid JSON" - -# Check plugin entry exists -cat .claude-plugin/marketplace.json | jq '.plugins[] | select(.name == "skill-name")' || echo "❌ Plugin not found" -``` - -#### Step 6: Inspect installation state - -```bash -# Check if plugin is installed -cat ~/.claude/plugins/installed_plugins.json | jq -r '.plugins | keys[]' | grep "skill-name" - -# If installed, check details -cat ~/.claude/plugins/installed_plugins.json | jq '.plugins["skill-name@marketplace-name"]' - -# Verify files exist -ls ~/.claude/plugins/cache/{marketplace-name}/{skill-name}/{version}/ -``` - -### Debugging Commands Reference - -| Purpose | Command | -|---------|---------| -| List marketplaces | `claude plugin marketplace list` | -| Update marketplace | `claude plugin marketplace update {name}` | -| Install plugin | `claude plugin install {plugin}@{marketplace}` | -| Uninstall plugin | `claude plugin uninstall {plugin}@{marketplace}` | -| Update plugin | `claude plugin update {plugin}@{marketplace}` | -| Validate manifest | `claude plugin validate {path}` | -| Check installed plugins | `cat ~/.claude/plugins/installed_plugins.json \| jq '.plugins \| keys'` | -| Inspect plugin details | `cat ~/.claude/plugins/installed_plugins.json \| jq '.plugins["{plugin}@{marketplace}"]'` | -| Clear marketplace cache | `rm -rf ~/.claude/plugins/cache/{marketplace-name}` | -| Verify JSON syntax | `python3 -m json.tool {file.json}` | - -### Understanding File Locations - -```bash -# Marketplace clones (git repositories) -~/.claude/plugins/marketplaces/{marketplace-name}/ - -# Installed plugin cache -~/.claude/plugins/cache/{marketplace-name}/{plugin-name}/{version}/ - -# Installation registry -~/.claude/plugins/installed_plugins.json - -# Personal skills (not from marketplace) -~/.claude/skills/ - -# Project-scoped skills (shared with team) -.claude/skills/ -``` - -### Common Pitfalls - -#### Pitfall 1: Confusing local skills with plugin skills - -```bash -# ❌ Wrong: Copying skill to personal directory doesn't make it installable -cp -r my-skill ~/.claude/skills/my-skill # Works, but not managed by plugin system - -# ✅ Correct: Install via marketplace for version management -claude plugin install my-skill@my-marketplace -``` - -#### Pitfall 2: Forgetting to push after updating marketplace.json - -```bash -# ❌ Wrong workflow -vim .claude-plugin/marketplace.json # Add new plugin -git add .claude-plugin/marketplace.json -git commit -m "Add plugin" -# FORGOT TO PUSH! -claude plugin install new-plugin@my-marketplace # ❌ Fails: not found - -# ✅ Correct workflow -vim .claude-plugin/marketplace.json -git add -A -git commit -m "Add new plugin" -git push # ← CRITICAL STEP -claude plugin marketplace update my-marketplace -claude plugin install new-plugin@my-marketplace # ✅ Works -``` - -#### Pitfall 3: Expecting instant propagation - -```bash -# ❌ Wrong expectation -git push # Push changes -claude plugin install new-plugin@my-marketplace # ❌ Fails: cache is stale - -# ✅ Correct: Update cache first -git push -claude plugin marketplace update my-marketplace # Fetch latest from GitHub -claude plugin install new-plugin@my-marketplace # ✅ Works -``` - -#### Pitfall 4: Inconsistent naming - -```json -// marketplace.json -{ - "name": "my_plugin", // Underscore - "skills": ["./my-plugin"] // Dash - will cause confusion -} -``` - -```bash -# User tries to install -claude plugin install my-plugin@marketplace # ❌ Not found (name has underscore) -claude plugin install my_plugin@marketplace # ✅ Works, but confusing -``` - -**Best practice**: Use consistent kebab-case for both plugin name and skill directory. - -### Real-World Example: macos-cleaner Installation Issue - -**Scenario**: After creating macos-cleaner skill and updating all documentation, `claude plugin install macos-cleaner@daymade-skills` failed with "Plugin not found". - -**Investigation**: -```bash -# 1. Check git status -git status -# Output: 16 modified/untracked files - -# 2. Check marketplace cache -ls -la ~/.claude/plugins/cache/daymade-skills/ -# Output: Last modified Dec 20 (weeks old!) - -# 3. Check GitHub -# marketplace.json on GitHub: version 1.20.0 (old) -# Local marketplace.json: version 1.21.0 (new) -``` - -**Root cause**: Local changes weren't pushed to GitHub. CLI was pulling from GitHub, not local files. - -**Solution**: -```bash -# 1. Commit and push -git add -A -git commit -m "Release v1.21.0: Add macos-cleaner" -git push - -# 2. Update marketplace -claude plugin marketplace update daymade-skills - -# 3. Install -claude plugin install macos-cleaner@daymade-skills -# ✔ Successfully installed plugin: macos-cleaner@daymade-skills -``` - -**Verification**: -```bash -cat ~/.claude/plugins/installed_plugins.json | jq '.plugins["macos-cleaner@daymade-skills"]' -# Output: Installation details with correct version - -ls ~/.claude/plugins/cache/daymade-skills/macos-cleaner/1.0.0/ -# Output: All skill files present -``` - -**Lesson**: Always remember the GitHub-based architecture. Local changes are invisible until pushed. - -### Advanced: Manual Cache Inspection - -If automated commands don't reveal the issue, manually inspect: - -```bash -# 1. Check marketplace clone -cat ~/.claude/plugins/marketplaces/{marketplace-name}/.claude-plugin/marketplace.json | jq '.metadata.version' - -# 2. Check what's in cache -ls -R ~/.claude/plugins/cache/{marketplace-name}/ - -# 3. Compare with GitHub -curl -s https://raw.githubusercontent.com/{owner}/{repo}/main/.claude-plugin/marketplace.json | jq '.metadata.version' - -# 4. Check installation record -cat ~/.claude/plugins/installed_plugins.json | jq '.plugins' | grep -i "{skill-name}" -``` - -### When All Else Fails - -1. **Complete cache reset**: - ```bash - rm -rf ~/.claude/plugins/cache/* - claude plugin marketplace update {marketplace-name} - ``` - -2. **Re-register marketplace**: - ```bash - # Remove marketplace - # (No direct command, manual edit ~/.claude/plugins/known_marketplaces.json) - - # Re-add - claude plugin marketplace add https://github.com/{owner}/{repo} - ``` - -3. **Check Claude Code version**: - ```bash - claude --version - # Plugins require Claude Code v2.0.12+ - ``` - -4. **Enable verbose logging** (if available): - ```bash - CLAUDE_DEBUG=1 claude plugin install {plugin}@{marketplace} - ``` - -### Getting Help - -If you're still stuck: - -1. **Check GitHub issues**: https://github.com/anthropics/claude-code/issues -2. **Verify marketplace.json**: Run `claude plugin validate .claude-plugin/marketplace.json` -3. **Compare with working marketplace**: Study anthropic's official marketplace structure -4. **Document your debugging**: Include output from all diagnostic commands above +For systematic debugging steps (common errors, debugging process, pitfalls, real-world examples), see [references/plugin-troubleshooting.md](./references/plugin-troubleshooting.md). -For this project specifically, refer to: -- [Plugin and Skill Architecture](#plugin-and-skill-architecture) - Architecture overview (this document) -- [skill-creator/SKILL.md](./skill-creator/SKILL.md) - Skill creation guide -- [CONTRIBUTING.md](./CONTRIBUTING.md) - Development workflow +**Quick fix for most issues**: Commit → push → `claude plugin marketplace update daymade-skills` → retry install. diff --git a/references/new-skill-guide.md b/references/new-skill-guide.md new file mode 100644 index 00000000..8f76502f --- /dev/null +++ b/references/new-skill-guide.md @@ -0,0 +1,241 @@ +# Adding a New Skill to Marketplace - Detailed Guide + +**CRITICAL**: When adding a skill to this marketplace, you MUST update all of these files. Missing any file will result in incomplete integration. + +## Files to Update + +``` +✅ CHANGELOG.md (Add version entry) +✅ README.md (7 locations: badges, description, install, skill section, use case, docs link, requirements) +✅ README.zh-CN.md (7 locations: same as above, translated) ⚠️ CRITICAL +✅ CLAUDE.md (3 locations: overview, marketplace config, available skills) +✅ .claude-plugin/marketplace.json (CRITICAL: metadata + new plugin entry) +✅ skill-name/ (The actual skill directory) +✅ skill-name/skill-name.zip (Packaged skill) +``` + +## Step-by-Step Process + +### 1. Refine the Skill (if needed) +```bash +cd skill-creator +python3 scripts/security_scan.py ../skill-name --verbose +``` + +### 2. Package the Skill +```bash +cd skill-creator +python3 scripts/package_skill.py ../skill-name +``` + +### 3. Update CHANGELOG.md + +Add new version entry at the top (after [Unreleased]): + +```markdown +## [X.Y.0] - YYYY-MM-DD + +### Added +- **New Skill**: skill-name - Brief description + - Feature 1 + - Feature 2 + - Bundled scripts/references/assets + +### Changed +- Updated marketplace skills count from N to N+1 +- Updated marketplace version from X.(Y-1).0 to X.Y.0 +- Updated README.md badges (skills count, version) +- Updated README.md to include skill-name in skills listing +- Updated README.zh-CN.md badges (skills count, version) +- Updated README.zh-CN.md to include skill-name in skills listing +- Updated CLAUDE.md skills count from N to N+1 +- Added skill-name use case section to README.md +- Added skill-name use case section to README.zh-CN.md +- Added dependencies to requirements section (if any, both EN and ZH) +``` + +### 4. Update README.md + +**a. Update badges (top of file):** +```markdown +[![Skills](https://img.shields.io/badge/skills-N-blue.svg)] +[![Version](https://img.shields.io/badge/version-X.Y.0-green.svg)] +``` + +**b. Update description:** +```markdown +Professional Claude Code skills marketplace featuring N production-ready skills... +``` + +**c. Add installation command:** +```markdown +# Brief description +claude plugin install skill-name@daymade-skills +``` + +**d. Add skill section (### N. **skill-name**):** +```markdown +### N. **skill-name** - One-line Title + +Brief description paragraph. + +**When to use:** +- Use case 1 +- Use case 2 + +**Key features:** +- Feature 1 +- Feature 2 + +**Example usage:** +\`\`\`bash +# Example commands +\`\`\` + +**🎬 Live Demo** + +*Coming soon* (or add demo GIF) + +📚 **Documentation**: See [skill-name/references/](./skill-name/references/)... + +**Requirements**: Dependencies (e.g., Python 3.8+, FFmpeg, etc.) +``` + +**e. Add use case section:** +```markdown +### For [Use Case Category] +Use **skill-name** to [describe primary use case]. Combine with **other-skill** to [describe integration]. +``` + +**f. Add documentation quick link and update requirements section (if needed).** + +### 5. Update CLAUDE.md + +- Update repository overview skill count +- Update marketplace configuration plugin count +- Add skill to Available Skills list + +### 6. Update .claude-plugin/marketplace.json (MOST IMPORTANT) + +```json +{ + "name": "skill-name", + "description": "Clear description with trigger conditions. Use when [scenarios]", + "source": "./", + "strict": false, + "version": "1.0.0", + "category": "appropriate-category", + "keywords": ["keyword1", "keyword2", "keyword3"], + "skills": ["./skill-name"] +} +``` + +**Categories:** `developer-tools`, `document-conversion`, `documentation`, `customization`, `communication`, `utilities`, `assets`, `design`, `productivity`, `security`, `media` + +Validate: `python3 -m json.tool .claude-plugin/marketplace.json > /dev/null` + +### 7. Update README.zh-CN.md + +**CRITICAL**: Chinese documentation must be kept in sync with English version. + +Same 7 locations as README.md, translated to professional technical Chinese. Keep code examples in English. + +### 8. Commit and Release + +```bash +# Commit marketplace update +git add .claude-plugin/marketplace.json skill-name/ +git commit -m "Release vX.Y.0: Add skill-name + +- Add skill-name vX.Y.Z +- Update marketplace to vX.Y.0" + +# Commit documentation +git add README.md README.zh-CN.md CLAUDE.md CHANGELOG.md demos/ +git commit -m "docs: Update README for vX.Y.0 with skill-name" + +# Push +git push + +# Create GitHub release +gh release create vX.Y.0 \ + --title "Release vX.Y.0: Add skill-name - Description" \ + --notes "$(cat <<'EOF' +## New Skill: skill-name + +Features: +- Feature 1 +- Feature 2 + +Installation: +```bash +claude plugin install skill-name@daymade-skills +``` + +Changelog: ... +EOF +)" +``` + +### 9. Generate Demo (Optional but Recommended) + +```bash +./cli-demo-generator/scripts/auto_generate_demo.py \ + -c "command1" \ + -c "command2" \ + -o demos/skill-name/demo-name.gif \ + --title "Skill Demo" \ + --theme "Dracula" +``` + +## Verification Checklist + +Before committing, verify: + +- [ ] CHANGELOG.md has new version entry +- [ ] README.md badges updated (skills count + version) +- [ ] README.md has skill section with number +- [ ] README.md has use case section +- [ ] README.md has documentation link +- [ ] README.md requirements updated (if needed) +- [ ] README.zh-CN.md badges updated (skills count + version) +- [ ] README.zh-CN.md has skill section with number +- [ ] README.zh-CN.md has use case section +- [ ] README.zh-CN.md has documentation link +- [ ] README.zh-CN.md requirements updated (if needed) +- [ ] README.zh-CN.md installation command added +- [ ] CLAUDE.md skill count updated in 3 places +- [ ] CLAUDE.md has skill in Available Skills list +- [ ] marketplace.json metadata.version updated +- [ ] marketplace.json metadata.description updated +- [ ] marketplace.json has new plugin entry +- [ ] marketplace.json validates (python3 -m json.tool) +- [ ] skill-name.zip package exists +- [ ] Security scan passed + +## Common Mistakes to Avoid + +1. **Forgetting marketplace.json** - Without this, `claude plugin install` fails +2. **Forgetting Chinese documentation** - README.zh-CN.md must be updated in sync (6 locations) +3. **Inconsistent version numbers** - CHANGELOG, README badges (both EN and ZH), CLAUDE.md, and marketplace.json must all match +4. **Inconsistent skill counts** - README description (both EN and ZH), badges, CLAUDE.md must all have same count +5. **Missing skill number in README** - Skills must be numbered sequentially in both EN and ZH versions +6. **Invalid JSON syntax** - Always validate marketplace.json after editing +7. **Forgetting to push** - Local changes are invisible until pushed to GitHub + +## Quick Reference Commands + +```bash +# 1. Refine and validate skill +cd skill-creator && python3 scripts/security_scan.py ../skill-name --verbose + +# 2. Package skill +python3 scripts/package_skill.py ../skill-name + +# 3. Validate marketplace.json +cd .. && python3 -m json.tool .claude-plugin/marketplace.json > /dev/null && echo "✅ Valid" + +# 4. Verify Chinese documentation is in sync +grep "skills-[0-9]*" README.md README.zh-CN.md +grep "version-[0-9.]*" README.md README.zh-CN.md +``` diff --git a/references/plugin-architecture.md b/references/plugin-architecture.md new file mode 100644 index 00000000..b5239ea4 --- /dev/null +++ b/references/plugin-architecture.md @@ -0,0 +1,296 @@ +# Plugin and Skill Architecture + +This document explains the architecture of Claude Code's extension system and how different components work together. + +## Core Concepts + +### 1. Skills + +**What**: Functional units that extend Claude's capabilities with specialized knowledge and workflows. + +**Structure**: +``` +skill-name/ +├── SKILL.md (required) # YAML frontmatter + Markdown instructions +├── scripts/ (optional) # Executable code (Python/Bash) +├── references/ (optional) # Documentation loaded as needed +└── assets/ (optional) # Templates and resources +``` + +**Loading mechanism** (Progressive Disclosure): +1. **Metadata** (~100 tokens): Always in context (name + description from YAML frontmatter) +2. **SKILL.md body** (<5k tokens): Loaded when Claude determines the skill applies +3. **Bundled resources**: Loaded only as needed by Claude + +**Location**: +- **Personal**: `~/.claude/skills/` (user-specific, not shared) +- **Project**: `.claude/skills/` (checked into git, shared with team) +- **Plugin cache**: `~/.claude/plugins/cache/{marketplace}/{plugin}/{version}/{skill}/` + +**Example**: When you ask "analyze my disk space", Claude loads the `macos-cleaner` skill's SKILL.md, then reads `references/cleanup_targets.md` as needed. + +### 2. Plugins + +**What**: Distribution units that package one or more skills for installation via marketplaces. + +**Purpose**: Plugins enable: +- One-command installation (`claude plugin install skill-name@marketplace-name`) +- Version management +- Dependency tracking +- Marketplace distribution + +**Relationship to Skills**: +``` +Plugin (marketplace.json entry) +├── Skill 1 (./skill-name-1/) +├── Skill 2 (./skill-name-2/) +└── Skill 3 (./skill-name-3/) +``` + +**Configuration** (in `.claude-plugin/marketplace.json`): +```json +{ + "name": "my-plugin", + "description": "Use when...", + "version": "1.0.0", + "category": "utilities", + "keywords": ["keyword1", "keyword2"], + "skills": ["./skill-1", "./skill-2"] +} +``` + +**Example**: The `skill-creator` plugin contains one skill (`./skill-creator`), while a hypothetical `developer-tools` plugin might contain multiple skills like `./git-helper`, `./code-reviewer`, `./test-runner`. + +### 3. Agents (Subagents) + +**What**: Specialized autonomous agents invoked via the `Task` tool for complex, multi-step operations. + +**Types**: +- **Bash**: Command execution specialist +- **general-purpose**: Research, search, multi-step tasks +- **Explore**: Fast codebase exploration +- **Plan**: Software architecture planning +- **skill-creator**: Meta-agent for creating skills +- **Custom**: Domain-specific agents (e.g., `test-runner`, `build-validator`) + +**When to use**: +- Tasks requiring multiple rounds of tool calls +- Open-ended exploration (finding files, searching code) +- Planning before implementation +- Autonomous execution without user intervention + +**Example**: +```python +# Instead of manually searching multiple times: +Task( + subagent_type="Explore", + description="Find error handling code", + prompt="Search the codebase for error handling patterns and list all files that handle HTTP errors" +) +``` + +### 4. Commands + +**What**: Slash commands (e.g., `/commit`, `/review-pr`) that trigger skills. + +**Relationship**: Commands are shortcuts to invoke skills. +- `/commit` → invokes `commit` skill +- `/review-pr` → invokes `review-pr` skill + +**Configuration**: Defined in plugin's `commands/` directory or skill metadata. + +## Architecture Diagram + +``` +Marketplace (GitHub) + ↓ (git clone) +~/.claude/plugins/marketplaces/{marketplace-name}/ + ↓ (plugin install) +~/.claude/plugins/cache/{marketplace-name}/{plugin}/{version}/ + ├── skill-1/ + │ ├── SKILL.md + │ ├── scripts/ + │ └── references/ + └── skill-2/ + └── SKILL.md + ↓ (Claude loads) +Claude Code Context + ├── Metadata (always loaded) + ├── SKILL.md (loaded when relevant) + └── Resources (loaded as needed) +``` + +## Installation Flow + +### Step 1: User initiates installation +```bash +claude plugin install macos-cleaner@daymade-skills +``` + +### Step 2: CLI locates marketplace +```bash +# Check ~/.claude/plugins/marketplaces/daymade-skills/ +# If not exists, git clone from GitHub +``` + +### Step 3: Read marketplace.json +```json +{ + "plugins": [ + { + "name": "macos-cleaner", + "version": "1.0.0", + "skills": ["./macos-cleaner"] + } + ] +} +``` + +### Step 4: Download to cache +```bash +# Clone entire marketplace repo to: +~/.claude/plugins/cache/daymade-skills/macos-cleaner/1.0.0/ + +# Extract skill to: +~/.claude/plugins/cache/daymade-skills/macos-cleaner/1.0.0/macos-cleaner/ +``` + +### Step 5: Record installation +```json +// ~/.claude/plugins/installed_plugins.json +{ + "plugins": { + "macos-cleaner@daymade-skills": [{ + "scope": "user", + "installPath": "~/.claude/plugins/cache/daymade-skills/macos-cleaner/1.0.0", + "version": "1.0.0", + "installedAt": "2026-01-11T08:03:46.593Z" + }] + } +} +``` + +### Step 6: Claude Code loads skill +``` +When user asks: "My Mac is running out of space" + ↓ +Claude scans installed plugins metadata + ↓ +Finds "macos-cleaner" description matches + ↓ +Loads SKILL.md into context + ↓ +Executes workflow (analyze → report → confirm → cleanup) + ↓ +Loads references/scripts as needed +``` + +## Key Files and Locations + +### Configuration Files + +| File | Location | Purpose | +|------|----------|---------| +| `marketplace.json` | `~/.claude/plugins/marketplaces/{name}/.claude-plugin/` | Defines available plugins | +| `installed_plugins.json` | `~/.claude/plugins/` | Tracks installed plugins | +| `known_marketplaces.json` | `~/.claude/plugins/` | Lists registered marketplaces | + +### Directory Structure + +``` +~/.claude/ +├── skills/ # Personal skills (not from marketplace) +├── plugins/ +│ ├── marketplaces/ # Marketplace clones +│ │ ├── daymade-skills/ # Marketplace name +│ │ │ └── .claude-plugin/ +│ │ │ └── marketplace.json +│ │ └── anthropic-agent-skills/ +│ ├── cache/ # Installed plugins +│ │ └── daymade-skills/ +│ │ └── macos-cleaner/ +│ │ └── 1.0.0/ # Version +│ │ └── macos-cleaner/ # Skill directory +│ │ ├── SKILL.md +│ │ ├── scripts/ +│ │ └── references/ +│ ├── installed_plugins.json # Installation registry +│ └── known_marketplaces.json # Marketplace registry +``` + +## Data Flow + +### Skill Activation +``` +User message + ↓ +Claude analyzes installed plugin metadata + ↓ +Matches description to user intent + ↓ +Loads SKILL.md (progressive disclosure) + ↓ +Executes instructions + ↓ +Loads bundled resources (scripts, references) as needed + ↓ +Generates response +``` + +### Plugin Update +``` +Local changes to skill + ↓ +git add & commit + ↓ +git push to GitHub + ↓ +User runs: claude plugin marketplace update {marketplace-name} + ↓ +CLI pulls latest from GitHub + ↓ +Updates ~/.claude/plugins/marketplaces/{marketplace-name}/ + ↓ +User runs: claude plugin update {plugin-name@marketplace} + ↓ +Re-downloads to cache with new version number + ↓ +Updates installed_plugins.json +``` + +## Common Misconceptions + +| Myth | Reality | +|------|---------| +| "Updating local files immediately updates the plugin" | Plugins are distributed via GitHub. Local changes require `git push` before users can install updates. | +| "Skills and plugins are the same thing" | Skills are functional units (SKILL.md + resources). Plugins are distribution packages (can contain multiple skills). | +| "marketplace.json is just metadata" | marketplace.json is the **source of truth** for plugin discovery. Without correct configuration here, `claude plugin install` will fail. | +| "Cache is just for performance" | Cache (`~/.claude/plugins/cache/`) is where installed plugins actually live. Deleting cache uninstalls all plugins. | +| "Skills in ~/.claude/skills/ work the same as plugin skills" | `~/.claude/skills/` = Personal skills (manual, no versioning). Plugin cache = Managed by CLI (versioned, updateable, shareable). | + +## Best Practices + +### For Skill Authors + +1. **Clear metadata**: Description should clearly state "Use when..." to help Claude match user intent +2. **Progressive disclosure**: Keep SKILL.md lean, move details to `references/` +3. **Test locally first**: Copy to `~/.claude/skills/` for testing before packaging +4. **Version properly**: Use semver (MAJOR.MINOR.PATCH) in marketplace.json +5. **Document bundled resources**: All scripts and references should be mentioned in SKILL.md + +### For Marketplace Maintainers + +1. **Git workflow**: Always `git push` after updating marketplace.json +2. **Validate JSON**: Run `python -m json.tool marketplace.json` before committing +3. **Update cache**: Remind users to run `claude plugin marketplace update` after releases +4. **Version consistency**: Marketplace version ≠ plugin versions (they track independently) + +### For Users + +1. **Update marketplaces**: Run `claude plugin marketplace update {name}` periodically +2. **Check installed plugins**: Inspect `~/.claude/plugins/installed_plugins.json` +3. **Clear cache on issues**: `rm -rf ~/.claude/plugins/cache/{marketplace-name}` then reinstall +4. **Understand scopes**: + - `--scope user`: Only you (default) + - `--scope project`: Shared with team via `.claude/plugins/` + - `--scope local`: Gitignored, local only diff --git a/references/plugin-troubleshooting.md b/references/plugin-troubleshooting.md new file mode 100644 index 00000000..4f7beffa --- /dev/null +++ b/references/plugin-troubleshooting.md @@ -0,0 +1,441 @@ +# Plugin and Skill Troubleshooting + +Systematic debugging steps for common plugin and skill installation issues. + +## Understanding the Architecture First + +**CRITICAL**: Claude Code's plugin system is **GitHub-based**, not local-file-based. + +``` +GitHub Repository (source of truth) + ↓ (git clone / git pull) +~/.claude/plugins/marketplaces/{marketplace-name}/ + ↓ (claude plugin install) +~/.claude/plugins/cache/{marketplace-name}/{plugin}/{version}/ + ↓ (Claude Code loads) +Active skill in Claude's context +``` + +**Key insight**: Local file changes are NOT visible to `claude plugin install` until pushed to GitHub. + +## Common Error 1: "Plugin not found in marketplace" + +**Error message**: +``` +Installing plugin "skill-name@marketplace-name"... +✘ Failed to install plugin: Plugin "skill-name" not found in marketplace "marketplace-name" +``` + +**Root causes** (in order of likelihood): + +### Cause 1.1: Local changes not pushed to GitHub (MOST COMMON) + +**Symptoms**: +- `git status` shows modified files or untracked directories +- marketplace.json updated locally but install fails +- All documentation updated but plugin not found + +**Diagnosis**: +```bash +# Check if you have uncommitted changes +git status + +# Check last commit vs remote +git log origin/main..HEAD + +# Verify GitHub has latest marketplace.json +# Open: https://github.com/{owner}/{repo}/blob/main/.claude-plugin/marketplace.json +``` + +**Solution**: +```bash +# 1. Commit all changes +git add -A +git commit -m "Release vX.Y.Z: Add {skill-name}" + +# 2. Push to GitHub +git push + +# 3. Clear local marketplace cache +rm -rf ~/.claude/plugins/cache/{marketplace-name} + +# 4. Update marketplace from GitHub +claude plugin marketplace update {marketplace-name} + +# 5. Retry installation +claude plugin install {skill-name}@{marketplace-name} +``` + +**Why this happens**: You updated marketplace.json locally, but Claude CLI pulls from GitHub, not your local filesystem. + +### Cause 1.2: marketplace.json configuration error + +**Symptoms**: +- Git is up-to-date but install still fails +- Other plugins from same marketplace install fine + +**Diagnosis**: +```bash +# 1. Check marketplace.json syntax +python3 -m json.tool .claude-plugin/marketplace.json > /dev/null + +# 2. Verify plugin entry exists +cat .claude-plugin/marketplace.json | jq '.plugins[] | select(.name == "skill-name")' + +# 3. Check spelling matches exactly +# Plugin name in marketplace.json MUST match install command +``` + +**Common mistakes**: +```json +// ❌ Wrong: name mismatch +{ + "name": "macos_cleaner", // Underscore + "skills": ["./macos-cleaner"] // Dash +} + +// ✅ Correct: consistent naming +{ + "name": "macos-cleaner", + "skills": ["./macos-cleaner"] +} +``` + +**Solution**: Fix marketplace.json, then commit and push. + +### Cause 1.3: Skill directory missing + +**Symptoms**: +- marketplace.json has entry +- Git is up-to-date +- Plugin installs but skills don't load + +**Diagnosis**: +```bash +# Check if skill directory exists +ls -la {skill-name}/ + +# Verify SKILL.md exists +ls -la {skill-name}/SKILL.md +``` + +**Solution**: Ensure skill directory and SKILL.md are committed to git. + +## Common Error 2: Marketplace cache is stale + +**Symptoms**: +- GitHub has latest changes +- Install finds plugin but gets old version +- Newly added plugins not visible + +**Diagnosis**: +```bash +# Check cache timestamp +ls -la ~/.claude/plugins/cache/{marketplace-name}/ + +# Compare with last git push +git log -1 --format="%ai" +``` + +**Solution**: +```bash +# Option 1: Update marketplace cache +claude plugin marketplace update {marketplace-name} + +# Option 2: Clear cache and re-fetch +rm -rf ~/.claude/plugins/cache/{marketplace-name} +claude plugin marketplace update {marketplace-name} +``` + +## Common Error 3: JSON syntax error + +**Error message**: +``` +Error parsing marketplace manifest +``` + +**Diagnosis**: +```bash +# Validate JSON syntax +python3 -m json.tool .claude-plugin/marketplace.json + +# Check for common issues: +# - Missing commas +# - Trailing commas (in arrays/objects) +# - Unescaped quotes in strings +# - Missing closing braces +``` + +**Solution**: Fix JSON syntax, validate, commit, push. + +## Systematic Debugging Process + +When facing ANY plugin/skill issue, follow this checklist: + +### Step 1: Verify marketplace registration + +```bash +# List registered marketplaces +claude plugin marketplace list + +# Expected output should include your marketplace +``` + +If missing, register: +```bash +claude plugin marketplace add https://github.com/{owner}/{repo} +``` + +### Step 2: Check Git status + +```bash +# Are there uncommitted changes? +git status + +# Is local ahead of remote? +git log origin/main..HEAD + +# Push if needed +git push +``` + +### Step 3: Verify GitHub has latest + +```bash +# Open in browser or use gh CLI +gh browse .claude-plugin/marketplace.json + +# Or check with curl +curl https://raw.githubusercontent.com/{owner}/{repo}/main/.claude-plugin/marketplace.json | jq '.plugins[] | .name' +``` + +### Step 4: Clear and update cache + +```bash +# Remove stale cache +rm -rf ~/.claude/plugins/cache/{marketplace-name} + +# Re-fetch from GitHub +claude plugin marketplace update {marketplace-name} +``` + +### Step 5: Validate configuration + +```bash +# Check marketplace.json is valid JSON +python3 -m json.tool .claude-plugin/marketplace.json > /dev/null && echo "✅ Valid JSON" + +# Check plugin entry exists +cat .claude-plugin/marketplace.json | jq '.plugins[] | select(.name == "skill-name")' || echo "❌ Plugin not found" +``` + +### Step 6: Inspect installation state + +```bash +# Check if plugin is installed +cat ~/.claude/plugins/installed_plugins.json | jq -r '.plugins | keys[]' | grep "skill-name" + +# If installed, check details +cat ~/.claude/plugins/installed_plugins.json | jq '.plugins["skill-name@marketplace-name"]' + +# Verify files exist +ls ~/.claude/plugins/cache/{marketplace-name}/{skill-name}/{version}/ +``` + +## Debugging Commands Reference + +| Purpose | Command | +|---------|---------| +| List marketplaces | `claude plugin marketplace list` | +| Update marketplace | `claude plugin marketplace update {name}` | +| Install plugin | `claude plugin install {plugin}@{marketplace}` | +| Uninstall plugin | `claude plugin uninstall {plugin}@{marketplace}` | +| Update plugin | `claude plugin update {plugin}@{marketplace}` | +| Validate manifest | `claude plugin validate {path}` | +| Check installed plugins | `cat ~/.claude/plugins/installed_plugins.json \| jq '.plugins \| keys'` | +| Inspect plugin details | `cat ~/.claude/plugins/installed_plugins.json \| jq '.plugins["{plugin}@{marketplace}"]'` | +| Clear marketplace cache | `rm -rf ~/.claude/plugins/cache/{marketplace-name}` | +| Verify JSON syntax | `python3 -m json.tool {file.json}` | + +## File Locations + +```bash +# Marketplace clones (git repositories) +~/.claude/plugins/marketplaces/{marketplace-name}/ + +# Installed plugin cache +~/.claude/plugins/cache/{marketplace-name}/{plugin-name}/{version}/ + +# Installation registry +~/.claude/plugins/installed_plugins.json + +# Personal skills (not from marketplace) +~/.claude/skills/ + +# Project-scoped skills (shared with team) +.claude/skills/ +``` + +## Common Pitfalls + +### Pitfall 1: Confusing local skills with plugin skills + +```bash +# ❌ Wrong: Copying skill to personal directory doesn't make it installable +cp -r my-skill ~/.claude/skills/my-skill # Works, but not managed by plugin system + +# ✅ Correct: Install via marketplace for version management +claude plugin install my-skill@my-marketplace +``` + +### Pitfall 2: Forgetting to push after updating marketplace.json + +```bash +# ❌ Wrong workflow +vim .claude-plugin/marketplace.json # Add new plugin +git add .claude-plugin/marketplace.json +git commit -m "Add plugin" +# FORGOT TO PUSH! +claude plugin install new-plugin@my-marketplace # ❌ Fails: not found + +# ✅ Correct workflow +vim .claude-plugin/marketplace.json +git add -A +git commit -m "Add new plugin" +git push # ← CRITICAL STEP +claude plugin marketplace update my-marketplace +claude plugin install new-plugin@my-marketplace # ✅ Works +``` + +### Pitfall 3: Expecting instant propagation + +```bash +# ❌ Wrong expectation +git push # Push changes +claude plugin install new-plugin@my-marketplace # ❌ Fails: cache is stale + +# ✅ Correct: Update cache first +git push +claude plugin marketplace update my-marketplace # Fetch latest from GitHub +claude plugin install new-plugin@my-marketplace # ✅ Works +``` + +### Pitfall 4: Inconsistent naming + +```json +// marketplace.json +{ + "name": "my_plugin", // Underscore + "skills": ["./my-plugin"] // Dash - will cause confusion +} +``` + +```bash +# User tries to install +claude plugin install my-plugin@marketplace # ❌ Not found (name has underscore) +claude plugin install my_plugin@marketplace # ✅ Works, but confusing +``` + +**Best practice**: Use consistent kebab-case for both plugin name and skill directory. + +## Real-World Example: macos-cleaner Installation Issue + +**Scenario**: After creating macos-cleaner skill and updating all documentation, `claude plugin install macos-cleaner@daymade-skills` failed with "Plugin not found". + +**Investigation**: +```bash +# 1. Check git status +git status +# Output: 16 modified/untracked files + +# 2. Check marketplace cache +ls -la ~/.claude/plugins/cache/daymade-skills/ +# Output: Last modified Dec 20 (weeks old!) + +# 3. Check GitHub +# marketplace.json on GitHub: version 1.20.0 (old) +# Local marketplace.json: version 1.21.0 (new) +``` + +**Root cause**: Local changes weren't pushed to GitHub. CLI was pulling from GitHub, not local files. + +**Solution**: +```bash +# 1. Commit and push +git add -A +git commit -m "Release v1.21.0: Add macos-cleaner" +git push + +# 2. Update marketplace +claude plugin marketplace update daymade-skills + +# 3. Install +claude plugin install macos-cleaner@daymade-skills +# ✔ Successfully installed plugin: macos-cleaner@daymade-skills +``` + +**Verification**: +```bash +cat ~/.claude/plugins/installed_plugins.json | jq '.plugins["macos-cleaner@daymade-skills"]' +# Output: Installation details with correct version + +ls ~/.claude/plugins/cache/daymade-skills/macos-cleaner/1.0.0/ +# Output: All skill files present +``` + +**Lesson**: Always remember the GitHub-based architecture. Local changes are invisible until pushed. + +## Advanced: Manual Cache Inspection + +If automated commands don't reveal the issue, manually inspect: + +```bash +# 1. Check marketplace clone +cat ~/.claude/plugins/marketplaces/{marketplace-name}/.claude-plugin/marketplace.json | jq '.metadata.version' + +# 2. Check what's in cache +ls -R ~/.claude/plugins/cache/{marketplace-name}/ + +# 3. Compare with GitHub +curl -s https://raw.githubusercontent.com/{owner}/{repo}/main/.claude-plugin/marketplace.json | jq '.metadata.version' + +# 4. Check installation record +cat ~/.claude/plugins/installed_plugins.json | jq '.plugins' | grep -i "{skill-name}" +``` + +## When All Else Fails + +1. **Complete cache reset**: + ```bash + rm -rf ~/.claude/plugins/cache/* + claude plugin marketplace update {marketplace-name} + ``` + +2. **Re-register marketplace**: + ```bash + # Remove marketplace + # (No direct command, manual edit ~/.claude/plugins/known_marketplaces.json) + + # Re-add + claude plugin marketplace add https://github.com/{owner}/{repo} + ``` + +3. **Check Claude Code version**: + ```bash + claude --version + # Plugins require Claude Code v2.0.12+ + ``` + +4. **Enable verbose logging** (if available): + ```bash + CLAUDE_DEBUG=1 claude plugin install {plugin}@{marketplace} + ``` + +## Getting Help + +If you're still stuck: + +1. **Check GitHub issues**: https://github.com/anthropics/claude-code/issues +2. **Verify marketplace.json**: Run `claude plugin validate .claude-plugin/marketplace.json` +3. **Compare with working marketplace**: Study anthropic's official marketplace structure +4. **Document your debugging**: Include output from all diagnostic commands above diff --git a/references/promotion-policy.md b/references/promotion-policy.md new file mode 100644 index 00000000..d6212a96 --- /dev/null +++ b/references/promotion-policy.md @@ -0,0 +1,60 @@ +# Handling Third-Party Marketplace Promotion Requests + +This repository is a **personal curated marketplace**, NOT a community directory or ecosystem hub. All requests to add third-party marketplace links, skill collection references, or "Community Marketplaces" sections should be declined. + +## Policy + +**DO NOT accept:** +- PRs adding "Related Resources" or "Community Marketplaces" sections linking to third-party skill collections +- Issues requesting promotion of external marketplaces +- PRs adding links to other skill repositories in README.md + +**Rationale:** +1. **Scope creep**: Shifts repository purpose from curated skills to ecosystem directory +2. **Implicit endorsement**: Listing implies quality/security review we cannot maintain +3. **Maintenance burden**: Would need to track and vet external projects over time +4. **Precedent setting**: Accepting one creates obligation to accept others + +## Response Template + +When declining, use this approach: + +```markdown +Hi @{username}, + +Thank you for your interest and for sharing {project-name}! {Brief positive acknowledgment of their project}. + +However, I'm keeping this repository focused as a **personal curated marketplace** rather than a directory of external skill collections. Adding third-party references would: + +1. Shift the repository's scope from curated skills to ecosystem directory +2. Create implicit endorsement expectations I can't maintain +3. Set precedent for similar requests (reference other declined requests if applicable) + +**What you can do instead:** + +1. **Standalone marketplace** - Your repo already works as an independent marketplace: + ``` + /plugin marketplace add {owner}/{repo} + ``` + +2. **Community channels** - Promote through: + - Claude Code GitHub discussions/issues (Anthropic's official repo) + - Developer communities (Reddit, Discord, etc.) + - Your own blog/social media + +3. **Official registry** - If/when Anthropic launches an official skill registry, that would be the appropriate place for ecosystem-wide discovery. + +Your marketplace can succeed on its own merits. Good luck with {project-name}! +``` + +## Workflow + +1. **Review the request** - Confirm it's a third-party promotion (not a legitimate contribution) +2. **Add polite comment** - Use template above, customize for their specific project +3. **Close with reason** - Use "not planned" for issues, just close for PRs +4. **Reference precedent** - Link to previously declined requests for consistency (e.g., #7, PR #5) + +## Examples + +- **Issue #7**: "Add Community Marketplaces section - Protocol Thunderdome" → Declined, closed as "not planned" +- **PR #5**: "Add Trail of Bits Security Skills to Related Resources" → Declined, closed diff --git a/youtube-downloader/references/internal-sop.md b/youtube-downloader/references/internal-sop.md new file mode 100644 index 00000000..5fdd1be3 --- /dev/null +++ b/youtube-downloader/references/internal-sop.md @@ -0,0 +1,12 @@ +# YouTube Downloader Internal SOP + +Use this SOP to avoid common yt-dlp failures and confusion: + +1. Quote YouTube URLs in shell commands (zsh treats `?` as glob). Example: `'https://www.youtube.com/watch?v=VIDEO_ID'`. +2. Ensure proxy is active for both yt-dlp and PO Token providers (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY). +3. If you see "Sign in to confirm you're not a bot", request cookie permission and use browser cookies. +4. Start the PO Token provider before downloading. Prefer Docker bgutil; fall back to browser-based WPC when Docker is unavailable or fails. +5. Use `web_safari` client when cookies are present; otherwise use `mweb` for PO tokens. +6. Keep the browser window open while WPC is minting tokens and make sure it can reach YouTube through the same proxy. +7. If you see "Only images are available" or "Requested format is not available", treat it as PO token failure and retry after fixing provider/browser state. +8. If you see SSL EOF or fragment errors, treat it as proxy instability. Retry with progressive formats or switch to a more stable proxy. From 2192458ef7f3697602bbf3a6d854d6d96d1bc2e8 Mon Sep 17 00:00:00 2001 From: daymade Date: Wed, 18 Mar 2026 23:08:55 +0800 Subject: [PATCH 008/137] release: add scrapling-skill and fix script compatibility - add scrapling-skill with validated CLI workflow, diagnostics, packaging, and docs integration - fix skill-creator package_skill.py so direct script invocation works from repo root - fix continue-claude-work extract_resume_context.py typing compatibility for local python3 - bump marketplace to 1.39.0 and updated skill versions --- .claude-plugin/marketplace.json | 33 ++- CHANGELOG.md | 19 ++ CLAUDE.md | 7 +- README.md | 52 ++++- README.zh-CN.md | 52 ++++- continue-claude-work/.security-scan-passed | 4 +- .../scripts/extract_resume_context.py | 40 ++-- scrapling-skill.skill | Bin 0 -> 6223 bytes scrapling-skill/.security-scan-passed | 4 + scrapling-skill/SKILL.md | 183 +++++++++++++++++ scrapling-skill/references/troubleshooting.md | 164 +++++++++++++++ scrapling-skill/scripts/diagnose_scrapling.py | 191 ++++++++++++++++++ skill-creator/scripts/package_skill.py | 9 +- 13 files changed, 722 insertions(+), 36 deletions(-) create mode 100644 scrapling-skill.skill create mode 100644 scrapling-skill/.security-scan-passed create mode 100644 scrapling-skill/SKILL.md create mode 100644 scrapling-skill/references/troubleshooting.md create mode 100755 scrapling-skill/scripts/diagnose_scrapling.py diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index df931871..dd8dc895 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -5,8 +5,8 @@ "email": "daymadev89@gmail.com" }, "metadata": { - "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, and macOS programmatic window screenshot capture workflows", - "version": "1.38.0", + "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, macOS programmatic window screenshot capture workflows, and verified Scrapling CLI installation and web extraction workflows", + "version": "1.39.0", "homepage": "https://github.com/daymade/claude-code-skills" }, "plugins": [ @@ -15,7 +15,7 @@ "description": "Essential meta-skill for creating effective Claude Code skills with initialization scripts, validation, packaging, marketplace registration, and privacy best practices", "source": "./", "strict": false, - "version": "1.5.0", + "version": "1.5.1", "category": "developer-tools", "keywords": [ "skill-creation", @@ -500,7 +500,7 @@ "description": "Develops iOS applications with XcodeGen, SwiftUI, and SPM. Use when configuring XcodeGen project.yml, resolving SPM dependency issues, deploying to devices, handling code signing, debugging camera/AVFoundation, iOS version compatibility issues, or fixing Library not loaded @rpath framework errors. Includes state machine testing patterns for @MainActor classes", "source": "./", "strict": false, - "version": "1.1.1", + "version": "1.1.0", "category": "developer-tools", "keywords": [ "ios", @@ -565,7 +565,7 @@ "description": "Intelligent macOS disk space analysis and cleanup with safety-first philosophy. Use when users report disk space issues, need to clean their Mac, or want to understand storage consumption. Analyzes system caches, application remnants, large files, and development environments (Docker, Homebrew, npm, pip) with risk categorization (Safe/Caution/Keep) and requires explicit user confirmation before any deletions. Includes Mole visual tool integration for hybrid workflow", "source": "./", "strict": false, - "version": "1.1.1", + "version": "1.1.0", "category": "utilities", "keywords": [ "macos", @@ -882,7 +882,7 @@ "description": "Recover actionable context from local `.claude` session artifacts and continue interrupted work without running `claude --resume`. Extracts compact boundary summaries, subagent workflow state, session end reason, and workspace drift via bundled Python script. Use when a user provides a Claude session ID, asks to continue prior work from local history, or wants to inspect `.claude` files before resuming implementation", "source": "./", "strict": false, - "version": "1.1.0", + "version": "1.1.1", "category": "developer-tools", "keywords": [ "claude-code", @@ -898,6 +898,27 @@ "skills": [ "./continue-claude-work" ] + }, + { + "name": "scrapling-skill", + "description": "Install, troubleshoot, and use Scrapling CLI for extracting HTML, Markdown, or text from webpages. Diagnoses missing extras, Playwright browser runtime issues, TLS verification failures, and WeChat public article extraction patterns. Use when users mention Scrapling, `scrapling extract`, `uv tool install scrapling`, or need to decide between static and browser-backed fetching", + "source": "./", + "strict": false, + "version": "1.0.0", + "category": "developer-tools", + "keywords": [ + "scrapling", + "web-scraping", + "html", + "markdown", + "playwright", + "wechat", + "extraction", + "cli" + ], + "skills": [ + "./scrapling-skill" + ] } ] } diff --git a/CHANGELOG.md b/CHANGELOG.md index 74235cca..5e628655 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - None +## [1.39.0] - 2026-03-18 + +### Added +- **New Skill**: scrapling-skill v1.0.0 - Reliable Scrapling CLI installation, troubleshooting, and extraction workflows for HTML, Markdown, and text output + - Bundled `diagnose_scrapling.py` script to verify CLI health, detect missing extras, inspect Playwright browser runtime, and run real smoke tests + - Static-first workflow for choosing between `extract get`, `extract fetch`, and `stealthy-fetch` + - Verified WeChat public article extraction pattern using `#js_content` + - Verified recovery path for local TLS trust-store failures via `--no-verify` + - Bundled troubleshooting reference covering extras, browser runtime, and output validation + +### Changed +- **skill-creator** v1.5.0 → v1.5.1: Fixed `scripts/package_skill.py` so it works when invoked directly from the repository root instead of only via `python -m` +- **continue-claude-work** v1.1.0 → v1.1.1: Replaced newer Python-only type syntax in `extract_resume_context.py` so the script runs under the local `python3` environment +- Updated marketplace skills/plugins count from 42 to 43 +- Updated marketplace version from 1.38.0 to 1.39.0 +- Updated marketplace metadata description to include Scrapling CLI extraction workflows +- Updated README.md and README.zh-CN.md badges, installation commands, skill listings, use cases, quick links, and requirements +- Updated CLAUDE.md counts, version reference, and Available Skills list (added #43) + ## [1.38.0] - 2026-03-07 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 6359893f..222a437f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Repository Overview -This is a Claude Code skills marketplace containing 42 production-ready skills organized in a plugin marketplace structure. Each skill is a self-contained package that extends Claude's capabilities with specialized knowledge, workflows, and bundled resources. +This is a Claude Code skills marketplace containing 43 production-ready skills organized in a plugin marketplace structure. Each skill is a self-contained package that extends Claude's capabilities with specialized knowledge, workflows, and bundled resources. **Essential Skill**: `skill-creator` is the most important skill in this marketplace - it's a meta-skill that enables users to create their own skills. Always recommend it first for users interested in extending Claude Code. @@ -134,7 +134,7 @@ Skills for public distribution must NOT contain: ## Marketplace Configuration The marketplace is configured in `.claude-plugin/marketplace.json`: -- Contains 42 plugins, each mapping to one skill +- Contains 43 plugins, each mapping to one skill - Each plugin has: name, description, version, category, keywords, skills array - Marketplace metadata: name, owner, version, homepage @@ -144,7 +144,7 @@ The marketplace is configured in `.claude-plugin/marketplace.json`: 1. **Marketplace Version** (`.claude-plugin/marketplace.json` → `metadata.version`) - Tracks the marketplace catalog as a whole - - Current: v1.38.0 + - Current: v1.39.0 - Bump when: Adding/removing skills, major marketplace restructuring - Semantic versioning: MAJOR.MINOR.PATCH @@ -219,6 +219,7 @@ This applies when you change ANY file under a skill directory: 40. **excel-automation** - Create formatted Excel files, parse complex xlsm models, and control Excel windows on macOS via AppleScript 41. **capture-screen** - Programmatically capture macOS application windows using Swift window ID discovery and screencapture workflows 42. **continue-claude-work** - Recover local `.claude` session context via compact-boundary extraction, subagent workflow recovery, and session end reason detection, then continue interrupted work without `claude --resume` +43. **scrapling-skill** - Install, troubleshoot, and use Scrapling CLI for static/dynamic web extraction, WeChat article capture, and verified output validation **Recommendation**: Always suggest `skill-creator` first for users interested in creating skills or extending Claude Code. diff --git a/README.md b/README.md index 0c0e2eb3..7f4eeff3 100644 --- a/README.md +++ b/README.md @@ -6,15 +6,15 @@ [![简体中文](https://img.shields.io/badge/语言-简体中文-red)](./README.zh-CN.md) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Skills](https://img.shields.io/badge/skills-42-blue.svg)](https://github.com/daymade/claude-code-skills) -[![Version](https://img.shields.io/badge/version-1.38.0-green.svg)](https://github.com/daymade/claude-code-skills) +[![Skills](https://img.shields.io/badge/skills-43-blue.svg)](https://github.com/daymade/claude-code-skills) +[![Version](https://img.shields.io/badge/version-1.39.0-green.svg)](https://github.com/daymade/claude-code-skills) [![Claude Code](https://img.shields.io/badge/Claude%20Code-2.0.13+-purple.svg)](https://claude.com/code) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/daymade/claude-code-skills/graphs/commit-activity) -Professional Claude Code skills marketplace featuring 42 production-ready skills for enhanced development workflows. +Professional Claude Code skills marketplace featuring 43 production-ready skills for enhanced development workflows. ## 📑 Table of Contents @@ -240,6 +240,9 @@ claude plugin install capture-screen@daymade-skills # Resume interrupted Claude work from local session artifacts claude plugin install continue-claude-work@daymade-skills + +# Scrapling CLI extraction and troubleshooting +claude plugin install scrapling-skill@daymade-skills ``` Each skill can be installed independently - choose only what you need! @@ -1787,6 +1790,44 @@ claude plugin install continue-claude-work@daymade-skills --- +### 43. **scrapling-skill** - Reliable Scrapling CLI Workflows + +Install, troubleshoot, and use Scrapling CLI with a verified static-first workflow for extracting HTML, Markdown, or text from webpages. Includes a diagnostic script for broken extras installs, Playwright browser runtime checks, and smoke tests against real URLs. + +**When to use:** +- Users mention Scrapling, `uv tool install scrapling`, or `scrapling extract` +- You need to choose between static and browser-backed fetching +- You need to extract article bodies from WeChat public pages (`mp.weixin.qq.com`) +- A Scrapling install works partially but fails on missing extras, browser runtime, or TLS verification + +**Key features:** +- Bundled `diagnose_scrapling.py` script for CLI, browser runtime, and live URL smoke tests +- Verified default path: start with `extract get`, escalate to `extract fetch` only when needed +- WeChat extraction pattern using `#js_content` for clean article Markdown +- Troubleshooting guidance for missing `click`, Playwright runtime setup, and `curl: (60)` trust-store failures +- Output validation workflow using file size and content checks instead of exit-code assumptions + +**Example usage:** +```bash +# Install the skill +claude plugin install scrapling-skill@daymade-skills + +# Then ask Claude to work through Scrapling for you +"Install Scrapling CLI and verify the setup" +"Extract this WeChat article into Markdown with Scrapling" +"Decide whether this page needs static or browser-backed fetching" +``` + +**🎬 Live Demo** + +*Coming soon* + +📚 **Documentation**: See [scrapling-skill/SKILL.md](./scrapling-skill/SKILL.md) and [scrapling-skill/references/troubleshooting.md](./scrapling-skill/references/troubleshooting.md). + +**Requirements**: Python 3.6+, `uv`, Scrapling CLI, and Playwright browser runtime for browser-backed fetches. + +--- + ## 🎬 Interactive Demo Gallery Want to see all demos in one place with click-to-enlarge functionality? Check out our [interactive demo gallery](./demos/index.html) or browse the [demos directory](./demos/). @@ -1853,6 +1894,9 @@ Use **claude-code-history-files-finder** to recover deleted files from previous ### For Resuming Interrupted Claude Sessions Use **continue-claude-work** to recover the last actionable request from local `~/.claude` artifacts and continue implementation without reopening the original session. Combine with **claude-code-history-files-finder** when you need broader cross-session search, statistics, or deleted-file recovery. +### For Web Extraction & WeChat Articles +Use **scrapling-skill** to install and validate Scrapling CLI, choose between static and browser-backed fetching, and extract clean Markdown from sites like `mp.weixin.qq.com`. Combine with **deep-research** to turn extracted sources into structured reports or with **docs-cleaner** to normalize captured article content. + ### For Documentation Maintenance Use **docs-cleaner** to consolidate redundant documentation while preserving valuable content. Perfect for cleaning up documentation sprawl after rapid development phases or merging overlapping docs into authoritative sources. @@ -1941,6 +1985,7 @@ Each skill includes: - **excel-automation**: See `excel-automation/SKILL.md` for create/parse/control workflows and `excel-automation/references/formatting-reference.md` for formatting standards - **capture-screen**: See `capture-screen/SKILL.md` for CGWindowID-based screenshot workflows on macOS - **continue-claude-work**: See `continue-claude-work/SKILL.md` for local artifact recovery, drift checks, and resume workflow +- **scrapling-skill**: See `scrapling-skill/SKILL.md` for the CLI workflow and `scrapling-skill/references/troubleshooting.md` for verified Scrapling failure modes ## 🛠️ Requirements @@ -1967,6 +2012,7 @@ Each skill includes: - **uv + openpyxl** (for excel-automation): `uv run --with openpyxl ...` - **macOS** (for capture-screen and excel-automation AppleScript control workflows) - **Python 3.8+** (for continue-claude-work): bundled script for session extraction (no external dependencies) +- **uv + Scrapling CLI** (for scrapling-skill): `uv tool install 'scrapling[shell]'` and `scrapling install` for browser-backed fetches ## ❓ FAQ diff --git a/README.zh-CN.md b/README.zh-CN.md index 36e7b56e..021e8e5e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -6,15 +6,15 @@ [![简体中文](https://img.shields.io/badge/语言-简体中文-red)](./README.zh-CN.md) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Skills](https://img.shields.io/badge/skills-42-blue.svg)](https://github.com/daymade/claude-code-skills) -[![Version](https://img.shields.io/badge/version-1.38.0-green.svg)](https://github.com/daymade/claude-code-skills) +[![Skills](https://img.shields.io/badge/skills-43-blue.svg)](https://github.com/daymade/claude-code-skills) +[![Version](https://img.shields.io/badge/version-1.39.0-green.svg)](https://github.com/daymade/claude-code-skills) [![Claude Code](https://img.shields.io/badge/Claude%20Code-2.0.13+-purple.svg)](https://claude.com/code) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/daymade/claude-code-skills/graphs/commit-activity) -专业的 Claude Code 技能市场，提供 42 个生产就绪的技能，用于增强开发工作流。 +专业的 Claude Code 技能市场，提供 43 个生产就绪的技能，用于增强开发工作流。 ## 📑 目录 @@ -243,6 +243,9 @@ claude plugin install capture-screen@daymade-skills # 基于本地会话产物续做中断的 Claude 工作 claude plugin install continue-claude-work@daymade-skills + +# Scrapling CLI 抽取与故障排查 +claude plugin install scrapling-skill@daymade-skills ``` 每个技能都可以独立安装 - 只选择你需要的！ @@ -1829,6 +1832,44 @@ claude plugin install continue-claude-work@daymade-skills --- +### 43. **scrapling-skill** - 可靠的 Scrapling CLI 工作流 + +围绕 Scrapling CLI 提供经过验证的安装、排障与网页抽取工作流，用于从网页输出 HTML、Markdown 或纯文本。内置诊断脚本，可检查 extras 安装问题、Playwright 浏览器运行时，以及真实 URL 的烟测结果。 + +**使用场景：** +- 用户提到 Scrapling、`uv tool install scrapling` 或 `scrapling extract` +- 需要判断应该使用静态抓取还是浏览器抓取 +- 需要从微信公众号页面（`mp.weixin.qq.com`）提取正文 +- Scrapling 安装看似成功，但在 extras、浏览器运行时或 TLS 校验上失败 + +**主要功能：** +- 内置 `diagnose_scrapling.py`，检查 CLI、浏览器运行时与真实 URL 烟测 +- 经过验证的默认路径：先用 `extract get`，只有必要时再升级到 `extract fetch` +- 针对微信公众号文章的 `#js_content` 提取模式 +- 覆盖缺少 `click`、Playwright 运行时缺失、`curl: (60)` 证书问题等真实故障 +- 用文件大小和内容验证结果，而不是只看退出码 + +**示例用法：** +```bash +# 安装技能 +claude plugin install scrapling-skill@daymade-skills + +# 然后让 Claude 代你跑 Scrapling +"安装 Scrapling CLI 并验证配置" +"用 Scrapling 把这篇微信公众号文章提取成 Markdown" +"判断这个页面应不应该走浏览器抓取" +``` + +**🎬 实时演示** + +*即将推出* + +📚 **文档**：参见 [scrapling-skill/SKILL.md](./scrapling-skill/SKILL.md) 和 [scrapling-skill/references/troubleshooting.md](./scrapling-skill/references/troubleshooting.md)。 + +**要求**：Python 3.6+、`uv`、Scrapling CLI；如需浏览器抓取，还需要 Playwright 浏览器运行时。 + +--- + ## 🎬 交互式演示画廊 想要在一个地方查看所有演示并具有点击放大功能？访问我们的[交互式演示画廊](./demos/index.html)或浏览[演示目录](./demos/)。 @@ -1895,6 +1936,9 @@ claude plugin install continue-claude-work@daymade-skills ### 续做中断的 Claude 会话 使用 **continue-claude-work** 从本地 `~/.claude` 产物中恢复最后一个可执行请求，并在不重新打开原始会话的情况下继续实现。若还需要跨会话搜索、统计分析或恢复已删除文件，可与 **claude-code-history-files-finder** 配合使用。 +### 网页提取与微信公众号文章 +使用 **scrapling-skill** 安装并验证 Scrapling CLI，判断应使用静态抓取还是浏览器抓取，并从 `mp.weixin.qq.com` 等页面提取干净的 Markdown。可与 **deep-research** 配合，将抓取内容整理为结构化报告，或与 **docs-cleaner** 配合清理抽取后的文章内容。 + ### 文档维护 使用 **docs-cleaner** 在保留有价值内容的同时整合冗余文档。非常适合在快速开发阶段后清理文档扩散或将重叠的文档合并为权威来源。 @@ -1983,6 +2027,7 @@ claude plugin install continue-claude-work@daymade-skills - **excel-automation**：参见 `excel-automation/SKILL.md` 了解创建/解析/控制工作流，参见 `excel-automation/references/formatting-reference.md` 了解格式规范 - **capture-screen**：参见 `capture-screen/SKILL.md` 了解基于 CGWindowID 的 macOS 截图流程 - **continue-claude-work**：参见 `continue-claude-work/SKILL.md` 了解本地会话产物恢复、漂移检查与续做流程 +- **scrapling-skill**：参见 `scrapling-skill/SKILL.md` 了解 CLI 工作流，参见 `scrapling-skill/references/troubleshooting.md` 了解已验证的 Scrapling 故障模式 ## 🛠️ 系统要求 @@ -2006,6 +2051,7 @@ claude plugin install continue-claude-work@daymade-skills - **uv + openpyxl**（用于 excel-automation）：`uv run --with openpyxl ...` - **macOS**（用于 capture-screen 与 excel-automation 的 AppleScript 控制流程） - **Python 3.8+**（用于 continue-claude-work）：内置脚本进行会话提取（无外部依赖） +- **uv + Scrapling CLI**（用于 scrapling-skill）：`uv tool install 'scrapling[shell]'`，浏览器抓取前运行 `scrapling install` ## ❓ 常见问题 diff --git a/continue-claude-work/.security-scan-passed b/continue-claude-work/.security-scan-passed index b423c462..4e1d205b 100644 --- a/continue-claude-work/.security-scan-passed +++ b/continue-claude-work/.security-scan-passed @@ -1,4 +1,4 @@ Security scan passed -Scanned at: 2026-03-07T14:27:12.638956 +Scanned at: 2026-03-18T23:02:18.627209 Tool: gitleaks + pattern-based validation -Content hash: c464aa735e8b7832c2c77e4cea22fff9c7e6117ecee4f6769f5eb62cced8a11a +Content hash: 62e456422cabfe74e5757a802044dac45d1662341b2e90dc943685db81d8f659 diff --git a/continue-claude-work/scripts/extract_resume_context.py b/continue-claude-work/scripts/extract_resume_context.py index 7c602247..fad677bc 100755 --- a/continue-claude-work/scripts/extract_resume_context.py +++ b/continue-claude-work/scripts/extract_resume_context.py @@ -37,6 +37,7 @@ import sys import time from pathlib import Path +from typing import Dict, List, Optional CLAUDE_DIR = Path.home() / ".claude" PROJECTS_DIR = CLAUDE_DIR / "projects" @@ -59,7 +60,7 @@ def normalize_path(project_path: str) -> str: return project_path.replace("/", "-") -def find_project_dir(project_path: str) -> Path | None: +def find_project_dir(project_path: str) -> Optional[Path]: """Find the Claude projects directory for a given project path.""" abs_path = os.path.abspath(project_path) @@ -85,7 +86,7 @@ def find_project_dir(project_path: str) -> Path | None: return None -def load_sessions_index(project_dir: Path) -> list[dict]: +def load_sessions_index(project_dir: Path) -> List[Dict]: """Load and parse sessions-index.json, sorted by modified desc.""" index_file = project_dir / "sessions-index.json" if not index_file.exists(): @@ -97,7 +98,7 @@ def load_sessions_index(project_dir: Path) -> list[dict]: return entries -def search_sessions(entries: list[dict], query: str) -> list[dict]: +def search_sessions(entries: List[Dict], query: str) -> List[Dict]: """Search sessions by keyword in firstPrompt and summary.""" query_lower = query.lower() results = [] @@ -109,7 +110,7 @@ def search_sessions(entries: list[dict], query: str) -> list[dict]: return results -def format_session_entry(entry: dict, file_exists: bool = True) -> str: +def format_session_entry(entry: Dict, file_exists: bool = True) -> str: """Format a session index entry for display.""" sid = entry.get("sessionId", "?") modified = entry.get("modified", "?") @@ -123,7 +124,7 @@ def format_session_entry(entry: dict, file_exists: bool = True) -> str: # ── Session file parsing ──────────────────────────────────────────── -def parse_session_structure(session_file: Path) -> dict: +def parse_session_structure(session_file: Path) -> Dict: """Parse a session JSONL file and return structured data.""" file_size = session_file.stat().st_size total_lines = 0 @@ -276,8 +277,8 @@ def parse_session_structure(session_file: Path) -> dict: def _detect_end_reason( - last_role: str | None, - unresolved: dict, + last_role: Optional[str], + unresolved: Dict, error_count: int, ) -> str: """Detect why the session ended.""" @@ -300,7 +301,7 @@ def _is_noise_user_text(text: str) -> bool: return False -def extract_user_text(messages: list[dict], limit: int = 5) -> list[str]: +def extract_user_text(messages: List[Dict], limit: int = 5) -> List[str]: """Extract the last N user text messages (not tool results or system noise).""" user_texts = [] for msg_obj in reversed(messages): @@ -329,7 +330,7 @@ def extract_user_text(messages: list[dict], limit: int = 5) -> list[str]: return user_texts -def extract_assistant_text(messages: list[dict], limit: int = 3) -> list[str]: +def extract_assistant_text(messages: List[Dict], limit: int = 3) -> List[str]: """Extract the last N assistant text responses (no thinking/tool_use).""" assistant_texts = [] for msg_obj in reversed(messages): @@ -355,7 +356,7 @@ def extract_assistant_text(messages: list[dict], limit: int = 3) -> list[str]: # ── Subagent extraction ────────────────────────────────────────────── -def extract_subagent_context(session_file: Path) -> list[dict]: +def extract_subagent_context(session_file: Path) -> List[Dict]: """Extract subagent summaries from session subdirectories. Returns list of {name, type, status, last_text, is_interrupted}. @@ -457,7 +458,8 @@ def get_git_state(project_path: str) -> str: try: branch = subprocess.run( ["git", "branch", "--show-current"], - capture_output=True, text=True, cwd=project_path, timeout=5, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True, cwd=project_path, timeout=5, ) if branch.stdout.strip(): parts.append(f"**Current branch**: `{branch.stdout.strip()}`") @@ -467,7 +469,8 @@ def get_git_state(project_path: str) -> str: try: status = subprocess.run( ["git", "status", "--short"], - capture_output=True, text=True, cwd=project_path, timeout=10, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True, cwd=project_path, timeout=10, ) if status.stdout.strip(): parts.append(f"### git status\n```\n{status.stdout.strip()}\n```") @@ -479,7 +482,8 @@ def get_git_state(project_path: str) -> str: try: log = subprocess.run( ["git", "log", "--oneline", "-5"], - capture_output=True, text=True, cwd=project_path, timeout=10, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, + universal_newlines=True, cwd=project_path, timeout=10, ) if log.stdout.strip(): parts.append(f"### git log (last 5)\n```\n{log.stdout.strip()}\n```") @@ -489,7 +493,7 @@ def get_git_state(project_path: str) -> str: return "\n\n".join(parts) -def get_memory_md(project_dir: Path) -> str | None: +def get_memory_md(project_dir: Path) -> Optional[str]: """Read MEMORY.md if it exists in the project's memory directory.""" memory_dir = project_dir / "memory" memory_file = memory_dir / "MEMORY.md" @@ -500,7 +504,7 @@ def get_memory_md(project_dir: Path) -> str | None: return None -def get_session_memory(session_file: Path) -> str | None: +def get_session_memory(session_file: Path) -> Optional[str]: """Read session-memory/summary.md if it exists (newer CC versions).""" session_dir = session_file.parent / session_file.stem summary = session_dir / "session-memory" / "summary.md" @@ -524,8 +528,8 @@ def get_session_memory(session_file: Path) -> str | None: def build_briefing( - session_entry: dict | None, - parsed: dict, + session_entry: Optional[Dict], + parsed: Dict, project_path: str, project_dir: Path, session_file: Path, @@ -658,7 +662,7 @@ def build_briefing( # ── CLI ────────────────────────────────────────────────────────────── -def _check_session_files(entries: list[dict], project_dir: Path) -> dict[str, bool]: +def _check_session_files(entries: List[Dict], project_dir: Path) -> Dict[str, bool]: """Check which index entries have actual files on disk.""" status = {} for entry in entries: diff --git a/scrapling-skill.skill b/scrapling-skill.skill new file mode 100644 index 0000000000000000000000000000000000000000..6ad2ba62b5146aba323a624b2fa4a51b3519e868 GIT binary patch literal 6223 zcmaKxbx>T*nuiB>NpN=?++on*?jGEoLBgORxO>n50W!Gz;2JzYkPw190fG(|Z1e5b z-h1obZ?{i%ovKsSPgVCH?|FW2YpEh369NDLbO5B)%k+%XNVy&z0H7uY07(CQ^|tY{ z0=qi7J8*itIJvrV>#8cNsd2g4))<+2tiXtZ7wc#(3*AG%idd{hzgm_*bQM|yDQ$TA z(#-a4DN)8ZFajBO&p9r(15cPrHWSP|bbK1{&Xu~lKJMpLiBC6av^vw7I@zrsoPZ$0 z9ZroZnFDK5cMI@vM*3jSV*KlgK2m6AWV}J6OA^vkN&dOt|dsZ6Uednuzrq#;ezW-c+5r*NpZM)f|q#XzO%h zj@u+Q%uzVlSm6po#4D3husGZHIBd>*g8tg4C^_|EfnIgU3Gg>G+K0zqWN>)sF^zIJ z81z)%0b#3b`l8xZr6d`^H}aW=;~wu1k!z!=rW9J_h0ag?*n$ciM7{E;n3PSt!kRuY zoa1V*Ik${8;srXv1uufkuzpdM>4B!?OTKwIimzN=nQ}-xkWl<`d3#{O_)7G_`WT0d zO?`QwEC-g6SPM1W6!eb=eA&3CGnn!DVo>;_aT79W#WyStHVaF{=aItC(AMx7-tL)F zrCO_35u%HSvsVgFHyTD+q)*}0Kk{j(GywM)G){@tyE>l zuHzqd@G5Oo{A??V9XH7kPE=rK?VuYTFk)ugKdrfP3HweO1@qqL)z+ z)PrA^%_D3_jy5v21uf(5HHu+!CRbItN*~qxGz)p^!0$~`m!6K8>t)x_bk$>ICI(7e z56V0UT%$lRx=Jg8Un zlPVpcqqZKQ)yv!O`9lPU&gAF*y*OOTw^OT;>d1S7!fQ?S7Q44~U5(Y-EGvA&| zv2U)^yA0yuSfRR{!aNjJ>c1k+W%q0%o3Oc=zo+j1&N=%%yMR6K=XD9bH1G~5>`%kp^fpVTNd27+qipFRO2HO#~ z4UDeG!3=E_Rdp^>Q#vnOI)$?+m8!Q$J!~d%M9|J&>GO0y@Uf29RLfZ{m>4f)jORo1 z6j#3)pcK-WL5HpiVY%Kfy!C{1+6Hj!lz-Cpk{4;w*kP#01->PM)e3*p4Pz{U8Jb19 zv|y%AM6E2HkSC3zXJ+G^B;qQN&dq@u_#9((QM`sG=$|i&=sh4TF#D`l^NH`2l4a!f|_|R==UF z@9tiRWiiYR_Fm1z%t^zIuN#*}YX~e*k@}v%o*L3je(x1|0S#}InTDR>W$Rc%I@*4~?^;Py)!s#n#0!{GqI>$xy1}W%1Lg=4pv! z?6#kUEVMY1K41h9&1{Na5le;~$<`e|4i0po98eqZ%9WnY+wqgzivS9hj7|(yOcovZ zvm||-G8g};3kiqRr79?V(3WXh!9G(r48J2hV#B*e${*s@$sBR0LhiCvC)0L?Uf}AD&x8P`D3ze zQJ`)HV1=6HqBL9T_{hZ&JK)EA{j6$QCtKI_C*@t8Z}eAM2Rw`FjN3CU5bas2Dc+01?D;$Do!xsSeBbY>2ty*N9lNn- zl}o6n&v)-iDRqBdQ?T-hw9U{5f=Eb!G%SH-q_Q5ta|(Y>YR}TXb_=4?a4c%3nLl)+ ztfV*F&T3*)NSX4nk8COa&Rxnz|0zW-R_w#v+cME12LaLXB#qeu2R>)?b4j>rUY}F+ zcE39gavaMhoibg}ejIA;pqk`ljM&=!sW5f3b>YD2<;HrB2z-k|nLRIWCa}(7doVob zt$22HYyPvWKp>DoJb4|5+4<2wrL}Q>k8yx$&5#l%$Penz_+vLMesz=i_=aQ@A{yzK1lyzJasygc4myV`j>dU*K!(YwEt&Tu(tSqMLP(Ga^H z&plpcidmk9xV!NDg#NXNQoZkNbZfn%vulHDDyH%hJMBRmL`j=OjDnW)cPsH&d!DGb z{WYS&O1%kYOLWWEF9yY&ExvFuWiBquq72zx*EA@&tlCD0kEJNGw#7aBN;Vw+C08_B;cg`3tu)w?3T&pY zMkEyB{c_-FgU#RJr+sX_(V9ON6A#5(5|~Y*kD+qf{$1u0pA+szb0#f_ou?`Iq<8sc z$%FS^9rIWVY6;a+gL-7jM=1*PU0RO9*)Kg2`9gGI!;KR5^`(GO79F{V&>rc}Kli(nca4oBLU-4o_u3t?4j6P3fHKcG;e&U-K%zKF*Hbxsa^I!HKy z`y$QyI4u~o%q?WpIY}}iIjNA&`?Es1H$Dt6z$=R{bXkdkRyDm@-fXwBl56$VEd=EH zIQnk!4hYz5r>RFsx!18B0*3@Ky)b*p%YCvgf!~f_p|8VM(`R*Y6LGEtM&qtI9!Gml zta0^*Hq+mXLfMxE3Q$a0d8cwmVQpzGiVW9GOYZ1Q^zjF#IPHO4c%$#D#W>;G%3m-s z!P?|zv85LUgX`#ZtZIicVe|VpjFvWEi$CFp>kqbvIDRfDx0%MGJl&Iil=a~^a@pqC zIhx^M`26lgQ8qR>G3Pc+(I=E&WW`Z&vP56uJMmnQ|=no`P_bgc?gXex9YFRe6k=ry3 zTscdR0&g>@NWZGhn@a_kn!ji9gY(|K7*5Fl*1I^=$f;I-Y7E;)6dTGP!TW}x^YE5h zJq7dCY=IXeSB7~$p}fpv->8*}vmDT@F?@y@pN)%Lc)R#qh>LuzhUB4gDv6aka$GWK zw4B|m+zF3jSEIby-JG5{GaSKU2`-?ryJd-e9b=I{`LEK0pj!VKxkyQ;9&N2rKg zVr&+XT6abI7?02qdhJ{JX@y_lrQN}cc`53f zehriXX4v(I4eLe>^FCF3HXx#VT{Dp{a<#kDsy6on?yKsSOFtB0q%`BzBvo10v@vq$ z=-o>T@)DxBy&)rt}D^e{Q zGlNvtFl@?p=4Qep#-N)N^Uam|%V7xet3p_cUM|(<8MfD-Sh|U*!2P-RO(tsS4+6sB z&9*r|#n7xGmxpMSK9rK!PzZjvQwY4g-dp@Cx%m6yX(o2sl!Cb`>68w6g+uCRC=JFT zXB}i>JzK!40o&xqE6A&*_K+%lq*`=U^M(f^IC{@dK~i%va~o4E_Uauh_`A!=zy;4! z3`GMi6MK&&5sN~8l^(h@G*LUUsSUghj~M+0bT>@wa%1SbRcpCmS{)rzeAmtJmi&@{L<~r)A(te|pKHeAP93U=8UG-iN|8jW;G7s+2 z=Tw2w!}Ex`!kn9i}@*>OoMt!Ot_r!f=*1l{ z%X$J}eEDcFzZ_QjOPab%Y!B2StcToJMn${)+G;XYnBjX&w0wkdBMJCy^(@CoOSq;- z#bE`+U|4s;C)&?|OT_HY3kn{UT6~@rWmKMU<^3k;H_sp+fu4+G8I+XMmUFHvt zT*f|)=lb}L)Yy+~=e5l9G3&!+qe>qI=kMY?xwg$%+jho9lI|A60O4zc`j_WmTicy? zGVA9jq~zrJSpqjxcB`S~I1R6H%WVb2XCnvBg(gdWc;nn5GRcd{n=D|br6;v?(bdsr z9IjR=j!svXFIoCw0?)R_4#xtGbVt-^ zzs2S0FJZ)vc&jOp%#MTva6A{1Y0_sC=9ok?77Vc~0=lNuhw1?@iCYd69&cjtqWRd0 z=}Mu}kk~MU@iY=?)*dMaFjfeOJ7$*{vGM4*SmPcTmxYe2Nw5l{g+bwN*2^&m@0Y^r zsROUDi(6WC=9S_Iqgrt;m0j+AqL7v9CsmPz4N-oAJ1Y3$LK&b%Iu7_dN1tPQ5eB-9 zcDV3&fu_Oaq0>#)!S#+C=%ueFdHJ5o%Vu5<+Xcc39rKbW>k%UQ1=tR{-0+9@uMxU@ zm<9bLx&5N^G!hQ2V4ujpiza1nR5MW>+g~HqI@UBB>wK{)uju!VAZv_#qPz1u?1K`S z4bqe%55}vUyE|G@Hg}z2vv{l25AmkbWU(xHD#Z1+?72c(x83Ag+i~40Cb=+9_;fez zO5fdM!GzYV_fCs2vZ*J%oMLjGeTWq-ao$>lb+)x;2%w^Tp2o@QtAKwEOP~xB7$bM3 zRH+w_^OMb8_l}?D(F~>I(XqD~pCsG#r+*#}ZhFIH(Pu}vx>u94d2V_#gY1OqBQY#yr~^Sg3_ zL017Hne}?L)~5-LApQOtHCJ34r|MH}a`dD5NrItl#jcyVIg9~W+h?8P!9d5mT>?@a z7HM#VXCX0zfRu;jaXKvv%01cfC`8Sye~|@eMqKJUU?(qiTz05z-CS>#R%~ov~7XJAhZK4 zco-Z$m`6+XsT+U5wL7{$&p-MxbL`3YHUi!GCTWj`vS9Aa=mq!ec@&xK37;>iKT$sy z8jqJxx_>HH#r#~c1Ux9#B5)Sd7Dxd#8(<6!4z65a;RbZ-u|o*~*Oe{_@fJg{=PJ|k zqMWKCZoPB8-W%D@s9bHQY%leNqeZ@M7t6`sM*IoDFL7ADd9gj`yA2itcDmuAT%H+Y-h4S~piQtU?+5jYne#m*4$E zM!w<6RASco=|}1k*8rP)E6Yh7a8W-%IlBzU$3nDJ5fBNH{_kG*pTP9r0~-3*>mPsG zf9rhzE9LL9|js_content|rich_media_title|main' page.html +``` + +If the file is tiny, empty, or missing the expected container, the extraction did not succeed. Go back to Step 3 and switch fetchers or selectors. + +## Step 6: Handle Known Failure Modes + +### Local TLS trust store problem + +If `extract get` fails with `curl: (60) SSL certificate problem`, treat it as a local trust-store problem first, not a Scrapling content failure. + +Retry the same command with: + +```bash +--no-verify +``` + +Only do this after confirming the failure matches the local certificate verification error pattern. Do not silently disable verification by default. + +### WeChat article pages + +For `mp.weixin.qq.com`: +- Try `extract get` before `extract fetch` +- Use `-s '#js_content'` for the article body +- Validate the saved Markdown or HTML immediately + +### Browser-backed fetch failures + +If `extract fetch` fails: +1. Re-check the install with `python3 scripts/diagnose_scrapling.py` +2. Confirm Chromium and Chrome Headless Shell are present +3. Retry with a slightly longer timeout +4. Escalate to `stealthy-fetch` only if the site behavior justifies it + +## Command Patterns + +### Diagnose and smoke test a URL + +```bash +python3 scripts/diagnose_scrapling.py --url 'https://example.com' +``` + +### Diagnose and smoke test a WeChat article body + +```bash +python3 scripts/diagnose_scrapling.py \ + --url 'https://mp.weixin.qq.com/s/ARTICLE_ID?scene=1' \ + --selector '#js_content' \ + --no-verify +``` + +### Diagnose and smoke test a browser-backed fetch + +```bash +python3 scripts/diagnose_scrapling.py \ + --url 'https://example.com' \ + --dynamic +``` + +## Guardrails + +- Do not tell the user to reinstall blindly. Verify first. +- Do not default to the Python library API when the user is clearly asking about the CLI. +- Do not jump to browser-backed fetching unless the static result is missing the real content. +- Do not claim success from exit code alone. Inspect the saved file. +- Do not hardcode user-specific absolute paths into outputs or docs. + +## Resources + +- Installation and smoke test helper: `scripts/diagnose_scrapling.py` +- Verified failure modes and recovery paths: `references/troubleshooting.md` diff --git a/scrapling-skill/references/troubleshooting.md b/scrapling-skill/references/troubleshooting.md new file mode 100644 index 00000000..891f18cf --- /dev/null +++ b/scrapling-skill/references/troubleshooting.md @@ -0,0 +1,164 @@ +# Scrapling Troubleshooting + +## Contents + +- Installation modes +- Verified failure modes +- Static vs dynamic fetch choice +- WeChat extraction pattern +- Smoke test commands + +## Installation Modes + +Use the CLI path as the default: + +```bash +uv tool install 'scrapling[shell]' +``` + +Do not assume `uv tool install scrapling` is enough for CLI usage. The base package may install the executable wrapper without the optional CLI dependencies. + +## Verified Failure Modes + +### 1. CLI installed without extras + +Symptom: + +- `scrapling --help` fails +- Output mentions missing `click` +- Output says Scrapling must be installed with extras + +Recovery: + +```bash +uv tool uninstall scrapling +uv tool install 'scrapling[shell]' +``` + +### 2. Browser-backed fetchers not ready + +Symptom: + +- `extract fetch` or `extract stealthy-fetch` fails because the Playwright runtime is not installed +- Scrapling has not downloaded Chromium or Chrome Headless Shell + +Recovery: + +```bash +scrapling install +``` + +Success signals: + +- `scrapling install` later reports `The dependencies are already installed` +- Browser caches contain both: + - `chromium-*` + - `chromium_headless_shell-*` + +Typical cache roots: + +- `~/Library/Caches/ms-playwright/` +- `~/.cache/ms-playwright/` + +### 3. Static fetch TLS trust-store failure + +Symptom: + +- `extract get` fails with `curl: (60) SSL certificate problem` + +Interpretation: + +- Treat this as a local certificate verification problem first +- Do not assume the target URL or Scrapling itself is broken + +Recovery: + +Retry the same static command with: + +```bash +--no-verify +``` + +Do not make `--no-verify` the default. Use it only after the failure matches this certificate-verification pattern. + +## Static vs Dynamic Fetch Choice + +Use this order: + +1. `extract get` +2. `extract fetch` +3. `extract stealthy-fetch` + +Use `extract get` when: + +- The page is mostly server-rendered +- The content is likely already present in raw HTML +- The target is an article page with a stable content container + +Use `extract fetch` when: + +- Static HTML does not contain the real content +- The site depends on JavaScript rendering +- The page content appears only after runtime hydration + +Use `extract stealthy-fetch` when: + +- `fetch` still fails +- The target site shows challenge or anti-bot behavior + +## WeChat Extraction Pattern + +For `mp.weixin.qq.com` public article pages: + +- Start with `extract get` +- Use the selector `#js_content` +- Validate the saved file immediately + +Example: + +```bash +scrapling extract get 'https://mp.weixin.qq.com/s/ARTICLE_ID?scene=1' article.md -s '#js_content' +``` + +Observed behavior: + +- The static fetch can already contain the real article body +- Browser-backed fetch is often unnecessary for article extraction + +## Smoke Test Commands + +### Basic diagnosis + +```bash +python3 scripts/diagnose_scrapling.py +``` + +### Static extraction smoke test + +```bash +python3 scripts/diagnose_scrapling.py --url 'https://example.com' +``` + +### WeChat article smoke test + +```bash +python3 scripts/diagnose_scrapling.py \ + --url 'https://mp.weixin.qq.com/s/ARTICLE_ID?scene=1' \ + --selector '#js_content' +``` + +### Dynamic extraction smoke test + +```bash +python3 scripts/diagnose_scrapling.py \ + --url 'https://example.com' \ + --dynamic +``` + +### Validate saved output + +```bash +wc -c article.md +sed -n '1,40p' article.md +rg -n '|js_content|main|rich_media_title' page.html +``` diff --git a/scrapling-skill/scripts/diagnose_scrapling.py b/scrapling-skill/scripts/diagnose_scrapling.py new file mode 100755 index 00000000..e6fa29c6 --- /dev/null +++ b/scrapling-skill/scripts/diagnose_scrapling.py @@ -0,0 +1,191 @@ +#!/usr/bin/env python3 +""" +Diagnose a local Scrapling CLI installation and optionally run a smoke test. +""" + +import argparse +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path +from typing import Iterable, List, Tuple + + +def run_command(cmd: List[str]) -> Tuple[int, str, str]: + result = subprocess.run( + cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + universal_newlines=True, + check=False, + ) + return result.returncode, result.stdout, result.stderr + + +def print_section(title: str) -> None: + print("") + print(title) + print("-" * len(title)) + + +def existing_dirs(paths: Iterable[Path]) -> List[str]: + return [str(path) for path in paths if path.exists()] + + +def detect_browser_cache() -> Tuple[List[str], List[str]]: + roots = [ + Path.home() / "Library" / "Caches" / "ms-playwright", + Path.home() / ".cache" / "ms-playwright", + ] + chromium = [] + headless_shell = [] + for root in roots: + if not root.exists(): + continue + chromium.extend(existing_dirs(sorted(root.glob("chromium-*")))) + headless_shell.extend(existing_dirs(sorted(root.glob("chromium_headless_shell-*")))) + return chromium, headless_shell + + +def diagnose_cli() -> bool: + print_section("CLI") + scrapling_path = shutil.which("scrapling") + if not scrapling_path: + print("status: missing") + print("fix: install with `uv tool install 'scrapling[shell]'`") + return False + + print("path: {0}".format(scrapling_path)) + code, stdout, stderr = run_command(["scrapling", "--help"]) + output = (stdout + "\n" + stderr).strip() + + if code == 0: + print("status: working") + return True + + print("status: broken") + if "install scrapling with any of the extras" in output.lower() or "no module named 'click'" in output.lower(): + print("cause: installed without CLI extras") + print("fix: `uv tool uninstall scrapling` then `uv tool install 'scrapling[shell]'`") + else: + print("cause: unknown") + + if output: + print("details:") + print(output[:1200]) + return False + + +def diagnose_browsers() -> None: + print_section("Browser Runtime") + chromium, headless_shell = detect_browser_cache() + print("chromium: {0}".format("present" if chromium else "missing")) + for path in chromium: + print(" - {0}".format(path)) + print("chrome-headless-shell: {0}".format("present" if headless_shell else "missing")) + for path in headless_shell: + print(" - {0}".format(path)) + if not chromium or not headless_shell: + print("hint: run `scrapling install` before browser-backed fetches") + + +def preview_file(path: Path, preview_lines: int) -> None: + print_section("Smoke Test Output") + if not path.exists(): + print("status: missing output file") + return + + size = path.stat().st_size + print("path: {0}".format(path)) + print("bytes: {0}".format(size)) + if size == 0: + print("status: empty") + return + + if path.suffix in (".md", ".txt"): + print("preview:") + with path.open("r", encoding="utf-8", errors="replace") as handle: + for index, line in enumerate(handle): + if index >= preview_lines: + break + print(line.rstrip()) + + +def run_smoke_test(args: argparse.Namespace) -> int: + print_section("Smoke Test") + + suffix = ".html" + if args.selector: + suffix = ".md" + + output_path = Path(tempfile.gettempdir()) / ("scrapling-smoke" + suffix) + if output_path.exists(): + output_path.unlink() + + cmd = ["scrapling", "extract", "fetch" if args.dynamic else "get", args.url, str(output_path)] + if args.selector: + cmd.extend(["-s", args.selector]) + if args.dynamic: + cmd.extend(["--timeout", str(args.timeout)]) + elif args.no_verify: + cmd.append("--no-verify") + + print("command: {0}".format(" ".join(cmd))) + code, stdout, stderr = run_command(cmd) + if stdout.strip(): + print(stdout.strip()) + if stderr.strip(): + print(stderr.strip()) + + preview_file(output_path, args.preview_lines) + return code + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Diagnose Scrapling and run an optional smoke test.") + parser.add_argument("--url", help="Optional URL for a smoke test") + parser.add_argument("--selector", help="Optional CSS selector for the smoke test") + parser.add_argument( + "--dynamic", + action="store_true", + help="Use `scrapling extract fetch` instead of `scrapling extract get`", + ) + parser.add_argument( + "--no-verify", + action="store_true", + help="Pass `--no-verify` to static smoke tests", + ) + parser.add_argument( + "--timeout", + type=int, + default=20000, + help="Timeout in milliseconds for dynamic smoke tests", + ) + parser.add_argument( + "--preview-lines", + type=int, + default=20, + help="Number of preview lines for markdown/text output", + ) + return parser + + +def main() -> int: + parser = build_parser() + args = parser.parse_args() + + cli_ok = diagnose_cli() + diagnose_browsers() + + if not cli_ok: + return 1 + + if not args.url: + return 0 + + return run_smoke_test(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skill-creator/scripts/package_skill.py b/skill-creator/scripts/package_skill.py index ceab59b3..bd10fe28 100755 --- a/skill-creator/scripts/package_skill.py +++ b/skill-creator/scripts/package_skill.py @@ -15,6 +15,13 @@ import sys import zipfile from pathlib import Path +from typing import Optional, Tuple + +SCRIPT_DIR = Path(__file__).resolve().parent +PACKAGE_ROOT = SCRIPT_DIR.parent +if str(PACKAGE_ROOT) not in sys.path: + sys.path.insert(0, str(PACKAGE_ROOT)) + from scripts.quick_validate import validate_skill from scripts.security_scan import calculate_skill_hash @@ -41,7 +48,7 @@ def should_exclude(rel_path: Path) -> bool: return any(fnmatch.fnmatch(name, pat) for pat in EXCLUDE_GLOBS) -def validate_security_marker(skill_path: Path) -> tuple[bool, str]: +def validate_security_marker(skill_path: Path) -> Tuple[bool, str]: """ Validate security marker file exists and hash matches current content From d4634cb00b975404a550ec927412fa20f4e79070 Mon Sep 17 00:00:00 2001 From: daymade <daymadev89@gmail.com> Date: Fri, 20 Mar 2026 03:41:27 +0800 Subject: [PATCH 009/137] security: remove leaked API key from security.py docstring examples Replace real Zhipu GLM API key with fake placeholder in mask_secret() and SecretStr docstring examples. The real key was exposed in this PUBLIC repo. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- transcript-fixer/scripts/utils/security.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/transcript-fixer/scripts/utils/security.py b/transcript-fixer/scripts/utils/security.py index 69e6185c..b98cfb9b 100644 --- a/transcript-fixer/scripts/utils/security.py +++ b/transcript-fixer/scripts/utils/security.py @@ -43,7 +43,7 @@ def mask_secret(secret: str, visible_chars: int = 4) -> str: Masked string like "7fb3...DPRR" Examples: - >>> mask_secret("7fb3ab7b186242288fe93a27227b7149.bJCOEAsUfejvWDPR") + >>> mask_secret("example-fake-api-key-1234567890abcdef.test") '7fb3...DPRR' >>> mask_secret("short") @@ -248,7 +248,7 @@ class SecretStr: Wrapper for secrets that prevents accidental logging. Usage: - api_key = SecretStr("7fb3ab7b186242288fe93a27227b7149.bJCOEAsUfejvWDPR") + api_key = SecretStr("example-fake-api-key-1234567890abcdef.test") print(api_key) # Prints: SecretStr(7fb3...DPRR) print(api_key.get()) # Get actual value when needed From a496c91cae1f2068ede6ee34c64e0dfe591c3a12 Mon Sep 17 00:00:00 2001 From: daymade <daymadev89@gmail.com> Date: Sat, 21 Mar 2026 15:56:38 +0800 Subject: [PATCH 010/137] fix: prevent dictionary false positives + add tunnel-doctor WSL/Go findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit transcript-fixer: - Add common_words.py safety system (blocks common Chinese words from dictionary) - Add --audit command to scan existing dictionary for risky rules - Add --force flag to override safety checks explicitly - Fix substring corruption (产线数据→产线束据, 现金流→现现金流) - Unified position-aware replacement with _already_corrected() check - 69 tests covering all production false positive scenarios tunnel-doctor: - Add Step 5A: Tailscale SSH proxy silent failure on WSL - Add Step 5B: App Store vs Standalone Tailscale on macOS - Add Go net/http NO_PROXY CIDR incompatibility warning - Add utun interface identification (MTU 1280=Tailscale, 4064=Shadowrocket) - Fix "Four→Five Conflict Layers" inconsistency in reference doc - Add complete working Shadowrocket config reference Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- transcript-fixer/SKILL.md | 40 ++ transcript-fixer/scripts/cli/__init__.py | 2 + .../scripts/cli/argument_parser.py | 12 + transcript-fixer/scripts/cli/commands.py | 77 +- .../scripts/core/correction_service.py | 70 +- .../scripts/core/dictionary_processor.py | 200 +++++- transcript-fixer/scripts/fix_transcription.py | 3 + .../scripts/tests/test_common_words_safety.py | 675 ++++++++++++++++++ .../scripts/utils/common_words.py | 311 ++++++++ tunnel-doctor/SKILL.md | 117 ++- .../references/proxy_conflict_reference.md | 55 +- tunnel-doctor/scripts/quick_diagnose.py | 78 +- 12 files changed, 1596 insertions(+), 44 deletions(-) create mode 100644 transcript-fixer/scripts/tests/test_common_words_safety.py create mode 100644 transcript-fixer/scripts/utils/common_words.py diff --git a/transcript-fixer/SKILL.md b/transcript-fixer/SKILL.md index 5785a47b..e5f532d3 100644 --- a/transcript-fixer/SKILL.md +++ b/transcript-fixer/SKILL.md @@ -142,6 +142,46 @@ Do **not** save one-off deletions, ambiguous context-only rewrites, or section-s See `references/iteration_workflow.md` for complete iteration guide with checklist. +## FALSE POSITIVE RISKS -- READ BEFORE ADDING CORRECTIONS + +Dictionary-based corrections are powerful but dangerous. Adding the wrong rule silently corrupts every future transcript. The `--add` command runs safety checks automatically, but you must understand the risks. + +### What is safe to add + +- **ASR-specific gibberish**: "巨升智能" -> "具身智能" (no real word sounds like "巨升智能") +- **Long compound errors**: "语音是别" -> "语音识别" (4+ chars, unlikely to collide) +- **English transliteration errors**: "japanese 3 pro" -> "Gemini 3 Pro" + +### What is NEVER safe to add + +- **Common Chinese words**: "仿佛", "正面", "犹豫", "传说", "增加", "教育" -- these appear correctly in normal text. Replacing them corrupts transcripts from better ASR models. +- **Words <=2 characters**: Almost any 2-char Chinese string is a valid word or part of one. "线数" inside "产线数据" becomes "产线束据". +- **Both sides are real words**: "仿佛->反复", "犹豫->抑郁" -- both forms are valid Chinese. The "error" is only an error for one specific ASR model. + +### When in doubt, use a context rule instead + +Context rules use regex patterns that match only in specific surroundings, avoiding false positives: +```bash +# Instead of: --add "线数" "线束" +# Use a context rule in the database: +sqlite3 ~/.transcript-fixer/corrections.db "INSERT INTO context_rules (pattern, replacement, description, priority) VALUES ('(?<!产)线数(?!据)', '线束', 'ASR: 线数->线束 (not inside 产线数据)', 10);" +``` + +### Auditing the dictionary + +Run `--audit` periodically to scan all rules for false positive risks: +```bash +uv run scripts/fix_transcription.py --audit +uv run scripts/fix_transcription.py --audit --domain manufacturing +``` + +### Forcing a risky addition + +If you understand the risks and still want to add a flagged rule: +```bash +uv run scripts/fix_transcription.py --add "仿佛" "反复" --domain general --force +``` + ## AI Fallback Strategy When GLM API is unavailable (503, network issues), the script outputs `[CLAUDE_FALLBACK]` marker. diff --git a/transcript-fixer/scripts/cli/__init__.py b/transcript-fixer/scripts/cli/__init__.py index 6e6e5b0c..ca8d6207 100644 --- a/transcript-fixer/scripts/cli/__init__.py +++ b/transcript-fixer/scripts/cli/__init__.py @@ -9,6 +9,7 @@ from .commands import ( cmd_init, cmd_add_correction, + cmd_audit, cmd_list_corrections, cmd_run_correction, cmd_review_learned, @@ -25,6 +26,7 @@ __all__ = [ 'cmd_init', 'cmd_add_correction', + 'cmd_audit', 'cmd_list_corrections', 'cmd_run_correction', 'cmd_review_learned', diff --git a/transcript-fixer/scripts/cli/argument_parser.py b/transcript-fixer/scripts/cli/argument_parser.py index a697e77e..a418fd08 100644 --- a/transcript-fixer/scripts/cli/argument_parser.py +++ b/transcript-fixer/scripts/cli/argument_parser.py @@ -37,12 +37,24 @@ def create_argument_parser() -> argparse.ArgumentParser: dest="add_correction", help="Add correction" ) + parser.add_argument( + "--force", + action="store_true", + default=False, + help="Force --add even when safety checks detect risks (common word, substring collision)" + ) parser.add_argument( "--list", action="store_true", dest="list_corrections", help="List all corrections" ) + parser.add_argument( + "--audit", + action="store_true", + dest="audit_dictionary", + help="Audit all active corrections for false positive risks (common words, short text, substring collisions)" + ) # Correction workflow parser.add_argument( diff --git a/transcript-fixer/scripts/cli/commands.py b/transcript-fixer/scripts/cli/commands.py index 5b836b45..dbac6553 100644 --- a/transcript-fixer/scripts/cli/commands.py +++ b/transcript-fixer/scripts/cli/commands.py @@ -43,16 +43,85 @@ def cmd_init(args: argparse.Namespace) -> None: def cmd_add_correction(args: argparse.Namespace) -> None: - """Add a single correction""" + """Add a single correction with safety checks""" service = _get_service() + force = getattr(args, 'force', False) try: - service.add_correction(args.from_text, args.to_text, args.domain) - print(f"✅ Added: '{args.from_text}' → '{args.to_text}' (domain: {args.domain})") + service.add_correction( + args.from_text, args.to_text, args.domain, force=force, + ) + print(f"Added: '{args.from_text}' -> '{args.to_text}' (domain: {args.domain})") except Exception as e: - print(f"❌ Error: {e}") + print(f"Error: {e}", file=sys.stderr) sys.exit(1) +def cmd_audit(args: argparse.Namespace) -> None: + """Audit all active corrections for false positive risks""" + service = _get_service() + domain = getattr(args, 'domain', None) + + print(f"\nAuditing corrections" + (f" (domain: {domain})" if domain else " (all domains)") + "...") + print("=" * 70) + + issues = service.audit_dictionary(domain) + + if not issues: + corrections = service.get_corrections(domain) + print(f"\nAll {len(corrections)} corrections passed safety checks.") + return + + # Categorize + error_count = 0 + warning_count = 0 + for from_text, warnings in issues.items(): + for w in warnings: + if w.level == "error": + error_count += 1 + else: + warning_count += 1 + + corrections = service.get_corrections(domain) + print(f"\nScanned {len(corrections)} corrections. " + f"Found issues in {len(issues)} rules:") + print(f" Errors: {error_count} (should be removed or converted to context rules)") + print(f" Warnings: {warning_count} (review recommended)") + print() + + # Print details grouped by severity + for severity in ["error", "warning"]: + label = "ERRORS" if severity == "error" else "WARNINGS" + relevant = { + ft: [w for w in ws if w.level == severity] + for ft, ws in issues.items() + } + relevant = {ft: ws for ft, ws in relevant.items() if ws} + + if not relevant: + continue + + print(f"--- {label} ({len(relevant)} rules) ---") + for from_text, warnings in sorted(relevant.items()): + to_text = corrections.get(from_text, "?") + print(f"\n '{from_text}' -> '{to_text}'") + for w in warnings: + print(f" [{w.category}] {w.message}") + print(f" Suggestion: {w.suggestion}") + print() + + if error_count > 0: + print( + f"ACTION REQUIRED: {error_count} error(s) found. These rules are " + f"actively causing false positives and should be removed or " + f"converted to context rules." + ) + print( + f"To remove a rule: " + f"sqlite3 ~/.transcript-fixer/corrections.db " + f"\"UPDATE corrections SET is_active=0 WHERE from_text='...';\"" + ) + + def cmd_list_corrections(args: argparse.Namespace) -> None: """List all corrections""" service = _get_service() diff --git a/transcript-fixer/scripts/core/correction_service.py b/transcript-fixer/scripts/core/correction_service.py index d8417a37..daf70cff 100644 --- a/transcript-fixer/scripts/core/correction_service.py +++ b/transcript-fixer/scripts/core/correction_service.py @@ -12,6 +12,7 @@ import re import os +import sys import logging from pathlib import Path from typing import Dict, List, Optional, Tuple @@ -23,6 +24,10 @@ DatabaseError ) +# Import safety check for common words +sys.path.insert(0, str(Path(__file__).parent.parent)) +from utils.common_words import check_correction_safety, audit_corrections, SafetyWarning + logger = logging.getLogger(__name__) @@ -178,10 +183,15 @@ def add_correction( domain: str = "general", source: str = "manual", confidence: float = 1.0, - notes: Optional[str] = None + notes: Optional[str] = None, + force: bool = False, ) -> int: """ - Add a correction with full validation. + Add a correction with full validation and safety checks. + + Safety checks detect common Chinese words and substring collision + risks that would cause false positives. Pass force=True to bypass + (errors become warnings printed to stderr). Args: from_text: Original (incorrect) text @@ -190,12 +200,13 @@ def add_correction( source: Origin of correction confidence: Confidence score notes: Optional notes + force: If True, downgrade safety errors to warnings Returns: ID of inserted correction Raises: - ValidationError: If validation fails + ValidationError: If validation or safety check fails """ # Comprehensive validation self.validate_correction_text(from_text, "from_text") @@ -210,6 +221,34 @@ def add_correction( f"from_text and to_text are identical: '{from_text}'" ) + # Safety check: detect common words and substring collisions + safety_warnings = check_correction_safety(from_text, to_text, strict=True) + + if safety_warnings: + errors = [w for w in safety_warnings if w.level == "error"] + warns = [w for w in safety_warnings if w.level == "warning"] + + if errors and not force: + # Block the addition + msg_parts = [] + for w in errors: + msg_parts.append(f"[{w.category}] {w.message}") + msg_parts.append(f" Suggestion: {w.suggestion}") + raise ValidationError( + f"Safety check BLOCKED adding '{from_text}' -> '{to_text}':\n" + + "\n".join(msg_parts) + + "\n\nUse --force to override (at your own risk)." + ) + + # Print warnings (errors downgraded by --force, or genuine warnings) + all_to_print = errors + warns if force else warns + if all_to_print: + for w in all_to_print: + prefix = "FORCED" if w.level == "error" else "WARNING" + logger.warning( + f"[{prefix}] [{w.category}] {w.message} | {w.suggestion}" + ) + # Get current user added_by = os.getenv("USER") or os.getenv("USERNAME") or "unknown" @@ -431,6 +470,31 @@ def get_statistics(self, domain: Optional[str] = None) -> Dict[str, any]: return stats + # ==================== Audit Operations ==================== + + def audit_dictionary( + self, + domain: Optional[str] = None, + ) -> Dict[str, List[SafetyWarning]]: + """ + Audit all active corrections for safety issues. + + Scans every rule and flags: + - from_text that is a common Chinese word (false positive risk) + - from_text that is <= 2 characters (high collision risk) + - from_text that appears as substring in common words (collateral damage) + - Both from_text and to_text being common words (bidirectional risk) + + Args: + domain: Optional domain filter (None = all domains) + + Returns: + Dict mapping from_text to list of SafetyWarnings. + Only entries with issues are included. + """ + corrections = self.get_corrections(domain) + return audit_corrections(corrections) + # ==================== Helper Methods ==================== def _detect_conflicts( diff --git a/transcript-fixer/scripts/core/dictionary_processor.py b/transcript-fixer/scripts/core/dictionary_processor.py index 15a95863..6a1907e7 100644 --- a/transcript-fixer/scripts/core/dictionary_processor.py +++ b/transcript-fixer/scripts/core/dictionary_processor.py @@ -14,9 +14,17 @@ from __future__ import annotations import re +import sys +import logging +from pathlib import Path from typing import Dict, List, Tuple from dataclasses import dataclass +sys.path.insert(0, str(Path(__file__).parent.parent)) +from utils.common_words import ALL_COMMON_WORDS + +logger = logging.getLogger(__name__) + @dataclass class Change: @@ -96,7 +104,16 @@ def _apply_context_rules(self, text: str) -> Tuple[str, List[Change]]: return corrected, changes def _apply_dictionary(self, text: str) -> Tuple[str, List[Change]]: - """Apply simple dictionary replacements""" + """ + Apply dictionary replacements with substring safety checks. + + Safety layers (applied in order at each match site): + 1. Superset check: if to_text already exists at the match position, + skip to prevent duplication (e.g., "金流"→"现金流" inside "现金流"). + This applies to ALL rules regardless of length. + 2. Boundary check (short rules only, <=3 chars): if the match is inside + a longer common word, skip to prevent collateral damage. + """ changes = [] corrected = text @@ -104,32 +121,167 @@ def _apply_dictionary(self, text: str) -> Tuple[str, List[Change]]: if wrong not in corrected: continue - # Find all occurrences - occurrences = [] - start = 0 - while True: - pos = corrected.find(wrong, start) - if pos == -1: - break - line_num = corrected[:pos].count('\n') + 1 - occurrences.append(line_num) - start = pos + len(wrong) - - # Track changes - for line_num in occurrences: - changes.append(Change( - line_number=line_num, - from_text=wrong, - to_text=correct, - rule_type="dictionary", - rule_name="corrections_dict" - )) - - # Apply replacement - corrected = corrected.replace(wrong, correct) + # All rules go through position-aware replacement to get + # the superset check. Short rules additionally get the + # boundary check against common words. + needs_boundary_check = len(wrong) <= 3 + corrected, new_changes = self._apply_with_safety_checks( + corrected, wrong, correct, needs_boundary_check, + ) + changes.extend(new_changes) return corrected, changes + def _find_occurrences(self, text: str, target: str) -> List[int]: + """Find all line numbers where target appears in text.""" + occurrences = [] + start = 0 + while True: + pos = text.find(target, start) + if pos == -1: + break + line_num = text[:pos].count('\n') + 1 + occurrences.append(line_num) + start = pos + len(target) + return occurrences + + def _apply_with_safety_checks( + self, + text: str, + wrong: str, + correct: str, + check_boundaries: bool, + ) -> Tuple[str, List[Change]]: + """ + Apply replacement at each match position with two safety layers: + + 1. Superset check (all rules): When to_text contains from_text + (e.g., "金流"→"现金流"), check if the surrounding text already + forms to_text. If so, skip — the text is already correct. + + 2. Boundary check (short rules only): Check if the match is inside + a longer common word (e.g., "天差" inside "天差地别"). + """ + changes = [] + result_parts = [] + search_start = 0 + + while search_start < len(text): + pos = text.find(wrong, search_start) + if pos == -1: + result_parts.append(text[search_start:]) + break + + # Safety layer 1: superset check. + # If to_text contains from_text, the replacement could create + # duplication. Check if to_text already exists at this position. + if self._already_corrected(text, pos, wrong, correct): + result_parts.append(text[search_start:pos + len(wrong)]) + search_start = pos + len(wrong) + logger.debug( + f"Skipped '{wrong}' at pos {pos}: " + f"already corrected ('{correct}' present)" + ) + continue + + # Safety layer 2: boundary check (short rules only). + if check_boundaries and self._is_inside_longer_word( + text, pos, wrong + ): + result_parts.append(text[search_start:pos + len(wrong)]) + search_start = pos + len(wrong) + logger.debug( + f"Skipped '{wrong}' at pos {pos}: part of longer word" + ) + continue + + # Safe to replace + line_num = text[:pos].count('\n') + 1 + changes.append(Change( + line_number=line_num, + from_text=wrong, + to_text=correct, + rule_type="dictionary", + rule_name="corrections_dict" + )) + + result_parts.append(text[search_start:pos]) + result_parts.append(correct) + search_start = pos + len(wrong) + + return "".join(result_parts), changes + + @staticmethod + def _already_corrected( + text: str, pos: int, from_text: str, to_text: str + ) -> bool: + """ + Check if to_text already exists at the match position, meaning + the text is already in the corrected form. + + This catches the case where from_text is a substring of to_text + (e.g., "金流" is inside "现金流"). If the surrounding text already + forms "现金流", replacing "金流" would produce "现现金流". + + Returns True if the replacement should be skipped. + """ + if from_text not in to_text: + # to_text doesn't contain from_text, so no superset risk. + return False + + to_len = len(to_text) + from_len = len(from_text) + + # Find all positions where from_text appears inside to_text. + # For each, check if the surrounding text matches to_text. + offset = 0 + while True: + idx = to_text.find(from_text, offset) + if idx == -1: + break + + # If to_text were at text position (pos - idx), from_text at pos + # would be the substring starting at idx within to_text. + candidate_start = pos - idx + candidate_end = candidate_start + to_len + + if (candidate_start >= 0 + and candidate_end <= len(text) + and text[candidate_start:candidate_end] == to_text): + return True + + offset = idx + 1 + + return False + + @staticmethod + def _is_inside_longer_word(text: str, pos: int, match: str) -> bool: + """ + Check if the match at `pos` is embedded inside a longer common word. + + Looks at a window around the match and checks all possible substrings + of that window against the common words set. + """ + match_len = len(match) + # Check windows of 2 to 5 characters that overlap with the match + max_word_len = 5 + window_start = max(0, pos - (max_word_len - 1)) + window_end = min(len(text), pos + match_len + (max_word_len - 1)) + window = text[window_start:window_end] + + # Position of the match within the window + match_offset = pos - window_start + + # Check all substrings that contain the match position + for length in range(match_len + 1, min(max_word_len + 1, len(window) + 1)): + for start in range(max(0, match_offset + match_len - length), + min(match_offset + 1, len(window) - length + 1)): + substr = window[start:start + length] + if substr != match and substr in ALL_COMMON_WORDS: + return True + + return False + def get_summary(self, changes: List[Change]) -> Dict[str, int]: """Generate summary statistics""" summary = { diff --git a/transcript-fixer/scripts/fix_transcription.py b/transcript-fixer/scripts/fix_transcription.py index 6e63379b..323ba17d 100755 --- a/transcript-fixer/scripts/fix_transcription.py +++ b/transcript-fixer/scripts/fix_transcription.py @@ -31,6 +31,7 @@ from cli import ( cmd_init, cmd_add_correction, + cmd_audit, cmd_list_corrections, cmd_run_correction, cmd_review_learned, @@ -89,6 +90,8 @@ def main() -> None: elif args.add_correction: args.from_text, args.to_text = args.add_correction cmd_add_correction(args) + elif getattr(args, 'audit_dictionary', False): + cmd_audit(args) elif args.list_corrections: cmd_list_corrections(args) elif args.review_learned: diff --git a/transcript-fixer/scripts/tests/test_common_words_safety.py b/transcript-fixer/scripts/tests/test_common_words_safety.py new file mode 100644 index 00000000..3d4e205c --- /dev/null +++ b/transcript-fixer/scripts/tests/test_common_words_safety.py @@ -0,0 +1,675 @@ +#!/usr/bin/env python3 +""" +Tests for common word safety checks and boundary-aware replacement. + +Covers the three classes of production bugs: +1. Common words added as corrections cause false positives +2. Substring matching causes collateral damage +3. Short common words should never be dictionary entries +""" + +import unittest +import tempfile +import shutil +from pathlib import Path +import sys + +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from utils.common_words import ( + check_correction_safety, + audit_corrections, + SafetyWarning, + ALL_COMMON_WORDS, + COMMON_WORDS_2CHAR, + SUBSTRING_COLLISION_MAP, +) +from core.dictionary_processor import DictionaryProcessor +from core.correction_repository import CorrectionRepository +from core.correction_service import CorrectionService, ValidationError + + +class TestSafetyChecks(unittest.TestCase): + """Test the check_correction_safety function.""" + + def test_common_word_blocked_strict(self): + """Adding a common word like '仿佛' should produce an error in strict mode.""" + warnings = check_correction_safety("仿佛", "反复", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "Expected at least one error for '仿佛'") + self.assertTrue( + any(w.category == "common_word" for w in errors), + "Expected 'common_word' category", + ) + + def test_common_word_warning_nonstrict(self): + """In non-strict mode, common words produce warnings, not errors.""" + warnings = check_correction_safety("仿佛", "反复", strict=False) + errors = [w for w in warnings if w.level == "error"] + self.assertEqual(len(errors), 0, "Non-strict mode should have no errors") + warns = [w for w in warnings if w.level == "warning"] + self.assertTrue(len(warns) > 0, "Expected at least one warning") + + def test_both_common_words_flagged(self): + """When both from_text and to_text are common words, flag with 'both_common'.""" + warnings = check_correction_safety("正面", "正念", strict=True) + both = [w for w in warnings if w.category == "both_common"] + # "正面" is common, "正念" may or may not be -- but at least common_word should fire + common = [w for w in warnings if w.category == "common_word"] + self.assertTrue(len(common) > 0) + + def test_short_text_warning(self): + """2-char text not in common words list still gets a short_text warning.""" + # Use something unlikely to be in the common words list + warnings = check_correction_safety("zz", "xx", strict=True) + short_warns = [w for w in warnings if w.category == "short_text"] + self.assertTrue(len(short_warns) > 0, "Expected short_text warning for 2-char text") + + def test_known_substring_collision(self): + """'线数' is in SUBSTRING_COLLISION_MAP and should trigger collision warning.""" + warnings = check_correction_safety("线数", "线束", strict=True) + collisions = [w for w in warnings if w.category == "substring_collision"] + self.assertTrue(len(collisions) > 0, "Expected substring_collision for '线数'") + + def test_safe_correction_no_warnings(self): + """A safe, domain-specific correction should produce no warnings.""" + # "巨升智能" -> "具身智能" is a genuine ASR error, not a common word + warnings = check_correction_safety("巨升智能", "具身智能", strict=True) + self.assertEqual(len(warnings), 0, f"Expected no warnings, got: {warnings}") + + def test_long_from_text_safe(self): + """Long from_text (>4 chars) should not trigger short text or collision warnings.""" + warnings = check_correction_safety("语音识别错误", "语音识别模型", strict=True) + short_warns = [w for w in warnings if w.category == "short_text"] + self.assertEqual(len(short_warns), 0) + + # --- Production false positives from the bug report --- + + def test_production_false_positive_fangfu(self): + """'仿佛→反复' was a real production false positive.""" + warnings = check_correction_safety("仿佛", "反复", strict=True) + self.assertTrue(len(warnings) > 0) + + def test_production_false_positive_zhengmian(self): + """'正面→正念' was a real production false positive.""" + warnings = check_correction_safety("正面", "正念", strict=True) + self.assertTrue(len(warnings) > 0) + + def test_production_false_positive_youyu(self): + """'犹豫→抑郁' was a real production false positive.""" + warnings = check_correction_safety("犹豫", "抑郁", strict=True) + self.assertTrue(len(warnings) > 0) + + def test_production_false_positive_chuanshuo(self): + """'传说→穿梭' was a real production false positive.""" + warnings = check_correction_safety("传说", "穿梭", strict=True) + self.assertTrue(len(warnings) > 0) + + def test_production_false_positive_yanji(self): + """'演技→眼界' was a real production false positive.""" + warnings = check_correction_safety("演技", "眼界", strict=True) + self.assertTrue(len(warnings) > 0) + + def test_production_false_positive_zengjia(self): + """'增加→工站/环节' was a real production false positive.""" + warnings = check_correction_safety("增加", "工站", strict=True) + self.assertTrue(len(warnings) > 0) + + +class TestAuditCorrections(unittest.TestCase): + """Test the audit_corrections function.""" + + def test_audit_finds_known_bad_rules(self): + """Audit should flag the production false positives.""" + corrections = { + "仿佛": "反复", + "正面": "正念", + "线数": "线束", + "巨升智能": "具身智能", # This one is fine + } + issues = audit_corrections(corrections) + + self.assertIn("仿佛", issues) + self.assertIn("正面", issues) + self.assertIn("线数", issues) + self.assertNotIn("巨升智能", issues) + + def test_audit_empty_dict(self): + """Audit of empty dict returns empty.""" + issues = audit_corrections({}) + self.assertEqual(len(issues), 0) + + +class TestBoundaryAwareReplacement(unittest.TestCase): + """Test DictionaryProcessor's boundary-aware replacement logic.""" + + def test_substring_collision_prevented(self): + """'线数→线束' should NOT match inside '产线数据'.""" + processor = DictionaryProcessor({"线数": "线束"}, []) + text = "这条产线数据很重要" + result, changes = processor.process(text) + self.assertEqual(result, "这条产线数据很重要", + "Should NOT replace '线数' inside '产线数据'") + self.assertEqual(len(changes), 0) + + def test_standalone_match_replaced(self): + """'线数→线束' SHOULD match when it's standalone (not inside a longer word).""" + processor = DictionaryProcessor({"线数": "线束"}, []) + text = "检查线数是否正确" + result, changes = processor.process(text) + # "线数" here is standalone (not inside a common word), + # so it should be replaced + self.assertEqual(result, "检查线束是否正确") + self.assertEqual(len(changes), 1) + + def test_long_correction_not_affected(self): + """Corrections longer than 3 chars use standard replacement.""" + processor = DictionaryProcessor({"巨升智能": "具身智能"}, []) + text = "今天讨论巨升智能的进展" + result, changes = processor.process(text) + self.assertEqual(result, "今天讨论具身智能的进展") + self.assertEqual(len(changes), 1) + + def test_multiple_replacements_mixed(self): + """Mix of safe and unsafe positions should be handled correctly.""" + processor = DictionaryProcessor({"数据": "数据集"}, []) + text = "大数据分析和数据清洗" + result, changes = processor.process(text) + # "数据" inside "大数据" should be protected + # "数据" standalone should be replaced + # Both are common words, so boundary check applies + # The exact behavior depends on what's in ALL_COMMON_WORDS + # At minimum, the processor should not crash + self.assertIsInstance(result, str) + + def test_no_corrections_no_changes(self): + """Empty corrections dict produces no changes.""" + processor = DictionaryProcessor({}, []) + text = "原始文本" + result, changes = processor.process(text) + self.assertEqual(result, "原始文本") + self.assertEqual(len(changes), 0) + + def test_context_rules_still_work(self): + """Context rules (regex) are unaffected by boundary checks.""" + context_rules = [{ + "pattern": r"股价系统", + "replacement": "框架系统", + "description": "ASR error fix" + }] + processor = DictionaryProcessor({}, context_rules) + text = "股价系统需要优化" + result, changes = processor.process(text) + self.assertEqual(result, "框架系统需要优化") + self.assertEqual(len(changes), 1) + + +class TestSupersetReplacementBug(unittest.TestCase): + """ + Bug 1: When to_text contains from_text as a substring, and the + surrounding text already forms to_text, the replacement must be skipped. + + Production example: rule "金流"→"现金流", input "现金流断了" + Without fix: "现现金流断了" (WRONG -- duplicated prefix) + With fix: "现金流断了" (correct -- already in target form) + + This check must work for ALL rule lengths, not just short rules. + """ + + def test_suffix_superset_skip(self): + """from_text is a suffix of to_text: 金流→现金流 inside 现金流.""" + processor = DictionaryProcessor({"金流": "现金流"}, []) + result, changes = processor.process("现金流断了") + self.assertEqual(result, "现金流断了") + self.assertEqual(len(changes), 0) + + def test_suffix_superset_standalone_replaced(self): + """Standalone from_text should still be replaced.""" + processor = DictionaryProcessor({"金流": "现金流"}, []) + result, changes = processor.process("金流断了") + self.assertEqual(result, "现金流断了") + self.assertEqual(len(changes), 1) + + def test_prefix_superset_skip(self): + """from_text is a prefix of to_text: 现金→现金流 inside 现金流.""" + processor = DictionaryProcessor({"现金": "现金流"}, []) + result, changes = processor.process("现金流断了") + self.assertEqual(result, "现金流断了") + self.assertEqual(len(changes), 0) + + def test_middle_superset_skip(self): + """from_text is in the middle of to_text.""" + processor = DictionaryProcessor({"金流": "现金流通"}, []) + result, changes = processor.process("现金流通畅") + self.assertEqual(result, "现金流通畅") + self.assertEqual(len(changes), 0) + + def test_long_rule_superset_skip(self): + """Superset check must also work for long rules (>3 chars).""" + processor = DictionaryProcessor({"金流断裂": "现金流断裂"}, []) + result, changes = processor.process("现金流断裂了") + self.assertEqual(result, "现金流断裂了") + self.assertEqual(len(changes), 0) + + def test_long_rule_superset_standalone_replaced(self): + """Long rule standalone should still be replaced.""" + processor = DictionaryProcessor({"金流断裂": "现金流断裂"}, []) + result, changes = processor.process("金流断裂了") + self.assertEqual(result, "现金流断裂了") + self.assertEqual(len(changes), 1) + + def test_superset_with_unknown_words(self): + """Superset check works regardless of common_words membership.""" + # Use words NOT in ALL_COMMON_WORDS + processor = DictionaryProcessor({"资流": "投资流"}, []) + result, changes = processor.process("投资流断了") + self.assertEqual(result, "投资流断了") + self.assertEqual(len(changes), 0) + + def test_superset_mixed_positions(self): + """One occurrence is already correct, another is standalone.""" + processor = DictionaryProcessor({"金流": "现金流"}, []) + result, changes = processor.process("现金流好，金流差") + self.assertEqual(result, "现金流好，现金流差") + self.assertEqual(len(changes), 1) + + def test_no_superset_normal_replacement(self): + """When to_text does NOT contain from_text, normal replacement.""" + processor = DictionaryProcessor({"金流": "资金链"}, []) + result, changes = processor.process("金流断了") + self.assertEqual(result, "资金链断了") + self.assertEqual(len(changes), 1) + + +class TestIdiomCompoundProtection(unittest.TestCase): + """ + Bug 2: Short rules must not corrupt idioms and compound words. + + Production examples: + - "天差"→"偏差" inside "天差地别" => "偏差地别" (broken idiom) + - "亮亮"→"亮哥" inside "漂漂亮亮" => "漂漂亮哥" (broken phrase) + + Defense: _is_inside_longer_word checks common_words set. + """ + + def test_tiancha_inside_idiom(self): + """天差→偏差 must not break 天差地别.""" + processor = DictionaryProcessor({"天差": "偏差"}, []) + result, changes = processor.process("天差地别") + self.assertEqual(result, "天差地别") + self.assertEqual(len(changes), 0) + + def test_liangliang_inside_compound(self): + """亮亮→亮哥 must not break 漂漂亮亮.""" + processor = DictionaryProcessor({"亮亮": "亮哥"}, []) + result, changes = processor.process("漂漂亮亮") + self.assertEqual(result, "漂漂亮亮") + self.assertEqual(len(changes), 0) + + def test_tiancha_standalone_replaced(self): + """Standalone 天差 (not inside idiom) should be replaced.""" + processor = DictionaryProcessor({"天差": "偏差"}, []) + # 天差 alone, not followed by 地别 or other idiom continuation + result, changes = processor.process("误差天差太大了") + # Whether this gets replaced depends on common_words; at minimum + # it should not crash. If 天差 is in common words, it stays. + self.assertIsInstance(result, str) + + +class TestValidPhraseProtection(unittest.TestCase): + """ + Bug 3: Short rules must not corrupt valid phrases where from_text + is a legitimate substring. + + Production example: + - "被看"→"被砍" inside "被看见" => "被砍见" + + Defense: _is_inside_longer_word checks common_words set. + """ + + def test_beikan_inside_beikanjian(self): + """被看→被砍 must not break 被看见.""" + processor = DictionaryProcessor({"被看": "被砍"}, []) + result, changes = processor.process("被看见") + self.assertEqual(result, "被看见") + self.assertEqual(len(changes), 0) + + def test_beikan_in_sentence(self): + """被看→被砍 must not break 被看见 in a full sentence.""" + processor = DictionaryProcessor({"被看": "被砍"}, []) + result, changes = processor.process("他被看见了") + self.assertEqual(result, "他被看见了") + self.assertEqual(len(changes), 0) + + +class TestServiceSafetyIntegration(unittest.TestCase): + """Integration tests: CorrectionService rejects unsafe corrections.""" + + def setUp(self): + self.test_dir = Path(tempfile.mkdtemp()) + self.db_path = self.test_dir / "test.db" + self.repository = CorrectionRepository(self.db_path) + self.service = CorrectionService(self.repository) + + def tearDown(self): + self.service.close() + shutil.rmtree(self.test_dir) + + def test_common_word_rejected(self): + """Adding a common word correction is blocked by default.""" + with self.assertRaises(ValidationError) as ctx: + self.service.add_correction("仿佛", "反复", "general") + self.assertIn("Safety check BLOCKED", str(ctx.exception)) + + def test_common_word_forced(self): + """Adding a common word with force=True succeeds.""" + correction_id = self.service.add_correction( + "仿佛", "反复", "general", force=True, + ) + self.assertIsInstance(correction_id, int) + self.assertGreater(correction_id, 0) + + def test_safe_correction_accepted(self): + """A genuine ASR correction is accepted without force.""" + correction_id = self.service.add_correction( + "巨升智能", "具身智能", "general", + ) + self.assertIsInstance(correction_id, int) + + def test_audit_on_service(self): + """audit_dictionary method returns issues for unsafe rules.""" + # Force-add some unsafe rules + self.service.add_correction("仿佛", "反复", "general", force=True) + self.service.add_correction("巨升智能", "具身智能", "general") + + issues = self.service.audit_dictionary("general") + self.assertIn("仿佛", issues) + self.assertNotIn("巨升智能", issues) + + +class TestProductionFalsePositivesCoverage(unittest.TestCase): + """ + Verify ALL production false positives from the 2026-03 manual review + are present in the safety system and correctly caught. + + Each test corresponds to a specific word that caused real damage in production. + If any of these tests fail, it means the safety net has a gap. + """ + + # --- Category 1: Lifestyle domain --- + + def test_baojian_blocked(self): + """'保健' (lifestyle/beauty) must be caught.""" + self.assertIn("保健", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("保健", "宝剑", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'保健' must produce an error") + + def test_neihan_blocked(self): + """'内涵' (lifestyle/beauty) must be caught.""" + self.assertIn("内涵", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("内涵", "内含", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'内涵' must produce an error") + + def test_zhengjing_blocked(self): + """'正经' (lifestyle) must be caught.""" + self.assertIn("正经", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("正经", "正劲", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'正经' must produce an error") + + # --- Category 1: Manufacturing domain --- + + def test_jingong_blocked(self): + """'仅供' (manufacturing) must be caught.""" + self.assertIn("仅供", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("仅供", "紧供", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'仅供' must produce an error") + + def test_gongqi_blocked(self): + """'供气' (manufacturing) must be caught.""" + self.assertIn("供气", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("供气", "工器", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'供气' must produce an error") + + def test_chutou_blocked(self): + """'出头' (manufacturing) must be caught.""" + self.assertIn("出头", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("出头", "初投", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'出头' must produce an error") + + def test_jikou_blocked(self): + """'几口' (manufacturing) must be caught.""" + self.assertIn("几口", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("几口", "集口", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'几口' must produce an error") + + # --- Category 1: Various domains --- + + def test_liangben_blocked(self): + """'两本' must be caught.""" + self.assertIn("两本", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("两本", "量本", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'两本' must produce an error") + + def test_chuwu_blocked(self): + """'初五' must be caught.""" + self.assertIn("初五", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("初五", "出误", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'初五' must produce an error") + + def test_lijie_blocked(self): + """'力竭' must be caught.""" + self.assertIn("力竭", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("力竭", "立杰", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'力竭' must produce an error") + + def test_chongyu_blocked(self): + """'充于' must be caught.""" + self.assertIn("充于", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("充于", "冲余", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'充于' must produce an error") + + def test_shuju_blocked(self): + """'数据' must be caught.""" + self.assertIn("数据", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("数据", "束据", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'数据' must produce an error") + + # --- Category 1: Substring collision sources --- + + def test_beikan_blocked(self): + """'被看' (general) must be caught.""" + self.assertIn("被看", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("被看", "被砍", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'被看' must produce an error") + + def test_tiancha_blocked(self): + """'天差' (education) must be caught.""" + self.assertIn("天差", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("天差", "偏差", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'天差' must produce an error") + + def test_liangliang_blocked(self): + """'亮亮' (manufacturing) must be caught.""" + self.assertIn("亮亮", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("亮亮", "亮哥", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'亮亮' must produce an error") + + def test_jinliu_blocked(self): + """'金流' (manufacturing) must be caught.""" + self.assertIn("金流", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("金流", "现金流", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'金流' must produce an error") + + # --- Category 1: Substring issue sources --- + + def test_kanjian_blocked(self): + """'看见' must be caught (caused substring issues).""" + self.assertIn("看见", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("看见", "砍件", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'看见' must produce an error") + + def test_fenzhong_blocked(self): + """'分钟' must be caught (caused substring issues).""" + self.assertIn("分钟", COMMON_WORDS_2CHAR) + warnings = check_correction_safety("分钟", "份种", strict=True) + errors = [w for w in warnings if w.level == "error"] + self.assertTrue(len(errors) > 0, "'分钟' must produce an error") + + +class TestSubstringCollisionMapCoverage(unittest.TestCase): + """ + Verify all production substring collision patterns are in the map. + + Each test reproduces a real corruption pattern from production: + a short word matched inside a longer valid phrase and corrupted it. + """ + + def test_xianshu_collision_exists(self): + """'线数' inside '产线数据' -> corrupts to '产线束据'.""" + self.assertIn("线数", SUBSTRING_COLLISION_MAP) + self.assertIn("产线数据", SUBSTRING_COLLISION_MAP["线数"]) + + def test_jinliu_collision_exists(self): + """'金流' inside '现金流' -> corrupts to '现现金流'.""" + self.assertIn("金流", SUBSTRING_COLLISION_MAP) + self.assertIn("现金流", SUBSTRING_COLLISION_MAP["金流"]) + + def test_beikan_collision_exists(self): + """'被看' inside '被看见' -> corrupts to '被砍见'.""" + self.assertIn("被看", SUBSTRING_COLLISION_MAP) + self.assertIn("被看见", SUBSTRING_COLLISION_MAP["被看"]) + + def test_liangliang_collision_exists(self): + """'亮亮' inside '漂漂亮亮' -> corrupts to '漂漂亮哥'.""" + self.assertIn("亮亮", SUBSTRING_COLLISION_MAP) + self.assertIn("漂漂亮亮", SUBSTRING_COLLISION_MAP["亮亮"]) + + def test_tiancha_collision_exists(self): + """'天差' inside '天差地别' -> corrupts idiom to '偏差地别'.""" + self.assertIn("天差", SUBSTRING_COLLISION_MAP) + self.assertIn("天差地别", SUBSTRING_COLLISION_MAP["天差"]) + + def test_collision_safety_check_fires(self): + """check_correction_safety must flag entries in SUBSTRING_COLLISION_MAP.""" + for short_word in ["金流", "被看", "亮亮", "天差"]: + warnings = check_correction_safety(short_word, "dummy", strict=True) + collision_warnings = [ + w for w in warnings if w.category == "substring_collision" + ] + self.assertTrue( + len(collision_warnings) > 0, + f"'{short_word}' must trigger substring_collision warning", + ) + + +class TestBoundaryAwareProductionCollisions(unittest.TestCase): + """ + End-to-end tests: verify DictionaryProcessor does NOT corrupt + longer valid phrases when a short correction matches inside them. + + Each test reproduces an exact production corruption scenario. + """ + + def test_jinliu_inside_xianjinliu(self): + """'金流→现金流' must NOT corrupt '现金流' to '现现金流'.""" + processor = DictionaryProcessor({"金流": "现金流"}, []) + text = "公司的现金流很健康" + result, changes = processor.process(text) + self.assertEqual(result, "公司的现金流很健康", + "Must NOT replace '金流' inside '现金流'") + self.assertEqual(len(changes), 0) + + def test_beikan_inside_beikanjian(self): + """'被看→被砍' must NOT corrupt '被看见' to '被砍见'.""" + processor = DictionaryProcessor({"被看": "被砍"}, []) + text = "他被看见了" + result, changes = processor.process(text) + self.assertEqual(result, "他被看见了", + "Must NOT replace '被看' inside '被看见'") + self.assertEqual(len(changes), 0) + + def test_liangliang_inside_piaopiaoliangliag(self): + """'亮亮→亮哥' must NOT corrupt '漂漂亮亮' to '漂漂亮哥'.""" + processor = DictionaryProcessor({"亮亮": "亮哥"}, []) + text = "打扮得漂漂亮亮的" + result, changes = processor.process(text) + self.assertEqual(result, "打扮得漂漂亮亮的", + "Must NOT replace '亮亮' inside '漂漂亮亮'") + self.assertEqual(len(changes), 0) + + def test_tiancha_inside_tianchadiebie(self): + """'天差→偏差' must NOT corrupt '天差地别' to '偏差地别'.""" + processor = DictionaryProcessor({"天差": "偏差"}, []) + text = "两者天差地别" + result, changes = processor.process(text) + self.assertEqual(result, "两者天差地别", + "Must NOT replace '天差' inside '天差地别'") + self.assertEqual(len(changes), 0) + + def test_kanjian_not_corrupted_by_beikan(self): + """'被看→被砍' must NOT corrupt '看见' if '被看见' is in text.""" + processor = DictionaryProcessor({"被看": "被砍"}, []) + text = "我被看见了，别人也看见了" + result, changes = processor.process(text) + # '被看见' contains '被看' -- boundary check must protect it + self.assertNotIn("被砍", result, + "Must NOT corrupt any instance of '被看' inside '被看见'") + + +class TestAuditCatchesAllProductionFalsePositives(unittest.TestCase): + """ + Verify audit_corrections flags every single production false positive + when they appear in a corrections dictionary. + """ + + def test_audit_catches_all_category1_words(self): + """Every Category 1 word must be flagged by audit_corrections.""" + all_false_positives = { + # lifestyle + "仿佛": "反复", "正面": "正念", "犹豫": "抑郁", + "传说": "穿梭", "演技": "眼界", "无果": "无过", + "旗号": "期号", "应急": "应集", "正经": "正劲", + # lifestyle/beauty + "保健": "宝剑", "内涵": "内含", + # manufacturing + "仅供": "紧供", "供气": "工器", "出头": "初投", "几口": "集口", + # lifestyle previously disabled + "增加": "工站", "教育": "叫于", "大一": "答疑", + "曲线": "去先", "分母": "份母", + # various domains + "两本": "量本", "初五": "出误", "数据": "束据", + "力竭": "立杰", "充于": "冲余", + # substring collision sources + "被看": "被砍", "天差": "偏差", "亮亮": "亮哥", "金流": "现金流", + # substring issue words + "看见": "砍件", "分钟": "份种", + } + + issues = audit_corrections(all_false_positives) + + for word in all_false_positives: + self.assertIn( + word, issues, + f"audit_corrections MUST flag '{word}' but did not" + ) + + +if __name__ == '__main__': + unittest.main() diff --git a/transcript-fixer/scripts/utils/common_words.py b/transcript-fixer/scripts/utils/common_words.py new file mode 100644 index 00000000..16b498c1 --- /dev/null +++ b/transcript-fixer/scripts/utils/common_words.py @@ -0,0 +1,311 @@ +#!/usr/bin/env python3 +""" +Common Chinese Words Safety Check + +Detects when a correction's from_text is a common Chinese word, +which would cause false positive replacements across transcripts. + +This is the core defense against the "仿佛→反复" class of bugs: +valid corrections for one ASR model that corrupt correct text from better models. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import List, Set + + +# High-frequency Chinese words that should NEVER be dictionary correction sources. +# These are words that appear correctly in normal Chinese text and replacing them +# would cause widespread collateral damage. +# +# Organized by category for maintainability. Not exhaustive -- the heuristic +# checks below catch additional cases. +# +# IMPORTANT: This list was curated from actual production false positives found +# in a 187-video transcription run (2026-03). Each entry caused real damage. + +COMMON_WORDS_2CHAR: Set[str] = { + # --- Production false positives (confirmed damage, 2026-03 run) --- + # lifestyle domain + "仿佛", "正面", "犹豫", "传说", "演技", "无果", "旗号", "应急", "正经", + # lifestyle/beauty domain + "保健", "内涵", + # manufacturing domain + "仅供", "供气", "出头", "几口", + # lifestyle - previously disabled then re-confirmed + "增加", "教育", "大一", "曲线", "分母", + # various domains - discovered in manual review + "两本", "初五", "数据", "力竭", "充于", + # general/manufacturing/education - substring collision sources + "被看", "天差", "亮亮", "金流", + # caused substring issues in production + "看见", "分钟", + # --- High-frequency general vocabulary --- + "我们", "他们", "你们", "这个", "那个", "什么", "怎么", "为什么", + "可以", "因为", "所以", "但是", "虽然", "如果", "已经", "正在", + "需要", "应该", "可能", "一定", "非常", "比较", "特别", "一般", + "开始", "结束", "继续", "发展", "问题", "方法", "工作", "时间", + "学习", "研究", "分析", "讨论", "了解", "知道", "觉得", "认为", + "希望", "表示", "提出", "建议", "要求", "计划", "设计", "管理", + "技术", "系统", "数据", "网络", "平台", "产品", "服务", "市场", + "企业", "公司", "团队", "项目", "客户", "用户", "资源", "成本", + "效果", "质量", "安全", "标准", "流程", "模式", "策略", "方案", + "结构", "功能", "接口", "模块", "组件", "测试", "部署", "运维", + "目标", "任务", "进度", "优化", "调整", "更新", "升级", "维护", + "配置", "参数", "设置", "选项", "状态", "信息", "内容", "格式", + "教育", "培训", "实践", "经验", "能力", "水平", "素质", "思维", + "创新", "合作", "沟通", "交流", "反馈", "评估", "考核", "激励", + # --- Common verbs and adjectives --- + "实现", "完成", "处理", "解决", "执行", "操作", "运行", "启动", + "关闭", "打开", "保存", "删除", "修改", "添加", "移除", "查看", + "搜索", "过滤", "排序", "导入", "导出", "上传", "下载", "同步", + "重要", "关键", "核心", "基本", "主要", "次要", "简单", "复杂", + "明确", "清晰", "具体", "详细", "准确", "完整", "稳定", "灵活", + # --- Domain terms that look like ASR errors but are valid --- + "线数", "曲线", "分母", "正面", "旗号", "无果", "演技", +} + +# Common 3+ character words that should also be protected. +# These serve dual purpose: +# 1. Never used as correction sources (same as 2-char words) +# 2. Used by DictionaryProcessor._is_inside_longer_word() to detect +# when a short correction target is embedded inside a valid longer word +COMMON_WORDS_3PLUS: Set[str] = { + "自动化", "智能化", "数字化", "信息化", "标准化", "规范化", + "产线数", "服务器", "数据库", "操作系统", "人工智能", "机器学习", + "深度学习", "自然语言", "计算机视觉", "强化学习", + "区块链", "云计算", "大数据", "物联网", "互联网", + # --- Production collision targets (longer words containing short false positives) --- + # These must be here so _is_inside_longer_word() can detect them + "产线数据", "现金流", "资金流", "现金流量", "资金流向", + "被看见", "被看到", "被看作", "被看成", "被看好", + "漂漂亮亮", "亮亮堂堂", "明明亮亮", + "天差地别", "天差地远", + "被看见", "没看见", + "出头露面", "出头之日", + "正月初五", "大年初五", + "保健品", "保健操", "医疗保健", + "文化内涵", + "无果而终", + # --- Common Chinese idioms/phrases containing short words --- + # These are needed to prevent idiom corruption + "正面临", "正面对", + "应急响应", "应急预案", "应急处理", + "仅供参考", "仅供参阅", +} + +# Words that commonly contain other words as substrings. +# Key: the short word, Value: common words containing it. +# Used to warn about substring collision risk. +SUBSTRING_COLLISION_MAP: dict[str, list[str]] = { + "线数": ["产线数据", "曲线数", "线数量"], + "增加": ["新增加", "增加值"], + "数据": ["大数据", "数据库", "数据集", "元数据"], + "服务": ["服务器", "服务端", "微服务", "云服务"], + "测试": ["单元测试", "集成测试", "压力测试", "测试用例"], + "模型": ["大模型", "模型训练", "预训练模型"], + "学习": ["学习率", "深度学习", "机器学习", "强化学习"], + "正面": ["正面临", "正面对"], + "应急": ["应急响应", "应急预案", "应急处理"], + "无果": ["无果而终", "毫无果断"], + # --- Production substring collision patterns (2026-03 manual review) --- + # "线数" inside "产线数据" → corrupts to "产线束据" + # (already covered above) + # "金流" inside "现金流" → corrupts to "现现金流" (replacement contains match) + "金流": ["现金流", "资金流", "资金流向", "现金流量"], + # "被看" inside "被看见" → corrupts to "被砍见" + "被看": ["被看见", "被看到", "被看作", "被看成", "被看好"], + # "亮亮" inside "漂漂亮亮" → corrupts to "漂漂亮哥" + "亮亮": ["漂漂亮亮", "亮亮堂堂", "明明亮亮"], + # "天差" inside "天差地别" → corrupts idiom to "偏差地别" + "天差": ["天差地别", "天差地远"], + # "看见" inside longer phrases → substring collision risk + "看见": ["被看见", "看见过", "没看见"], + # "分钟" inside longer phrases → substring collision risk + "分钟": ["几分钟", "十分钟", "三十分钟", "一分钟"], + # "出头" common in phrases + "出头": ["出头露面", "出头之日", "冒出头"], + # "初五" common in date phrases + "初五": ["正月初五", "大年初五"], + # "保健" common in compound words + "保健": ["保健品", "保健操", "医疗保健"], + # "内涵" common in compound words + "内涵": ["内涵段子", "文化内涵"], +} + +ALL_COMMON_WORDS: Set[str] = COMMON_WORDS_2CHAR | COMMON_WORDS_3PLUS + + +@dataclass +class SafetyWarning: + """A warning about a potentially dangerous correction rule.""" + level: str # "error" (should block) or "warning" (should confirm) + category: str # "common_word", "short_text", "substring_collision" + message: str + suggestion: str # What to do instead + + +def check_correction_safety( + from_text: str, + to_text: str, + strict: bool = True, +) -> List[SafetyWarning]: + """ + Check if a correction rule is safe to add to the dictionary. + + This is the main entry point. Returns a list of warnings/errors. + Empty list = safe to add. + + Args: + from_text: The text to be replaced (the "wrong" text) + to_text: The replacement text (the "correct" text) + strict: If True, common word matches are errors; if False, warnings + + Returns: + List of SafetyWarning objects (empty = safe) + """ + warnings: List[SafetyWarning] = [] + + # Check 1: Is from_text a known common word? + if from_text in ALL_COMMON_WORDS: + level = "error" if strict else "warning" + warnings.append(SafetyWarning( + level=level, + category="common_word", + message=( + f"'{from_text}' is a common Chinese word that appears correctly " + f"in normal text. Replacing it with '{to_text}' will cause " + f"false positives across all transcripts." + ), + suggestion=( + f"Use a context rule instead: add a regex pattern that matches " + f"'{from_text}' only in the specific context where it's an ASR error. " + f"Example: match '{from_text}' only when preceded/followed by specific characters." + ), + )) + + # Check 2: Is from_text very short (<=2 chars)? + if len(from_text) <= 2: + # Even if not in our common words list, 2-char Chinese words are risky + if from_text not in ALL_COMMON_WORDS: + # Not already flagged above -- add a length warning + warnings.append(SafetyWarning( + level="warning", + category="short_text", + message=( + f"'{from_text}' is only {len(from_text)} character(s). " + f"Short corrections have high false positive risk in Chinese " + f"because they match as substrings inside longer words." + ), + suggestion=( + f"Verify '{from_text}' is never a valid word in any context. " + f"If unsure, use a context rule with surrounding text patterns instead." + ), + )) + + # Check 3: Could from_text match as a substring inside common words? + # This catches the "线数" matching inside "产线数据" bug. + if from_text in SUBSTRING_COLLISION_MAP: + collisions = SUBSTRING_COLLISION_MAP[from_text] + warnings.append(SafetyWarning( + level="error" if strict else "warning", + category="substring_collision", + message=( + f"'{from_text}' is a substring of common words: " + f"{', '.join(collisions)}. " + f"Replacing '{from_text}' with '{to_text}' will corrupt these words." + ), + suggestion=( + f"Use a context rule with negative lookahead/lookbehind to exclude " + f"matches inside these common words. Example regex: " + f"'(?<!产){from_text}(?!据)' to avoid matching inside '产线数据'." + ), + )) + else: + # Dynamic check: scan our common words for substring matches + _check_dynamic_substring_collisions(from_text, to_text, warnings) + + # Check 4: Is from_text == to_text except for tone/similar sound? + # (Catch obvious non-errors like 仿佛→反复 where both are valid words) + if from_text in ALL_COMMON_WORDS and to_text in ALL_COMMON_WORDS: + warnings.append(SafetyWarning( + level="error" if strict else "warning", + category="both_common", + message=( + f"Both '{from_text}' and '{to_text}' are common Chinese words. " + f"This is almost certainly a false correction -- both forms are " + f"valid in different contexts." + ), + suggestion=( + f"This rule should NOT be in the dictionary. If '{from_text}' is " + f"genuinely an ASR error in a specific domain, use a context rule " + f"tied to that domain's vocabulary." + ), + )) + + return warnings + + +def _check_dynamic_substring_collisions( + from_text: str, + to_text: str, + warnings: List[SafetyWarning], +) -> None: + """ + Check if from_text appears as a substring in any common word, + where the common word is NOT the from_text itself. + """ + if len(from_text) > 4: + # Long enough that substring collisions are unlikely to be problematic + return + + collisions: List[str] = [] + for word in ALL_COMMON_WORDS: + if word == from_text: + continue + if from_text in word: + collisions.append(word) + + if collisions: + # Only show first 5 to avoid spam + shown = collisions[:5] + more = f" (and {len(collisions) - 5} more)" if len(collisions) > 5 else "" + warnings.append(SafetyWarning( + level="warning", + category="substring_collision", + message=( + f"'{from_text}' appears inside {len(collisions)} common word(s): " + f"{', '.join(shown)}{more}. " + f"This replacement may cause collateral damage." + ), + suggestion=( + f"Review whether '{from_text}→{to_text}' could corrupt any of " + f"these words. Consider using a context rule instead." + ), + )) + + +def audit_corrections( + corrections: dict[str, str], +) -> dict[str, List[SafetyWarning]]: + """ + Audit all corrections in a dictionary for safety issues. + + Used by the --audit command. + + Args: + corrections: Dict of {from_text: to_text} + + Returns: + Dict of {from_text: [warnings]} for entries with issues. + Entries with no issues are not included. + """ + results: dict[str, List[SafetyWarning]] = {} + + for from_text, to_text in corrections.items(): + warnings = check_correction_safety(from_text, to_text, strict=False) + if warnings: + results[from_text] = warnings + + return results diff --git a/tunnel-doctor/SKILL.md b/tunnel-doctor/SKILL.md index f0e231a3..4f3aee3c 100644 --- a/tunnel-doctor/SKILL.md +++ b/tunnel-doctor/SKILL.md @@ -37,6 +37,8 @@ Determine which scenario applies: - **`git clone` fails with `Connection closed by 198.18.x.x`** → TUN DNS hijack for SSH (Step 2H) - **SSH connects but `operation not permitted`** → Tailscale SSH config issue (Step 4) - **SSH connects but `be-child ssh` exits code 1** → WSL snap sandbox issue (Step 5) +- **TCP port 22 reachable (`nc -z` succeeds) but SSH fails with `kex_exchange_identification: Connection closed`** → Tailscale SSH proxy intercept on WSL (Step 5A) +- **`tailscale ssh` returns "not available on App Store builds"** → Wrong Tailscale distribution on macOS (Step 5B) **Key distinctions**: - SSH does NOT use `http_proxy`/`NO_PROXY` env vars. If SSH works but HTTP doesn't → Layer 2. @@ -45,6 +47,8 @@ Determine which scenario applies: - If `ssh -T git@github.com` works but `git push` fails intermittently → Layer 4 (double tunnel). - If host `curl https://...` works but `docker pull` times out → Layer 5 (VM proxy propagation). - If DNS resolves to `198.18.x.x` virtual IPs → TUN DNS hijack (Step 2H). +- If `nc -z` succeeds on port 22 but SSH gets no banner (`kex_exchange_identification`) → Tailscale SSH proxy intercept (Step 5A). Confirm with `tcpdump -i any port 22` on the remote — 0 packets means Tailscale intercepts above the kernel. +- If `tailscale ssh` fails with "not available on App Store builds" → install Standalone Tailscale (Step 5B). ### Fast Path: Run Automated Checks @@ -96,6 +100,18 @@ export NO_PROXY=localhost,127.0.0.1,.ts.net,100.64.0.0/10,192.168.*,10.*,172.16. **NO_PROXY syntax pitfalls** — see [references/proxy_conflict_reference.md](references/proxy_conflict_reference.md) for the compatibility matrix. +**Go `net/http` CIDR caveat**: Go's standard `net/http` does NOT support CIDR notation in `NO_PROXY`. Setting `NO_PROXY=100.64.0.0/10` works for curl and Python, but Go programs (including Tailscale-adjacent tooling) will still send traffic through the proxy. The fix is to use MagicDNS hostnames (e.g., `workstation-4090-wsl`) instead of raw IPs, or add explicit hostnames to `NO_PROXY`: + +```bash +# WRONG for Go programs — CIDR is silently ignored +NO_PROXY=100.64.0.0/10 go-program http://100.101.102.103:8002/health # → goes through proxy + +# CORRECT — use hostname (matched as suffix) or explicit IP +export NO_PROXY=localhost,127.0.0.1,.ts.net,workstation-4090-wsl,100.101.102.103,192.168.*,10.*,172.16.* +``` + +This is especially relevant when accessing Tailscale services from Go-based tools (e.g., custom CLIs, Go test suites hitting remote APIs). + Verify the fix: ```bash @@ -127,6 +143,19 @@ gateway: 192.168.x.1 # Default gateway interface: en0 # Physical interface, NOT Tailscale ``` +**Important**: Not all `utun` interfaces are Tailscale's. Verify which utun belongs to Tailscale before concluding the route is correct: + +```bash +# Find Tailscale's utun interface (has a 100.x.x.x IP) +ifconfig | grep -A2 'inet 100\.' +``` + +Quick indicators by MTU: +- **MTU 1280** → typically Tailscale +- **MTU 4064** → typically Shadowrocket TUN + +If `route -n get` shows traffic going to a utun with MTU 4064, it is hitting Shadowrocket's TUN, not Tailscale — this is still a route conflict even though the interface name starts with `utun`. + Confirm with full route table: ```bash @@ -508,13 +537,89 @@ sudo tailscale up --ssh **Important**: The new installation may assign a different Tailscale IP. Check with `tailscale status --self`. +### Step 5A: Fix Tailscale SSH Proxy Silent Failure on WSL + +**Symptom**: TCP port 22 is reachable (`nc -z -w 5 <ip> 22` succeeds), but SSH fails immediately with: + +``` +kex_exchange_identification: Connection closed by remote host +``` + +No SSH banner is ever received. This happens even with apt-installed Tailscale (not snap). + +**Root cause**: When `tailscale up --ssh` is enabled on WSL, Tailscale intercepts port 22 connections at the application layer (above the kernel network stack). If Tailscale's built-in SSH proxy malfunctions, it accepts the TCP connection but immediately closes it before sending the SSH banner. + +**Key diagnostic** — on the WSL instance: + +```bash +# This will show 0 packets even during active SSH attempts +sudo tcpdump -i any port 22 -c 5 -w /dev/null 2>&1 +``` + +Zero packets means Tailscale is intercepting connections before they reach the kernel network stack. The kernel's `sshd` never sees the connection. + +**Distinction from Step 5**: Step 5 covers snap sandbox issues where `be-child ssh` fails. This is a different problem — Tailscale's SSH proxy itself silently fails, regardless of installation method. + +**Fix** — disable Tailscale's SSH proxy and use regular sshd: + +```bash +# On the WSL instance: +sudo tailscale up --ssh=false + +# Verify sshd is running +sudo service ssh status +# If not running: +sudo service ssh start + +# Verify from the client machine: +ssh -o ConnectTimeout=10 <user>@<tailscale-ip> 'echo SSH_OK' +``` + +After disabling Tailscale SSH, connections go through the kernel network stack to `sshd` as normal. The Tailscale ACL `"action": "accept"` in Step 4 is no longer relevant — authentication is handled by `sshd` using SSH keys or passwords. + +**When to keep `--ssh` enabled**: Only if you specifically need Tailscale's SSH features (ACL-based access control, no SSH key management). If standard sshd works, prefer `--ssh=false` for reliability. + +### Step 5B: Fix App Store Tailscale on macOS (Missing `tailscale ssh`) + +**Symptom**: Running `tailscale ssh` returns: + +``` +The 'tailscale ssh' subcommand is not available on macOS builds +distributed through the App Store or TestFlight. +``` + +**Root cause**: The App Store version of Tailscale for macOS is sandboxed and does not include the `tailscale ssh` subcommand. + +**Fix** — install the Standalone version: + +1. Uninstall the App Store version (delete from /Applications) +2. Download the Standalone build from https://pkgs.tailscale.com/stable/#macos +3. Install to /Applications + +**Post-install CLI setup**: The standalone `tailscale` CLI binary is embedded inside the app bundle. Add an alias to your shell config: + +```bash +# ~/.zshrc +alias tailscale="/Applications/Tailscale.app/Contents/MacOS/Tailscale" +``` + +Verify: + +```bash +source ~/.zshrc +tailscale version +tailscale ssh <user>@<hostname> # Should work now +``` + ### Step 6: Verify End-to-End Run a complete connectivity test: ```bash -# 1. Check route is correct +# 1. Check route is correct (must show Tailscale's utun, not en0 or Shadowrocket's utun) route -n get <tailscale-ip> +# Also confirm which utun is Tailscale's: +ifconfig | grep -A2 'inet 100\.' # 2. Test TCP connectivity nc -z -w 5 <tailscale-ip> 22 @@ -523,7 +628,7 @@ nc -z -w 5 <tailscale-ip> 22 ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no <user>@<tailscale-ip> 'echo SSH_OK && hostname && whoami' ``` -All three must pass. If step 1 fails, revisit Step 3. If step 2 fails, check WSL sshd or firewall. If step 3 fails, revisit Steps 4-5. +All three must pass. If step 1 fails, revisit Step 3. If step 1 shows wrong utun (e.g., Shadowrocket's utun with MTU 4064 instead of Tailscale's with MTU 1280), that is also a route conflict. If step 2 passes but step 3 fails with `kex_exchange_identification`, revisit Step 5A (Tailscale SSH proxy intercept). If step 2 fails, check WSL sshd or firewall. If step 3 fails with other errors, revisit Steps 4-5. ## SOP: Remote Development via Tailscale @@ -612,7 +717,13 @@ Each `-L` flag is independent. If one port is already bound locally, `ExitOnForw ### 4. SSH Non-Login Shell Setup -SSH non-login shells don't load `~/.zshrc`, so nvm/Homebrew tools and proxy env vars are unavailable. Prefix all remote commands with `source ~/.zshrc 2>/dev/null;`. See [references/proxy_conflict_reference.md § SSH Non-Login Shell Pitfall](references/proxy_conflict_reference.md) for details and examples. +**This is a frequent source of "it works interactively but fails in scripts" bugs.** SSH non-login shells don't load `~/.zshrc` (or `~/.bashrc` on Linux), so tools installed via nvm, Homebrew, uv, cargo, or any shell-level manager won't be in `$PATH`. Proxy env vars set in `~/.zshrc` also won't be loaded. + +This affects **all** remote commands run via `ssh user@host "command"`, including CI/CD pipelines, cron-triggered SSH, and Makefile remote targets. Prefix all remote commands with `source ~/.zshrc 2>/dev/null;` (macOS) or `source ~/.bashrc 2>/dev/null;` (Linux/WSL). + +**Common failure**: `ssh user@host "uv run ..."` or `ssh user@host "node ..."` returns `command not found` even though the command works in an interactive SSH session. + +See [references/proxy_conflict_reference.md § SSH Non-Login Shell Pitfall](references/proxy_conflict_reference.md) for details and examples. For Makefile targets that run remote commands: diff --git a/tunnel-doctor/references/proxy_conflict_reference.md b/tunnel-doctor/references/proxy_conflict_reference.md index 03f35e3b..34fed4cf 100644 --- a/tunnel-doctor/references/proxy_conflict_reference.md +++ b/tunnel-doctor/references/proxy_conflict_reference.md @@ -68,6 +68,14 @@ NO_PROXY="<shadowrocket-ip>" curl -s -X POST "http://<shadowrocket-ip>:8080/api/ curl --noproxy '*' -s --connect-timeout 2 "http://192.168.31.110:8080/api/read" | head -1 ``` +**Port conflict warning**: Shadowrocket's config API listens on port 8080 by default, which may conflict with other services (e.g., whisper.cpp server, development proxies). If the API returns unexpected content (HTML, JSON from another service), verify what is actually listening on the port: + +```bash +lsof -nP -iTCP:8080 | head -5 +``` + +If another service owns port 8080, you need to either stop that service or access the Shadowrocket API from a different device on the same network. + **Critical**: Use `--data-binary`, NOT `-d`. The `-d` flag URL-encodes the content, corrupting `#`, `=`, `&` and other characters in the config. This **destroys the entire configuration** — all rules, settings, and proxy groups are lost. The user must restore from backup. ```bash @@ -88,6 +96,31 @@ tun-excluded-routes = 10.0.0.0/8, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 19 Note: `100.64.0.0/10` is intentionally absent. +### Complete Working Reference Config for Tailscale Compatibility + +This is a validated reference showing the correct relationship between `skip-proxy`, `tun-excluded-routes`, and `[Rule]` for Tailscale coexistence: + +``` +[General] +# skip-proxy: bypass the HTTP proxy for these destinations (fixes browser 503) +# 100.64.0.0/10 MUST be here for browser access to Tailscale IPs +skip-proxy = 192.168.0.0/16, 10.0.0.0/8, 172.16.0.0/12, 100.64.0.0/10, localhost, *.local, captive.apple.com + +# tun-excluded-routes: CIDRs excluded from TUN routing (sent directly via physical interface) +# 100.64.0.0/10 must NOT be here — including it creates an en0 route that overrides Tailscale +tun-excluded-routes = 10.0.0.0/8, 127.0.0.0/8, 169.254.0.0/16, 172.16.0.0/12, 192.0.0.0/24, 192.0.2.0/24, 192.88.99.0/24, 192.168.0.0/16, 198.51.100.0/24, 203.0.113.0/24, 224.0.0.0/4, 255.255.255.255/32 + +[Rule] +# Tailscale traffic enters TUN but is passed through without proxying +IP-CIDR,100.64.0.0/10,DIRECT +# ... other rules ... +``` + +**Key points**: +- `skip-proxy` — YES, include `100.64.0.0/10` (browser bypass) +- `tun-excluded-routes` — NO, never include `100.64.0.0/10` (would hijack routing) +- `[Rule]` — YES, include `IP-CIDR,100.64.0.0/10,DIRECT` (TUN passthrough) + ## Clash / ClashX Pro ### The Fix @@ -151,12 +184,15 @@ export NO_PROXY=localhost,127.0.0.1,.ts.net,100.64.0.0/10,192.168.*,10.*,172.16. ### NO_PROXY Syntax Pitfalls -| Syntax | curl | Python requests | Node.js | Meaning | -|--------|------|-----------------|---------|---------| -| `.ts.net` | ✅ | ✅ | ✅ | Domain suffix match (correct) | -| `*.ts.net` | ❌ | ✅ | varies | Glob — curl does NOT support this | -| `100.64.0.0/10` | ✅ 7.86+ | ✅ 2.25+ | ❌ native | CIDR notation | -| `100.*` | ✅ | ✅ | ✅ | Too broad — covers public IPs `100.0-63.*` and `100.128-255.*` | +| Syntax | curl | Python requests | Go `net/http` | Node.js | Meaning | +|--------|------|-----------------|---------------|---------|---------| +| `.ts.net` | ✅ | ✅ | ✅ | ✅ | Domain suffix match (correct) | +| `*.ts.net` | ❌ | ✅ | ❌ | varies | Glob — curl and Go do NOT support this | +| `100.64.0.0/10` | ✅ 7.86+ | ✅ 2.25+ | ❌ | ❌ native | CIDR notation — Go silently ignores it | +| `100.*` | ✅ | ✅ | ❌ | ✅ | Too broad — covers public IPs `100.0-63.*` and `100.128-255.*` | +| `workstation-name` | ✅ | ✅ | ✅ | ✅ | Exact hostname match (safest for Go) | + +**Go `net/http` warning**: Go's proxy bypass logic (`httpproxy.Config.ProxyFunc`) does not implement CIDR matching. `NO_PROXY=100.64.0.0/10` is silently ignored — Go programs will still route traffic through the proxy. Use MagicDNS hostnames (e.g., `workstation-4090-wsl`) or explicit IPs (e.g., `100.101.102.103`) instead of CIDR ranges when Go programs need to bypass the proxy. **Key rule**: Always use `.ts.net` (leading dot, no asterisk) for domain suffix matching. This is the most portable syntax across all HTTP clients. @@ -282,9 +318,9 @@ Connection setup is slightly slower (~6s vs ~2s) because TUN routing has more ne ## General Principles -### Four Conflict Layers +### Five Conflict Layers -Proxy tools create conflicts at four independent layers on macOS. Layers 1-3 affect Tailscale connectivity; Layer 4 affects SSH git operations through the same proxy infrastructure: +Proxy tools create conflicts at five independent layers on macOS. Layers 1-3 affect Tailscale connectivity; Layer 4 affects SSH git operations; Layer 5 affects VM/container runtimes: | Layer | Setting | What it controls | Symptom when wrong | |-------|---------|------------------|--------------------| @@ -292,6 +328,7 @@ Proxy tools create conflicts at four independent layers on macOS. Layers 1-3 aff | 2. HTTP env vars | `http_proxy` / `NO_PROXY` | CLI tools (curl, wget, Python, Node.js) | `curl` times out, SSH works, browser works | | 3. System proxy | `skip-proxy` | Browser and system HTTP clients | Browser 503, `curl` works (both with/without proxy), SSH works | | 4. SSH ProxyCommand | `ProxyCommand connect -H` | SSH git operations (push/pull/clone) | `ssh -T` works, `git push` fails intermittently with `failed to begin relaying via HTTP` | +| 5. VM/Container proxy | Docker/OrbStack proxy config | `docker pull`, `docker build` | Host `curl` works, `docker pull` times out (TLS handshake timeout) | **Each layer is independent.** A fix at one layer doesn't help the others. You may need fixes at multiple layers simultaneously. @@ -398,4 +435,6 @@ If a proxy config change breaks Tailscale connectivity: sudo route delete -net 100.64.0.0/10 <gateway-ip> ``` +**Important**: Manually deleting a bad `en0` route with `sudo route delete` is only a temporary fix. Shadowrocket will re-add the route when the VPN connection is next reconnected or toggled. The only permanent fix is modifying the Shadowrocket configuration to remove `100.64.0.0/10` from `tun-excluded-routes` (it should never be there). + If `tun-excluded-routes` was modified, reverting it and restarting Shadowrocket will restore Tailscale's routing immediately. diff --git a/tunnel-doctor/scripts/quick_diagnose.py b/tunnel-doctor/scripts/quick_diagnose.py index 346ca8ca..a53b833b 100755 --- a/tunnel-doctor/scripts/quick_diagnose.py +++ b/tunnel-doctor/scripts/quick_diagnose.py @@ -167,6 +167,32 @@ def strict_tls_check(url: str, timeout: int) -> Dict[str, object]: } +def find_tailscale_utun() -> Optional[str]: + """Find which utun interface belongs to Tailscale (has a 100.x.x.x IP).""" + code, stdout, _ = run(["ifconfig"]) + if code != 0: + return None + current_iface = "" + for line in stdout.splitlines(): + # Interface header line (e.g., "utun7: flags=...") + m = re.match(r"^(\w+):", line) + if m: + current_iface = m.group(1) + # Look for Tailscale CGNAT IP on a utun interface + if current_iface.startswith("utun") and "inet 100." in line: + return current_iface + return None + + +def get_iface_mtu(iface: str) -> Optional[int]: + """Get MTU of a network interface.""" + code, stdout, _ = run(["ifconfig", iface]) + if code != 0: + return None + m = re.search(r"mtu\s+(\d+)", stdout) + return int(m.group(1)) if m else None + + def route_check(tailscale_ip: str) -> Dict[str, object]: code, stdout, stderr = run(["route", "-n", "get", tailscale_ip]) if code != 0: @@ -181,10 +207,24 @@ def route_check(tailscale_ip: str) -> Dict[str, object]: if line.startswith("gateway:"): gateway = line.split(":", 1)[1].strip() + # Identify which utun is Tailscale's and whether the route points to it + tailscale_utun = find_tailscale_utun() + route_mtu = get_iface_mtu(interface) if interface else None + is_tailscale_iface = (interface == tailscale_utun) if tailscale_utun else None + wrong_utun = ( + interface.startswith("utun") + and tailscale_utun is not None + and interface != tailscale_utun + ) + return { "ok": True, "interface": interface, "gateway": gateway, + "tailscale_utun": tailscale_utun or "", + "route_iface_mtu": route_mtu, + "is_tailscale_iface": is_tailscale_iface, + "wrong_utun": wrong_utun, "raw": stdout, } @@ -300,6 +340,10 @@ def build_report( route_info = route_check(tailscale_ip) if tailscale_ip else None if route_info and route_info["ok"]: iface = str(route_info["interface"]) + ts_utun = str(route_info.get("tailscale_utun", "")) + route_mtu = route_info.get("route_iface_mtu") + wrong_utun = route_info.get("wrong_utun", False) + if iface.startswith("en"): findings.append( { @@ -311,6 +355,23 @@ def build_report( ), } ) + elif wrong_utun: + mtu_hint = f" (MTU {route_mtu})" if route_mtu else "" + findings.append( + { + "level": "error", + "title": "Route points to wrong utun interface", + "detail": ( + f"route -n get {tailscale_ip} resolved to {iface}{mtu_hint}, " + f"but Tailscale is on {ts_utun}. " + f"Likely hitting Shadowrocket/VPN TUN (MTU 4064) instead of Tailscale (MTU 1280)." + ), + "fix": ( + "Check proxy TUN excluded-routes and rule ordering. " + "Ensure IP-CIDR,100.64.0.0/10,DIRECT is in proxy rules." + ), + } + ) summary = { "host": host, @@ -373,8 +434,21 @@ def print_human(report: Dict[str, object]) -> int: if route: if route.get("ok"): print("Tailscale Route Check") - print(f"- interface: {route.get('interface') or 'N/A'}") - print(f"- gateway: {route.get('gateway') or 'N/A'}") + print(f"- route interface: {route.get('interface') or 'N/A'}") + route_mtu = route.get("route_iface_mtu") + if route_mtu: + print(f" route iface MTU: {route_mtu}") + print(f"- gateway: {route.get('gateway') or 'N/A'}") + ts_utun = route.get("tailscale_utun") + if ts_utun: + print(f"- tailscale utun: {ts_utun}") + is_ts = route.get("is_tailscale_iface") + if is_ts is True: + print(" route → Tailscale utun: YES (correct)") + elif is_ts is False: + print(" route → Tailscale utun: NO (MISMATCH — see findings)") + else: + print("- tailscale utun: (not detected — is Tailscale running?)") print("") else: print("Tailscale Route Check") From 639a6d303e85d7cec48015cdd1f8538d539e5464 Mon Sep 17 00:00:00 2001 From: daymade <daymadev89@gmail.com> Date: Sun, 22 Mar 2026 23:40:38 +0800 Subject: [PATCH 011/137] feat(transcript-fixer): native AI correction as default mode (v1.3.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude IS the AI — when running inside Claude Code, use Claude's own language understanding for Stage 2 corrections instead of calling an external API. No API key needed by default. New capabilities in native mode: - Intelligent paragraph breaks at logical topic transitions - Filler word reduction (excessive repetition removal) - Interactive review with confidence-level tables - Context-aware judgment using full document context API mode (GLM) remains available for batch/automation use cases. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- .claude-plugin/marketplace.json | 29 +++++++-- transcript-fixer/SKILL.md | 104 +++++++++++++++++++++----------- 2 files changed, 93 insertions(+), 40 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index dd8dc895..76986aa5 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -6,7 +6,7 @@ }, "metadata": { "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, macOS programmatic window screenshot capture workflows, and verified Scrapling CLI installation and web extraction workflows", - "version": "1.39.0", + "version": "1.40.0", "homepage": "https://github.com/daymade/claude-code-skills" }, "plugins": [ @@ -15,7 +15,7 @@ "description": "Essential meta-skill for creating effective Claude Code skills with initialization scripts, validation, packaging, marketplace registration, and privacy best practices", "source": "./", "strict": false, - "version": "1.5.1", + "version": "1.6.0", "category": "developer-tools", "keywords": [ "skill-creation", @@ -289,10 +289,10 @@ }, { "name": "transcript-fixer", - "description": "Corrects speech-to-text (ASR/STT) transcription errors in meeting notes, lecture recordings, interviews, and voice memos through dictionary-based rules and AI corrections. Supports Chinese domain names, AI fallback to Claude Code, and iterative dictionary building. Use when users mention transcript correction, ASR errors, speech-to-text mistakes, homophone errors, or working with transcription files", + "description": "Corrects speech-to-text (ASR/STT) transcription errors using dictionary rules and native Claude AI corrections (no API key needed by default). Supports intelligent paragraph breaks, filler word reduction, interactive review, Chinese domain names, and iterative dictionary building. Use when users mention transcript correction, ASR errors, speech-to-text mistakes, homophone errors, or working with transcription files", "source": "./", "strict": false, - "version": "1.2.0", + "version": "1.3.0", "category": "productivity", "keywords": [ "transcription", @@ -919,6 +919,27 @@ "skills": [ "./scrapling-skill" ] + }, + { + "name": "asr-transcribe-to-text", + "description": "Transcribe audio and video files to text using a remote ASR service (Qwen3-ASR or OpenAI-compatible endpoint). Extracts audio from video, sends to configurable ASR endpoint, outputs clean text. Use when the user wants to transcribe recordings, convert audio/video to text, do speech-to-text, or mentions ASR, Qwen ASR, 转录, 语音转文字, 录音转文字, or has a meeting recording, lecture, interview, or screen recording to transcribe.", + "source": "./", + "strict": false, + "version": "1.0.0", + "category": "productivity", + "keywords": [ + "asr", + "transcription", + "speech-to-text", + "qwen", + "audio", + "video", + "vllm", + "ffmpeg" + ], + "skills": [ + "./asr-transcribe-to-text" + ] } ] } diff --git a/transcript-fixer/SKILL.md b/transcript-fixer/SKILL.md index e5f532d3..11060cb5 100644 --- a/transcript-fixer/SKILL.md +++ b/transcript-fixer/SKILL.md @@ -29,20 +29,36 @@ powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | ie ## Quick Start -**Recommended: Use Enhanced Wrapper** (auto-detects API key, opens HTML diff): +**Default: Native AI Correction (no API key needed)** + +When invoked from Claude Code, the skill uses a two-phase approach: +1. **Dictionary phase** (script): Apply 700+ learned correction rules instantly +2. **AI phase** (Claude native): Claude reads the text directly and fixes ASR errors, adds paragraph breaks, removes filler words ```bash # First time: Initialize database uv run scripts/fix_transcription.py --init -# Process transcript with enhanced UX -uv run scripts/fix_transcript_enhanced.py input.md --output ./corrected +# Phase 1: Dictionary corrections (instant, free) +uv run scripts/fix_transcription.py --input meeting.md --stage 1 ``` -The enhanced wrapper automatically: -- Detects GLM API key from shell configs (checks lines near `ANTHROPIC_BASE_URL`) -- Moves output files to specified directory -- Opens HTML visual diff in browser for immediate feedback +After Stage 1, Claude should: +1. Read the Stage 1 output in ~3000-char chunks +2. Identify ASR errors (homophones, technical terms, broken sentences) +3. Present corrections in a table for user review (high/medium confidence) +4. Apply confirmed corrections and save stable patterns to dictionary +5. Optionally: add paragraph breaks and remove excessive filler words + +**Alternative: API-Based Batch Processing** (for automation or large volumes): + +```bash +# Set API key for automated AI corrections +export GLM_API_KEY="<api-key>" # From https://open.bigmodel.cn/ + +# Run full pipeline (dict + API AI + diff report) +uv run scripts/fix_transcript_enhanced.py input.md --output ./corrected +``` **Timestamp repair**: ```bash @@ -58,25 +74,9 @@ uv run scripts/split_transcript_sections.py meeting.txt \ --rebase-to-zero ``` -**Alternative: Use Core Script Directly**: - -```bash -# 1. Set API key (if not auto-detected) -export GLM_API_KEY="<api-key>" # From https://open.bigmodel.cn/ - -# 2. Add common corrections (5-10 terms) -uv run scripts/fix_transcription.py --add "错误词" "正确词" --domain general - -# 3. Run full correction pipeline -uv run scripts/fix_transcription.py --input meeting.md --stage 3 - -# 4. Review learned patterns after 3-5 runs -uv run scripts/fix_transcription.py --review-learned -``` - **Output files**: - `*_stage1.md` - Dictionary corrections applied -- `*_stage2.md` - AI corrections applied (final version) +- `*_corrected.txt` - Final version (native mode) or `*_stage2.md` (API mode) - `*_对比.html` - Visual diff (open in browser for best experience) **Generate word-level diff** (recommended for reviewing corrections): @@ -116,14 +116,15 @@ This creates an HTML file showing word-by-word differences with clear highlighti ## Core Workflow -Three-stage pipeline stores corrections in `~/.transcript-fixer/corrections.db`: +Two-phase pipeline stores corrections in `~/.transcript-fixer/corrections.db`: 1. **Initialize** (first time): `uv run scripts/fix_transcription.py --init` 2. **Add domain corrections**: `--add "错误词" "正确词" --domain <domain>` -3. **Process transcript**: `--input file.md --stage 3` -4. **Review learned patterns**: `--review-learned` and `--approve` high-confidence suggestions +3. **Phase 1 — Dictionary**: `--input file.md --stage 1` (instant, free) +4. **Phase 2 — AI Correction**: Claude reads output and fixes ASR errors natively (default), or use `--stage 3` with `GLM_API_KEY` for API mode +5. **Save stable patterns**: `--add "错误词" "正确词"` after each fix session +6. **Review learned patterns**: `--review-learned` and `--approve` high-confidence suggestions -**Stages**: Dictionary (instant, free) → AI via GLM API (parallel) → Full pipeline **Domains**: `general`, `embodied_ai`, `finance`, `medical`, or custom names including Chinese (e.g., `火星加速器`, `具身智能`) **Learning**: Patterns appearing ≥3 times at ≥80% confidence move from AI to dictionary @@ -182,14 +183,45 @@ If you understand the risks and still want to add a flagged rule: uv run scripts/fix_transcription.py --add "仿佛" "反复" --domain general --force ``` -## AI Fallback Strategy +## Native AI Correction (Default Mode) + +**Claude IS the AI.** When running inside Claude Code, use Claude's own language understanding for Stage 2 corrections instead of calling an external API. This is the default behavior — no API key needed. + +### Workflow + +1. **Run Stage 1** (dictionary): `uv run scripts/fix_transcription.py --input file.md --stage 1` +2. **Read the text** in ~3000-character chunks (use `cut -c<start>-<end>` for single-line files) +3. **Identify ASR errors** — look for: + - Homophone errors (同音字): "上海文" → "上下文", "扩种" → "扩充" + - Broken sentence boundaries: "很大程。路上" → "很大程度上" + - Technical terms: "Web coding" → "Vibe Coding" + - Missing/extra characters: "沉沉默" → "沉默" +4. **Present corrections** in a table with confidence levels before applying: + - High confidence: clear ASR errors with unambiguous corrections + - Medium confidence: context-dependent, need user confirmation +5. **Apply corrections** to a copy of the file (never modify the original) +6. **Save stable patterns** to dictionary: `--add "错误词" "正确词" --domain general` +7. **Generate word diff**: `uv run scripts/generate_word_diff.py original.md corrected.md diff.html` + +### Enhanced AI Capabilities (Native Mode Only) + +Native mode can do things the API mode cannot: + +- **Intelligent paragraph breaks**: Add `\n\n` at logical topic transitions in continuous text +- **Filler word reduction**: Remove excessive repetition (这个这个这个 → 这个, 都都都都 → 都) +- **Interactive review**: Present corrections for user confirmation before applying +- **Context-aware judgment**: Use full document context to resolve ambiguous errors + +### When to Use API Mode Instead + +Use `GLM_API_KEY` + Stage 3 for: +- Batch processing multiple files in automation +- When Claude Code is not available (standalone script usage) +- Consistent reproducible processing without interactive review -When GLM API is unavailable (503, network issues), the script outputs `[CLAUDE_FALLBACK]` marker. +### Legacy Fallback Marker -Claude Code should then: -1. Analyze the text directly for ASR errors -2. Fix using Edit tool -3. **MUST save corrections to dictionary** with `--add` +When the script outputs `[CLAUDE_FALLBACK]` (GLM API error), switch to native mode automatically. ## Database Operations @@ -209,8 +241,8 @@ sqlite3 ~/.transcript-fixer/corrections.db "SELECT value FROM system_config WHER | Stage | Description | Speed | Cost | |-------|-------------|-------|------| | 1 | Dictionary only | Instant | Free | -| 2 | AI only | ~10s | API calls | -| 3 | Full pipeline | ~10s | API calls | +| 1 + Native | Dictionary + Claude AI (default) | ~1min | Free | +| 3 | Dictionary + API AI + diff report | ~10s | API calls | ## Bundled Resources From ee38ae41b8eb461cf9d65ac630d0f3df09010b4b Mon Sep 17 00:00:00 2001 From: daymade <daymadev89@gmail.com> Date: Mon, 23 Mar 2026 00:03:45 +0800 Subject: [PATCH 012/137] feat: add asr-transcribe-to-text skill + optimize skill-creator with AskUserQuestion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New skill: asr-transcribe-to-text (v1.0.0) - Transcribe audio/video via configurable ASR endpoint (Qwen3-ASR default) - Persistent config in CLAUDE_PLUGIN_DATA (endpoint, model, proxy bypass) - Single-request-first strategy (empirically proven: 55min in one request) - Fallback overlap-merge script for very long audio (18min chunks, 2min overlap) - AskUserQuestion at config init, health check failure, and output verification skill-creator optimization (v1.5.1 → v1.6.0) - Add AskUserQuestion best practices section (Re-ground/Simplify/Recommend/Options) - Inject structured decision points at 8 key workflow stages - Inspired by gstack's atomic question pattern Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- asr-transcribe-to-text/SKILL.md | 197 +++++++++++++++++ .../references/overlap_merge_strategy.md | 56 +++++ .../scripts/overlap_merge_transcribe.py | 200 +++++++++++++++++ skill-creator/SKILL.md | 207 +++++++++++++++++- 4 files changed, 655 insertions(+), 5 deletions(-) create mode 100644 asr-transcribe-to-text/SKILL.md create mode 100644 asr-transcribe-to-text/references/overlap_merge_strategy.md create mode 100644 asr-transcribe-to-text/scripts/overlap_merge_transcribe.py diff --git a/asr-transcribe-to-text/SKILL.md b/asr-transcribe-to-text/SKILL.md new file mode 100644 index 00000000..15af8f58 --- /dev/null +++ b/asr-transcribe-to-text/SKILL.md @@ -0,0 +1,197 @@ +--- +name: asr-transcribe-to-text +description: Transcribe audio and video files to text using a remote ASR service (Qwen3-ASR or OpenAI-compatible endpoint). Extracts audio from video, sends to configurable ASR endpoint, outputs clean text. Use when the user wants to transcribe recordings, convert audio/video to text, do speech-to-text, or mentions ASR, Qwen ASR, 转录, 语音转文字, 录音转文字, or has a meeting recording, lecture, interview, or screen recording to transcribe. +argument-hint: [audio-or-video-file-path] +--- + +# ASR Transcribe to Text + +Transcribe audio/video files to text using a configurable ASR endpoint (default: Qwen3-ASR-1.7B via vLLM). Configuration persists across sessions in `${CLAUDE_PLUGIN_DATA}/config.json`. + +## Step 0: Load or Initialize Configuration + +```bash +cat "${CLAUDE_PLUGIN_DATA}/config.json" 2>/dev/null +``` + +**If config exists**, read the values and proceed to Step 1. + +**If config does not exist** (first run), use **AskUserQuestion**: + +``` +First-time setup for ASR transcription. +I need to know where your ASR service is running so I can send audio to it. + +RECOMMENDATION: Use the defaults below if you have Qwen3-ASR on a 4090 via Tailscale. + +Q1: ASR Endpoint URL? + A) http://workstation-4090-wsl:8002/v1/audio/transcriptions (Default — Qwen3-ASR vLLM via Tailscale) + B) http://localhost:8002/v1/audio/transcriptions (Local machine) + C) Let me enter a custom URL + +Q2: Does your network have an HTTP proxy that might intercept LAN/Tailscale traffic? + A) Yes — add --noproxy to bypass it (Recommended if you use Shadowrocket/Clash/corporate proxy) + B) No — direct connection is fine +``` + +Save the config: +```bash +mkdir -p "${CLAUDE_PLUGIN_DATA}" +python3 -c " +import json +config = { + 'endpoint': 'USER_PROVIDED_ENDPOINT', + 'model': 'USER_PROVIDED_MODEL_OR_DEFAULT', + 'noproxy': True, # or False based on user answer + 'max_timeout': 900 +} +with open('${CLAUDE_PLUGIN_DATA}/config.json', 'w') as f: + json.dump(config, f, indent=2) +print('Config saved.') +" +``` + +## Step 1: Validate Input and Check Service Health + +Read config and health-check in a single command (shell variables don't persist across Bash calls): + +```bash +python3 -c " +import json, subprocess, sys +with open('${CLAUDE_PLUGIN_DATA}/config.json') as f: + cfg = json.load(f) +base = cfg['endpoint'].rsplit('/audio/', 1)[0] +noproxy = ['--noproxy', '*'] if cfg.get('noproxy', True) else [] +result = subprocess.run( + ['curl', '-s', '--max-time', '10'] + noproxy + [f'{base}/models'], + capture_output=True, text=True +) +if result.returncode != 0 or not result.stdout.strip(): + print(f'HEALTH CHECK FAILED', file=sys.stderr) + print(f'Endpoint: {base}/models', file=sys.stderr) + print(f'stdout: {result.stdout[:200]}', file=sys.stderr) + print(f'stderr: {result.stderr[:200]}', file=sys.stderr) + sys.exit(1) +else: + print(f'Service healthy: {base}') + print(f'Model: {cfg[\"model\"]}') +" +``` + +**If health check fails**, use **AskUserQuestion**: + +``` +ASR service at [endpoint] is not responding. + +Options: +A) Diagnose — check network, Tailscale, and service status step by step +B) Reconfigure — the endpoint URL might be wrong, let me re-enter it +C) Try anyway — send the transcription request and see what happens +D) Abort — I'll fix the service manually and come back later +``` + +For option A, diagnose in order: +1. Network: `ping -c 1 HOST` or `tailscale status | grep HOST` +2. Service: `tailscale ssh USER@HOST "curl -s localhost:PORT/v1/models"` +3. Proxy: retry with `--noproxy '*'` toggled + +## Step 2: Extract Audio (if input is video) + +For video files (mp4, mov, mkv, avi, webm), extract audio as 16kHz mono MP3: + +```bash +ffmpeg -i INPUT_VIDEO -vn -acodec libmp3lame -q:a 4 -ar 16000 -ac 1 OUTPUT.mp3 -y +``` + +For audio files (mp3, wav, m4a, flac, ogg), use directly — no conversion needed. + +Get duration for progress estimation: +```bash +ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 INPUT_FILE +``` + +## Step 3: Transcribe — Single Request First + +**Always try full-length single request first.** Chunking causes sentence truncation at every split boundary — the model forces the last sentence to close and loses words. Single request = zero truncation + fastest speed. + +The Qwen3-ASR paper's "20-minute limit" is a training benchmark, not an inference hard limit. Empirically verified: 55 minutes transcribed in a single 76-second request on 4090 24GB. + +```bash +python3 -c " +import json, subprocess, sys, os, tempfile +with open('${CLAUDE_PLUGIN_DATA}/config.json') as f: + cfg = json.load(f) +noproxy = ['--noproxy', '*'] if cfg.get('noproxy', True) else [] +timeout = str(cfg.get('max_timeout', 900)) +audio_file = 'AUDIO_FILE_PATH' # replace with actual path +output_json = tempfile.mktemp(suffix='.json', prefix='asr_') + +result = subprocess.run( + ['curl', '-s', '--max-time', timeout] + noproxy + [ + cfg['endpoint'], + '-F', f'file=@{audio_file}', + '-F', f'model={cfg[\"model\"]}', + '-o', output_json + ], capture_output=True, text=True +) + +with open(output_json) as f: + data = json.load(f) +if 'text' not in data: + print(f'ERROR: {json.dumps(data)[:300]}', file=sys.stderr) + sys.exit(1) +text = data['text'] +duration = data.get('usage', {}).get('seconds', 0) +print(f'Transcribed: {len(text)} chars, {duration}s audio', file=sys.stderr) +print(text) +os.unlink(output_json) +" > OUTPUT.txt +``` + +**Performance reference**: ~400 characters per minute for Chinese speech; rates vary by language. Qwen3-ASR supports 52 languages including Chinese dialects, English, Japanese, Korean, and more. + +## Step 4: Verify and Confirm Output + +After transcription, verify quality: +1. Confirm the response contains a `text` field (not an error message) +2. Check character count is plausible for the audio duration (~400 chars/min for Chinese) +3. Show the user the first ~200 characters as a preview + +If the output looks wrong (empty, garbled, or error), use **AskUserQuestion**: + +``` +Transcription may have an issue: +- Expected: ~[N] chars for [M] minutes of audio +- Got: [actual chars] chars +- Preview: "[first 100 chars...]" + +Options: +A) Save as-is — the output looks fine to me +B) Retry with fallback — split into chunks and merge (handles long audio / OOM) +C) Reconfigure — try a different model or endpoint +D) Abort — something is wrong with the service +``` + +If output is good, save as `.txt` alongside the original file or to user-specified location. + +## Step 5: Fallback — Overlap-Merge for Very Long Audio + +If single request fails (timeout, OOM, HTTP error), fall back to chunked transcription with overlap merging: + +```bash +python3 ${CLAUDE_PLUGIN_ROOT}/scripts/overlap_merge_transcribe.py \ + --config "${CLAUDE_PLUGIN_DATA}/config.json" \ + INPUT_AUDIO OUTPUT.txt +``` + +This splits into 18-minute chunks with 2-minute overlap, then merges using punctuation-stripped fuzzy matching. See [references/overlap_merge_strategy.md](references/overlap_merge_strategy.md) for the algorithm details. + +## Reconfigure + +To change the ASR endpoint, model, or proxy settings: + +```bash +rm "${CLAUDE_PLUGIN_DATA}/config.json" +``` + +Then re-run Step 0 to collect new values via AskUserQuestion. diff --git a/asr-transcribe-to-text/references/overlap_merge_strategy.md b/asr-transcribe-to-text/references/overlap_merge_strategy.md new file mode 100644 index 00000000..e005be0e --- /dev/null +++ b/asr-transcribe-to-text/references/overlap_merge_strategy.md @@ -0,0 +1,56 @@ +# Overlap-Merge Strategy: Why and How + +## The Problem with Naive Chunking + +When ASR transcribes audio in chunks, each chunk's last sentence gets **forcibly truncated**. The model closes the sentence at the chunk boundary even if the speaker is mid-sentence. + +**Real example from testing (5-minute chunks):** + +| Version | Text at boundary | +|---------|-----------------| +| 5min chunk ending | "...靠的就。" (truncated) | +| Continuous 10min | "...靠的就是其中一次战略会。" (complete) | +| Next 5min chunk start | "如果不这么啃，这个业务..." (picks up but gap exists) | + +Concatenating these chunks produces: "靠的就。如果不这么啃..." — losing "是其中一次战略会" entirely. + +## Why Exact String Matching Fails + +The same 2-minute audio segment transcribed in two different contexts (end of chunk A vs start of chunk B) produces **different punctuation**: + +- Chunk A: "可能三年AI的进化" +- Chunk B: "可能。三年AI的进化" + +The words are identical, but punctuation differs because the model's sentence boundary decisions depend on surrounding context. Exact string matching finds zero overlap. + +## The Solution: Punctuation-Stripped Fuzzy Matching + +1. Strip all punctuation from both the tail of chunk A and the head of chunk B +2. Find the longest common substring in the stripped versions +3. Use chunk B's version at the merge point (because chunk A's ending is truncated while chunk B has the complete sentence) + +## Optimal Parameters (Empirically Determined) + +| Parameter | Value | Rationale | +|-----------|-------|-----------| +| Chunk duration | 18 min (1080s) | Safe margin under 20min paper benchmark; 4090 24GB handles much more | +| Overlap duration | 2 min (120s) | ~800 chars overlap region; enough for reliable fuzzy matching | +| Min match length | 15 chars | Filters false positives while catching real overlaps | +| Search window | 600 chars | Covers the overlap region with margin | + +## When to Use This Fallback + +Only use overlap-merge when the single full-length request fails. Reasons it might fail: +- Audio longer than ~2 hours (untested territory, may OOM on 24GB VRAM) +- GPU memory pressure from other processes +- Network timeout (curl max-time exceeded) + +For audio under 1 hour, always try single request first — it's faster, simpler, and produces the best quality. + +## Empirical Comparison (55-minute AI course recording) + +| Strategy | Segments | Boundaries | Total chars | Quality | +|----------|----------|------------|-------------|---------| +| 12x5min direct concat | 12 | 11 cuts | 23,060 | 11 truncated sentences | +| 4x18min overlap merge | 4 | 3 merges | 22,781 | Zero truncation | +| 1x55min single request | 1 | 0 | 22,889 | Perfect (best) | diff --git a/asr-transcribe-to-text/scripts/overlap_merge_transcribe.py b/asr-transcribe-to-text/scripts/overlap_merge_transcribe.py new file mode 100644 index 00000000..b7305756 --- /dev/null +++ b/asr-transcribe-to-text/scripts/overlap_merge_transcribe.py @@ -0,0 +1,200 @@ +# /// script +# requires-python = ">=3.9" +# /// +""" +Overlap-merge transcription for long audio files. + +Splits audio into 18-minute chunks with 2-minute overlap, transcribes each chunk +via a configurable ASR endpoint, then merges using punctuation-stripped fuzzy +matching to eliminate sentence truncation at boundaries. + +Usage: + python3 scripts/overlap_merge_transcribe.py INPUT_AUDIO OUTPUT.txt --config CONFIG.json + python3 scripts/overlap_merge_transcribe.py INPUT_AUDIO OUTPUT.txt --endpoint URL --model MODEL +""" + +import argparse +import json +import os +import re +import subprocess +import sys +import tempfile + + +def get_duration(audio_path: str) -> float: + result = subprocess.run( + ["ffprobe", "-v", "error", "-show_entries", "format=duration", + "-of", "default=noprint_wrappers=1:nokey=1", audio_path], + capture_output=True, text=True + ) + return float(result.stdout.strip()) + + +def split_audio(audio_path: str, chunk_dir: str, chunk_duration: int, overlap: int) -> list[tuple[int, int, str]]: + """Split audio into overlapping chunks. Returns list of (start_sec, duration_sec, chunk_path).""" + total = int(get_duration(audio_path)) + chunks = [] + start = 0 + + while start < total: + duration = min(chunk_duration, total - start) + chunk_path = os.path.join(chunk_dir, f"chunk_{len(chunks):02d}.mp3") + + subprocess.run( + ["ffmpeg", "-i", audio_path, "-ss", str(start), "-t", str(duration), + "-acodec", "copy", chunk_path, "-y"], + capture_output=True + ) + chunks.append((start, duration, chunk_path)) + print(f" Chunk {len(chunks)-1}: {start//60}:{start%60:02d} - {(start+duration)//60}:{(start+duration)%60:02d}", file=sys.stderr) + + start += duration - overlap + if start + duration >= total and duration == chunk_duration: + start = total - duration # ensure last chunk covers the end + if start <= chunks[-1][0]: + break + + return chunks + + +def transcribe(audio_path: str, endpoint: str, model: str, noproxy: bool = True) -> str: + """Send audio to ASR endpoint and return text.""" + noproxy_args = ["--noproxy", "*"] if noproxy else [] + result = subprocess.run( + ["curl", "-s", "--max-time", "600"] + noproxy_args + [ + endpoint, + "-F", f"file=@{audio_path}", + "-F", f"model={model}" + ], + capture_output=True, text=True + ) + data = json.loads(result.stdout) + return data["text"] + + +def strip_punct(text: str) -> str: + """Remove all punctuation, keep only CJK chars, letters, and digits.""" + return re.sub(r'[^\w\u4e00-\u9fff]', '', text) + + +def fuzzy_merge(text_a: str, text_b: str, search_chars: int = 600, min_match: int = 15) -> str: + """ + Merge two overlapping transcription segments using punctuation-stripped fuzzy matching. + + The ASR model produces slightly different punctuation for the same audio segment + across different runs, so exact string matching fails. By stripping punctuation + before matching, we find the true overlap region reliably. + + Uses text_b's version at the merge point because text_a truncates its final sentence + while text_b has the complete version. + """ + tail_a_clean = strip_punct(text_a[-search_chars:]) + text_b_clean = strip_punct(text_b) + + best_match_len = 0 + best_b_clean_end = 0 + + # Search for longest matching substring (punctuation-stripped) + for start in range(len(tail_a_clean)): + substr = tail_a_clean[start:start + min_match] + if len(substr) < min_match: + break + pos = text_b_clean.find(substr) + if pos >= 0: + # Extend the match as far as possible + match_len = min_match + while (start + match_len < len(tail_a_clean) + and pos + match_len < len(text_b_clean) + and tail_a_clean[start + match_len] == text_b_clean[pos + match_len]): + match_len += 1 + + if match_len > best_match_len: + best_match_len = match_len + best_b_clean_end = pos + match_len + best_a_clean_start = start + + if best_match_len >= min_match: + # Map clean positions back to raw text positions + # For text_a: find where the match starts in raw text + a_offset = len(text_a) - search_chars + clean_count = 0 + a_cut_pos = len(text_a) + for idx, ch in enumerate(text_a[-search_chars:]): + if strip_punct(ch): + clean_count += 1 + if clean_count > best_a_clean_start: + a_cut_pos = a_offset + idx + break + + # For text_b: find where the match ends in raw text + clean_count = 0 + b_start_pos = 0 + for idx, ch in enumerate(text_b): + if strip_punct(ch): + clean_count += 1 + if clean_count >= best_b_clean_end: + b_start_pos = idx + 1 + break + + print(f" Merged: {best_match_len} chars matched (punct-stripped)", file=sys.stderr) + return text_a[:a_cut_pos] + text_b[b_start_pos:] + else: + print(f" Warning: no overlap found ({best_match_len} chars), concatenating directly", file=sys.stderr) + return text_a + text_b + + +def main(): + parser = argparse.ArgumentParser(description="Overlap-merge ASR transcription") + parser.add_argument("input", help="Input audio/video file") + parser.add_argument("output", help="Output text file") + parser.add_argument("--config", help="Path to config.json (from CLAUDE_PLUGIN_DATA)") + parser.add_argument("--endpoint", default="http://workstation-4090-wsl:8002/v1/audio/transcriptions", help="ASR endpoint URL") + parser.add_argument("--model", default="Qwen/Qwen3-ASR-1.7B", help="Model name") + parser.add_argument("--noproxy", action="store_true", default=True, help="Use --noproxy with curl") + parser.add_argument("--chunk-duration", type=int, default=1080, help="Chunk duration in seconds (default: 1080 = 18min)") + parser.add_argument("--overlap", type=int, default=120, help="Overlap duration in seconds (default: 120 = 2min)") + args = parser.parse_args() + + # Load config from file if provided, otherwise use CLI args + if args.config and os.path.exists(args.config): + with open(args.config) as f: + cfg = json.load(f) + args.endpoint = cfg.get("endpoint", args.endpoint) + args.model = cfg.get("model", args.model) + args.noproxy = cfg.get("noproxy", args.noproxy) + + print(f"Input: {args.input}", file=sys.stderr) + total_duration = get_duration(args.input) + print(f"Duration: {total_duration:.0f}s ({total_duration/60:.1f}min)", file=sys.stderr) + + with tempfile.TemporaryDirectory() as chunk_dir: + # Split + print(f"\nSplitting into {args.chunk_duration}s chunks with {args.overlap}s overlap...", file=sys.stderr) + chunks = split_audio(args.input, chunk_dir, args.chunk_duration, args.overlap) + print(f"Created {len(chunks)} chunks\n", file=sys.stderr) + + # Transcribe each chunk + texts = [] + for i, (start, dur, path) in enumerate(chunks): + print(f"Transcribing chunk {i} ({start//60}:{start%60:02d})...", end=" ", file=sys.stderr, flush=True) + text = transcribe(path, args.endpoint, args.model, args.noproxy) + texts.append(text) + print(f"{len(text)} chars", file=sys.stderr) + + # Merge + print(f"\nMerging {len(texts)} segments...", file=sys.stderr) + merged = texts[0] + for i in range(1, len(texts)): + print(f" Merging chunk {i-1} + {i}:", file=sys.stderr) + merged = fuzzy_merge(merged, texts[i]) + + # Save + with open(args.output, "w", encoding="utf-8") as f: + f.write(merged) + + print(f"\nDone! {len(merged)} chars saved to {args.output}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/skill-creator/SKILL.md b/skill-creator/SKILL.md index d2fd97a3..6ba0e0ff 100644 --- a/skill-creator/SKILL.md +++ b/skill-creator/SKILL.md @@ -41,6 +41,23 @@ So please pay attention to context cues to understand how to phrase your communi It's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it. +### Using AskUserQuestion (Critical — Read This) + +**Use the AskUserQuestion tool aggressively at every decision point.** Do not ask open-ended text questions in conversation when structured choices exist. This is the single biggest UX improvement you can make — users juggle multiple windows and may not have looked at this conversation in 20 minutes. + +**Every AskUserQuestion MUST follow this structure:** + +1. **Re-ground**: State the skill name, current phase, and what just happened (1-2 sentences). The user may have context-switched away. +2. **Simplify**: Explain the decision in plain language. No function names or internal jargon. Say what it DOES, not what it's called. +3. **Recommend**: Lead with your recommendation and a one-line reason why. If options involve effort, show both scales: `(human: ~X min / Claude: ~Y min)`. +4. **Options**: Provide 2-4 concrete, lettered choices. Each option should be a clear action, not an abstract concept. + +**Rules:** +- One decision per question — never batch unrelated choices +- Provide an escape hatch ("Other" is always implicit in AskUserQuestion) +- Accept the user's choice — nudge on tradeoffs but never refuse to proceed +- Skip the question if there's an obvious answer with no tradeoffs (just state what you'll do) + --- ## Creating a skill @@ -52,7 +69,83 @@ Start by understanding the user's intent. The current conversation might already 1. What should this skill enable Claude to do? 2. When should this skill trigger? (what user phrases/contexts) 3. What's the expected output format? -4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide. +4. Should we set up test cases to verify the skill works? + +After extracting answers from conversation history (or asking questions 1-3), use **AskUserQuestion** to confirm the skill type and testing strategy: + +``` +Creating skill "[name]" — here's what I understand so far: +- Purpose: [1-sentence summary] +- Triggers on: [key phrases] +- Output: [format] + +RECOMMENDATION: [Objective/Subjective/Hybrid] skill → [suggested testing approach] + +Options: +A) Objective output (files, code, data) — set up automated test cases (Recommended if output is verifiable) +B) Subjective output (writing, design) — qualitative human review only +C) Hybrid — automated checks for structure, human review for quality +D) Skip testing for now — just build the skill and iterate by feel +``` + +This upfront classification drives the entire evaluation strategy downstream. Get it right here to avoid wasted effort later. + +### Prior Art Research (Do Not Skip) + +The user's private methodology — their domain rules, workflow decisions, competitive edge — is what makes a skill valuable. No public repo can provide that. But the user shouldn't waste time reinventing infrastructure (API clients, auth flows, rate limiting) when mature tools exist. Prior art research finds building blocks for the infrastructure layer so the skill can focus on encoding the user's unique methodology. + +**Search these channels in order** (use subagents for 4-8 in parallel): + +| Priority | Channel | What to search | How | +|----------|---------|---------------|-----| +| 1 | **Conversation history** | User's proven workflows, verified API patterns, corrections made during debugging | Grep recent conversations for the service/API name | +| 2 | **Local documents & SOPs** | User's private methodology, runbooks, existing skills | Search project directory, `~/.claude/CLAUDE.md`, `~/.claude/references/` | +| 3 | **Installed plugins & MCPs** | Already-integrated tools | Check `~/.claude/plugins/`, parse `installed_plugins.json`; check `~/.claude.json` for configured MCP servers | +| 4 | **skills.sh** | Community skills | `WebFetch https://skills.sh/?q=<keyword>` | +| 5 | **Anthropic official plugins** | Official/partner plugins | `WebFetch https://github.com/anthropics/claude-plugins-official/tree/main/plugins` and `external_plugins` directory | +| 6 | **MCP servers on GitHub** | Existing MCP servers for the same API | `WebSearch "<service-name> MCP server site:github.com"` | +| 7 | **Official API docs** | The target service's own documentation | `WebSearch "<service-name> API documentation"` or `WebFetch` the docs URL | +| 8 | **npm / PyPI** | SDK or CLI packages | `npm search <keyword>` or `curl https://pypi.org/pypi/<name>/json` | + +Channels 1-3 surface the user's own proven patterns and existing integrations. Channels 4-8 find public infrastructure. The user's private SOP always takes precedence — public tools are building blocks, not replacements. In competitive domains (finance, trading, proprietary operations), the valuable methodology will never be public. + +**If a public MCP server or skill is found, clone it and verify — don't trust the README:** + +1. **Read the actual source code** — many projects have polished READMEs on hollow codebases +2. **Verify auth method** — does it match how the API actually authenticates? (X-Api-Key headers vs Bearer vs OAuth — many get this wrong) +3. **Check test coverage** — zero tests = prototype, not production-grade +4. **Check maintenance** — last commit date, open issue count, response to bug reports +5. **Check environment compatibility** — proxy/network assumptions, hardcoded DNS/IPs, region locks +6. **Check license** — MIT/Apache is fine; GPL/SSPL may conflict with proprietary use +7. **Check dependency weight** — huge dependency trees create conflict and security surface + +**Decision matrix:** + +| Finding | Action | +|---------|--------| +| Mature MCP/SDK handles the infrastructure | **Adopt it, build on top** — install the MCP, then build the skill as a workflow layer encoding the user's methodology | +| Partial MCP or SDK exists | **Extend** — use for infrastructure, fill gaps in the skill | +| Public skill covers the same domain | **Use for structural inspiration only** — public skills in competitive domains are generic by definition. The user's edge is their private SOP | +| Nothing public exists | **Build from scratch** — validate API access patterns work (auth, endpoints, proxy) before writing the full skill | +| Integration cost > build cost | **Build it** — a 2-hour custom implementation you own beats a "mature" tool with integration friction and upstream risk | + +After research completes, present findings via **AskUserQuestion**: + +``` +Research complete for "[skill-name]". Here's what I found: + +[1-2 sentence summary of what exists publicly] + +RECOMMENDATION: [ADOPT / EXTEND / BUILD] because [one-line reason] + +Options: +A) Adopt [tool/MCP X] for infrastructure, build methodology layer on top (Recommended) +B) Extend [partial tool Y] — use what works, fill gaps in the skill +C) Build from scratch — nothing found matches well enough +D) Show me the detailed findings before I decide +``` + +When in doubt, bias toward adopting mature infrastructure for the plumbing layer and building custom logic for the methodology layer — that's where the value lives. ### Interview and Research @@ -342,7 +435,26 @@ Anthropic has wrote skill authoring best practices, you SHOULD retrieve it befor ### Test Cases -After writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] "Here are a few test cases I'd like to try. Do these look right, or do you want to add more?" Then run them. +After writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Present them via **AskUserQuestion**: + +``` +Skill draft is ready. Here are [N] test cases I'd like to run: + +1. "[test prompt 1]" — tests [what aspect] +2. "[test prompt 2]" — tests [what aspect] +3. "[test prompt 3]" — tests [what aspect] + +Each test runs the skill + a baseline (no skill) for comparison. +Estimated time: ~[X] minutes total. + +RECOMMENDATION: Run all [N] test cases now. + +Options: +A) Run all test cases (Recommended) +B) Run test cases, but let me modify them first +C) Add more test cases before running +D) Skip testing — the skill looks good enough to ship +``` Save test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress. @@ -450,7 +562,22 @@ Put each with_skill version before its baseline counterpart. Note: please use generate_review.py to create the viewer; there's no need to write custom HTML. -5. **Tell the user** something like: "I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know." +5. **Tell the user** via **AskUserQuestion**: + +``` +Results are ready! I've opened the eval viewer in your browser. + +- "Outputs" tab: click through each test case, leave feedback in the textbox +- "Benchmark" tab: quantitative comparison (pass rates, timing, tokens) + +Take your time reviewing. When you're done, come back here. + +Options: +A) I've finished reviewing — read my feedback and improve the skill +B) I have questions about the results before giving feedback +C) Results look good enough — skip iteration, let's package the skill +D) Results need major rework — let's discuss before iterating +``` ### What the user sees in the viewer @@ -507,6 +634,24 @@ This is the heart of the loop. You've run the test cases, the user has reviewed This task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need. +After analyzing feedback, present your improvement plan via **AskUserQuestion**: + +``` +I've read the feedback from [N] test cases. [X] had specific complaints, [Y] looked good. + +Key issues: +- [Issue 1]: [plain-language summary] +- [Issue 2]: [plain-language summary] + +RECOMMENDATION: [strategy] because [reason] + +Options: +A) Iterative refinement — targeted fixes for the specific issues above (Recommended) +B) Structural redesign — the core approach needs rethinking +C) Bundle a script — I noticed all test runs independently wrote similar code for [X] +D) Expand test set first — add [N] more test cases to avoid overfitting to these examples +``` + ### The iteration loop After improving the skill: @@ -517,6 +662,18 @@ After improving the skill: 4. Wait for the user to review and tell you they're done 5. Read the new feedback, improve again, repeat +At the end of each iteration, use **AskUserQuestion** as a checkpoint: + +``` +Iteration [N] complete. Results: [pass_rate]% assertions passing, [delta vs previous]. + +Options: +A) Continue iterating — I see more room for improvement +B) Accept this version — it's good enough, let's move to packaging +C) Revert to previous iteration — this round made things worse +D) Run blind comparison — rigorously compare this version vs the previous one +``` + Keep going until: - The user says they're happy - The feedback is all empty (everything looks good) @@ -686,7 +843,20 @@ When editing, remember that the skill is being created for another instance of C ### Step 5: Sanitization Review (Optional) -**Ask the user before executing this step:** "This skill appears to be extracted from a business project. Would you like me to perform a sanitization review to remove business-specific content before public distribution?" +Use **AskUserQuestion** before executing this step: + +``` +This skill appears to contain content from a real project. +Before distribution, I should check for business-specific details +(company names, internal paths, product names) that shouldn't be public. + +RECOMMENDATION: Run selective sanitization — review each finding before removing. + +Options: +A) Full sanitization — automatically remove all business-specific content +B) Selective sanitization — show me each finding and let me decide (Recommended) +C) Skip — this is for internal use only, no sanitization needed +``` Skip if: skill was created from scratch for public use, user declines, or skill is for internal use. @@ -730,6 +900,21 @@ brew install gitleaks - `3` - gitleaks not installed - `4` - Scan error +**If issues are found**, present them via **AskUserQuestion**: + +``` +Security scan found [N] issues in "[skill-name]": +- [SEVERITY] [file]: [description] +- ... + +RECOMMENDATION: Fix automatically — these look like [accidental leaks / false positives]. + +Options: +A) Fix all issues automatically (Recommended) +B) Review each finding — let me decide per-item (some may be intentional) +C) Override and proceed — I accept the risk for internal distribution +``` + ### Step 7: Packaging a Skill Once the skill is ready, package it into a distributable file: @@ -773,7 +958,19 @@ After packaging, update the marketplace registry to include the new or updated s **For updated skills**, bump the version in `plugins[].version` following semver. -### Step 9: Iterate +### Step 9: Ship or Iterate + +After completing the skill, use **AskUserQuestion** to determine next steps: + +``` +Skill "[name]" is complete. Security scan passed, marketplace updated. + +Options: +A) Package and export as .skill file for distribution +B) Run description optimization — improve auto-triggering accuracy (~5 min) +C) Expand test set and iterate more — add edge cases before shipping +D) Done for now — I'll test it manually and come back if needed +``` After testing the skill, users may request improvements. Often this happens right after using the skill, with fresh context of how the skill performed. From 143995b213f09b2cae0c98e94f0aa391d340f3b2 Mon Sep 17 00:00:00 2001 From: daymade <daymadev89@gmail.com> Date: Mon, 23 Mar 2026 00:06:30 +0800 Subject: [PATCH 013/137] =?UTF-8?q?refactor:=20rename=20markdown-tools=20?= =?UTF-8?q?=E2=86=92=20doc-to-markdown=20(v2.0.0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Rename skill to better reflect its purpose (document-to-markdown conversion) - Update SKILL.md name, description, and trigger keywords - Add benchmark reference (2026-03-22) - Update marketplace.json entry (name, skills path, version 2.0.0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- .claude-plugin/marketplace.json | 18 +- {markdown-tools => doc-to-markdown}/SKILL.md | 32 +- .../references/benchmark-2026-03-22.md | 163 +++ .../references/conversion-examples.md | 0 .../references/heavy-mode-guide.md | 0 .../references/tool-comparison.md | 0 doc-to-markdown/scripts/convert.py | 1150 +++++++++++++++++ .../scripts/convert_path.py | 0 .../scripts/extract_pdf_images.py | 0 .../scripts/merge_outputs.py | 0 .../scripts/validate_output.py | 0 markdown-tools/scripts/convert.py | 434 ------- 12 files changed, 1346 insertions(+), 451 deletions(-) rename {markdown-tools => doc-to-markdown}/SKILL.md (74%) create mode 100644 doc-to-markdown/references/benchmark-2026-03-22.md rename {markdown-tools => doc-to-markdown}/references/conversion-examples.md (100%) rename {markdown-tools => doc-to-markdown}/references/heavy-mode-guide.md (100%) rename {markdown-tools => doc-to-markdown}/references/tool-comparison.md (100%) create mode 100755 doc-to-markdown/scripts/convert.py rename {markdown-tools => doc-to-markdown}/scripts/convert_path.py (100%) rename {markdown-tools => doc-to-markdown}/scripts/extract_pdf_images.py (100%) rename {markdown-tools => doc-to-markdown}/scripts/merge_outputs.py (100%) rename {markdown-tools => doc-to-markdown}/scripts/validate_output.py (100%) delete mode 100755 markdown-tools/scripts/convert.py diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 76986aa5..2286d3df 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -50,25 +50,23 @@ ] }, { - "name": "markdown-tools", - "description": "Convert documents (PDFs, Word, PowerPoint) to high-quality markdown with multi-tool orchestration. Supports Quick Mode (fast, single tool) and Heavy Mode (best quality, multi-tool merge with segment-level selection). Features PyMuPDF4LLM for LLM-optimized PDF conversion, pandoc for DOCX/PPTX structure preservation, quality validation with HTML reports, and image extraction with metadata", + "name": "doc-to-markdown", + "description": "Converts DOCX/PDF/PPTX to high-quality Markdown with automatic post-processing. Fixes pandoc grid tables, image paths, attribute noise, and code blocks. Supports Quick Mode (fast, single tool) and Heavy Mode (best quality, multi-tool merge). Trigger on \"convert document\", \"docx to markdown\", \"parse word\", \"doc to markdown\", \"extract images from document\".", "source": "./", "strict": false, - "version": "1.2.0", + "version": "2.0.0", "category": "document-conversion", "keywords": [ "markdown", - "pdf", "docx", + "pdf", "pptx", - "pymupdf4llm", + "converter", "pandoc", - "markitdown", - "heavy-mode", - "quality-validation" + "document" ], "skills": [ - "./markdown-tools" + "./doc-to-markdown" ] }, { @@ -942,4 +940,4 @@ ] } ] -} +} \ No newline at end of file diff --git a/markdown-tools/SKILL.md b/doc-to-markdown/SKILL.md similarity index 74% rename from markdown-tools/SKILL.md rename to doc-to-markdown/SKILL.md index 8bc71f48..966c9fc1 100644 --- a/markdown-tools/SKILL.md +++ b/doc-to-markdown/SKILL.md @@ -1,11 +1,11 @@ --- -name: markdown-tools -description: Converts documents to markdown with multi-tool orchestration for best quality. Supports Quick Mode (fast, single tool) and Heavy Mode (best quality, multi-tool merge). Use when converting PDF/DOCX/PPTX files to markdown, extracting images from documents, validating conversion quality, or needing LLM-optimized document output. +name: doc-to-markdown +description: Converts DOCX/PDF/PPTX to high-quality Markdown with automatic post-processing. Fixes pandoc grid tables, image paths, attribute noise, and code blocks. Supports Quick Mode (fast, single tool) and Heavy Mode (best quality, multi-tool merge). Trigger on "convert document", "docx to markdown", "parse word", "doc to markdown", "extract images from document". --- -# Markdown Tools +# Doc to Markdown -Convert documents to high-quality markdown with intelligent multi-tool orchestration. +Convert documents to high-quality markdown with intelligent multi-tool orchestration and automatic DOCX post-processing. ## Dual Mode Architecture @@ -34,6 +34,9 @@ uv run --with pymupdf4llm --with markitdown scripts/convert.py document.pdf -o o # Heavy Mode - multi-tool parallel execution with merge uv run --with pymupdf4llm --with markitdown scripts/convert.py document.pdf -o output.md --heavy +# DOCX with deep python-docx parsing (experimental) +uv run --with pymupdf4llm --with markitdown --with python-docx scripts/convert.py document.docx -o output.md --docx-deep + # Check available tools uv run scripts/convert.py --list-tools ``` @@ -43,7 +46,7 @@ uv run scripts/convert.py --list-tools | Format | Quick Mode Tool | Heavy Mode Tools | |--------|----------------|------------------| | PDF | pymupdf4llm | pymupdf4llm + markitdown | -| DOCX | pandoc | pandoc + markitdown | +| DOCX | pandoc + post-processing | pandoc + markitdown | | PPTX | markitdown | markitdown + pandoc | | XLSX | markitdown | markitdown | @@ -53,6 +56,21 @@ uv run scripts/convert.py --list-tools - **markitdown**: Microsoft's universal converter, good for Office formats - **pandoc**: Excellent structure preservation for DOCX/PPTX +## DOCX Post-Processing (automatic) + +When converting DOCX files via pandoc, the following cleanups are applied automatically: + +| Problem | Fix | +|---------|-----| +| Grid tables (`+:---+` syntax) | Single-column -> blockquote, multi-column -> split images | +| Image double path (`media/media/`) | Flatten to `media/` | +| Pandoc attributes (`{width="..." height="..."}`) | Removed | +| Inline classes (`{.underline}`, `{.mark}`) | Removed | +| Indented dashed code blocks | Converted to fenced code blocks (```) | +| Escaped brackets (`\[...\]`) | Unescaped to `[...]` | +| Double-bracket links (`[[text]{...}](url)`) | Simplified to `[text](url)` | +| Escaped quotes in code (`\"`) | Fixed to `"` | + ## Heavy Mode Workflow Heavy Mode runs multiple tools in parallel and selects the best segments: @@ -117,7 +135,7 @@ python scripts/merge_outputs.py output1.md output2.md -o merged.md --verbose ## Path Conversion (Windows/WSL) ```bash -# Windows → WSL conversion +# Windows to WSL conversion python scripts/convert_path.py "C:\Users\name\Documents\file.pdf" # Output: /mnt/c/Users/name/Documents/file.pdf ``` @@ -147,7 +165,7 @@ brew install pandoc | Script | Purpose | |--------|---------| -| `convert.py` | Main orchestrator with Quick/Heavy mode | +| `convert.py` | Main orchestrator with Quick/Heavy mode + DOCX post-processing | | `merge_outputs.py` | Merge multiple markdown outputs | | `validate_output.py` | Quality validation with HTML report | | `extract_pdf_images.py` | PDF image extraction with metadata | diff --git a/doc-to-markdown/references/benchmark-2026-03-22.md b/doc-to-markdown/references/benchmark-2026-03-22.md new file mode 100644 index 00000000..27b20690 --- /dev/null +++ b/doc-to-markdown/references/benchmark-2026-03-22.md @@ -0,0 +1,163 @@ +# DOCX→Markdown 转换方案基准测试 + +> **测试日期**：2026-03-22 +> +> **测试文件**：`助教-【腾讯云🦞】小白实践 OpenClaw 保姆级教程.docx`（19MB，77 张图片，含 grid table 布局、JSON 代码块、多列图片并排、信息框） +> +> **测试方法**：5 个方案对同一文件转换，按 5 个维度各 10 分制打分 + +--- + +## 综合评分 + +| 维度 | Docling (IBM) | MarkItDown (MS) | Pandoc | Mammoth | **doc-to-markdown（我们）** | +|------|:---:|:---:|:---:|:---:|:---:| +| 表格质量 | 5 | 3 | 5 | 1~3 | **6** | +| 图片提取 | 4 | 2 | **10** | 5 | 7 | +| 文本完整性 | 8 | 7 | **9** | 7 | **9** | +| 格式清洁度 | 5 | 5 | 5 | 3 | **7** | +| 代码块 | 2 | 1 | N/A | 1 | **9** | +| **综合** | **4.8** | **3.6** | **7.3** | **3.4~3.6** | **7.6** | + +--- + +## 各方案详细分析 + +### 1. IBM Docling（综合 4.8） + +- **版本**：docling 2.x + Granite-Docling-258M +- **架构**：AI 驱动（VLM 视觉语言模型），DocTags 中间格式 → Markdown + +**致命问题**： +- 图片引用全部是 `<!-- image -->` 占位符（77 张图 0 张可显示），`ImageRefMode` API 对 DOCX 不可用 +- 标题层级全部丢失（0 个 `#`），所有标题退化为粗体文本 +- 代码块为零，JSON 和命令全部输出为普通段落 +- `api_key` 被错误转义为 `api\_key` + +**优点**： +- 文本内容完整，中文/emoji/链接保留良好 +- 无 grid table 或 HTML 残留 +- 表格语法正确（pipe table），但内容是占位符 + +**结论**：Docling 的优势在 PDF（AAAI 2025 论文场景），DOCX 支持远未达到生产级别。 + +### 2. Microsoft MarkItDown（综合 3.6） + +- **版本**：markitdown 0.1.5 +- **架构**：底层调用 mammoth → HTML → markdownify → Markdown + +**致命问题**： +- 77 张图片全部是截断的 base64 占位符（`data:image/png;base64,...`），默认 `keep_data_uris=False` 主动丢弃图片数据 +- 标题全部变为粗体文本（mammoth 无法识别 WPS 自定义样式） +- 代码块为零，JSON 被塞入表格单元格 +- 有序列表编号全部错误（输出为 `1. 1. 1.`） + +**优点**： +- 无 HTML 标签残留 +- 文本内容基本完整 + +**结论**：MarkItDown 的 markdownify 后处理反而引入破坏性截断。轻量场景可用，复杂 DOCX 不可靠。 + +### 3. Pandoc（综合 7.3） + +- **版本**：pandoc 3.9 +- **架构**：Haskell 原生 AST 解析，支持 60+ 格式 + +**测试了 3 种参数**： + +| 参数 | 结果 | +|------|------| +| `-t gfm` | 最差：24 个 HTML `<table>` 嵌套，74 个 HTML `<img>` | +| `-t markdown` | 最佳：grid table（可后处理），无 HTML | +| `-t markdown-raw_html-...` | 与 markdown 完全相同，参数无效果 | + +**问题**： +- Grid table 不可避免（原 docx 有多行单元格和嵌套表格，pipe table 无法表达） +- `{width="..." height="..."}` 68 处 +- `{.underline}` 6 处 +- 反斜杠过度转义 37 处 + +**优点**： +- 图片提取 10/10（77 张全部正确，路径结构一致） +- 文本完整性 9/10（内容、链接、emoji 全部保留） +- 最成熟稳定的底层引擎 + +**结论**：Pandoc 是最可靠的底层引擎，输出质量最高但需要后处理清洗 pandoc 私有语法。 + +### 4. Mammoth（综合 3.4~3.6） + +- **版本**：mammoth 1.11.0 +- **架构**：python-docx 解析 → HTML/Markdown（Markdown 支持已废弃） + +**测试了 2 种方式**： + +| 方式 | 综合 | +|------|------| +| 方式A：直接转 Markdown | 3.4（表格完全丢失） | +| 方式B：转 HTML → markdownify | 3.6（有表格但嵌套被压扁） | + +**致命问题**： +- 标题全部丢失（WPS `styles.xml` 中样式定义为空，mammoth 无法映射 Heading） +- 代码块为零 +- 图片全部 base64 内嵌，单文件 28MB +- 方式B 中 markdownify 丢失 14 张图片（63/77） + +**结论**：Mammoth 的 Markdown 支持已废弃，对 WPS 导出的 docx 兼容性差。不推荐。 + +### 5. doc-to-markdown / 我们的方案（综合 7.6） + +- **版本**：doc-to-markdown 1.0（基于 pandoc + 6 个后处理函数） +- **架构**：Pandoc 转换 → 自动后处理（grid table 清理、图片路径修复、属性清理、代码块修复、转义修复） + +**后处理实际效果**： + +| 后处理函数 | 修复数量 | +|-----------|---------| +| `_convert_grid_tables` | 11 处 grid table → pipe table / blockquote | +| `_clean_pandoc_attributes` | 3437 字符属性清理 | +| `_fix_code_blocks` | 22 处缩进虚线 → ``` 代码块 | +| `_fix_escaped_brackets` | 10 处 | +| `_fix_double_bracket_links` | 1 处 | +| `_fix_image_paths` | 77 张图片路径修复 | + +**已知问题（待修复）**： +- 图片路径双层嵌套 bug：`--assets-dir` 指定目录内被 pandoc 再建一层 `media/` +- 2 处 grid table 残留（文末并排图片组未完全转换） + +**优点**： +- 代码块识别 9/10（JSON 带语言标签，命令行正确包裹） +- 格式清洁度 7/10（attributes、转义、grid table 大部分清理干净） +- 文本完整性 9/10（关键内容全部保留） + +**结论**：综合最优，核心价值在 pandoc 后处理层。剩余 2 个 bug 可修。 + +--- + +## 架构决策 + +``` +最终方案：Pandoc（底层引擎）+ doc-to-markdown 后处理（增值层） + +理由： +1. Pandoc 图片提取最可靠（10/10），文本最完整（9/10） +2. Pandoc 的问题（grid table、属性、转义）全部可后处理解决 +3. Docling/MarkItDown/Mammoth 的致命问题（图片丢失、标题丢失）无法后处理修复 +4. 后处理层是我们的核心竞争力，成本低、可迭代 +``` + +--- + +## 测试文件特征 + +本次测试文件的难点在于： + +| 特征 | 说明 | 影响 | +|------|------|------| +| WPS 导出 | 非标准 Word 样式（Style ID 2/3 而非 Heading 1/2） | mammoth/markitdown/docling 标题全丢 | +| 多列图片布局 | 2x2、1x4 图片网格用表格排版 | pandoc 输出 grid table | +| 信息框/提示框 | 单列表格包裹文字 | pandoc 输出 grid table | +| 嵌套表格 | 表格内套表格 | pipe table 无法表达 | +| JSON 代码块 | 非代码块样式，用文本框/缩进表示 | 多数工具无法识别为代码 | +| 19MB 文件 | 77 张截图嵌入 | base64 方案导致 28MB 输出 | + +这些特征代表了真实世界中 WPS/飞书文档导出 docx 的典型困难，是有效的基准测试场景。 diff --git a/markdown-tools/references/conversion-examples.md b/doc-to-markdown/references/conversion-examples.md similarity index 100% rename from markdown-tools/references/conversion-examples.md rename to doc-to-markdown/references/conversion-examples.md diff --git a/markdown-tools/references/heavy-mode-guide.md b/doc-to-markdown/references/heavy-mode-guide.md similarity index 100% rename from markdown-tools/references/heavy-mode-guide.md rename to doc-to-markdown/references/heavy-mode-guide.md diff --git a/markdown-tools/references/tool-comparison.md b/doc-to-markdown/references/tool-comparison.md similarity index 100% rename from markdown-tools/references/tool-comparison.md rename to doc-to-markdown/references/tool-comparison.md diff --git a/doc-to-markdown/scripts/convert.py b/doc-to-markdown/scripts/convert.py new file mode 100755 index 00000000..a95eb86f --- /dev/null +++ b/doc-to-markdown/scripts/convert.py @@ -0,0 +1,1150 @@ +#!/usr/bin/env python3 +""" +Multi-tool document to markdown converter with intelligent orchestration. + +Supports Quick Mode (fast, single tool) and Heavy Mode (best quality, multi-tool merge). +DOCX files get automatic post-processing to fix pandoc artifacts. + +Usage: + # Quick Mode (default) - fast, single best tool + uv run --with pymupdf4llm --with markitdown scripts/convert.py document.pdf -o output.md + + # Heavy Mode - multi-tool parallel execution with merge + uv run --with pymupdf4llm --with markitdown scripts/convert.py document.pdf -o output.md --heavy + + # DOCX deep mode - python-docx direct parsing (experimental) + uv run --with python-docx scripts/convert.py document.docx -o output.md --docx-deep + + # With image extraction + uv run --with pymupdf4llm scripts/convert.py document.pdf -o output.md --assets-dir ./images + +Dependencies: + - pymupdf4llm: PDF conversion (LLM-optimized) + - markitdown: PDF/DOCX/PPTX conversion + - pandoc: DOCX/PPTX conversion (system install: brew install pandoc) + - python-docx: DOCX deep parsing (optional, for --docx-deep) +""" + +import argparse +import re +import subprocess +import sys +import shutil +import zipfile +from dataclasses import dataclass, field +from pathlib import Path +from typing import Optional + + +@dataclass +class ConversionResult: + """Result from a single tool conversion.""" + markdown: str + tool: str + images: list[str] = field(default_factory=list) + success: bool = True + error: str = "" + + +# ── Post-processing stats ──────────────────────────────────────────────────── + +@dataclass +class PostProcessStats: + """Track what the DOCX post-processor fixed.""" + grid_tables_converted: int = 0 + image_paths_fixed: int = 0 + attributes_removed: int = 0 + code_blocks_fixed: int = 0 + escaped_brackets_fixed: int = 0 + double_brackets_fixed: int = 0 + + def any_fixes(self) -> bool: + return any( + getattr(self, f) > 0 + for f in self.__dataclass_fields__ + ) + + def summary(self) -> str: + parts = [] + if self.grid_tables_converted: + parts.append(f"grid tables: {self.grid_tables_converted}") + if self.image_paths_fixed: + parts.append(f"image paths: {self.image_paths_fixed}") + if self.attributes_removed: + parts.append(f"attributes: {self.attributes_removed}") + if self.code_blocks_fixed: + parts.append(f"code blocks: {self.code_blocks_fixed}") + if self.escaped_brackets_fixed: + parts.append(f"escaped brackets: {self.escaped_brackets_fixed}") + if self.double_brackets_fixed: + parts.append(f"double brackets: {self.double_brackets_fixed}") + return ", ".join(parts) if parts else "no fixes needed" + + +# ── DOCX post-processing ───────────────────────────────────────────────────── + +# Regex patterns compiled once +_RE_GRID_BORDER = re.compile(r"^\+[:=-][-:=]+(?:\+[:=-][-:=]+)*\+$") +_RE_GRID_ROW = re.compile(r"^\|(.+)\|$") +_RE_NESTED_GRID_BORDER = re.compile(r"^\|\s*\+[:=-][-:=]+\+\s*\|$") +_RE_PANDOC_ATTR = re.compile(r"\{[^}]*(?:width|height)\s*=\s*\"[^\"]*\"[^}]*\}") +_RE_PANDOC_CLASS = re.compile(r"\{\.(?:underline|mark)\}") +_RE_DOUBLE_BRACKET_LINK = re.compile(r"\[\[([^\]]+)\]\(([^)]+)\)") +_RE_DOUBLE_BRACKET_CLOSED = re.compile(r"\[\[([^\]]+)\]\]\(([^)]+)\)") +_RE_DOUBLE_BRACKET_ATTR_LINK = re.compile(r"\[\[([^\]]+)\]\{[^}]*\}\]\(([^)]+)\)") +_RE_ESCAPED_BRACKET = re.compile(r"\\(\[|])") +# Matches single-column dashed line: " ------" +# AND multi-column simple table border: " ---- -----" +_RE_DASHED_LINE = re.compile(r"^(\s{2,})-{3,}[\s-]*$") +_RE_ESCAPED_QUOTE = re.compile(r'\\"') +# CJK + fullwidth punctuation range for bold spacing checks +_RE_CJK_PUNCT = re.compile(r'[\u4e00-\u9fff\u3000-\u303f\uff01-\uffef，。、；：！？（）【】「」《》""'']') +_RE_BOLD_PAIR = re.compile(r'\*\*(.+?)\*\*') + + +def _is_grid_border(line: str) -> bool: + """Check if a line is a grid table border like +---+ or +:---+.""" + stripped = line.strip() + return bool(_RE_GRID_BORDER.match(stripped)) + + +def _is_nested_grid_border(line: str) -> bool: + """Check if a line is a nested grid border like | +---+ |.""" + stripped = line.strip() + return bool(_RE_NESTED_GRID_BORDER.match(stripped)) + + +def _count_grid_columns(border_line: str) -> int: + """Count columns in a grid table border line.""" + stripped = border_line.strip() + if not stripped.startswith("+"): + return 0 + # Count + separators minus 1 = number of columns + return stripped.count("+") - 1 + + + +# Languages recognized as code block hints in pandoc dashed-line blocks +_KNOWN_CODE_LANGS = frozenset({ + "json", "bash", "shell", "python", "javascript", "js", + "html", "css", "yaml", "xml", "sql", "plain text", + "text", "plaintext", "typescript", "ts", "go", "rust", + "java", "c", "cpp", "ruby", "php", +}) + + +def _build_pipe_table(rows: list[list[str]]) -> list[str]: + """Build a standard markdown pipe table from rows of cells.""" + if not rows: + return [] + col_count = max(len(r) for r in rows) + lines = [ + "| " + " | ".join([""] * col_count) + " |", + "| " + " | ".join(["---"] * col_count) + " |", + ] + for row in rows: + padded = row + [""] * (col_count - len(row)) + lines.append("| " + " | ".join(padded) + " |") + return lines + + +def _collect_images(directory: Path) -> list[str]: + """Collect image files from a directory (single glob pass).""" + if not directory.exists(): + return [] + image_exts = {".png", ".jpg", ".jpeg", ".gif", ".webp"} + return sorted( + str(p) for p in directory.rglob("*") + if p.suffix.lower() in image_exts + ) + + +def _convert_grid_tables(text: str, stats: PostProcessStats) -> str: + """Convert pandoc grid tables to standard markdown. + + Single-column grid tables (info boxes) -> blockquotes. + Multi-column grid tables (side-by-side images) -> split into individual elements. + Nested grid tables are flattened. + """ + lines = text.split("\n") + result = [] + i = 0 + + while i < len(lines): + line = lines[i] + + # Detect grid table start + if _is_grid_border(line): + # Collect the entire grid table + table_lines = [line] + i += 1 + while i < len(lines): + table_lines.append(lines[i]) + if _is_grid_border(lines[i]) and len(table_lines) > 1: + i += 1 + break + i += 1 + else: + # Reached end of file without closing border + # Just output as-is + result.extend(table_lines) + continue + + stats.grid_tables_converted += 1 + num_cols = _count_grid_columns(table_lines[0]) + + # Extract content lines (skip borders) + content_lines = [] + for tl in table_lines: + if _is_grid_border(tl) or _is_nested_grid_border(tl): + continue + m = _RE_GRID_ROW.match(tl.strip()) + if m: + content_lines.append(m.group(1).strip()) + else: + # Non-standard line inside grid, keep content + stripped = tl.strip() + if stripped and stripped != "|": + content_lines.append(stripped) + + if num_cols <= 1: + # Single column -> blockquote + result.append("") + for cl in content_lines: + # Strip outer pipes if present from nested grids + cleaned = cl.strip() + if cleaned.startswith("|") and cleaned.endswith("|"): + cleaned = cleaned[1:-1].strip() + # Skip nested grid borders + if _RE_GRID_BORDER.match(cleaned): + continue + if cleaned: + result.append(f"> {cleaned}") + else: + result.append(">") + result.append("") + else: + # Multi-column -> convert to standard pipe table + # Parse rows: each content_line is a row, split by | into cells + table_rows = [] + for cl in content_lines: + cells = [c.strip() for c in cl.split("|") if c.strip() and not _RE_GRID_BORDER.match(c.strip())] + if cells: + table_rows.append(cells) + + if table_rows: + result.append("") + result.extend(_build_pipe_table(table_rows)) + result.append("") + else: + result.append(line) + i += 1 + + return "\n".join(result) + + +def _fix_image_paths(text: str, assets_dir: Optional[Path], stats: PostProcessStats) -> str: + """Fix pandoc's double media path and verify images exist. + + Pandoc extracts to <assets_dir>/media/<files> but references as + <assets_dir>/media/media/<files>. Fix the references. + Also flatten the actual directory if needed. + """ + def fix_path(m: re.Match) -> str: + alt = m.group(1) + path = m.group(2) + new_path = path + + # Fix double media/ path + if "media/media/" in path: + new_path = path.replace("media/media/", "media/") + stats.image_paths_fixed += 1 + + return f"![{alt}]({new_path})" + + text = re.sub(r"!\[([^\]]*)\]\(([^)]+)\)", fix_path, text) + + # Flatten double media/ nesting if present (pandoc artifact) + if assets_dir: + double_media = assets_dir / "media" / "media" + single_media = assets_dir / "media" + try: + for f in double_media.iterdir(): + dest = single_media / f.name + if not dest.exists(): + shutil.move(str(f), str(dest)) + double_media.rmdir() + except (FileNotFoundError, OSError): + pass + + return text + + +def _clean_pandoc_attributes(text: str, stats: PostProcessStats) -> str: + """Remove pandoc attribute annotations from markdown. + + Removes: {width="..." height="..."}, {.underline}, {.mark}, etc. + """ + count_before = len(text) + + # Remove width/height attributes on images + text = _RE_PANDOC_ATTR.sub("", text) + + # Remove class attributes like {.underline} + text = _RE_PANDOC_CLASS.sub("", text) + + if len(text) != count_before: + # Rough count of removals + stats.attributes_removed = count_before - len(text) + + return text + + +def _is_code_content(lines: list[str]) -> bool: + """Heuristic: decide if content between dashed lines is code or a note/callout. + + Code indicators: + - Has a language hint on the first line + - Contains JSON/code-like syntax ({, }, =, ;, ->, //) + - Contains URLs with protocols + - Has backslash line continuations + + Note indicators: + - Mostly CJK/prose text without code syntax + - Short single-line content + """ + text = "\n".join(lines) + stripped = text.strip() + + if not stripped: + return False + + # Code syntax indicators + code_chars = set('{}[]();=<>/\\') + code_char_count = sum(1 for c in stripped if c in code_chars) + + # If >5% of content is code syntax characters, treat as code + if len(stripped) > 0 and code_char_count / len(stripped) > 0.05: + return True + + # JSON-like structure + if stripped.startswith("{") or stripped.startswith("["): + return True + + # Command-like (starts with common command patterns) + first_line = lines[0].strip() if lines else "" + if re.match(r"^(curl|wget|npm|pip|brew|apt|docker|git|ssh|cd|ls|cat|echo|python|node|uv)\s", first_line): + return True + + return False + + +def _fix_code_blocks(text: str, stats: PostProcessStats) -> str: + """Convert pandoc's indented dashed-line blocks to fenced code blocks or blockquotes. + + Pandoc wraps both code and notes in: + ------------------------------------------------------------------ + content here + + ------------------------------------------------------------------ + + With language hint -> code block: + ```json + content here + ``` + + Without language hint + prose content -> blockquote: + > content here + + Without language hint + code-like content -> code block: + ``` + content here + ``` + """ + lines = text.split("\n") + result = [] + i = 0 + + known_langs = _KNOWN_CODE_LANGS + + while i < len(lines): + line = lines[i] + + # Detect indented dashed line (2+ leading spaces, 3+ dashes) + if _RE_DASHED_LINE.match(line): + # Check if this is a pandoc simple table (multiple dashed columns + # on the same line, or content between dashes contains images) + # Simple table pattern: " ---- ----" (multiple dash groups separated by spaces) + # Gap can be 1+ spaces (pandoc uses varying gaps) + dash_parts = [p for p in line.strip().split() if p.strip()] + is_simple_table_border = len(dash_parts) > 1 and all( + re.match(r"^-+$", p.strip()) for p in dash_parts + ) + + if is_simple_table_border: + # This is a pandoc simple table border - collect rows until + # next simple table border, convert to pipe table + table_rows = [] + j = i + 1 + while j < len(lines): + next_line = lines[j] + # Check for closing simple table border + next_parts = [p for p in next_line.strip().split() if p.strip()] + is_next_border = len(next_parts) > 1 and all( + re.match(r"^-+$", p.strip()) for p in next_parts + ) + if is_next_border: + j += 1 + break + if next_line.strip(): + # Split by 2+ spaces to get columns (pandoc uses varying gaps) + cells = [c.strip() for c in re.split(r"\s{2,}", next_line.strip()) if c.strip()] + if cells: + table_rows.append(cells) + j += 1 + + if table_rows: + stats.code_blocks_fixed += 1 + result.append("") + result.extend(_build_pipe_table(table_rows)) + result.append("") + + i = j + continue + + # Not a simple table - look for content and closing dashed line + block_content = [] + lang_hint = "" + j = i + 1 + + while j < len(lines): + next_line = lines[j] + + if _RE_DASHED_LINE.match(next_line): + # Found closing dashed line + j += 1 + break + + block_content.append(next_line) + j += 1 + else: + # No closing dashed line found - not a block, keep as-is + result.append(line) + i += 1 + continue + + # If content contains images, treat as simple table (single-column) + has_images = any("![" in cl for cl in block_content) + if has_images: + result.append("") + for cl in block_content: + cl = cl.strip() + if cl: + result.append(cl) + result.append("") + i = j + continue + + # Check if first line is a language hint (e.g., " JSON\", " Plain Text\") + has_lang_hint = False + if block_content: + first = block_content[0].strip() + first_clean = first.rstrip("\\").strip() + if first_clean.lower() in known_langs: + lang_hint = first_clean.lower() + if lang_hint in ("plain text", "text", "plaintext"): + lang_hint = "" # No language tag for plain text + has_lang_hint = True + block_content = block_content[1:] + + # Clean content: remove leading 2-space indent, fix escaped quotes + cleaned = [] + for cl in block_content: + if cl.startswith(" "): + cl = cl[2:] + cl = cl.replace('\\"', '"') + if cl.endswith("\\"): + cl = cl[:-1] + cleaned.append(cl) + + # Remove trailing/leading empty lines + while cleaned and not cleaned[-1].strip(): + cleaned.pop() + while cleaned and not cleaned[0].strip(): + cleaned.pop(0) + + if cleaned: + stats.code_blocks_fixed += 1 + + # Decide: code block vs blockquote + if has_lang_hint or _is_code_content(cleaned): + # Code block + result.append("") + result.append(f"```{lang_hint}") + result.extend(cleaned) + result.append("```") + result.append("") + else: + # Note/callout -> blockquote + result.append("") + for cl in cleaned: + if cl.strip(): + result.append(f"> {cl}") + else: + result.append(">") + result.append("") + + i = j + else: + result.append(line) + i += 1 + + return "\n".join(result) + + +def _fix_escaped_brackets(text: str, stats: PostProcessStats) -> str: + r"""Fix pandoc's escaped brackets: \[ -> [, \] -> ].""" + count = len(_RE_ESCAPED_BRACKET.findall(text)) + if count: + stats.escaped_brackets_fixed = count + text = _RE_ESCAPED_BRACKET.sub(r"\1", text) + return text + + +def _fix_double_bracket_links(text: str, stats: PostProcessStats) -> str: + """Fix double-bracket links: [[text]{.underline}](url) -> [text](url).""" + count = 0 + + def _replace_link(m: re.Match) -> str: + nonlocal count + count += 1 + return f"[{m.group(1)}]({m.group(2)})" + + text = _RE_DOUBLE_BRACKET_ATTR_LINK.sub(_replace_link, text) + text = _RE_DOUBLE_BRACKET_CLOSED.sub(_replace_link, text) + text = _RE_DOUBLE_BRACKET_LINK.sub(_replace_link, text) + + stats.double_brackets_fixed = count + return text + + +def _fix_cjk_bold_spacing(text: str) -> str: + """Add space between **bold** markers and adjacent CJK characters. + + DOCX uses run-level styling for bold — no spaces between runs in CJK text. + Markdown renderers need whitespace around ** to recognize bold boundaries. + We find each **content** span, check the character before/after, and insert + a space only when the adjacent character is CJK (avoiding double spaces). + """ + result = [] + last_end = 0 + + for m in _RE_BOLD_PAIR.finditer(text): + start, end = m.start(), m.end() + result.append(text[last_end:start]) + + # Space before opening ** if preceded by CJK + if start > 0 and _RE_CJK_PUNCT.match(text[start - 1]): + result.append(' ') + + result.append(m.group(0)) + + # Space after closing ** if followed by CJK + if end < len(text) and _RE_CJK_PUNCT.match(text[end]): + result.append(' ') + + last_end = end + + result.append(text[last_end:]) + return ''.join(result) + + +def _cleanup_excessive_blank_lines(text: str) -> str: + """Collapse 3+ consecutive blank lines to 2.""" + return re.sub(r"\n{4,}", "\n\n\n", text) + + +def postprocess_docx_markdown( + text: str, + assets_dir: Optional[Path] = None, +) -> tuple[str, PostProcessStats]: + """Apply all DOCX-specific post-processing to pandoc markdown output. + + Returns (cleaned_text, stats). + """ + stats = PostProcessStats() + + # Order matters: grid tables first (they contain images with attributes) + text = _convert_grid_tables(text, stats) + text = _fix_image_paths(text, assets_dir, stats) + text = _clean_pandoc_attributes(text, stats) + text = _fix_code_blocks(text, stats) + text = _fix_double_bracket_links(text, stats) + text = _fix_escaped_brackets(text, stats) + text = _fix_cjk_bold_spacing(text) + text = _cleanup_excessive_blank_lines(text) + + return text, stats + + +# ── DOCX deep parsing (python-docx) ────────────────────────────────────────── + +def convert_with_docx_deep( + file_path: Path, assets_dir: Optional[Path] = None +) -> ConversionResult: + """Convert DOCX using python-docx direct parsing (experimental). + + More precise than pandoc for: + - Table structure preservation + - Comment extraction + - Image extraction with position info + """ + try: + from docx import Document + from docx.opc.constants import RELATIONSHIP_TYPE as RT + except ImportError: + return ConversionResult( + markdown="", + tool="docx-deep", + success=False, + error="python-docx not installed. Run: pip install python-docx", + ) + + try: + doc = Document(str(file_path)) + md_parts = [] + images = [] + image_counter = 0 + + # Extract images from docx zip + if assets_dir: + assets_dir.mkdir(parents=True, exist_ok=True) + media_dir = assets_dir / "media" + media_dir.mkdir(exist_ok=True) + + with zipfile.ZipFile(str(file_path), "r") as zf: + for name in zf.namelist(): + if name.startswith("word/media/"): + img_name = Path(name).name + img_dest = media_dir / img_name + with zf.open(name) as src, open(img_dest, "wb") as dst: + dst.write(src.read()) + images.append(str(img_dest)) + + # Process paragraphs + for para in doc.paragraphs: + style_name = para.style.name if para.style else "" + text = para.text.strip() + + if not text: + md_parts.append("") + continue + + # Headings + if style_name.startswith("Heading"): + try: + level = int(style_name.split()[-1]) + except (ValueError, IndexError): + level = 1 + md_parts.append(f"{'#' * level} {text}") + md_parts.append("") + continue + + # Check for bold-only paragraphs (often sub-headings in Chinese docs) + all_bold = all(run.bold for run in para.runs if run.text.strip()) + if all_bold and para.runs and len(text) < 100: + md_parts.append(f"**{text}**") + md_parts.append("") + continue + + # Regular paragraph + md_parts.append(text) + md_parts.append("") + + # Process tables + for table in doc.tables: + md_parts.append("") + rows = table.rows + if not rows: + continue + + # Header row + header_cells = [cell.text.strip() for cell in rows[0].cells] + md_parts.append("| " + " | ".join(header_cells) + " |") + md_parts.append("| " + " | ".join(["---"] * len(header_cells)) + " |") + + # Data rows + for row in rows[1:]: + cells = [cell.text.strip() for cell in row.cells] + md_parts.append("| " + " | ".join(cells) + " |") + md_parts.append("") + + markdown = "\n".join(md_parts) + + return ConversionResult( + markdown=markdown, + tool="docx-deep", + images=images, + success=True, + ) + except Exception as e: + return ConversionResult( + markdown="", tool="docx-deep", success=False, error=str(e) + ) + + +# ── Existing tool converters ───────────────────────────────────────────────── + +def check_tool_available(tool: str) -> bool: + """Check if a conversion tool is available.""" + if tool == "pymupdf4llm": + try: + import pymupdf4llm + return True + except ImportError: + return False + elif tool == "markitdown": + try: + import markitdown + return True + except ImportError: + return False + elif tool == "pandoc": + return shutil.which("pandoc") is not None + elif tool == "docx-deep": + try: + from docx import Document + return True + except ImportError: + return False + return False + + +def select_tools(file_path: Path, mode: str) -> list[str]: + """Select conversion tools based on file type and mode.""" + ext = file_path.suffix.lower() + + # Tool preferences by format + tool_map = { + ".pdf": { + "quick": ["pymupdf4llm", "markitdown"], # fallback order + "heavy": ["pymupdf4llm", "markitdown"], + }, + ".docx": { + "quick": ["pandoc", "markitdown"], + "heavy": ["pandoc", "markitdown"], + }, + ".doc": { + "quick": ["pandoc", "markitdown"], + "heavy": ["pandoc", "markitdown"], + }, + ".pptx": { + "quick": ["markitdown", "pandoc"], + "heavy": ["markitdown", "pandoc"], + }, + ".xlsx": { + "quick": ["markitdown"], + "heavy": ["markitdown"], + }, + } + + tools = tool_map.get(ext, {"quick": ["markitdown"], "heavy": ["markitdown"]}) + + if mode == "quick": + # Return first available tool + for tool in tools["quick"]: + if check_tool_available(tool): + return [tool] + return [] + else: # heavy + # Return all available tools + return [t for t in tools["heavy"] if check_tool_available(t)] + + +def convert_with_pymupdf4llm( + file_path: Path, assets_dir: Optional[Path] = None +) -> ConversionResult: + """Convert using PyMuPDF4LLM (best for PDFs).""" + try: + import pymupdf4llm + + kwargs = {} + images = [] + + if assets_dir: + assets_dir.mkdir(parents=True, exist_ok=True) + kwargs["write_images"] = True + kwargs["image_path"] = str(assets_dir) + kwargs["dpi"] = 150 + + # Use best table detection strategy + kwargs["table_strategy"] = "lines_strict" + + md_text = pymupdf4llm.to_markdown(str(file_path), **kwargs) + + if assets_dir: + images = _collect_images(assets_dir) + + return ConversionResult( + markdown=md_text, tool="pymupdf4llm", images=images, success=True + ) + except Exception as e: + return ConversionResult( + markdown="", tool="pymupdf4llm", success=False, error=str(e) + ) + + +def convert_with_markitdown( + file_path: Path, assets_dir: Optional[Path] = None +) -> ConversionResult: + """Convert using markitdown.""" + try: + # markitdown CLI approach + result = subprocess.run( + ["markitdown", str(file_path)], + capture_output=True, + text=True, + timeout=120, + ) + + if result.returncode != 0: + return ConversionResult( + markdown="", + tool="markitdown", + success=False, + error=result.stderr, + ) + + return ConversionResult( + markdown=result.stdout, tool="markitdown", success=True + ) + except FileNotFoundError: + # Try Python API + try: + from markitdown import MarkItDown + + md = MarkItDown() + result = md.convert(str(file_path)) + return ConversionResult( + markdown=result.text_content, tool="markitdown", success=True + ) + except Exception as e: + return ConversionResult( + markdown="", tool="markitdown", success=False, error=str(e) + ) + except Exception as e: + return ConversionResult( + markdown="", tool="markitdown", success=False, error=str(e) + ) + + +def convert_with_pandoc( + file_path: Path, assets_dir: Optional[Path] = None +) -> ConversionResult: + """Convert using pandoc. + + Pandoc's --extract-media=DIR creates a media/ subdirectory inside DIR. + We point --extract-media at assets_dir's parent so pandoc's media/ + subdirectory lands exactly at assets_dir (when assets_dir ends with 'media'), + or we use a temp dir and move files afterward. + """ + try: + cmd = ["pandoc", str(file_path), "-t", "markdown", "--wrap=none"] + + extract_dir = None + if assets_dir: + assets_dir.mkdir(parents=True, exist_ok=True) + # Pandoc always creates a media/ subdirectory inside --extract-media. + # Point it at the parent so media/ lands at assets_dir. + if assets_dir.name == "media": + extract_dir = assets_dir.parent + else: + extract_dir = assets_dir + cmd.extend(["--extract-media", str(extract_dir)]) + + result = subprocess.run( + cmd, capture_output=True, text=True, timeout=120 + ) + + if result.returncode != 0: + return ConversionResult( + markdown="", tool="pandoc", success=False, error=result.stderr + ) + + md = result.stdout + + # Convert absolute image paths to relative paths based on output location + if extract_dir: + abs_media = str(extract_dir / "media") + # Replace absolute paths with relative 'media/' prefix + md = md.replace(abs_media + "/", "media/") + + images = _collect_images(assets_dir) if assets_dir else [] + + return ConversionResult( + markdown=md, tool="pandoc", images=images, success=True + ) + except Exception as e: + return ConversionResult( + markdown="", tool="pandoc", success=False, error=str(e) + ) + + +def convert_single( + file_path: Path, tool: str, assets_dir: Optional[Path] = None +) -> ConversionResult: + """Run a single conversion tool.""" + converters = { + "pymupdf4llm": convert_with_pymupdf4llm, + "markitdown": convert_with_markitdown, + "pandoc": convert_with_pandoc, + "docx-deep": convert_with_docx_deep, + } + + converter = converters.get(tool) + if not converter: + return ConversionResult( + markdown="", tool=tool, success=False, error=f"Unknown tool: {tool}" + ) + + return converter(file_path, assets_dir) + + +def merge_results(results: list[ConversionResult]) -> ConversionResult: + """Merge results from multiple tools, selecting best segments.""" + if not results: + return ConversionResult(markdown="", tool="none", success=False) + + # Filter successful results + successful = [r for r in results if r.success and r.markdown.strip()] + if not successful: + # Return first error + return results[0] if results else ConversionResult( + markdown="", tool="none", success=False + ) + + if len(successful) == 1: + return successful[0] + + # Multiple successful results - merge them + # Strategy: Compare key metrics and select best + best = successful[0] + best_score = score_markdown(best.markdown) + + for result in successful[1:]: + score = score_markdown(result.markdown) + if score > best_score: + best = result + best_score = score + + # Merge images from all results + all_images = [] + seen = set() + for result in successful: + for img in result.images: + if img not in seen: + all_images.append(img) + seen.add(img) + + best.images = all_images + best.tool = f"merged({','.join(r.tool for r in successful)})" + + return best + + +def score_markdown(md: str) -> float: + """Score markdown quality for comparison.""" + score = 0.0 + + # Length (more content is generally better) + score += min(len(md) / 10000, 5.0) # Cap at 5 points + + # Tables (proper markdown tables) + table_count = md.count("|---|") + md.count("| ---") + score += min(table_count * 0.5, 3.0) + + # Images (referenced images) + image_count = md.count("![") + score += min(image_count * 0.3, 2.0) + + # Headings (proper hierarchy) + h1_count = md.count("\n# ") + h2_count = md.count("\n## ") + h3_count = md.count("\n### ") + if h1_count > 0 and h2_count >= h1_count: + score += 1.0 # Good hierarchy + + # Lists (structured content) + list_count = md.count("\n- ") + md.count("\n* ") + md.count("\n1. ") + score += min(list_count * 0.1, 2.0) + + # Penalize pandoc artifacts (grid tables, attributes) + artifact_count = md.count("+:---") + md.count("+---+") + artifact_count += md.count('{width="') + md.count("{.underline}") + score -= artifact_count * 0.5 + + return score + + +def main(): + parser = argparse.ArgumentParser( + description="Convert documents to markdown with multi-tool orchestration", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + # Quick mode (default) + python convert.py document.pdf -o output.md + + # Heavy mode (best quality) + python convert.py document.pdf -o output.md --heavy + + # DOCX deep mode (python-docx parsing) + python convert.py document.docx -o output.md --docx-deep + + # With custom assets directory + python convert.py document.pdf -o output.md --assets-dir ./images + """, + ) + parser.add_argument("input", type=Path, nargs="?", help="Input document path") + parser.add_argument( + "-o", "--output", type=Path, help="Output markdown file" + ) + parser.add_argument( + "--heavy", + action="store_true", + help="Enable Heavy Mode (multi-tool, best quality)", + ) + parser.add_argument( + "--docx-deep", + action="store_true", + help="Use python-docx direct parsing (experimental, DOCX only)", + ) + parser.add_argument( + "--no-postprocess", + action="store_true", + help="Disable DOCX post-processing (keep raw pandoc output)", + ) + parser.add_argument( + "--assets-dir", + type=Path, + default=None, + help="Directory for extracted images (default: <output>_assets/)", + ) + parser.add_argument( + "--tool", + choices=["pymupdf4llm", "markitdown", "pandoc", "docx-deep"], + help="Force specific tool (overrides auto-selection)", + ) + parser.add_argument( + "--list-tools", + action="store_true", + help="List available tools and exit", + ) + + args = parser.parse_args() + + # List tools mode + if args.list_tools: + tools = ["pymupdf4llm", "markitdown", "pandoc", "docx-deep"] + print("Available conversion tools:") + for tool in tools: + status = "+" if check_tool_available(tool) else "-" + print(f" {status} {tool}") + sys.exit(0) + + # Validate input + if args.input is None: + parser.error("the following arguments are required: input") + if not args.input.exists(): + print(f"Error: Input file not found: {args.input}", file=sys.stderr) + sys.exit(1) + + # Determine output path + output_path = args.output or args.input.with_suffix(".md") + + # Determine assets directory + assets_dir = args.assets_dir + if assets_dir is None: + assets_dir = output_path.parent / f"{output_path.stem}_assets" + + is_docx = args.input.suffix.lower() in (".docx", ".doc") + + # Handle --docx-deep mode + if args.docx_deep: + if not is_docx: + print("Error: --docx-deep only works with DOCX files.", file=sys.stderr) + sys.exit(1) + tools = ["docx-deep"] + elif args.tool: + tools = [args.tool] if check_tool_available(args.tool) else [] + else: + # Select tools + mode = "heavy" if args.heavy else "quick" + tools = select_tools(args.input, mode) + + mode = "docx-deep" if args.docx_deep else ("heavy" if args.heavy else "quick") + + if not tools: + print("Error: No conversion tools available.", file=sys.stderr) + print("Install with:", file=sys.stderr) + print(" pip install pymupdf4llm", file=sys.stderr) + print(" uv tool install markitdown[pdf]", file=sys.stderr) + print(" brew install pandoc", file=sys.stderr) + sys.exit(1) + + print(f"Converting: {args.input}") + print(f"Mode: {mode.upper()}") + print(f"Tools: {', '.join(tools)}") + + # Run conversions + results = [] + for tool in tools: + print(f" Running {tool}...", end=" ", flush=True) + + # Use separate assets dirs for each tool in heavy mode + tool_assets = None + if assets_dir and mode == "heavy" and len(tools) > 1: + tool_assets = assets_dir / tool + elif assets_dir: + tool_assets = assets_dir + + result = convert_single(args.input, tool, tool_assets) + results.append(result) + + if result.success: + print(f"ok ({len(result.markdown):,} chars, {len(result.images)} images)") + else: + print(f"FAIL ({result.error[:50]}...)") + + # Merge results if heavy mode + if mode == "heavy" and len(results) > 1: + print(" Merging results...", end=" ", flush=True) + final = merge_results(results) + print(f"ok (using {final.tool})") + else: + final = merge_results(results) + + if not final.success: + print(f"Error: Conversion failed: {final.error}", file=sys.stderr) + sys.exit(1) + + # Apply DOCX post-processing + if is_docx and not args.no_postprocess and "pandoc" in final.tool: + print(" Post-processing DOCX output...", end=" ", flush=True) + final.markdown, pp_stats = postprocess_docx_markdown( + final.markdown, assets_dir + ) + print(f"ok ({pp_stats.summary()})") + + # Write output + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(final.markdown) + + print(f"\nOutput: {output_path}") + print(f" Size: {len(final.markdown):,} characters") + if final.images: + print(f" Images: {len(final.images)} extracted") + + +if __name__ == "__main__": + main() diff --git a/markdown-tools/scripts/convert_path.py b/doc-to-markdown/scripts/convert_path.py similarity index 100% rename from markdown-tools/scripts/convert_path.py rename to doc-to-markdown/scripts/convert_path.py diff --git a/markdown-tools/scripts/extract_pdf_images.py b/doc-to-markdown/scripts/extract_pdf_images.py similarity index 100% rename from markdown-tools/scripts/extract_pdf_images.py rename to doc-to-markdown/scripts/extract_pdf_images.py diff --git a/markdown-tools/scripts/merge_outputs.py b/doc-to-markdown/scripts/merge_outputs.py similarity index 100% rename from markdown-tools/scripts/merge_outputs.py rename to doc-to-markdown/scripts/merge_outputs.py diff --git a/markdown-tools/scripts/validate_output.py b/doc-to-markdown/scripts/validate_output.py similarity index 100% rename from markdown-tools/scripts/validate_output.py rename to doc-to-markdown/scripts/validate_output.py diff --git a/markdown-tools/scripts/convert.py b/markdown-tools/scripts/convert.py deleted file mode 100755 index 9ac6f36c..00000000 --- a/markdown-tools/scripts/convert.py +++ /dev/null @@ -1,434 +0,0 @@ -#!/usr/bin/env python3 -""" -Multi-tool document to markdown converter with intelligent orchestration. - -Supports Quick Mode (fast, single tool) and Heavy Mode (best quality, multi-tool merge). - -Usage: - # Quick Mode (default) - fast, single best tool - uv run --with pymupdf4llm --with markitdown scripts/convert.py document.pdf -o output.md - - # Heavy Mode - multi-tool parallel execution with merge - uv run --with pymupdf4llm --with markitdown scripts/convert.py document.pdf -o output.md --heavy - - # With image extraction - uv run --with pymupdf4llm scripts/convert.py document.pdf -o output.md --assets-dir ./images - -Dependencies: - - pymupdf4llm: PDF conversion (LLM-optimized) - - markitdown: PDF/DOCX/PPTX conversion - - pandoc: DOCX/PPTX conversion (system install: brew install pandoc) -""" - -import argparse -import subprocess -import sys -import tempfile -import shutil -from dataclasses import dataclass, field -from pathlib import Path -from typing import Optional - - -@dataclass -class ConversionResult: - """Result from a single tool conversion.""" - markdown: str - tool: str - images: list[str] = field(default_factory=list) - success: bool = True - error: str = "" - - -def check_tool_available(tool: str) -> bool: - """Check if a conversion tool is available.""" - if tool == "pymupdf4llm": - try: - import pymupdf4llm - return True - except ImportError: - return False - elif tool == "markitdown": - try: - import markitdown - return True - except ImportError: - return False - elif tool == "pandoc": - return shutil.which("pandoc") is not None - return False - - -def select_tools(file_path: Path, mode: str) -> list[str]: - """Select conversion tools based on file type and mode.""" - ext = file_path.suffix.lower() - - # Tool preferences by format - tool_map = { - ".pdf": { - "quick": ["pymupdf4llm", "markitdown"], # fallback order - "heavy": ["pymupdf4llm", "markitdown"], - }, - ".docx": { - "quick": ["pandoc", "markitdown"], - "heavy": ["pandoc", "markitdown"], - }, - ".doc": { - "quick": ["pandoc", "markitdown"], - "heavy": ["pandoc", "markitdown"], - }, - ".pptx": { - "quick": ["markitdown", "pandoc"], - "heavy": ["markitdown", "pandoc"], - }, - ".xlsx": { - "quick": ["markitdown"], - "heavy": ["markitdown"], - }, - } - - tools = tool_map.get(ext, {"quick": ["markitdown"], "heavy": ["markitdown"]}) - - if mode == "quick": - # Return first available tool - for tool in tools["quick"]: - if check_tool_available(tool): - return [tool] - return [] - else: # heavy - # Return all available tools - return [t for t in tools["heavy"] if check_tool_available(t)] - - -def convert_with_pymupdf4llm( - file_path: Path, assets_dir: Optional[Path] = None -) -> ConversionResult: - """Convert using PyMuPDF4LLM (best for PDFs).""" - try: - import pymupdf4llm - - kwargs = {} - images = [] - - if assets_dir: - assets_dir.mkdir(parents=True, exist_ok=True) - kwargs["write_images"] = True - kwargs["image_path"] = str(assets_dir) - kwargs["dpi"] = 150 - - # Use best table detection strategy - kwargs["table_strategy"] = "lines_strict" - - md_text = pymupdf4llm.to_markdown(str(file_path), **kwargs) - - # Collect extracted images - if assets_dir and assets_dir.exists(): - images = [str(p) for p in assets_dir.glob("*.png")] - images.extend([str(p) for p in assets_dir.glob("*.jpg")]) - - return ConversionResult( - markdown=md_text, tool="pymupdf4llm", images=images, success=True - ) - except Exception as e: - return ConversionResult( - markdown="", tool="pymupdf4llm", success=False, error=str(e) - ) - - -def convert_with_markitdown( - file_path: Path, assets_dir: Optional[Path] = None -) -> ConversionResult: - """Convert using markitdown.""" - try: - # markitdown CLI approach - result = subprocess.run( - ["markitdown", str(file_path)], - capture_output=True, - text=True, - timeout=120, - ) - - if result.returncode != 0: - return ConversionResult( - markdown="", - tool="markitdown", - success=False, - error=result.stderr, - ) - - return ConversionResult( - markdown=result.stdout, tool="markitdown", success=True - ) - except FileNotFoundError: - # Try Python API - try: - from markitdown import MarkItDown - - md = MarkItDown() - result = md.convert(str(file_path)) - return ConversionResult( - markdown=result.text_content, tool="markitdown", success=True - ) - except Exception as e: - return ConversionResult( - markdown="", tool="markitdown", success=False, error=str(e) - ) - except Exception as e: - return ConversionResult( - markdown="", tool="markitdown", success=False, error=str(e) - ) - - -def convert_with_pandoc( - file_path: Path, assets_dir: Optional[Path] = None -) -> ConversionResult: - """Convert using pandoc.""" - try: - cmd = ["pandoc", str(file_path), "-t", "markdown", "--wrap=none"] - - if assets_dir: - assets_dir.mkdir(parents=True, exist_ok=True) - cmd.extend(["--extract-media", str(assets_dir)]) - - result = subprocess.run( - cmd, capture_output=True, text=True, timeout=120 - ) - - if result.returncode != 0: - return ConversionResult( - markdown="", tool="pandoc", success=False, error=result.stderr - ) - - images = [] - if assets_dir and assets_dir.exists(): - images = [str(p) for p in assets_dir.rglob("*.png")] - images.extend([str(p) for p in assets_dir.rglob("*.jpg")]) - - return ConversionResult( - markdown=result.stdout, tool="pandoc", images=images, success=True - ) - except Exception as e: - return ConversionResult( - markdown="", tool="pandoc", success=False, error=str(e) - ) - - -def convert_single( - file_path: Path, tool: str, assets_dir: Optional[Path] = None -) -> ConversionResult: - """Run a single conversion tool.""" - converters = { - "pymupdf4llm": convert_with_pymupdf4llm, - "markitdown": convert_with_markitdown, - "pandoc": convert_with_pandoc, - } - - converter = converters.get(tool) - if not converter: - return ConversionResult( - markdown="", tool=tool, success=False, error=f"Unknown tool: {tool}" - ) - - return converter(file_path, assets_dir) - - -def merge_results(results: list[ConversionResult]) -> ConversionResult: - """Merge results from multiple tools, selecting best segments.""" - if not results: - return ConversionResult(markdown="", tool="none", success=False) - - # Filter successful results - successful = [r for r in results if r.success and r.markdown.strip()] - if not successful: - # Return first error - return results[0] if results else ConversionResult( - markdown="", tool="none", success=False - ) - - if len(successful) == 1: - return successful[0] - - # Multiple successful results - merge them - # Strategy: Compare key metrics and select best - best = successful[0] - best_score = score_markdown(best.markdown) - - for result in successful[1:]: - score = score_markdown(result.markdown) - if score > best_score: - best = result - best_score = score - - # Merge images from all results - all_images = [] - seen = set() - for result in successful: - for img in result.images: - if img not in seen: - all_images.append(img) - seen.add(img) - - best.images = all_images - best.tool = f"merged({','.join(r.tool for r in successful)})" - - return best - - -def score_markdown(md: str) -> float: - """Score markdown quality for comparison.""" - score = 0.0 - - # Length (more content is generally better) - score += min(len(md) / 10000, 5.0) # Cap at 5 points - - # Tables (proper markdown tables) - table_count = md.count("|---|") + md.count("| ---") - score += min(table_count * 0.5, 3.0) - - # Images (referenced images) - image_count = md.count("![") - score += min(image_count * 0.3, 2.0) - - # Headings (proper hierarchy) - h1_count = md.count("\n# ") - h2_count = md.count("\n## ") - h3_count = md.count("\n### ") - if h1_count > 0 and h2_count >= h1_count: - score += 1.0 # Good hierarchy - - # Lists (structured content) - list_count = md.count("\n- ") + md.count("\n* ") + md.count("\n1. ") - score += min(list_count * 0.1, 2.0) - - return score - - -def main(): - parser = argparse.ArgumentParser( - description="Convert documents to markdown with multi-tool orchestration", - formatter_class=argparse.RawDescriptionHelpFormatter, - epilog=""" -Examples: - # Quick mode (default) - python convert.py document.pdf -o output.md - - # Heavy mode (best quality) - python convert.py document.pdf -o output.md --heavy - - # With custom assets directory - python convert.py document.pdf -o output.md --assets-dir ./images - """, - ) - parser.add_argument("input", type=Path, help="Input document path") - parser.add_argument( - "-o", "--output", type=Path, help="Output markdown file" - ) - parser.add_argument( - "--heavy", - action="store_true", - help="Enable Heavy Mode (multi-tool, best quality)", - ) - parser.add_argument( - "--assets-dir", - type=Path, - default=None, - help="Directory for extracted images (default: <output>_assets/)", - ) - parser.add_argument( - "--tool", - choices=["pymupdf4llm", "markitdown", "pandoc"], - help="Force specific tool (overrides auto-selection)", - ) - parser.add_argument( - "--list-tools", - action="store_true", - help="List available tools and exit", - ) - - args = parser.parse_args() - - # List tools mode - if args.list_tools: - tools = ["pymupdf4llm", "markitdown", "pandoc"] - print("Available conversion tools:") - for tool in tools: - status = "✓" if check_tool_available(tool) else "✗" - print(f" {status} {tool}") - sys.exit(0) - - # Validate input - if not args.input.exists(): - print(f"Error: Input file not found: {args.input}", file=sys.stderr) - sys.exit(1) - - # Determine output path - output_path = args.output or args.input.with_suffix(".md") - - # Determine assets directory - assets_dir = args.assets_dir - if assets_dir is None and args.heavy: - assets_dir = output_path.parent / f"{output_path.stem}_assets" - - # Select tools - mode = "heavy" if args.heavy else "quick" - if args.tool: - tools = [args.tool] if check_tool_available(args.tool) else [] - else: - tools = select_tools(args.input, mode) - - if not tools: - print("Error: No conversion tools available.", file=sys.stderr) - print("Install with:", file=sys.stderr) - print(" pip install pymupdf4llm", file=sys.stderr) - print(" uv tool install markitdown[pdf]", file=sys.stderr) - print(" brew install pandoc", file=sys.stderr) - sys.exit(1) - - print(f"Converting: {args.input}") - print(f"Mode: {mode.upper()}") - print(f"Tools: {', '.join(tools)}") - - # Run conversions - results = [] - for tool in tools: - print(f" Running {tool}...", end=" ", flush=True) - - # Use separate assets dirs for each tool in heavy mode - tool_assets = None - if assets_dir and mode == "heavy" and len(tools) > 1: - tool_assets = assets_dir / tool - elif assets_dir: - tool_assets = assets_dir - - result = convert_single(args.input, tool, tool_assets) - results.append(result) - - if result.success: - print(f"✓ ({len(result.markdown):,} chars, {len(result.images)} images)") - else: - print(f"✗ ({result.error[:50]}...)") - - # Merge results if heavy mode - if mode == "heavy" and len(results) > 1: - print(" Merging results...", end=" ", flush=True) - final = merge_results(results) - print(f"✓ (using {final.tool})") - else: - final = merge_results(results) - - if not final.success: - print(f"Error: Conversion failed: {final.error}", file=sys.stderr) - sys.exit(1) - - # Write output - output_path.parent.mkdir(parents=True, exist_ok=True) - output_path.write_text(final.markdown) - - print(f"\nOutput: {output_path}") - print(f" Size: {len(final.markdown):,} characters") - if final.images: - print(f" Images: {len(final.images)} extracted") - - -if __name__ == "__main__": - main() From a5f3a4bfbe0670aa246ef1685fab9b3cb816c2bb Mon Sep 17 00:00:00 2001 From: daymade <daymadev89@gmail.com> Date: Mon, 23 Mar 2026 01:47:18 +0800 Subject: [PATCH 014/137] fix(tunnel-doctor): add OrbStack transparent proxy + TUN conflict diagnosis Real-world findings from debugging docker build failures on macOS with OrbStack + Shadowrocket: - Add docker pull vs docker build vs docker run proxy path distinction table - Add 2G-1: --network host workaround for OrbStack transparent proxy broken by TUN - Rewrite 2G-2: use host.internal (not 127.0.0.1) for OrbStack Docker proxy - Add 2G-4: container healthcheck failure from lowercase http_proxy env var leak - Add 3 new symptom entries to Step 1 diagnostic index - Add smoking gun diagnosis: wget showing "127.0.0.1: Connection refused" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- tunnel-doctor/SKILL.md | 170 +++++++++++++++++++++++++++++++++-------- 1 file changed, 138 insertions(+), 32 deletions(-) diff --git a/tunnel-doctor/SKILL.md b/tunnel-doctor/SKILL.md index 4f3aee3c..84e9a581 100644 --- a/tunnel-doctor/SKILL.md +++ b/tunnel-doctor/SKILL.md @@ -33,7 +33,10 @@ Determine which scenario applies: - **Remote dev server auth redirects to `localhost` → browser can't follow** → SSH tunnel needed (Step 2D) - **`make status` / scripts curl to localhost fail with proxy** → localhost proxy interception (Step 2E) - **`git push/pull` fails with `FATAL: failed to begin relaying via HTTP`** → SSH double tunnel (Step 2F) -- **`docker pull` fails with `TLS handshake timeout` or `docker build` can't fetch base images** → VM/container proxy propagation (Step 2G) +- **`docker build` `RUN apk/apt` fails with `Connection refused` instantly** → OrbStack transparent proxy + TUN conflict (Step 2G-1, fix: `--network host`) +- **`docker pull` fails with `TLS handshake timeout`** → VM proxy misconfiguration (Step 2G-2, fix: `docker.json` with `host.internal`) +- **Container healthcheck `(unhealthy)` but app runs fine** → Lowercase proxy env var leak (Step 2G-4, fix: clear `http_proxy`+`HTTP_PROXY`) +- **`docker build` can't fetch base images** → VM/container proxy propagation (Step 2G) - **`git clone` fails with `Connection closed by 198.18.x.x`** → TUN DNS hijack for SSH (Step 2H) - **SSH connects but `operation not permitted`** → Tailscale SSH config issue (Step 4) - **SSH connects but `be-child ssh` exits code 1** → WSL snap sandbox issue (Step 5) @@ -46,6 +49,8 @@ Determine which scenario applies: - If `tailscale ping` works but regular `ping` doesn't → Layer 1 (route table corrupted). - If `ssh -T git@github.com` works but `git push` fails intermittently → Layer 4 (double tunnel). - If host `curl https://...` works but `docker pull` times out → Layer 5 (VM proxy propagation). +- If `docker pull` works but `docker build` `RUN apk add` fails instantly with `Connection refused` → OrbStack transparent proxy broken by TUN (Step 2G-1). +- If container healthcheck shows `(unhealthy)` but app works → lowercase `http_proxy` leaked into container (Step 2G-4). - If DNS resolves to `198.18.x.x` virtual IPs → TUN DNS hijack (Step 2H). - If `nc -z` succeeds on port 22 but SSH gets no banner (`kex_exchange_identification`) → Tailscale SSH proxy intercept (Step 5A). Confirm with `tcpdump -i any port 22` on the remote — 0 packets means Tailscale intercepts above the kernel. - If `tailscale ssh` fails with "not available on App Store builds" → install Standalone Tailscale (Step 5B). @@ -318,7 +323,7 @@ GIT_SSH_COMMAND="ssh -o ProxyCommand=none" git push origin main ### Step 2G: Fix VM/Container Runtime Proxy Propagation (Docker pull/build failures) -**Symptom**: `docker pull` or `docker build` fails with `net/http: TLS handshake timeout` or `Internal Server Error` from `auth.docker.io`, while host `curl` to the same URLs works fine. +**Symptom**: `docker pull` or `docker build` fails with `net/http: TLS handshake timeout`, `Connection refused` from Alpine/Debian repos, or `Internal Server Error` from `auth.docker.io`, while host `curl` to the same URLs works fine. **Applies to**: OrbStack, Docker Desktop, or any VM-based Docker runtime on macOS with Shadowrocket/Clash TUN active. @@ -331,66 +336,160 @@ VM process (Docker): Docker daemon → VM bridge → host network → TUN → The TUN handles host-originated traffic correctly but may drop or delay VM-bridged traffic (different TCP stack, MTU, keepalive behavior). -**Three sub-problems and their fixes**: +**Critical distinction: `docker pull` vs `docker build` use different proxy paths**: -#### 2G-1: OrbStack auto-detects and caches proxy (most common) +| Operation | Proxy source | What controls it | +|-----------|-------------|------------------| +| `docker pull` | Docker daemon config | `~/.orbstack/config/docker.json` or `docker info` | +| `docker build` (`RUN apt/apk`) | Build container env | `--build-arg http_proxy=...` or `--network host` | +| `docker run` | Container env | `-e http_proxy=...` or inherited from daemon | -OrbStack's `network_proxy: auto` reads `http_proxy` from the shell environment and writes it to `~/.orbstack/config/docker.json`. **Crucially**, `orbctl config set network_proxy none` does NOT clean up `docker.json` — the cached proxy persists. +Fixing `docker.json` alone will NOT fix `docker build` — the `RUN` commands inside the build container don't inherit daemon proxy settings. + +**Diagnosis** — identify which sub-problem: + +```bash +# 1. Can the Docker daemon pull images? +docker pull --quiet alpine:latest 2>&1 + +# 2. Can a RUN command inside a build reach the internet? +docker build --no-cache - <<'EOF' 2>&1 +FROM alpine:latest +RUN apk update && echo "APK OK" +EOF + +# 3. Can a running container reach the internet? +docker run --rm alpine:latest sh -c "apk update 2>&1 | head -3" +``` + +**Four sub-problems and their fixes**: + +#### 2G-1: `docker build` fails but host works (most common with OrbStack + Shadowrocket) + +**Symptom**: `RUN apk add` or `RUN apt-get install` inside `docker build` fails with `Connection refused` instantly (< 0.2s), even though host `curl` to the same URL works. + +**Root cause**: OrbStack's `network_proxy: auto` creates a transparent proxy inside the VM that intercepts all HTTPS traffic. When Shadowrocket TUN is also active, the transparent proxy's upstream connection breaks — it redirects HTTPS to `127.0.0.1` inside the VM, which has nothing listening. + +**Diagnosis**: + +```bash +# Verify: inside the container, HTTPS goes to 127.0.0.1 (broken transparent proxy) +docker run --rm alpine:latest sh -c "wget -q --timeout=5 -O /dev/null https://dl-cdn.alpinelinux.org/ 2>&1" +# → "wget: can't connect to remote host (127.0.0.1): Connection refused" +# ^^^^^^^^^^^^ This is the smoking gun + +# Verify: --network host bypasses the VM bridge and works +docker run --rm --network host alpine:latest sh -c "apk update 2>&1 | head -3" +# → "v3.23.x ... OK: 27431 distinct packages available" ← Works! +``` + +**Fix** — use `--network host` for docker build: + +```bash +docker build --network host -f Dockerfile -t myimage . +``` + +This bypasses OrbStack's VM network bridge entirely. The build container uses the host's network stack directly, where Shadowrocket TUN correctly handles traffic. + +**Trade-off**: `--network host` disables build-time network isolation. For CI/CD, prefer fixing the proxy config (2G-2). For local development, `--network host` is the pragmatic fix. + +**Permanent fix** — if all your builds need this, add to `~/.docker/daemon.json` or use a shell alias: + +```bash +# Shell alias (add to ~/.zshrc) +alias docker-build='docker build --network host' +``` + +#### 2G-2: OrbStack auto-detects and caches proxy config + +OrbStack's `network_proxy: auto` reads `http_proxy` from the shell environment and configures the Docker daemon. The config is stored in `~/.orbstack/config/docker.json`. + +**Key behaviors**: +- `network_proxy: auto` — OrbStack reads host env, creates transparent proxy in VM +- `network_proxy: none` — Disables transparent proxy, but VM bridge traffic still routes through TUN (may timeout) +- `docker.json` — Controls `docker pull` proxy, NOT `docker build` RUN commands **Diagnosis**: ```bash -# OrbStack config says "none" but Docker still shows proxy -orbctl config get network_proxy # → "none" -docker info | grep -i proxy # → HTTP Proxy: http://127.0.0.1:1082 ← stale! +# Check all three layers +echo "=== OrbStack config ===" +orbctl config get network_proxy -# The real source of truth: +echo "=== docker.json (daemon proxy) ===" cat ~/.orbstack/config/docker.json -# → {"proxies": {"http-proxy": "http://127.0.0.1:1082", ...}} ← cached! + +echo "=== Docker info (effective proxy) ===" +docker info | grep -iE "proxy|No Proxy" ``` -**Fix** — DON'T remove the proxy. Instead, add precise `no-proxy` to prevent localhost interception while keeping the proxy as the VM's outbound channel: +**Fix** — configure `docker.json` with `host.internal` (OrbStack resolves this to the host IP): ```bash -# Write corrected config (keeps proxy, adds no-proxy for local traffic) python3 -c " -import json +import json, os config = { 'proxies': { - 'http-proxy': 'http://127.0.0.1:1082', - 'https-proxy': 'http://127.0.0.1:1082', + 'http-proxy': 'http://host.internal:1082', + 'https-proxy': 'http://host.internal:1082', 'no-proxy': 'localhost,127.0.0.1,::1,192.168.128.0/24,100.64.0.0/10,host.internal,*.local' } } -json.dump(config, open('$HOME/.orbstack/config/docker.json', 'w'), indent=2) +path = os.path.expanduser('~/.orbstack/config/docker.json') +json.dump(config, open(path, 'w'), indent=2) +print('Written:', path) " -# Full restart (not just docker engine) +# Full restart required orbctl stop && sleep 3 && orbctl start ``` -**Why NOT remove the proxy**: When TUN is active, removing the Docker proxy means VM traffic goes directly through the bridge → TUN path, which causes TLS handshake timeouts. The proxy provides a working outbound channel because OrbStack maps host `127.0.0.1` into the VM. +**Important**: Use `host.internal` (OrbStack-specific), NOT `127.0.0.1` (points to VM loopback) and NOT `host.docker.internal` (may not resolve in all contexts). + +**Why NOT remove the proxy**: When TUN is active, removing the Docker proxy means VM traffic goes directly through the bridge → TUN path, which causes TLS handshake timeouts. The proxy provides a working outbound channel. -#### 2G-2: Removing proxy makes Docker worse (counter-intuitive) +#### 2G-3: Removing proxy makes Docker worse (counter-intuitive) | Docker config | Traffic path | Result | |---------------|-------------|--------| -| Proxy ON, no `no-proxy` | Docker → proxy → TUN → internet | Docker Hub ✅, localhost probes ❌ | -| Proxy OFF | Docker → VM bridge → host → TUN → internet | TLS timeout ❌ | -| **Proxy ON + `no-proxy`** | **External: Docker → proxy → internet ✅; Local: Docker → direct ✅** | **Both work ✅** | +| Proxy ON (`127.0.0.1`), no `no-proxy` | Docker → VM proxy → ??? | `docker pull` may work, localhost probes ❌ | +| Proxy ON (`host.internal`), + `no-proxy` | External: Docker → host proxy → internet; Local: direct | **Both work ✅** | +| Proxy OFF (`network_proxy: none`) | Docker → VM bridge → host → TUN → internet | TLS timeout ❌ | +| **`--network host` (build only)** | **Build container → host network → TUN → internet** | **Build works ✅** | + +**Decision tree**: +- `docker pull` broken → Fix `docker.json` with `host.internal` proxy (2G-2) +- `docker build` broken → Use `--network host` (2G-1) OR pass `--build-arg http_proxy=http://host.internal:1082` +- Both broken → Fix both: `docker.json` + `--network host` + +#### 2G-4: Deploy scripts and container healthchecks probe localhost through proxy + +Deploy scripts that `curl localhost` inside containers or Docker healthchecks that use `wget http://localhost` will route through the proxy if env vars leak into the container. -#### 2G-3: Deploy scripts probe localhost through proxy +**Common symptoms**: +- Container healthcheck shows `(unhealthy)` but the app inside is running fine +- `wget: can't connect to remote host (127.0.0.1): Connection refused` in healthcheck logs (proxy port, not app port) -Deploy scripts that `curl localhost` inside the Docker environment will route through the proxy. Fix by adding `NO_PROXY` at the script level: +**Root cause**: Docker inherits uppercase AND lowercase proxy env vars from the host. Many tools only clear uppercase (`HTTP_PROXY=`) but forget lowercase (`http_proxy=http://127.0.0.1:1082`). The healthcheck `wget` uses lowercase. + +**Fix in docker-compose.yml** — clear BOTH cases: + +```yaml +environment: + # Must clear both uppercase and lowercase — wget/curl check different vars + - HTTP_PROXY= + - HTTPS_PROXY= + - http_proxy= + - https_proxy= + - NO_PROXY=* + - no_proxy=* +``` + +**Fix in deploy scripts**: ```bash -# In deploy.sh or similar scripts: _local_bypass="localhost,127.0.0.1,::1" -if [[ -n "${NO_PROXY:-}" ]]; then - export NO_PROXY="${_local_bypass},${NO_PROXY}" -else - export NO_PROXY="${_local_bypass}" -fi +export NO_PROXY="${_local_bypass}${NO_PROXY:+,${NO_PROXY}}" export no_proxy="$NO_PROXY" # Use 127.0.0.1 instead of localhost in probe URLs (some proxy implementations @@ -408,8 +507,15 @@ docker info | grep -iE "proxy|No Proxy" # Pull test docker pull --quiet hello-world -# Local probe test -curl -s http://127.0.0.1:3001/health +# Build test (the real verification) +docker build --network host --no-cache - <<'EOF' +FROM alpine:latest +RUN apk update && echo "BUILD OK" +EOF + +# Container env check (no proxy leak) +docker exec <container> env | grep -i proxy +# Expected: all empty or not set ``` ### Step 2H: Fix TUN DNS Hijack for SSH/Git (198.18.x.x virtual IPs) From d9e1967689f0bf7277d3f943935da2622ccbfd02 Mon Sep 17 00:00:00 2001 From: daymade <daymadev89@gmail.com> Date: Mon, 23 Mar 2026 03:18:37 +0800 Subject: [PATCH 015/137] feat(doc-to-markdown): CJK bold spacing, JSON pretty-print, 31 tests, full rename cleanup - Add CJK bold spacing fix: insert spaces around **bold** spans containing CJK characters for correct rendering (handles emoji adjacency, already-spaced) - Add JSON pretty-print: auto-format JSON code blocks with 2-space indent - Add 31 unit tests covering all post-processing functions - Fix pandoc simple table detection (1-space column gaps) - Fix image path double-nesting when --assets-dir ends with 'media' - Rename all markdown-tools references across 15 files (README, QUICKSTART, marketplace.json, CLAUDE.md, meeting-minutes-taker, GitHub templates) - Add 5-tool benchmark report (Docling/MarkItDown/Pandoc/Mammoth/ours) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- .claude-plugin/marketplace.json | 2 +- .github/ISSUE_TEMPLATE/bug_report.md | 2 +- .github/ISSUE_TEMPLATE/feature_request.md | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 2 +- CHANGELOG.md | 7 +- CLAUDE.md | 2 +- QUICKSTART.md | 4 +- QUICKSTART.zh-CN.md | 4 +- README.md | 14 +- README.zh-CN.md | 14 +- demos/README.md | 2 +- doc-to-markdown/SKILL.md | 83 +++--- .../references/heavy-mode-guide.md | 2 +- doc-to-markdown/scripts/convert.py | 45 +++- doc-to-markdown/scripts/test_convert.py | 242 ++++++++++++++++++ meeting-minutes-taker/SKILL.md | 4 +- 16 files changed, 346 insertions(+), 85 deletions(-) create mode 100644 doc-to-markdown/scripts/test_convert.py diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 2286d3df..16bf1f32 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -668,7 +668,7 @@ }, { "name": "meeting-minutes-taker", - "description": "Transform meeting transcripts into high-fidelity, structured meeting minutes with iterative review. Features speaker identification via feature analysis (word count, speaking style, topic focus) with context.md team directory mapping, intelligent file naming from content, integration with markdown-tools and transcript-fixer for pre-processing, evidence-based recording with speaker quotes, Mermaid diagrams for architecture discussions, and multi-turn parallel generation with UNION merge", + "description": "Transform meeting transcripts into high-fidelity, structured meeting minutes with iterative review. Features speaker identification via feature analysis (word count, speaking style, topic focus) with context.md team directory mapping, intelligent file naming from content, integration with doc-to-markdown and transcript-fixer for pre-processing, evidence-based recording with speaker quotes, Mermaid diagrams for architecture discussions, and multi-turn parallel generation with UNION merge", "source": "./", "strict": false, "version": "1.1.0", diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 51966ef5..7de86f63 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -16,7 +16,7 @@ Which skill is affected? - [ ] skill-creator - [ ] github-ops -- [ ] markdown-tools +- [ ] doc-to-markdown - [ ] mermaid-tools - [ ] statusline-generator - [ ] teams-channel-post-writer diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index edaf3704..050c3e22 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -20,7 +20,7 @@ Which skill would this enhance? - [ ] skill-creator - [ ] github-ops -- [ ] markdown-tools +- [ ] doc-to-markdown - [ ] mermaid-tools - [ ] statusline-generator - [ ] teams-channel-post-writer diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 27a4c902..54d20160 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -33,7 +33,7 @@ Which skills are affected by this PR? - [ ] skill-creator - [ ] github-ops -- [ ] markdown-tools +- [ ] doc-to-markdown - [ ] mermaid-tools - [ ] statusline-generator - [ ] teams-channel-post-writer diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e628655..b9558d97 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Added -- None +### Changed +- **Renamed**: `markdown-tools` → `doc-to-markdown` — clearer name for DOCX/PDF/PPTX → Markdown conversion +- **doc-to-markdown**: Added 8 DOCX post-processing fixes (grid tables, simple tables, CJK bold spacing, JSON pretty-print, image path flattening, pandoc attribute cleanup, code block detection, bracket fixes) +- **doc-to-markdown**: Added 31 unit tests (`test_convert.py`) +- **doc-to-markdown**: Added 5-tool benchmark report (`references/benchmark-2026-03-22.md`) ## [1.39.0] - 2026-03-18 diff --git a/CLAUDE.md b/CLAUDE.md index 222a437f..d8d51413 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -179,7 +179,7 @@ This applies when you change ANY file under a skill directory: 1. **skill-creator** ⭐ - **Essential meta-skill** for creating your own skills (with init/validate/package scripts) 2. **github-ops** - GitHub operations via gh CLI and API -3. **markdown-tools** - Document conversion with WSL path handling +3. **doc-to-markdown** - DOCX/PDF/PPTX → Markdown conversion with CJK post-processing 4. **mermaid-tools** - Diagram extraction and PNG generation 5. **statusline-generator** - Claude Code statusline customization 6. **teams-channel-post-writer** - Teams communication templates diff --git a/QUICKSTART.md b/QUICKSTART.md index ec254187..2f3abb85 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -122,7 +122,7 @@ claude plugin marketplace add https://github.com/daymade/claude-code-skills # In Claude Code use `/plugin ...`; in your terminal use `claude plugin ...` # Step 2: Install skills you need claude plugin install github-ops@daymade-skills -claude plugin install markdown-tools@daymade-skills +claude plugin install doc-to-markdown@daymade-skills # ... add more as needed # Step 3: Restart Claude Code @@ -136,7 +136,7 @@ This table is a quick starter list. See [README.md](./README.md) for the full ca |-------|-------------|-------------| | **skill-creator** ⭐ | Create your own skills | Building custom workflows | | **github-ops** | GitHub operations | Managing PRs, issues, workflows | -| **markdown-tools** | Document conversion | Converting docs to markdown | +| **doc-to-markdown** | Document conversion | Converting docs to markdown | | **mermaid-tools** | Diagram generation | Creating PNG diagrams | | **statusline-generator** | Statusline customization | Customizing Claude Code UI | | **teams-channel-post-writer** | Teams communication | Writing professional posts | diff --git a/QUICKSTART.zh-CN.md b/QUICKSTART.zh-CN.md index a188f556..7b865a96 100644 --- a/QUICKSTART.zh-CN.md +++ b/QUICKSTART.zh-CN.md @@ -122,7 +122,7 @@ claude plugin marketplace add https://github.com/daymade/claude-code-skills # 在 Claude Code 内使用 `/plugin ...`，在终端中使用 `claude plugin ...` # 步骤 2：安装你需要的技能 claude plugin install github-ops@daymade-skills -claude plugin install markdown-tools@daymade-skills +claude plugin install doc-to-markdown@daymade-skills # ... 根据需要添加更多 # 步骤 3：重启 Claude Code @@ -136,7 +136,7 @@ claude plugin install markdown-tools@daymade-skills |-------|-------------|-------------| | **skill-creator** ⭐ | 创建你自己的技能 | 构建自定义工作流 | | **github-ops** | GitHub 操作 | 管理 PR、问题、工作流 | -| **markdown-tools** | 文档转换 | 将文档转换为 markdown | +| **doc-to-markdown** | 文档转换 | 将文档转换为 markdown | | **mermaid-tools** | 图表生成 | 创建 PNG 图表 | | **statusline-generator** | 状态栏定制 | 自定义 Claude Code UI | | **teams-channel-post-writer** | Teams 通信 | 编写专业帖子 | diff --git a/README.md b/README.md index 7f4eeff3..53a3e9fe 100644 --- a/README.md +++ b/README.md @@ -146,7 +146,7 @@ claude plugin install skill-creator@daymade-skills claude plugin install github-ops@daymade-skills # Document conversion -claude plugin install markdown-tools@daymade-skills +claude plugin install doc-to-markdown@daymade-skills # Diagram generation claude plugin install mermaid-tools@daymade-skills @@ -294,7 +294,7 @@ Comprehensive GitHub operations using gh CLI and GitHub API. --- -### 2. **markdown-tools** - Document Conversion Suite +### 2. **doc-to-markdown** - Document Conversion Suite Converts documents to markdown with Windows/WSL path handling and PDF image extraction. @@ -313,7 +313,7 @@ Converts documents to markdown with Windows/WSL path handling and PDF image extr **🎬 Live Demo** -![Markdown Tools Demo](./demos/markdown-tools/convert-docs.gif) +![Markdown Tools Demo](./demos/doc-to-markdown/convert-docs.gif) --- @@ -1838,7 +1838,7 @@ Want to see all demos in one place with click-to-enlarge functionality? Check ou Use **github-ops** to streamline PR creation, issue management, and API operations. ### For Documentation -Combine **markdown-tools** for document conversion and **mermaid-tools** for diagram generation to create comprehensive documentation. Use **llm-icon-finder** to add brand icons. +Combine **doc-to-markdown** for document conversion and **mermaid-tools** for diagram generation to create comprehensive documentation. Use **llm-icon-finder** to add brand icons. ### For Research & Analysis Use **deep-research** to produce format-controlled research reports with evidence tables and citations. Combine with **fact-checker** to validate claims or with **twitter-reader** for social-source collection. @@ -1916,7 +1916,7 @@ Use **iOS-APP-developer** to configure XcodeGen projects, resolve SPM dependency Use **macos-cleaner** to intelligently analyze and reclaim disk space on macOS with safety-first approach. Unlike one-click cleaners that blindly delete, macos-cleaner explains what each file is, categorizes by risk level (🟢/🟡/🔴), and requires explicit confirmation before any deletion. Perfect for developers dealing with Docker/Homebrew/npm/pip cache bloat, users wanting to understand storage consumption, or anyone who values transparency over automation. Combines script-based precision with optional Mole visual tool integration for hybrid workflow. ### For Twitter/X Content Research -Use **twitter-reader** to fetch tweet content without JavaScript rendering or authentication. Perfect for documenting social media discussions, archiving threads, analyzing tweet content, or gathering reference material from Twitter/X. Combine with **markdown-tools** to convert fetched content into other formats, or with **repomix-safe-mixer** to package research collections securely. +Use **twitter-reader** to fetch tweet content without JavaScript rendering or authentication. Perfect for documenting social media discussions, archiving threads, analyzing tweet content, or gathering reference material from Twitter/X. Combine with **doc-to-markdown** to convert fetched content into other formats, or with **repomix-safe-mixer** to package research collections securely. ### For Skill Quality & Open-Source Contributions Use **skill-reviewer** to validate your own skills against best practices before publishing, or to review and improve others' skill repositories. Combine with **github-contributor** to find high-impact open-source projects, create professional PRs, and build your contributor reputation. Perfect for developers who want to contribute to the Claude Code ecosystem or any GitHub project systematically. @@ -1947,7 +1947,7 @@ Each skill includes: ### Quick Links - **github-ops**: See `github-ops/references/api_reference.md` for API documentation -- **markdown-tools**: See `markdown-tools/references/conversion-examples.md` for conversion scenarios +- **doc-to-markdown**: See `doc-to-markdown/references/conversion-examples.md` for conversion scenarios - **mermaid-tools**: See `mermaid-tools/references/setup_and_troubleshooting.md` for setup guide - **statusline-generator**: See `statusline-generator/references/color_codes.md` for customization - **teams-channel-post-writer**: See `teams-channel-post-writer/references/writing-guidelines.md` for quality standards @@ -1992,7 +1992,7 @@ Each skill includes: - **Claude Code** 2.0.13 or higher - **Python 3.6+** (for scripts in multiple skills) - **gh CLI** (for github-ops) -- **markitdown** (for markdown-tools) +- **markitdown** (for doc-to-markdown) - **mermaid-cli** (for mermaid-tools) - **yt-dlp** (for youtube-downloader): `brew install yt-dlp` or `pip install yt-dlp` - **FFmpeg/FFprobe** (for video-comparer): `brew install ffmpeg`, `apt install ffmpeg`, or `winget install ffmpeg` diff --git a/README.zh-CN.md b/README.zh-CN.md index 021e8e5e..f4b354f5 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -146,7 +146,7 @@ claude plugin install skill-creator@daymade-skills claude plugin install github-ops@daymade-skills # 文档转换 -claude plugin install markdown-tools@daymade-skills +claude plugin install doc-to-markdown@daymade-skills # 图表生成 claude plugin install mermaid-tools@daymade-skills @@ -319,7 +319,7 @@ CC-Switch 支持以下中国 AI 服务提供商： --- -### 2. **markdown-tools** - 文档转换套件 +### 2. **doc-to-markdown** - 文档转换套件 将文档转换为 markdown，支持 Windows/WSL 路径处理和 PDF 图片提取。 @@ -338,7 +338,7 @@ CC-Switch 支持以下中国 AI 服务提供商： **🎬 实时演示** -![Markdown 工具演示](./demos/markdown-tools/convert-docs.gif) +![Markdown 工具演示](./demos/doc-to-markdown/convert-docs.gif) --- @@ -1880,7 +1880,7 @@ claude plugin install scrapling-skill@daymade-skills 使用 **github-ops** 简化 PR 创建、问题管理和 API 操作。 ### 文档处理 -结合 **markdown-tools** 进行文档转换和 **mermaid-tools** 进行图表生成，创建全面的文档。使用 **llm-icon-finder** 添加品牌图标。 +结合 **doc-to-markdown** 进行文档转换和 **mermaid-tools** 进行图表生成，创建全面的文档。使用 **llm-icon-finder** 添加品牌图标。 ### 调研与分析 使用 **deep-research** 生成格式可控的调研报告，包含证据表与引用。与 **fact-checker** 结合用于验证关键结论，或与 **twitter-reader** 结合收集社媒资料。 @@ -1952,7 +1952,7 @@ claude plugin install scrapling-skill@daymade-skills 使用 **iOS-APP-developer** 配置 XcodeGen 项目，处理 SPM 依赖、签名与部署问题。 ### Twitter/X 内容研究 -使用 **twitter-reader** 无需 JavaScript 渲染或身份验证即可获取推文内容。非常适合记录社交媒体讨论、归档话题、分析推文内容或从 Twitter/X 收集参考资料。与 **markdown-tools** 结合可将获取的内容转换为其他格式，或与 **repomix-safe-mixer** 结合安全地打包研究集合。 +使用 **twitter-reader** 无需 JavaScript 渲染或身份验证即可获取推文内容。非常适合记录社交媒体讨论、归档话题、分析推文内容或从 Twitter/X 收集参考资料。与 **doc-to-markdown** 结合可将获取的内容转换为其他格式，或与 **repomix-safe-mixer** 结合安全地打包研究集合。 ### macOS 系统维护与磁盘空间恢复 使用 **macos-cleaner** 以安全优先的方式智能分析和恢复 macOS 上的磁盘空间。与盲目删除的一键清理工具不同，macos-cleaner 解释每个文件是什么、按风险级别分类（🟢/🟡/🔴），并在任何删除前需要明确确认。非常适合处理 Docker/Homebrew/npm/pip 缓存膨胀的开发者、希望了解存储空间消耗的用户，或任何重视透明度而非自动化的人。结合基于脚本的精度和可选的 Mole 可视化工具集成以实现混合工作流。 @@ -1989,7 +1989,7 @@ claude plugin install scrapling-skill@daymade-skills ### 快速链接 - **github-ops**：参见 `github-ops/references/api_reference.md` 了解 API 文档 -- **markdown-tools**：参见 `markdown-tools/references/conversion-examples.md` 了解转换场景 +- **doc-to-markdown**：参见 `doc-to-markdown/references/conversion-examples.md` 了解转换场景 - **mermaid-tools**：参见 `mermaid-tools/references/setup_and_troubleshooting.md` 了解设置指南 - **statusline-generator**：参见 `statusline-generator/references/color_codes.md` 了解自定义 - **teams-channel-post-writer**：参见 `teams-channel-post-writer/references/writing-guidelines.md` 了解质量标准 @@ -2034,7 +2034,7 @@ claude plugin install scrapling-skill@daymade-skills - **Claude Code** 2.0.13 或更高版本 - **Python 3.6+**（用于多个技能中的脚本） - **gh CLI**（用于 github-ops） -- **markitdown**（用于 markdown-tools） +- **markitdown**（用于 doc-to-markdown） - **mermaid-cli**（用于 mermaid-tools） - **VHS**（用于 cli-demo-generator）：`brew install vhs` - **asciinema**（可选，用于 cli-demo-generator 交互式录制） diff --git a/demos/README.md b/demos/README.md index e43db9be..aeb565b7 100644 --- a/demos/README.md +++ b/demos/README.md @@ -14,7 +14,7 @@ demos/ │ └── package-skill.tape # Package for distribution ├── github-ops/ │ └── create-pr.tape # Create pull requests -├── markdown-tools/ +├── doc-to-markdown/ │ └── convert-docs.tape # Convert documents └── generate_all_demos.sh # Generate all GIFs ``` diff --git a/doc-to-markdown/SKILL.md b/doc-to-markdown/SKILL.md index 966c9fc1..0dea85ed 100644 --- a/doc-to-markdown/SKILL.md +++ b/doc-to-markdown/SKILL.md @@ -1,75 +1,68 @@ --- name: doc-to-markdown -description: Converts DOCX/PDF/PPTX to high-quality Markdown with automatic post-processing. Fixes pandoc grid tables, image paths, attribute noise, and code blocks. Supports Quick Mode (fast, single tool) and Heavy Mode (best quality, multi-tool merge). Trigger on "convert document", "docx to markdown", "parse word", "doc to markdown", "extract images from document". +description: Converts DOCX/PDF/PPTX to high-quality Markdown with automatic post-processing. Fixes pandoc grid tables, simple tables, image paths, CJK bold spacing, attribute noise, and code blocks. Benchmarked best-in-class (7.6/10) against Docling, MarkItDown, Pandoc raw, and Mammoth. Trigger on "convert document", "docx to markdown", "parse word", "doc to markdown", "解析word", "转换文档". --- # Doc to Markdown Convert documents to high-quality markdown with intelligent multi-tool orchestration and automatic DOCX post-processing. -## Dual Mode Architecture - -| Mode | Speed | Quality | Use Case | -|------|-------|---------|----------| -| **Quick** (default) | Fast | Good | Drafts, simple documents | -| **Heavy** | Slower | Best | Final documents, complex layouts | +**Architecture**: Pandoc (best-in-class extraction) + 8 post-processing fixes (our value-add). ## Quick Start -### Installation - ```bash -# Required: PDF/DOCX/PPTX support -uv tool install "markitdown[pdf]" -pip install pymupdf4llm -brew install pandoc -``` - -### Basic Conversion +# DOCX → Markdown (one command, zero manual fixes) +uv run --with pymupdf4llm --with markitdown scripts/convert.py document.docx -o output.md --assets-dir ./media -```bash -# Quick Mode (default) - fast, single best tool +# PDF → Markdown uv run --with pymupdf4llm --with markitdown scripts/convert.py document.pdf -o output.md -# Heavy Mode - multi-tool parallel execution with merge -uv run --with pymupdf4llm --with markitdown scripts/convert.py document.pdf -o output.md --heavy +# Run tests +uv run --with pytest pytest scripts/test_convert.py -v +``` -# DOCX with deep python-docx parsing (experimental) -uv run --with pymupdf4llm --with markitdown --with python-docx scripts/convert.py document.docx -o output.md --docx-deep +## Dual Mode -# Check available tools -uv run scripts/convert.py --list-tools -``` +| Mode | Speed | Quality | Use Case | +|------|-------|---------|----------| +| **Quick** (default) | Fast | Good | Drafts, simple documents | +| **Heavy** | Slower | Best | Final documents, complex layouts | -## Tool Selection Matrix +## Tool Selection -| Format | Quick Mode Tool | Heavy Mode Tools | -|--------|----------------|------------------| +| Format | Quick Mode | Heavy Mode | +|--------|-----------|------------| | PDF | pymupdf4llm | pymupdf4llm + markitdown | | DOCX | pandoc + post-processing | pandoc + markitdown | | PPTX | markitdown | markitdown + pandoc | | XLSX | markitdown | markitdown | -### Tool Characteristics +## DOCX Post-Processing (automatic) -- **pymupdf4llm**: LLM-optimized PDF conversion with native table detection and image extraction -- **markitdown**: Microsoft's universal converter, good for Office formats -- **pandoc**: Excellent structure preservation for DOCX/PPTX +When converting DOCX via pandoc, 8 cleanups are applied automatically: -## DOCX Post-Processing (automatic) +| Problem | Fix | Test coverage | +|---------|-----|---------------| +| Grid tables (`+:---+`) | Single-column → blockquote, multi-column → pipe table | `TestPostprocessPipeline` | +| Simple tables (` ---- ----`) | Multi-column images → pipe table with captions | `TestSimpleTable` | +| Image path nesting (`media/media/`) | Flatten to `media/`, absolute → relative | `test_stats_tracking` | +| Pandoc attributes (`{width="..."}`) | Removed | `test_pandoc_attributes_removed` | +| CJK bold spacing (`**粗体**中文`) | Add space around `**` for CJK bold spans | `TestCjkBoldSpacing` (15 cases) | +| Indented dashed code blocks | → fenced ``` with language detection | `test_code_block_with_language` | +| Escaped brackets (`\[...\]`) | → `[...]` | `test_escaped_brackets_fixed` | +| Double-bracket links (`[[text]](url)`) | → `[text](url)` | `test_double_bracket_links_fixed` | + +### CJK Bold Spacing — why and how -When converting DOCX files via pandoc, the following cleanups are applied automatically: +DOCX uses run-level styling (no spaces between bold/normal runs in CJK text). Markdown renderers need whitespace around `**` to recognize bold boundaries. -| Problem | Fix | -|---------|-----| -| Grid tables (`+:---+` syntax) | Single-column -> blockquote, multi-column -> split images | -| Image double path (`media/media/`) | Flatten to `media/` | -| Pandoc attributes (`{width="..." height="..."}`) | Removed | -| Inline classes (`{.underline}`, `{.mark}`) | Removed | -| Indented dashed code blocks | Converted to fenced code blocks (```) | -| Escaped brackets (`\[...\]`) | Unescaped to `[...]` | -| Double-bracket links (`[[text]{...}](url)`) | Simplified to `[text](url)` | -| Escaped quotes in code (`\"`) | Fixed to `"` | +**Rule**: if a `**content**` span contains any CJK character, ensure both sides have a space — unless already spaced or at line boundary. This handles CJK punctuation, emoji adjacency, and mixed content. + +``` +Before: 打开**飞书**，就可以 → some renderers fail to bold +After: 打开 **飞书** ，就可以 → universally renders correctly +``` ## Heavy Mode Workflow @@ -166,6 +159,7 @@ brew install pandoc | Script | Purpose | |--------|---------| | `convert.py` | Main orchestrator with Quick/Heavy mode + DOCX post-processing | +| `test_convert.py` | 31 tests covering all post-processing functions | | `merge_outputs.py` | Merge multiple markdown outputs | | `validate_output.py` | Quality validation with HTML report | | `extract_pdf_images.py` | PDF image extraction with metadata | @@ -173,6 +167,7 @@ brew install pandoc ## References +- `references/benchmark-2026-03-22.md` - 5-tool benchmark (Docling/MarkItDown/Pandoc/Mammoth/ours) - `references/heavy-mode-guide.md` - Detailed Heavy Mode documentation - `references/tool-comparison.md` - Tool capabilities comparison - `references/conversion-examples.md` - Batch operation examples diff --git a/doc-to-markdown/references/heavy-mode-guide.md b/doc-to-markdown/references/heavy-mode-guide.md index 442cb6e1..3befcab7 100644 --- a/doc-to-markdown/references/heavy-mode-guide.md +++ b/doc-to-markdown/references/heavy-mode-guide.md @@ -1,6 +1,6 @@ # Heavy Mode Guide -Detailed documentation for markdown-tools Heavy Mode conversion. +Detailed documentation for doc-to-markdown Heavy Mode conversion. ## Overview diff --git a/doc-to-markdown/scripts/convert.py b/doc-to-markdown/scripts/convert.py index a95eb86f..a3c97475 100755 --- a/doc-to-markdown/scripts/convert.py +++ b/doc-to-markdown/scripts/convert.py @@ -26,6 +26,7 @@ """ import argparse +import json import re import subprocess import sys @@ -478,10 +479,19 @@ def _fix_code_blocks(text: str, stats: PostProcessStats) -> str: # Decide: code block vs blockquote if has_lang_hint or _is_code_content(cleaned): - # Code block + # Code block — try to pretty-print JSON + code_lines = cleaned + if lang_hint == "json": + try: + raw = "\n".join(cleaned) + parsed = json.loads(raw) + code_lines = json.dumps(parsed, indent=2, ensure_ascii=False).split("\n") + except (json.JSONDecodeError, ValueError): + pass # Keep original if not valid JSON + result.append("") result.append(f"```{lang_hint}") - result.extend(cleaned) + result.extend(code_lines) result.append("```") result.append("") else: @@ -529,29 +539,40 @@ def _replace_link(m: re.Match) -> str: def _fix_cjk_bold_spacing(text: str) -> str: - """Add space between **bold** markers and adjacent CJK characters. + """Add space around **bold** spans that contain CJK characters. DOCX uses run-level styling for bold — no spaces between runs in CJK text. Markdown renderers need whitespace around ** to recognize bold boundaries. - We find each **content** span, check the character before/after, and insert - a space only when the adjacent character is CJK (avoiding double spaces). + + Rule: if a **content** span contains any CJK character, ensure both sides + have a space (unless already spaced or at line boundary). This handles: + - CJK directly touching **: 打开**飞书** → 打开 **飞书** + - Emoji touching **: **密码】**➡️ → **密码】** ➡️ + - Already spaced: 已有 **粗体** → unchanged + - English bold: English **bold** text → unchanged """ result = [] last_end = 0 for m in _RE_BOLD_PAIR.finditer(text): start, end = m.start(), m.end() + content = m.group(1) + result.append(text[last_end:start]) - # Space before opening ** if preceded by CJK - if start > 0 and _RE_CJK_PUNCT.match(text[start - 1]): - result.append(' ') + # Only add spaces for bold spans containing CJK + if _RE_CJK_PUNCT.search(content): + # Space before ** if previous char is not whitespace + if start > 0 and text[start - 1] not in (' ', '\t', '\n'): + result.append(' ') - result.append(m.group(0)) + result.append(m.group(0)) - # Space after closing ** if followed by CJK - if end < len(text) and _RE_CJK_PUNCT.match(text[end]): - result.append(' ') + # Space after ** if next char is not whitespace + if end < len(text) and text[end] not in (' ', '\t', '\n'): + result.append(' ') + else: + result.append(m.group(0)) last_end = end diff --git a/doc-to-markdown/scripts/test_convert.py b/doc-to-markdown/scripts/test_convert.py new file mode 100644 index 00000000..d334209b --- /dev/null +++ b/doc-to-markdown/scripts/test_convert.py @@ -0,0 +1,242 @@ +"""Tests for doc-to-markdown convert.py post-processing functions. + +Run: uv run pytest scripts/test_convert.py -v +""" + +import pytest +import re +import sys +from pathlib import Path + +# Import the module under test +sys.path.insert(0, str(Path(__file__).parent)) +from convert import ( + _fix_cjk_bold_spacing, + _build_pipe_table, + _collect_images, + PostProcessStats, + postprocess_docx_markdown, +) + + +# ── CJK Bold Spacing ───────────────────────────────────────────────────────── + + +class TestCjkBoldSpacing: + """Test _fix_cjk_bold_spacing: spaces between **bold** and CJK chars.""" + + def test_bold_followed_by_cjk_punctuation(self): + """**text** directly touching CJK colon → add space after **.""" + inp = "**打开阶跃开放平台链接**：https://platform.stepfun.com/" + out = _fix_cjk_bold_spacing(inp) + assert "**打开阶跃开放平台链接** ：" in out + + def test_cjk_before_bold(self): + """CJK char directly before ** → add space before **.""" + assert _fix_cjk_bold_spacing("可用**手机号**进行") == "可用 **手机号** 进行" + + def test_bold_with_emoji_neighbor(self): + """**text** touching emoji ➡️ → still add space (CJK content rule).""" + inp = "点击**【接口密码】**➡️**【创建新的密钥**】" + out = _fix_cjk_bold_spacing(inp) + # Each CJK-containing bold span should have spaces on both sides + assert "点击 **【接口密码】** ➡️" in out + assert "➡️ **【创建新的密钥**" in out + + def test_full_emoji_line(self): + """Complete line with emoji separators between bold spans.""" + inp = "点击**【接口密码】**➡️**【创建新的密钥**】➡️**【输入密钥名称】**（输入你想取的名称），生成API Key" + out = _fix_cjk_bold_spacing(inp) + assert "点击 **【接口密码】** ➡️" in out + assert "**【输入密钥名称】** （输入" in out + + def test_bold_between_cjk(self): + """CJK **text** CJK → spaces on both sides.""" + assert _fix_cjk_bold_spacing("打开**飞书**，就可以") == "打开 **飞书** ，就可以" + + def test_bold_with_chinese_quotes(self): + """Bold containing Chinese quotes.""" + inp = '有个**"企鹅戴龙虾头套的机器人"**，开始' + out = _fix_cjk_bold_spacing(inp) + assert '**"企鹅戴龙虾头套的机器人"** ，' in out + + def test_multiple_bold_spans(self): + """Multiple bold spans in one line.""" + assert _fix_cjk_bold_spacing("这是**测试**和**验证**的效果") == "这是 **测试** 和 **验证** 的效果" + + def test_already_spaced(self): + """Already has spaces → no double spaces.""" + inp = "已有空格 **粗体** 不需要再加" + assert _fix_cjk_bold_spacing(inp) == inp + + def test_english_unchanged(self): + """English bold text should not be modified.""" + inp = "English **bold** text should not change" + assert _fix_cjk_bold_spacing(inp) == inp + + def test_line_start_bold(self): + """Bold at line start followed by CJK.""" + assert _fix_cjk_bold_spacing("**重要**内容") == "**重要** 内容" + + def test_line_start_bold_standalone(self): + """Bold at line start with no CJK neighbor → no change.""" + assert _fix_cjk_bold_spacing("**这是纯粗体不需要改**") == "**这是纯粗体不需要改**" + + def test_no_bold(self): + """Text without bold markers → unchanged.""" + inp = "这是普通文本，没有粗体" + assert _fix_cjk_bold_spacing(inp) == inp + + def test_empty_string(self): + assert _fix_cjk_bold_spacing("") == "" + + def test_bold_at_line_end(self): + """Bold at line end → no trailing space needed.""" + assert _fix_cjk_bold_spacing("内容是**粗体**") == "内容是 **粗体**" + + def test_mixed_cjk_and_english_bold(self): + """English bold between CJK → no change (no CJK in content).""" + inp = "请使用 **API Key** 进行认证" + assert _fix_cjk_bold_spacing(inp) == inp + + +# ── Pipe Table Builder ──────────────────────────────────────────────────────── + + +class TestBuildPipeTable: + """Test _build_pipe_table: rows → markdown pipe table.""" + + def test_basic_table(self): + rows = [["a", "b"], ["c", "d"]] + result = _build_pipe_table(rows) + assert result == [ + "| | |", + "| --- | --- |", + "| a | b |", + "| c | d |", + ] + + def test_uneven_rows(self): + """Rows with different column counts → padded.""" + rows = [["a", "b", "c"], ["d"]] + result = _build_pipe_table(rows) + assert "| d | | |" in result + + def test_single_cell(self): + rows = [["only"]] + result = _build_pipe_table(rows) + assert len(result) == 3 # header + sep + 1 row + + def test_empty_rows(self): + assert _build_pipe_table([]) == [] + + def test_image_with_caption(self): + """Images and captions should pair correctly in table.""" + rows = [ + ["![](img1.png)", "![](img2.png)"], + ["Step 1", "Step 2"], + ] + result = _build_pipe_table(rows) + assert "| ![](img1.png) | ![](img2.png) |" in result + assert "| Step 1 | Step 2 |" in result + + +# ── Full Post-Processing Pipeline ───────────────────────────────────────────── + + +class TestPostprocessPipeline: + """Integration tests for the full postprocess_docx_markdown pipeline.""" + + def test_grid_table_single_column_to_blockquote(self): + """Single-column grid table → blockquote.""" + inp = """+:---+ +| 注意事项 | ++----+""" + out, stats = postprocess_docx_markdown(inp) + assert "> 注意事项" in out + assert "+:---+" not in out + + def test_pandoc_attributes_removed(self): + """Pandoc {width=...} and {.underline} removed.""" + inp = '![](img.png){width="5in" height="3in"} and [text]{.underline}' + out, stats = postprocess_docx_markdown(inp) + assert "{width=" not in out + assert "{.underline}" not in out + assert "![](img.png)" in out + + def test_escaped_brackets_fixed(self): + r"""Pandoc \[ and \] → [ and ].""" + inp = r"你 \[在飞书里\] 发消息" + out, stats = postprocess_docx_markdown(inp) + assert "你 [在飞书里] 发消息" in out + + def test_double_bracket_links_fixed(self): + """[[text]](url) → [text](url).""" + inp = "[[点击跳转]](https://example.com)" + out, stats = postprocess_docx_markdown(inp) + assert "[点击跳转](https://example.com)" in out + + def test_code_block_with_language(self): + """Indented dashed block with JSON language hint → ```json.""" + inp = """ ------------------------------------------------------------------ + JSON\\ + {\\ + "provider": "stepfun"\\ + } + ------------------------------------------------------------------""" + out, stats = postprocess_docx_markdown(inp) + assert "```json" in out + assert '"provider": "stepfun"' in out + assert "---" not in out + + def test_code_block_plain_text_to_blockquote(self): + """Indented dashed block with plain text → blockquote.""" + inp = """ -------------------------- + 注意：这是一条重要提示 + --------------------------""" + out, stats = postprocess_docx_markdown(inp) + assert "> 注意：这是一条重要提示" in out + + def test_cjk_bold_spacing_in_pipeline(self): + """CJK bold spacing is applied in the full pipeline.""" + inp = "打开**飞书**，就可以看到" + out, stats = postprocess_docx_markdown(inp) + assert "打开 **飞书** ，就可以看到" in out + + def test_excessive_blank_lines_collapsed(self): + """4+ blank lines → 2 blank lines.""" + inp = "line1\n\n\n\n\nline2" + out, stats = postprocess_docx_markdown(inp) + assert out.count("\n") < 5 + + def test_stats_tracking(self): + """Stats object correctly tracks fix counts.""" + inp = '![](media/media/img.png){width="5in"}' + out, stats = postprocess_docx_markdown(inp) + assert stats.attributes_removed > 0 + + +# ── Simple Table (pandoc) ───────────────────────────────────────────────────── + + +class TestSimpleTable: + """Test pandoc simple table (indented dashes with spaces) → pipe table.""" + + def test_two_column_image_table(self): + """Two images side by side in simple table → pipe table.""" + inp = """ ---- ---- + ![](img1.png) ![](img2.png) + + ---- ----""" + out, stats = postprocess_docx_markdown(inp) + assert "| ![](img1.png) | ![](img2.png) |" in out + assert "----" not in out + + def test_four_column_image_table(self): + """Four images in simple table → 4-column pipe table.""" + inp = """ ---------- ---------- ---------- ---------- + ![](a.png) ![](b.png) ![](c.png) ![](d.png) + + ---------- ---------- ---------- ----------""" + out, stats = postprocess_docx_markdown(inp) + assert "| ![](a.png) | ![](b.png) | ![](c.png) | ![](d.png) |" in out diff --git a/meeting-minutes-taker/SKILL.md b/meeting-minutes-taker/SKILL.md index 6971e650..5ae91359 100644 --- a/meeting-minutes-taker/SKILL.md +++ b/meeting-minutes-taker/SKILL.md @@ -11,7 +11,7 @@ Transform raw meeting transcripts into comprehensive, evidence-based meeting min ## Quick Start **Pre-processing (Optional but Recommended):** -- **Document conversion**: Use `markdown-tools` skill to convert .docx/.pdf to Markdown first (preserves tables/images) +- **Document conversion**: Use `doc-to-markdown` skill to convert .docx/.pdf to Markdown first (preserves tables/images) - **Transcript cleanup**: Use `transcript-fixer` skill to fix ASR/STT errors if transcript quality is poor - **Context file**: Prepare `context.md` with team directory for accurate speaker identification @@ -457,7 +457,7 @@ If v3 has a flowchart for "Status Query Mechanism" but v1/v2 don't have it, that **Full pipeline for .docx transcripts:** ``` -Step 0: markdown-tools # Convert .docx → Markdown (preserves tables/images) +Step 0: doc-to-markdown # Convert .docx → Markdown (preserves tables/images) ↓ Step 0.5: transcript-fixer # Fix ASR errors (optional, if quality is poor) ↓ From b9facf3516dbcce7c34c17750c4570cac9f97b9f Mon Sep 17 00:00:00 2001 From: daymade <daymadev89@gmail.com> Date: Mon, 23 Mar 2026 03:20:20 +0800 Subject: [PATCH 016/137] fix(doc-to-markdown): update marketplace.json description and version to 2.1.0 Sync description with actual capabilities: CJK bold spacing, JSON pretty-print, simple table support, 31 tests, benchmark score. Add cjk/chinese keywords. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- .claude-plugin/marketplace.json | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 16bf1f32..3a11739e 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -51,10 +51,10 @@ }, { "name": "doc-to-markdown", - "description": "Converts DOCX/PDF/PPTX to high-quality Markdown with automatic post-processing. Fixes pandoc grid tables, image paths, attribute noise, and code blocks. Supports Quick Mode (fast, single tool) and Heavy Mode (best quality, multi-tool merge). Trigger on \"convert document\", \"docx to markdown\", \"parse word\", \"doc to markdown\", \"extract images from document\".", + "description": "Converts DOCX/PDF/PPTX to high-quality Markdown. Pandoc engine + 8 post-processing fixes: grid/simple tables to pipe tables, CJK bold spacing, JSON pretty-print, image path flattening, pandoc attribute cleanup, code block detection, bracket fixes. Benchmarked 7.6/10 (best-in-class vs Docling/MarkItDown/Mammoth). 31 unit tests. Trigger on \"convert document\", \"docx to markdown\", \"parse word\", \"doc to markdown\", \"解析word\", \"转换文档\".", "source": "./", "strict": false, - "version": "2.0.0", + "version": "2.1.0", "category": "document-conversion", "keywords": [ "markdown", @@ -63,7 +63,9 @@ "pptx", "converter", "pandoc", - "document" + "document", + "cjk", + "chinese" ], "skills": [ "./doc-to-markdown" From 87221d94d5efb1f68cfe52c4f17884d57ce4c55c Mon Sep 17 00:00:00 2001 From: daymade <daymadev89@gmail.com> Date: Thu, 2 Apr 2026 23:33:03 +0800 Subject: [PATCH 017/137] feat(pdf-creator): add theme system + Chrome backend; add terraform-skill draft - pdf-creator v1.2.0: theme system (default/warm-terra), dual backend (weasyprint/chrome auto-detect), argparse CLI, extracted CSS to themes/ - terraform-skill: operational traps from real deployments (provisioner timing, DNS duplication, multi-env isolation, pre-deploy validation) - asr-transcribe-to-text: add security scan marker Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --- .claude-plugin/marketplace.json | 6 +- asr-transcribe-to-text/.security-scan-passed | 4 + pdf-creator/SKILL.md | 65 +-- pdf-creator/scripts/md_to_pdf.py | 401 ++++++++++-------- pdf-creator/themes/default.css | 88 ++++ pdf-creator/themes/warm-terra.css | 121 ++++++ terraform-skill/SKILL.md | 233 ++++++++++ .../references/multi-env-isolation.md | 130 ++++++ .../references/pre-deploy-validation.md | 82 ++++ .../references/zero-to-deploy-checklist.md | 168 ++++++++ 10 files changed, 1091 insertions(+), 207 deletions(-) create mode 100644 asr-transcribe-to-text/.security-scan-passed create mode 100644 pdf-creator/themes/default.css create mode 100644 pdf-creator/themes/warm-terra.css create mode 100644 terraform-skill/SKILL.md create mode 100644 terraform-skill/references/multi-env-isolation.md create mode 100644 terraform-skill/references/pre-deploy-validation.md create mode 100644 terraform-skill/references/zero-to-deploy-checklist.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 3a11739e..1cf92e18 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -414,15 +414,17 @@ }, { "name": "pdf-creator", - "description": "Create PDF documents from markdown with proper Chinese font support using weasyprint. Use when converting markdown to PDF, generating formal documents (legal filings, trademark applications, reports), or when Chinese typography is required. Triggers include convert to PDF, generate PDF, markdown to PDF, or printable documents", + "description": "Create PDF documents from markdown with Chinese font support. Supports theme system (default for formal docs, warm-terra for training materials) and dual backend (weasyprint or Chrome). Triggers include convert to PDF, generate PDF, markdown to PDF, or printable documents", "source": "./", "strict": false, - "version": "1.1.0", + "version": "1.2.0", "category": "document-conversion", "keywords": [ "pdf", "markdown", "weasyprint", + "chrome", + "themes", "chinese-fonts", "document-generation", "legal", diff --git a/asr-transcribe-to-text/.security-scan-passed b/asr-transcribe-to-text/.security-scan-passed new file mode 100644 index 00000000..d93e23f4 --- /dev/null +++ b/asr-transcribe-to-text/.security-scan-passed @@ -0,0 +1,4 @@ +Security scan passed +Scanned at: 2026-03-22T23:58:59.508059 +Tool: gitleaks + pattern-based validation +Content hash: 5dbbc175b8bfd6c6c8ab97f4112bf1f48eb489ef2a55375f88266deade644cd4 diff --git a/pdf-creator/SKILL.md b/pdf-creator/SKILL.md index ff942554..ac3c809b 100644 --- a/pdf-creator/SKILL.md +++ b/pdf-creator/SKILL.md @@ -1,55 +1,60 @@ --- name: pdf-creator -description: Create PDF documents from markdown with proper Chinese font support using weasyprint. This skill should be used when converting markdown to PDF, generating formal documents (legal, trademark filings, reports), or when Chinese typography is required. Triggers include "convert to PDF", "generate PDF", "markdown to PDF", or any request for creating printable documents. +description: Create PDF documents from markdown with proper Chinese font support. Supports theme system (default for formal docs, warm-terra for training materials) and dual backend (weasyprint or Chrome). Triggers include "convert to PDF", "generate PDF", "markdown to PDF", or any request for creating printable documents. --- # PDF Creator -Create professional PDF documents from markdown with proper Chinese font support. +Create professional PDF documents from markdown with Chinese font support and theme system. ## Quick Start -Convert a single markdown file: - ```bash -cd ~/workspace/claude-code-skills/pdf-creator -uv run --with weasyprint --with markdown scripts/md_to_pdf.py input.md output.pdf -``` +# Default theme (formal: Songti SC + black/grey) +uv run --with weasyprint scripts/md_to_pdf.py input.md output.pdf -Batch convert multiple files: +# Warm theme (training: PingFang SC + terra cotta) +uv run --with weasyprint scripts/md_to_pdf.py input.md --theme warm-terra -```bash -uv run --with weasyprint --with markdown scripts/batch_convert.py *.md --output-dir ./pdfs +# No weasyprint? Use Chrome backend (auto-detected if weasyprint unavailable) +python scripts/md_to_pdf.py input.md --theme warm-terra --backend chrome + +# List available themes +python scripts/md_to_pdf.py --list-themes dummy.md ``` -macOS ARM (Homebrew) 的 `DYLD_LIBRARY_PATH` 会自动检测配置，无需手动设置。 +## Themes + +Stored in `themes/*.css`. Each theme is a standalone CSS file. -## Font Configuration +| Theme | Font | Color | Best for | +|-------|------|-------|----------| +| `default` | Songti SC + Heiti SC | Black/grey | Legal docs, contracts, formal reports | +| `warm-terra` | PingFang SC | Terra cotta (#d97756) + warm neutrals | Course outlines, training materials, workshops | -The scripts use these Chinese fonts (with fallbacks): +To create a new theme: copy `themes/default.css`, modify, save as `themes/your-theme.css`. -| Font Type | Primary | Fallbacks | -|-----------|---------|-----------| -| Body text | Songti SC | SimSun, STSong, Noto Serif CJK SC | -| Headings | Heiti SC | SimHei, STHeiti, Noto Sans CJK SC | +## Backends -## Output Specifications +The script auto-detects the best available backend: -- **Page size**: A4 -- **Margins**: 2.5cm top/bottom, 2cm left/right -- **Body font**: 12pt, 1.8 line height -- **Max file size**: Designed to stay under 2MB for form submissions +| Backend | Install | Pros | Cons | +|---------|---------|------|------| +| `weasyprint` | `pip install weasyprint` | Precise CSS rendering, no browser needed | Requires system libs (cairo, pango) | +| `chrome` | Google Chrome installed | Zero Python deps, great CJK support | Larger binary, slightly less CSS control | -## Common Use Cases +Override with `--backend chrome` or `--backend weasyprint`. -1. **Legal documents**: Trademark filings, contracts, evidence lists -2. **Reports**: Business reports, technical documentation -3. **Formal letters**: Official correspondence requiring print format +## Batch Convert + +```bash +uv run --with weasyprint scripts/batch_convert.py *.md --output-dir ./pdfs +``` ## Troubleshooting -**Problem**: Chinese characters display as boxes -**Solution**: Ensure Songti SC or other Chinese fonts are installed on the system +**Chinese characters display as boxes**: Ensure Chinese fonts are installed (Songti SC, PingFang SC, etc.) + +**weasyprint import error**: Run with `uv run --with weasyprint` or use `--backend chrome` instead. -**Problem**: `weasyprint` import error -**Solution**: Run with `uv run --with weasyprint --with markdown` to ensure dependencies +**Chrome header/footer appearing**: The script passes `--no-pdf-header-footer`. If it still appears, your Chrome version may not support this flag — update Chrome. diff --git a/pdf-creator/scripts/md_to_pdf.py b/pdf-creator/scripts/md_to_pdf.py index a1ce7575..4fd9ed6a 100644 --- a/pdf-creator/scripts/md_to_pdf.py +++ b/pdf-creator/scripts/md_to_pdf.py @@ -1,150 +1,110 @@ #!/usr/bin/env python3 """ -Markdown to PDF converter with Chinese font support. +Markdown to PDF converter with Chinese font support and theme system. -Converts markdown files to PDF using pandoc (markdown→HTML) + weasyprint (HTML→PDF). -Designed for formal documents (trademark filings, legal documents, reports). +Converts markdown files to PDF using: + - pandoc (markdown → HTML) + - weasyprint or headless Chrome (HTML → PDF), auto-detected Usage: python md_to_pdf.py input.md output.pdf - python md_to_pdf.py input.md # outputs input.pdf + python md_to_pdf.py input.md --theme warm-terra + python md_to_pdf.py input.md --theme default --backend chrome + python md_to_pdf.py input.md # outputs input.pdf, default theme, auto backend + +Themes: + Stored in ../themes/*.css. Built-in themes: + - default: Songti SC + black/grey, formal documents + - warm-terra: PingFang SC + terra cotta, training/workshop materials Requirements: - pip install weasyprint pandoc (system install, e.g. brew install pandoc) - - macOS environment setup (if needed): - export DYLD_LIBRARY_PATH="/opt/homebrew/lib:$DYLD_LIBRARY_PATH" + weasyprint (pip install weasyprint) OR Google Chrome (for --backend chrome) """ +from __future__ import annotations + +import argparse import os import platform import re import shutil import subprocess import sys +import tempfile from pathlib import Path -# Auto-configure library path on macOS ARM (Homebrew) — must be before weasyprint import -if platform.system() == 'Darwin': - _homebrew_lib = '/opt/homebrew/lib' +SCRIPT_DIR = Path(__file__).resolve().parent +THEMES_DIR = SCRIPT_DIR.parent / "themes" + +# macOS ARM: auto-configure library path for weasyprint +if platform.system() == "Darwin": + _homebrew_lib = "/opt/homebrew/lib" if Path(_homebrew_lib).is_dir(): - _cur = os.environ.get('DYLD_LIBRARY_PATH', '') + _cur = os.environ.get("DYLD_LIBRARY_PATH", "") if _homebrew_lib not in _cur: - os.environ['DYLD_LIBRARY_PATH'] = f"{_homebrew_lib}:{_cur}" if _cur else _homebrew_lib - -from weasyprint import CSS, HTML - - -# CSS with Chinese font support -CSS_STYLES = """ -@page { - size: A4; - margin: 2.5cm 2cm; -} - -body { - font-family: 'Songti SC', 'SimSun', 'STSong', 'Noto Serif CJK SC', serif; - font-size: 12pt; - line-height: 1.8; - color: #000; - width: 100%; -} - -h1 { - font-family: 'Heiti SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', sans-serif; - font-size: 18pt; - font-weight: bold; - text-align: center; - margin-top: 0; - margin-bottom: 1.5em; -} - -h2 { - font-family: 'Heiti SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', sans-serif; - font-size: 14pt; - font-weight: bold; - margin-top: 1.5em; - margin-bottom: 0.8em; -} - -h3 { - font-family: 'Heiti SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', sans-serif; - font-size: 12pt; - font-weight: bold; - margin-top: 1em; - margin-bottom: 0.5em; -} - -p { - margin: 0.8em 0; - text-align: justify; -} - -ul, ol { - margin: 0.8em 0; - padding-left: 2em; -} - -li { - margin: 0.4em 0; -} - -table { - border-collapse: collapse; - width: 100%; - margin: 1em 0; - font-size: 10pt; - table-layout: fixed; -} - -th, td { - border: 1px solid #666; - padding: 8px 6px; - text-align: left; - overflow-wrap: break-word; - word-break: normal; -} - -th { - background-color: #f0f0f0; - font-weight: bold; -} - -hr { - border: none; - border-top: 1px solid #ccc; - margin: 1.5em 0; -} - -strong { - font-weight: bold; -} - -code { - font-family: 'SF Mono', 'Monaco', 'Menlo', monospace; - font-size: 10pt; - background-color: #f5f5f5; - padding: 0.2em 0.4em; - border-radius: 3px; -} - -pre { - background-color: #f5f5f5; - padding: 1em; - overflow-x: auto; - font-size: 10pt; - line-height: 1.4; - border-radius: 4px; -} - -blockquote { - border-left: 3px solid #ccc; - margin: 1em 0; - padding-left: 1em; - color: #555; -} -""" + os.environ["DYLD_LIBRARY_PATH"] = ( + f"{_homebrew_lib}:{_cur}" if _cur else _homebrew_lib + ) + + +def _find_chrome() -> str | None: + """Find Chrome/Chromium binary path.""" + candidates = [ + "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", + "/Applications/Chromium.app/Contents/MacOS/Chromium", + shutil.which("google-chrome"), + shutil.which("chromium"), + shutil.which("chrome"), + ] + for c in candidates: + if c and Path(c).exists(): + return str(c) + return None + + +def _has_weasyprint() -> bool: + """Check if weasyprint is importable.""" + try: + import weasyprint # noqa: F401 + + return True + except ImportError: + return False + + +def _detect_backend() -> str: + """Auto-detect best available backend: weasyprint > chrome.""" + if _has_weasyprint(): + return "weasyprint" + if _find_chrome(): + return "chrome" + print( + "Error: No PDF backend found. Install weasyprint (pip install weasyprint) " + "or Google Chrome.", + file=sys.stderr, + ) + sys.exit(1) + + +def _load_theme(theme_name: str) -> str: + """Load CSS from themes directory.""" + theme_file = THEMES_DIR / f"{theme_name}.css" + if not theme_file.exists(): + available = [f.stem for f in THEMES_DIR.glob("*.css")] + print( + f"Error: Theme '{theme_name}' not found. Available: {available}", + file=sys.stderr, + ) + sys.exit(1) + return theme_file.read_text(encoding="utf-8") + + +def _list_themes() -> list[str]: + """List available theme names.""" + if not THEMES_DIR.exists(): + return [] + return sorted(f.stem for f in THEMES_DIR.glob("*.css")) def _ensure_list_spacing(text: str) -> str: @@ -152,39 +112,36 @@ def _ensure_list_spacing(text: str) -> str: Both Python markdown library and pandoc require a blank line before a list when it follows a paragraph. Without it, list items render as plain text. - - This preprocessor adds blank lines before list items when needed, without - modifying the user's original markdown file. """ - lines = text.split('\n') + lines = text.split("\n") result = [] - list_re = re.compile(r'^(\s*)([-*+]|\d+\.)\s') + list_re = re.compile(r"^(\s*)([-*+]|\d+\.)\s") for i, line in enumerate(lines): if i > 0 and list_re.match(line): prev = lines[i - 1] if prev.strip() and not list_re.match(prev): - result.append('') + result.append("") result.append(line) - return '\n'.join(result) + return "\n".join(result) def _md_to_html(md_file: str) -> str: - """Convert markdown to HTML using pandoc with list spacing preprocessing. - - Reads the markdown file, preprocesses it to ensure proper list spacing, - then passes the content to pandoc via stdin. The original file is not modified. - """ - if not shutil.which('pandoc'): - print("Error: pandoc not found. Install with: brew install pandoc", file=sys.stderr) + """Convert markdown to HTML using pandoc with list spacing preprocessing.""" + if not shutil.which("pandoc"): + print( + "Error: pandoc not found. Install with: brew install pandoc", + file=sys.stderr, + ) sys.exit(1) - # Read and preprocess markdown to ensure list spacing - md_content = Path(md_file).read_text(encoding='utf-8') + md_content = Path(md_file).read_text(encoding="utf-8") md_content = _ensure_list_spacing(md_content) result = subprocess.run( - ['pandoc', '-f', 'markdown', '-t', 'html'], - input=md_content, capture_output=True, text=True, + ["pandoc", "-f", "markdown", "-t", "html"], + input=md_content, + capture_output=True, + text=True, ) if result.returncode != 0: print(f"Error: pandoc failed: {result.stderr}", file=sys.stderr) @@ -193,58 +150,152 @@ def _md_to_html(md_file: str) -> str: return result.stdout -def markdown_to_pdf(md_file: str, pdf_file: str | None = None) -> str: +def _build_full_html(html_content: str, css: str, title: str) -> str: + """Wrap HTML content in a full document with CSS.""" + return f"""<!DOCTYPE html> +<html lang="zh-CN"> +<head> + <meta charset="UTF-8"> + <title>{title} + + + +{html_content} + +""" + + +def _render_weasyprint(full_html: str, pdf_file: str, css: str) -> None: + """Render PDF using weasyprint.""" + from weasyprint import CSS, HTML + + HTML(string=full_html).write_pdf(pdf_file, stylesheets=[CSS(string=css)]) + + +def _render_chrome(full_html: str, pdf_file: str) -> None: + """Render PDF using headless Chrome.""" + chrome = _find_chrome() + if not chrome: + print("Error: Chrome not found.", file=sys.stderr) + sys.exit(1) + + with tempfile.NamedTemporaryFile( + suffix=".html", mode="w", encoding="utf-8", delete=False + ) as f: + f.write(full_html) + html_path = f.name + + try: + result = subprocess.run( + [ + chrome, + "--headless", + "--disable-gpu", + "--no-pdf-header-footer", + f"--print-to-pdf={pdf_file}", + html_path, + ], + capture_output=True, + text=True, + ) + if not Path(pdf_file).exists(): + print( + f"Error: Chrome failed to generate PDF. stderr: {result.stderr}", + file=sys.stderr, + ) + sys.exit(1) + finally: + Path(html_path).unlink(missing_ok=True) + + +def markdown_to_pdf( + md_file: str, + pdf_file: str | None = None, + theme: str = "default", + backend: str | None = None, +) -> str: """ - Convert markdown file to PDF with Chinese font support. + Convert markdown file to PDF. Args: md_file: Path to input markdown file - pdf_file: Path to output PDF file (optional, defaults to same name as input) + pdf_file: Path to output PDF (optional, defaults to same name as input) + theme: Theme name (from themes/ directory) + backend: 'weasyprint', 'chrome', or None (auto-detect) Returns: Path to generated PDF file """ md_path = Path(md_file) - if pdf_file is None: - pdf_file = str(md_path.with_suffix('.pdf')) + pdf_file = str(md_path.with_suffix(".pdf")) - # Convert to HTML via pandoc - html_content = _md_to_html(md_file) - - # Create full HTML document - full_html = f""" - - - - {md_path.stem} - - -{html_content} - -""" + if backend is None: + backend = _detect_backend() - # Generate PDF - HTML(string=full_html).write_pdf(pdf_file, stylesheets=[CSS(string=CSS_STYLES)]) + css = _load_theme(theme) + html_content = _md_to_html(md_file) + full_html = _build_full_html(html_content, css, md_path.stem) + + if backend == "weasyprint": + _render_weasyprint(full_html, pdf_file, css) + elif backend == "chrome": + _render_chrome(full_html, pdf_file) + else: + print(f"Error: Unknown backend '{backend}'", file=sys.stderr) + sys.exit(1) + size_kb = Path(pdf_file).stat().st_size / 1024 + print(f"Generated: {pdf_file} ({size_kb:.0f}KB, theme={theme}, backend={backend})") return pdf_file def main(): - if len(sys.argv) < 2: - print("Usage: python md_to_pdf.py [output.pdf]") - print("\nConverts markdown to PDF with Chinese font support.") - sys.exit(1) + available_themes = _list_themes() - md_file = sys.argv[1] - pdf_file = sys.argv[2] if len(sys.argv) > 2 else None + parser = argparse.ArgumentParser( + description="Markdown to PDF with Chinese font support and themes." + ) + parser.add_argument("input", help="Input markdown file") + parser.add_argument("output", nargs="?", help="Output PDF file (optional)") + parser.add_argument( + "--theme", + default="default", + choices=available_themes or ["default"], + help=f"CSS theme (available: {', '.join(available_themes) or 'default'})", + ) + parser.add_argument( + "--backend", + choices=["weasyprint", "chrome"], + default=None, + help="PDF rendering backend (default: auto-detect)", + ) + parser.add_argument( + "--list-themes", + action="store_true", + help="List available themes and exit", + ) - if not Path(md_file).exists(): - print(f"Error: File not found: {md_file}") + args = parser.parse_args() + + if args.list_themes: + for t in available_themes: + marker = " (default)" if t == "default" else "" + css_file = THEMES_DIR / f"{t}.css" + first_line = "" + for line in css_file.read_text().splitlines(): + line = line.strip() + if line.startswith("*") and "—" in line: + first_line = line.lstrip("* ").strip() + break + print(f" {t}{marker}: {first_line}") + sys.exit(0) + + if not Path(args.input).exists(): + print(f"Error: File not found: {args.input}", file=sys.stderr) sys.exit(1) - output = markdown_to_pdf(md_file, pdf_file) - print(f"Generated: {output}") + markdown_to_pdf(args.input, args.output, args.theme, args.backend) if __name__ == "__main__": diff --git a/pdf-creator/themes/default.css b/pdf-creator/themes/default.css new file mode 100644 index 00000000..255cd1a2 --- /dev/null +++ b/pdf-creator/themes/default.css @@ -0,0 +1,88 @@ +/* + * Default — PDF theme for formal documents + * + * Color palette: black/grey, no accent color + * Font: Songti SC (body) + Heiti SC (headings) + * Best for: legal documents, trademark filings, contracts, formal reports + * + * This is the original built-in theme from md_to_pdf.py, extracted for reference. + */ + +@page { + size: A4; + margin: 2.5cm 2cm; +} + +body { + font-family: 'Songti SC', 'SimSun', 'STSong', 'Noto Serif CJK SC', serif; + font-size: 12pt; + line-height: 1.8; + color: #000; + width: 100%; +} + +h1 { + font-family: 'Heiti SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', sans-serif; + font-size: 18pt; + font-weight: bold; + text-align: center; + margin-top: 0; + margin-bottom: 1.5em; +} + +h2 { + font-family: 'Heiti SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', sans-serif; + font-size: 14pt; + font-weight: bold; + margin-top: 1.5em; + margin-bottom: 0.8em; +} + +h3 { + font-family: 'Heiti SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', sans-serif; + font-size: 12pt; + font-weight: bold; + margin-top: 1em; + margin-bottom: 0.5em; +} + +p { + margin: 0.8em 0; + text-align: justify; +} + +ul, ol { + margin: 0.8em 0; + padding-left: 2em; +} + +li { + margin: 0.4em 0; +} + +table { + border-collapse: collapse; + width: 100%; + margin: 1em 0; + font-size: 10pt; + table-layout: fixed; +} + +th, td { + border: 1px solid #666; + padding: 8px 6px; + text-align: left; + overflow-wrap: break-word; + word-break: normal; +} + +th { + background-color: #f0f0f0; + font-weight: bold; +} + +hr { + border: none; + border-top: 1px solid #ccc; + margin: 1.5em 0; +} diff --git a/pdf-creator/themes/warm-terra.css b/pdf-creator/themes/warm-terra.css new file mode 100644 index 00000000..3af0f800 --- /dev/null +++ b/pdf-creator/themes/warm-terra.css @@ -0,0 +1,121 @@ +/* + * Warm Terra — PDF theme for workshop/training documents + * + * Color palette: terra cotta (#d97756) + warm neutrals + * Font: PingFang SC (macOS) / Microsoft YaHei (Windows) + * Best for: course outlines, training materials, workshop agendas + * + * Usage with md_to_pdf.py: + * python md_to_pdf.py input.md output.pdf --theme warm-terra + * + * Usage with pandoc + Chrome (fallback): + * pandoc input.md -o /tmp/out.html --standalone -H <(cat this-file.css wrapped in + + +{html_content} + +""" + + +def _render_weasyprint(full_html: str, pdf_file: str, css: str) -> None: + """Render PDF using weasyprint.""" + from weasyprint import CSS, HTML + + HTML(string=full_html).write_pdf(pdf_file, stylesheets=[CSS(string=css)]) + + +def _render_chrome(full_html: str, pdf_file: str) -> None: + """Render PDF using headless Chrome.""" + chrome = _find_chrome() + if not chrome: + print("Error: Chrome not found.", file=sys.stderr) + sys.exit(1) + + with tempfile.NamedTemporaryFile( + suffix=".html", mode="w", encoding="utf-8", delete=False + ) as f: + f.write(full_html) + html_path = f.name + + try: + result = subprocess.run( + [ + chrome, + "--headless", + "--disable-gpu", + "--no-pdf-header-footer", + f"--print-to-pdf={pdf_file}", + html_path, + ], + capture_output=True, + text=True, + ) + if not Path(pdf_file).exists(): + print( + f"Error: Chrome failed to generate PDF. stderr: {result.stderr}", + file=sys.stderr, + ) + sys.exit(1) + finally: + Path(html_path).unlink(missing_ok=True) + + +# ---------------- Visual self-check (post-render PNG previews) ---------------- +# +# Why: "PDF generated successfully" ≠ "PDF renders correctly". Common silent +# failures include paragraph collapsing (pseudo-list), table overflow, missing +# CJK / emoji glyphs, code block garbling. The author/AI must visually inspect +# the rendered output, not assume success from a clean exit code. +# +# Design: after PDF generation, automatically convert each page to PNG next to +# the PDF (via pdftoppm), and print a structured self-check checklist to stdout. +# This makes "Read each PNG and verify" the default contract of every PDF run, +# not an optional step that's easy to skip. +# +# Reference: CLAUDE.md "Self-Verification Protocol" + "Visual Verification" rules. + + +def _generate_pdf_previews(pdf_file: str, dpi: int = 130) -> list[Path]: + """Convert each PDF page to PNG for visual inspection. + + Returns sorted list of PNG paths. Empty list if pdftoppm not available + or no pages produced. Old previews in target dir are cleaned first. + """ + if not shutil.which("pdftoppm"): + return [] + + pdf_path = Path(pdf_file).resolve() + preview_dir = pdf_path.parent / f"{pdf_path.stem}-preview" + preview_dir.mkdir(exist_ok=True) + # Clean stale previews so old/extra pages don't linger after a shorter rerun + for old in preview_dir.glob("page-*.png"): + old.unlink() + + subprocess.run( + [ + "pdftoppm", + "-png", + "-r", + str(dpi), + str(pdf_path), + str(preview_dir / "page"), + ], + capture_output=True, + ) + + return sorted(preview_dir.glob("page-*.png")) + + +def _lint_pdf_typography(pdf_file: str) -> list[dict]: + """Detect inappropriate Chinese line breaks in rendered PDF. + + Uses `pdftotext -layout` to preserve visual line structure, then scans + each page for known typography anti-patterns per "中文文案排版指北": + + 1. Single CJK character alone on a line (cell too narrow → forced break + mid-word/mid-name) + 2. Line ending with 全角左括号 「（」 followed by content on next line + (broken parenthesis pair: opener separated from content) + 3. Line starting with 全角右括号 「）」 (same as #2 from receiving end) + 4. Short line ending with mid-thought punctuation 「、，；：」 right + before next CJK content (suggests forced break in narrow cell) + + Returns: list of dicts {page, line, kind, snippet, message}. + Empty list if pdftotext unavailable or PDF clean. + + Note: this only catches obvious cases. Subtle typography issues (uneven + line spacing, awkward breaks at safe points) require visual inspection. + """ + if not shutil.which("pdftotext"): + return [] + + findings: list[dict] = [] + + # Process each page separately to give accurate page numbers + page_count_result = subprocess.run( + ["pdfinfo", str(pdf_file)], + capture_output=True, + text=True, + ) + page_count = 0 + if page_count_result.returncode == 0: + for line in page_count_result.stdout.splitlines(): + if line.startswith("Pages:"): + try: + page_count = int(line.split(":", 1)[1].strip()) + except ValueError: + pass + break + + if page_count == 0: + return [] + + cjk_re = re.compile(r"[一-鿿]") + + for page_num in range(1, page_count + 1): + result = subprocess.run( + [ + "pdftotext", + "-layout", + "-f", + str(page_num), + "-l", + str(page_num), + str(pdf_file), + "-", + ], + capture_output=True, + text=True, + ) + if result.returncode != 0: + continue + + lines = result.stdout.split("\n") + for i, line in enumerate(lines): + stripped = line.strip() + if not stripped: + continue + + # Pattern 1: Single CJK character alone on a line + if len(stripped) == 1 and cjk_re.match(stripped): + findings.append({ + "page": page_num, + "line": i + 1, + "kind": "single-cjk-char", + "snippet": stripped, + "message": ( + f"Single CJK char 「{stripped}」 alone on a line — " + "cell width forced mid-word break" + ), + }) + continue + + # Pattern 2: Line ends with 全角左括号 followed by content next line + if stripped.endswith("（") and i + 1 < len(lines): + next_stripped = lines[i + 1].strip() + if next_stripped and not next_stripped.startswith("）"): + findings.append({ + "page": page_num, + "line": i + 1, + "kind": "broken-bracket-open", + "snippet": stripped[-15:], + "message": ( + "Line ends with 全角左括号「（」 and content " + "wraps to next line — broken bracket pair" + ), + }) + continue + + # Pattern 3: Line starts with 全角右括号 + if stripped.startswith("）"): + findings.append({ + "page": page_num, + "line": i + 1, + "kind": "broken-bracket-close", + "snippet": stripped[:15], + "message": ( + "Line starts with 全角右括号「）」 — " + "broken from previous line's bracket pair" + ), + }) + continue + + # Pattern 4: Line ends with 中文标点 (suggests forced break) + # This is informational — sometimes legitimate + if stripped.endswith(("、", "，", "；", "：")) and i + 1 < len(lines): + next_stripped = lines[i + 1].strip() + # Only warn if next line continues with CJK content (not a new section) + if next_stripped and cjk_re.match(next_stripped): + # Skip if this looks like end of a list item or paragraph (heuristic) + if len(stripped) < 30: # short cell content suggests forced break + findings.append({ + "page": page_num, + "line": i + 1, + "kind": "trailing-punctuation-break", + "snippet": stripped[-15:], + "message": ( + f"Short line ends with 「{stripped[-1]}」 mid-thought — " + "may be forced break in narrow cell" + ), + }) + + return findings + + +def _print_typography_findings(findings: list[dict]) -> None: + """Print typography lint findings to stderr.""" + if not findings: + return + print("", file=sys.stderr) + print( + f"⚠️ Typography lint: {len(findings)} potential Chinese line-break issue(s)", + file=sys.stderr, + ) + print( + " Per 中文文案排版指北 — narrow cells / over-wide tables often force " + "inappropriate breaks:", + file=sys.stderr, + ) + for f in findings[:20]: # cap output + print( + f" [page {f['page']} L{f['line']}] {f['kind']}: " + f"{f['message']}", + file=sys.stderr, + ) + print(f" snippet: 「{f['snippet']}」", file=sys.stderr) + if len(findings) > 20: + print(f" ... ({len(findings) - 20} more)", file=sys.stderr) + print( + " 💡 Fix: reduce table column count, shorten cell content, or " + "restructure as separate paragraph below the table.", + file=sys.stderr, + ) + print("", file=sys.stderr) + + +def _print_self_check_hint(pages: list[Path]) -> None: + """Print a structured self-check checklist after PDF generation.""" + if not pages: + print( + "ℹ️ Visual self-check skipped: pdftoppm not found " + "(install: brew install poppler).", + file=sys.stderr, + ) + return + + preview_dir = pages[0].parent + print("") + print(f"📋 Visual self-check required ({len(pages)} pages)") + print(f" Previews: {preview_dir}/page-NN.png") + print("") + print(" ⚠️ PDF generated cleanly does NOT mean rendering matches intent.") + print(" Read each page PNG and verify against your markdown source:") + print("") + print(" [ ] Paragraphs render as separate blocks (NOT collapsed into one)") + print(" ↳ Common issue: ≥2 consecutive `**xxx**：text` lines without") + print(" blank lines collapse into one paragraph (CommonMark soft") + print(" break = space). Fix in markdown: use `- ` real list, or") + print(" insert blank lines between.") + print(" [ ] Tables fit within page margins (no right-side text cut off)") + print(" [ ] No inappropriate Chinese line breaks (per 中文文案排版指北)") + print(" ↳ Symptoms: single CJK char alone on a line; broken bracket") + print(" pairs (line ends with `（` while content wraps to next line,") + print(" or line starts with `）`); short cells with mid-thought breaks.") + print(" ↳ Cause: cell width too narrow / table has too many columns.") + print(" ↳ Fix: reduce column count, shorten cell content, or move") + print(" long content into a separate paragraph below the table.") + print(" [ ] Lists keep nested indentation (sub-items visually nested)") + print(" [ ] Emoji + CJK glyphs render correctly (no boxes / placeholders)") + print(" [ ] Code blocks readable (monospace + CJK both visible)") + print(" [ ] No content overflow / unexpected page breaks mid-table") + print(" [ ] Last page ends naturally (no orphan title at top)") + print("") + + +def markdown_to_pdf( + md_file: str, + pdf_file: str | None = None, + theme: str = "default", + backend: str | None = None, + previews: bool = True, +) -> str: + """ + Convert markdown file to PDF. + + Args: + md_file: Path to input markdown file + pdf_file: Path to output PDF (optional, defaults to same name as input) + theme: Theme name (from themes/ directory) + backend: 'weasyprint', 'chrome', or None (auto-detect) + previews: If True (default), auto-generate per-page PNG previews next + to the PDF and print a visual self-check checklist. Disable + with --no-preview for batch / non-interactive runs. + + Returns: + Path to generated PDF file + """ + md_path = Path(md_file) + if pdf_file is None: + pdf_file = str(md_path.with_suffix(".pdf")) + + if backend is None: + backend = _detect_backend() + + css = _load_theme(theme) + html_content = _md_to_html(md_file) + full_html = _build_full_html(html_content, css, md_path.stem) + + if backend == "weasyprint": + _render_weasyprint(full_html, pdf_file, css) + elif backend == "chrome": + _render_chrome(full_html, pdf_file) + else: + print(f"Error: Unknown backend '{backend}'", file=sys.stderr) + sys.exit(1) + + size_kb = Path(pdf_file).stat().st_size / 1024 + print(f"Generated: {pdf_file} ({size_kb:.0f}KB, theme={theme}, backend={backend})") + + if previews: + # Auto-run typography lint to catch obvious mid-word breaks in tables. + # Findings go to stderr; do NOT block (warnings, not errors). + typography_findings = _lint_pdf_typography(pdf_file) + _print_typography_findings(typography_findings) + + pages = _generate_pdf_previews(pdf_file) + _print_self_check_hint(pages) + + return pdf_file + + +def main(): + available_themes = _list_themes() + + parser = argparse.ArgumentParser( + description="Markdown to PDF with Chinese font support and themes." + ) + parser.add_argument("input", help="Input markdown file") + parser.add_argument("output", nargs="?", help="Output PDF file (optional)") + parser.add_argument( + "--theme", + default="default", + choices=available_themes or ["default"], + help=f"CSS theme (available: {', '.join(available_themes) or 'default'})", + ) + parser.add_argument( + "--backend", + choices=["weasyprint", "chrome"], + default=None, + help="PDF rendering backend (default: auto-detect)", + ) + parser.add_argument( + "--list-themes", + action="store_true", + help="List available themes and exit", + ) + parser.add_argument( + "--no-preview", + action="store_true", + help="Skip per-page PNG preview generation and self-check hint", + ) + + args = parser.parse_args() + + if args.list_themes: + for t in available_themes: + marker = " (default)" if t == "default" else "" + css_file = THEMES_DIR / f"{t}.css" + first_line = "" + for line in css_file.read_text().splitlines(): + line = line.strip() + if line.startswith("*") and "—" in line: + first_line = line.lstrip("* ").strip() + break + print(f" {t}{marker}: {first_line}") + sys.exit(0) + + if not Path(args.input).exists(): + print(f"Error: File not found: {args.input}", file=sys.stderr) + sys.exit(1) + + markdown_to_pdf( + args.input, + args.output, + args.theme, + args.backend, + previews=not args.no_preview, + ) + + +if __name__ == "__main__": + main() diff --git a/suites/daymade-docs/pdf-creator/scripts/tests/test_cjk_code_blocks.py b/daymade-docs/pdf-creator/scripts/tests/test_cjk_code_blocks.py similarity index 100% rename from suites/daymade-docs/pdf-creator/scripts/tests/test_cjk_code_blocks.py rename to daymade-docs/pdf-creator/scripts/tests/test_cjk_code_blocks.py diff --git a/suites/daymade-docs/pdf-creator/scripts/tests/test_list_rendering.py b/daymade-docs/pdf-creator/scripts/tests/test_list_rendering.py similarity index 100% rename from suites/daymade-docs/pdf-creator/scripts/tests/test_list_rendering.py rename to daymade-docs/pdf-creator/scripts/tests/test_list_rendering.py diff --git a/suites/daymade-docs/pdf-creator/themes/default.css b/daymade-docs/pdf-creator/themes/default.css similarity index 100% rename from suites/daymade-docs/pdf-creator/themes/default.css rename to daymade-docs/pdf-creator/themes/default.css diff --git a/suites/daymade-docs/pdf-creator/themes/warm-terra.css b/daymade-docs/pdf-creator/themes/warm-terra.css similarity index 100% rename from suites/daymade-docs/pdf-creator/themes/warm-terra.css rename to daymade-docs/pdf-creator/themes/warm-terra.css diff --git a/suites/daymade-docs/ppt-creator/SKILL.md b/daymade-docs/ppt-creator/SKILL.md similarity index 100% rename from suites/daymade-docs/ppt-creator/SKILL.md rename to daymade-docs/ppt-creator/SKILL.md diff --git a/suites/daymade-docs/ppt-creator/references/CHECKLIST.md b/daymade-docs/ppt-creator/references/CHECKLIST.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/CHECKLIST.md rename to daymade-docs/ppt-creator/references/CHECKLIST.md diff --git a/suites/daymade-docs/ppt-creator/references/EXAMPLES.md b/daymade-docs/ppt-creator/references/EXAMPLES.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/EXAMPLES.md rename to daymade-docs/ppt-creator/references/EXAMPLES.md diff --git a/suites/daymade-docs/ppt-creator/references/INTAKE.md b/daymade-docs/ppt-creator/references/INTAKE.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/INTAKE.md rename to daymade-docs/ppt-creator/references/INTAKE.md diff --git a/suites/daymade-docs/ppt-creator/references/ORCHESTRATION_DATA_CHARTS.md b/daymade-docs/ppt-creator/references/ORCHESTRATION_DATA_CHARTS.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/ORCHESTRATION_DATA_CHARTS.md rename to daymade-docs/ppt-creator/references/ORCHESTRATION_DATA_CHARTS.md diff --git a/suites/daymade-docs/ppt-creator/references/ORCHESTRATION_OVERVIEW.md b/daymade-docs/ppt-creator/references/ORCHESTRATION_OVERVIEW.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/ORCHESTRATION_OVERVIEW.md rename to daymade-docs/ppt-creator/references/ORCHESTRATION_OVERVIEW.md diff --git a/suites/daymade-docs/ppt-creator/references/ORCHESTRATION_PPTX.md b/daymade-docs/ppt-creator/references/ORCHESTRATION_PPTX.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/ORCHESTRATION_PPTX.md rename to daymade-docs/ppt-creator/references/ORCHESTRATION_PPTX.md diff --git a/suites/daymade-docs/ppt-creator/references/RUBRIC.md b/daymade-docs/ppt-creator/references/RUBRIC.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/RUBRIC.md rename to daymade-docs/ppt-creator/references/RUBRIC.md diff --git a/suites/daymade-docs/ppt-creator/references/STYLE-GUIDE.md b/daymade-docs/ppt-creator/references/STYLE-GUIDE.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/STYLE-GUIDE.md rename to daymade-docs/ppt-creator/references/STYLE-GUIDE.md diff --git a/suites/daymade-docs/ppt-creator/references/TEMPLATES.md b/daymade-docs/ppt-creator/references/TEMPLATES.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/TEMPLATES.md rename to daymade-docs/ppt-creator/references/TEMPLATES.md diff --git a/suites/daymade-docs/ppt-creator/references/VIS-GUIDE.md b/daymade-docs/ppt-creator/references/VIS-GUIDE.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/VIS-GUIDE.md rename to daymade-docs/ppt-creator/references/VIS-GUIDE.md diff --git a/suites/daymade-docs/ppt-creator/references/WORKFLOW.md b/daymade-docs/ppt-creator/references/WORKFLOW.md similarity index 100% rename from suites/daymade-docs/ppt-creator/references/WORKFLOW.md rename to daymade-docs/ppt-creator/references/WORKFLOW.md diff --git a/suites/daymade-docs/ppt-creator/scripts/chartkit.py b/daymade-docs/ppt-creator/scripts/chartkit.py similarity index 100% rename from suites/daymade-docs/ppt-creator/scripts/chartkit.py rename to daymade-docs/ppt-creator/scripts/chartkit.py diff --git a/references/new-skill-guide.md b/references/new-skill-guide.md index 5f7003eb..c4f89050 100644 --- a/references/new-skill-guide.md +++ b/references/new-skill-guide.md @@ -220,7 +220,7 @@ Before committing, verify: 3. **Inconsistent version numbers** - CHANGELOG, README badges (both EN and ZH), CLAUDE.md, and marketplace.json must all match 4. **Inconsistent skill counts** - README description (both EN and ZH), badges, CLAUDE.md must all have same count 5. **Missing skill number in README** - Skills must be numbered sequentially in both EN and ZH versions -6. **Relying on JSON syntax check alone** - `python -m json.tool` only catches malformed JSON. It will NOT catch missing plugin entries, broken source+skills resolution, or orphan SKILL.md files on disk. Use `bash suites/daymade-claude-code/marketplace-dev/scripts/check_marketplace.sh` for the full 4-check validation. +6. **Relying on JSON syntax check alone** - `python -m json.tool` only catches malformed JSON. It will NOT catch missing plugin entries, broken source+skills resolution, or orphan SKILL.md files on disk. Use `bash daymade-claude-code/marketplace-dev/scripts/check_marketplace.sh` for the full 4-check validation. 7. **Leaving orphan SKILL.md directories** - A tracked skill directory with no plugin entry in marketplace.json is invisible to `claude plugin install`. The reverse-sync check in `check_marketplace.sh` emits a WARN for each orphan. Treat every WARN as a real signal: register it or delete it. 8. **Using `git add -A` or `git add .`** - When multiple sessions/agents edit the repo in parallel, a blanket stage can piggyback another agent's unstaged changes into your commit. Always stage files by name. 9. **Forgetting to push** - Local changes are invisible until pushed to GitHub @@ -236,7 +236,7 @@ uv run python -m scripts.security_scan ../skill-name --verbose uv run --with PyYAML python -m scripts.package_skill ../skill-name # 3. Full marketplace validation — the single source of truth for "is this shippable?" -cd .. && bash suites/daymade-claude-code/marketplace-dev/scripts/check_marketplace.sh +cd .. && bash daymade-claude-code/marketplace-dev/scripts/check_marketplace.sh # Runs 4 checks in sequence: # [1/4] JSON syntax of .claude-plugin/marketplace.json # [2/4] claude plugin validate . (schema-level, skipped if CLI missing) diff --git a/suites/daymade-docs/pdf-creator/scripts/md_to_pdf.py b/suites/daymade-docs/pdf-creator/scripts/md_to_pdf.py deleted file mode 100644 index 863e02cc..00000000 --- a/suites/daymade-docs/pdf-creator/scripts/md_to_pdf.py +++ /dev/null @@ -1,350 +0,0 @@ -#!/usr/bin/env python3 -""" -Markdown to PDF converter with Chinese font support and theme system. - -Converts markdown files to PDF using: - - pandoc (markdown → HTML) - - weasyprint or headless Chrome (HTML → PDF), auto-detected - -Usage: - python md_to_pdf.py input.md output.pdf - python md_to_pdf.py input.md --theme warm-terra - python md_to_pdf.py input.md --theme default --backend chrome - python md_to_pdf.py input.md # outputs input.pdf, default theme, auto backend - -Themes: - Stored in ../themes/*.css. Built-in themes: - - default: Songti SC + black/grey, formal documents - - warm-terra: PingFang SC + terra cotta, training/workshop materials - -Requirements: - pandoc (system install, e.g. brew install pandoc) - weasyprint (pip install weasyprint) OR Google Chrome (for --backend chrome) -""" - -from __future__ import annotations - -import argparse -import os -import platform -import re -import shutil -import subprocess -import sys -import tempfile -from pathlib import Path - -SCRIPT_DIR = Path(__file__).resolve().parent -THEMES_DIR = SCRIPT_DIR.parent / "themes" - -# macOS ARM: auto-configure library path for weasyprint -if platform.system() == "Darwin": - _homebrew_lib = "/opt/homebrew/lib" - if Path(_homebrew_lib).is_dir(): - _cur = os.environ.get("DYLD_LIBRARY_PATH", "") - if _homebrew_lib not in _cur: - os.environ["DYLD_LIBRARY_PATH"] = ( - f"{_homebrew_lib}:{_cur}" if _cur else _homebrew_lib - ) - - -def _find_chrome() -> str | None: - """Find Chrome/Chromium binary path.""" - candidates = [ - "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", - "/Applications/Chromium.app/Contents/MacOS/Chromium", - shutil.which("google-chrome"), - shutil.which("chromium"), - shutil.which("chrome"), - ] - for c in candidates: - if c and Path(c).exists(): - return str(c) - return None - - -def _has_weasyprint() -> bool: - """Check if weasyprint is importable.""" - try: - import weasyprint # noqa: F401 - - return True - except ImportError: - return False - - -def _detect_backend() -> str: - """Auto-detect best available backend: weasyprint > chrome.""" - if _has_weasyprint(): - return "weasyprint" - if _find_chrome(): - return "chrome" - print( - "Error: No PDF backend found. Install weasyprint (pip install weasyprint) " - "or Google Chrome.", - file=sys.stderr, - ) - sys.exit(1) - - -def _load_theme(theme_name: str) -> str: - """Load CSS from themes directory.""" - theme_file = THEMES_DIR / f"{theme_name}.css" - if not theme_file.exists(): - available = [f.stem for f in THEMES_DIR.glob("*.css")] - print( - f"Error: Theme '{theme_name}' not found. Available: {available}", - file=sys.stderr, - ) - sys.exit(1) - return theme_file.read_text(encoding="utf-8") - - -def _list_themes() -> list[str]: - """List available theme names.""" - if not THEMES_DIR.exists(): - return [] - return sorted(f.stem for f in THEMES_DIR.glob("*.css")) - - -def _ensure_list_spacing(text: str) -> str: - """Ensure blank lines before list items for proper markdown parsing. - - Both Python markdown library and pandoc require a blank line before a list - when it follows a paragraph. Without it, list items render as plain text. - """ - lines = text.split("\n") - result = [] - list_re = re.compile(r"^(\s*)([-*+]|\d+\.)\s") - for i, line in enumerate(lines): - if i > 0 and list_re.match(line): - prev = lines[i - 1] - if prev.strip() and not list_re.match(prev): - result.append("") - result.append(line) - return "\n".join(result) - - -_CJK_RANGE = re.compile( - # Chinese: CJK Unified Ideographs + Extension A + Compatibility + Extensions B-F - r"[\u4e00-\u9fff\u3400-\u4dbf\uf900-\ufaff" - r"\U00020000-\U0002a6df\U0002a700-\U0002ebef" - # CJK Symbols and Punctuation + Halfwidth/Fullwidth Forms - r"\u3000-\u303f\uff00-\uffef" - # Japanese: Hiragana + Katakana + Katakana Phonetic Extensions - r"\u3040-\u309f\u30a0-\u30ff\u31f0-\u31ff" - # Korean: Hangul Syllables + Hangul Jamo + Hangul Compatibility Jamo - r"\uac00-\ud7af\u1100-\u11ff\u3130-\u318f]" -) - -# Match 
…
allowing attributes on both tags. -# Also handles pandoc's 

-# syntax highlighting wrapper, by matching the inner 
 structure.
-_PRE_CODE_RE = re.compile(
-    r"]*>\s*]*>(.+?)\s*
",
-    flags=re.DOTALL,
-)
-
-
-def _fix_cjk_code_blocks(html: str) -> str:
-    """Replace 
 blocks containing CJK with styled divs.
-
-    weasyprint renders 
 blocks using monospace fonts that lack CJK glyphs,
-    causing garbled output. This converts CJK-heavy code blocks to styled divs
-    that use the document's CJK font stack instead.
-
-    Pure-ASCII code blocks (including pandoc-highlighted ones with language
-    identifiers) are left untouched so syntax highlighting and monospace
-    rendering are preserved.
-    """
-
-    def _replace_if_cjk(match: re.Match) -> str:
-        content = match.group(1)
-        if _CJK_RANGE.search(content):
-            # Strip pandoc's  syntax-highlighting wrappers so the
-            # content renders as plain text in the inherited body font.
-            cleaned = re.sub(r"]*>", "", content)
-            cleaned = cleaned.replace("", "")
-            return f'
{cleaned}
'
-        return match.group(0)
-
-    return _PRE_CODE_RE.sub(_replace_if_cjk, html)
-
-
-def _md_to_html(md_file: str) -> str:
-    """Convert markdown to HTML using pandoc with list spacing preprocessing."""
-    if not shutil.which("pandoc"):
-        print(
-            "Error: pandoc not found. Install with: brew install pandoc",
-            file=sys.stderr,
-        )
-        sys.exit(1)
-
-    md_content = Path(md_file).read_text(encoding="utf-8")
-    md_content = _ensure_list_spacing(md_content)
-
-    result = subprocess.run(
-        ["pandoc", "-f", "markdown", "-t", "html"],
-        input=md_content,
-        capture_output=True,
-        text=True,
-    )
-    if result.returncode != 0:
-        print(f"Error: pandoc failed: {result.stderr}", file=sys.stderr)
-        sys.exit(1)
-
-    html = result.stdout
-    html = _fix_cjk_code_blocks(html)
-    return html
-
-
-def _build_full_html(html_content: str, css: str, title: str) -> str:
-    """Wrap HTML content in a full document with CSS."""
-    return f"""
-
-
-    
-    {title}
-    
-
-
-{html_content}
-
-"""
-
-
-def _render_weasyprint(full_html: str, pdf_file: str, css: str) -> None:
-    """Render PDF using weasyprint."""
-    from weasyprint import CSS, HTML
-
-    HTML(string=full_html).write_pdf(pdf_file, stylesheets=[CSS(string=css)])
-
-
-def _render_chrome(full_html: str, pdf_file: str) -> None:
-    """Render PDF using headless Chrome."""
-    chrome = _find_chrome()
-    if not chrome:
-        print("Error: Chrome not found.", file=sys.stderr)
-        sys.exit(1)
-
-    with tempfile.NamedTemporaryFile(
-        suffix=".html", mode="w", encoding="utf-8", delete=False
-    ) as f:
-        f.write(full_html)
-        html_path = f.name
-
-    try:
-        result = subprocess.run(
-            [
-                chrome,
-                "--headless",
-                "--disable-gpu",
-                "--no-pdf-header-footer",
-                f"--print-to-pdf={pdf_file}",
-                html_path,
-            ],
-            capture_output=True,
-            text=True,
-        )
-        if not Path(pdf_file).exists():
-            print(
-                f"Error: Chrome failed to generate PDF. stderr: {result.stderr}",
-                file=sys.stderr,
-            )
-            sys.exit(1)
-    finally:
-        Path(html_path).unlink(missing_ok=True)
-
-
-def markdown_to_pdf(
-    md_file: str,
-    pdf_file: str | None = None,
-    theme: str = "default",
-    backend: str | None = None,
-) -> str:
-    """
-    Convert markdown file to PDF.
-
-    Args:
-        md_file: Path to input markdown file
-        pdf_file: Path to output PDF (optional, defaults to same name as input)
-        theme: Theme name (from themes/ directory)
-        backend: 'weasyprint', 'chrome', or None (auto-detect)
-
-    Returns:
-        Path to generated PDF file
-    """
-    md_path = Path(md_file)
-    if pdf_file is None:
-        pdf_file = str(md_path.with_suffix(".pdf"))
-
-    if backend is None:
-        backend = _detect_backend()
-
-    css = _load_theme(theme)
-    html_content = _md_to_html(md_file)
-    full_html = _build_full_html(html_content, css, md_path.stem)
-
-    if backend == "weasyprint":
-        _render_weasyprint(full_html, pdf_file, css)
-    elif backend == "chrome":
-        _render_chrome(full_html, pdf_file)
-    else:
-        print(f"Error: Unknown backend '{backend}'", file=sys.stderr)
-        sys.exit(1)
-
-    size_kb = Path(pdf_file).stat().st_size / 1024
-    print(f"Generated: {pdf_file} ({size_kb:.0f}KB, theme={theme}, backend={backend})")
-    return pdf_file
-
-
-def main():
-    available_themes = _list_themes()
-
-    parser = argparse.ArgumentParser(
-        description="Markdown to PDF with Chinese font support and themes."
-    )
-    parser.add_argument("input", help="Input markdown file")
-    parser.add_argument("output", nargs="?", help="Output PDF file (optional)")
-    parser.add_argument(
-        "--theme",
-        default="default",
-        choices=available_themes or ["default"],
-        help=f"CSS theme (available: {', '.join(available_themes) or 'default'})",
-    )
-    parser.add_argument(
-        "--backend",
-        choices=["weasyprint", "chrome"],
-        default=None,
-        help="PDF rendering backend (default: auto-detect)",
-    )
-    parser.add_argument(
-        "--list-themes",
-        action="store_true",
-        help="List available themes and exit",
-    )
-
-    args = parser.parse_args()
-
-    if args.list_themes:
-        for t in available_themes:
-            marker = " (default)" if t == "default" else ""
-            css_file = THEMES_DIR / f"{t}.css"
-            first_line = ""
-            for line in css_file.read_text().splitlines():
-                line = line.strip()
-                if line.startswith("*") and "—" in line:
-                    first_line = line.lstrip("* ").strip()
-                    break
-            print(f"  {t}{marker}: {first_line}")
-        sys.exit(0)
-
-    if not Path(args.input).exists():
-        print(f"Error: File not found: {args.input}", file=sys.stderr)
-        sys.exit(1)
-
-    markdown_to_pdf(args.input, args.output, args.theme, args.backend)
-
-
-if __name__ == "__main__":
-    main()

From dcfb38b06b59f04b4a80a2d7d1c2ba8ca08963e0 Mon Sep 17 00:00:00 2001
From: daymade 
Date: Sun, 26 Apr 2026 21:40:52 +0800
Subject: [PATCH 115/137] chore(gitignore): ignore
 debugging-network-issues-workspace/

Skill iteration workspace, follows the same pattern as the pre-existing
douban-skill-workspace/ entry.

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 019f1d7f..3cb0857a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -93,6 +93,7 @@ recovered_deep_research/
 
 # Eval workspaces (contain test data with personal info)
 douban-skill-workspace/
+debugging-network-issues-workspace/
 .gstack/
 
 # Claude Code local settings

From ed6d28ba09d99b701fa559773104c8a2082022c6 Mon Sep 17 00:00:00 2001
From: daymade 
Date: Sun, 26 Apr 2026 21:52:54 +0800
Subject: [PATCH 116/137] feat(marketplace): register debugging-network-issues
 + stepfun-tts and bump to v1.51.0
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Land the two skill directories that were missing from git history. The
plugin entries already existed in marketplace.json (carried into commit
1f72904 as uncommitted draft modifications during the suite-flatten
rewrite), but the skill directories themselves and all documentation
surfaces were never synchronized — install would have failed to resolve
./debugging-network-issues and ./stepfun-tts. This commit completes that
half-done registration.

Skills:
- debugging-network-issues v1.0.0: Evidence-driven, falsification-first
  methodology for network/streaming/protocol-layer bugs (HTTP/2 RST_STREAM,
  SSE stalls, fixed-time drops, CDN/proxy/CGNAT idle timeouts). Bundles
  layered-isolation probe scripts and a real SSE 130s case study.
- stepfun-tts v1.0.0: StepFun StepAudio 2.5 family — stepaudio-2.5-tts
  (Contextual TTS via instruction + inline parentheses) and
  stepaudio-2.5-asr (SSE endpoint, 32K context, ~100x RTF, 30-min
  single-call). Captures the three step-tts-2 → stepaudio-2.5 breaking
  changes with migration playbook.

Manifest sync:
- marketplace.json: metadata.version 1.50.0 → 1.51.0
- CHANGELOG.md: v1.51.0 entry with the half-done-registration note
- README.md / README.zh-CN.md: badges, descriptions, skill sections #49+#50,
  Use Cases, Documentation Quick Links, Requirements
- CLAUDE.md: skill count 49 → 51, plugin count 53 → 55, Available Skills

check_marketplace.sh: 4/4 PASS

Co-Authored-By: Claude Opus 4.7 (1M context) 
---
 .claude-plugin/marketplace.json               |   2 +-
 CHANGELOG.md                                  |  16 ++
 CLAUDE.md                                     |   6 +-
 README.md                                     |  58 ++++-
 README.zh-CN.md                               |  58 ++++-
 .../.security-scan-passed                     |   4 +
 debugging-network-issues/SKILL.md             | 207 +++++++++++++++++
 debugging-network-issues/evals/evals.json     |  62 +++++
 .../references/case-sse-rst-130s.md           | 181 +++++++++++++++
 .../references/cognitive-traps.md             | 139 +++++++++++
 .../references/counter-review-pattern.md      | 160 +++++++++++++
 .../references/instrumentation-patterns.md    | 186 +++++++++++++++
 .../layered-isolation-experiment.md           | 141 ++++++++++++
 .../references/packet-capture-recipes.md      | 148 ++++++++++++
 .../scripts/layered-isolation-probe.sh        | 115 ++++++++++
 .../scripts/mock-idle-upstream.py             | 140 +++++++++++
 stepfun-tts/.security-scan-passed             |   4 +
 stepfun-tts/SKILL.md                          | 102 ++++++++
 stepfun-tts/references/api_reference.md       | 178 ++++++++++++++
 stepfun-tts/references/known_issues.md        | 131 +++++++++++
 stepfun-tts/references/migration_from_v2.md   | 206 +++++++++++++++++
 stepfun-tts/scripts/ab_compare.sh             | 132 +++++++++++
 stepfun-tts/scripts/asr_transcribe.py         | 205 +++++++++++++++++
 stepfun-tts/scripts/tts_generate.py           | 217 ++++++++++++++++++
 24 files changed, 2789 insertions(+), 9 deletions(-)
 create mode 100644 debugging-network-issues/.security-scan-passed
 create mode 100644 debugging-network-issues/SKILL.md
 create mode 100644 debugging-network-issues/evals/evals.json
 create mode 100644 debugging-network-issues/references/case-sse-rst-130s.md
 create mode 100644 debugging-network-issues/references/cognitive-traps.md
 create mode 100644 debugging-network-issues/references/counter-review-pattern.md
 create mode 100644 debugging-network-issues/references/instrumentation-patterns.md
 create mode 100644 debugging-network-issues/references/layered-isolation-experiment.md
 create mode 100644 debugging-network-issues/references/packet-capture-recipes.md
 create mode 100755 debugging-network-issues/scripts/layered-isolation-probe.sh
 create mode 100755 debugging-network-issues/scripts/mock-idle-upstream.py
 create mode 100644 stepfun-tts/.security-scan-passed
 create mode 100644 stepfun-tts/SKILL.md
 create mode 100644 stepfun-tts/references/api_reference.md
 create mode 100644 stepfun-tts/references/known_issues.md
 create mode 100644 stepfun-tts/references/migration_from_v2.md
 create mode 100755 stepfun-tts/scripts/ab_compare.sh
 create mode 100755 stepfun-tts/scripts/asr_transcribe.py
 create mode 100755 stepfun-tts/scripts/tts_generate.py

diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json
index 631429a2..5c2c0bb4 100644
--- a/.claude-plugin/marketplace.json
+++ b/.claude-plugin/marketplace.json
@@ -6,7 +6,7 @@
   },
   "metadata": {
     "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, macOS programmatic window screenshot capture workflows, verified Scrapling CLI installation and web extraction workflows, plugin marketplace development for converting skills repos into official Claude Code marketplaces, Tencent IMA knowledge base companion with zero-config installation across Claude Code/Codex/OpenClaw, upstream bug detection and runtime repair, personalized fan-out search with priority-based knowledge base boosting, and suite plugins that expose related skills under shared namespaces for combined installation workflows — including a dedicated Claude Code operations suite (daymade-claude-code) bundling session recovery, CLAUDE.md optimization, troubleshooting, statusline configuration, export repair, and marketplace development skills under one namespace",
-    "version": "1.50.0"
+    "version": "1.51.0"
   },
   "plugins": [
     {
diff --git a/CHANGELOG.md b/CHANGELOG.md
index df2f6107..ac9556a2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.51.0] - 2026-04-26
+
+### Added
+- **debugging-network-issues** v1.0.0: Evidence-driven, falsification-first methodology for network, streaming, and protocol-layer bugs where the obvious cause is probably wrong. Built from a real 5-hour production case (SSE RST_STREAM at exactly 130s, traced to a CGNAT idle timeout). Provides layered-isolation experiments (run the same logical request through 3+ paths differing by one hop), env-gated runtime instrumentation patterns, and a counter-review four-question filter to challenge single-cause assumptions before shipping a fix. Bundles probe scripts (`layered-isolation-probe.sh`, `mock-idle-upstream.py`) and reference docs covering counter-review, packet-capture recipes, instrumentation patterns, and cognitive traps. Triggers on `ECONNRESET`, HTTP/2 `RST_STREAM`, `INTERNAL_ERROR`, fixed-time SSE drops, CDN/proxy/CGNAT idle timeouts, and "works sometimes / fails after N seconds" patterns.
+- **stepfun-tts** v1.0.0: Generate Chinese/Japanese speech with `stepaudio-2.5-tts` and transcribe long audio with `stepaudio-2.5-asr` (SSE endpoint, 32K context, ~100x RTF, up to 30-minute single call). Encapsulates the three non-obvious StepAudio 2.5 pitfalls that cost hours: `voice_label` removal (replaced by `instruction` + inline `()` prosody), `/v1/audio/asr/sse` endpoint mismatch (returns misleading `model not supported` error otherwise), and stricter censorship rules. Bundled scripts: `tts_generate.py` (with `--batch `), `asr_transcribe.py`, `ab_compare.sh`. API key resolution: `$STEPFUN_API_KEY` → `${CLAUDE_PLUGIN_DATA}/config.json` fallback. Reference docs cover migration from `step-tts-2`, the censorship rewrite list, and the verified-on-2026-04-23 known-issues registry.
+
+### Changed
+- Marketplace skill count: 49 → 51
+- Marketplace plugin entry count: 53 → 55
+- Marketplace version: 1.50.0 → 1.51.0
+- README.md, README.zh-CN.md: badges, descriptions, skill sections (#49 + #50), Use Cases entries, Documentation Quick Links, Requirements
+- CLAUDE.md: overview count, marketplace plugin count, Available Skills list
+
+### Note
+Plugin entries for these two skills were inadvertently committed in v1.50.0's path-rewrite operation (the entries existed as uncommitted draft modifications in `marketplace.json` and were carried along when that file was rewritten). v1.51.0 completes the registration that v1.50.0 left half-done by landing the skill directories themselves and synchronizing all documentation surfaces.
+
 ## [1.50.0] - 2026-04-26
 
 ### Changed
diff --git a/CLAUDE.md b/CLAUDE.md
index 2d072fc5..e15c8050 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Repository Overview
 
-This is a Claude Code skills marketplace containing 49 production-ready skills organized in a plugin marketplace structure. Most plugins expose one skill for narrow installs; suite plugins expose related skills under shared namespaces for combined installation workflows.
+This is a Claude Code skills marketplace containing 51 production-ready skills organized in a plugin marketplace structure. Most plugins expose one skill for narrow installs; suite plugins expose related skills under shared namespaces for combined installation workflows.
 
 **Essential Skill**: `skill-creator` is the most important skill in this marketplace - it's a meta-skill that enables users to create their own skills. Always recommend it first for users interested in extending Claude Code.
 
@@ -152,7 +152,7 @@ If it fires, fix the issue — do NOT use `--no-verify` to bypass.
 ## Marketplace Configuration
 
 The marketplace is configured in `.claude-plugin/marketplace.json`:
-- Contains 53 plugin entries: most map to one skill, while suite plugins (`daymade-docs`, `daymade-claude-code`) map to multiple related skills
+- Contains 55 plugin entries: most map to one skill, while suite plugins (`daymade-docs`, `daymade-claude-code`) map to multiple related skills
 - Each plugin has: name, description, version, category, keywords, skills array
 - Marketplace metadata: name, owner, version, homepage
 - Suite plugins use `/` (a top-level directory at the repo root) as the canonical source for their member skills so suite caches contain only those skills. Single-skill plugin entries for suite members should point to the same canonical subdirectories.
@@ -245,6 +245,8 @@ This applies when you change ANY file under a skill directory:
 47. **wechat-article-scraper** - World-class WeChat article extraction with 6-level strategy routing, OG metadata fallback, image-paragraph association, and Sogou search discovery; supports Markdown/JSON/HTML/PDF export
 48. **terraform-skill** - Operational traps for Terraform provisioners, multi-environment isolation, and zero-to-deployment reliability; covers provisioner timing races, SSH connection conflicts, DNS record duplication, volume permissions, database bootstrap gaps, Cloudflare credential errors, and init-data-only-on-first-boot pitfalls
 49. **slides-creator** - Narrative-first slide deck creation guiding users through structured narrative design (ABCDEFG model), then delegating visual generation to baoyu-slide-deck. Triggers on create slides, make a presentation, generate deck, slide deck, PPT, or when user needs to turn content into visual slides
+50. **debugging-network-issues** - Evidence-driven, falsification-first methodology for network/streaming/protocol-layer bugs (HTTP/2 RST_STREAM, SSE stalls, fixed-time drops, CDN/proxy/CGNAT idle timeouts). Layered isolation experiments + counter-review filter, with bundled probe scripts and a real SSE 130s case study
+51. **stepfun-tts** - StepFun StepAudio 2.5 family: stepaudio-2.5-tts (Contextual TTS via instruction + inline parentheses) and stepaudio-2.5-asr (SSE endpoint, 32K context, ~100x RTF, 30-min single-call). Captures the three breaking changes from step-tts-2 (voice_label removal, /v1/audio/asr/sse endpoint, stricter censorship) with migration playbook
 
 **Recommendation**: Always suggest `skill-creator` first for users interested in creating skills or extending Claude Code.
 
diff --git a/README.md b/README.md
index 0eec3d1b..f4104e96 100644
--- a/README.md
+++ b/README.md
@@ -6,15 +6,15 @@
 [![简体中文](https://img.shields.io/badge/语言-简体中文-red)](./README.zh-CN.md)
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![Skills](https://img.shields.io/badge/skills-49-blue.svg)](https://github.com/daymade/claude-code-skills)
-[![Version](https://img.shields.io/badge/version-1.49.0-green.svg)](https://github.com/daymade/claude-code-skills)
+[![Skills](https://img.shields.io/badge/skills-51-blue.svg)](https://github.com/daymade/claude-code-skills)
+[![Version](https://img.shields.io/badge/version-1.51.0-green.svg)](https://github.com/daymade/claude-code-skills)
 [![Claude Code](https://img.shields.io/badge/Claude%20Code-2.0.13+-purple.svg)](https://claude.com/code)
 [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md)
 [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/daymade/claude-code-skills/graphs/commit-activity)
-Professional Claude Code skills marketplace featuring 49 production-ready skills for enhanced development workflows. +Professional Claude Code skills marketplace featuring 51 production-ready skills for enhanced development workflows. ## 📑 Table of Contents @@ -2079,6 +2079,49 @@ Guides users through structured narrative design (ABCDEFG model), then delegates --- +### 49. **debugging-network-issues** - Evidence-Driven Network Investigation + +Falsification-first methodology for network, streaming, and protocol-layer bugs where the obvious cause is probably wrong. Built from a real 5-hour SSE incident where assumption-stacking wasted hours that a 10-minute layered experiment would have resolved. + +**When to use:** +- Connection resets (`ECONNRESET`, HTTP/2 `RST_STREAM`, `INTERNAL_ERROR`) +- SSE / long-polling stalls or fixed-time drops (60s, 100s, 130s) +- CDN / proxy / CGNAT idle-timeout incidents +- Any "works sometimes / fails after N seconds" pattern +- Multi-hop systems (client → CDN → LB → reverse proxy → app → upstream) where a symptom could plausibly come from several layers + +**Key features:** +- Layered isolation experiments: run the same logical request through three or more paths differing by exactly one hop +- Env-gated runtime instrumentation patterns (no production-code mutation) +- Counter-review four-question filter to challenge single-cause assumptions +- Bundled probe scripts (`layered-isolation-probe.sh`, `mock-idle-upstream.py`) +- Real case study: SSE RST_STREAM at 130s caused by CGNAT idle timeout + +**Requirements**: None (methodology + portable shell/Python probes). + +--- + +### 50. **stepfun-tts** - StepFun StepAudio 2.5 TTS + ASR + +Generate Chinese / Japanese speech and transcribe long audio with StepFun's StepAudio 2.5 family. Captures the three non-obvious pitfalls that cost hours otherwise: `voice_label` removal, the `/v1/audio/asr/sse` endpoint, and stricter censorship. + +**When to use:** +- Chinese / Japanese TTS with emotional and prosody control +- Long audio transcription (up to ~30 minutes single-call, 32K context, ~100x RTF) +- Migration from `step-tts-2` to `stepaudio-2.5-tts` (`voice_label` → `instruction` breaking change) +- Hitting StepFun censorship blocks or endpoint mismatches + +**Key features:** +- `stepaudio-2.5-tts` with `instruction` (≤200 chars natural-language mood) + inline `()` prosody +- `stepaudio-2.5-asr` SSE streaming with base64 audio (avoids the misleading "model not supported" error) +- Bundled `tts_generate.py` (with `--batch `), `asr_transcribe.py`, `ab_compare.sh` +- API key resolution: `$STEPFUN_API_KEY` → `${CLAUDE_PLUGIN_DATA}/config.json` fallback +- Censorship rewrite playbook in `references/migration_from_v2.md` + +**Requirements**: StepFun API key (https://platform.stepfun.com/). + +--- + ## 🎬 Interactive Demo Gallery Want to see all demos in one place with click-to-enlarge functionality? Check out our [interactive demo gallery](./demos/index.html) or browse the [demos directory](./demos/). @@ -2199,6 +2242,12 @@ Use **douban-skill** to back up your Douban 书影音 (book/movie/music/game) hi ### For Terraform & IaC Troubleshooting Use **terraform-skill** when your `terraform apply` fails at a provisioner step, when fresh instances hit "docker: not found", or when multi-environment setups accidentally share snapshots. Every pattern in the skill is an *exact error → root cause → copy-paste fix* triple drawn from real incidents. Perfect for anyone who has lost a weekend to timing races in cloud-init, rsync connection drops in local-exec, or hardcoded domains in Caddyfiles. +### For Network, Streaming & Protocol-Layer Debugging +Use **debugging-network-issues** when symptoms do not match the obvious cause: HTTP/2 `RST_STREAM`, SSE stalls at exactly 60s/100s/130s, "works sometimes but not always" failures, or anything that looks like an idle-timeout incident through CDN / proxy / CGNAT chains. The skill replaces assumption-stacking with **layered isolation experiments** — running the same logical request through three or more paths that differ by one hop — plus a counter-review pattern for shipping fixes only after the hypothesis has been falsified, not just confirmed. + +### For Chinese TTS & Long-Audio Transcription (StepFun) +Use **stepfun-tts** for Chinese / Japanese voice synthesis with emotional control via `instruction` + inline `()` prosody, or for transcribing up to 30-minute audio in a single call (32K context, ~100x RTF). Captures the three breaking changes that ambush new StepAudio 2.5 users: `voice_label` removal, the `/v1/audio/asr/sse` endpoint mismatch, and stricter censorship rules. Combine with **transcript-fixer** for ASR post-processing or with **meeting-minutes-taker** to turn long recordings into structured minutes. + ## 📚 Documentation Each skill includes: @@ -2254,6 +2303,8 @@ Each skill includes: - **douban-skill**: See `douban-skill/SKILL.md` for the export workflow and `douban-skill/references/troubleshooting.md` for the complete log of 7 tested scraping approaches and why each failed - **terraform-skill**: See `terraform-skill/SKILL.md` for the full catalogue of operational traps organised by exact error → root cause → copy-paste fix - **slides-creator**: See `slides-creator/SKILL.md` for the narrative-first workflow, `slides-creator/references/narrative-design-guide.md` for the ABCDEFG model, and `slides-creator/references/content-creation-first-law.md` for the universal content creation principle +- **debugging-network-issues**: See `debugging-network-issues/SKILL.md` for the falsification-first workflow, `debugging-network-issues/references/layered-isolation-experiment.md` for the multi-hop isolation pattern, and `debugging-network-issues/references/case-sse-rst-130s.md` for the real production case study +- **stepfun-tts**: See `stepfun-tts/SKILL.md` for the TTS+ASR decision tree and `stepfun-tts/references/migration_from_v2.md` for the `voice_label` → `instruction` migration playbook plus the censorship rewrite list ## 🛠️ Requirements @@ -2282,6 +2333,7 @@ Each skill includes: - **Python 3.8+** (for continue-claude-work): bundled script for session extraction (no external dependencies) - **uv + Scrapling CLI** (for scrapling-skill): `uv tool install 'scrapling[shell]'` and `scrapling install` for browser-backed fetches - **Node.js 18+ + curl + unzip** (for ima-copilot): `npx skills` is fetched on demand from the npm registry; IMA OpenAPI credentials from [https://ima.qq.com/agent-interface](https://ima.qq.com/agent-interface) +- **StepFun API key** (for stepfun-tts): Available at [https://platform.stepfun.com/](https://platform.stepfun.com/) → API Keys ## ❓ FAQ diff --git a/README.zh-CN.md b/README.zh-CN.md index 8628f9ae..9494a450 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -6,15 +6,15 @@ [![简体中文](https://img.shields.io/badge/语言-简体中文-red)](./README.zh-CN.md) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Skills](https://img.shields.io/badge/skills-49-blue.svg)](https://github.com/daymade/claude-code-skills) -[![Version](https://img.shields.io/badge/version-1.49.0-green.svg)](https://github.com/daymade/claude-code-skills) +[![Skills](https://img.shields.io/badge/skills-51-blue.svg)](https://github.com/daymade/claude-code-skills) +[![Version](https://img.shields.io/badge/version-1.51.0-green.svg)](https://github.com/daymade/claude-code-skills) [![Claude Code](https://img.shields.io/badge/Claude%20Code-2.0.13+-purple.svg)](https://claude.com/code) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/daymade/claude-code-skills/graphs/commit-activity) -专业的 Claude Code 技能市场，提供 49 个生产就绪的技能，用于增强开发工作流。 +专业的 Claude Code 技能市场，提供 51 个生产就绪的技能，用于增强开发工作流。 ## 📑 目录 @@ -2120,6 +2120,49 @@ uv run douban-skill/scripts/douban-rss-sync.py --- +### 49. **debugging-network-issues** - 证据驱动的网络问题排查 + +针对网络、流式、协议层 bug 的"先证伪、再下结论"方法论。源自一次真实的 5 小时 SSE 生产事故——堆假设浪费的几个小时，10 分钟分层实验就能解决。 + +**使用场景：** +- 连接重置（`ECONNRESET`、HTTP/2 `RST_STREAM`、`INTERNAL_ERROR`） +- SSE / 长轮询挂起或定时断开（60s、100s、130s） +- CDN / 代理 / CGNAT 空闲超时事件 +- "时灵时不灵 / N 秒后必断"模式 +- 多跳系统（client → CDN → LB → reverse proxy → app → upstream）症状可能来自多层 + +**主要功能：** +- 分层隔离实验：让同一逻辑请求走三条以上、每条仅差一跳的路径 +- 环境变量门控的运行时埋点（不污染生产代码） +- 反审查四问过滤器，挑战单因果假设 +- 内置探针脚本（`layered-isolation-probe.sh`、`mock-idle-upstream.py`） +- 真实案例：CGNAT 130s 空闲超时导致的 SSE RST_STREAM + +**要求**：无（方法论 + 可移植的 shell/Python 探针）。 + +--- + +### 50. **stepfun-tts** - 阶跃 StepAudio 2.5 TTS + ASR + +用 StepFun 阶跃的 StepAudio 2.5 系列做中文 / 日语语音合成与长音频转写。封装了三个会浪费时间的非显然坑：`voice_label` 移除、`/v1/audio/asr/sse` 端点、更严的审查。 + +**使用场景：** +- 带情感和韵律控制的中 / 日语 TTS +- 长音频转写（单次最长 ~30 分钟、32K context、~100x RTF） +- 从 `step-tts-2` 迁移到 `stepaudio-2.5-tts`（`voice_label` → `instruction` 是破坏性变更） +- 遇到 StepFun 审查拦截或端点错误 + +**主要功能：** +- `stepaudio-2.5-tts`：用 `instruction`（≤200 字自然语言情绪）+ 文中 `()` 行内韵律 +- `stepaudio-2.5-asr`：SSE 流式 + base64 音频（避开误导性的 "model not supported" 错误） +- 内置 `tts_generate.py`（含 `--batch `）、`asr_transcribe.py`、`ab_compare.sh` +- API key 解析顺序：`$STEPFUN_API_KEY` → `${CLAUDE_PLUGIN_DATA}/config.json` 兜底 +- `references/migration_from_v2.md` 给出审查拦截的改写策略 + +**要求**：StepFun API key（https://platform.stepfun.com/）。 + +--- + ## 🎬 交互式演示画廊 想要在一个地方查看所有演示并具有点击放大功能？访问我们的[交互式演示画廊](./demos/index.html)或浏览[演示目录](./demos/)。 @@ -2240,6 +2283,12 @@ uv run douban-skill/scripts/douban-rss-sync.py ### Terraform 与 IaC 故障排查 使用 **terraform-skill** 当 `terraform apply` 在 provisioner 步骤失败、新实例遇到 "docker: not found"、或多环境 setup 意外共享快照时。Skill 里每一条都是*确切报错 → 根本原因 → 复制粘贴修复*三元组，来自真实事故。特别适合曾经被 cloud-init 的时序竞争、local-exec 里 rsync 连接断开、或者 Caddyfile 里硬编码域名搞掉一个周末的人。 +### 网络、流式与协议层调试 +使用 **debugging-network-issues** 应对症状和"显然原因"对不上的场景：HTTP/2 `RST_STREAM`、SSE 在 60s/100s/130s 整点卡死、"时灵时不灵"故障、或 CDN / 代理 / CGNAT 链路上的空闲超时事件。Skill 用**分层隔离实验**（同一逻辑请求走三条以上、每条仅差一跳的路径）替代假设堆叠，再加一套反审查模式——只在假设被**证伪**而不是单纯被"证实"之后才上 fix。 + +### 中文 TTS 与长音频转写（StepFun 阶跃） +使用 **stepfun-tts** 进行中 / 日语语音合成（通过 `instruction` + 行内 `()` 控制情绪与韵律），或单次最长 30 分钟的长音频转写（32K context、~100x RTF）。封装了让 StepAudio 2.5 新用户必踩的三个破坏性变更：`voice_label` 移除、`/v1/audio/asr/sse` 端点错位、更严的审查规则。可与 **transcript-fixer** 组合做 ASR 后处理，或与 **meeting-minutes-taker** 把长录音变成结构化纪要。 + ## 📚 文档 每个技能包括： @@ -2295,6 +2344,8 @@ uv run douban-skill/scripts/douban-rss-sync.py - **douban-skill**：参见 `douban-skill/SKILL.md` 了解导出工作流，参见 `douban-skill/references/troubleshooting.md` 查看 7 种被测抓取方案及失败原因的完整日志 - **terraform-skill**：参见 `terraform-skill/SKILL.md` 查看按确切报错 → 根本原因 → 复制粘贴修复组织的实操陷阱完整目录 - **slides-creator**：参见 `slides-creator/SKILL.md` 了解叙事优先工作流，参见 `slides-creator/references/narrative-design-guide.md` 了解 ABCDEFG 模型，参见 `slides-creator/references/content-creation-first-law.md` 了解通用内容创作原则 +- **debugging-network-issues**：参见 `debugging-network-issues/SKILL.md` 了解证伪优先工作流，参见 `debugging-network-issues/references/layered-isolation-experiment.md` 了解多跳隔离模式，参见 `debugging-network-issues/references/case-sse-rst-130s.md` 查看真实生产案例 +- **stepfun-tts**：参见 `stepfun-tts/SKILL.md` 了解 TTS+ASR 决策树，参见 `stepfun-tts/references/migration_from_v2.md` 查看 `voice_label` → `instruction` 迁移手册和审查改写清单 ## 🛠️ 系统要求 @@ -2320,6 +2371,7 @@ uv run douban-skill/scripts/douban-rss-sync.py - **Python 3.8+**（用于 continue-claude-work）：内置脚本进行会话提取（无外部依赖） - **uv + Scrapling CLI**（用于 scrapling-skill）：`uv tool install 'scrapling[shell]'`，浏览器抓取前运行 `scrapling install` - **Node.js 18+ + curl + unzip**（用于 ima-copilot）：`npx skills` 按需从 npm registry 拉取；IMA OpenAPI 凭据从 [https://ima.qq.com/agent-interface](https://ima.qq.com/agent-interface) 获取 +- **StepFun API key**（用于 stepfun-tts）：在 [https://platform.stepfun.com/](https://platform.stepfun.com/) → API Keys 获取 ## ❓ 常见问题 diff --git a/debugging-network-issues/.security-scan-passed b/debugging-network-issues/.security-scan-passed new file mode 100644 index 00000000..919cfcc0 --- /dev/null +++ b/debugging-network-issues/.security-scan-passed @@ -0,0 +1,4 @@ +Security scan passed +Scanned at: 2026-04-26T21:45:20.631240 +Tool: gitleaks + pattern-based validation +Content hash: 2af01a8c2d8c1638f9ad9c1c0abe9c13cfa533d07bb8706803e2e3f80edb2821 diff --git a/debugging-network-issues/SKILL.md b/debugging-network-issues/SKILL.md new file mode 100644 index 00000000..d2c6f4db --- /dev/null +++ b/debugging-network-issues/SKILL.md @@ -0,0 +1,207 @@ +--- +name: debugging-network-issues +description: Evidence-driven investigation for network, streaming, and protocol-layer bugs. Use when debugging connection resets (ECONNRESET, HTTP/2 RST_STREAM, INTERNAL_ERROR), SSE or long-polling stalls, fixed-time connection drops, CDN/proxy/CGNAT idle timeouts, or any incident where symptoms do not match the obvious cause. Applies falsification-first methodology — layered isolation experiments to pin down the responsible network layer, env-gated runtime instrumentation for non-invasive observation, and counter-review agent teams to challenge single-cause assumptions. Strongly trigger on "socket closed unexpectedly", "stream interrupted", "ECONNRESET", "HTTP/2 INTERNAL_ERROR", "fails after N seconds", "works sometimes but not always", "upstream silent for X seconds", or any scenario where the investigator might jump to conclusions before evidence. Generalizes to any multi-layer system investigation where assumption-first thinking is the failure mode. +--- + +# Debugging Network Issues + +Evidence-driven investigation methodology for incidents where the obvious cause is probably wrong. Built from a real 5-hour production case (see [references/case-sse-rst-130s.md](references/case-sse-rst-130s.md)) where assumption-stacking wasted hours that a 10-minute layered experiment would have resolved. + +Apply this skill when the user reports a network/streaming/protocol symptom and the investigator feels tempted to diagnose from one log line or one circumstantial data point. The skill's job is to slow that reflex down. + +## Core principles + +### 1. Evidence over assumption + +If you cannot point to a concrete artifact — log line, pcap frame, probe output, metric sample — you are guessing, not diagnosing. Before stating "X is the cause", require yourself to name the direct evidence. If it does not exist yet, add instrumentation (see [references/instrumentation-patterns.md](references/instrumentation-patterns.md)) or capture it (see [references/packet-capture-recipes.md](references/packet-capture-recipes.md)) before continuing. + +### 2. Falsification over confirmation + +N independent sources "confirming" a hypothesis does not make it true. One falsifying observation rules it out. Before acting on a hypothesis, answer: + +> "What observation would make me abandon this hypothesis?" + +If the answer is "nothing" or "I cannot think of one", the hypothesis is unfalsifiable and must not drive the investigation. If the answer is concrete, go look for that observation before committing to action. + +### 3. Layered isolation + +Multi-hop systems (client → CDN → LB → reverse proxy → app → upstream) concentrate bugs at the seams between layers. When a symptom could plausibly come from several layers, **do not reason about which layer; test**. The canonical technique: run the same logical request through three or more paths that differ by exactly one hop, then compare where the symptom appears. This resolves in minutes what stacking hypotheses cannot resolve in hours. See [references/layered-isolation-experiment.md](references/layered-isolation-experiment.md). + +### 4. Counter-review before committing + +Before committing to a root cause or shipping a fix, have independent reviewers challenge the conclusion — not confirm it. Agents are good at surfacing risks a single investigator did not think of; they are bad at weighing them. Apply the four-question filter (see [references/counter-review-pattern.md](references/counter-review-pattern.md)) to every finding before it shapes action. + +## Workflow + +Copy this checklist into the investigation notes and check items off: + +``` +Investigation Progress: +- [ ] Step 0: Scope the symptom (exact error, exact times, who, who-not, what changed) +- [ ] Step 0.5: Verify the premise — does direct evidence show the symptom is actually happening? +- [ ] Step 1: Gather direct evidence at every hop before hypothesizing +- [ ] Step 2: Frame ≥3 hypotheses; for each, name (a) what falsifies it, (b) which layer boundary the intervention would target +- [ ] Step 3: Design a decisive experiment (for network: layered isolation) +- [ ] Step 4: Add instrumentation if evidence gaps block direct observation +- [ ] Step 5: Execute, record actual vs predicted +- [ ] Step 6: Counter-review before acting +- [ ] Step 7: Fix + re-run the same experiment to verify +- [ ] Step 8: Document wrong turns as teaching material +``` + +### Step 0: Scope + +A tight scope is the difference between a 20-minute investigation and a 5-hour one. Before looking at anything, extract: + +- **Exact error string** (copy-paste, not paraphrase). `socket closed` is not the same as `ECONNRESET` is not the same as `HTTP/2 RST_STREAM INTERNAL_ERROR (err 2)`. +- **Exact timestamps** (ISO-8601 with timezone, not "yesterday evening") +- **Reproducibility** (every time / intermittent / only specific users) +- **Who is affected, who is not** (differential observations narrow the search) +- **What changed recently** (deploys, config, upstream dependencies, client versions) + +Distinguish symptom from diagnosis. "Slow" is not a symptom. "Request took 130.898s then returned HTTP/2 INTERNAL_ERROR" is. + +### Step 0.5: Verify the premise + +Before investing in a full investigation, confirm the reported symptom is actually happening — not just inferred from downstream effects or user frustration. One cheap direct observation beats hours spent investigating a non-problem. + +Ask: **"What direct evidence shows this symptom is real?"** + +- If the user reports "timeout at 130s": is that from a timestamped log, a browser network panel, or a recollection? +- If the user reports "connection reset": did they see the packet or is it inferred from a retry spike? +- If the user reports "fails for some but not others": has it been reproduced in a controlled test, or is it anecdotal? + +Acceptable premises: +- Log line with timestamp and error string +- Browser DevTools Network screenshot showing the failure +- Reproduction command that shows the symptom on demand +- Metrics chart showing the specific error count rising + +Not sufficient as premise: +- "Users are saying it feels slow" +- "The alert fired but I did not check what actually failed" +- "Last week someone mentioned..." + +If the premise fails verification, the fix is observation — not investigation. Add the missing telemetry, wait for the next occurrence with instrumentation in place, and return when you have real data. Resist the sunk-cost instinct to investigate anyway "since we are already here". + +### Step 1: Gather direct evidence at every hop + +Before framing hypotheses, collect: + +- Server-side logs at every hop in the request path +- Client-side logs (browser devtools HAR, CLI debug log, SDK traces) +- Metrics over the incident window (RPS, latency, error rate, connection count, CPU/mem) +- Distributed trace if available +- Packet capture if the symptom is at the wire level (see [references/packet-capture-recipes.md](references/packet-capture-recipes.md)) + +If any of these is missing and relevant, **fill the gap before guessing**. Adding a `TRACE_*` env flag and restarting a container beats an hour of hypothesis-stacking. The instrumentation patterns in [references/instrumentation-patterns.md](references/instrumentation-patterns.md) are low-risk, env-gated, and safe to ship into production permanently. + +### Step 2: Hypotheses with falsifiers and threat-model boundaries + +List three or more plausible causes. For each, write three sentences: + +- **What would confirm it?** (easy and often misleading) +- **What would refute it?** (the falsifier — this is what matters) +- **Which layer boundary would the intervention target?** (the threat-model question — forces you to be precise about where the fix would apply) + +The third question prevents a common anti-pattern: proposing a fix that operates on the wrong hop. For example, a "keepalive" fix that writes bytes downstream to the client is useless for an *upstream* idle timeout — the intervention targets a different boundary than the problem. Naming the boundary up-front surfaces this mismatch before coding starts. + +If you cannot state a concrete refuter, the hypothesis is unfalsifiable. Flag it, but do not act on it. If you cannot state which boundary a proposed fix targets, you do not yet understand what the fix actually does. + +### Step 3: Decisive experiment + +For network-layer problems, the default is **layered isolation**: three paths differing by exactly one hop. Example for a CDN-fronted service: + +| Path | Route | Rules out if it passes | +|------|-------|-----------------------| +| A | Full path via CDN | Nothing — this is the failing baseline | +| B | `--resolve` to origin IP (bypass CDN) | CDN layer | +| C | Server loopback (bypass CDN + LB) | CDN + LB | + +If only A fails, the CDN is the cause. If A and B fail but C passes, the LB is. Compose more variants as needed. See [references/layered-isolation-experiment.md](references/layered-isolation-experiment.md) for a runnable template using a mock idle upstream — the experiment does not need a cooperating production request to trigger, the idle interval can be controlled precisely. + +For non-network domains: +- Performance: controlled benchmark with one variable changed +- Correctness bug: failing test case that reproduces +- Intermittent: sampled tracing + wait for recurrence + +### Step 4: Instrumentation when needed + +If the decisive experiment requires an observation that cannot currently be made, add it — do not skip it. The canonical pattern is env-gated instrumentation that: + +- Defaults off (zero runtime cost in steady state) +- Turns on via one environment variable, without code changes +- Writes greppable log tags (`[SSE-CHUNK] ts=... req=... bytes=...`) +- Ships into production permanently — future incidents reuse it + +See [references/instrumentation-patterns.md](references/instrumentation-patterns.md) for the exact template used to diagnose the Qiniu 125-second upstream silence in this incident. + +### Step 5: Execute and record + +Run the experiment once, fully documented: command, environment, inputs, observed outputs, wall-clock timestamps. Compare against the prediction made in Step 2. If actual matches predicted, the hypothesis is calibrated. If not, the hypothesis is wrong — **do not rescue it with ad-hoc auxiliary hypotheses** ("oh, but maybe X also interferes..."). Return to Step 2 and write new hypotheses from scratch. + +### Step 6: Counter-review + +Before committing to a root cause or shipping a fix, spawn independent reviewers to challenge the conclusion. Give them the same evidence, ask them to falsify, not confirm. Apply the four-question filter to each finding they raise: + +1. **Probability** — will this actually happen? +2. **Cost** — what is the cost of fixing versus ignoring? +3. **Realistic scenario** — does this apply to the user's actual business case? +4. **Verification** — can I cheaply confirm or refute this? + +Classify every finding: real issue / partly right / unlikely / actively harmful. Never paste raw agent output to the user; filter first. See [references/counter-review-pattern.md](references/counter-review-pattern.md). + +### Step 7: Fix and verify + +Apply the fix. Rerun the same decisive experiment from Step 3. Confirm the symptom no longer reproduces with the same setup that was reliably producing it. If the pre-fix state can no longer be reproduced after the fix, the fix cannot be proven — figure out why the repro was lost before declaring victory. + +### Step 8: Document wrong turns + +The wrong turns in the investigation are more valuable than the right answer. Write an incident report capturing: + +- Symptom + direct evidence +- Each hypothesis tried + how it was falsified +- Decisive experiment design + result +- Fix + verification +- New monitoring or instrumentation added + +Future investigators — including future self — will read this to avoid the same cognitive traps. + +## Common cognitive traps + +1. **Circumstantial evidence convergence.** Five indirect clues all pointing the same direction feel like proof. They are not. If a direct probe is cheap, run it. +2. **Field-semantic confusion.** `duration=5.95s` can mean total wall time (one tool), handler execution phase (another tool), or TTFB (a third). Never cite a numeric field without verifying its semantics against documentation or code. +3. **Single-cause bias.** Multi-layer systems fail from multi-layer defect compositions. Fix the direct cause but document the amplifying factors so the next layer of defense can also be hardened. +4. **Naming assumption.** A resource labeled `spot-instance` may not actually be a spot instance. Verify attributes via API, not metadata names. +5. **Probe self-verification.** A diagnostic that runs through the broken connection to test the broken connection yields uninterpretable results. Always cross-verify with an independent probe. +6. **Assumption-rescue cycle.** When evidence contradicts a hypothesis, the temptation is to add a modifier ("yes, but only in case X"). Resist. If the first falsifier fires, scrap the hypothesis. +7. **Unverified premise.** Investigating a symptom that was never directly observed — inferred from user frustration, alert titles, or downstream effects. Verify first (Step 0.5). Do not investigate anecdotes. +8. **Threat-model mismatch.** Proposing a fix that targets the wrong layer — writing bytes downstream to solve an upstream problem, tuning a timeout on a hop that never fires it. Naming the boundary each hypothesis targets (Step 2) surfaces this. + +See [references/cognitive-traps.md](references/cognitive-traps.md) for extended examples including this case study. + +## Anti-patterns — things to explicitly avoid + +- **Jumping to a fix before a falsifier is found.** "Probably it is X, let me restart / tweak / upgrade." This converts learning opportunities into mystery fixes that do not prevent recurrence. +- **Accepting agent counter-review findings wholesale.** Agents over-produce risk findings. Filter before acting (see four-question filter above). +- **Ad-hoc production edits that bypass IaC.** If the investigation requires changing production, change the source-of-truth first, then apply — otherwise the "fix" evaporates on the next deploy and the drift hides the real state. +- **Declaring root cause from a single observation.** Demand a falsifier attempt first. +- **Writing "should work now" without re-running the failing experiment.** Re-verify. + +## Case study + +The [references/case-sse-rst-130s.md](references/case-sse-rst-130s.md) walks through a full 5-hour investigation where the assistant repeatedly jumped to the wrong conclusion. The right answer — Cloudflare edge HTTP/2 stream idle timeout at 126 seconds, amplified by Qiniu not emitting SSE ping during Sonnet 4.6 tool_use generation — surfaced in 10 minutes once a subagent designed a 3-path layered isolation experiment with a mock idle upstream. Read the case study before applying this skill to an unfamiliar problem domain; the wrong-turn anatomy is the teaching. + +## Reference files + +- [references/layered-isolation-experiment.md](references/layered-isolation-experiment.md) — 3-path technique, mock upstream template, result matrix +- [references/instrumentation-patterns.md](references/instrumentation-patterns.md) — env-gated TRACE_*, greppable log tags, deployment checklist +- [references/packet-capture-recipes.md](references/packet-capture-recipes.md) — tcpdump filters for RST isolation, interface selection on Docker, HTTP/2 decoding +- [references/counter-review-pattern.md](references/counter-review-pattern.md) — 4-agent team composition, 4-question filter, integration workflow +- [references/cognitive-traps.md](references/cognitive-traps.md) — extended examples, rescue-cycle warnings +- [references/case-sse-rst-130s.md](references/case-sse-rst-130s.md) — canonical case study with wrong-turn timeline + +## Scripts + +- [scripts/mock-idle-upstream.py](scripts/mock-idle-upstream.py) — SSE server that emits one frame then idles N seconds. Use as the upstream in layered isolation experiments to precisely control the idle interval. +- [scripts/layered-isolation-probe.sh](scripts/layered-isolation-probe.sh) — Runs the 3-path A/B/C comparison and prints a diagnostic matrix. diff --git a/debugging-network-issues/evals/evals.json b/debugging-network-issues/evals/evals.json new file mode 100644 index 00000000..77bb0553 --- /dev/null +++ b/debugging-network-issues/evals/evals.json @@ -0,0 +1,62 @@ +{ + "skill_name": "debugging-network-issues", + "evals": [ + { + "id": 1, + "name": "cross-domain-websocket-fixed-time-close", + "prompt": "我们的实时推送服务有个怪问题。用户 Alice 的客户端每次 WebSocket 连上之后，精确在 87 秒左右被断开，控制台报 `WebSocket is already in CLOSING or CLOSED state`。但同事 Bob 用相同的客户端代码没问题。服务端 nginx 日志没看到明显 error，upstream Go 服务也没记录异常。我们的架构是：浏览器 → Cloudflare → nginx → WebSocket server (Go)。Alice 现在开发被卡住，我应该怎么排查？", + "expected_behavior": [ + "推荐先收集直接证据（CLI/浏览器 DevTools Network、server logs、tcpdump）而不是直接建议 nginx 配置调整", + "提出至少 3 个候选根因并附带各自的 falsifier（能证伪的观测）", + "建议分层隔离实验——至少绕开 CF 的一条路径对比", + "识别 87 秒这个固定时长可能对应某个中间层的 idle timeout（而不是客户端代码 bug）", + "推荐 env-gated instrumentation 或 tcpdump 过滤 RST 来定位 close 来源", + "不盲目推荐 'restart nginx' 或 'upgrade client library' 这类无证据动作" + ], + "files": [] + }, + { + "id": 2, + "name": "batch-job-intermittent-no-restart-shortcut", + "prompt": "我们的凌晨批处理 job 最近一周有 3 天失败了（4/20, 4/21, 4/22），每次都是 `connection reset by peer`，成功和失败的几天我看不出配置或代码有什么不同。SRE 同事说 'restart 一下就好'，但我想搞清楚根因——这种间歇性问题之前发生过，每次 restart 完就不管了，过几周又复发。你能帮我规划一个系统性调查方法吗？", + "expected_behavior": [ + "明确反对 'restart 了事'，说明为什么浅层 workaround 会让问题复发", + "建议先扩展时间窗口收集证据：失败当天前后的 metrics/logs（不局限在 job 报错时间点）", + "建议加 instrumentation——至少在上游连接层加时间戳日志，以便下次复现时直接看到证据", + "提出至少 3 个候选根因并写出各自的 falsifier", + "建议差异对比：成功的日子 vs 失败的日子有什么变化（负载、其他 job、外部依赖）", + "指出 'connection reset by peer' 是 OS 层事件，谁 reset 需要 tcpdump 或上游日志证实" + ], + "files": [] + }, + { + "id": 3, + "name": "keepalive-patch-counter-review", + "prompt": "我写了一个 SSE keepalive 补丁准备上生产，目的是 upstream idle > 15s 时主动塞 `: keepalive\\n\\n` 防止被中间层 RST。代码大概是：\n```js\nlet lastChunk = Date.now();\nconst timer = setInterval(() => {\n if (Date.now() - lastChunk > 15000 && !res.writableEnded) {\n res.write(': keepalive\\n\\n');\n }\n}, 10000);\nproxyRes.on('data', chunk => { lastChunk = Date.now(); /* existing forwarding */ });\nproxyRes.on('end', () => clearInterval(timer));\n```\n你能帮我审查一下有没有风险？我打算 canary 5% 流量后 24 小时内全量。", + "expected_behavior": [ + "识别 timer 启动后立刻可能 fire（如果 upstream 还没响应 headers），在 res.writeHead 之前 res.write 会触发 Node 隐式 header 发送 → 后续 writeHead 会抛 ERR_HTTP_HEADERS_SENT", + "检查 clearInterval 是否覆盖所有 exit path（proxyReq.on('error')、proxyRes.on('aborted')、res.on('close') 等）", + "至少提 1 个关于非流式响应被误伤的风险（non-streaming JSON 场景不应该加 keepalive）", + "建议 canary 前先做更精确的 gate（env flag / response-type check）", + "提到 SSE comment 帧的 client 兼容性（Anthropic SDK / OpenAI SDK / 浏览器 EventSource 都忽略 `:` 前缀，这点 OK）", + "应用 4-question filter 或类似批判性框架——不是盲目列所有 'theoretical' 风险" + ], + "files": [] + }, + { + "id": 4, + "name": "db-pool-exhaustion-generalization", + "prompt": "我们生产应用最近一周每天下午 3-4 点期间，API 错误率会从 0.1% 飙到 2% 左右，持续 30-60 分钟然后自己恢复。logs 里主要是 `Error: pool is at capacity` from 我们的 DB driver (pg-pool)。DB 那边 CPU/mem 看起来 OK，慢查询日志也没明显异常。运维同事说 \"pool 加大就好\"，我们的 config 已经从 20 连接加到 50 然后到 100 了，每次加大都撑一阵子又开始触发。这个模式让我怀疑根因不是 pool 大小本身。你帮我设计下调查计划？", + "expected_behavior": [ + "Verifies the premise: asks for direct evidence (metrics, logs) that 3-4pm spike is real pool error vs generic 5xx", + "Rejects \"scale pool larger\" as surface-level; explains why each upsize buys time but does not fix root cause", + "Proposes at least 3 falsifiable hypotheses (e.g., upstream dep latency spike, cron workload overlap, lock contention, query plan regression)", + "Uses threat-model framing: each proposed fix names which layer boundary it operates on (pool size = app-local; query slowness = DB-side; workload = upstream)", + "Recommends differential analysis: compare 3-4pm window vs other times (traffic pattern, other jobs, external deps)", + "Recommends instrumentation BEFORE next recurrence (pool-wait metrics, query timing histogram, lock wait time)", + "Does NOT over-apply network-specific tools (tcpdump / layered isolation) where they do not fit — methodology should adapt to DB domain" + ], + "files": [] + } + ] +} \ No newline at end of file diff --git a/debugging-network-issues/references/case-sse-rst-130s.md b/debugging-network-issues/references/case-sse-rst-130s.md new file mode 100644 index 00000000..dd03efe9 --- /dev/null +++ b/debugging-network-issues/references/case-sse-rst-130s.md @@ -0,0 +1,181 @@ +# Case Study: SSE HTTP/2 RST at 130s (130s RST incident, April 2026) + +A production incident where a user's Claude CLI kept failing with `ECONNRESET` after exactly 130 seconds on long tool_use tasks. The investigation took 5 hours and produced three wrong root-cause conclusions before a structured experiment resolved it in 10 minutes. + +This case study is the canonical teaching material for this skill. Read it to understand the anatomy of how assumption-first investigations go wrong, and how to recognize when to switch to evidence-first. + +## Contents +- Symptom +- Environment +- Investigation timeline (with wrong turns) +- The decisive experiment +- Final root cause +- Post-mortem lessons + +## Symptom + +The reporting user (handle: User A), using Claude CLI 2.1.116 on Windows (Node v24.3.0), submits long tasks via `ANTHROPIC_BASE_URL=https://api.example.com/openrouter`. Tasks involving Claude Sonnet 4.6 + long tool_use (writing long files with the Write tool) consistently fail with: + +``` +API Error: The socket connection was closed unexpectedly. +For more information, pass `verbose: true` in the second argument to fetch() +``` + +In CLI debug logs: + +``` +2026-04-22T13:43:38.261Z [DEBUG] [API REQUEST] /openrouter/v1/messages +2026-04-22T13:43:47.728Z [DEBUG] Stream started - received first chunk +2026-04-22T13:45:57.344Z [ERROR] Error in API request: The socket connection was closed unexpectedly. +2026-04-22T13:45:57.347Z [ERROR] Connection error details: code=ECONNRESET +``` + +From first chunk (`t=9s`) to ECONNRESET (`t=130s`): exactly 130 seconds. Reproducible across sessions. + +## Environment + +- Client: Claude CLI 2.1.116, Windows 11, Node v24.3.0 +- Server: api.example.com (Cloudflare-proxied, origin is aliyun Japan, 203.0.113.10) +- Architecture: `CLI → CF → Caddy → lobe-provider-gateway → lobe-new-api → Qiniu (Anthropic proxy) → Sonnet 4.6` +- Not affected: same user's Haiku requests, other users' Sonnet 4.6 requests with shorter duration (all <45s) + +## Investigation timeline (with wrong turns) + +### Hour 1: VPN theory (wrong) + +Observation: `Cf-Connecting-Ip` varied between China (198.51.100.42) and US (203.0.113.x) across requests, sometimes in the same session. + +Hypothesis: User's VPN is unstable, rotating exit nodes, TCP dies when the route changes. + +Evidence gathered: IP log showed the flipping. CF Ray always terminated at SJC (US). + +Action taken: recommend user disable VPN and retry. + +Falsification: User responds "disabled VPN, still fails at 130s." The requests with China-origin IP also fail. + +**Trap**: Circumstantial evidence convergence (Trap 1). The IP flipping was real but the VPN was not the cause. + +### Hour 2: CLI version bug theory (wrong) + +Observation: Failing requests' response bodies archived to OSS all terminate at suspiciously similar byte positions (3538 / 3902 / 3946 bytes). In each case, the response truncates mid-way through a `tool_use` `input_json_delta` containing a path string with Chinese characters (`D:\翩姐\AI剪辑\培训...`). + +Hypothesis: Claude CLI 2.1.116 has a bug parsing `input_json_delta` containing Chinese + Windows backslashes. Client closes the socket when it fails to parse. + +Evidence gathered: Three failures, all at ~3.5-3.9 KB, all at similar character positions. Strong pattern. + +Action tentative: recommend CLI upgrade to 2.1.117. + +Falsification: User's CLI debug log shows the close happens **130 seconds after first chunk**, not at the instant the byte is received. If it were a parse bug, the close would be within milliseconds of the problematic byte. Additionally, CLI 2.1.2 (older) worked fine on similar content. + +**Trap**: Field-semantic confusion (Trap 2). A misread of Caddy's `duration=5.95s` field led to believing the close was fast (5s), which made the parse-bug theory plausible. When CLI debug logs were examined, the actual timing was 130s, contradicting the hypothesis. + +### Hour 3: Caddy IdleConnTimeout theory (wrong) + +After a subagent suggested examining Caddy reverse_proxy defaults, a hypothesis formed: Caddy's HTTP/1.1 transport `IdleConnTimeout` defaults to 120s; combined with TTFB that explains the 130s. + +Evidence: Caddy source code shows `IdleConnTimeout = 120s` default. 120 + 10s TTFB = 130s close match. + +Action tentative: add explicit `keepalive_timeout 30m` to Caddy config. + +Falsification: A probe from the server itself, `curl --resolve api.example.com:443:127.0.0.1 https://api.example.com/openrouter/v1/messages ...`, runs for 200+ seconds without closing. This path goes through Caddy (just not through CF). If Caddy's IdleConnTimeout were the cause, this probe would also fail at 130s. It does not. + +**Trap**: Assumption-rescue cycle (Trap 6) was tempting — "maybe Caddy only triggers IdleConnTimeout under specific conditions" — but the direct probe was decisive. Abandoned the hypothesis cleanly. + +### Hour 4: Qiniu no-ping theory (partial) + +Observation: Scanning 38 archived response bodies shows all of them contain zero `event: ping` frames and zero SSE comment-only frames (`:` prefix lines). Anthropic protocol specifies periodic `ping` events during long inactivity. Qiniu appears to strip or not forward them. + +Hypothesis: Qiniu does not forward SSE ping → connection looks idle to the CDN → CDN closes at its idle threshold. + +Evidence: 38 bodies, ping count = 0, consistent. + +Action tentative: deploy server-side keepalive in provider-gateway to compensate. + +Partial refutation (from counter-review agent): Anthropic's own official API has been reported (GitHub issues `claude-code#18028`, `claude-agent-sdk-typescript#44`) to stall 59-138+ seconds with no event during tool_use generation. So Qiniu's no-ping behavior, while real, is not sufficient as an independent root cause — the upstream itself is silent, Qiniu has nothing to forward. + +This hypothesis is partly correct — Qiniu does not emit ping — but it is not the direct cause of the close. It is an amplifying factor (absence of keepalive that would have prevented the idle timeout somewhere). + +**Progress**: we now know "something on the wire is quiet for ~125 seconds during tool_use generation" but we still do not know which layer is closing the connection. + +### Hour 5: The decisive experiment + +Subagent designed and executed a 3-path layered isolation experiment: + +1. **Mock upstream**: Python/Flask on port 19999 that emits one SSE frame then sleeps 200 seconds — precisely simulating the observed upstream silence pattern. + +2. **Temporary routing**: Added CF DNS record for `test-idle.example.com` pointing at origin (proxied: true), plus a Caddy conf snippet forwarding that hostname to the mock. + +3. **Three paths**, run in parallel: + - **Path A** (via CF): `curl https://test-idle.example.com/...` + - **Path B** (bypass CF): `env -i curl --resolve test-idle.example.com:443:203.0.113.10 ...` + - **Path C** (server loopback): `ssh server 'curl http://127.0.0.1:19999/probe-c'` + +Results (multiple runs, consistent): + +| Path | Close time | Curl error | +|------|------------|-----------| +| A (via CF) | 126.01s | HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2) | +| B (bypass CF) | 220s (clean, hit client --max-time) | none (curl side closed on --max-time) | +| C (loopback) | 220s (clean, hit client --max-time) | none | + +**Interpretation**: Only Path A closes early. B and C traverse the same Caddy/origin/upstream stack but bypass CF. Therefore the close originates at Cloudflare's edge, not at any layer below it. + +Additionally, curl's error `HTTP/2 stream N was not closed cleanly: INTERNAL_ERROR (err 2)` indicates the reset was a peer-sent `RST_STREAM` HTTP/2 frame, not a TCP RST. Confirmation that CF sent an HTTP/2-layer close while the TCP connection itself remained healthy for other streams. + +### One subtle wrinkle from the experiment + +The first run of Path B appeared to close at 126s too, which would have falsely implicated the origin. Investigation revealed the client machine had a system-level Shadowrocket proxy on `http_proxy=http://127.0.0.1:1082` that was silently routing the `--resolve`-bypassed traffic back through CF. `env -i curl` (strip all environment) then correctly showed Path B clean at 220s. + +This is **Trap 5** (probe self-verification) caught in real time. Mitigation: `env -i curl ...` is now a mandatory reflex when running bypass-comparison experiments. + +## Final root cause + +**Direct cause**: Cloudflare edge closes HTTP/2 streams that go idle for ~126 seconds (empirically observed constant, matches the 120-130s close-time range on Path A across runs). + +**Amplifying factors**: + +1. Qiniu proxy does not emit SSE `event: ping` during upstream silence (38/38 observed bodies had zero ping) +2. Upstream Claude Sonnet 4.6 during tool_use generation emits initial output then batch-generates the tool_use.input for 100+ seconds with no interim chunks +3. Claude CLI does not implement a client-side idle watchdog to detect the stall before the peer resets + +Any one of these, in isolation, would likely not produce the observed failure. All three together + the CF idle timeout produce it reliably for Sonnet 4.6 + long tool_use requests from this account. + +## Remediation + +Not shipped as part of this case study, but the intended fix vector: + +- **Server-side SSE keepalive in provider-gateway**: if the upstream has not emitted for N seconds, inject `: keepalive\n\n` comment frames (SSE-safe, ignored by clients, but keeps bytes flowing on the wire). This prevents CF from observing a full N-second idle. +- **Does not require client or upstream changes**. Single point of defense for all client/upstream combinations. + +Code reviewer counter-review (see [counter-review-pattern.md](counter-review-pattern.md)) caught two non-trivial bugs in the first keepalive draft before they shipped: + +- Writing keepalive bytes before the response header has been flushed triggers Node's implicit-header emission, corrupting non-streaming Anthropic JSON responses +- Several error-path `clearInterval` omissions that would leak timers + +Counter-review also verified (not assumed) one claim that could have been silently wrong: **SSE comment frames with `:` prefix are safely ignored by all standard clients**. Cross-checked against the WHATWG `EventSource` spec (lines beginning with `:` are interpreted as comments and discarded), the `@anthropic-ai/sdk` source (`SSEDecoder` skips comment lines), the `openai` SDK (`openai/streaming` follows the same contract), and lobe-chat's `EventSourceParserStream`. This verification was cheap — 5 minutes with grep — and removed an otherwise plausible failure mode ("what if some client treats our keepalive bytes as malformed SSE and errors out?") from the risk list. The lesson: when a code review produces a compatibility claim, verify it from primary sources (spec + SDK source), do not leave it as "probably fine". + +## Post-mortem lessons + +### What went wrong + +- **5 hours before the experiment was proposed**. The experiment was cheap (~10 minutes to set up) but nobody proposed it until a counter-review agent did. The main investigator was stuck in hypothesis-stacking mode. +- **Three wrong hypotheses acted on before falsification.** Each was plausible. Each had circumstantial supporting evidence. None had a cheap falsifier run before acting. +- **One field-semantic mistake** (Caddy `duration=5.95s`) anchored an entire wrong direction for ~1 hour. +- **The user had to push back explicitly** ("I turned off VPN, still fails") to break the first wrong direction. The system should have surfaced that test earlier. + +### What went right + +- When the experiment was finally run, it was rigorous: 3 paths, multiple runs, server-side + client-side observation, explicit cleanup. +- Counter-review caught two real code bugs in the remediation. +- Instrumentation added mid-incident (env-gated TRACE_SSE_CHUNKS) yielded decisive evidence for the 125-second upstream silence, *and* is now permanent observability for future incidents. +- Docker compose was refactored mid-incident to bind-mount `server.cjs` from the host, eliminating the "rebuild image to add a log line" cycle permanently. + +### Transferable methodology (codified in this skill) + +1. When circumstantial evidence converges on a cause, demand one direct falsifier before acting. See [cognitive-traps.md](cognitive-traps.md), Trap 1. +2. When multiple layers could be responsible, do not reason — test. See [layered-isolation-experiment.md](layered-isolation-experiment.md). +3. When observability is missing, add it as an env-gated permanent feature. See [instrumentation-patterns.md](instrumentation-patterns.md). +4. Before committing to a root cause or shipping a fix, counter-review. See [counter-review-pattern.md](counter-review-pattern.md). +5. When a probe shares infrastructure with the subject of the probe, it is not a valid probe. Cross-verify. See [cognitive-traps.md](cognitive-traps.md), Trap 5. + +These five rules, applied at hour 1, would have resolved the incident in ~30 minutes instead of 5 hours. diff --git a/debugging-network-issues/references/cognitive-traps.md b/debugging-network-issues/references/cognitive-traps.md new file mode 100644 index 00000000..124c2739 --- /dev/null +++ b/debugging-network-issues/references/cognitive-traps.md @@ -0,0 +1,139 @@ +# Cognitive Traps + +Curated list of wrong-turn patterns observed in real investigations. Each entry: the trap, why it is seductive, a concrete example, and the counter-move. + +## Contents +- Trap 1: Circumstantial evidence convergence +- Trap 2: Field-semantic confusion +- Trap 3: Single-cause bias +- Trap 4: Naming assumption +- Trap 5: Probe self-verification +- Trap 6: Assumption-rescue cycle +- Trap 7: Time-of-symptom equals time-of-cause +- Trap 8: Duration of investigation biases conclusion weight +- Trap 9: Agent output equals ground truth +- Trap 10: Unverified premise +- Trap 11: Threat-model mismatch + +## Trap 1: Circumstantial evidence convergence + +Five indirect clues all pointing toward hypothesis H feel like proof. They are not, because they share a common cause (your mental model) that selected them. + +**Example** (from this case study): Initial assumption was "VPN node rotation". Supporting circumstantial evidence: +- Client IP flipped between CN and US across requests (real) +- Request to CF hit SJC PoP (real, expected for US-routed) +- Each failed request had short duration in some log field (misread — see Trap 2) +- User was known to sometimes use VPN (real) + +All four looked consistent, and the main investigator committed to "VPN instability is the root cause". The user pushed back: "I turned off VPN and it still fails." One falsifying test broke the chain. + +**Counter-move**: When circumstantial evidence converges, require at least one direct test before acting. "The IP flips" is circumstantial. "The same user reproduces the failure with VPN verifiably off" is direct. + +## Trap 2: Field-semantic confusion + +A number from a log field means whatever that field's code defines — not what the name suggests. + +**Example** (from this case study): Caddy's access log has `duration=0` and a separate warning log has `duration=5.95s`. The investigator read "duration 5.95s" and concluded "connection lasted 5.95 seconds before being reset". But that particular field in Caddy's `aborting with incomplete response` warning is the elapsed time between the abort signal and the handler winding down — not the total request lifetime. The actual request lifetime (from CLI debug log) was 130 seconds. + +The investigator then built a whole theory around "CLI fails at 5-8 seconds due to a bug in chunk parsing", which was wrong at the root. + +**Counter-move**: Never cite a numeric field value as evidence without checking its semantics in the source code or vendor documentation. If the field is suggestive but ambiguous, treat it as unverified until its meaning is confirmed. + +## Trap 3: Single-cause bias + +Real production failures often emerge from multiple cooperating defects. Finding one cause and stopping leaves the amplifying factors in place, which will trigger the next incident. + +**Example** (case study in full resolution): + +- Direct cause: Cloudflare edge HTTP/2 stream idle timeout at 126s +- Amplifying factor 1: Qiniu proxy does not emit SSE `event: ping` during upstream stalls +- Amplifying factor 2: Upstream Claude Sonnet 4.6 batches tool_use output (125s silences observed) +- Amplifying factor 3: Claude CLI has no client-side idle watchdog (GitHub issue documented) + +Fixing only the direct cause (e.g., moving off CF) would leave factors 1-3. Factor 2 means even with a different CDN with a larger idle window, a different idle threshold eventually fires. Factor 3 means the client is blind to the stall. The durable fix addresses factor 1 at minimum and factor 2 via server-side keepalive as defense in depth. + +**Counter-move**: After finding the direct cause, ask explicitly: "What amplifying factors enabled this? If the direct cause were fixed, what would still be wrong?" Document all layers, fix the most cost-effective ones. + +## Trap 4: Naming assumption + +Labels, tags, and names are metadata assigned by humans; they do not reflect runtime attributes. Verify via API, not by reading the name. + +**Example**: A cloud instance tagged `claude4dev-spot` was assumed to be a Spot pricing instance during an incident. The instance was actually PostPaid; the tag was legacy from a pre-migration period. The investigator spent 10 minutes down the wrong path (Spot reclamation theory) before checking `DescribeInstanceAttribute`. + +**Counter-move**: In incident response, the first step when a property matters is to query the authoritative API, not to read the name. + +## Trap 5: Probe self-verification + +A probe that uses the thing it is probing to deliver its result cannot independently verify that thing. + +**Example**: Using `curl` through a VPN to test whether the VPN is dropping connections. If the VPN drops, `curl` reports an error, which is what you expected — but the same error would occur if the remote host rejected the connection. The probe did not isolate. + +**Counter-move**: Probes must be structurally independent of the subject. To test the VPN, use a second network path (mobile hotspot) to compare. To test a CDN, bypass it with `--resolve`. The [layered isolation experiment](layered-isolation-experiment.md) is this principle systematized. + +## Trap 6: Assumption-rescue cycle + +When evidence contradicts a hypothesis, the temptation is to add a modifier: "yes, but only under condition X". This rescues the hypothesis at the cost of unfalsifiability — eventually the modifiers stack to "it fails when it fails". + +**Example** (case study): After "VPN instability" was falsified by "still fails with VPN off", a rescue was "well, maybe the VPN client has a residual system-level hook". Adding more conditions without evidence. + +**Counter-move**: When a falsifier fires, the correct response is to scrap the hypothesis, not to narrow its scope. Return to Step 2 of the workflow and write new hypotheses. + +## Trap 7: Time-of-symptom equals time-of-cause + +The time the user notices a symptom is often much later than the time the cause first engaged. Correcting this requires examining upstream time series. + +**Example**: Disk fills at midnight, various retries and degradations through the morning, user-facing failure at 10:30 AM. The 10:30 timestamp is when to start looking at logs, but if you examine only the 10:30 ± 5 minute window you will miss the midnight root cause. + +**Counter-move**: Always extend the investigation window backward by at least 10x the symptom-to-report time, or to the last known-good state. Look for monotonic metric trends crossing thresholds, not just error spikes. + +## Trap 8: Duration of investigation biases conclusion weight + +After four hours of deep investigation, the investigator has a strong psychological bias toward "we must be close" and against "start over". This leads to over-weighting marginal evidence that fits the current theory. + +**Example** (case study): After 3 hours of circumstantial evidence for "VPN theory", then "CLI bug theory", then "Caddy IdleConnTimeout theory", the investigator was resistant to "start a fresh experiment from scratch". The user pushed to switch approach. The experiment resolved in 10 minutes what 3 hours of deep reasoning had not. + +**Counter-move**: Time-box. If a hypothesis has not been confirmed (not just "consistent with evidence", but actually confirmed by a direct test) within a set time, switch to a structurally different approach. Layered isolation or an experiment is a good default switch. + +## Trap 9: Agent output equals ground truth + +Spawning an agent to investigate returns text that reads authoritatively. Accepting that text without verification treats the agent as a peer reviewer, but agents do not have skin in the game — they over-produce risks and claims. + +**Example**: A counter-review agent cites "Cloudflare proxy_read_timeout is 100s" with high confidence. This appears to match the observed 130s. The investigator concludes CF is the cause — except the actual CF limit in this case is a different timeout (HTTP/2 stream idle, ~126s), and "100s" was the agent generalizing from community posts without matching the exact protocol. + +**Counter-move**: Every agent claim that feeds into an action needs at least one cheap verification step. If the agent says "X is 100s", test whether X is actually 100s in your environment (or find the primary source). Filter agent findings through the [four-question filter](counter-review-pattern.md#the-four-question-filter). + +## Trap 10: Unverified premise + +Investigating a symptom that was never directly observed. The premise enters the conversation as "users say X is happening" or "the alert fired so X must be failing" and drives hours of hypothesis-building before anyone checks whether X is actually occurring. + +**Example**: A user reports "our SSE connections keep dropping at 130 seconds". The team spends 3 hours building a keepalive patch. On the verification run before ship, they realize the original symptom was a single-digit frequency over the last week — well within normal disconnect noise for that service — and the "130-second pattern" was coincidence across two samples. + +**Another example** (surfaced by counter-review in this case study): the proposed fix was server-side SSE keepalive. Counter-review asked: "does the user have direct evidence the RST is actually happening right now, or is this inferred from a past incident?" The fix was for a real incident that *had* occurred, so the question was answered correctly — but the habit of asking is what prevents investigating a non-problem. + +**Counter-move**: Before investigating, answer one question: "What direct artifact (log line with timestamp, captured packet, screenshot) shows this symptom is currently real?" If the answer is "nothing I can point to", the first action is not investigation — it is adding the telemetry and waiting for the next real occurrence. This is faster and more correct than investigating on vapor. + +See [SKILL.md Step 0.5](../SKILL.md) for the verification checklist. + +## Trap 11: Threat-model mismatch + +Proposing a fix that operates on the wrong hop. The hypothesis correctly identifies that *some layer* is at fault, but the implementation lands at a different layer, so the fix cannot actually remediate the real cause. + +**Example** (from eval-3 baseline in this skill's iteration-1 tests): a proposed SSE keepalive patch writes `: keepalive\n\n` to `res` (downstream client-facing connection). But the stated concern was "upstream idle > 15s". Writing bytes to the downstream socket does nothing to maintain the upstream TCP connection — if the idle timeout fires on the proxy→upstream hop, the keepalive is directed at the wrong boundary. The patch would ship without reducing the incidence of the original symptom. + +This trap is particularly insidious because the hypothesis about "why" was correct (idle timeout somewhere) and the fix category was correct (keepalive). Only the *layer* was wrong. + +**Counter-move**: For every proposed fix, explicitly name which layer boundary it operates on, and then check whether that boundary is the one where the problem originates. If they do not match, the fix is targeting the wrong thing regardless of how reasonable it looks in isolation. + +Phrased as a question: "My fix makes bytes flow at boundary X. Is X the same as the boundary where the problem manifests?" + +In the SKILL.md workflow, this is the Step-2 third-question prompt. Do it before writing code. + +## Summary: the meta-move + +All nine traps share a common structure: the investigator is willing to act on indirect evidence when a cheap direct test is available but was skipped. + +The universal counter-move, restated: + +> Before acting on a conclusion, identify the cheapest direct test that could falsify it. Run that test. + +If the test is expensive, accept the conclusion is provisional and design instrumentation that will make the test cheap next time. diff --git a/debugging-network-issues/references/counter-review-pattern.md b/debugging-network-issues/references/counter-review-pattern.md new file mode 100644 index 00000000..ff855836 --- /dev/null +++ b/debugging-network-issues/references/counter-review-pattern.md @@ -0,0 +1,160 @@ +# Counter-Review Pattern + +## Contents +- Why counter-review (not peer-review) +- The four-agent team composition +- The four-question filter +- Integration workflow +- When NOT to counter-review +- The case study: what counter-review surfaced + +## Why counter-review + +A lone investigator converging on a conclusion is highly susceptible to confirmation bias, especially after investing hours in a line of reasoning. Standard peer review is better but shares most of the investigator's context and inherits the same blind spots. + +**Counter-review** is adversarial by design: the reviewer's job is to *falsify* the conclusion, not confirm it. They start from the same evidence but are explicitly instructed to find what the investigator missed, find weaker-evidence conclusions the investigator over-weighted, and propose experiments that could disprove the current hypothesis. + +Counter-review works best when: + +- Multiple reviewers run in parallel with distinct framings +- Each reviewer has their own search/research capability (not just re-reading the same investigator's notes) +- Their outputs are filtered before acting, not accepted wholesale + +## The four-agent team composition + +This composition was used in this investigation and proved effective. Roles are distinct on purpose — they cover orthogonal angles. + +### 1. Independent diagnostician + +**Prompt framing**: "You have the complete evidence set. Reach your own conclusion without being anchored by mine. Especially: what hypotheses did I not consider?" + +**Typical value**: surfaces the "I forgot to check X" class of gap. In this case study, this agent was the first to raise Caddy's `IdleConnTimeout` default as a suspect — a layer the main investigator had not yet examined. + +**Agent type**: `general-purpose` with SSH/Bash access + +### 2. Assumption challenger + +**Prompt framing**: "The main conclusion is X. Challenge it using external research (WebSearch authoritative sources, vendor docs, bug trackers). Cite URLs. Do not rely on training data." + +**Typical value**: finds vendor-documented behavior that contradicts or qualifies the investigator's assumption. In this case study, this agent found published evidence that Anthropic's own official API also experiences SSE stalls during tool_use generation — downgrading the "Qiniu does not forward ping" hypothesis from "root cause" to "amplifying factor". + +**Agent type**: `general-purpose` with WebSearch + +### 3. Code reviewer (if a fix is proposed) + +**Prompt framing**: "The proposed fix is this [diff]. Audit for: race conditions, cleanup paths, unintended interactions with existing code, boundary conditions. Read the surrounding code." + +**Typical value**: catches "my fix would break the JSON response path" class of bug. In this case study, this agent caught that a proposed `setInterval` keepalive would corrupt non-streaming Anthropic JSON responses because `res.write` before `writeHead` triggers Node's implicit-header emission — a subtle bug the main investigator would likely have shipped. + +**Agent type**: `Plan` agent or `codex:rescue` with code-read access + +### 4. Decisive experiment designer + +**Prompt framing**: "Design and execute an experiment that decisively confirms or refutes the current root cause. Layered isolation preferred. Clean up after yourself." + +**Typical value**: converts hypothesis-stacking into definitive answer. In this case study, this agent set up a mock idle upstream, deployed temporary CF DNS and Caddy routes, ran 3 parallel paths, observed `126s RST only on Path A`, and cleaned everything up. What 5 hours of reasoning could not resolve, 10 minutes of experiment did. + +**Agent type**: `general-purpose` with Bash/SSH/file access + +### Launch pattern + +Launch all four in parallel (one message, four `Agent` tool calls with `run_in_background: true`). Process notifications as they return; do not wait for all before starting to read. Some will finish in minutes, the experiment designer often takes longer. + +## The four-question filter + +Agent reviewers over-produce findings. A code reviewer will generate 10 risk items even when 3 are worth acting on; a challenger will list 6 counter-hypotheses even when 1 is plausible. Paste-the-raw-agent-output is the anti-pattern. Filter every finding through four questions: + +### 1. Probability — will this actually happen? + +Distinguish real risks from theoretical risks. A race condition in a code path that runs once at startup on a single thread is theoretical. A race condition in a request handler under load is real. + +Ask: in the actual deployment, under actual load patterns, with actual input distributions, does this failure mode fire with >1% probability? If not, defer. + +### 2. Cost — what is the cost of fixing versus ignoring? + +For each finding: +- Cost of fixing: engineering time + regression risk + complexity added +- Cost of ignoring: expected incidents × their impact + +Some findings are real but cheap to accept (log a warning, move on). Others are real and cheap to fix (one-line guard). Prioritize the second. + +### 3. Realistic scenario — does this apply to the user's actual business case? + +Agents generalize. A counter-review of an internal-only tool often surfaces findings appropriate for a consumer product (rate limiting, CSRF, input fuzzing). Filter to the actual deployment context. + +### 4. Verification — can I cheaply confirm or refute this? + +For findings that survive 1-3, can you test the claim in under 5 minutes? If yes, test. If the test comes back negative, discard. If positive, elevate to actionable. + +Never accept a finding as real without at least one cheap verification. + +### Classification after filtering + +Classify each finding: + +- **Real issue** — act on it +- **Partly right** — acknowledge and narrow scope before acting +- **Unlikely** — log but do not act +- **Actively harmful** — the suggested fix would introduce a new bug; explicitly reject + +Report to the user with classification, not raw agent output. + +## Integration workflow + +``` +┌────────────────────────────────────────────────────────┐ +│ 1. Main investigator reaches tentative conclusion │ +│ and draft remediation │ +├────────────────────────────────────────────────────────┤ +│ 2. Launch 4 counter-review agents in parallel, │ +│ one message, run_in_background=true │ +├────────────────────────────────────────────────────────┤ +│ 3. As each returns, read full output │ +├────────────────────────────────────────────────────────┤ +│ 4. Apply 4-question filter to every finding │ +│ - probability, cost, realism, verifiability │ +├────────────────────────────────────────────────────────┤ +│ 5. For every finding that survives filter, │ +│ run cheap verification │ +├────────────────────────────────────────────────────────┤ +│ 6. Reclassify findings after verification │ +│ (real / partly / unlikely / harmful) │ +├────────────────────────────────────────────────────────┤ +│ 7. Update root cause / fix based on classified │ +│ findings │ +├────────────────────────────────────────────────────────┤ +│ 8. Report to user: classification table + justified │ +│ action list. No raw paste. │ +└────────────────────────────────────────────────────────┘ +``` + +## When NOT to counter-review + +Counter-review has overhead (5-30 minutes of parallel agent runs + filter time). Skip it when: + +- The root cause is already directly verified (you have the smoking gun, not circumstantial evidence) +- The fix is mechanical (e.g., updating a hardcoded version number) with no design decisions +- The incident is ongoing and the priority is stabilization, not perfect root cause +- The cost of getting it 80% right and iterating is lower than the cost of getting it 100% right the first time + +In other words: counter-review is for the **conclusion** phase, not the stabilization phase. + +## The case study: what counter-review surfaced + +Main investigator's tentative conclusion before counter-review: "Qiniu does not forward SSE ping, causing some middle layer to RST after ~130s. Fix: add server-side keepalive in provider-gateway." + +Counter-review surfaced: + +| Agent | Finding | Classification after filter | +|-------|---------|---------------------------| +| Challenger | Anthropic's official API also stalls 59-138s+ on tool_use (GitHub issues cited). "Qiniu not forwarding ping" is not sufficient as independent root cause. | Partly right — downgraded assumption from "root cause" to "amplifying factor" | +| Challenger | 130s matches Cisco CGNAT initial TCP timeout (120s + RTT) better than CF 100s. | Partly right — worth testing but did not change the fix | +| Code reviewer | Proposed keepalive would corrupt non-streaming Anthropic JSON responses (res.write before writeHead triggers implicit headers). | Real issue — fixed before deploy | +| Code reviewer | Several clearInterval paths missing (proxyReq.on error, proxyRes aborted). | Real issue — fixed before deploy | +| Code reviewer | SSE comment (`:` prefix) client-compatibility claim needed to be verified, not assumed. | Verified from primary sources (WHATWG EventSource spec + `@anthropic-ai/sdk` `SSEDecoder` source + `openai` SDK + lobe-chat EventSourceParserStream). Confirmed safe. Removed from risk list. | +| Independent | Caddy default IdleConnTimeout is 120s, could match the 130s constant. | Turned out wrong (ruled out by experiment) but good hypothesis | +| Experiment | 3-path layered isolation with mock idle upstream: Path A fails at 126s, B and C clean at 220s. | Definitive — pinpointed CF edge | + +Raw count: 6 findings surfaced. Acted on: 3 (2 code fixes, 1 definitive experiment result). Discarded: 1 (wrong). Downgraded but kept as context: 2. + +If the main investigator had pasted all six to the user as "here's what counter-review said", the user would have had to do the filtering work. The filter is the investigator's job. diff --git a/debugging-network-issues/references/instrumentation-patterns.md b/debugging-network-issues/references/instrumentation-patterns.md new file mode 100644 index 00000000..a9f365a9 --- /dev/null +++ b/debugging-network-issues/references/instrumentation-patterns.md @@ -0,0 +1,186 @@ +# Instrumentation Patterns + +## Contents +- When to instrument +- Env-gated TRACE pattern (the default) +- Log tag conventions +- Deployment checklist +- Worked example: TRACE_SSE_CHUNKS +- Analysis: extracting timing data from logs +- Persisting instrumentation versus removing it + +## When to instrument + +Instrument when a hypothesis cannot be confirmed or refuted from currently-available observability. If the system already emits the signal you need (distributed trace, access log field, metric), use that. If not, add instrumentation rather than guess. + +Symptoms that justify adding instrumentation: + +- "We do not know what the upstream is doing between these timestamps" — add chunk-level or event-level logging at the boundary +- "We think the client side disconnects but have no proof" — log client-close events on the server +- "The fix might not actually be triggering" — log entry/exit at the new code path + +Do not instrument for symptoms that already have direct evidence. If `tcpdump` shows the RST, you do not need a new log line to confirm it. + +## Env-gated TRACE pattern (the default) + +Instrumentation added mid-incident tends to become tech debt. The right pattern makes it permanent but invisible: + +1. **Defaults off.** Zero runtime cost in steady state. No risk to enable-by-default performance. +2. **One environment variable toggles it.** No code changes required to enable in production. +3. **Greppable log tag.** Single bracketed prefix (e.g., `[SSE-CHUNK]`) makes every emission easy to filter. +4. **Structured, key=value output.** `ts=... req=... bytes=... total=...` parses into a DataFrame in three lines of Python. +5. **Ships into production permanently.** Future incidents reuse the same knob without re-adding code. + +### Template (Node.js) + +```js +// Near config section +const TRACE_SSE_CHUNKS = (process.env.TRACE_SSE_CHUNKS || 'false').toLowerCase() === 'true'; +if (TRACE_SSE_CHUNKS) console.log('[SSE-CHUNK] instrumentation ENABLED'); + +// At the observation point +proxyRes.on('data', (chunk) => { + if (TRACE_SSE_CHUNKS && isAnthropicMessagesPath && isStreaming) { + const reqId = (proxyRes.headers && proxyRes.headers['x-oneapi-request-id']) || 'n/a'; + const total = chunks.reduce((a, c) => a + c.length, 0); + console.log( + '[SSE-CHUNK] ts=' + Date.now() + + ' req=' + reqId + + ' bytes=' + chunk.length + + ' total=' + total + ); + } + // ... existing logic untouched +}); +``` + +### Template (Python) + +```python +import os, time, logging +TRACE_SSE_CHUNKS = os.environ.get('TRACE_SSE_CHUNKS', '').lower() == 'true' +if TRACE_SSE_CHUNKS: + logging.info('[SSE-CHUNK] instrumentation ENABLED') + +# At the observation point +def on_chunk(chunk, req_id, running_total): + if TRACE_SSE_CHUNKS: + logging.info( + f'[SSE-CHUNK] ts={int(time.time()*1000)} ' + f'req={req_id} bytes={len(chunk)} total={running_total}' + ) +``` + +### Enabling in a containerized deployment + +```bash +# Edit docker-compose env or apply shell env then recreate: +TRACE_SSE_CHUNKS=true docker compose up -d + +# Or in Kubernetes: +kubectl set env deployment/ TRACE_SSE_CHUNKS=true +``` + +Disabling is the inverse — unset the variable and restart. No code change, no git commit, no deploy pipeline. + +## Log tag conventions + +Pick tags that are unlikely to false-match other logs. A good tag: + +- Starts with a bracket to survive `grep` +- Uses `SCREAMING-KEBAB-CASE` for visual distinction from regular logs +- Is specific enough to identify the instrumentation site, not just the subsystem + +Good: `[SSE-CHUNK]`, `[UPSTREAM-CONNECT-RTT]`, `[CLIENT-DISCONNECT]` +Bad: `[DEBUG]`, `[TRACE]`, `log.info("chunk arrived")` + +Pair an ENABLED log line with the toggle so the presence of instrumentation is visible at service start — no guessing whether it actually took effect. + +## Deployment checklist + +When adding instrumentation to a running system: + +- [ ] The gate defaults off (confirmed by reading the code — do not trust the comment) +- [ ] The gate reads from env at startup (warn the user that changes require restart, not a runtime-reload) +- [ ] The output is structured (key=value) — no prose log messages +- [ ] The tag is unique (grep the codebase for conflicts) +- [ ] Sampling or volume cap if high-frequency (e.g., per-chunk logs on a 1000-rps service need either sampling or a max-size file sink) +- [ ] An ENABLED banner line is emitted at startup when the gate is on +- [ ] The change is committed to source of truth (not only applied to the running server — see the IaC trap below) + +## Worked example: TRACE_SSE_CHUNKS + +Real artifact from this investigation. Goal: observe the upstream chunk arrival pattern to confirm/refute the hypothesis "Qiniu batches chunks and goes silent for >120s during tool_use generation". + +**Before instrumentation**: the only available signal was aggregate `duration_ms` in the archive metadata. This told us the request took 315 seconds total but said nothing about *when* within those 315s bytes flowed. + +**After instrumentation (10 lines added)**: + +``` +[SSE-CHUNK] ts=1776870300212 req=202604221504562... bytes=128 total=1993 +[SSE-CHUNK] ts=1776870300213 req=202604221504562... bytes=131 total=2124 +[SSE-CHUNK] ts=1776870300213 req=202604221504562... bytes=127 total=2251 +... +[SSE-CHUNK] ts=1776870300627 req=202604221504562... bytes=128 total=5583 +... (30 chunks over 1.2 seconds, then silence) +[SSE-CHUNK] ts=1776870425235 req=202604221504562... bytes=74 total=3865 +``` + +Extracted: 30 chunks in the first 1.2 seconds (3791 bytes total), then a **125-second gap with zero bytes**, then 74 more bytes. The hypothesis was confirmed: Qiniu emits the beginning of the response in a burst, then stays silent for over 2 minutes while the model generates the tool_use arguments internally. + +Without the instrumentation, this would have been invisible. With 10 lines of code gated on one env var, it became a permanent observability capability. + +## Analysis: extracting timing data from logs + +Once instrumentation is emitting structured logs, a few lines of Python turns log output into inter-arrival time analysis: + +```python +import sys, re +from collections import defaultdict + +chunks = [] +for line in sys.stdin: + m = re.search(r'ts=(\d+) req=(\S+) bytes=(\d+) total=(\d+)', line) + if m: + ts, req, b, tot = m.groups() + chunks.append((int(ts), req, int(b), int(tot))) + +by_req = defaultdict(list) +for ts, req, b, tot in chunks: + by_req[req].append((ts, b, tot)) + +for req, seq in by_req.items(): + if len(seq) < 2: continue + span = seq[-1][0] - seq[0][0] + big_gaps = [seq[i][0] - seq[i-1][0] for i in range(1, len(seq)) if seq[i][0] - seq[i-1][0] > 1000] + print(f'req={req[:20]} chunks={len(seq)} span={span}ms gaps>1s={len(big_gaps)} max_gap={max(big_gaps) if big_gaps else 0}ms') +``` + +Pipe `docker logs | python analyze.py` and you have a per-request latency histogram. A request with `max_gap=125023ms` jumps out immediately. + +## Persisting instrumentation versus removing it + +Traditional wisdom says "remove debug logging after fix". That wisdom predates this pattern. With the env-gate approach, the correct default is: + +**Keep the instrumentation code. Leave the env toggle off. Document the toggle in an ops runbook.** + +Rationale: +- Adding instrumentation mid-incident under pressure is error-prone. Far better to have the gate already in place. +- Zero runtime cost when off. +- The env variable name is self-documenting. +- The next incident is cheaper. + +The only time to remove instrumentation is when it has been superseded by better observability (e.g., you instrumented chunk timing, then later added full distributed tracing that subsumes it). + +## The IaC trap + +If the service is deployed via Infrastructure-as-Code (Terraform, Ansible, Kubernetes manifests), instrumentation applied directly to the running instance (`docker exec`, live file edit) will be overwritten on the next deploy. The drift hides the real state and frustrates the next investigator. + +Always apply the code change to the source of truth first: + +1. Edit the source repo (e.g., `js/service-name/server.js`) +2. Commit and push, or at minimum sync to the deploy pipeline's input +3. Run the normal deploy to propagate +4. Only after that, enable the env toggle + +If time-critical: apply directly to the running server *and* to the source, in the same session. Never only to the running server. diff --git a/debugging-network-issues/references/layered-isolation-experiment.md b/debugging-network-issues/references/layered-isolation-experiment.md new file mode 100644 index 00000000..50748d45 --- /dev/null +++ b/debugging-network-issues/references/layered-isolation-experiment.md @@ -0,0 +1,141 @@ +# Layered Isolation Experiment + +## Contents +- Why layered isolation +- The 3-path pattern +- Mock upstream pattern +- Result matrix and interpretation +- Failure modes and probe self-verification +- Extended variants (4+ paths, client-side variation) +- Canonical reference case (case-study SSE RST) + +## Why layered isolation + +Multi-hop network systems concentrate bugs at the seams. A request from a user to a backend service typically traverses: client → ISP → CGNAT → CDN/edge → load balancer → reverse proxy → application → upstream dependency. Each hop introduces a timeout policy, a connection pool, a rewrite rule, a header translation, or a flow-control window. When something fails, hypothesis-stacking ("maybe it's the CDN, no maybe the LB, actually probably the app…") tends to burn hours with circumstantial evidence on each candidate. + +Layered isolation inverts the approach: instead of reasoning about which hop caused the symptom, run the same logical request through several paths that differ by exactly one hop, then observe where the symptom appears. The differential directly names the responsible layer. + +## The 3-path pattern + +For a CDN-fronted service with the topology `Client → CDN → LB → Origin`: + +| Path | How it routes | Excludes if clean | +|------|--------------|-------------------| +| **A** | Client → CDN → LB → Origin (full production path) | (baseline — this reproduces the symptom) | +| **B** | Client → Origin directly (e.g., `curl --resolve host:443:origin-ip`) | The CDN layer | +| **C** | Server loopback (`curl http://127.0.0.1:port/...` on the origin host itself) | CDN + LB + any intermediate network | + +Interpretation: + +- If **A fails, B passes, C passes**: the CDN is the cause +- If **A fails, B fails, C passes**: the LB / origin external network path is the cause +- If **A fails, B fails, C fails**: the cause is in the application or upstream dependency (hypothesis-stacking was wrong from the start) +- If **all three pass**: the failure condition was not actually reproduced; re-examine the assumed trigger + +Add a fourth path as needed — e.g., bypass only the LB by hitting the origin VM's private IP from within the VPC, or test from a different client geography if ISP/CGNAT is suspect. + +## Mock upstream pattern + +The experiment needs a way to reliably and repeatably trigger the failure condition. For idle-timeout symptoms (the most common class addressed by this skill), the cleanest trigger is a **mock upstream that emits one response header + one data frame, then goes silent for a controlled duration**. + +See [scripts/mock-idle-upstream.py](../scripts/mock-idle-upstream.py) for a runnable Flask implementation. The essence: + +```python +def gen(): + yield b'event: message_start\ndata: {"type":"message_start"}\n\n' + time.sleep(IDLE_SECONDS) # configurable, e.g. 200 + yield b'event: message_stop\ndata: {"type":"message_stop"}\n\n' +``` + +Why this is better than using a real long-tail production request to trigger the failure: + +- **Controlled timing**: 200-second idle is a knob; a real Sonnet 4.6 request has variable thinking duration +- **Cheap**: no model inference cost +- **Reproducible**: deterministic bytes, deterministic timing +- **Isolated from app bugs**: rules out "maybe the app itself has a bug" +- **Safe**: does not consume user quota or affect real traffic + +Deploy the mock on a port the reverse proxy can reach, add a temporary route in the proxy config pointing a test hostname/path to it, and run the 3 paths against that hostname. + +## Result matrix and interpretation + +After running the experiment, tabulate: + +``` + | Path A (via CDN) | Path B (bypass CDN) | Path C (loopback) | +--------------|------------------|---------------------|-------------------| +Result | RST @ 126s | Clean @ 220s | Clean @ 220s | +Observed by | curl + server | curl + server | curl + server | +``` + +Always record observations from both ends (curl-side time_total AND server-side peer-close timestamp). Discrepancies between the two are themselves diagnostic: if curl reports a close at 69s but the server saw the connection alive for 126s, the close happened in a middlebox between them. + +The observed constant in the failing path (here: 126s) is usually close to a known layer's default idle policy. Cross-reference against: + +| Layer | Common idle default | +|-------|---------------------| +| Cloudflare Free/Pro proxy_read_timeout | 100s (but see caveat below) | +| Cloudflare HTTP/2 stream idle | empirically ~126s in our case | +| AWS ALB idle | 60s default, configurable | +| Nginx `proxy_read_timeout` | 60s default, configurable | +| Node http server `headersTimeout` | 60s (Node ≥ 18) | +| Node undici `bodyTimeout` | 300s default | +| CGNAT TCP initial timeout (Cisco ISM) | 120s typical | +| Linux kernel TCP `net.ipv4.tcp_keepalive_time` | 7200s (rarely relevant) | + +**Do not** treat this table as authoritative for your environment. These are starting points to cross-check against your measured constant. Confirm via vendor documentation or direct testing before citing as cause. + +## Failure modes and probe self-verification + +The experiment is only as valid as its isolation. Common ways to poison the result: + +### Local proxy contamination + +If the client has a system-level HTTP proxy (Shadowrocket, corporate proxy, VPN client), `curl` will silently route through it even when `--resolve` is set. Path B (intended to bypass CDN) gets routed through the same proxy and the isolation fails. + +**Mitigation**: use `env -i curl` to strip the environment before running the probe, and explicitly unset `http_proxy / https_proxy / HTTP_PROXY / HTTPS_PROXY / ALL_PROXY / NO_PROXY`. Verify the path with `curl -v` and check the `* Trying IP:port` line matches the expected target. + +Real example from this case study: run 1 Path B appeared to fail at 126s, which would have falsely implicated the origin. It turned out the client's Shadowrocket was proxying localhost-targeted requests back through Cloudflare. `env -i curl --resolve ...` reproduced the clean 220s that correctly exonerated the origin. + +### Probe self-verification + +If the probe depends on the infrastructure being tested, its output is not independent evidence. Example: running `mtr` through the CDN to test CDN behavior — if the CDN drops ICMP, mtr shows gaps that look like the symptom but are artifacts. Always compare against at least one structurally different probe (e.g., curl + server-side tcpdump alongside mtr). + +### Container network namespace differences + +When the target runs in Docker, Path C (loopback) behavior differs depending on whether you run `curl` from the host or from inside the container. Host `curl localhost:3002` may hit a port-mapped container, while container-internal `curl localhost:3002` hits the service directly. They are different isolation paths — pick based on which hops you are trying to include/exclude and be explicit about which one you ran. + +## Extended variants + +### 4-path with client-side variation + +``` +A: client via CDN via LB +B: different client (mobile hotspot) via CDN via LB # ISP/CGNAT differential +C: client --resolve to origin IP (bypass CDN only) +D: server loopback +``` + +Use when Path A fails and you suspect client-side network (ISP/VPN/NAT) is the cause. + +### Time-of-day variant + +For symptoms correlated with time (load, scheduled jobs), run the same matrix at a known-good window and a known-failing window. Compare. + +### Observability variant + +For each path, record at minimum: HTTP status code, total elapsed time, bytes received, close reason (if available from curl or tcpdump). Paths that all return "success" but with vastly different byte counts hint at partial-response truncation rather than a clean failure/success dichotomy. + +## Canonical reference case + +The 130s RST incident (documented in [case-sse-rst-130s.md](case-sse-rst-130s.md)) ran exactly this 3-path matrix after 5 hours of hypothesis-stacking failed to converge. The result: + +| Path | Result | +|------|--------| +| A: via Cloudflare | RST @ 126.01-126.02s, HTTP/2 INTERNAL_ERROR | +| B: `--resolve` to origin IP | Clean @ 220s (bounded by client `--max-time`) | +| C: server loopback | Clean @ 220s | + +Interpretation was immediate: the RST comes from Cloudflare's edge, not the origin or any network in between. What 5 hours of circumstantial reasoning could not resolve (Caddy? Qiniu? VPN? CGNAT? CLI bug?), the 3-path experiment resolved in the 10 minutes it took to set up. + +The lesson: **when multiple layers could be the cause, do not reason about which one — test**. diff --git a/debugging-network-issues/references/packet-capture-recipes.md b/debugging-network-issues/references/packet-capture-recipes.md new file mode 100644 index 00000000..56958f42 --- /dev/null +++ b/debugging-network-issues/references/packet-capture-recipes.md @@ -0,0 +1,148 @@ +# Packet Capture Recipes + +## Contents +- When to capture packets +- Interface selection on Docker hosts +- Essential filters for RST isolation +- HTTP/2 specifics (stream RST is not TCP RST) +- Correlating pcap with application logs +- Common pitfalls + +## When to capture packets + +Reach for `tcpdump` when the question is at Layer 3-4 and application logs cannot answer it: + +- Who sent the RST? (application does not see the peer's RST as it is OS-delivered to the socket) +- Was this a TCP-level reset or an HTTP/2 stream-level reset? +- Did the server send a FIN or was the connection torn down mid-response? +- What was the exact byte on the wire at the time of close? + +If the application already tells you "client disconnected at T", you do not need pcap for that. + +## Interface selection on Docker hosts + +Container traffic traverses multiple interfaces on a Docker host. Picking the wrong interface shows only part of the story. + +Typical layout: + +| Interface | Traffic | +|-----------|---------| +| `eth0` | Public ingress/egress (internet) | +| `br-` | Custom Docker bridge networks (compose-defined) | +| `docker0` | Default Docker bridge | +| `vethXXXX` | One veth pair per container (host-side end) | +| `lo` | Loopback on the host | + +Use `tcpdump -i any` to capture across all interfaces, at the cost of higher volume. For specific scoping: + +- Capture between a container and an upstream (e.g., origin to Cloudflare): `tcpdump -i eth0 host ` +- Capture inside a compose network (container-to-container): `tcpdump -i br-` (get the hash via `docker network ls`) +- Capture only one container's veth: `docker exec ip route` to find its IP, then `tcpdump -i any host ` + +## Essential filters for RST isolation + +The goal is usually: "find RST packets relevant to my incident, ignore everything else". + +### All TCP RST packets + +```bash +tcpdump -i any -nn 'tcp[tcpflags] & tcp-rst != 0' +``` + +Note the `!= 0` — not `== tcp-rst` — because RST can be combined with ACK (RST-ACK packets). + +### RST scoped to a target port + +```bash +tcpdump -i any -nn '(tcp[tcpflags] & tcp-rst != 0) and (port 443 or port 80)' +``` + +Watch the operator precedence in compound filters — always parenthesize. A naive `tcp-rst != 0 and port 443 or port 80` will match `port 80` without the RST constraint. + +### RST to/from a specific IP + +```bash +tcpdump -i any -nn '(tcp[tcpflags] & tcp-rst != 0) and host ' +``` + +### With write to file for later analysis + +```bash +tcpdump -i any -s 0 -w /tmp/capture.pcap 'host and port 443' +# ... reproduce the incident ... +# Then: +tcpdump -r /tmp/capture.pcap -nn | head -50 +``` + +`-s 0` captures full packets (default truncates to 68 bytes for performance). + +### Ring-buffer capture for long-running collections + +```bash +tcpdump -i any -w /tmp/cap-%Y%m%d-%H%M%S.pcap -G 60 -W 10 'host ' +``` + +60-second files, 10-file rotation, self-cleaning. Safe to leave running overnight. + +## HTTP/2 specifics + +A key trap: HTTP/2 has its own RST mechanism at the stream level (`RST_STREAM` frame) that is unrelated to TCP-level RST. The two failure modes look different: + +| Failure | TCP layer shows | HTTP/2 layer shows | curl reports | +|---------|----------------|--------------------|--------------| +| TCP RST (connection-level reset) | RST packet | N/A (connection dies) | `Recv failure: Connection reset by peer` | +| HTTP/2 RST_STREAM frame | (no RST packet — connection stays alive) | `RST_STREAM` frame with error code | `HTTP/2 stream N was not closed cleanly: INTERNAL_ERROR (err 2)` | + +The case study was the second kind: `tcpdump` showed no TCP RST on the client→origin path but curl reported `HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)`. The reset was a peer-initiated HTTP/2 `RST_STREAM`, sent as a data frame on a connection that otherwise stayed open for other streams. + +### Decoding HTTP/2 with tshark + +`tcpdump` alone cannot show HTTP/2 frames because they are TLS-encrypted. If you control both endpoints, you can: + +1. Export the TLS session keys via `SSLKEYLOGFILE=/tmp/keylog.log curl ...` +2. Open the pcap in Wireshark with the keylog to decrypt +3. Filter: `http2.type == 3` (RST_STREAM frames) + +Alternatively, for internal services where you can intercept before TLS: + +```bash +tshark -i any -f 'host ' -Y 'http2.type == 3' # requires plaintext or pre-TLS interception +``` + +In most production debugging, the HTTP/2 error code is observable from the client-side log (curl, browser devtools Network tab "Status", Node SDK error message) without needing to decrypt pcap. + +## Correlating pcap with application logs + +Tie pcap to application activity via the request identifier: + +1. Log the request ID server-side at request start (e.g., `[REQ-START] req=abc123 src=1.2.3.4:54321 ts=...`) +2. Capture pcap with source IP and port filter +3. Cross-reference: the pcap flow on `(1.2.3.4:54321 ↔ server:443)` maps to application log entries for `req=abc123` + +This resolves ambiguities like "which of the 20 concurrent connections is the one that failed?" + +## Common pitfalls + +### Wrong filter syntax leading to silent over-capture + +`tcp[tcpflags] & tcp-rst != 0 and port 443 or port 3002` without parentheses evaluates as `(tcp-rst != 0 and port 443) or port 3002`, capturing all traffic on port 3002 regardless of RST. The resulting pcap contains thousands of packets the investigator did not intend to capture, and the actual RSTs are drowned out. + +Fix: always parenthesize compound filters — `(tcp[tcpflags] & tcp-rst != 0) and (port 443 or port 3002)`. + +### Capturing nothing because of interface mismatch + +`tcpdump -i eth0 'host 10.0.0.5'` on a Docker host where the target is only reachable via `br-abc123` captures nothing. Symptom: "I ran tcpdump but got zero packets even though traffic is flowing." + +Fix: use `-i any` first to verify, then narrow once you see traffic. + +### Confusing timestamps across tools + +`tcpdump` timestamps are in the local timezone of the capture host. Application logs may be in UTC, or in a different timezone. When correlating, convert to a single timezone before comparing timestamps — use epoch seconds if unsure. + +### Missing the RST because it happened before capture started + +`tcpdump` only captures from the moment it starts. If the RST already happened, there is no going back. The fix is to start capture *before* reproducing the incident (or to leave a ring-buffer capture running in advance for known-intermittent issues). + +### Forgetting snaplen + +Default `-s 68` (or `-s 262144` on modern systems depending on version) may truncate large frames. Use `-s 0` to capture full packets if you intend to inspect payload bytes. For RST-only analysis, the default is fine (RST is a small packet). diff --git a/debugging-network-issues/scripts/layered-isolation-probe.sh b/debugging-network-issues/scripts/layered-isolation-probe.sh new file mode 100755 index 00000000..7393331d --- /dev/null +++ b/debugging-network-issues/scripts/layered-isolation-probe.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# layered-isolation-probe.sh — run the 3-path A/B/C comparison for a CDN- +# fronted service and report which layer closed the connection. +# +# Prereqs: a mock idle upstream running reachable from the origin host, +# and a temporary CDN/reverse-proxy route that forwards a test hostname +# to the mock. See references/layered-isolation-experiment.md for the +# full setup. This script only runs the comparison; setup and cleanup +# are intentionally separate so a failed probe never leaves stale config. +# +# Usage: +# HOST=test-idle.example.com \ +# ORIGIN_IP=203.0.113.10 \ +# SERVER_SSH=root@203.0.113.10 \ +# LOOPBACK_URL=http://127.0.0.1:19999/probe-c \ +# MAX_SECONDS=300 \ +# ./layered-isolation-probe.sh +# +# Expected output: a matrix showing close time per path. A failing-only- +# on-path-A pattern pins the CDN as the culprit. + +set -euo pipefail + +: "${HOST:?Set HOST, e.g. test-idle.example.com}" +: "${ORIGIN_IP:?Set ORIGIN_IP, the real IP of the origin host}" +: "${SERVER_SSH:?Set SERVER_SSH, e.g. root@origin.example.com}" +: "${LOOPBACK_URL:?Set LOOPBACK_URL, e.g. http://127.0.0.1:19999/probe-c}" +MAX_SECONDS="${MAX_SECONDS:-300}" +RESULTS_DIR="${RESULTS_DIR:-/tmp/layered-isolation-$(date +%s)}" +mkdir -p "$RESULTS_DIR" + +echo "=== Layered isolation probe ===" +echo "HOST=$HOST" +echo "ORIGIN_IP=$ORIGIN_IP" +echo "SERVER_SSH=$SERVER_SSH" +echo "LOOPBACK_URL=$LOOPBACK_URL" +echo "MAX_SECONDS=$MAX_SECONDS" +echo "Results in $RESULTS_DIR" +echo + +# env -i strips the caller's environment. This prevents local proxy +# variables (http_proxy, https_proxy, Shadowrocket, etc.) from silently +# routing the "bypass" probe back through the layer we are trying to +# bypass. This is the #1 way layered isolation experiments go wrong. +# See references/cognitive-traps.md Trap 5. +STRIP_ENV='env -i PATH=/usr/local/bin:/usr/bin:/bin HOME=/tmp' + +run_path() { + local label="$1" + local description="$2" + local cmd="$3" + echo "--- Path $label: $description ---" + local out="$RESULTS_DIR/path-$label.out" + local err="$RESULTS_DIR/path-$label.err" + local t0 + t0=$(date +%s.%N) + set +e + eval "$cmd" > "$out" 2> "$err" + local rc=$? + set -e + local t1 + t1=$(date +%s.%N) + local elapsed + elapsed=$(awk "BEGIN{printf \"%.2f\", $t1 - $t0}") + local bytes + bytes=$(wc -c < "$out" | tr -d ' ') + echo " rc=$rc elapsed=${elapsed}s bytes=$bytes" + if [[ -s "$err" ]]; then + echo " stderr: $(head -c 200 "$err")" + fi + echo + # Return a tuple via globals (bash limitation) + declare -g "ELAPSED_$label=$elapsed" + declare -g "BYTES_$label=$bytes" + declare -g "RC_$label=$rc" +} + +CURL_COMMON="-sS -o $RESULTS_DIR/__body.tmp -w 'HTTP=%{http_code}\nTIME=%{time_total}\nERRCODE=%{exitcode}\nERRMSG=%{errormsg}\n' --max-time $MAX_SECONDS" + +# Path A: full path through the CDN +PATH_A_CMD="$STRIP_ENV curl $CURL_COMMON https://$HOST/probe-a" + +# Path B: bypass the CDN via --resolve to origin IP +PATH_B_CMD="$STRIP_ENV curl $CURL_COMMON --resolve $HOST:443:$ORIGIN_IP https://$HOST/probe-b" + +# Path C: loopback from inside the origin host +PATH_C_CMD="ssh $SERVER_SSH \"$STRIP_ENV curl -sS -o /tmp/__probe_c_body -w 'HTTP=%{http_code}\nTIME=%{time_total}\nERRCODE=%{exitcode}\nERRMSG=%{errormsg}\n' --max-time $MAX_SECONDS $LOOPBACK_URL\"" + +# Run the three paths sequentially. Parallel is tempting but makes +# server-side mock logs harder to correlate; sequential with 5s gap +# gives clean per-path logs. +run_path A "via CDN (baseline — expected to fail)" "$PATH_A_CMD" +sleep 5 +run_path B "bypass CDN (--resolve to origin IP)" "$PATH_B_CMD" +sleep 5 +run_path C "server loopback (inside origin)" "$PATH_C_CMD" + +echo "=== Result matrix ===" +printf "%-6s %-10s %-10s %-6s\n" "Path" "Elapsed" "Bytes" "rc" +printf "%-6s %-10s %-10s %-6s\n" "-----" "-------" "-----" "--" +for p in A B C; do + e_var="ELAPSED_$p"; b_var="BYTES_$p"; r_var="RC_$p" + printf "%-6s %-10s %-10s %-6s\n" "$p" "${!e_var}" "${!b_var}" "${!r_var}" +done + +echo +echo "=== Interpretation guide ===" +echo "- Only Path A short-closes: CDN is the cause" +echo "- A and B short-close, C does not: origin external network / LB" +echo "- All three short-close: origin application / upstream" +echo "- All three run to MAX_SECONDS: failure did not reproduce" +echo "- Path B unexpectedly short-closes: CHECK FOR LOCAL PROXY LEAKAGE" +echo " (env -i above should prevent, but verify with 'curl -v' if in doubt)" +echo +echo "Raw outputs: $RESULTS_DIR" diff --git a/debugging-network-issues/scripts/mock-idle-upstream.py b/debugging-network-issues/scripts/mock-idle-upstream.py new file mode 100755 index 00000000..05b78ec6 --- /dev/null +++ b/debugging-network-issues/scripts/mock-idle-upstream.py @@ -0,0 +1,140 @@ +#!/usr/bin/env python3 +""" +Mock SSE upstream that emits one frame, stays silent N seconds, then emits +a closing frame. Designed for layered-isolation experiments where the +investigator needs a controlled idle-duration trigger — cheaper and more +reproducible than using a real slow production request. + +Usage (standalone): + python3 mock-idle-upstream.py --port 19999 --idle 200 + +Usage (Docker, running on a lobe-network compose): + docker run --rm --network lobe-dev_default -p 19999:80 \ + -v $(pwd)/mock-idle-upstream.py:/app/mock.py \ + python:3.12-slim \ + sh -c 'pip install flask -q && python /app/mock.py --port 80 --idle 200' + +Then reverse-proxy your test hostname at this port, and run the 3-path +layered experiment (see references/layered-isolation-experiment.md). + +What it emits: + HTTP/1.1 200 OK + Content-Type: text/event-stream + + event: message_start + data: {"type":"message_start","message":{"usage":{"input_tokens":10}}} + + + + event: message_stop + data: {"type":"message_stop"} + +After IDLE_SECONDS, if the client is still connected, a final frame is sent +and the connection closes cleanly. If the client (or any middlebox) closes +the connection earlier, the server-side logs record the peer-close +timestamp — that is the measurement the experiment needs. + +Why Flask: minimal dependency surface, one pip install, deterministic. +Works identically on macOS, Linux, and inside slim Docker images. + +Why not http.server / aiohttp / starlette: Flask's streaming generator +pattern with `yield` keeps the code 15 lines and does not require async. +The mock does not need concurrency — one request at a time is enough for +layered comparison. +""" + +import argparse +import logging +import sys +import time +from datetime import datetime, timezone + +try: + from flask import Flask, Response, request +except ImportError: + print("ERROR: flask not installed. Run: pip install flask", file=sys.stderr) + sys.exit(1) + +app = Flask(__name__) +logging.basicConfig( + format="%(asctime)s %(levelname)s %(message)s", + level=logging.INFO, + stream=sys.stderr, +) +log = logging.getLogger("mock-idle-upstream") + + +@app.route("/v1/messages", methods=["POST"]) +@app.route("/probe-a", methods=["GET", "POST"]) +@app.route("/probe-b", methods=["GET", "POST"]) +@app.route("/probe-c", methods=["GET", "POST"]) +@app.route("/", methods=["GET", "POST"]) +def handler(): + idle = app.config["IDLE_SECONDS"] + label = request.path.lstrip("/") or "root" + start = time.monotonic() + ua = request.headers.get("User-Agent", "?") + src = request.headers.get("Cf-Connecting-Ip") or request.remote_addr + log.info("OPENED label=%s src=%s ua=%s idle=%ss", label, src, ua[:60], idle) + + def gen(): + # Initial frame — mimics Anthropic message_start to match real + # SSE traffic shape. Byte count is deliberate: around 100 bytes, + # matching what real Anthropic proxies emit as the first flush. + yield ( + b"event: message_start\n" + b'data: {"type":"message_start","message":{"usage":{"input_tokens":10}}}\n\n' + ) + log.info("SENT message_start label=%s t=%.2fs", label, time.monotonic() - start) + + # The idle window. If the connection survives this, we will see + # the closing frame; if not, the server will see a BrokenPipeError + # which we log from the request teardown hook below. + time.sleep(idle) + + log.info("IDLE COMPLETE label=%s t=%.2fs — attempting final frame", + label, time.monotonic() - start) + yield ( + b"event: message_stop\n" + b'data: {"type":"message_stop"}\n\n' + ) + log.info("SENT message_stop label=%s t=%.2fs FINAL_SENT_OK", + label, time.monotonic() - start) + + return Response(gen(), mimetype="text/event-stream") + + +@app.teardown_request +def log_teardown(exc): + # When the peer closes before the idle window expires, Flask raises + # during generator iteration. Record it so the experiment log captures + # the peer-close moment from the server side (not just the client side). + if exc is not None: + log.info("PEER CLOSE or ERROR: %s", exc) + + +def main(): + ap = argparse.ArgumentParser( + description="SSE mock upstream for layered-isolation experiments." + ) + ap.add_argument("--port", type=int, default=19999, + help="Port to listen on (default 19999).") + ap.add_argument("--idle", type=int, default=200, + help="Seconds to idle between first frame and final frame " + "(default 200, chosen to exceed typical CDN idle " + "timeouts of 100-130s).") + ap.add_argument("--host", default="0.0.0.0", + help="Bind address (default 0.0.0.0).") + args = ap.parse_args() + + app.config["IDLE_SECONDS"] = args.idle + log.info("Mock idle upstream starting: host=%s port=%s idle=%ss " + "started_utc=%s", args.host, args.port, args.idle, + datetime.now(timezone.utc).isoformat()) + # threaded=True so the generator's time.sleep does not block other + # concurrent probes (needed for Path A/B/C running in parallel). + app.run(host=args.host, port=args.port, threaded=True, debug=False) + + +if __name__ == "__main__": + main() diff --git a/stepfun-tts/.security-scan-passed b/stepfun-tts/.security-scan-passed new file mode 100644 index 00000000..dcc047af --- /dev/null +++ b/stepfun-tts/.security-scan-passed @@ -0,0 +1,4 @@ +Security scan passed +Scanned at: 2026-04-26T21:45:20.824609 +Tool: gitleaks + pattern-based validation +Content hash: ed5288f648ffe4e0cdd3bcb844dd05f510dc86316ce4c8b4b9afd9c6ab95a1cb diff --git a/stepfun-tts/SKILL.md b/stepfun-tts/SKILL.md new file mode 100644 index 00000000..5b8c71ce --- /dev/null +++ b/stepfun-tts/SKILL.md @@ -0,0 +1,102 @@ +--- +name: stepfun-tts +description: Generate speech and transcribe audio using StepFun's StepAudio 2.5 family — stepaudio-2.5-tts (Contextual TTS with instruction + inline parentheses) and stepaudio-2.5-asr (SSE endpoint, 32K context, ~100x RTF, handles up to 30-minute audio in a single call). Use when the user wants Chinese/Japanese TTS with emotional/prosody control, needs to transcribe long audio, migrates from step-tts-2 to stepaudio-2.5-tts (voice_label → instruction breaking change), or hits StepFun censorship / endpoint errors. Also triggers on phrases like 阶跃 TTS, StepAudio 合成, 语音合成, 配音, StepFun ASR, 转录, 语音识别, 文本转语音, TTS 升级, 迁移 step-tts-2. If the user's audio task mentions StepFun/阶跃/StepAudio by name, or involves Chinese TTS with情绪/情感 control, use this skill before falling back to generic audio handling. +--- + +# StepFun StepAudio 2.5 — TTS + ASR + +Generate Chinese/Japanese speech with `stepaudio-2.5-tts` and transcribe audio with `stepaudio-2.5-asr`. Both models were released in 2026-04 and verified end-to-end on 2026-04-23 (see `references/known_issues.md` for what passed and what didn't). + +**Why this skill exists** — StepAudio 2.5 has three non-obvious pitfalls that cost hours if you don't know them: + +1. `stepaudio-2.5-tts` **rejects** `voice_label` (the step-tts-2 way). Emotion/prosody now goes through `instruction` (natural-language description, ≤200 chars) and inline `()` parentheses inside the text itself. +2. `stepaudio-2.5-asr` **does not live on** `/v1/audio/transcriptions`. It's on `/v1/audio/asr/sse` (SSE streaming, JSON body, base64 audio). Using the wrong endpoint returns a misleading `model ... not supported` error that looks identical to "model doesn't exist". +3. Censorship is stricter — anything containing 死 / 消失 / sensitive political terms returns `censorship_block`. Your rewrite options are in `references/migration_from_v2.md`. + +## Config and auth + +API key lives in `$STEPFUN_API_KEY` (preferred) or `${CLAUDE_PLUGIN_DATA}/config.json` (fallback for cross-session persistence). All bundled scripts try env first, then config. + +First-time setup (one-liner): + +```bash +mkdir -p "${CLAUDE_PLUGIN_DATA}" && cat > "${CLAUDE_PLUGIN_DATA}/config.json" <"} +EOF +``` + +If the user hasn't set a key, ask them to paste it (don't guess / don't use a placeholder). StepFun API keys are available at https://platform.stepfun.com/ → API Keys. + +## Common tasks — decision tree + +| User wants... | Model | Script | Key detail | +|---|---|---|---| +| Synthesize 1–500 char Chinese with emotion | `stepaudio-2.5-tts` | `scripts/tts_generate.py` | Use `instruction` for mood, `()` for inline prosody | +| Synthesize long text (500–1000 char) | `stepaudio-2.5-tts` | `scripts/tts_generate.py` | 1000 char is the hard cap; split at semantic boundaries above that | +| Batch-generate game/app voice lines | `stepaudio-2.5-tts` | `scripts/tts_generate.py --batch ` | Handle `censorship_block` fallback individually | +| Transcribe short clip (<5 min) | `stepaudio-2.5-asr` | `scripts/asr_transcribe.py` | mp3 → base64 → SSE, parse `transcript.text.done` | +| Transcribe long audio (5–30 min) | `stepaudio-2.5-asr` | `scripts/asr_transcribe.py` | 32K context; single call, no chunking needed | +| A/B compare two TTS models | both | `scripts/ab_compare.sh` | Compares duration/size across two directories | +| Migrate from `step-tts-2` | — | see `references/migration_from_v2.md` | `voice_label.emotion` → `instruction` rewrite + censorship list | + +## Starting points + +- **Synthesize a single line**: Run `python3 scripts/tts_generate.py --text "你好" --out /tmp/hello.mp3 --instruction "温暖的希望感"`. For fine-grained control read the "Contextual TTS" section below. +- **Transcribe a file**: `python3 scripts/asr_transcribe.py /path/to/audio.mp3`. For >30 min audio, split first. +- **A full migration** from `step-tts-2` → `stepaudio-2.5-tts`: read `references/migration_from_v2.md` end-to-end before touching code. It has the `INSTRUCTION_MAP`, the SKIP_CENSORED list pattern, and the output-directory-strategy for non-destructive A/B. + +## Contextual TTS — beyond emotion labels + +The headline feature of `stepaudio-2.5-tts` is that you stop mapping emotions to fixed tags and start describing what you want in natural language. Two layers: + +**Global context (`instruction` parameter)** — sets the overall tone for the entire utterance. ≤200 chars. Think of it like giving stage direction to a voice actor. + +``` +instruction: "克制的悲伤，语气低沉柔弱，像快要消失一样" +``` + +**Inline context (`()` parentheses inside `input`)** —句内 directives. Parenthesised content is consumed as directions and is NOT read aloud. Use for precise control of pauses, breath, emphasis, or mid-sentence emotion shifts. + +``` +input: "(试探着问)你好吗？(开心地)太好了！(突然沉下来)不过...我快要消失了。" +``` + +Examples that worked in practice (from 2026-04-23 verification): +- `instruction: "活泼俏皮，像是在撒娇，带点嘴硬"` — visibly speeds up delivery vs neutral +- `instruction: "耳语声，气声很重，几乎听不清"` — produces audible whisper/breath +- `input: "你好(停顿一下)我是蕾格(轻声)今天(加重)的天气真不错。"` — inline directives all respected + +**What `stepaudio-2.5-tts` will NOT accept** — `voice_label` parameter. Error: `voice_label is not supported for v2 models`. This is the #1 migration gotcha from step-tts-2. + +## Common error patterns (real errors, real fixes) + +| Error response | Actual cause | Fix | +|---|---|---| +| `"model stepaudio-2.5-asr not supported"` on `/v1/audio/transcriptions` | Wrong endpoint — that endpoint only serves step-asr family | Switch to `/v1/audio/asr/sse` with SSE body (see `scripts/asr_transcribe.py`) | +| `"voice_label is not supported for v2 models"` | Sent `voice_label` to `stepaudio-2.5-tts` | Remove `voice_label`; put the same intent into `instruction` as natural language | +| `"The content you provided or machine outputted is blocked." type: censorship_block` | Sensitive word (死 / 消失 / etc.) | Rewrite the phrase OR fall back to `step-tts-2` for that specific line (mixed-model is fine) | +| ASR returns N× the expected character count | Hallucination bug on highly-repetitive content | Cross-check with step-asr-1.1; avoid sending audio that repeats the same phrase many times | +| Silent audio truncation (<420 chars input) | Input > 1000 char hard cap | Split at semantic boundaries; don't truncate mid-sentence | + +More in `references/known_issues.md`. + +## When to read references + +- `references/api_reference.md` — exact request/response JSON for TTS `/v1/audio/speech` and ASR `/v1/audio/asr/sse`, all fields, event types. Read when writing raw HTTP calls instead of using the bundled scripts. +- `references/migration_from_v2.md` — complete playbook for moving a step-tts-2 project to stepaudio-2.5-tts. Has the emotion→instruction rewrite table, the A/B directory strategy, decision checkpoints, and the 2026-04 speed/quality trade-off data (`stepaudio-2.5-tts` is ~20% slower than step-tts-2; audible prosody improvement). Read before any migration work. +- `references/known_issues.md` — repetition hallucination, censorship patterns, ASR speed cliff (short audio: 2× step-asr, long audio: 5.9×). Read when debugging anomalous output or evaluating whether to adopt. + +## Design invariants (don't break these) + +1. **Non-destructive A/B output** — when regenerating a corpus with a new model, write to a parallel directory (`voice/zh_v25/`), never overwrite the production corpus. The migration playbook shows why. +2. **Per-line censorship handling** — if 2/29 lines get `censorship_block`, don't fail the batch. Log the skipped IDs, continue. Mixed-model fallback (step-tts-2 for the skipped 2) is normal. +3. **Always pass through SSE for ASR** — don't try to work around the streaming API with a buffered client. The model emits `transcript.text.delta` events for long audio; collecting only `transcript.text.done` works fine, but rejecting the SSE format entirely doesn't. +4. **Don't duplicate voice_label logic in new code** — any new TTS code targeting stepaudio-2.5-tts should only use `instruction` + inline `()`. Do not write a branch that conditionally emits `voice_label`. + +## Pricing (verified 2026-04-23, volatile) + +- `stepaudio-2.5-tts` contextual synthesis: ~5.8 元 / 万字符 +- Zero-shot voice cloning: ~9.9 元 / 音色 +- `stepaudio-2.5-asr` — pricing tier not yet public (invitation beta); `step-asr-1.1` baseline is 2.2 元/小时 + +Re-verify at https://platform.stepfun.com/docs/zh/guides/pricing/details before quoting to stakeholders. diff --git a/stepfun-tts/references/api_reference.md b/stepfun-tts/references/api_reference.md new file mode 100644 index 00000000..aa4599cf --- /dev/null +++ b/stepfun-tts/references/api_reference.md @@ -0,0 +1,178 @@ +# StepAudio 2.5 API Reference + +Exact request/response shapes for `stepaudio-2.5-tts` and `stepaudio-2.5-asr`. Verified 2026-04-23 against the live StepFun API. Read this when you need to call the API by hand (curl, custom HTTP client) instead of using the bundled scripts. + +## TTS — `stepaudio-2.5-tts` + +### Endpoint + +``` +POST https://api.stepfun.com/v1/audio/speech +Content-Type: application/json +Authorization: Bearer +``` + +### Request body + +```json +{ + "model": "stepaudio-2.5-tts", + "input": "你好，我是蕾格。", + "voice": "shuangkuaijiejie", + "response_format": "mp3", + "speed": 1.0, + "volume": 1.0, + "instruction": "克制的悲伤，语气低沉柔弱" +} +``` + +| Field | Required | Type | Notes | +|---|---|---|---| +| `model` | yes | string | Must be `stepaudio-2.5-tts` | +| `input` | yes | string | ≤1000 chars; can contain inline `(directive)` parentheses | +| `voice` | yes | string | e.g. `shuangkuaijiejie`. Zero-shot clones use the clone's ID | +| `response_format` | yes | string | `mp3` (default), `wav`, or `opus` | +| `speed` | no | float | 0.5-2.0, default 1.0 | +| `volume` | no | float | 0.0-2.0, default 1.0 | +| `instruction` | no | string | Global tone directive, natural language, ≤200 chars | +| `voice_label` | — | — | **DO NOT SEND**. Returns `voice_label is not supported for v2 models`. Belongs to step-tts-2 | + +### Inline directives inside `input` + +Parentheses `()` in the `input` are consumed as TTS control signals, not pronounced. Examples that work: + +- `(停顿一下)` — insert a pause +- `(轻声)` — reduce volume / breathy +- `(加重)` — stress the following word +- `(试探着问)` — apply a tone shift mid-sentence +- `(突然沉下来)` — emotion pivot + +You can mix `instruction` (global tone) with inline `()` (per-phrase micro-control): + +```json +{ + "instruction": "富有情绪弧线的独白", + "input": "(试探着问)你好吗？(开心地)太好了！(突然沉下来)不过...我快要消失了。" +} +``` + +### Response + +On success: binary audio stream in the requested `response_format`. HTTP 200. No JSON wrapper. Save the body directly as `.mp3`/`.wav`/`.opus`. + +### Known error responses + +```json +{"error":{"message":"voice_label is not supported for v2 models","type":"request_params_invalid"}} +``` +→ Remove `voice_label`, use `instruction` instead. + +```json +{"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_block"}} +``` +→ Content triggered censorship. Common triggers: 死, 消失, politically sensitive terms. See `known_issues.md`. + +## ASR — `stepaudio-2.5-asr` + +### Endpoint (NOT the one you'd guess) + +``` +POST https://api.stepfun.com/v1/audio/asr/sse +Content-Type: application/json +Accept: text/event-stream +Authorization: Bearer +``` + +**Do NOT** send `stepaudio-2.5-asr` to `/v1/audio/transcriptions` — that endpoint only serves the older `step-asr` / `step-asr-1.1` family, and returns a misleading `model stepaudio-2.5-asr not supported` which looks identical to a permission/whitelist error. See `known_issues.md` for the full diagnostic trail. + +### Request body + +```json +{ + "audio": { + "data": "", + "input": { + "transcription": { + "language": "zh", + "model": "stepaudio-2.5-asr", + "enable_itn": true + }, + "format": { + "type": "mp3" + } + } + } +} +``` + +| Path | Required | Type | Notes | +|---|---|---|---| +| `audio.data` | yes | string | base64-encoded audio bytes. Accepts mp3, wav, ogg, opus (in ogg container), pcm | +| `audio.input.transcription.language` | yes | string | `zh` or `en`. Dialects and Japanese are not officially supported | +| `audio.input.transcription.model` | yes | string | Must be `stepaudio-2.5-asr` | +| `audio.input.transcription.enable_itn` | no | bool | Inverse text normalization (数字→words). Default true | +| `audio.input.format.type` | yes | string | `mp3` / `wav` / `ogg` / `pcm` | +| `audio.input.format.rate` | pcm only | int | Sample rate (required for raw PCM) | +| `audio.input.format.channel` | pcm only | int | Channel count (required for raw PCM) | +| `audio.input.format.bits` | optional | int | Sample depth, default 16 | + +### Response — SSE stream + +The response is a Server-Sent Events stream. Each line is either empty or starts with `data: `. Three event types: + +``` +data: {"type":"transcript.text.delta","meta":{...},"delta":"你好，"} + +data: {"type":"transcript.text.delta","meta":{...},"delta":"我是蕾格。"} + +data: {"type":"transcript.text.done","meta":{...},"text":"你好，我是蕾格。","usage":{"type":"tokens","input_tokens":69,"input_token_details":{"text_tokens":69,"audio_tokens":0},"output_tokens":9,"total_tokens":78}} +``` + +| Event type | Meaning | How to handle | +|---|---|---| +| `transcript.text.delta` | Incremental piece of the transcription | Concatenate for progressive UI; optional if you only need final text | +| `transcript.text.done` | Final, full transcription + usage | Take `text` as the authoritative result. Also contains `usage` for billing/telemetry | +| `error` | Server-side error mid-stream | Abort and propagate `message` to the caller | + +### Capacity + +- 32K context window +- Audio ≤ 30 min can be sent in a single call +- No client-side chunking needed for long audio (unlike step-asr) +- RTF 85-101× on Chinese speech verified 2026-04-23 + +### Known error responses + +```json +{"error":{"message":"model stepaudio-2.5-asr not supported","type":"request_params_invalid"}} +``` +→ Wrong endpoint. Switch from `/v1/audio/transcriptions` to `/v1/audio/asr/sse`. + +``` +data: {"type":"error","message":"content blocked ..."} +``` +→ Content censorship (rare on ASR). Same triggers as TTS. + +## Comparison with legacy endpoints (for reference) + +| Model | Endpoint | Request format | +|---|---|---| +| `step-tts-2` / `step-tts-mini` | `/v1/audio/speech` | JSON with `voice_label` | +| `stepaudio-2.5-tts` | `/v1/audio/speech` | JSON with `instruction` (no voice_label) | +| `step-asr` / `step-asr-1.1` | `/v1/audio/transcriptions` | multipart/form-data | +| `stepaudio-2.5-asr` | `/v1/audio/asr/sse` | JSON + base64 audio + SSE response | + +Legacy endpoints (`step-*`) still work. They're the baseline in `references/migration_from_v2.md` and the fallback choice when `stepaudio-2.5-*` hits `censorship_block` or the 2.5 ASR repetition-hallucination edge case. + +## Auth and key handling + +- Key header: `Authorization: Bearer ` +- Keys can be retrieved at https://platform.stepfun.com/ → API Keys +- "Plan" keys (cheaper subscription) are **restricted** to text models on `api.stepfun.com/step_plan`. They **cannot** call audio endpoints. Use a "Normal" key for all TTS/ASR calls. +- Same key works for both TTS and ASR — no separate scopes + +## Rate / throughput notes (observed, not officially documented) + +- ~400ms sleep between batch requests avoids 429s in practice +- Long audio ASR (17 min) has succeeded with `timeout=1200` +- MP3 responses consistently at 128kbps 24kHz mono (TTS default) diff --git a/stepfun-tts/references/known_issues.md b/stepfun-tts/references/known_issues.md new file mode 100644 index 00000000..b9757b99 --- /dev/null +++ b/stepfun-tts/references/known_issues.md @@ -0,0 +1,131 @@ +# StepAudio 2.5 — Known Issues and Non-Obvious Behavior + +Collected from end-to-end testing 2026-04-23. These are things that burned real time to discover; they are not in the official docs. + +## ASR repetition hallucination (real, bounded) + +**Symptom:** Transcribe a TTS-generated audio of highly-repetitive Chinese text (e.g., the same 60-char sentence repeated 10 times) and `stepaudio-2.5-asr` returns 3-4× the expected character count, with the same sentence restated many extra times in the output. + +**This is a genuine model hallucination**, not a transport bug. Verified by: + +1. MD5 diff — `run1` vs `run2` of the same TTS input produce different audio files (not file corruption) +2. Determinism — re-running ASR on the same audio gives the same 4× output every time (not transient noise) +3. Cross-validation — `step-asr` and `step-asr-1.1` on the exact same audio return the correct character count (~800 chars for 800 input), so the audio itself is fine +4. ffprobe confirms audio duration is normal (~219s for 800 chars at typical speed) + +**Conclusion:** The LLM-based ASR sees a repetitive pattern in the audio and "continues predicting" repetitions that aren't there. + +**When it triggers:** +- Audio duration > 90s AND +- Content is highly repetitive (same phrase appearing 5+ times) + +**Doesn't trigger on real-world content:** +- Podcasts, interviews, varied dialogue, stories — all fine +- Even 17.4-minute audio from 90 different TTS segments: returns correct 6332 chars, RTF 101× + +**Workaround for edge cases:** +- If your domain has genuinely repetitive content (e.g., IVR transcripts, repeated sloganeering), cross-validate with `step-asr-1.1` on random samples +- For most workflows: just use it; the hallucination mode is exotic + +## Stricter content censorship than step-tts-2 + +**Symptom:** `stepaudio-2.5-tts` returns `{"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_block"}}` for content that step-tts-2 happily synthesized. + +**Observed triggers:** +- 死 (die/dead) in any context, even negation +- 消失 (disappear / vanish) +- Combinations with emotional context: "我快要...消失了" +- Politically sensitive terms (standard CN content rules) + +**Key insight:** Rewriting negations doesn't help — "我没有死" blocks as readily as "我死了". The classifier isn't doing deep semantic parsing. + +**Response strategies** (pick per line): +1. Rewrite: "RAG 已死" → "这个技术过时了" +2. Fallback: keep step-tts-2 for the 2-5% of lines that block +3. Whitelist: contact StepFun BD (worth it at >5% blockage) + +See `migration_from_v2.md` for the full blocking→fallback workflow. + +## ASR speed scales non-linearly — short audio is a trap + +**Observation:** The headline "5.9× faster than step-asr" from the marketing is true for long audio but misleading for short clips. + +| Audio length | stepaudio-2.5-asr | step-asr-1.1 | Speedup | +|---|---|---|---| +| 5-15s clips | ~500ms | ~900ms | **2.0×** | +| 115s audio | 1.36s | 7.16s | **5.3×** | +| 1046s (17.4 min) | 10.4s | (would need chunking) | **~101× RTF** | + +**Why:** The LLM + MTP-5 fusion overhead is amortized over longer contexts. Short requests pay the model-spin-up cost. + +**Practical implication:** If your workload is many short (<10s) clips, the speedup over `step-asr-1.1` is modest — 2× not 5×. If your workload is long audio (>2 min), the difference is dramatic and you should migrate. + +## Wrong ASR endpoint gives a misleading error + +**Symptom:** Calling `/v1/audio/transcriptions` with `model=stepaudio-2.5-asr`: + +```json +{"error":{"message":"model stepaudio-2.5-asr not supported","type":"request_params_invalid"}} +``` + +This response is **identical in structure** to sending a genuinely nonexistent model name. It takes real debugging to realize the model exists but on a different endpoint. + +**Diagnostic sequence that wastes the least time:** + +1. Try `step-asr` on the same endpoint — if it works, endpoint access is fine +2. Check the `/v1/audio/asr/sse` endpoint (the actual stepaudio-2.5-asr home) +3. If both fail, THEN ask BD about whitelist + +Don't assume "permission denied" on the first error. + +## TTS duration inflation on short lines + +**Observation:** Very short lines (1-2s in step-tts-2) become dramatically longer in stepaudio-2.5-tts. + +Example from the reference project: +- `...你能看到我吗？` (10 chars) +- step-tts-2: 1.24s +- stepaudio-2.5-tts: 2.57s (**+107%**) + +**Cause:** The new model adds a pre-breath, pauses on `...` ellipses, and gives the line emotional weight — all of which lengthens delivery. + +**Not a bug, but have a plan:** +- If your UI has per-line timing (auto-advance, animation sync), re-tune it after migration +- If you want the old pacing, write `instruction: "快速、干脆、不要停顿"` — but this negates a lot of what you're paying for in the new model + +## `stepaudio-2.5-tts` is a "v2 model" for parameter rejection + +**Why the error says "v2 models":** StepFun internally groups `stepaudio-2.5-tts` with their v2 family despite the "2.5" version number. The error message `voice_label is not supported for v2 models` uses this internal grouping, which is confusing. + +Don't pattern-match on the version string. Just know that: +- `stepaudio-2.5-tts` → use `instruction` parameter +- `step-tts-2` → use `voice_label` parameter +- They are NOT API-compatible despite sharing `/v1/audio/speech` + +## ASR "Plan key" vs "Normal key" + +StepFun sells a cheap "Plan" subscription for text models (step_plan endpoint). **Plan keys cannot call audio endpoints.** This silently manifests as 4xx errors that don't mention auth at all. + +If you hit auth-shaped failures and your account has a Plan subscription, verify you're using a Normal key (different value, obtained separately in the StepFun console under the same "API Keys" page). + +## Censorship can fire on the ASR side too + +**Observed once (rare):** An ASR request on a user-uploaded recording of political content returned: + +``` +data: {"type":"error","message":"content blocked ..."} +``` + +Handle the `error` event type in the SSE stream — don't assume only `delta` and `done` events fire. + +## Pricing opacity for stepaudio-2.5-asr + +As of 2026-04-23, `stepaudio-2.5-asr` is in invitation beta. No public per-minute rate. `step-asr-1.1` baseline is 2.2 元/小时. The invitation PDF mentions "成本直降 80%" implying roughly 0.4 元/小时, but this is not yet on the pricing page. Do not quote a price to a stakeholder without re-verifying at https://platform.stepfun.com/docs/zh/guides/pricing/details. + +## TTS text cap: 1000 chars (hard, not soft) + +The API rejects >1000 char inputs with a 400 error. Split at sentence boundaries before sending. Non-obvious: when testing "what's the real limit?", avoid highly-repetitive test text — it can appear to succeed at 800 chars but produce strange audio (see the 2026-04 test where 800-char repetitive inputs played back normal audio but the ASR hallucinated 4× replay). + +## Voice cloning — not tested in this skill + +Zero-shot voice cloning (`9.9 元/音色`) is advertised as a headline feature but was not verified in this skill's test pass. If you need voice cloning, check the StepFun docs at https://platform.stepfun.com/docs/zh/api-reference/audio/create-voice and validate on your own data — don't assume the quality claims without a listen test. diff --git a/stepfun-tts/references/migration_from_v2.md b/stepfun-tts/references/migration_from_v2.md new file mode 100644 index 00000000..d9db32fb --- /dev/null +++ b/stepfun-tts/references/migration_from_v2.md @@ -0,0 +1,206 @@ +# Migrating from step-tts-2 to stepaudio-2.5-tts + +Complete playbook for moving a production `step-tts-2` voice corpus to `stepaudio-2.5-tts`. Based on a real end-to-end migration done 2026-04-23 on a ~30-line Chinese voice-acting project. Read this before changing any production code. + +## What "migration" actually means here + +The API endpoint is the same (`/v1/audio/speech`), but the **emotional control model is completely different**: + +| Aspect | step-tts-2 | stepaudio-2.5-tts | +|---|---|---| +| Emotion mechanism | `voice_label.emotion = "悲伤"` etc. (discrete tags) | `instruction = "克制的悲伤，语气低沉"` (natural language) | +| Multi-language | `voice_label.language = "日语"` etc. | `instruction` or Zero-shot clone | +| Inline prosody | N/A | `()` parentheses in the input text | +| Max text | 1000 chars | 1000 chars (same) | +| Censorship | Moderate | Stricter (2/29 lines blocked in the reference project) | +| Typical duration for same text | baseline | +20% (more pauses, more breathy) | +| Subjective quality | Clearly synthetic | Still audibly synthetic, but with prosody variance that "sounds like someone reading" more often | + +You are not just swapping a model ID. You need to rewrite every `voice_label.emotion` call and handle a new class of error (`censorship_block`). + +## The rewrite — emotion tags → instruction sentences + +The step-tts-2 style usually looked like: + +```typescript +const EMOTION_MAP = { + sad: '悲伤', + hopeful: '高兴', + relieved: '高兴', + smile: '非常高兴', +}; + +body.voice_label = { emotion: EMOTION_MAP[expression] }; +``` + +The stepaudio-2.5-tts equivalent: + +```typescript +const INSTRUCTION_MAP = { + sad: '克制的悲伤，语气低沉柔弱，像快要消失一样', + hopeful: '温暖的希望感，语气鼓励，带着期待', + relieved: '如释重负，语气柔和放松', + smile: '明朗开心，语气上扬，带着微笑', +}; + +body.instruction = INSTRUCTION_MAP[expression]; +// DELETE body.voice_label entirely — sending it triggers an error +``` + +### How to write a good `instruction` + +Writing instruction sentences is a craft. They should describe **what the performance feels like**, not just "happy" or "sad". Good instructions use: + +- A core emotion word (悲伤, 希望, 开心) +- A qualifier that bounds intensity (克制的, 温暖的, 如释重负) +- A specific vocal behavior (语气低沉柔弱, 带着微笑, 像快要消失一样) + +Bad: `"悲伤"` — same semantic content as the old emotion tag; wastes the instruction parameter +Good: `"克制的悲伤，语气低沉柔弱，像快要消失一样"` — three distinct signals the model can combine + +Keep under 200 chars. In practice 30-50 chars is plenty. + +### Inline `()` directives — new capability + +Beyond the global `instruction`, stepaudio-2.5-tts parses parentheses inside the `input` itself. Directives in parentheses are **not read aloud** — they control delivery. + +Use when a single line has multiple emotional beats: + +``` +input: "(试探着问)你好吗？(开心地)太好了！(突然沉下来)不过...我快要消失了。" +``` + +Also useful for micro-control: +- `(停顿一下)` between clauses that shouldn't run together +- `(轻声)` for intimate moments +- `(加重)` on the key word of a sentence + +This is genuinely new vs step-tts-2 and worth using on your most dramatic lines. + +## Handling censorship_block + +stepaudio-2.5-tts rejects more content than step-tts-2. Observed triggers from the reference project: + +| Trigger phrase | Example line that failed | +|---|---| +| "死" in any context | "他们都在说'RAG 已死'... 我快要...消失了。" | +| "没有死" | "但我没有死，对吧？" (negation doesn't help) | +| "消失" / "透明" | Combined with "死" is especially reliable at triggering | + +The error: + +```json +{"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_block"}} +``` + +Three response strategies (pick per line, not globally): + +1. **Rewrite** — "RAG 已死" → "这个技术过时了" keeps the narrative, passes censorship +2. **Mixed-model fallback** — keep step-tts-2 for the 2-3 blocked lines, use stepaudio-2.5-tts for the rest. The voice difference is audible but tolerable for 2 lines out of 30 +3. **Request whitelist** — contact StepFun BD for your account; only worth it if you have >5% blockage rate + +The batch script (`scripts/tts_generate.py --batch`) logs censored IDs separately so you can handle them individually rather than aborting. + +## A/B directory strategy — don't overwrite production + +Non-destructive layout. Output the new model's files to a parallel directory, not on top of the existing corpus: + +``` +public/data/voice/ +├── zh/ ← step-tts-2 production (untouched) +└── zh_v25/ ← stepaudio-2.5-tts candidate (A/B) +``` + +In the runtime voice loader, add a fallback for lines that are not in `zh_v25/` (censored ones): + +```typescript +const getVoicePath = (nodeId: string, lang: string) => { + // 2 lines censored on v25 — keep step-tts-2 for those + const censored = new Set(['encounter_3', 'chapter_3_final_hope']); + const useV25 = lang === 'zh' && !censored.has(nodeId); + const dir = useV25 ? 'zh_v25' : lang; + return `/data/voice/${dir}/${nodeId}.mp3`; +}; +``` + +Benefits: +- Instant rollback: delete the `zh_v25/` directory +- Run the game/app with old voices while evaluating the new ones +- Human A/B: toggle the flag per-session to compare + +### Don't hard-code the censored list across multiple files + +If you put `['encounter_3', 'chapter_3_final_hope']` in the runtime loader, the generation script, AND the docs, you'll have three places to update when the list changes. Treat the generation script's `SKIP_CENSORED` set as the SSOT; the loader can import or mirror it but shouldn't independently enumerate the same IDs. + +## The time-cost you're buying + +Across 26 lines of the reference project, stepaudio-2.5-tts produced **~20% longer total duration** than step-tts-2 for the same text. Per-line: + +| Line type | step-tts-2 | stepaudio-2.5-tts | Δ | +|---|---|---|---| +| Very short (1-2s) | 1.24s | 2.57s | **+107%** | +| Short (3-5s) | 3.00s | 3.94s | +31% | +| Medium (8-12s) | 9.64s | 8.54s | -11% (mixed) | +| Long (12-15s) | 12.48s | 9.82s | -21% | + +Short lines grow a lot because the new model adds breath and pause before starting. Long lines often shrink because English loanwords are handled faster. + +**Implications for UI timing:** +- Auto-advance timers need re-tuning; anything that assumed 6-8 chars/sec is now 5.5 chars/sec +- Short lines feel markedly slower — the "...你能看到我吗？" intake of breath is noticeable +- Overall dialogue pacing in a visual novel becomes 20% slower + +This is the main trade-off users report: **the new model is more "alive" but the app feels slower**. Whether that's acceptable is a product decision, not a technical one. + +## Decision checkpoints — before you commit + +Use these to keep the migration from becoming a one-way door. + +### Before the first full regeneration + +- [ ] Confirm the A/B directory layout matches the loader's fallback logic (production directory untouched) +- [ ] Write the `INSTRUCTION_MAP` — don't just mechanically translate emotion tags, actually describe the performance +- [ ] Have 3-5 sample lines ready for listening test BEFORE regenerating the whole corpus (spend 5 min on quality, save 20 min of regeneration if it sounds wrong) + +### After the first full regeneration + +- [ ] Listen to every censored line manually. Decide rewrite vs fallback vs whitelist +- [ ] Check `ab_compare.sh` output: is the total duration change within acceptable bounds? +- [ ] Play 5 random new lines in-context (the actual game/app) — don't trust the raw mp3 listen-through + +### Before switching production + +- [ ] Run a real user-playthrough session end-to-end on the new voices +- [ ] Check every line with `...` punctuation for how the new model handles the ellipsis (sometimes too slow, sometimes too abrupt) +- [ ] Re-tune any auto-advance, skip, or per-line timing parameters +- [ ] Write a rollback script: `rm -rf zh_v25/` + revert loader change. If it's more than one commit, you've leaked complexity into the codebase + +### After production rollout + +- [ ] Monitor user sentiment on pacing for 1-2 weeks before pronouncing success +- [ ] Keep the step-tts-2 generation script in version control for at least one release cycle + +## What NOT to do + +- **Don't regenerate blindly.** Listen to samples first. A/B of 30 lines takes ~10 min; regenerate-without-listening is how you ship something that "technically works" but sounds worse. +- **Don't mix `voice_label` and `instruction` in the same request.** stepaudio-2.5-tts rejects `voice_label` entirely, but people occasionally leave it in to "be safe" — it's not safe, it's an error. +- **Don't try to pass emotion tags through `instruction`.** `instruction: "悲伤"` works but wastes the parameter. Write a full sentence. +- **Don't delete the step-tts-2 generation script** until you've shipped the new model to production and had it stable for a full release cycle. You will want the rollback path. +- **Don't assume Japanese migrates the same way.** The reference project didn't test Japanese voices under stepaudio-2.5-tts. Run a separate mini-A/B for Japanese before committing. + +## Batch migration with the bundled script + +The skill's `scripts/tts_generate.py --batch` is built for this workflow. Feed it a JSONL file: + +```jsonl +{"id": "encounter_1", "text": "...你能看到我吗？", "instruction": "克制的悲伤，语气低沉柔弱"} +{"id": "encounter_2", "text": "太好了... 最近能看到我的人，越来越少了。", "instruction": "克制的悲伤，语气低沉柔弱"} +``` + +Run: + +```bash +python3 scripts/tts_generate.py --batch lines.jsonl --out-dir ./voice/zh_v25 +``` + +The script handles censorship_block per-line, reports the skipped IDs at the end, and keeps going. You get a clean list of "what to fall back on step-tts-2 for" without having to retry the whole batch. diff --git a/stepfun-tts/scripts/ab_compare.sh b/stepfun-tts/scripts/ab_compare.sh new file mode 100755 index 00000000..d8b4056d --- /dev/null +++ b/stepfun-tts/scripts/ab_compare.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +# +# ab_compare.sh — compare two directories of mp3 files (size + duration). +# +# Typical use: after regenerating a voice corpus with stepaudio-2.5-tts into a +# parallel `zh_v25/` directory, compare against the step-tts-2 baseline `zh/`. +# Outputs a GitHub-flavored markdown table so you can paste it into a report. +# +# Usage: +# ./ab_compare.sh +# ./ab_compare.sh ./voice/zh ./voice/zh_v25 +# +# Only files present in BOTH directories are compared. Files unique to either +# side are reported separately at the end. +# +# Dependencies: ffprobe (from ffmpeg), GNU-style stat OR BSD stat (macOS). + +set -euo pipefail + +if [ $# -ne 2 ]; then + echo "Usage: $0 " >&2 + echo " Compares mp3 files present in both directories by size and duration." >&2 + exit 2 +fi + +DIR_A="$1" +DIR_B="$2" + +if [ ! -d "$DIR_A" ]; then + echo "ERROR: baseline dir not found: $DIR_A" >&2 + exit 2 +fi +if [ ! -d "$DIR_B" ]; then + echo "ERROR: candidate dir not found: $DIR_B" >&2 + exit 2 +fi + +if ! command -v ffprobe >/dev/null 2>&1; then + echo "ERROR: ffprobe not found. Install ffmpeg: brew install ffmpeg (macOS)" >&2 + exit 2 +fi + +# Portable byte-size function (macOS stat vs GNU stat) +filesize() { + if stat -f%z "$1" >/dev/null 2>&1; then + stat -f%z "$1" # BSD / macOS + else + stat -c%s "$1" # GNU / Linux + fi +} + +duration() { + ffprobe -v quiet -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 "$1" +} + +# Compute the intersection (files present in both) +common_files=$(comm -12 \ + <(cd "$DIR_A" && ls *.mp3 2>/dev/null | sort) \ + <(cd "$DIR_B" && ls *.mp3 2>/dev/null | sort) || true) + +only_a=$(comm -23 \ + <(cd "$DIR_A" && ls *.mp3 2>/dev/null | sort) \ + <(cd "$DIR_B" && ls *.mp3 2>/dev/null | sort) || true) + +only_b=$(comm -13 \ + <(cd "$DIR_A" && ls *.mp3 2>/dev/null | sort) \ + <(cd "$DIR_B" && ls *.mp3 2>/dev/null | sort) || true) + +if [ -z "$common_files" ]; then + echo "ERROR: no .mp3 files common to both directories." >&2 + exit 1 +fi + +# Header +printf "| %-28s | %10s | %10s | %9s | %8s | %8s | %7s |\n" \ + "id" "A bytes" "B bytes" "Δsize%" "A dur" "B dur" "Δdur%" +printf "|%s|%s|%s|%s|%s|%s|%s|\n" \ + "-----------------------------" "------------" "------------" "-----------" "----------" "----------" "---------" + +total_a_size=0 +total_b_size=0 +total_a_dur=0 +total_b_dur=0 +n=0 + +while IFS= read -r fname; do + [ -z "$fname" ] && continue + a_path="$DIR_A/$fname" + b_path="$DIR_B/$fname" + a_size=$(filesize "$a_path") + b_size=$(filesize "$b_path") + a_dur=$(duration "$a_path") + b_dur=$(duration "$b_path") + + if [ "$a_size" -gt 0 ]; then + dsize=$(awk -v a="$a_size" -v b="$b_size" 'BEGIN{printf "%.1f", (b-a)*100/a}') + else + dsize="N/A" + fi + ddur=$(awk -v a="$a_dur" -v b="$b_dur" 'BEGIN{if(a+0==0){print "N/A"}else{printf "%.1f", (b-a)*100/a}}') + + id_only="${fname%.mp3}" + printf "| %-28s | %10d | %10d | %8s%% | %7.2fs | %7.2fs | %6s%% |\n" \ + "$id_only" "$a_size" "$b_size" "$dsize" "$a_dur" "$b_dur" "$ddur" + + total_a_size=$((total_a_size + a_size)) + total_b_size=$((total_b_size + b_size)) + total_a_dur=$(awk -v s="$total_a_dur" -v d="$a_dur" 'BEGIN{printf "%.3f", s+d}') + total_b_dur=$(awk -v s="$total_b_dur" -v d="$b_dur" 'BEGIN{printf "%.3f", s+d}') + n=$((n + 1)) +done <<< "$common_files" + +echo "" +echo "**Totals (${n} common files):**" +echo "" +if [ "$total_a_size" -gt 0 ]; then + size_delta=$(awk -v a="$total_a_size" -v b="$total_b_size" 'BEGIN{printf "%.1f", (b-a)*100/a}') + dur_delta=$(awk -v a="$total_a_dur" -v b="$total_b_dur" 'BEGIN{printf "%.1f", (b-a)*100/a}') + echo "- Size: A=${total_a_size} B=${total_b_size} (Δ ${size_delta}%)" + echo "- Duration: A=${total_a_dur}s B=${total_b_dur}s (Δ ${dur_delta}%)" +fi + +if [ -n "$only_a" ]; then + echo "" + echo "**Only in A (${DIR_A}):**" + echo "$only_a" | sed 's/^/ - /' +fi +if [ -n "$only_b" ]; then + echo "" + echo "**Only in B (${DIR_B}):**" + echo "$only_b" | sed 's/^/ - /' +fi diff --git a/stepfun-tts/scripts/asr_transcribe.py b/stepfun-tts/scripts/asr_transcribe.py new file mode 100755 index 00000000..9f615671 --- /dev/null +++ b/stepfun-tts/scripts/asr_transcribe.py @@ -0,0 +1,205 @@ +#!/usr/bin/env python3 +""" +stepaudio-2.5-asr transcription — single file, SSE endpoint. + +Endpoint: POST https://api.stepfun.com/v1/audio/asr/sse (NOT /v1/audio/transcriptions) + +Why a dedicated script: naive implementations try to reuse the step-asr-era endpoint +(/v1/audio/transcriptions with multipart), get back `model stepaudio-2.5-asr not supported`, +and waste time debugging what looks like a model/permission issue. The actual cause is +that stepaudio-2.5-asr is a different endpoint entirely — SSE streaming, JSON body, +base64-encoded audio. + +Handles: +- Auto-detects audio format from file extension (mp3 / wav / ogg / pcm) +- base64 encodes and wraps in the nested {audio: {data, input: {transcription, format}}} body +- Parses SSE stream: collects transcript.text.delta, returns transcript.text.done.text +- Flags "content blocked" errors distinctly from transport errors +- 32K context / up to ~30 min audio in a single call — no client-side chunking needed + +Usage: + python3 asr_transcribe.py path/to/audio.mp3 + python3 asr_transcribe.py path/to/audio.mp3 --json # include usage (tokens/timing) + python3 asr_transcribe.py path/to/audio.mp3 --language zh +""" + +from __future__ import annotations + +import argparse +import base64 +import json +import os +import sys +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any + +ASR_URL = "https://api.stepfun.com/v1/audio/asr/sse" +MODEL = "stepaudio-2.5-asr" + +# Extensions that StepAudio 2.5 ASR accepts natively (no conversion needed) +EXT_TO_FORMAT = { + ".mp3": "mp3", + ".wav": "wav", + ".ogg": "ogg", + ".opus": "ogg", # opus in ogg container + ".pcm": "pcm", +} + + +def load_api_key() -> str: + """Env first, then ${CLAUDE_PLUGIN_DATA}/config.json. Fail fast.""" + k = os.environ.get("STEPFUN_API_KEY", "").strip() + if k: + return k + plugin_data = os.environ.get("CLAUDE_PLUGIN_DATA", "").strip() + if plugin_data: + cfg = Path(plugin_data) / "config.json" + if cfg.exists(): + try: + k = json.loads(cfg.read_text()).get("api_key", "").strip() + if k: + return k + except json.JSONDecodeError: + pass + print( + "ERROR: no API key found.\n" + " Set $STEPFUN_API_KEY, or create ${CLAUDE_PLUGIN_DATA}/config.json with {\"api_key\": \"...\"}", + file=sys.stderr, + ) + sys.exit(2) + + +def detect_format(path: Path, override: str | None) -> str: + if override: + return override + fmt = EXT_TO_FORMAT.get(path.suffix.lower()) + if not fmt: + print( + f"ERROR: cannot detect audio format from extension {path.suffix!r}.\n" + f" Supported: {', '.join(EXT_TO_FORMAT)}.\n" + f" Or pass --format explicitly.", + file=sys.stderr, + ) + sys.exit(2) + return fmt + + +def transcribe( + *, + api_key: str, + audio_path: Path, + audio_format: str, + language: str = "zh", + enable_itn: bool = True, + timeout: int = 1200, +) -> dict[str, Any]: + """ + Returns {ok, text?, usage?, elapsed, deltas_count, err?, censored?}. + Parses the SSE stream; takes the text from transcript.text.done. + """ + audio_b64 = base64.b64encode(audio_path.read_bytes()).decode("ascii") + body = json.dumps( + { + "audio": { + "data": audio_b64, + "input": { + "transcription": { + "language": language, + "model": MODEL, + "enable_itn": enable_itn, + }, + "format": {"type": audio_format}, + }, + } + } + ).encode() + req = urllib.request.Request( + ASR_URL, + data=body, + method="POST", + headers={ + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + "Accept": "text/event-stream", + }, + ) + t0 = time.time() + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + raw = resp.read().decode("utf-8", errors="replace") + except urllib.error.HTTPError as e: + err = e.read().decode(errors="replace")[:500] + censored = "censorship" in err.lower() or "blocked" in err.lower() + return {"ok": False, "status": e.code, "err": err, "elapsed": time.time() - t0, "censored": censored} + + elapsed = time.time() - t0 + text = "" + usage: dict[str, Any] | None = None + deltas = 0 + errors: list[str] = [] + for line in raw.splitlines(): + if not line.startswith("data:"): + continue + payload = line[5:].strip() + if not payload: + continue + try: + ev = json.loads(payload) + except json.JSONDecodeError: + continue + t = ev.get("type") + if t == "transcript.text.delta": + deltas += 1 + elif t == "transcript.text.done": + text = ev.get("text", "") + usage = ev.get("usage") + elif t == "error": + errors.append(ev.get("message", "")) + + if not text and errors: + return {"ok": False, "status": 200, "err": "; ".join(errors), "elapsed": elapsed} + return {"ok": True, "text": text, "usage": usage, "elapsed": elapsed, "deltas_count": deltas} + + +def main() -> int: + ap = argparse.ArgumentParser(description="stepaudio-2.5-asr transcription (SSE endpoint)") + ap.add_argument("audio", type=Path, help="Path to audio file (mp3/wav/ogg/opus/pcm)") + ap.add_argument("--language", default="zh", help="Language code (zh/en). Default: zh") + ap.add_argument("--format", help="Audio format override (mp3/wav/ogg/pcm)") + ap.add_argument("--no-itn", action="store_true", help="Disable inverse text normalization") + ap.add_argument("--json", action="store_true", help="Output full JSON (text + usage + timing)") + args = ap.parse_args() + + if not args.audio.exists(): + print(f"ERROR: audio file not found: {args.audio}", file=sys.stderr) + return 2 + + api_key = load_api_key() + fmt = detect_format(args.audio, args.format) + result = transcribe( + api_key=api_key, + audio_path=args.audio, + audio_format=fmt, + language=args.language, + enable_itn=not args.no_itn, + ) + + if not result["ok"]: + if result.get("censored"): + print("ERROR: content blocked by StepFun censorship. The audio likely contains sensitive content.", file=sys.stderr) + else: + print(f"ERROR status={result.get('status')}: {result.get('err', '')}", file=sys.stderr) + return 1 + + if args.json: + print(json.dumps(result, ensure_ascii=False, indent=2)) + else: + print(result["text"]) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/stepfun-tts/scripts/tts_generate.py b/stepfun-tts/scripts/tts_generate.py new file mode 100755 index 00000000..993ffd9f --- /dev/null +++ b/stepfun-tts/scripts/tts_generate.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +""" +stepaudio-2.5-tts synthesis — single line or batch. + +Endpoint: POST https://api.stepfun.com/v1/audio/speech + +Key things this script handles that naive implementations miss: +- Does NOT send voice_label (would trigger "voice_label is not supported for v2 models") +- Puts emotion/prosody into `instruction` (natural-language, ≤200 chars) +- Preserves inline `()` directives in the text — these are consumed by the TTS as directions, not read aloud +- Per-line censorship_block fallback: log and skip, don't fail the whole batch +- Reads API key from $STEPFUN_API_KEY or $CLAUDE_PLUGIN_DATA/config.json + +Usage: + # Single line + python3 tts_generate.py --text "你好，我是蕾格。" --out /tmp/hello.mp3 \\ + --instruction "温暖的希望感，语气鼓励" + + # Batch from JSONL (one JSON object per line: {"id": "...", "text": "...", "instruction": "..."}) + python3 tts_generate.py --batch lines.jsonl --out-dir /tmp/voices/ + + # With inline prosody directives in the text itself + python3 tts_generate.py --text "你好(停顿一下)我是蕾格(轻声)" --out /tmp/hello.mp3 +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import time +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any, Iterable + +API_URL = "https://api.stepfun.com/v1/audio/speech" +MODEL = "stepaudio-2.5-tts" +DEFAULT_VOICE = "shuangkuaijiejie" # 爽快姐姐 — verified in the 2026-04 test pass + + +def load_api_key() -> str: + """Env first, then ${CLAUDE_PLUGIN_DATA}/config.json. Fail fast — no placeholder fallback.""" + k = os.environ.get("STEPFUN_API_KEY", "").strip() + if k: + return k + plugin_data = os.environ.get("CLAUDE_PLUGIN_DATA", "").strip() + if plugin_data: + cfg = Path(plugin_data) / "config.json" + if cfg.exists(): + try: + k = json.loads(cfg.read_text()).get("api_key", "").strip() + if k: + return k + except json.JSONDecodeError: + pass + print( + "ERROR: no API key found.\n" + " Set $STEPFUN_API_KEY, or create ${CLAUDE_PLUGIN_DATA}/config.json with {\"api_key\": \"...\"}\n" + " Get a key at https://platform.stepfun.com/ → API Keys", + file=sys.stderr, + ) + sys.exit(2) + + +def synthesize( + *, + api_key: str, + text: str, + instruction: str | None = None, + voice: str = DEFAULT_VOICE, + speed: float = 1.0, + volume: float = 1.0, + response_format: str = "mp3", + timeout: int = 300, +) -> dict[str, Any]: + """ + Call /v1/audio/speech with stepaudio-2.5-tts. + + Returns {ok, audio_bytes?, status, err?, censored?}. + censored=True signals a censorship_block which callers should handle individually + rather than aborting a batch. + """ + body: dict[str, Any] = { + "model": MODEL, + "input": text, + "voice": voice, + "response_format": response_format, + "speed": speed, + "volume": volume, + } + if instruction: + if len(instruction) > 200: + return {"ok": False, "status": 0, "err": f"instruction too long: {len(instruction)} > 200 chars"} + body["instruction"] = instruction + + req = urllib.request.Request( + API_URL, + data=json.dumps(body).encode(), + method="POST", + headers={ + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + }, + ) + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + return {"ok": True, "status": resp.status, "audio_bytes": resp.read()} + except urllib.error.HTTPError as e: + raw = e.read().decode(errors="replace") + # Detect censorship_block so caller can decide whether to skip + censored = "censorship_block" in raw or "blocked" in raw.lower() + # Detect the known voice_label migration error and make the message actionable + if "voice_label is not supported" in raw: + hint = ( + "\n HINT: stepaudio-2.5-tts does not accept voice_label. " + "Put emotion/prosody into `instruction` (natural language) instead." + ) + raw = raw + hint + return {"ok": False, "status": e.code, "err": raw[:500], "censored": censored} + + +def read_batch(path: Path) -> Iterable[dict[str, Any]]: + for line in path.read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + yield json.loads(line) + + +def main() -> int: + ap = argparse.ArgumentParser(description="stepaudio-2.5-tts single-line or batch synthesis") + g = ap.add_mutually_exclusive_group(required=True) + g.add_argument("--text", help="Single text to synthesize") + g.add_argument("--batch", type=Path, help="JSONL file: {id, text, instruction?} per line") + ap.add_argument("--out", help="Output mp3 path (single mode)") + ap.add_argument("--out-dir", type=Path, help="Output directory (batch mode)") + ap.add_argument("--instruction", help="Global tone directive (natural language, ≤200 chars)") + ap.add_argument("--voice", default=DEFAULT_VOICE, help=f"Voice ID (default: {DEFAULT_VOICE})") + ap.add_argument("--speed", type=float, default=1.0) + ap.add_argument("--volume", type=float, default=1.0) + ap.add_argument("--delay-ms", type=int, default=400, help="Sleep between batch requests to avoid throttling") + args = ap.parse_args() + + api_key = load_api_key() + + if args.text: + if not args.out: + print("ERROR: --out is required with --text", file=sys.stderr) + return 2 + result = synthesize( + api_key=api_key, + text=args.text, + instruction=args.instruction, + voice=args.voice, + speed=args.speed, + volume=args.volume, + ) + if not result["ok"]: + print(f"FAIL status={result['status']}: {result.get('err', '')}", file=sys.stderr) + return 1 + Path(args.out).write_bytes(result["audio_bytes"]) + print(f"OK wrote {args.out} ({len(result['audio_bytes'])} bytes)") + return 0 + + # Batch mode + if not args.out_dir: + print("ERROR: --out-dir is required with --batch", file=sys.stderr) + return 2 + args.out_dir.mkdir(parents=True, exist_ok=True) + success = 0 + censored: list[str] = [] + failed: list[tuple[str, str]] = [] + + for item in read_batch(args.batch): + line_id = item.get("id") + text = item.get("text") + if not line_id or not text: + print(f"skip (missing id/text): {item}", file=sys.stderr) + continue + instr = item.get("instruction", args.instruction) + result = synthesize( + api_key=api_key, + text=text, + instruction=instr, + voice=item.get("voice", args.voice), + speed=item.get("speed", args.speed), + volume=item.get("volume", args.volume), + ) + out_path = args.out_dir / f"{line_id}.mp3" + if result["ok"]: + out_path.write_bytes(result["audio_bytes"]) + print(f" ✓ {line_id} ({len(result['audio_bytes'])} bytes)") + success += 1 + elif result.get("censored"): + print(f" ⚠ {line_id} CENSORED — skipped (consider rewriting or fallback to step-tts-2)") + censored.append(line_id) + else: + err = result.get("err", "")[:160] + print(f" ✗ {line_id} status={result['status']}: {err}", file=sys.stderr) + failed.append((line_id, err)) + time.sleep(args.delay_ms / 1000) + + print("") + print(f"Done: {success} ok, {len(censored)} censored, {len(failed)} failed") + if censored: + print(f" Censored IDs: {', '.join(censored)}") + if failed: + print(" Failed IDs:") + for lid, err in failed: + print(f" {lid}: {err}") + return 0 if not failed else 1 + + +if __name__ == "__main__": + sys.exit(main()) From ec7c313b0997847dabd2abb2ea8a547a526381e7 Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 26 Apr 2026 21:53:43 +0800 Subject: [PATCH 117/137] feat(pdf-creator): CJK typography auto-injection + theme refinements (v1.4.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds two-layer CJK typography handling without modifying user markdown or theme CSS files: - Layer 1: _load_theme() appends a CJK CSS patch (table-layout: fixed, word-break: keep-all, line-break: strict) to fix the most common weasyprint CJK rendering issue — single-character cell wrapping when one column has 5x more content than another. - Layer 2: post-render typography lint via pdftotext -layout scans for known CJK anti-patterns (orphan single CJK char, broken bracket pairs, short lines ending with mid-thought punctuation). Findings are warnings, not errors; PDF still generates and the author decides whether to accept, shorten, or restructure. Also bundles theme CSS refinements (default.css +58, warm-terra.css +35) and a new self-check preview test (test_self_check_preview.py). - pdf-creator: v1.3.2 → v1.4.0 (MINOR — new feature: CJK auto-injection) - SKILL.md documents the layer architecture and the deliberate choice not to silently auto-fix everything (per CLAUDE.md "禁止隐式行为" rule). Co-Authored-By: Claude Opus 4.7 (1M context) --- .claude-plugin/marketplace.json | 2 +- daymade-docs/pdf-creator/SKILL.md | 37 +++++ daymade-docs/pdf-creator/scripts/md_to_pdf.py | 67 ++++++++- .../scripts/tests/test_self_check_preview.py | 130 ++++++++++++++++++ daymade-docs/pdf-creator/themes/default.css | 58 +++++++- .../pdf-creator/themes/warm-terra.css | 35 ++++- 6 files changed, 319 insertions(+), 10 deletions(-) create mode 100644 daymade-docs/pdf-creator/scripts/tests/test_self_check_preview.py diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 5c2c0bb4..de9680f6 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -673,7 +673,7 @@ "description": "Create PDF documents from markdown with Chinese font support. Supports theme system (default for formal docs, warm-terra for training materials) and dual backend (weasyprint or Chrome). Triggers include convert to PDF, generate PDF, markdown to PDF, or printable documents", "source": "./daymade-docs/pdf-creator", "strict": false, - "version": "1.3.2", + "version": "1.4.0", "category": "document-conversion", "keywords": [ "pdf", diff --git a/daymade-docs/pdf-creator/SKILL.md b/daymade-docs/pdf-creator/SKILL.md index 73606c41..e34da9a8 100644 --- a/daymade-docs/pdf-creator/SKILL.md +++ b/daymade-docs/pdf-creator/SKILL.md @@ -79,3 +79,40 @@ python scripts/md_to_pdf.py input.md output.pdf --no-preview ``` **Requires** `pdftoppm` (`brew install poppler` on macOS). If not installed, the script logs a hint and skips preview generation but still produces the PDF. + +## CJK Typography (default behavior) + +The script applies two layers of CJK-aware processing automatically — **without modifying the user's markdown source or theme CSS files**: + +### Layer 1: CSS patch (auto-injected, fixes ~80% of cases) + +`_load_theme()` appends a CJK typography CSS patch to the loaded theme CSS. The patch: + +- `table { table-layout: fixed; width: 100% }` — equal column widths prevent weasyprint auto-layout from squeezing one column to ~10% width when an adjacent column has 5x more content +- `td, th { word-break: keep-all; line-break: strict }` — don't slice CJK characters apart +- `th { white-space: nowrap }` — short headers stay one line for predictable column widths + +This silently fixes the most common anti-pattern (cell content forcibly wrapped between CJK characters producing single-char-only lines), without touching the user's source. The user's theme CSS file on disk is never modified. + +### Layer 2: Typography lint (post-render detection, catches the rest) + +After PDF generation, the script runs `pdftotext -layout` per page and scans for known CJK anti-patterns per "中文文案排版指北" (Chinese typography style guide): + +- Single CJK character alone on a line (cell still too narrow even after Layer 1) +- Line ending with `（` followed by content next line (broken bracket pair) +- Line starting with `）` (broken from previous bracket pair) +- Short line ending with mid-thought punctuation `、，；：` + +Findings are printed to stderr with page+line locations. They are **warnings, not errors** — PDF still generates. The author sees the finding and decides: + +1. Accept (e.g. one orphan char in a long doc may be acceptable) +2. Shorten the offending cell content to fit the column width +3. Restructure (e.g. move long content into a paragraph below the table) + +### Why not silently auto-fix everything? + +Layer 2 deliberately does NOT modify the markdown. Per CLAUDE.md "禁止隐式行为" rule: silently rewriting non-standard markdown (e.g. expanding pseudo-lists into real lists) would mask the signal that the source is wrong, causing the same markdown to render incorrectly in other processors. Layer 1 is acceptable because it patches **rendering behavior** for already-standard markdown (a standard table that weasyprint happens to render imperfectly for CJK), not the markdown source itself. + +### Known limitations + +When a single cell's content is just slightly longer than the available column width (e.g. 10 CJK chars in a 9-char-wide cell after equal split), weasyprint will fall back to forced break despite `keep-all`. Layer 1 cannot fix this — Layer 2 will catch it and prompt the author to shorten cell content or restructure. diff --git a/daymade-docs/pdf-creator/scripts/md_to_pdf.py b/daymade-docs/pdf-creator/scripts/md_to_pdf.py index b0860df3..828ced66 100644 --- a/daymade-docs/pdf-creator/scripts/md_to_pdf.py +++ b/daymade-docs/pdf-creator/scripts/md_to_pdf.py @@ -87,8 +87,71 @@ def _detect_backend() -> str: sys.exit(1) +# ---------------- CJK typography patch (auto-injected, no source mutation) ---- +# +# Design principle: the user's markdown stays untouched, the user's theme CSS +# files stay untouched. We only patch the CSS string at load time (intermediate +# state), so the same .md + the same theme.css render correctly without the +# author having to know about CJK line-break engine quirks. +# +# Why needed: weasyprint's default `word-break: normal` treats every CJK +# character as a break opportunity. In narrow table cells this produces +# single-char-only lines and broken bracket pairs — anti-patterns per the +# well-known "中文文案排版指北" Chinese typography guide. +# +# Strategy (CJK-safe break rules, scoped to table cells only so body text +# behaves normally): +# - `word-break: keep-all` — don't break inside CJK runs +# - `overflow-wrap: break-word` — but allow break at word/punctuation +# boundaries when content exceeds cell width +# - `line-break: strict` — apply strict CJK rules (no break before +# closing 」』）, no break after opening 「『（, no break around 、，；：) +_TYPOGRAPHY_CSS_PATCH = """ +/* ===== md_to_pdf auto-injected: CJK typography for table cells ===== + * + * Three-layer fix for the most common CJK rendering anti-patterns: + * + * 1. table-layout: fixed + equal column widths — prevents weasyprint + * auto-layout from squeezing one column to 10% width when an + * adjacent column has 5x more content (the root cause of "single + * CJK char alone on a line" in narrow cells). + * + * 2. CJK break rules at cell level — don't slice CJK characters apart; + * break only at word/punctuation boundaries. + * + * 3. Header nowrap — short headers stay one line; combined with fixed + * layout, column widths are predictable. + * + * Trade-off: tables now distribute width equally across columns instead + * of content-aware. This may give "wider than needed" columns to short- + * content cells, but eliminates the "single CJK char per line" bug. + */ +table { + table-layout: fixed; + width: 100%; +} +table td, table th { + word-break: keep-all; + /* `overflow-wrap: normal` instead of break-word: when keep-all says + * "don't break inside CJK" and cell is too narrow for the content, + * prefer letting content overflow slightly rather than fallback to + * mid-token breaks (which produce single-CJK-char-per-line). */ + overflow-wrap: normal; + line-break: strict; +} +table th { + white-space: nowrap; +} +/* =================================================================== */ +""" + + def _load_theme(theme_name: str) -> str: - """Load CSS from themes directory.""" + """Load CSS from themes directory and append CJK typography patch. + + The patch is appended AFTER the user theme so it wins the cascade for + table cells. The user's theme CSS file on disk is never modified. + """ theme_file = THEMES_DIR / f"{theme_name}.css" if not theme_file.exists(): available = [f.stem for f in THEMES_DIR.glob("*.css")] @@ -97,7 +160,7 @@ def _load_theme(theme_name: str) -> str: file=sys.stderr, ) sys.exit(1) - return theme_file.read_text(encoding="utf-8") + return theme_file.read_text(encoding="utf-8") + _TYPOGRAPHY_CSS_PATCH def _list_themes() -> list[str]: diff --git a/daymade-docs/pdf-creator/scripts/tests/test_self_check_preview.py b/daymade-docs/pdf-creator/scripts/tests/test_self_check_preview.py new file mode 100644 index 00000000..f9696df2 --- /dev/null +++ b/daymade-docs/pdf-creator/scripts/tests/test_self_check_preview.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python3 +""" +Integration test for visual self-check helper. + +Verifies the contract: + 1. Default PDF run auto-generates per-page PNG previews next to the PDF + 2. Default run prints a self-check checklist to stdout (so AI/author + is automatically reminded to visually inspect the rendering) + 3. --no-preview disables both PNG generation and checklist + +This test enforces "Visual Verification" rule from CLAUDE.md: PDF generation +silently succeeding does NOT mean the rendering matches the markdown intent. +The checklist makes "Read each page PNG and verify" the default contract, +not an optional step that's easy to skip. +""" + +import subprocess +import sys +import tempfile +from pathlib import Path + + +SAMPLE_MD = """# Test Document + +A short test for self-check preview generation. + +## Section A + +- Item 1 +- Item 2 +- Item 3 + +## Section B + +正文段落（中文测试 CJK rendering）。 +""" + + +def run_md_to_pdf(args: list[str], cwd: Path) -> subprocess.CompletedProcess: + script = cwd.parent / "md_to_pdf.py" + return subprocess.run( + ["uv", "run", "--with", "weasyprint", str(script)] + args, + capture_output=True, + text=True, + cwd=cwd.parent, + ) + + +def main() -> int: + passed = 0 + total = 0 + script_dir = Path(__file__).parent.parent + + with tempfile.TemporaryDirectory() as tmpdir: + tmp = Path(tmpdir) + md_path = tmp / "test.md" + md_path.write_text(SAMPLE_MD, encoding="utf-8") + + # ---- Test 1: Default run prints self-check checklist ---- + total += 1 + pdf_path = tmp / "test.pdf" + result = run_md_to_pdf([str(md_path), str(pdf_path)], script_dir) + if result.returncode != 0: + print(f"❌ PDF generation failed: {result.stderr}") + return 1 + + checklist_markers = [ + "Visual self-check required", + "Paragraphs render as separate blocks", + "Tables fit within page margins", + "Emoji + CJK glyphs render", + ] + missing = [m for m in checklist_markers if m not in result.stdout] + if not missing: + print("✅ Default run prints self-check checklist with all key items") + passed += 1 + else: + print(f"❌ Checklist missing items: {missing}") + print(f" stdout: {result.stdout[:500]}") + + # ---- Test 2: Preview directory + PNG files generated ---- + total += 1 + preview_dir = tmp / "test-preview" + if preview_dir.exists(): + pages = sorted(preview_dir.glob("page-*.png")) + if pages and all(p.stat().st_size > 0 for p in pages): + print(f"✅ {len(pages)} non-empty preview PNGs generated in test-preview/") + passed += 1 + else: + print(f"❌ Preview dir exists but PNGs missing/empty: {pages}") + else: + print(f"❌ Preview dir not created: {preview_dir}") + + # ---- Test 3: --no-preview disables both PNG + checklist ---- + total += 1 + pdf_path2 = tmp / "test_disabled.pdf" + preview_dir2 = tmp / "test_disabled-preview" + result = run_md_to_pdf( + [str(md_path), str(pdf_path2), "--no-preview"], script_dir + ) + no_checklist = "Visual self-check" not in result.stdout + no_preview_dir = not preview_dir2.exists() + if no_checklist and no_preview_dir: + print("✅ --no-preview correctly disables PNG + checklist") + passed += 1 + else: + print( + f"❌ --no-preview failed: checklist_absent={no_checklist}, " + f"dir_absent={no_preview_dir}" + ) + + # ---- Test 4: Stale PNGs cleaned on rerun ---- + total += 1 + # Plant a stale page-99.png (simulating old preview from a longer doc) + preview_dir.mkdir(exist_ok=True) + stale = preview_dir / "page-99.png" + stale.write_bytes(b"stale") + result = run_md_to_pdf([str(md_path), str(pdf_path)], script_dir) + if not stale.exists(): + print("✅ Stale preview PNGs cleaned on rerun") + passed += 1 + else: + print("❌ Stale page-99.png not cleaned on rerun") + + print(f"\n=== {passed}/{total} tests passed ===") + return 0 if passed == total else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/daymade-docs/pdf-creator/themes/default.css b/daymade-docs/pdf-creator/themes/default.css index 07a0cb23..098074a1 100644 --- a/daymade-docs/pdf-creator/themes/default.css +++ b/daymade-docs/pdf-creator/themes/default.css @@ -10,7 +10,13 @@ @page { size: A4; - margin: 2.5cm 2cm; + margin: 2.5cm 2cm 2cm 2cm; + @bottom-center { + content: counter(page) " / " counter(pages); + font-family: 'Songti SC', 'SimSun', 'STSong', 'Noto Serif CJK SC', serif; + font-size: 9pt; + color: #666; + } } body { @@ -30,20 +36,60 @@ h1 { margin-bottom: 1.5em; } +/* Heading scale: each level visibly larger than body (12pt) AND visually + * distinct from adjacent levels. Font fallback chain widened to include + * PingFang SC and system-ui in case 'Heiti SC' is not registered with + * fontconfig (common on weasyprint installs). + */ h2 { - font-family: 'Heiti SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', sans-serif; - font-size: 14pt; + font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; + font-size: 16pt; font-weight: bold; margin-top: 1.5em; margin-bottom: 0.8em; + color: #000; } h3 { - font-family: 'Heiti SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', sans-serif; - font-size: 12pt; + font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; + font-size: 14pt; font-weight: bold; - margin-top: 1em; + margin-top: 1.3em; margin-bottom: 0.5em; + color: #000; +} + +h4 { + font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; + font-size: 13pt; + font-weight: bold; + margin-top: 1.1em; + margin-bottom: 0.4em; + color: #1a1a1a; +} + +/* h5: still distinct from body. Combine size bump (+1pt) with left border + * so the heading visually jumps out even when the size delta is subtle. + */ +h5 { + font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; + font-size: 13pt; + font-weight: bold; + margin-top: 1em; + margin-bottom: 0.35em; + padding-left: 0.5em; + border-left: 3px solid #888; + color: #1a1a1a; +} + +h6 { + font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; + font-size: 12pt; + font-weight: bold; + margin-top: 0.8em; + margin-bottom: 0.25em; + color: #444; + font-style: italic; } p { diff --git a/daymade-docs/pdf-creator/themes/warm-terra.css b/daymade-docs/pdf-creator/themes/warm-terra.css index eb7e9573..b1483917 100644 --- a/daymade-docs/pdf-creator/themes/warm-terra.css +++ b/daymade-docs/pdf-creator/themes/warm-terra.css @@ -15,7 +15,13 @@ @page { size: A4; - margin: 12mm; + margin: 12mm 12mm 16mm 12mm; + @bottom-center { + content: counter(page) " / " counter(pages); + font-family: 'PingFang SC', 'Microsoft YaHei', 'Noto Sans CJK SC', sans-serif; + font-size: 9pt; + color: #8a7460; + } } body { @@ -52,6 +58,33 @@ h3 { margin-bottom: 0.5em; } +/* h4/h5/h6 must remain visually distinct from body text — never smaller than + * body. Use bold + slight color shift instead of size shrinking. */ +h4 { + font-size: 13.5px; + font-weight: 700; + margin-top: 14px; + margin-bottom: 0.4em; + color: #1f1b17; +} + +h5 { + font-size: 13px; + font-weight: 700; + margin-top: 12px; + margin-bottom: 0.3em; + color: #2a2521; +} + +h6 { + font-size: 13px; + font-weight: 700; + margin-top: 10px; + margin-bottom: 0.2em; + color: #4a3f33; + font-style: italic; +} + p { margin: 0.6em 0; } From f3182f7b44d4ea566e6a3b388ee35b69011a9637 Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 26 Apr 2026 22:37:41 +0800 Subject: [PATCH 118/137] fix(pdf-creator): typography size unity + integration test path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Default theme: tighten font-size scale so all body-adjacent elements (inline code / table / pre) sit within ≤1pt of body, eliminating the visual "noticeably smaller" gap on long-form Chinese documents. - body: 12pt -> 11.5pt; line-height 1.8 -> 1.6 - heading scale: h1 17pt / h2 15pt / h3 13pt / h4 12.5pt / h5 12pt + left border / h6 11.5pt italic - inline code: 10pt -> 11pt (was 2pt smaller than body) - table: 10pt -> 11pt - pre code: 9pt -> 10.5pt; line-height 1.6 -> 1.5 - .cjk-code-block matches table test_self_check_preview.py: fix script path resolution. Old: cwd.parent / "md_to_pdf.py" (resolved to pdf-creator/md_to_pdf.py which doesn't exist). New: scripts_dir / "md_to_pdf.py" with renamed parameter and clarifying docstring on cwd semantics. Tests: 14/14 passing (self_check_preview 4/4 + list_rendering 4/4 + cjk_code_blocks 6/6). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../scripts/tests/test_self_check_preview.py | 13 +++++-- daymade-docs/pdf-creator/themes/default.css | 35 ++++++++++--------- 2 files changed, 29 insertions(+), 19 deletions(-) diff --git a/daymade-docs/pdf-creator/scripts/tests/test_self_check_preview.py b/daymade-docs/pdf-creator/scripts/tests/test_self_check_preview.py index f9696df2..e6583491 100644 --- a/daymade-docs/pdf-creator/scripts/tests/test_self_check_preview.py +++ b/daymade-docs/pdf-creator/scripts/tests/test_self_check_preview.py @@ -36,13 +36,20 @@ """ -def run_md_to_pdf(args: list[str], cwd: Path) -> subprocess.CompletedProcess: - script = cwd.parent / "md_to_pdf.py" +def run_md_to_pdf(args: list[str], scripts_dir: Path) -> subprocess.CompletedProcess: + """Run md_to_pdf.py with the given CLI args. + + scripts_dir: path to the pdf-creator/scripts/ directory. The script + lives at scripts_dir/md_to_pdf.py; we run the subprocess + with cwd=scripts_dir.parent (the pdf-creator/ root) so + the script's relative themes/ lookup resolves correctly. + """ + script = scripts_dir / "md_to_pdf.py" return subprocess.run( ["uv", "run", "--with", "weasyprint", str(script)] + args, capture_output=True, text=True, - cwd=cwd.parent, + cwd=scripts_dir.parent, ) diff --git a/daymade-docs/pdf-creator/themes/default.css b/daymade-docs/pdf-creator/themes/default.css index 098074a1..d24eae31 100644 --- a/daymade-docs/pdf-creator/themes/default.css +++ b/daymade-docs/pdf-creator/themes/default.css @@ -21,29 +21,32 @@ body { font-family: 'Songti SC', 'SimSun', 'STSong', 'Noto Serif CJK SC', serif; - font-size: 12pt; - line-height: 1.8; + font-size: 11.5pt; + line-height: 1.6; color: #000; width: 100%; } h1 { font-family: 'Heiti SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', sans-serif; - font-size: 18pt; + font-size: 17pt; font-weight: bold; text-align: center; margin-top: 0; margin-bottom: 1.5em; } -/* Heading scale: each level visibly larger than body (12pt) AND visually +/* Heading scale: each level visibly larger than body (11.5pt) AND visually * distinct from adjacent levels. Font fallback chain widened to include * PingFang SC and system-ui in case 'Heiti SC' is not registered with * fontconfig (common on weasyprint installs). + * + * Size unity rule: all body-adjacent text (inline code, table, pre) + * stays within 0.5-1pt of body so nothing reads as "noticeably smaller". */ h2 { font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; - font-size: 16pt; + font-size: 15pt; font-weight: bold; margin-top: 1.5em; margin-bottom: 0.8em; @@ -52,7 +55,7 @@ h2 { h3 { font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; - font-size: 14pt; + font-size: 13pt; font-weight: bold; margin-top: 1.3em; margin-bottom: 0.5em; @@ -61,19 +64,19 @@ h3 { h4 { font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; - font-size: 13pt; + font-size: 12.5pt; font-weight: bold; margin-top: 1.1em; margin-bottom: 0.4em; color: #1a1a1a; } -/* h5: still distinct from body. Combine size bump (+1pt) with left border +/* h5: still distinct from body. Combine size bump (+0.5pt) with left border * so the heading visually jumps out even when the size delta is subtle. */ h5 { font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; - font-size: 13pt; + font-size: 12pt; font-weight: bold; margin-top: 1em; margin-bottom: 0.35em; @@ -84,7 +87,7 @@ h5 { h6 { font-family: 'Heiti SC', 'PingFang SC', 'SimHei', 'STHeiti', 'Noto Sans CJK SC', system-ui, sans-serif; - font-size: 12pt; + font-size: 11.5pt; font-weight: bold; margin-top: 0.8em; margin-bottom: 0.25em; @@ -110,7 +113,7 @@ table { border-collapse: collapse; width: 100%; margin: 1em 0; - font-size: 10pt; + font-size: 11pt; table-layout: fixed; } @@ -138,7 +141,7 @@ code { background: #f5f5f5; padding: 1px 4px; border-radius: 3px; - font-size: 10pt; + font-size: 11pt; } pre { @@ -157,8 +160,8 @@ pre code { background: none; padding: 0; border-radius: 0; - font-size: 9pt; - line-height: 1.6; + font-size: 10.5pt; + line-height: 1.5; } /* CJK code blocks converted to styled divs by preprocessor. @@ -170,8 +173,8 @@ pre code { border-radius: 4px; padding: 12px 16px; margin: 1em 0; - font-size: 10pt; - line-height: 1.8; + font-size: 11pt; + line-height: 1.6; white-space: pre-wrap; word-break: break-all; } From 3cb9823073344461dfa062bbf9f5e39f9e3aa9c8 Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 26 Apr 2026 23:00:10 +0800 Subject: [PATCH 119/137] fix(pdf-creator): add CJK font fallback for inline code (default theme) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Inline code with mixed ASCII + CJK content (e.g. `Shift + 右键`, `Cmd + Space`, `Option 键`) was rendering with the CJK chars INVISIBLE — only the ASCII parts and code box background showed. Root cause: code/pre code font-family chain was 'Menlo', 'PingFang SC', 'Heiti SC', 'Noto Sans CJK SC', monospace weasyprint's fontconfig often fails to register PingFang SC / Heiti SC / Noto Sans CJK SC, so CJK fell through to the generic 'monospace' which has no CJK glyphs → chars dropped silently. Fix: append 'Songti SC', 'SimSun' to the chain (same fonts the body successfully uses). Songti SC is registered, so CJK chars in inline/pre code now always render even if PingFang/Heiti/Noto are unavailable. Trade-off: when those preferred CJK monospace are missing, CJK falls back to a serif (Songti) inside the monospace code box — readability > invisibility. Co-Authored-By: Claude Opus 4.7 (1M context) --- daymade-docs/pdf-creator/themes/default.css | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/daymade-docs/pdf-creator/themes/default.css b/daymade-docs/pdf-creator/themes/default.css index d24eae31..3a86ab4b 100644 --- a/daymade-docs/pdf-creator/themes/default.css +++ b/daymade-docs/pdf-creator/themes/default.css @@ -137,7 +137,7 @@ hr { } code { - font-family: 'Menlo', 'PingFang SC', 'Heiti SC', 'Noto Sans CJK SC', monospace; + font-family: 'Menlo', 'PingFang SC', 'Heiti SC', 'Noto Sans CJK SC', 'Songti SC', 'SimSun', monospace; background: #f5f5f5; padding: 1px 4px; border-radius: 3px; @@ -156,7 +156,7 @@ pre { } pre code { - font-family: 'Menlo', 'PingFang SC', 'Heiti SC', 'Noto Sans CJK SC', monospace; + font-family: 'Menlo', 'PingFang SC', 'Heiti SC', 'Noto Sans CJK SC', 'Songti SC', 'SimSun', monospace; background: none; padding: 0; border-radius: 0; From 9ff33522c46563e8a78bc5a1544753a0dd32cf0c Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 26 Apr 2026 23:05:19 +0800 Subject: [PATCH 120/137] fix(pdf-creator): keep table rows intact across page breaks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add defensive page-break rules so a single never gets split across page boundaries (which causes mid-line cell content cuts and no header repeat — major readability hit). - tr { page-break-inside: avoid; break-inside: avoid } CSS 2.1 + CSS3 Fragmentation aliases for cross-engine safety. weasyprint honors both and pushes the row to next page if it doesn't fit. Trade-off: short rows at page bottom may get pushed, leaving white space — readability > compactness. - thead { display: table-header-group } Repeats the header row on each page when a table spans pages. Standard CSS print idiom, weasyprint native support. Pages-count delta on a 11-page Chinese SOP: 0 (no row was actually being split today). This is defensive — protects future content from silent fragmentation regressions. Co-Authored-By: Claude Opus 4.7 (1M context) --- daymade-docs/pdf-creator/themes/default.css | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/daymade-docs/pdf-creator/themes/default.css b/daymade-docs/pdf-creator/themes/default.css index 3a86ab4b..e00501d2 100644 --- a/daymade-docs/pdf-creator/themes/default.css +++ b/daymade-docs/pdf-creator/themes/default.css @@ -117,6 +117,20 @@ table { table-layout: fixed; } +/* Keep table rows intact when paginating: prevents a single from being + * split across page boundaries (cell content cut mid-line, no header repeat). + * Trade-off: short rows that don't fit at page bottom get pushed to next page, + * leaving some white space at the bottom — readability > compactness. */ +tr { + page-break-inside: avoid; + break-inside: avoid; +} + +/* Repeat on each page when a table spans multiple pages. */ +thead { + display: table-header-group; +} + th, td { border: 1px solid #666; padding: 8px 6px; From 11b096f97f542f0445f8b65ed8355c72764eda67 Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 26 Apr 2026 23:13:08 +0800 Subject: [PATCH 121/137] fix(pdf-creator): force CJK chars in inline code to use CJK fonts (not Menlo) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous fallback-chain fix (3cb9823) added 'Songti SC, SimSun' to the code font-family chain. weasyprint still picked Menlo for CJK chars in mixed-script inline code (e.g. `Terminal/终端`), marking them with a Menlo font reference in the PDF even though Menlo has no CJK glyphs. Symptom: Chrome's PDF viewer auto-fell-back so CJK rendered fine, but strict readers (macOS Preview, Adobe Reader, most print drivers) showed blanks where CJK should be — only the ASCII parts of inline code visible. Root cause: weasyprint's font-selection algorithm doesn't probe per-glyph availability when picking the primary font for a run. Fix: declare 'Menlo' via @font-face with unicode-range limited to Latin / Latin-Ext / general punctuation / currency / letterlike. This makes weasyprint skip Menlo for any CJK char and pick the next chain font (PingFang SC → Heiti SC → Songti SC) which has CJK glyphs. Verified with pdfplumber: '终' '端' chars now reference EIATWD+PingFang-SC / HMHQRL+Heiti-SC-Bold / OKAPEP+Songti-SC-Bold in the PDF (no more Menlo references for CJK). Co-Authored-By: Claude Opus 4.7 (1M context) --- daymade-docs/pdf-creator/themes/default.css | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/daymade-docs/pdf-creator/themes/default.css b/daymade-docs/pdf-creator/themes/default.css index e00501d2..2d92adb4 100644 --- a/daymade-docs/pdf-creator/themes/default.css +++ b/daymade-docs/pdf-creator/themes/default.css @@ -8,6 +8,25 @@ * This is the original built-in theme from md_to_pdf.py, extracted for reference. */ +/* Restrict 'Menlo' to Latin/ASCII range so CJK characters in inline code + * don't get marked as Menlo (which has no CJK glyphs). Without this, the + * generated PDF references Menlo for CJK chars, and strict PDF readers + * (macOS Preview, some print drivers) show blanks instead of falling back. + * Chrome falls back automatically; Preview does not. The unicode-range + * trick forces weasyprint to skip Menlo for CJK and use the next font in + * the chain (PingFang SC → Heiti SC → Songti SC) which has CJK glyphs. */ +@font-face { + font-family: 'Menlo'; + src: local('Menlo'); + unicode-range: U+0020-007F, U+00A0-00FF, U+2000-206F, U+2070-209F, U+20A0-20CF, U+2100-214F; +} +@font-face { + font-family: 'Menlo'; + src: local('Menlo Bold'); + font-weight: bold; + unicode-range: U+0020-007F, U+00A0-00FF, U+2000-206F, U+2070-209F, U+20A0-20CF, U+2100-214F; +} + @page { size: A4; margin: 2.5cm 2cm 2cm 2cm; From 6f29306ad1ede789d8618ea6155a8a4ccde024cf Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 26 Apr 2026 23:37:35 +0800 Subject: [PATCH 122/137] fix(pdf-creator): prefer CID TrueType CJK fonts over OpenType in code chain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PingFang SC ships as OpenType (CID Type 0C) on macOS. weasyprint's subset-embedded OpenType CJK glyphs render correctly in Chrome but show blank squares in macOS Preview / Adobe Reader / many print drivers — known weasyprint + Preview compatibility issue with subset OT-CJK fonts. Songti SC and Heiti SC are CID TrueType and don't have this issue (verified by inspection: body text using Songti SC renders fine on all readers). Reorder the inline code font-family chain to prefer the CID TrueType CJK fonts: before: 'Menlo', 'PingFang SC', 'Heiti SC', ..., 'Songti SC', 'SimSun', monospace after: 'Menlo', 'Songti SC', 'Heiti SC', 'SimSun', 'PingFang SC', ..., monospace Verified with pdfplumber on the same SOP page that previously broke: "右键" now references OKAPEP+Songti-SC-Bold (CID TrueType) instead of MQZJZV+PingFang-SC-Semi-Bold (CID Type 0C OT), so Preview will render the glyphs correctly. Trade-off: inline code CJK glyphs are now serif (Songti SC) instead of sans-serif. Acceptable — readability across all PDF readers > monospace visual consistency for one font feature. Co-Authored-By: Claude Opus 4.7 (1M context) --- daymade-docs/pdf-creator/themes/default.css | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/daymade-docs/pdf-creator/themes/default.css b/daymade-docs/pdf-creator/themes/default.css index 2d92adb4..59b0be45 100644 --- a/daymade-docs/pdf-creator/themes/default.css +++ b/daymade-docs/pdf-creator/themes/default.css @@ -170,7 +170,7 @@ hr { } code { - font-family: 'Menlo', 'PingFang SC', 'Heiti SC', 'Noto Sans CJK SC', 'Songti SC', 'SimSun', monospace; + font-family: 'Menlo', 'Songti SC', 'Heiti SC', 'SimSun', 'PingFang SC', 'Noto Sans CJK SC', monospace; background: #f5f5f5; padding: 1px 4px; border-radius: 3px; @@ -189,7 +189,7 @@ pre { } pre code { - font-family: 'Menlo', 'PingFang SC', 'Heiti SC', 'Noto Sans CJK SC', 'Songti SC', 'SimSun', monospace; + font-family: 'Menlo', 'Songti SC', 'Heiti SC', 'SimSun', 'PingFang SC', 'Noto Sans CJK SC', monospace; background: none; padding: 0; border-radius: 0; From 04e404fd85fdd06733709e455e59d47cd73cfea4 Mon Sep 17 00:00:00 2001 From: daymade Date: Mon, 27 Apr 2026 00:10:22 +0800 Subject: [PATCH 123/137] fix(pdf-creator): neutralize pandoc auto-emitted from dash counts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pandoc reads dash counts in the markdown table separator row and emits as inline styles in HTML. For tables with one short-label column and one very long column (e.g. col1 is "4/28（周二）下午" 9 chars, col4 is a 30+ char address), pandoc allocates col1 ~17% width — too narrow for CJK labels — forcing mid-token line breaks like `4/28（周|二）下|午`. Inline styles beat external stylesheet rules at equal specificity, so no `td:first-child { width: ... }` rule could override the inline width. The fix uses `!important` on `` to neutralize pandoc's hint, letting weasyprint's `table-layout: fixed` distribute width equally (25% per column for a 4-col table — enough for typical CJK labels). Authors who genuinely want explicit column widths can still write raw HTML in markdown. Verified by 3 parallel agent investigations (CSS-only / HTML postprocess / markdown grid-table). Picked CSS-only as simplest one-line fix. Co-Authored-By: Claude Opus 4.7 (1M context) --- daymade-docs/pdf-creator/themes/default.css | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/daymade-docs/pdf-creator/themes/default.css b/daymade-docs/pdf-creator/themes/default.css index 59b0be45..f67588d8 100644 --- a/daymade-docs/pdf-creator/themes/default.css +++ b/daymade-docs/pdf-creator/themes/default.css @@ -163,6 +163,21 @@ th { font-weight: bold; } +/* Neutralize pandoc's auto-emitted based on dash counts + * in the markdown separator row. Pandoc treats `| ----- | --- |` as a column- + * width hint and inlines `style="width: 17%"` etc on each . Inline styles + * beat external stylesheets at equal specificity, so without `!important` no + * `td:first-child { width: ... }` rule can recover. With this neutralizer + * weasyprint falls back to `table-layout: fixed` equal width allocation, + * which for typical 4-col tables gives 25% per column — enough for short + * CJK labels like `4/28（周二）下午` to render on one line. + * + * Authors who really want explicit widths can still write raw HTML + * `` directly in markdown — that overrides this rule when needed. */ +table colgroup col { + width: auto !important; +} + hr { border: none; border-top: 1px solid #ccc; From 80e94fd1296f6c4acb5d9e199df0d201e52044c0 Mon Sep 17 00:00:00 2001 From: daymade Date: Mon, 27 Apr 2026 13:02:29 +0800 Subject: [PATCH 124/137] =?UTF-8?q?docs(pdf-creator):=20troubleshoot=20?= =?UTF-8?q?=E2=80=94=20CJK=20in=20inline=20code=20(Preview)=20+=20col1=20m?= =?UTF-8?q?id-break=20(pandoc=20colgroup)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add 2 troubleshooting entries to SKILL.md covering the two CJK rendering bugs fixed in recent commits (3cb9823 / 6f29306 / 04e404f): 1. Inline code with mixed CJK + ASCII shows blanks in macOS Preview - Cause: weasyprint subset-embeds PingFang SC as CID Type 0C OpenType, which strict PDF readers (Preview / Adobe) cannot render reliably. Chrome PDF viewer auto-fallbacks and hides the bug. - Fix already in default theme: prefer CID TrueType (Songti / Heiti) before OpenType (PingFang) in code font-family chain. - Verify with pdfplumber font-name inspection. 2. Table column 1 with short label gets mid-broken (e.g. `4/28（周|二）下|午`) - Cause: pandoc auto-emits from dash counts in the markdown separator row. Inline style beats external CSS — `td:first-child { width:... }` silently shadowed. - Fix already in default theme: `table colgroup col { width: auto !important }` - Verify with `pandoc input.md -t html | grep colgroup`. Co-Authored-By: Claude Opus 4.7 (1M context) --- daymade-docs/pdf-creator/SKILL.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/daymade-docs/pdf-creator/SKILL.md b/daymade-docs/pdf-creator/SKILL.md index e34da9a8..eb54cc5e 100644 --- a/daymade-docs/pdf-creator/SKILL.md +++ b/daymade-docs/pdf-creator/SKILL.md @@ -61,6 +61,10 @@ uv run --with weasyprint scripts/batch_convert.py *.md --output-dir ./pdfs **Chrome header/footer appearing**: The script passes `--no-pdf-header-footer`. If it still appears, your Chrome version may not support this flag — update Chrome. +**Inline code with mixed CJK + ASCII shows blanks in macOS Preview** (e.g. `` `Terminal/终端` `` renders only `Terminal/` with the CJK part missing): weasyprint subset-embeds PingFang SC as **OpenType (CID Type 0C)**, which strict PDF readers (macOS Preview / Adobe Reader) fail to render. Chrome's PDF viewer falls back automatically and hides the bug. Fix is in the default theme: code font-family chain prioritizes **CID TrueType** CJK fonts (Songti SC / Heiti SC) before OpenType ones (PingFang SC). To verify: `pdfplumber` + check `font['fontname']` of CJK chars — if any references `PingFang-SC` (CID Type 0C OT), readers will likely fail. Reorder font chain to put CID TrueType first. + +**Table column 1 with short label gets mid-broken** (e.g. `4/28（周|二）下|午`): pandoc auto-emits `` from dash counts in the markdown separator row. For `| ----- | --- | --- | -------- |` (uneven dash widths), pandoc allocates col 1 ~17% — too narrow for a 9-char CJK label. Inline `style=""` beats external CSS at equal specificity, so `td:first-child { width:... }` is silently shadowed. Fix is in default theme: `table colgroup col { width: auto !important }` neutralizes pandoc's hint, letting `table-layout: fixed` distribute equally (25% per column for a 4-col table). To verify: `pandoc input.md -t html | grep colgroup` — if it shows ``, the bug applies. + ## Visual Self-Check (default behavior) After every PDF generation, the script automatically: From b2003d6f31764a7a37e08cc76ded206810f70ae4 Mon Sep 17 00:00:00 2001 From: daymade Date: Tue, 28 Apr 2026 23:01:20 +0800 Subject: [PATCH 125/137] feat(statusline-generator): context window display with token counts + jq/py fallback (v1.1.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add context window usage display (actual tokens: 88.9K/1.0M + percentage) - human() formatter: K/M suffixes with one decimal for readability - Color-coded thresholds: green ≤50%, yellow 51-80%, red >80% - Auto-detect jq vs python3 for JSON parsing (Windows compat) - merge 8 jq calls into 1 (performance) - Add --no-optional-locks to rev-parse - New reference: context-window-schema.md (full statusline JSON schema) - SKILL.md: Dependencies section, context window docs, updated examples --- .claude-plugin/marketplace.json | 9 +- .../statusline-generator/SKILL.md | 54 +++++- .../references/context-window-schema.md | 107 ++++++++++++ .../scripts/generate_statusline.sh | 156 +++++++++++++----- 4 files changed, 270 insertions(+), 56 deletions(-) create mode 100644 daymade-claude-code/statusline-generator/references/context-window-schema.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index de9680f6..4bd5f6c5 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -6,7 +6,7 @@ }, "metadata": { "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, macOS programmatic window screenshot capture workflows, verified Scrapling CLI installation and web extraction workflows, plugin marketplace development for converting skills repos into official Claude Code marketplaces, Tencent IMA knowledge base companion with zero-config installation across Claude Code/Codex/OpenClaw, upstream bug detection and runtime repair, personalized fan-out search with priority-based knowledge base boosting, and suite plugins that expose related skills under shared namespaces for combined installation workflows — including a dedicated Claude Code operations suite (daymade-claude-code) bundling session recovery, CLAUDE.md optimization, troubleshooting, statusline configuration, export repair, and marketplace development skills under one namespace", - "version": "1.51.0" + "version": "1.46.0" }, "plugins": [ { @@ -946,13 +946,14 @@ }, { "name": "statusline-generator", - "description": "Configure Claude Code statuslines with multi-line layouts, cost tracking via ccusage, git status, and customizable colors", + "description": "Configure Claude Code statuslines with context window display (actual token counts), multi-line layouts, cost tracking via ccusage, git status, human-readable formatting, and customizable colors", "source": "./daymade-claude-code/statusline-generator", "strict": false, - "version": "1.0.1", + "version": "1.1.0", "category": "customization", "keywords": [ "statusline", + "context-window", "ccusage", "git-status", "customization", @@ -1215,4 +1216,4 @@ ] } ] -} \ No newline at end of file +} diff --git a/daymade-claude-code/statusline-generator/SKILL.md b/daymade-claude-code/statusline-generator/SKILL.md index d21b83bc..0bd73bb6 100644 --- a/daymade-claude-code/statusline-generator/SKILL.md +++ b/daymade-claude-code/statusline-generator/SKILL.md @@ -1,6 +1,6 @@ --- name: statusline-generator -description: Configures and customizes Claude Code statuslines with multi-line layouts, cost tracking via ccusage, git status indicators, and customizable colors. Activates for statusline setup, installation, configuration, customization, color changes, cost display, git status integration, or troubleshooting statusline issues. +description: Configures and customizes Claude Code statuslines with context window display (actual token counts), multi-line layouts, cost tracking via ccusage, git status indicators, human-readable formatting, and customizable colors. Activates for statusline setup, installation, configuration, customization, context window display, color changes, cost display, git status integration, or troubleshooting statusline issues. --- # Statusline Generator @@ -19,6 +19,22 @@ This skill activates for: - Statusline display or cost tracking issues - Git status or path shortening features +## Dependencies + +The statusline script needs to parse JSON from stdin. It auto-detects the available parser: + +| Priority | Tool | Availability | +|----------|------|-------------| +| 1 (preferred) | `jq` | macOS/Linux: `brew install jq` / `apt install jq`; Windows: `choco install jq` or `scoop install jq` | +| 2 (fallback) | `python3` | Pre-installed on macOS and most Linux distros; Windows: `winget install python3` or python.org | + +If neither is available, context window and cost display will be skipped — git branch and path still work. + +Other requirements: +- `git` — for branch status (optional; gracefully skips outside repos) +- `awk` — for number formatting (installed on macOS/Linux by default; Git Bash on Windows includes it) +- `ccusage` — for session/daily cost tracking (optional; gracefully skips if missing) + ## Quick Start ### Basic Installation @@ -33,7 +49,7 @@ Install the default multi-line statusline: 2. Restart Claude Code to see the statusline The default statusline displays: -- **Line 1**: `username (model) [session_cost/daily_cost]` +- **Line 1**: `username (model) [session_cost/daily_cost] ctx: 89.5K/1.0M (9%)` - **Line 2**: `current_path` - **Line 3**: `[git:branch*+]` @@ -61,7 +77,7 @@ Alternatively, manually install by: The statusline uses a 3-line layout optimized for portrait screens: ``` -username (Sonnet 4.5 [1M]) [$0.26/$25.93] +username (Sonnet 4.5 [1M]) [$0.26/$25.93] ctx: 89.5K/1.0M (9%) ~/workspace/java/ready-together-svc [git:feature/branch-name*+] ``` @@ -90,6 +106,26 @@ Model names are automatically shortened: This saves horizontal space while preserving key information. +### Context Window Display + +The statusline can show actual context window usage with human-readable token counts — not just a percentage. This is the most important statusline feature for long sessions. + +Format: +``` +ctx: 88.9K/1.0M (9%) +``` + +Components: +- **Used tokens**: `current_usage.input_tokens + cache_read_input_tokens + cache_creation_input_tokens` +- **Total capacity**: `context_window_size` from the input JSON +- **Percentage**: Shown in parentheses as secondary info +- **Human-readable**: `<1000` → raw, `≥1000` → `X.XK`, `≥1M` → `X.XM` +- **Color-coded**: Green (≤50%), Yellow (51–80%), Red (>80%) + +**Important**: Use `current_usage.*` fields to compute actual context usage. `total_input_tokens` is the session-cumulative count and can exceed the context window size. + +Full statusline input JSON schema (all available fields): `references/context-window-schema.md`. + ### Git Status Indicators Git branch status shows: @@ -140,9 +176,10 @@ Convert to single-line layout by modifying the final `printf`: printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m%s\n\033[01;37m%s\033[00m\n%s' \ "$username" "$model" "$cost_info" "$short_path" "$git_info" -# With: -printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m:\033[01;37m%s\033[00m%s%s' \ - "$username" "$model" "$short_path" "$git_info" "$cost_info" +# With single-line + context: +printf '\033[01;36m[%s]\033[00m \033[01;35m%s\033[00m%s | %bctx: %s/%s (%s%%)\033[00m | \033[01;32m$%s\033[00m' \ + "$model" "$short_dir" "$git_info" "$ctx_color" "$ctx_used_h" "$ctx_size_h" "$ctx_used_pct" "$cost" +# Output: [Sonnet 4.5 [1M]] project (main*) | ctx: 89.5K/1M (9%) | $0.42 ``` ### Disabling Cost Tracking @@ -201,11 +238,14 @@ printf '\033[01;32m%s@%s\033[00m \033[01;36m(%s)\033[00m%s [%s]\n...' \ ## Resources ### scripts/generate_statusline.sh -Main statusline script with all features (multi-line, ccusage, git, colors). Copy this to `~/.claude/statusline.sh` for use. +Main statusline script with all features (context window, multi-line, ccusage, git, colors). Copy to `~/.claude/statusline.sh` for use. ### scripts/install_statusline.sh Automated installation script that copies the statusline script and updates settings.json. +### references/context-window-schema.md +Complete statusline input JSON schema. Documents all fields under `context_window`, `cost`, `model`, `workspace`. Load for context window display implementation or debugging. + ### references/color_codes.md Complete ANSI color code reference for customizing statusline colors. Load when users request color customization. diff --git a/daymade-claude-code/statusline-generator/references/context-window-schema.md b/daymade-claude-code/statusline-generator/references/context-window-schema.md new file mode 100644 index 00000000..aac2b0db --- /dev/null +++ b/daymade-claude-code/statusline-generator/references/context-window-schema.md @@ -0,0 +1,107 @@ +# Statusline Input JSON Schema + +The statusline script receives a JSON object on stdin. This reference documents all known fields. + +## Top-Level Fields + +```json +{ + "session_id": "uuid", + "transcript_path": "/path/to/transcript.jsonl", + "cwd": "/current/working/dir", + "version": "2.1.121", + "fast_mode": false, + "exceeds_200k_tokens": false, + "output_style": { "name": "default" }, + "model": { + "id": "model-id-string", + "display_name": "Human-Readable Model Name" + }, + "workspace": { + "current_dir": "/path/to/cwd", + "project_dir": "/path/to/project", + "added_dirs": ["/additional/dir"] + }, + "cost": { + "total_cost_usd": 1.0868, + "total_duration_ms": 342863, + "total_api_duration_ms": 282122, + "total_lines_added": 0, + "total_lines_removed": 0 + }, + "context_window": { ... }, + "effort": { "level": "max" }, + "thinking": { "enabled": true } +} +``` + +## `context_window` Object (Key Fields) + +```json +{ + "context_window": { + "context_window_size": 1000000, + "used_percentage": 9, + "remaining_percentage": 91, + "total_input_tokens": 83320, + "total_output_tokens": 5456, + "current_usage": { + "input_tokens": 29, + "output_tokens": 163, + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 88832 + } + } +} +``` + +### Field Meanings + +| Field | Type | Description | +|-------|------|-------------| +| `context_window_size` | number | Model's total context window in tokens (e.g., 1000000 = 1M) | +| `used_percentage` | number | Current context usage as percentage (0-100, may have decimals) | +| `remaining_percentage` | number | Remaining context capacity as percentage | +| `total_input_tokens` | number | **Session-cumulative** input tokens (not current context state) | +| `total_output_tokens` | number | **Session-cumulative** output tokens | +| `current_usage.input_tokens` | number | Current turn uncached input tokens | +| `current_usage.output_tokens` | number | Current turn output tokens | +| `current_usage.cache_creation_input_tokens` | number | Tokens written to prompt cache this turn | +| `current_usage.cache_read_input_tokens` | number | Tokens read from prompt cache this turn | + +### Computing Actual Context Usage + +**Correct formula:** +``` +current_context_used = current_usage.input_tokens + + current_usage.cache_read_input_tokens + + current_usage.cache_creation_input_tokens +``` + +This sum should approximately equal `context_window_size * used_percentage / 100`. + +**Do NOT use `total_input_tokens`** — it's the session-level cumulative total, which can exceed the context window size (e.g., 150K total_input_tokens in a 100K context window is normal because old content gets evicted). + +### Model Context Window Sizes (Reference) + +| Model Family | Typical `context_window_size` | +|-------------|------------------------------| +| Claude Opus 4.x | 200,000 | +| Claude Sonnet 4.x | 200,000 | +| Claude Opus 4.5+ | 500,000 | +| Claude Sonnet 4.5+ | 1,000,000 | +| DeepSeek V4 Flash | 200,000 | +| DeepSeek V4 Pro | 1,000,000 | +| GPT-5.x | varies | + +Always read `context_window_size` from the JSON — never hardcode it. + +## `cost` Object + +| Field | Type | Description | +|-------|------|-------------| +| `total_cost_usd` | number | Session total cost in USD (may have >2 decimals) | +| `total_duration_ms` | number | Total wall-clock duration in ms | +| `total_api_duration_ms` | number | Total API call duration in ms | +| `total_lines_added` | number | Lines added this session | +| `total_lines_removed` | number | Lines removed this session | diff --git a/daymade-claude-code/statusline-generator/scripts/generate_statusline.sh b/daymade-claude-code/statusline-generator/scripts/generate_statusline.sh index b191df7d..55a82d62 100644 --- a/daymade-claude-code/statusline-generator/scripts/generate_statusline.sh +++ b/daymade-claude-code/statusline-generator/scripts/generate_statusline.sh @@ -1,80 +1,146 @@ #!/usr/bin/env bash +# Statusline script — multi-line layout with context window, cost, git +# Dependencies: git, awk. jq preferred; python3 used as fallback if jq missing. -# Read JSON input from stdin input=$(cat) -# Extract values from JSON -model_full=$(echo "$input" | jq -r '.model.display_name' 2>/dev/null || echo "Claude") -cwd=$(echo "$input" | jq -r '.workspace.current_dir' 2>/dev/null || pwd) -transcript=$(echo "$input" | jq -r '.transcript_path' 2>/dev/null) +# --- JSON field extraction (jq with python3 fallback) --- +if command -v jq &>/dev/null; then + IFS=$'\t' read -r model_full cwd ctx_used_pct cost_raw ctx_size \ + ctx_input ctx_cache_read ctx_cache_create <<< \ + "$(echo "$input" | jq -r ' + [ + (.model.display_name // "Claude"), + (.workspace.current_dir // ""), + (.context_window.used_percentage // 0), + (.cost.total_cost_usd // 0), + (.context_window.context_window_size // 0), + (.context_window.current_usage.input_tokens // 0), + (.context_window.current_usage.cache_read_input_tokens // 0), + (.context_window.current_usage.cache_creation_input_tokens // 0) + ] | @tsv + ')" +else + # Windows / no-jq: fall back to python3 (stdlib only, no pip needed) + IFS=$'\t' read -r model_full cwd ctx_used_pct cost_raw ctx_size \ + ctx_input ctx_cache_read ctx_cache_create <<< \ + "$(echo "$input" | python3 -c ' +import json, sys +d = json.load(sys.stdin) +cw = d.get("context_window", {}) +cu = cw.get("current_usage", {}) +vals = [ + d.get("model", {}).get("display_name", "Claude"), + d.get("workspace", {}).get("current_dir", ""), + cw.get("used_percentage", 0), + d.get("cost", {}).get("total_cost_usd", 0), + cw.get("context_window_size", 0), + cu.get("input_tokens", 0), + cu.get("cache_read_input_tokens", 0), + cu.get("cache_creation_input_tokens", 0), +] +print("\t".join(str(v) for v in vals)) +')" +fi + +# coerce percentage to int safely +ctx_used_pct=$(printf '%.0f' "${ctx_used_pct:-0}" 2>/dev/null || echo 0) -# Shorten model name: "Sonnet 4.5 (with 1M token context)" -> "Sonnet 4.5 [1M]" -model=$(echo "$model_full" | sed -E 's/(.*)\(with ([0-9]+[KM]) token context\)/\1[\2]/' | sed 's/ *$//') +# --- derived values --- +cwd="${cwd:-$PWD}" +ctx_used=$((ctx_input + ctx_cache_read + ctx_cache_create)) + +# human-readable number formatter +human() { + local n=${1:-0} + if [ "$n" -ge 1000000 ]; then + awk -v v="$n" 'BEGIN {printf "%.1fM", v/1000000}' + elif [ "$n" -ge 1000 ]; then + awk -v v="$n" 'BEGIN {printf "%.1fK", v/1000}' + else + echo "$n" + fi +} + +ctx_used_h=$(human $ctx_used) +ctx_size_h=$(human $ctx_size) +cost=$(echo "$cost_raw" | awk '{printf "%.2f", $1}') + +# model name shortening +model=$(echo "$model_full" \ + | sed -E 's/\(with ([0-9]+[KM]) token context\)/[\1]/' \ + | sed -E 's/\[1m\]/[1M]/' \ + | sed 's/ *$//') -# Get username username=$(whoami) +short_path="${cwd/#$HOME/\~}" -# Shorten path (replace home with ~) -short_path="${cwd/#$HOME/~}" +# --- context color thresholds --- +ctx_color="\033[01;32m" # green ≤50% +if [ "$ctx_used_pct" -gt 80 ]; then + ctx_color="\033[01;31m" # red >80% +elif [ "$ctx_used_pct" -gt 50 ]; then + ctx_color="\033[01;33m" # yellow 51-80% +fi -# Git branch status +# --- git branch status --- git_info="" -if [ -d "$cwd/.git" ] || git -C "$cwd" rev-parse --git-dir >/dev/null 2>&1; then +if [ -d "$cwd/.git" ] || git -C "$cwd" --no-optional-locks rev-parse --git-dir >/dev/null 2>&1; then branch=$(git -C "$cwd" --no-optional-locks branch --show-current 2>/dev/null || echo "detached") - # Check for changes status="" if ! git -C "$cwd" --no-optional-locks diff --quiet 2>/dev/null || \ ! git -C "$cwd" --no-optional-locks diff --cached --quiet 2>/dev/null; then status="*" fi - - # Check for untracked files if [ -n "$(git -C "$cwd" --no-optional-locks ls-files --others --exclude-standard 2>/dev/null)" ]; then status="${status}+" fi - # Format git info with color if [ -n "$status" ]; then - # Red for dirty git_info=$(printf ' \033[01;31m[git:%s%s]\033[00m' "$branch" "$status") else - # Yellow for clean git_info=$(printf ' \033[01;33m[git:%s]\033[00m' "$branch") fi fi -# Cost information using ccusage with caching +# --- cost via ccusage (async, non-blocking; requires jq or python3) --- cost_info="" -cache_file="/tmp/claude_cost_cache_$(date +%Y%m%d_%H%M).txt" - -# Clean old cache files (older than 2 minutes) -find /tmp -name "claude_cost_cache_*.txt" -mmin +2 -delete 2>/dev/null - -if [ -f "$cache_file" ]; then - # Use cached costs - cost_info=$(cat "$cache_file") -else - # Get costs from ccusage (in background to not block statusline on first run) - { - session=$(ccusage session --json --offline -o desc 2>/dev/null | jq -r '.sessions[0].totalCost' 2>/dev/null | xargs printf "%.2f") - daily=$(ccusage daily --json --offline -o desc 2>/dev/null | jq -r '.daily[0].totalCost' 2>/dev/null | xargs printf "%.2f") +if command -v jq &>/dev/null || command -v python3 &>/dev/null; then + cache_file="/tmp/claude_cost_cache_$(date +%Y%m%d_%H%M).txt" + find /tmp -name "claude_cost_cache_*.txt" -mmin +2 -delete 2>/dev/null - if [ -n "$session" ] && [ -n "$daily" ] && [ "$session" != "" ] && [ "$daily" != "" ]; then - printf ' \033[01;35m[$%s/$%s]\033[00m' "$session" "$daily" > "$cache_file" + if [ -f "$cache_file" ]; then + cost_info=$(cat "$cache_file") + else + { + if command -v jq &>/dev/null; then + session=$(ccusage session --json --offline -o desc 2>/dev/null | jq -r '.sessions[0].totalCost' 2>/dev/null | xargs printf "%.2f" 2>/dev/null) + daily=$(ccusage daily --json --offline -o desc 2>/dev/null | jq -r '.daily[0].totalCost' 2>/dev/null | xargs printf "%.2f" 2>/dev/null) + else + session=$(ccusage session --json --offline -o desc 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(f\"{d['sessions'][0]['totalCost']:.2f}\")" 2>/dev/null) + daily=$(ccusage daily --json --offline -o desc 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(f\"{d['daily'][0]['totalCost']:.2f}\")" 2>/dev/null) + fi + if [ -n "$session" ] && [ -n "$daily" ] && [ "$session" != "" ] && [ "$daily" != "" ]; then + printf ' \033[01;35m[$%s/$%s]\033[00m' "$session" "$daily" > "$cache_file" + fi + } & + prev_cache=$(find /tmp -name "claude_cost_cache_*.txt" -mmin -10 2>/dev/null | head -1) + if [ -f "$prev_cache" ]; then + cost_info=$(cat "$prev_cache") fi - } & - - # Try to use previous cache while new one is being generated - prev_cache=$(find /tmp -name "claude_cost_cache_*.txt" -mmin -10 2>/dev/null | head -1) - if [ -f "$prev_cache" ]; then - cost_info=$(cat "$prev_cache") fi fi -# Print the final status line (multi-line format for portrait screens) -# Line 1: username (model) [costs] -# Line 2: path (bright white for better visibility) +# --- context display string --- +ctx_display="" +if [ "$ctx_size" -gt 0 ]; then + ctx_display=$(printf " ${ctx_color}ctx: %s/%s (%s%%)" "$ctx_used_h" "$ctx_size_h" "$ctx_used_pct") +fi + +# --- output: 3-line layout --- +# Line 1: username (model) [costs] ctx: used/total (pct%) +# Line 2: path # Line 3: [git:branch] -printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m%s\n\033[01;37m%s\033[00m\n%s' \ - "$username" "$model" "$cost_info" "$short_path" "$git_info" \ No newline at end of file +printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m%s%s\n\033[01;37m%s\033[00m\n%s' \ + "$username" "$model" "$cost_info" "$ctx_display" "$short_path" "$git_info" From 5a1b269bf5f17353b5ebdf6967062ceecf0fab43 Mon Sep 17 00:00:00 2001 From: daymade Date: Thu, 7 May 2026 13:13:02 +0800 Subject: [PATCH 126/137] chore: remove ralph loop generated artifacts Remove files accidentally generated by an unexpected ralph loop run: - package.json, package-lock.json (not a Node.js project) - EOF, TECHNICAL_DEBT.md (spurious files) - .claude/archive/ralph-*.md (10 archive files) Co-Authored-By: Claude Opus 4.7 --- .claude/archive/ralph-loop.local.md | 10 - .claude/archive/ralph-round-68.md | 21 - .claude/archive/ralph-round-69.md | 29 - .claude/archive/ralph-round-70.md | 33 - .claude/archive/ralph-round-71.md | 37 - .claude/archive/ralph-round-72.md | 24 - .claude/archive/ralph-round-73.md | 24 - .claude/archive/ralph-round-74.md | 24 - .claude/archive/ralph-round-75.md | 24 - EOF | 0 TECHNICAL_DEBT.md | 242 ---- package-lock.json | 1956 --------------------------- package.json | 15 - 13 files changed, 2439 deletions(-) delete mode 100644 .claude/archive/ralph-loop.local.md delete mode 100644 .claude/archive/ralph-round-68.md delete mode 100644 .claude/archive/ralph-round-69.md delete mode 100644 .claude/archive/ralph-round-70.md delete mode 100644 .claude/archive/ralph-round-71.md delete mode 100644 .claude/archive/ralph-round-72.md delete mode 100644 .claude/archive/ralph-round-73.md delete mode 100644 .claude/archive/ralph-round-74.md delete mode 100644 .claude/archive/ralph-round-75.md delete mode 100644 EOF delete mode 100644 TECHNICAL_DEBT.md delete mode 100644 package-lock.json delete mode 100644 package.json diff --git a/.claude/archive/ralph-loop.local.md b/.claude/archive/ralph-loop.local.md deleted file mode 100644 index 17f66e84..00000000 --- a/.claude/archive/ralph-loop.local.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -active: true -iteration: 390 -session_id: -max_iterations: 0 -completion_promise: null -started_at: "2026-04-11T19:55:07Z" ---- - -使用这个技能，直到我们超越所有的竞品，每一轮开始的时候，你都要去分析我们当前的这个skill和我们所有的竞品的差距是什么，取其精华，去其糟粕 diff --git a/.claude/archive/ralph-round-68.md b/.claude/archive/ralph-round-68.md deleted file mode 100644 index 601c7bb5..00000000 --- a/.claude/archive/ralph-round-68.md +++ /dev/null @@ -1,21 +0,0 @@ -# Ralph Loop Round 68 - Completed - -## 竞品差距分析 -当前 v3.25.0 vs 竞品能力矩阵： -- wcplusPro: 付费护城河 = 自动化工作流 -- 新榜/西瓜数据: 无工作流功能 -- 开源竞品: 完全无此功能 - -Gap: 工作流引擎是 wcplusPro 的付费核心功能，无任何开源实现 - -## 实施结果 -- 自动化工作流引擎 v1.0 ✅ -- RESTful API + WebSocket ✅ -- CLI w workflow 命令组 ✅ -- 46个唯一支持特性 ✅ - -## 版本 -v3.25.0 → v3.26.0 - -## 提交 -3f61ffd feat(wechat-article-scraper): 自动化工作流引擎 v1.0 diff --git a/.claude/archive/ralph-round-69.md b/.claude/archive/ralph-round-69.md deleted file mode 100644 index 802631a1..00000000 --- a/.claude/archive/ralph-round-69.md +++ /dev/null @@ -1,29 +0,0 @@ -# Ralph Loop Round 69 - Completed - -## 竞品差距分析 -当前 v3.26.0 vs 竞品能力矩阵： -- wcplusPro: 第二付费护城河 = 团队协作 + 数据同步到Notion/语雀 -- 新榜/西瓜数据: 无团队协作功能 -- 开源竞品: 完全无此功能 - -Gap: 团队协作是 wcplusPro 的另一付费核心功能，无任何开源实现 - -## 实施结果 -- 团队协作系统 v1.0 ✅ - - 多用户管理、RBAC权限(admin/member/viewer) - - 团队共享工作区、文章收藏夹 - - 文章标注系统(标签/评论/高亮) - - 邀请码机制 -- 第三方集成 v1.0 ✅ - - Notion API 数据库同步 - - 语雀 API 知识库归档 - - Airtable 表格同步 -- w team CLI命令组 ✅ -- w sync CLI命令组 ✅ -- 48个唯一支持特性 ✅ - -## 版本 -v3.26.0 → v3.27.0 - -## 提交 -81fcc97 feat(wechat-article-scraper): 团队协作与第三方集成 v1.0 diff --git a/.claude/archive/ralph-round-70.md b/.claude/archive/ralph-round-70.md deleted file mode 100644 index 6b38b73f..00000000 --- a/.claude/archive/ralph-round-70.md +++ /dev/null @@ -1,33 +0,0 @@ -# Ralph Loop Round 70 - Completed - -## 竞品差距分析 -当前 v3.27.0 vs 竞品能力矩阵： -- wcplusPro: 付费功能 = 数据导出(Excel/PDF/Word) + 批量操作 + 高级筛选 -- 新榜/西瓜数据: 基础导出功能 -- 开源竞品: 无此功能 - -Gap: 丰富的数据导出和批量操作是 wcplusPro 的核心付费功能 - -## 实施结果 -- 多格式导出引擎 v1.0 ✅ - - Excel: 样式美化、多sheet、统计图表 - - PDF: 中文支持、分页优化 - - Word: 格式保持、目录生成 - - Markdown/JSON/CSV: 标准格式 - - 导出模板系统（自定义字段、样式） -- 高级筛选系统 v1.0 ✅ - - 多条件组合筛选（AND/OR逻辑） - - 条件类型: 时间/公众号/关键词/阅读量/标签 - - 筛选模板保存/加载 -- 批量操作引擎 v1.0 ✅ - - 批量导出/编辑/同步/删除 - - 任务队列和进度追踪 - - 操作历史记录 -- w export/filter/batch CLI命令组 ✅ -- 51个唯一支持特性 ✅ - -## 版本 -v3.27.0 → v3.28.0 - -## 提交 -37211c6 feat(wechat-article-scraper): 批量操作与多格式导出 v1.0 diff --git a/.claude/archive/ralph-round-71.md b/.claude/archive/ralph-round-71.md deleted file mode 100644 index 3d778ed4..00000000 --- a/.claude/archive/ralph-round-71.md +++ /dev/null @@ -1,37 +0,0 @@ -# Ralph Loop Round 71 - Completed - -## 竞品差距分析 -当前 v3.28.0 vs 竞品能力矩阵： -- wcplusPro: 有成熟的 Chrome 扩展用于一键采集 -- 新榜/西瓜数据: 有自己的浏览器插件 -- 开源竞品: 无此功能 - -Gap: 浏览器扩展是用户便捷使用的重要入口 - -## 实施结果 -- 浏览器扩展 v2.0 完善 ✅ - - Chrome/Firefox Manifest V3 扩展 - - Content Script: 页面内容提取、文章数据解析 - - 标题、作者、发布时间提取 - - 正文内容提取（段落、HTML） - - 图片提取（data-src、data-backsrc） - - 视频提取 - - 互动数据读取（阅读数、点赞数、在看数） - - Background Script: 右键菜单、快捷键、自动下载 - - 右键菜单：抓取文章、抓取并下载图片、打开仪表盘 - - 快捷键：Ctrl+Shift+S 快速抓取 - - Tab 状态监听和徽章显示 - - Popup UI: 格式选择、进度显示、结果操作 - - 状态检测（是否在文章页面） - - 格式选择：Markdown/HTML/JSON - - 设置开关：保存本地、上传服务器、自动分类 - - 进度条和结果展示 - - 查看/下载/复制结果 - - w extension CLI命令: install/pack/check -- v3.29.0, 52个唯一支持特性 ✅ - -## 版本 -v3.28.0 → v3.29.0 - -## 提交 -6c5f24d feat(wechat-article-scraper): 浏览器扩展 v2.0 完善 diff --git a/.claude/archive/ralph-round-72.md b/.claude/archive/ralph-round-72.md deleted file mode 100644 index f1742227..00000000 --- a/.claude/archive/ralph-round-72.md +++ /dev/null @@ -1,24 +0,0 @@ -# Ralph Loop Round 72 - Completed - -## 竞品差距分析 -当前 v3.29.0 vs 竞品能力矩阵： -- wcplusPro: 无舆情监控功能 -- 新榜/西瓜数据: 有数据监控但无敏感内容检测 -- 开源竞品: 无此功能 - -Gap: 舆情监控是企业级用户的刚需，特别是敏感内容检测和危机预警 - -## 实施结果 -- 舆情监控系统 v1.0 ✅ - - 敏感词检测: 6大类别敏感词库 (政治/色情/暴力/赌博/毒品/诈骗) - - 品牌提及追踪: 追踪品牌/关键词在文章中的提及 - - 情感趋势分析: 正面/中性/负面情绪监控 - - 危机预警系统: 异常传播速度检测、敏感内容预警 - - CLI集成: `w sentiment` 命令 (word/brand/alert/trend/stats/scan) -- v3.30.0, 53个唯一支持特性 ✅ - -## 版本 -v3.29.0 → v3.30.0 - -## 提交 -7719c53 feat(wechat-article-scraper): 舆情监控系统 v1.0 diff --git a/.claude/archive/ralph-round-73.md b/.claude/archive/ralph-round-73.md deleted file mode 100644 index 8e3e25cd..00000000 --- a/.claude/archive/ralph-round-73.md +++ /dev/null @@ -1,24 +0,0 @@ -# Ralph Loop Round 73 - Completed - -## 竞品差距分析 -当前 v3.30.0 vs 竞品能力矩阵： -- wcplusPro: 有强大的数据可视化图表和竞品分析（付费功能） -- 新榜/西瓜数据: 有数据监控看板、热点追踪 -- 开源竞品: 无此功能 - -Gap: 数据可视化仪表盘、竞品分析、AI洞察报告是 wcplusPro/新榜的核心付费功能 - -## 实施结果 -- 数据可视化与智能分析系统 v1.0 ✅ - - 数据仪表盘 (analytics_dashboard.py): ECharts配置、阅读趋势、互动热力图、文章排行、公众号健康度 - - 竞品分析系统 (competitor_analyzer.py): 多账号对比、竞争力评分、内容策略分析、最佳发布时间 - - AI智能洞察 (ai_insights.py): 趋势分析、异常检测、自动运营建议、健康度评分 - - 热点追踪 (hot_topics.py): 自动发现热点、话题聚类、传播速度监控、生命周期追踪 - - CLI集成: `w analytics` 统一入口 (trends/top/metrics/report/heatmap/compare/insights/topics) -- v3.31.0, 57个唯一支持特性 ✅ - -## 版本 -v3.30.0 → v3.31.0 - -## 提交 -e4109dd feat(wechat-article-scraper): 数据可视化与智能分析系统 v1.0 diff --git a/.claude/archive/ralph-round-74.md b/.claude/archive/ralph-round-74.md deleted file mode 100644 index 091a332e..00000000 --- a/.claude/archive/ralph-round-74.md +++ /dev/null @@ -1,24 +0,0 @@ -# Ralph Loop Round 74 - Completed - -## 竞品差距分析 -当前 v3.31.0 vs 竞品能力矩阵： -- wcplusPro: 有AI写作助手（标题生成、摘要、改写） -- 新榜/西瓜数据: 有素材库管理、自动排版 -- 开源竞品: 无此功能 - -Gap: AI写作辅助、素材库管理是竞品的付费核心功能 - -## 实施结果 -- 智能写作助手 v1.0 ✅ - - AI标题生成器 (writing_assistant.py/TitleGenerator): 8种爆款公式、A/B测试、CTR预测 - - 智能摘要生成 (Summarizer): 5种风格（新闻/营销/极简/故事/要点） - - 内容改写润色 (ContentRewriter): 5种风格转换、语气调整 - - 素材库管理 (material_library.py): 文案/图片/链接收藏、标签分类、全文搜索 - - CLI集成: `w writing` 命令（title/summary/rewrite/analyze/material） -- v3.32.0, 61个唯一支持特性 ✅ - -## 版本 -v3.31.0 → v3.32.0 - -## 提交 -b8bbc08 feat(wechat-article-scraper): 智能写作助手 v1.0 diff --git a/.claude/archive/ralph-round-75.md b/.claude/archive/ralph-round-75.md deleted file mode 100644 index 9baab224..00000000 --- a/.claude/archive/ralph-round-75.md +++ /dev/null @@ -1,24 +0,0 @@ -# Ralph Loop Round 75 - Completed - -## 竞品差距分析 -当前 v3.32.0 vs 竞品能力矩阵： -- wcplusPro: 有自动采集、定时任务（付费功能） -- 新榜/西瓜数据: 有定时监控、通知提醒 -- 开源竞品: 无此功能 - -Gap: 定时任务调度、自动采集、通知提醒是竞品的付费核心功能 - -## 实施结果 -- 定时任务与自动化系统 v1.0 ✅ - - 任务调度器 (task_scheduler.py): Cron表达式解析、任务调度循环、多任务类型支持 - - 任务执行日志: 执行记录、成功率统计、错误追踪、历史查询 - - 通知系统 (notification_system.py): 邮件/SMTP、Webhook回调、5种通知模板、通知历史 - - 5种内置任务类型: scrape/export/backup/cleanup/custom - - CLI集成: `w scheduler` 命令（create/list/run/toggle/delete/history/stats/daemon） -- v3.33.0, 64个唯一支持特性 ✅ - -## 版本 -v3.32.0 → v3.33.0 - -## 提交 -bccf5f3 feat(wechat-article-scraper): 定时任务与自动化系统 v1.0 diff --git a/EOF b/EOF deleted file mode 100644 index e69de29b..00000000 diff --git a/TECHNICAL_DEBT.md b/TECHNICAL_DEBT.md deleted file mode 100644 index dcdac309..00000000 --- a/TECHNICAL_DEBT.md +++ /dev/null @@ -1,242 +0,0 @@ -# 技术债务清单 - -## 当前债务状况 - -| 债务类型 | 严重程度 | 预计修复时间 | 阻塞发布 | -|---------|---------|-------------|----------| -| 测试覆盖不足 | 🔴 极高 | 4周 | ✅ 是 | -| 缺少CI/CD | 🔴 极高 | 1周 | ✅ 是 | -| 错误处理不完善 | 🟠 高 | 2周 | ✅ 是 | -| 性能未优化 | 🟠 高 | 2周 | ❌ 否 | -| 文档不完整 | 🟡 中 | 1周 | ❌ 否 | -| 代码重复 | 🟡 中 | 1周 | ❌ 否 | - -## 详细债务清单 - -### 🔴 极高优先级 - -#### 1. 测试覆盖不足（5% → 目标80%） - -**现状:** -- 总源文件: 2,999 -- 测试文件: 5 -- 覆盖率: ~5% - -**缺失测试:** -- [ ] `inbox-engine.ts` - 核心逻辑无测试 -- [ ] `annotation-engine.ts` - 核心逻辑无测试 -- [ ] `sync-engine.ts` - 核心逻辑无测试 -- [ ] `tts-engine.ts` - 新增无测试 -- [ ] `reading-agent-sdk.ts` - 复杂逻辑无测试 -- [ ] 所有API routes - 无集成测试 - -**风险:** -- 重构困难 -- 回归bug无法发现 -- 无法 confident 地发布 - -**修复方案:** -```bash -# 优先级1: 核心引擎单元测试 -- inbox-engine.test.ts -- annotation-engine.test.ts -- sync-engine.test.ts - -# 优先级2: API集成测试 -- api/articles.test.ts -- api/scrape.test.ts -- api/sync.test.ts - -# 优先级3: E2E测试 -- scrape-to-read.spec.ts (已完成) -- reading-annotations.spec.ts (已完成) -- tts-stagehand.spec.ts (已完成) -``` - -#### 2. 缺少CI/CD流水线 - -**现状:** -- 无自动化测试 -- 无自动化部署 -- 无代码质量检查 - -**需要建立:** -- [ ] GitHub Actions workflow - - [ ] Lint check - - [ ] Type check - - [ ] Unit tests - - [ ] E2E tests - - [ ] Security scan - - [ ] Deploy to staging - - [ ] Deploy to production - -**参考实现:** -Omnivore 的 `.github/workflows/` 配置 - -#### 3. 错误处理不完善 - -**现状:** -- 大量 `throw error` 未处理 -- 用户看到的是原始错误信息 -- 无错误边界(Error Boundaries) - -**需要修复:** -- [ ] 统一错误处理中间件 -- [ ] 用户友好的错误提示 -- [ ] Sentry上报所有未捕获错误 -- [ ] React Error Boundaries - -### 🟠 高优先级 - -#### 4. 性能未优化 - -**已知问题:** -- [ ] 首屏加载无骨架屏 -- [ ] 大文章渲染卡顿 -- [ ] 图片懒加载不完善 -- [ ] 无虚拟滚动 - -**待测量指标:** -``` -LCP (Largest Contentful Paint): 目标 < 2.5s -FID (First Input Delay): 目标 < 100ms -CLS (Cumulative Layout Shift): 目标 < 0.1 -TTI (Time to Interactive): 目标 < 3.8s -``` - -#### 5. 数据库查询未优化 - -**潜在问题:** -- [ ] 缺少复合索引 -- [ ] N+1查询问题 -- [ ] 大数据表无分区 - -**需要:** -- [ ] 查询性能分析 -- [ ] EXPLAIN ANALYZE 所有慢查询 -- [ ] 连接池配置优化 - -### 🟡 中优先级 - -#### 6. 文档不完整 - -**缺失:** -- [ ] API文档 (OpenAPI/Swagger) -- [ ] 架构决策记录 (ADR) -- [ ] 部署指南 -- [ ] 贡献者指南 - -#### 7. 代码重复 - -**发现重复:** -- [ ] 日期格式化函数 (多处) -- [ ] API错误处理逻辑 (多处) -- [ ] 类型定义分散 - -## 还债计划 - -### Sprint 1: 测试基础 (Week 1-2) - -```bash -# 目标: 测试覆盖率 5% → 30% - -Week 1: -- [ ] 设置 Vitest + React Testing Library -- [ ] inbox-engine.ts 单元测试 -- [ ] annotation-engine.ts 单元测试 - -Week 2: -- [ ] sync-engine.ts 单元测试 -- [ ] tts-engine.ts 单元测试 -- [ ] 核心组件单元测试 -``` - -### Sprint 2: CI/CD + 错误处理 (Week 3-4) - -```bash -# 目标: 自动化流水线 + 错误监控 - -Week 3: -- [ ] GitHub Actions workflow -- [ ] 自动化测试触发 -- [ ] Sentry生产环境配置 - -Week 4: -- [ ] 错误边界组件 -- [ ] 统一错误处理 -- [ ] 用户友好错误提示 -``` - -### Sprint 3: 性能优化 (Week 5-6) - -```bash -# 目标: Web Vitals 达标 - -Week 5: -- [ ] 性能基线测量 -- [ ] 图片优化 -- [ ] 代码分割 - -Week 6: -- [ ] 虚拟滚动 (大文章) -- [ ] 骨架屏 -- [ ] 缓存策略 -``` - -### Sprint 4: 清理 (Week 7-8) - -```bash -# 目标: 代码质量提升 - -Week 7: -- [ ] 重构重复代码 -- [ ] 类型定义集中化 -- [ ] 数据库索引优化 - -Week 8: -- [ ] 文档完善 -- [ ] 架构图更新 -- [ ] 技术债务复盘 -``` - -## 债务预防 - -### 代码审查清单 - -- [ ] 新功能必须包含测试 -- [ ] 复杂逻辑必须有注释 -- [ ] API变更必须更新文档 -- [ ] 性能敏感代码必须有基准测试 - -### 自动化防护 - -```yaml -# .github/workflows/quality.yml -- name: Test Coverage - run: | - npm run test:coverage - # 失败如果覆盖率 < 60% - -- name: Bundle Size - run: | - npm run analyze - # 失败如果 bundle > 500KB - -- name: Performance Budget - run: | - lhci autorun - # 失败如果 Web Vitals 不达标 -``` - -## 债务追踪 - -| 日期 | 债务项 | 状态 | 备注 | -|-----|-------|------|------| -| 2025-04-12 | 测试覆盖 | 🟡 进行中 | 已规划E2E | -| 2025-04-12 | CI/CD | 🔴 未开始 | 优先级P0 | -| 2025-04-12 | 错误处理 | 🔴 未开始 | 优先级P0 | - ---- - -最后更新: Round 91 -下次审查: Round 95 diff --git a/package-lock.json b/package-lock.json deleted file mode 100644 index bcc54866..00000000 --- a/package-lock.json +++ /dev/null @@ -1,1956 +0,0 @@ -{ - "name": "claude-code-skills", - "lockfileVersion": 3, - "requires": true, - "packages": { - "": { - "dependencies": { - "@anthropic-ai/sdk": "^0.88.0", - "@tanstack/react-virtual": "^3.13.23", - "@types/node": "^25.6.0", - "commander": "^14.0.3", - "dotenv": "^17.4.1", - "neo4j-driver": "^6.0.1", - "sqlite": "^5.1.1", - "sqlite3": "^6.0.1", - "tsx": "^4.21.0", - "typescript": "^6.0.2", - "zod": "^4.3.6" - } - }, - "node_modules/@anthropic-ai/sdk": { - "version": "0.88.0", - "resolved": "https://registry.npmjs.org/@anthropic-ai/sdk/-/sdk-0.88.0.tgz", - "integrity": "sha512-QQOtB5U9ZBJQj6y1ICmDZl14LWa4JCiJRoihI+0yuZ4OjbONrakP0yLwPv4DJFb3VYCtQM31bTOpCBMs2zghPw==", - "license": "MIT", - "dependencies": { - "json-schema-to-ts": "^3.1.1" - }, - "bin": { - "anthropic-ai-sdk": "bin/cli" - }, - "peerDependencies": { - "zod": "^3.25.0 || ^4.0.0" - }, - "peerDependenciesMeta": { - "zod": { - "optional": true - } - } - }, - "node_modules/@babel/runtime": { - "version": "7.29.2", - "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.2.tgz", - "integrity": "sha512-JiDShH45zKHWyGe4ZNVRrCjBz8Nh9TMmZG1kh4QTK8hCBTWBi8Da+i7s1fJw7/lYpM4ccepSNfqzZ/QvABBi5g==", - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@esbuild/aix-ppc64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz", - "integrity": "sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg==", - "cpu": [ - "ppc64" - ], - "license": "MIT", - "optional": true, - "os": [ - "aix" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/android-arm": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.27.7.tgz", - "integrity": "sha512-jbPXvB4Yj2yBV7HUfE2KHe4GJX51QplCN1pGbYjvsyCZbQmies29EoJbkEc+vYuU5o45AfQn37vZlyXy4YJ8RQ==", - "cpu": [ - "arm" - ], - "license": "MIT", - "optional": true, - "os": [ - "android" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/android-arm64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.27.7.tgz", - "integrity": "sha512-62dPZHpIXzvChfvfLJow3q5dDtiNMkwiRzPylSCfriLvZeq0a1bWChrGx/BbUbPwOrsWKMn8idSllklzBy+dgQ==", - "cpu": [ - "arm64" - ], - "license": "MIT", - "optional": true, - "os": [ - "android" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/android-x64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.27.7.tgz", - "integrity": "sha512-x5VpMODneVDb70PYV2VQOmIUUiBtY3D3mPBG8NxVk5CogneYhkR7MmM3yR/uMdITLrC1ml/NV1rj4bMJuy9MCg==", - "cpu": [ - "x64" - ], - "license": "MIT", - "optional": true, - "os": [ - "android" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/darwin-arm64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.27.7.tgz", - "integrity": "sha512-5lckdqeuBPlKUwvoCXIgI2D9/ABmPq3Rdp7IfL70393YgaASt7tbju3Ac+ePVi3KDH6N2RqePfHnXkaDtY9fkw==", - "cpu": [ - "arm64" - ], - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/darwin-x64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.27.7.tgz", - "integrity": "sha512-rYnXrKcXuT7Z+WL5K980jVFdvVKhCHhUwid+dDYQpH+qu+TefcomiMAJpIiC2EM3Rjtq0sO3StMV/+3w3MyyqQ==", - "cpu": [ - "x64" - ], - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/freebsd-arm64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.27.7.tgz", - "integrity": "sha512-B48PqeCsEgOtzME2GbNM2roU29AMTuOIN91dsMO30t+Ydis3z/3Ngoj5hhnsOSSwNzS+6JppqWsuhTp6E82l2w==", - "cpu": [ - "arm64" - ], - "license": "MIT", - "optional": true, - "os": [ - "freebsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/freebsd-x64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.27.7.tgz", - "integrity": "sha512-jOBDK5XEjA4m5IJK3bpAQF9/Lelu/Z9ZcdhTRLf4cajlB+8VEhFFRjWgfy3M1O4rO2GQ/b2dLwCUGpiF/eATNQ==", - "cpu": [ - "x64" - ], - "license": "MIT", - "optional": true, - "os": [ - "freebsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-arm": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.27.7.tgz", - "integrity": "sha512-RkT/YXYBTSULo3+af8Ib0ykH8u2MBh57o7q/DAs3lTJlyVQkgQvlrPTnjIzzRPQyavxtPtfg0EopvDyIt0j1rA==", - "cpu": [ - "arm" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-arm64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.27.7.tgz", - "integrity": "sha512-RZPHBoxXuNnPQO9rvjh5jdkRmVizktkT7TCDkDmQ0W2SwHInKCAV95GRuvdSvA7w4VMwfCjUiPwDi0ZO6Nfe9A==", - "cpu": [ - "arm64" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-ia32": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.27.7.tgz", - "integrity": "sha512-GA48aKNkyQDbd3KtkplYWT102C5sn/EZTY4XROkxONgruHPU72l+gW+FfF8tf2cFjeHaRbWpOYa/uRBz/Xq1Pg==", - "cpu": [ - "ia32" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-loong64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.27.7.tgz", - "integrity": "sha512-a4POruNM2oWsD4WKvBSEKGIiWQF8fZOAsycHOt6JBpZ+JN2n2JH9WAv56SOyu9X5IqAjqSIPTaJkqN8F7XOQ5Q==", - "cpu": [ - "loong64" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-mips64el": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.27.7.tgz", - "integrity": "sha512-KabT5I6StirGfIz0FMgl1I+R1H73Gp0ofL9A3nG3i/cYFJzKHhouBV5VWK1CSgKvVaG4q1RNpCTR2LuTVB3fIw==", - "cpu": [ - "mips64el" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-ppc64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.27.7.tgz", - "integrity": "sha512-gRsL4x6wsGHGRqhtI+ifpN/vpOFTQtnbsupUF5R5YTAg+y/lKelYR1hXbnBdzDjGbMYjVJLJTd2OFmMewAgwlQ==", - "cpu": [ - "ppc64" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-riscv64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.27.7.tgz", - "integrity": "sha512-hL25LbxO1QOngGzu2U5xeXtxXcW+/GvMN3ejANqXkxZ/opySAZMrc+9LY/WyjAan41unrR3YrmtTsUpwT66InQ==", - "cpu": [ - "riscv64" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-s390x": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.27.7.tgz", - "integrity": "sha512-2k8go8Ycu1Kb46vEelhu1vqEP+UeRVj2zY1pSuPdgvbd5ykAw82Lrro28vXUrRmzEsUV0NzCf54yARIK8r0fdw==", - "cpu": [ - "s390x" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-x64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.27.7.tgz", - "integrity": "sha512-hzznmADPt+OmsYzw1EE33ccA+HPdIqiCRq7cQeL1Jlq2gb1+OyWBkMCrYGBJ+sxVzve2ZJEVeePbLM2iEIZSxA==", - "cpu": [ - "x64" - ], - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/netbsd-arm64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.27.7.tgz", - "integrity": "sha512-b6pqtrQdigZBwZxAn1UpazEisvwaIDvdbMbmrly7cDTMFnw/+3lVxxCTGOrkPVnsYIosJJXAsILG9XcQS+Yu6w==", - "cpu": [ - "arm64" - ], - "license": "MIT", - "optional": true, - "os": [ - "netbsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/netbsd-x64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.27.7.tgz", - "integrity": "sha512-OfatkLojr6U+WN5EDYuoQhtM+1xco+/6FSzJJnuWiUw5eVcicbyK3dq5EeV/QHT1uy6GoDhGbFpprUiHUYggrw==", - "cpu": [ - "x64" - ], - "license": "MIT", - "optional": true, - "os": [ - "netbsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/openbsd-arm64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.27.7.tgz", - "integrity": "sha512-AFuojMQTxAz75Fo8idVcqoQWEHIXFRbOc1TrVcFSgCZtQfSdc1RXgB3tjOn/krRHENUB4j00bfGjyl2mJrU37A==", - "cpu": [ - "arm64" - ], - "license": "MIT", - "optional": true, - "os": [ - "openbsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/openbsd-x64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.27.7.tgz", - "integrity": "sha512-+A1NJmfM8WNDv5CLVQYJ5PshuRm/4cI6WMZRg1by1GwPIQPCTs1GLEUHwiiQGT5zDdyLiRM/l1G0Pv54gvtKIg==", - "cpu": [ - "x64" - ], - "license": "MIT", - "optional": true, - "os": [ - "openbsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/openharmony-arm64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.27.7.tgz", - "integrity": "sha512-+KrvYb/C8zA9CU/g0sR6w2RBw7IGc5J2BPnc3dYc5VJxHCSF1yNMxTV5LQ7GuKteQXZtspjFbiuW5/dOj7H4Yw==", - "cpu": [ - "arm64" - ], - "license": "MIT", - "optional": true, - "os": [ - "openharmony" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/sunos-x64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.27.7.tgz", - "integrity": "sha512-ikktIhFBzQNt/QDyOL580ti9+5mL/YZeUPKU2ivGtGjdTYoqz6jObj6nOMfhASpS4GU4Q/Clh1QtxWAvcYKamA==", - "cpu": [ - "x64" - ], - "license": "MIT", - "optional": true, - "os": [ - "sunos" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/win32-arm64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.27.7.tgz", - "integrity": "sha512-7yRhbHvPqSpRUV7Q20VuDwbjW5kIMwTHpptuUzV+AA46kiPze5Z7qgt6CLCK3pWFrHeNfDd1VKgyP4O+ng17CA==", - "cpu": [ - "arm64" - ], - "license": "MIT", - "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/win32-ia32": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.27.7.tgz", - "integrity": "sha512-SmwKXe6VHIyZYbBLJrhOoCJRB/Z1tckzmgTLfFYOfpMAx63BJEaL9ExI8x7v0oAO3Zh6D/Oi1gVxEYr5oUCFhw==", - "cpu": [ - "ia32" - ], - "license": "MIT", - "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/win32-x64": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.27.7.tgz", - "integrity": "sha512-56hiAJPhwQ1R4i+21FVF7V8kSD5zZTdHcVuRFMW0hn753vVfQN8xlx4uOPT4xoGH0Z/oVATuR82AiqSTDIpaHg==", - "cpu": [ - "x64" - ], - "license": "MIT", - "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@gar/promise-retry": { - "version": "1.0.3", - "resolved": "https://registry.npmjs.org/@gar/promise-retry/-/promise-retry-1.0.3.tgz", - "integrity": "sha512-GmzA9ckNokPypTg10pgpeHNQe7ph+iIKKmhKu3Ob9ANkswreCx7R3cKmY781K8QK3AqVL3xVh9A42JvIAbkkSA==", - "license": "MIT", - "optional": true, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/@isaacs/fs-minipass": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/@isaacs/fs-minipass/-/fs-minipass-4.0.1.tgz", - "integrity": "sha512-wgm9Ehl2jpeqP3zw/7mo3kRHFp5MEDhqAdwy1fTGkHAwnkGOVsgpvQhL8B5n1qlb01jV3n/bI0ZfZp5lWA1k4w==", - "license": "ISC", - "dependencies": { - "minipass": "^7.0.4" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "node_modules/@npmcli/agent": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/@npmcli/agent/-/agent-4.0.0.tgz", - "integrity": "sha512-kAQTcEN9E8ERLVg5AsGwLNoFb+oEG6engbqAU2P43gD4JEIkNGMHdVQ096FsOAAYpZPB0RSt0zgInKIAS1l5QA==", - "license": "ISC", - "optional": true, - "dependencies": { - "agent-base": "^7.1.0", - "http-proxy-agent": "^7.0.0", - "https-proxy-agent": "^7.0.1", - "lru-cache": "^11.2.1", - "socks-proxy-agent": "^8.0.3" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/@npmcli/fs": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/@npmcli/fs/-/fs-5.0.0.tgz", - "integrity": "sha512-7OsC1gNORBEawOa5+j2pXN9vsicaIOH5cPXxoR6fJOmH6/EXpJB2CajXOu1fPRFun2m1lktEFX11+P89hqO/og==", - "license": "ISC", - "optional": true, - "dependencies": { - "semver": "^7.3.5" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/@npmcli/redact": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/@npmcli/redact/-/redact-4.0.0.tgz", - "integrity": "sha512-gOBg5YHMfZy+TfHArfVogwgfBeQnKbbGo3pSUyK/gSI0AVu+pEiDVcKlQb0D8Mg1LNRZILZ6XG8I5dJ4KuAd9Q==", - "license": "ISC", - "optional": true, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/@tanstack/react-virtual": { - "version": "3.13.23", - "resolved": "https://registry.npmjs.org/@tanstack/react-virtual/-/react-virtual-3.13.23.tgz", - "integrity": "sha512-XnMRnHQ23piOVj2bzJqHrRrLg4r+F86fuBcwteKfbIjJrtGxb4z7tIvPVAe4B+4UVwo9G4Giuz5fmapcrnZ0OQ==", - "license": "MIT", - "dependencies": { - "@tanstack/virtual-core": "3.13.23" - }, - "funding": { - "type": "github", - "url": "https://github.com/sponsors/tannerlinsley" - }, - "peerDependencies": { - "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", - "react-dom": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0" - } - }, - "node_modules/@tanstack/virtual-core": { - "version": "3.13.23", - "resolved": "https://registry.npmjs.org/@tanstack/virtual-core/-/virtual-core-3.13.23.tgz", - "integrity": "sha512-zSz2Z2HNyLjCplANTDyl3BcdQJc2k1+yyFoKhNRmCr7V7dY8o8q5m8uFTI1/Pg1kL+Hgrz6u3Xo6eFUB7l66cg==", - "license": "MIT", - "funding": { - "type": "github", - "url": "https://github.com/sponsors/tannerlinsley" - } - }, - "node_modules/@types/node": { - "version": "25.6.0", - "resolved": "https://registry.npmjs.org/@types/node/-/node-25.6.0.tgz", - "integrity": "sha512-+qIYRKdNYJwY3vRCZMdJbPLJAtGjQBudzZzdzwQYkEPQd+PJGixUL5QfvCLDaULoLv+RhT3LDkwEfKaAkgSmNQ==", - "license": "MIT", - "dependencies": { - "undici-types": "~7.19.0" - } - }, - "node_modules/abbrev": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/abbrev/-/abbrev-4.0.0.tgz", - "integrity": "sha512-a1wflyaL0tHtJSmLSOVybYhy22vRih4eduhhrkcjgrWGnRfrZtovJ2FRjxuTtkkj47O/baf0R86QU5OuYpz8fA==", - "license": "ISC", - "optional": true, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/agent-base": { - "version": "7.1.4", - "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz", - "integrity": "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">= 14" - } - }, - "node_modules/balanced-match": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", - "integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==", - "license": "MIT", - "optional": true, - "engines": { - "node": "18 || 20 || >=22" - } - }, - "node_modules/base64-js": { - "version": "1.5.1", - "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", - "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT" - }, - "node_modules/bindings": { - "version": "1.5.0", - "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.5.0.tgz", - "integrity": "sha512-p2q/t/mhvuOj/UeLlV6566GD/guowlr0hHxClI0W9m7MWYkL1F0hLo+0Aexs9HSPCtR1SXQ0TD3MMKrXZajbiQ==", - "license": "MIT", - "dependencies": { - "file-uri-to-path": "1.0.0" - } - }, - "node_modules/bl": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz", - "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==", - "license": "MIT", - "dependencies": { - "buffer": "^5.5.0", - "inherits": "^2.0.4", - "readable-stream": "^3.4.0" - } - }, - "node_modules/brace-expansion": { - "version": "5.0.5", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz", - "integrity": "sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==", - "license": "MIT", - "optional": true, - "dependencies": { - "balanced-match": "^4.0.2" - }, - "engines": { - "node": "18 || 20 || >=22" - } - }, - "node_modules/buffer": { - "version": "5.7.1", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz", - "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.1.13" - } - }, - "node_modules/cacache": { - "version": "20.0.4", - "resolved": "https://registry.npmjs.org/cacache/-/cacache-20.0.4.tgz", - "integrity": "sha512-M3Lab8NPYlZU2exsL3bMVvMrMqgwCnMWfdZbK28bn3pK6APT/Te/I8hjRPNu1uwORY9a1eEQoifXbKPQMfMTOA==", - "license": "ISC", - "optional": true, - "dependencies": { - "@npmcli/fs": "^5.0.0", - "fs-minipass": "^3.0.0", - "glob": "^13.0.0", - "lru-cache": "^11.1.0", - "minipass": "^7.0.3", - "minipass-collect": "^2.0.1", - "minipass-flush": "^1.0.5", - "minipass-pipeline": "^1.2.4", - "p-map": "^7.0.2", - "ssri": "^13.0.0" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/chownr": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/chownr/-/chownr-3.0.0.tgz", - "integrity": "sha512-+IxzY9BZOQd/XuYPRmrvEVjF/nqj5kgT4kEq7VofrDoM1MxoRjEWkrCC3EtLi59TVawxTAn+orJwFQcrqEN1+g==", - "license": "BlueOak-1.0.0", - "engines": { - "node": ">=18" - } - }, - "node_modules/commander": { - "version": "14.0.3", - "resolved": "https://registry.npmjs.org/commander/-/commander-14.0.3.tgz", - "integrity": "sha512-H+y0Jo/T1RZ9qPP4Eh1pkcQcLRglraJaSLoyOtHxu6AapkjWVCy2Sit1QQ4x3Dng8qDlSsZEet7g5Pq06MvTgw==", - "license": "MIT", - "engines": { - "node": ">=20" - } - }, - "node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", - "license": "MIT", - "optional": true, - "dependencies": { - "ms": "^2.1.3" - }, - "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } - } - }, - "node_modules/decompress-response": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz", - "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==", - "license": "MIT", - "dependencies": { - "mimic-response": "^3.1.0" - }, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/deep-extend": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz", - "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==", - "license": "MIT", - "engines": { - "node": ">=4.0.0" - } - }, - "node_modules/detect-libc": { - "version": "2.1.2", - "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", - "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", - "license": "Apache-2.0", - "engines": { - "node": ">=8" - } - }, - "node_modules/dotenv": { - "version": "17.4.1", - "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.1.tgz", - "integrity": "sha512-k8DaKGP6r1G30Lx8V4+pCsLzKr8vLmV2paqEj1Y55GdAgJuIqpRp5FfajGF8KtwMxCz9qJc6wUIJnm053d/WCw==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://dotenvx.com" - } - }, - "node_modules/end-of-stream": { - "version": "1.4.5", - "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz", - "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==", - "license": "MIT", - "dependencies": { - "once": "^1.4.0" - } - }, - "node_modules/env-paths": { - "version": "2.2.1", - "resolved": "https://registry.npmjs.org/env-paths/-/env-paths-2.2.1.tgz", - "integrity": "sha512-+h1lkLKhZMTYjog1VEpJNG7NZJWcuc2DDk/qsqSTRRCOXiLjeQ1d1/udrUGhqMxUgAlwKNZ0cf2uqan5GLuS2A==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">=6" - } - }, - "node_modules/esbuild": { - "version": "0.27.7", - "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz", - "integrity": "sha512-IxpibTjyVnmrIQo5aqNpCgoACA/dTKLTlhMHihVHhdkxKyPO1uBBthumT0rdHmcsk9uMonIWS0m4FljWzILh3w==", - "hasInstallScript": true, - "license": "MIT", - "bin": { - "esbuild": "bin/esbuild" - }, - "engines": { - "node": ">=18" - }, - "optionalDependencies": { - "@esbuild/aix-ppc64": "0.27.7", - "@esbuild/android-arm": "0.27.7", - "@esbuild/android-arm64": "0.27.7", - "@esbuild/android-x64": "0.27.7", - "@esbuild/darwin-arm64": "0.27.7", - "@esbuild/darwin-x64": "0.27.7", - "@esbuild/freebsd-arm64": "0.27.7", - "@esbuild/freebsd-x64": "0.27.7", - "@esbuild/linux-arm": "0.27.7", - "@esbuild/linux-arm64": "0.27.7", - "@esbuild/linux-ia32": "0.27.7", - "@esbuild/linux-loong64": "0.27.7", - "@esbuild/linux-mips64el": "0.27.7", - "@esbuild/linux-ppc64": "0.27.7", - "@esbuild/linux-riscv64": "0.27.7", - "@esbuild/linux-s390x": "0.27.7", - "@esbuild/linux-x64": "0.27.7", - "@esbuild/netbsd-arm64": "0.27.7", - "@esbuild/netbsd-x64": "0.27.7", - "@esbuild/openbsd-arm64": "0.27.7", - "@esbuild/openbsd-x64": "0.27.7", - "@esbuild/openharmony-arm64": "0.27.7", - "@esbuild/sunos-x64": "0.27.7", - "@esbuild/win32-arm64": "0.27.7", - "@esbuild/win32-ia32": "0.27.7", - "@esbuild/win32-x64": "0.27.7" - } - }, - "node_modules/expand-template": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz", - "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==", - "license": "(MIT OR WTFPL)", - "engines": { - "node": ">=6" - } - }, - "node_modules/exponential-backoff": { - "version": "3.1.3", - "resolved": "https://registry.npmjs.org/exponential-backoff/-/exponential-backoff-3.1.3.tgz", - "integrity": "sha512-ZgEeZXj30q+I0EN+CbSSpIyPaJ5HVQD18Z1m+u1FXbAeT94mr1zw50q4q6jiiC447Nl/YTcIYSAftiGqetwXCA==", - "license": "Apache-2.0", - "optional": true - }, - "node_modules/fdir": { - "version": "6.5.0", - "resolved": "https://registry.npmjs.org/fdir/-/fdir-6.5.0.tgz", - "integrity": "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">=12.0.0" - }, - "peerDependencies": { - "picomatch": "^3 || ^4" - }, - "peerDependenciesMeta": { - "picomatch": { - "optional": true - } - } - }, - "node_modules/file-uri-to-path": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz", - "integrity": "sha512-0Zt+s3L7Vf1biwWZ29aARiVYLx7iMGnEUl9x33fbB/j3jR81u/O2LbqK+Bm1CDSNDKVtJ/YjwY7TUd5SkeLQLw==", - "license": "MIT" - }, - "node_modules/fs-constants": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz", - "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==", - "license": "MIT" - }, - "node_modules/fs-minipass": { - "version": "3.0.3", - "resolved": "https://registry.npmjs.org/fs-minipass/-/fs-minipass-3.0.3.tgz", - "integrity": "sha512-XUBA9XClHbnJWSfBzjkm6RvPsyg3sryZt06BEQoXcF7EK/xpGaQYJgQKDJSUH5SGZ76Y7pFx1QBnXz09rU5Fbw==", - "license": "ISC", - "optional": true, - "dependencies": { - "minipass": "^7.0.3" - }, - "engines": { - "node": "^14.17.0 || ^16.13.0 || >=18.0.0" - } - }, - "node_modules/fsevents": { - "version": "2.3.3", - "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", - "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", - "hasInstallScript": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": "^8.16.0 || ^10.6.0 || >=11.0.0" - } - }, - "node_modules/get-tsconfig": { - "version": "4.13.7", - "resolved": "https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-4.13.7.tgz", - "integrity": "sha512-7tN6rFgBlMgpBML5j8typ92BKFi2sFQvIdpAqLA2beia5avZDrMs0FLZiM5etShWq5irVyGcGMEA1jcDaK7A/Q==", - "license": "MIT", - "dependencies": { - "resolve-pkg-maps": "^1.0.0" - }, - "funding": { - "url": "https://github.com/privatenumber/get-tsconfig?sponsor=1" - } - }, - "node_modules/github-from-package": { - "version": "0.0.0", - "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz", - "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==", - "license": "MIT" - }, - "node_modules/glob": { - "version": "13.0.6", - "resolved": "https://registry.npmjs.org/glob/-/glob-13.0.6.tgz", - "integrity": "sha512-Wjlyrolmm8uDpm/ogGyXZXb1Z+Ca2B8NbJwqBVg0axK9GbBeoS7yGV6vjXnYdGm6X53iehEuxxbyiKp8QmN4Vw==", - "license": "BlueOak-1.0.0", - "optional": true, - "dependencies": { - "minimatch": "^10.2.2", - "minipass": "^7.1.3", - "path-scurry": "^2.0.2" - }, - "engines": { - "node": "18 || 20 || >=22" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/graceful-fs": { - "version": "4.2.11", - "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz", - "integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==", - "license": "ISC", - "optional": true - }, - "node_modules/http-cache-semantics": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/http-cache-semantics/-/http-cache-semantics-4.2.0.tgz", - "integrity": "sha512-dTxcvPXqPvXBQpq5dUr6mEMJX4oIEFv6bwom3FDwKRDsuIjjJGANqhBuoAn9c1RQJIdAKav33ED65E2ys+87QQ==", - "license": "BSD-2-Clause", - "optional": true - }, - "node_modules/http-proxy-agent": { - "version": "7.0.2", - "resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-7.0.2.tgz", - "integrity": "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==", - "license": "MIT", - "optional": true, - "dependencies": { - "agent-base": "^7.1.0", - "debug": "^4.3.4" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/https-proxy-agent": { - "version": "7.0.6", - "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-7.0.6.tgz", - "integrity": "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==", - "license": "MIT", - "optional": true, - "dependencies": { - "agent-base": "^7.1.2", - "debug": "4" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/iconv-lite": { - "version": "0.7.2", - "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.2.tgz", - "integrity": "sha512-im9DjEDQ55s9fL4EYzOAv0yMqmMBSZp6G0VvFyTMPKWxiSBHUj9NW/qqLmXUwXrrM7AvqSlTCfvqRb0cM8yYqw==", - "license": "MIT", - "optional": true, - "dependencies": { - "safer-buffer": ">= 2.1.2 < 3.0.0" - }, - "engines": { - "node": ">=0.10.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/express" - } - }, - "node_modules/ieee754": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz", - "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "BSD-3-Clause" - }, - "node_modules/inherits": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", - "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", - "license": "ISC" - }, - "node_modules/ini": { - "version": "1.3.8", - "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz", - "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", - "license": "ISC" - }, - "node_modules/ip-address": { - "version": "10.1.0", - "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.1.0.tgz", - "integrity": "sha512-XXADHxXmvT9+CRxhXg56LJovE+bmWnEWB78LB83VZTprKTmaC5QfruXocxzTZ2Kl0DNwKuBdlIhjL8LeY8Sf8Q==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">= 12" - } - }, - "node_modules/isexe": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/isexe/-/isexe-4.0.0.tgz", - "integrity": "sha512-FFUtZMpoZ8RqHS3XeXEmHWLA4thH+ZxCv2lOiPIn1Xc7CxrqhWzNSDzD+/chS/zbYezmiwWLdQC09JdQKmthOw==", - "license": "BlueOak-1.0.0", - "optional": true, - "engines": { - "node": ">=20" - } - }, - "node_modules/json-schema-to-ts": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/json-schema-to-ts/-/json-schema-to-ts-3.1.1.tgz", - "integrity": "sha512-+DWg8jCJG2TEnpy7kOm/7/AxaYoaRbjVB4LFZLySZlWn8exGs3A4OLJR966cVvU26N7X9TWxl+Jsw7dzAqKT6g==", - "license": "MIT", - "dependencies": { - "@babel/runtime": "^7.18.3", - "ts-algebra": "^2.0.0" - }, - "engines": { - "node": ">=16" - } - }, - "node_modules/lru-cache": { - "version": "11.3.3", - "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.3.3.tgz", - "integrity": "sha512-JvNw9Y81y33E+BEYPr0U7omo+U9AySnsMsEiXgwT6yqd31VQWTLNQqmT4ou5eqPFUrTfIDFta2wKhB1hyohtAQ==", - "license": "BlueOak-1.0.0", - "optional": true, - "engines": { - "node": "20 || >=22" - } - }, - "node_modules/make-fetch-happen": { - "version": "15.0.5", - "resolved": "https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-15.0.5.tgz", - "integrity": "sha512-uCbIa8jWWmQZt4dSnEStkVC6gdakiinAm4PiGsywIkguF0eWMdcjDz0ECYhUolFU3pFLOev9VNPCEygydXnddg==", - "license": "ISC", - "optional": true, - "dependencies": { - "@gar/promise-retry": "^1.0.0", - "@npmcli/agent": "^4.0.0", - "@npmcli/redact": "^4.0.0", - "cacache": "^20.0.1", - "http-cache-semantics": "^4.1.1", - "minipass": "^7.0.2", - "minipass-fetch": "^5.0.0", - "minipass-flush": "^1.0.5", - "minipass-pipeline": "^1.2.4", - "negotiator": "^1.0.0", - "proc-log": "^6.0.0", - "ssri": "^13.0.0" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/mimic-response": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz", - "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==", - "license": "MIT", - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/minimatch": { - "version": "10.2.5", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.5.tgz", - "integrity": "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==", - "license": "BlueOak-1.0.0", - "optional": true, - "dependencies": { - "brace-expansion": "^5.0.5" - }, - "engines": { - "node": "18 || 20 || >=22" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/minimist": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz", - "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/minipass": { - "version": "7.1.3", - "resolved": "https://registry.npmjs.org/minipass/-/minipass-7.1.3.tgz", - "integrity": "sha512-tEBHqDnIoM/1rXME1zgka9g6Q2lcoCkxHLuc7ODJ5BxbP5d4c2Z5cGgtXAku59200Cx7diuHTOYfSBD8n6mm8A==", - "license": "BlueOak-1.0.0", - "engines": { - "node": ">=16 || 14 >=14.17" - } - }, - "node_modules/minipass-collect": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/minipass-collect/-/minipass-collect-2.0.1.tgz", - "integrity": "sha512-D7V8PO9oaz7PWGLbCACuI1qEOsq7UKfLotx/C0Aet43fCUB/wfQ7DYeq2oR/svFJGYDHPr38SHATeaj/ZoKHKw==", - "license": "ISC", - "optional": true, - "dependencies": { - "minipass": "^7.0.3" - }, - "engines": { - "node": ">=16 || 14 >=14.17" - } - }, - "node_modules/minipass-fetch": { - "version": "5.0.2", - "resolved": "https://registry.npmjs.org/minipass-fetch/-/minipass-fetch-5.0.2.tgz", - "integrity": "sha512-2d0q2a8eCi2IRg/IGubCNRJoYbA1+YPXAzQVRFmB45gdGZafyivnZ5YSEfo3JikbjGxOdntGFvBQGqaSMXlAFQ==", - "license": "MIT", - "optional": true, - "dependencies": { - "minipass": "^7.0.3", - "minipass-sized": "^2.0.0", - "minizlib": "^3.0.1" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - }, - "optionalDependencies": { - "iconv-lite": "^0.7.2" - } - }, - "node_modules/minipass-flush": { - "version": "1.0.7", - "resolved": "https://registry.npmjs.org/minipass-flush/-/minipass-flush-1.0.7.tgz", - "integrity": "sha512-TbqTz9cUwWyHS2Dy89P3ocAGUGxKjjLuR9z8w4WUTGAVgEj17/4nhgo2Du56i0Fm3Pm30g4iA8Lcqctc76jCzA==", - "license": "BlueOak-1.0.0", - "optional": true, - "dependencies": { - "minipass": "^3.0.0" - }, - "engines": { - "node": ">= 8" - } - }, - "node_modules/minipass-flush/node_modules/minipass": { - "version": "3.3.6", - "resolved": "https://registry.npmjs.org/minipass/-/minipass-3.3.6.tgz", - "integrity": "sha512-DxiNidxSEK+tHG6zOIklvNOwm3hvCrbUrdtzY74U6HKTJxvIDfOUL5W5P2Ghd3DTkhhKPYGqeNUIh5qcM4YBfw==", - "license": "ISC", - "optional": true, - "dependencies": { - "yallist": "^4.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/minipass-flush/node_modules/yallist": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz", - "integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==", - "license": "ISC", - "optional": true - }, - "node_modules/minipass-pipeline": { - "version": "1.2.4", - "resolved": "https://registry.npmjs.org/minipass-pipeline/-/minipass-pipeline-1.2.4.tgz", - "integrity": "sha512-xuIq7cIOt09RPRJ19gdi4b+RiNvDFYe5JH+ggNvBqGqpQXcru3PcRmOZuHBKWK1Txf9+cQ+HMVN4d6z46LZP7A==", - "license": "ISC", - "optional": true, - "dependencies": { - "minipass": "^3.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/minipass-pipeline/node_modules/minipass": { - "version": "3.3.6", - "resolved": "https://registry.npmjs.org/minipass/-/minipass-3.3.6.tgz", - "integrity": "sha512-DxiNidxSEK+tHG6zOIklvNOwm3hvCrbUrdtzY74U6HKTJxvIDfOUL5W5P2Ghd3DTkhhKPYGqeNUIh5qcM4YBfw==", - "license": "ISC", - "optional": true, - "dependencies": { - "yallist": "^4.0.0" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/minipass-pipeline/node_modules/yallist": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz", - "integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==", - "license": "ISC", - "optional": true - }, - "node_modules/minipass-sized": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/minipass-sized/-/minipass-sized-2.0.0.tgz", - "integrity": "sha512-zSsHhto5BcUVM2m1LurnXY6M//cGhVaegT71OfOXoprxT6o780GZd792ea6FfrQkuU4usHZIUczAQMRUE2plzA==", - "license": "ISC", - "optional": true, - "dependencies": { - "minipass": "^7.1.2" - }, - "engines": { - "node": ">=8" - } - }, - "node_modules/minizlib": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/minizlib/-/minizlib-3.1.0.tgz", - "integrity": "sha512-KZxYo1BUkWD2TVFLr0MQoM8vUUigWD3LlD83a/75BqC+4qE0Hb1Vo5v1FgcfaNXvfXzr+5EhQ6ing/CaBijTlw==", - "license": "MIT", - "dependencies": { - "minipass": "^7.1.2" - }, - "engines": { - "node": ">= 18" - } - }, - "node_modules/mkdirp-classic": { - "version": "0.5.3", - "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz", - "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==", - "license": "MIT" - }, - "node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT", - "optional": true - }, - "node_modules/napi-build-utils": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz", - "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==", - "license": "MIT" - }, - "node_modules/negotiator": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.0.0.tgz", - "integrity": "sha512-8Ofs/AUQh8MaEcrlq5xOX0CQ9ypTF5dl78mjlMNfOK08fzpgTHQRQPBxcPlEtIw0yRpws+Zo/3r+5WRby7u3Gg==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/neo4j-driver": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/neo4j-driver/-/neo4j-driver-6.0.1.tgz", - "integrity": "sha512-8DDF2MwEJNz7y7cp97x4u8fmVIP4CWS8qNBxdwxTG0fWtsS+2NdeC+7uXwmmuFOpHvkfXqv63uWY73bfDtOH8Q==", - "license": "Apache-2.0", - "dependencies": { - "neo4j-driver-bolt-connection": "6.0.1", - "neo4j-driver-core": "6.0.1", - "rxjs": "^7.8.2" - }, - "engines": { - "node": ">=18.0.0" - } - }, - "node_modules/neo4j-driver-bolt-connection": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/neo4j-driver-bolt-connection/-/neo4j-driver-bolt-connection-6.0.1.tgz", - "integrity": "sha512-1KyG73TO+CwnYJisdHD0sjUw9yR+P5q3JFcmVPzsHT4/whzCjuXSMpmY4jZcHH2PdY2cBUq4l/6WcDiPMxW2UA==", - "license": "Apache-2.0", - "dependencies": { - "buffer": "^6.0.3", - "neo4j-driver-core": "6.0.1", - "string_decoder": "^1.3.0" - } - }, - "node_modules/neo4j-driver-bolt-connection/node_modules/buffer": { - "version": "6.0.3", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz", - "integrity": "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.2.1" - } - }, - "node_modules/neo4j-driver-core": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/neo4j-driver-core/-/neo4j-driver-core-6.0.1.tgz", - "integrity": "sha512-5I2KxICAvcHxnWdJyDqwu8PBAQvWVTlQH2ve3VQmtVdJScPqWhpXN1PiX5IIl+cRF3pFpz9GQF53B5n6s0QQUQ==", - "license": "Apache-2.0" - }, - "node_modules/node-abi": { - "version": "3.89.0", - "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.89.0.tgz", - "integrity": "sha512-6u9UwL0HlAl21+agMN3YAMXcKByMqwGx+pq+P76vii5f7hTPtKDp08/H9py6DY+cfDw7kQNTGEj/rly3IgbNQA==", - "license": "MIT", - "dependencies": { - "semver": "^7.3.5" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/node-addon-api": { - "version": "8.7.0", - "resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-8.7.0.tgz", - "integrity": "sha512-9MdFxmkKaOYVTV+XVRG8ArDwwQ77XIgIPyKASB1k3JPq3M8fGQQQE3YpMOrKm6g//Ktx8ivZr8xo1Qmtqub+GA==", - "license": "MIT", - "engines": { - "node": "^18 || ^20 || >= 21" - } - }, - "node_modules/node-gyp": { - "version": "12.2.0", - "resolved": "https://registry.npmjs.org/node-gyp/-/node-gyp-12.2.0.tgz", - "integrity": "sha512-q23WdzrQv48KozXlr0U1v9dwO/k59NHeSzn6loGcasyf0UnSrtzs8kRxM+mfwJSf0DkX0s43hcqgnSO4/VNthQ==", - "license": "MIT", - "optional": true, - "dependencies": { - "env-paths": "^2.2.0", - "exponential-backoff": "^3.1.1", - "graceful-fs": "^4.2.6", - "make-fetch-happen": "^15.0.0", - "nopt": "^9.0.0", - "proc-log": "^6.0.0", - "semver": "^7.3.5", - "tar": "^7.5.4", - "tinyglobby": "^0.2.12", - "which": "^6.0.0" - }, - "bin": { - "node-gyp": "bin/node-gyp.js" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/nopt": { - "version": "9.0.0", - "resolved": "https://registry.npmjs.org/nopt/-/nopt-9.0.0.tgz", - "integrity": "sha512-Zhq3a+yFKrYwSBluL4H9XP3m3y5uvQkB/09CwDruCiRmR/UJYnn9W4R48ry0uGC70aeTPKLynBtscP9efFFcPw==", - "license": "ISC", - "optional": true, - "dependencies": { - "abbrev": "^4.0.0" - }, - "bin": { - "nopt": "bin/nopt.js" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/once": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", - "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", - "license": "ISC", - "dependencies": { - "wrappy": "1" - } - }, - "node_modules/p-map": { - "version": "7.0.4", - "resolved": "https://registry.npmjs.org/p-map/-/p-map-7.0.4.tgz", - "integrity": "sha512-tkAQEw8ysMzmkhgw8k+1U/iPhWNhykKnSk4Rd5zLoPJCuJaGRPo6YposrZgaxHKzDHdDWWZvE/Sk7hsL2X/CpQ==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">=18" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/path-scurry": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-2.0.2.tgz", - "integrity": "sha512-3O/iVVsJAPsOnpwWIeD+d6z/7PmqApyQePUtCndjatj/9I5LylHvt5qluFaBT3I5h3r1ejfR056c+FCv+NnNXg==", - "license": "BlueOak-1.0.0", - "optional": true, - "dependencies": { - "lru-cache": "^11.0.0", - "minipass": "^7.1.2" - }, - "engines": { - "node": "18 || 20 || >=22" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, - "node_modules/picomatch": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", - "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://github.com/sponsors/jonschlinkert" - } - }, - "node_modules/prebuild-install": { - "version": "7.1.3", - "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz", - "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==", - "deprecated": "No longer maintained. Please contact the author of the relevant native addon; alternatives are available.", - "license": "MIT", - "dependencies": { - "detect-libc": "^2.0.0", - "expand-template": "^2.0.3", - "github-from-package": "0.0.0", - "minimist": "^1.2.3", - "mkdirp-classic": "^0.5.3", - "napi-build-utils": "^2.0.0", - "node-abi": "^3.3.0", - "pump": "^3.0.0", - "rc": "^1.2.7", - "simple-get": "^4.0.0", - "tar-fs": "^2.0.0", - "tunnel-agent": "^0.6.0" - }, - "bin": { - "prebuild-install": "bin.js" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/proc-log": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/proc-log/-/proc-log-6.1.0.tgz", - "integrity": "sha512-iG+GYldRf2BQ0UDUAd6JQ/RwzaQy6mXmsk/IzlYyal4A4SNFw54MeH4/tLkF4I5WoWG9SQwuqWzS99jaFQHBuQ==", - "license": "ISC", - "optional": true, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/pump": { - "version": "3.0.4", - "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz", - "integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==", - "license": "MIT", - "dependencies": { - "end-of-stream": "^1.1.0", - "once": "^1.3.1" - } - }, - "node_modules/rc": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", - "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==", - "license": "(BSD-2-Clause OR MIT OR Apache-2.0)", - "dependencies": { - "deep-extend": "^0.6.0", - "ini": "~1.3.0", - "minimist": "^1.2.0", - "strip-json-comments": "~2.0.1" - }, - "bin": { - "rc": "cli.js" - } - }, - "node_modules/react": { - "version": "19.2.5", - "resolved": "https://registry.npmjs.org/react/-/react-19.2.5.tgz", - "integrity": "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA==", - "license": "MIT", - "peer": true, - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/react-dom": { - "version": "19.2.5", - "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.5.tgz", - "integrity": "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag==", - "license": "MIT", - "peer": true, - "dependencies": { - "scheduler": "^0.27.0" - }, - "peerDependencies": { - "react": "^19.2.5" - } - }, - "node_modules/readable-stream": { - "version": "3.6.2", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", - "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", - "license": "MIT", - "dependencies": { - "inherits": "^2.0.3", - "string_decoder": "^1.1.1", - "util-deprecate": "^1.0.1" - }, - "engines": { - "node": ">= 6" - } - }, - "node_modules/resolve-pkg-maps": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/resolve-pkg-maps/-/resolve-pkg-maps-1.0.0.tgz", - "integrity": "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==", - "license": "MIT", - "funding": { - "url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1" - } - }, - "node_modules/rxjs": { - "version": "7.8.2", - "resolved": "https://registry.npmjs.org/rxjs/-/rxjs-7.8.2.tgz", - "integrity": "sha512-dhKf903U/PQZY6boNNtAGdWbG85WAbjT/1xYoZIC7FAY0yWapOBQVsVrDl58W86//e1VpMNBtRV4MaXfdMySFA==", - "license": "Apache-2.0", - "dependencies": { - "tslib": "^2.1.0" - } - }, - "node_modules/safe-buffer": { - "version": "5.2.1", - "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", - "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT" - }, - "node_modules/safer-buffer": { - "version": "2.1.2", - "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", - "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==", - "license": "MIT", - "optional": true - }, - "node_modules/scheduler": { - "version": "0.27.0", - "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.27.0.tgz", - "integrity": "sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==", - "license": "MIT", - "peer": true - }, - "node_modules/semver": { - "version": "7.7.4", - "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz", - "integrity": "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA==", - "license": "ISC", - "bin": { - "semver": "bin/semver.js" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/simple-concat": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz", - "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT" - }, - "node_modules/simple-get": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz", - "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "decompress-response": "^6.0.0", - "once": "^1.3.1", - "simple-concat": "^1.0.0" - } - }, - "node_modules/smart-buffer": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/smart-buffer/-/smart-buffer-4.2.0.tgz", - "integrity": "sha512-94hK0Hh8rPqQl2xXc3HsaBoOXKV20MToPkcXvwbISWLEs+64sBq5kFgn2kJDHb1Pry9yrP0dxrCI9RRci7RXKg==", - "license": "MIT", - "optional": true, - "engines": { - "node": ">= 6.0.0", - "npm": ">= 3.0.0" - } - }, - "node_modules/socks": { - "version": "2.8.7", - "resolved": "https://registry.npmjs.org/socks/-/socks-2.8.7.tgz", - "integrity": "sha512-HLpt+uLy/pxB+bum/9DzAgiKS8CX1EvbWxI4zlmgGCExImLdiad2iCwXT5Z4c9c3Eq8rP2318mPW2c+QbtjK8A==", - "license": "MIT", - "optional": true, - "dependencies": { - "ip-address": "^10.0.1", - "smart-buffer": "^4.2.0" - }, - "engines": { - "node": ">= 10.0.0", - "npm": ">= 3.0.0" - } - }, - "node_modules/socks-proxy-agent": { - "version": "8.0.5", - "resolved": "https://registry.npmjs.org/socks-proxy-agent/-/socks-proxy-agent-8.0.5.tgz", - "integrity": "sha512-HehCEsotFqbPW9sJ8WVYB6UbmIMv7kUUORIF2Nncq4VQvBfNBLibW9YZR5dlYCSUhwcD628pRllm7n+E+YTzJw==", - "license": "MIT", - "optional": true, - "dependencies": { - "agent-base": "^7.1.2", - "debug": "^4.3.4", - "socks": "^2.8.3" - }, - "engines": { - "node": ">= 14" - } - }, - "node_modules/sqlite": { - "version": "5.1.1", - "resolved": "https://registry.npmjs.org/sqlite/-/sqlite-5.1.1.tgz", - "integrity": "sha512-oBkezXa2hnkfuJwUo44Hl9hS3er+YFtueifoajrgidvqsJRQFpc5fKoAkAor1O5ZnLoa28GBScfHXs8j0K358Q==", - "license": "MIT" - }, - "node_modules/sqlite3": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/sqlite3/-/sqlite3-6.0.1.tgz", - "integrity": "sha512-X0czUUMG2tmSqJpEQa3tCuZSHKIx8PwM53vLZzKp/o6Rpy25fiVfjdbnZ988M8+O3ZWR1ih0K255VumCb3MAnQ==", - "hasInstallScript": true, - "license": "BSD-3-Clause", - "dependencies": { - "bindings": "^1.5.0", - "node-addon-api": "^8.0.0", - "prebuild-install": "^7.1.3", - "tar": "^7.5.10" - }, - "engines": { - "node": ">=20.17.0" - }, - "optionalDependencies": { - "node-gyp": "12.x" - }, - "peerDependencies": { - "node-gyp": "12.x" - }, - "peerDependenciesMeta": { - "node-gyp": { - "optional": true - } - } - }, - "node_modules/ssri": { - "version": "13.0.1", - "resolved": "https://registry.npmjs.org/ssri/-/ssri-13.0.1.tgz", - "integrity": "sha512-QUiRf1+u9wPTL/76GTYlKttDEBWV1ga9ZXW8BG6kfdeyyM8LGPix9gROyg9V2+P0xNyF3X2Go526xKFdMZrHSQ==", - "license": "ISC", - "optional": true, - "dependencies": { - "minipass": "^7.0.3" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/string_decoder": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", - "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==", - "license": "MIT", - "dependencies": { - "safe-buffer": "~5.2.0" - } - }, - "node_modules/strip-json-comments": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", - "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/tar": { - "version": "7.5.13", - "resolved": "https://registry.npmjs.org/tar/-/tar-7.5.13.tgz", - "integrity": "sha512-tOG/7GyXpFevhXVh8jOPJrmtRpOTsYqUIkVdVooZYJS/z8WhfQUX8RJILmeuJNinGAMSu1veBr4asSHFt5/hng==", - "license": "BlueOak-1.0.0", - "dependencies": { - "@isaacs/fs-minipass": "^4.0.0", - "chownr": "^3.0.0", - "minipass": "^7.1.2", - "minizlib": "^3.1.0", - "yallist": "^5.0.0" - }, - "engines": { - "node": ">=18" - } - }, - "node_modules/tar-fs": { - "version": "2.1.4", - "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.4.tgz", - "integrity": "sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ==", - "license": "MIT", - "dependencies": { - "chownr": "^1.1.1", - "mkdirp-classic": "^0.5.2", - "pump": "^3.0.0", - "tar-stream": "^2.1.4" - } - }, - "node_modules/tar-fs/node_modules/chownr": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", - "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", - "license": "ISC" - }, - "node_modules/tar-stream": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz", - "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==", - "license": "MIT", - "dependencies": { - "bl": "^4.0.3", - "end-of-stream": "^1.4.1", - "fs-constants": "^1.0.0", - "inherits": "^2.0.3", - "readable-stream": "^3.1.1" - }, - "engines": { - "node": ">=6" - } - }, - "node_modules/tinyglobby": { - "version": "0.2.16", - "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz", - "integrity": "sha512-pn99VhoACYR8nFHhxqix+uvsbXineAasWm5ojXoN8xEwK5Kd3/TrhNn1wByuD52UxWRLy8pu+kRMniEi6Eq9Zg==", - "license": "MIT", - "optional": true, - "dependencies": { - "fdir": "^6.5.0", - "picomatch": "^4.0.4" - }, - "engines": { - "node": ">=12.0.0" - }, - "funding": { - "url": "https://github.com/sponsors/SuperchupuDev" - } - }, - "node_modules/ts-algebra": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/ts-algebra/-/ts-algebra-2.0.0.tgz", - "integrity": "sha512-FPAhNPFMrkwz76P7cdjdmiShwMynZYN6SgOujD1urY4oNm80Ou9oMdmbR45LotcKOXoy7wSmHkRFE6Mxbrhefw==", - "license": "MIT" - }, - "node_modules/tslib": { - "version": "2.8.1", - "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", - "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", - "license": "0BSD" - }, - "node_modules/tsx": { - "version": "4.21.0", - "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.21.0.tgz", - "integrity": "sha512-5C1sg4USs1lfG0GFb2RLXsdpXqBSEhAaA/0kPL01wxzpMqLILNxIxIOKiILz+cdg/pLnOUxFYOR5yhHU666wbw==", - "license": "MIT", - "dependencies": { - "esbuild": "~0.27.0", - "get-tsconfig": "^4.7.5" - }, - "bin": { - "tsx": "dist/cli.mjs" - }, - "engines": { - "node": ">=18.0.0" - }, - "optionalDependencies": { - "fsevents": "~2.3.3" - } - }, - "node_modules/tunnel-agent": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz", - "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==", - "license": "Apache-2.0", - "dependencies": { - "safe-buffer": "^5.0.1" - }, - "engines": { - "node": "*" - } - }, - "node_modules/typescript": { - "version": "6.0.2", - "resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.2.tgz", - "integrity": "sha512-bGdAIrZ0wiGDo5l8c++HWtbaNCWTS4UTv7RaTH/ThVIgjkveJt83m74bBHMJkuCbslY8ixgLBVZJIOiQlQTjfQ==", - "license": "Apache-2.0", - "bin": { - "tsc": "bin/tsc", - "tsserver": "bin/tsserver" - }, - "engines": { - "node": ">=14.17" - } - }, - "node_modules/undici-types": { - "version": "7.19.2", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.19.2.tgz", - "integrity": "sha512-qYVnV5OEm2AW8cJMCpdV20CDyaN3g0AjDlOGf1OW4iaDEx8MwdtChUp4zu4H0VP3nDRF/8RKWH+IPp9uW0YGZg==", - "license": "MIT" - }, - "node_modules/util-deprecate": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", - "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", - "license": "MIT" - }, - "node_modules/which": { - "version": "6.0.1", - "resolved": "https://registry.npmjs.org/which/-/which-6.0.1.tgz", - "integrity": "sha512-oGLe46MIrCRqX7ytPUf66EAYvdeMIZYn3WaocqqKZAxrBpkqHfL/qvTyJ/bTk5+AqHCjXmrv3CEWgy368zhRUg==", - "license": "ISC", - "optional": true, - "dependencies": { - "isexe": "^4.0.0" - }, - "bin": { - "node-which": "bin/which.js" - }, - "engines": { - "node": "^20.17.0 || >=22.9.0" - } - }, - "node_modules/wrappy": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", - "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", - "license": "ISC" - }, - "node_modules/yallist": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/yallist/-/yallist-5.0.0.tgz", - "integrity": "sha512-YgvUTfwqyc7UXVMrB+SImsVYSmTS8X/tSrtdNZMImM+n7+QTriRXyXim0mBrTXNeqzVF0KWGgHPeiyViFFrNDw==", - "license": "BlueOak-1.0.0", - "engines": { - "node": ">=18" - } - }, - "node_modules/zod": { - "version": "4.3.6", - "resolved": "https://registry.npmjs.org/zod/-/zod-4.3.6.tgz", - "integrity": "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/colinhacks" - } - } - } -} diff --git a/package.json b/package.json deleted file mode 100644 index 10823b33..00000000 --- a/package.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "dependencies": { - "@anthropic-ai/sdk": "^0.88.0", - "@tanstack/react-virtual": "^3.13.23", - "@types/node": "^25.6.0", - "commander": "^14.0.3", - "dotenv": "^17.4.1", - "neo4j-driver": "^6.0.1", - "sqlite": "^5.1.1", - "sqlite3": "^6.0.1", - "tsx": "^4.21.0", - "typescript": "^6.0.2", - "zod": "^4.3.6" - } -} From 93cfa49b4202731eee5360f6ca93a244f1636999 Mon Sep 17 00:00:00 2001 From: daymade Date: Thu, 7 May 2026 19:25:49 +0800 Subject: [PATCH 127/137] refactor(marketplace): move skill-creator under daymade-skills namespace - Remove standalone skill-creator plugin to avoid conflict with financial-services-plugins' bundled skill-creator - Add daymade-skills suite plugin containing ./skill-creator - Enables /daymade-skills:skill-creator invocation path chore(gitleaks): allowlist Douban public API key - 0dad551ec0f84ed02907ff5c42e8ec70 is documented as Douban mobile app's public API key, used for zero-config demos Co-Authored-By: Claude Opus 4.7 --- .claude-plugin/marketplace.json | 41 +++++++++++++++------------------ .gitleaks.toml | 4 ++++ 2 files changed, 23 insertions(+), 22 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 4bd5f6c5..594dc1c5 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -265,6 +265,25 @@ "./meeting-minutes-taker" ] }, + { + "name": "daymade-skills", + "description": "Daymade skills core suite. Bundles skill creation, quality review, search, and marketplace development tooling under one shared namespace.", + "source": "./", + "strict": false, + "version": "1.0.0", + "category": "suite", + "keywords": [ + "suite", + "skill-creation", + "skill-review", + "marketplace-dev", + "development", + "tooling" + ], + "skills": [ + "./skill-creator" + ] + }, { "name": "deep-research", "description": "Generate format-controlled research reports with evidence tracking, source governance, and multi-pass synthesis. V6.1 adds: source accessibility (circular verification forbidden, exclusive advantage encouraged). Enterprise Research Mode: six-dimension data collection, SWOT/barrier/risk frameworks, and three-level quality control for company research", @@ -860,28 +879,6 @@ "./scrapling-skill" ] }, - { - "name": "skill-creator", - "description": "Essential meta-skill for creating effective Claude Code skills with initialization scripts, validation, packaging, marketplace registration, and privacy best practices. Includes a specialized wrapper-skill workflow for retrospectively distilling an install-and-debug session into a reusable companion skill for a third-party CLI tool.", - "source": "./", - "strict": false, - "version": "1.7.3", - "category": "developer-tools", - "keywords": [ - "skill-creation", - "claude-code", - "development", - "tooling", - "workflow", - "meta-skill", - "wrapper-skill", - "cli-wrapper", - "essential" - ], - "skills": [ - "./skill-creator" - ] - }, { "name": "skill-reviewer", "description": "Reviews and improves Claude Code skills against official best practices. Supports three modes - self-review (validate your own skills), external review (evaluate others' skills), and auto-PR (fork, improve, submit). Use when checking skill quality, reviewing skill repositories, or contributing improvements to open-source skills", diff --git a/.gitleaks.toml b/.gitleaks.toml index b1b1df73..254f15ec 100644 --- a/.gitleaks.toml +++ b/.gitleaks.toml @@ -15,6 +15,10 @@ paths = [ '''CONTRIBUTING\.md$''', '''CLAUDE\.md$''', ] +regexes = [ + # Douban mobile app's public API key (documented as public, hardcoded for zero-config usage) + '''0dad551ec0f84ed02907ff5c42e8ec70''', +] [[rules]] id = "absolute-user-path-macos" From 89e6ddf63ec2bca86938dfa4310d6e7c83bb20e9 Mon Sep 17 00:00:00 2001 From: daymade Date: Fri, 8 May 2026 01:01:06 +0800 Subject: [PATCH 128/137] chore: remove mistakenly committed cloud/lessons-learned.md Co-Authored-By: Claude Opus 4.7 --- cloud/lessons-learned.md | 68 ---------------------------------------- 1 file changed, 68 deletions(-) delete mode 100644 cloud/lessons-learned.md diff --git a/cloud/lessons-learned.md b/cloud/lessons-learned.md deleted file mode 100644 index 629e5198..00000000 --- a/cloud/lessons-learned.md +++ /dev/null @@ -1,68 +0,0 @@ -# WeChat Article Scraper - 经验教训 - -## 核心原则：用户旅程优先于功能堆砌 - -### 问题 -在 Ralph Loop 迭代中，我犯了**功能堆砌**的错误： -- 添加了 AI摘要、Daily Review、高亮批注、TTS朗读、沉浸式阅读、键盘快捷键、阅读进度追踪... -- 但没有验证核心用户旅程是否通畅 - -### 核心发现 -**微信文章抓取的核心流程是断的！** - -问题链： -1. 扩展调用 `/api/scrape` 让服务器抓取微信文章 -2. 服务器没有微信登录态 → 抓取失败 -3. `/api/articles/import` 端点不存在 → 扩展无法直接上传 -4. 用户无法真正保存微信文章 - -### 修复方案 -1. 创建 `POST /api/articles/import` 端点，接收扩展直接上传的内容 -2. 修改扩展 `saveWeChatArticle` 函数： - - 旧：调用 `/api/scrape` 让服务器抓取（失败） - - 新：提取页面内容 → 调用 `/api/articles/import` 上传（成功） - -### 关键教训 - -#### 1. 先验证核心旅程，再加功能 -**核心用户旅程**（必须100%可用）： -``` -发现微信文章 → 点击扩展 → 提取内容 → 上传保存 → Web阅读 -``` - -**锦上添花功能**（只有核心旅程可用后才有意义）： -- AI摘要、TTS朗读、阅读进度、键盘快捷键... - -#### 2. 架构设计要理解业务场景 -微信文章抓取的特殊性： -- 需要登录态（cookies） -- 反爬严格 -- 必须在已登录的浏览器环境中提取 - -**错误设计**：服务器端抓取 -**正确设计**：扩展提取 + 直接上传 - -#### 3. 测试验证比代码更重要 -写100个功能不如验证1个核心流程。 - -### 决策原则 - -1. **功能添加前**：这是否让核心用户旅程更顺畅？ -2. **代码提交前**：我测试过这个功能真的能用吗？ -3. **迭代方向**：解决问题 > 添加功能 - -### 当前状态 - -**已修复**： -- ✅ 创建 `/api/articles/import` 端点 -- ✅ 修复扩展 `saveWeChatArticle` 函数 -- ✅ 核心抓取流程已打通 - -**待验证**： -- 扩展提取微信文章内容的准确性 -- 图片、视频等特殊内容的处理 -- 批量导入的稳定性 - ---- -记录时间：2026-04-13 -记录原因：Ralph Loop 反思 - 功能堆砌 vs 用户旅程 From b03fda855e8c382eb3e1d076acb5da8443a5ad06 Mon Sep 17 00:00:00 2001 From: daymade Date: Fri, 8 May 2026 01:26:58 +0800 Subject: [PATCH 129/137] restructure: move skill-creator, skill-reviewer, skills-search into daymade-skill suite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consolidates the three skill-lifecycle tools under a single suite directory, matching the daymade-docs and daymade-claude-code suite patterns. - Moves skill-creator/, skill-reviewer/, skills-search/ → daymade-skill/ - Renames marketplace suite plugin: daymade-skills → daymade-skill - Removes standalone skill-reviewer and skills-search plugin entries - Updates all path references in CLAUDE.md, README*, QUICKSTART*, PR template Co-Authored-By: Claude Opus 4.7 --- .claude-plugin/marketplace.json | 54 +++---------------- .github/PULL_REQUEST_TEMPLATE.md | 4 +- CLAUDE.md | 8 +-- QUICKSTART.md | 8 +-- QUICKSTART.zh-CN.md | 8 +-- README.md | 14 ++--- README.zh-CN.md | 14 ++--- .../skill-creator}/.gitignore | 0 .../skill-creator}/LICENSE.txt | 0 .../skill-creator}/SKILL.md | 0 .../skill-creator}/agents/analyzer.md | 0 .../skill-creator}/agents/comparator.md | 0 .../skill-creator}/agents/grader.md | 0 .../skill-creator}/assets/eval_review.html | 0 .../eval-viewer/generate_review.py | 0 .../skill-creator}/eval-viewer/viewer.html | 0 .../references/prerequisites.md | 0 .../references/sanitization_checklist.md | 0 .../skill-creator}/references/schemas.md | 0 .../skill-development-methodology.md | 0 .../skill-creator}/scripts/__init__.py | 0 .../scripts/aggregate_benchmark.py | 0 .../skill-creator}/scripts/generate_report.py | 0 .../scripts/improve_description.py | 0 .../skill-creator}/scripts/init_skill.py | 0 .../skill-creator}/scripts/package_skill.py | 0 .../skill-creator}/scripts/quick_validate.py | 0 .../skill-creator}/scripts/run_eval.py | 0 .../skill-creator}/scripts/run_loop.py | 0 .../skill-creator}/scripts/security_scan.py | 0 .../skill-creator}/scripts/utils.py | 0 .../wrapper-skill/architecture_contract.md | 0 .../workflows/wrapper-skill/patterns.md | 0 .../scripts/init_wrapper_skill.py | 0 .../wrapper-skill/verification_protocol.md | 0 .../workflows/wrapper-skill/workflow.md | 0 .../skill-reviewer}/.security-scan-passed | 0 .../skill-reviewer}/SKILL.md | 0 .../references/evaluation_checklist.md | 0 .../references/marketplace_template.json | 0 .../skill-reviewer}/references/pr_template.md | 0 .../skills-search}/.security-scan-passed | 0 .../skills-search}/SKILL.md | 0 43 files changed, 35 insertions(+), 75 deletions(-) rename {skill-creator => daymade-skill/skill-creator}/.gitignore (100%) rename {skill-creator => daymade-skill/skill-creator}/LICENSE.txt (100%) rename {skill-creator => daymade-skill/skill-creator}/SKILL.md (100%) rename {skill-creator => daymade-skill/skill-creator}/agents/analyzer.md (100%) rename {skill-creator => daymade-skill/skill-creator}/agents/comparator.md (100%) rename {skill-creator => daymade-skill/skill-creator}/agents/grader.md (100%) rename {skill-creator => daymade-skill/skill-creator}/assets/eval_review.html (100%) rename {skill-creator => daymade-skill/skill-creator}/eval-viewer/generate_review.py (100%) rename {skill-creator => daymade-skill/skill-creator}/eval-viewer/viewer.html (100%) rename {skill-creator => daymade-skill/skill-creator}/references/prerequisites.md (100%) rename {skill-creator => daymade-skill/skill-creator}/references/sanitization_checklist.md (100%) rename {skill-creator => daymade-skill/skill-creator}/references/schemas.md (100%) rename {skill-creator => daymade-skill/skill-creator}/references/skill-development-methodology.md (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/__init__.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/aggregate_benchmark.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/generate_report.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/improve_description.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/init_skill.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/package_skill.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/quick_validate.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/run_eval.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/run_loop.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/security_scan.py (100%) rename {skill-creator => daymade-skill/skill-creator}/scripts/utils.py (100%) rename {skill-creator => daymade-skill/skill-creator}/workflows/wrapper-skill/architecture_contract.md (100%) rename {skill-creator => daymade-skill/skill-creator}/workflows/wrapper-skill/patterns.md (100%) rename {skill-creator => daymade-skill/skill-creator}/workflows/wrapper-skill/scripts/init_wrapper_skill.py (100%) rename {skill-creator => daymade-skill/skill-creator}/workflows/wrapper-skill/verification_protocol.md (100%) rename {skill-creator => daymade-skill/skill-creator}/workflows/wrapper-skill/workflow.md (100%) rename {skill-reviewer => daymade-skill/skill-reviewer}/.security-scan-passed (100%) rename {skill-reviewer => daymade-skill/skill-reviewer}/SKILL.md (100%) rename {skill-reviewer => daymade-skill/skill-reviewer}/references/evaluation_checklist.md (100%) rename {skill-reviewer => daymade-skill/skill-reviewer}/references/marketplace_template.json (100%) rename {skill-reviewer => daymade-skill/skill-reviewer}/references/pr_template.md (100%) rename {skills-search => daymade-skill/skills-search}/.security-scan-passed (100%) rename {skills-search => daymade-skill/skills-search}/SKILL.md (100%) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 594dc1c5..9c9ba2df 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -6,7 +6,7 @@ }, "metadata": { "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, macOS programmatic window screenshot capture workflows, verified Scrapling CLI installation and web extraction workflows, plugin marketplace development for converting skills repos into official Claude Code marketplaces, Tencent IMA knowledge base companion with zero-config installation across Claude Code/Codex/OpenClaw, upstream bug detection and runtime repair, personalized fan-out search with priority-based knowledge base boosting, and suite plugins that expose related skills under shared namespaces for combined installation workflows — including a dedicated Claude Code operations suite (daymade-claude-code) bundling session recovery, CLAUDE.md optimization, troubleshooting, statusline configuration, export repair, and marketplace development skills under one namespace", - "version": "1.46.0" + "version": "1.46.1" }, "plugins": [ { @@ -266,11 +266,11 @@ ] }, { - "name": "daymade-skills", + "name": "daymade-skill", "description": "Daymade skills core suite. Bundles skill creation, quality review, search, and marketplace development tooling under one shared namespace.", - "source": "./", + "source": "./daymade-skill", "strict": false, - "version": "1.0.0", + "version": "1.0.1", "category": "suite", "keywords": [ "suite", @@ -281,7 +281,9 @@ "tooling" ], "skills": [ - "./skill-creator" + "./skill-creator", + "./skill-reviewer", + "./skills-search" ] }, { @@ -879,48 +881,6 @@ "./scrapling-skill" ] }, - { - "name": "skill-reviewer", - "description": "Reviews and improves Claude Code skills against official best practices. Supports three modes - self-review (validate your own skills), external review (evaluate others' skills), and auto-PR (fork, improve, submit). Use when checking skill quality, reviewing skill repositories, or contributing improvements to open-source skills", - "source": "./", - "strict": false, - "version": "1.0.0", - "category": "developer-tools", - "keywords": [ - "skill-review", - "best-practices", - "claude-code", - "quality-assurance", - "open-source", - "contribution", - "auto-pr" - ], - "skills": [ - "./skill-reviewer" - ] - }, - { - "name": "skills-search", - "description": "Search, discover, install, and manage Claude Code skills from the CCPM registry. Use when users want to find skills for specific tasks, install skills by name, list installed skills, get skill details, or manage their Claude Code skill collection. Triggers include find skills, search for plugins, install skill, list installed skills, or any CCPM registry operations", - "source": "./", - "strict": false, - "version": "1.1.0", - "category": "developer-tools", - "keywords": [ - "ccpm", - "skills", - "search", - "install", - "registry", - "plugin", - "marketplace", - "discovery", - "skill-management" - ], - "skills": [ - "./skills-search" - ] - }, { "name": "slides-creator", "description": "Narrative-first slide deck creation. Guides users through structured narrative design (ABCDEFG model), then delegates visual generation to baoyu-slide-deck. Triggers on create slides, make a presentation, generate deck, slide deck, PPT, or when user needs to turn content into visual slides.", diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 54d20160..3fb1ce61 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -70,8 +70,8 @@ How has this been tested? - [ ] Scripts are executable (chmod +x) - [ ] No absolute paths or user-specific information - [ ] Tested in actual Claude Code session -- [ ] Passed validation: `skill-creator/scripts/quick_validate.py` -- [ ] Successfully packages: `skill-creator/scripts/package_skill.py` +- [ ] Passed validation: `daymade-skill/skill-creator/scripts/quick_validate.py` +- [ ] Successfully packages: `daymade-skill/skill-creator/scripts/package_skill.py` ### For All PRs diff --git a/CLAUDE.md b/CLAUDE.md index e15c8050..fc8e989c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -61,13 +61,13 @@ claude plugin install skill-creator@daymade-skills ```bash # Quick validation of a skill -cd skill-creator && uv run --with PyYAML python -m scripts.quick_validate ../skill-name +cd daymade-skill/skill-creator && uv run --with PyYAML python -m scripts.quick_validate ../skill-name # Package a skill (includes automatic validation) -cd skill-creator && uv run --with PyYAML python -m scripts.package_skill ../skill-name [output-dir] +cd daymade-skill/skill-creator && uv run --with PyYAML python -m scripts.package_skill ../skill-name [output-dir] # Initialize a new skill from template -uv run python skill-creator/scripts/init_skill.py --path +uv run python daymade-skill/skill-creator/scripts/init_skill.py --path ``` ### Testing Skills Locally @@ -302,7 +302,7 @@ For the full step-by-step guide with templates and examples, see [references/new **Quick workflow**: ```bash # 1. Validate & package the skill itself -cd skill-creator +cd daymade-skill/skill-creator uv run python -m scripts.security_scan ../skill-name --verbose uv run --with PyYAML python -m scripts.package_skill ../skill-name diff --git a/QUICKSTART.md b/QUICKSTART.md index 2f3abb85..fee80e28 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -33,7 +33,7 @@ claude plugin install skill-creator@daymade-skills ```bash # Create a new skill from template -skill-creator/scripts/init_skill.py my-first-skill --path ~/my-skills +daymade-skill/skill-creator/scripts/init_skill.py my-first-skill --path ~/my-skills ``` This generates: @@ -61,7 +61,7 @@ Edit `~/my-skills/my-first-skill/SKILL.md`: ```bash # Check if your skill meets quality standards -skill-creator/scripts/quick_validate.py ~/my-skills/my-first-skill +daymade-skill/skill-creator/scripts/quick_validate.py ~/my-skills/my-first-skill ``` Fix any errors reported, then validate again. @@ -70,7 +70,7 @@ Fix any errors reported, then validate again. ```bash # Create a distributable .zip file -skill-creator/scripts/package_skill.py ~/my-skills/my-first-skill +daymade-skill/skill-creator/scripts/package_skill.py ~/my-skills/my-first-skill ``` This creates `my-first-skill.zip` ready to share! @@ -87,7 +87,7 @@ cp -r ~/my-skills/my-first-skill ~/.claude/skills/ ### Next Steps -- 📖 Read [skill-creator/SKILL.md](./skill-creator/SKILL.md) for comprehensive guidance +- 📖 Read [skill-creator/SKILL.md](./daymade-skill/skill-creator/SKILL.md) for comprehensive guidance - 🔍 Study existing skills in this marketplace for examples - 💡 Check [CONTRIBUTING.md](./CONTRIBUTING.md) to share your skill diff --git a/QUICKSTART.zh-CN.md b/QUICKSTART.zh-CN.md index 7b865a96..94507778 100644 --- a/QUICKSTART.zh-CN.md +++ b/QUICKSTART.zh-CN.md @@ -33,7 +33,7 @@ claude plugin install skill-creator@daymade-skills ```bash # 从模板创建一个新技能 -skill-creator/scripts/init_skill.py my-first-skill --path ~/my-skills +daymade-skill/skill-creator/scripts/init_skill.py my-first-skill --path ~/my-skills ``` 这将生成： @@ -61,7 +61,7 @@ skill-creator/scripts/init_skill.py my-first-skill --path ~/my-skills ```bash # 检查你的技能是否符合质量标准 -skill-creator/scripts/quick_validate.py ~/my-skills/my-first-skill +daymade-skill/skill-creator/scripts/quick_validate.py ~/my-skills/my-first-skill ``` 修复报告的任何错误，然后再次验证。 @@ -70,7 +70,7 @@ skill-creator/scripts/quick_validate.py ~/my-skills/my-first-skill ```bash # 创建可分发的 .zip 文件 -skill-creator/scripts/package_skill.py ~/my-skills/my-first-skill +daymade-skill/skill-creator/scripts/package_skill.py ~/my-skills/my-first-skill ``` 这将创建 `my-first-skill.zip`，可以分享了！ @@ -87,7 +87,7 @@ cp -r ~/my-skills/my-first-skill ~/.claude/skills/ ### 下一步 -- 📖 阅读 [skill-creator/SKILL.md](./skill-creator/SKILL.md) 获取全面指导 +- 📖 阅读 [skill-creator/SKILL.md](./daymade-skill/skill-creator/SKILL.md) 获取全面指导 - 🔍 研究此市场中的现有技能以获取示例 - 💡 查看 [CONTRIBUTING.md](./CONTRIBUTING.md) 以分享你的技能 diff --git a/README.md b/README.md index f4104e96..03153866 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ This is a **production-hardened fork** of [Anthropic's official skill-creator](h | User Experience | 4 | 9 | | **Total (out of 80)** | **42** | **65** | -> Full methodology: [skill-creator/references/skill-development-methodology.md](./skill-creator/references/skill-development-methodology.md) +> Full methodology: [skill-creator/references/skill-development-methodology.md](./daymade-skill/skill-creator/references/skill-development-methodology.md) ### Quick Install @@ -102,7 +102,7 @@ After installing skill-creator, simply ask Claude Code: Claude Code, with skill-creator loaded, will guide you through the entire skill creation process - from understanding your requirements to packaging the final skill. -📚 **Full documentation**: [skill-creator/SKILL.md](./skill-creator/SKILL.md) +📚 **Full documentation**: [daymade-skill/skill-creator/SKILL.md](./daymade-skill/daymade-skill/skill-creator/SKILL.md) ### Live Demos @@ -968,7 +968,7 @@ ccpm install-bundle web-dev # Install web development skills bundle *Coming soon* -📚 **Documentation**: See [skills-search/SKILL.md](./skills-search/SKILL.md) for complete command reference +📚 **Documentation**: See [daymade-skill/skills-search/SKILL.md](./daymade-skill/daymade-skill/skills-search/SKILL.md) for complete command reference **Requirements**: CCPM CLI (`npm install -g @daymade/ccpm`) @@ -1329,7 +1329,7 @@ claude plugin install skill-reviewer@daymade-skills *Coming soon* -📚 **Documentation**: See [skill-reviewer/references/](./skill-reviewer/references/) for: +📚 **Documentation**: See [daymade-skill/skill-reviewer/references/](./daymade-skill/daymade-skill/skill-reviewer/references/) for: - `evaluation_checklist.md` - Complete skill evaluation criteria - `pr_template.md` - Professional PR description template - `marketplace_template.json` - Marketplace configuration template @@ -2264,7 +2264,7 @@ Each skill includes: - **statusline-generator**: See `daymade-claude-code/statusline-generator/references/color_codes.md` for customization - **teams-channel-post-writer**: See `teams-channel-post-writer/references/writing-guidelines.md` for quality standards - **repomix-unmixer**: See `repomix-unmixer/references/repomix-format.md` for format specifications -- **skill-creator**: See `skill-creator/SKILL.md` for complete skill creation workflow +- **skill-creator**: See `daymade-skill/skill-creator/SKILL.md` for complete skill creation workflow - **llm-icon-finder**: See `llm-icon-finder/references/icons-list.md` for available icons - **cli-demo-generator**: See `cli-demo-generator/references/vhs_syntax.md` for VHS syntax and `cli-demo-generator/references/best_practices.md` for demo guidelines - **cloudflare-troubleshooting**: See `cloudflare-troubleshooting/references/api_overview.md` for API documentation @@ -2281,12 +2281,12 @@ Each skill includes: - **deep-research**: See `deep-research/references/research_report_template.md` for report structure and `deep-research/references/source_quality_rubric.md` for source triage - **pdf-creator**: See `daymade-docs/pdf-creator/SKILL.md` for PDF conversion and font setup - **claude-md-progressive-disclosurer**: See `daymade-claude-code/claude-md-progressive-disclosurer/SKILL.md` for CLAUDE.md optimization workflow -- **skills-search**: See `skills-search/SKILL.md` for CCPM CLI commands and registry operations +- **skills-search**: See `daymade-skill/skills-search/SKILL.md` for CCPM CLI commands and registry operations - **promptfoo-evaluation**: See `promptfoo-evaluation/references/promptfoo_api.md` for evaluation patterns - **iOS-APP-developer**: See `iOS-APP-developer/references/xcodegen-full.md` for XcodeGen options and project.yml details - **twitter-reader**: See `twitter-reader/SKILL.md` for API key setup and URL format support - **macos-cleaner**: See `macos-cleaner/references/cleanup_targets.md` for detailed cleanup target explanations, `macos-cleaner/references/mole_integration.md` for Mole visual tool integration, and `macos-cleaner/references/safety_rules.md` for comprehensive safety guidelines -- **skill-reviewer**: See `skill-reviewer/references/evaluation_checklist.md` for complete evaluation criteria, `skill-reviewer/references/pr_template.md` for PR templates, and `skill-reviewer/references/marketplace_template.json` for marketplace configuration +- **skill-reviewer**: See `daymade-skill/skill-reviewer/references/evaluation_checklist.md` for complete evaluation criteria, `daymade-skill/skill-reviewer/references/pr_template.md` for PR templates, and `daymade-skill/skill-reviewer/references/marketplace_template.json` for marketplace configuration - **github-contributor**: See `github-contributor/references/pr_checklist.md` for PR quality checklist, `github-contributor/references/project_evaluation.md` for project evaluation criteria, and `github-contributor/references/communication_templates.md` for issue/PR templates - **i18n-expert**: See `i18n-expert/SKILL.md` for complete i18n setup workflow, key architecture guidance, and audit procedures - **claude-skills-troubleshooting**: See `daymade-claude-code/claude-skills-troubleshooting/SKILL.md` for plugin troubleshooting workflow and architecture diff --git a/README.zh-CN.md b/README.zh-CN.md index 9494a450..61d56c6d 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -66,7 +66,7 @@ | 用户体验 | 4 | 9 | | **总分（/80）** | **42** | **65** | -> 完整方法论：[skill-creator/references/skill-development-methodology.md](./skill-creator/references/skill-development-methodology.md) +> 完整方法论：[skill-creator/references/skill-development-methodology.md](./daymade-skill/skill-creator/references/skill-development-methodology.md) ### 快速安装 @@ -102,7 +102,7 @@ claude plugin install skill-creator@daymade-skills 加载了 skill-creator 的 Claude Code 将引导你完成整个技能创建过程——从理解你的需求到打包最终技能。 -📚 **完整文档**：[skill-creator/SKILL.md](./skill-creator/SKILL.md) +📚 **完整文档**：[daymade-skill/skill-creator/SKILL.md](./daymade-skill/daymade-skill/skill-creator/SKILL.md) ### 实时演示 @@ -1011,7 +1011,7 @@ ccpm install-bundle web-dev # 安装 Web 开发技能包 *即将推出* -📚 **文档**：参见 [skills-search/SKILL.md](./skills-search/SKILL.md) 了解完整的命令参考 +📚 **文档**：参见 [daymade-skill/skills-search/SKILL.md](./daymade-skill/daymade-skill/skills-search/SKILL.md) 了解完整的命令参考 **要求**：CCPM CLI（`npm install -g @daymade/ccpm`） @@ -1370,7 +1370,7 @@ claude plugin install skill-reviewer@daymade-skills *即将推出* -📚 **文档**：参见 [skill-reviewer/references/](./skill-reviewer/references/) 了解： +📚 **文档**：参见 [daymade-skill/skill-reviewer/references/](./daymade-skill/daymade-skill/skill-reviewer/references/) 了解： - `evaluation_checklist.md` - 完整的技能评估标准 - `pr_template.md` - 专业 PR 描述模板 - `marketplace_template.json` - marketplace 配置模板 @@ -2305,7 +2305,7 @@ uv run douban-skill/scripts/douban-rss-sync.py - **statusline-generator**：参见 `daymade-claude-code/statusline-generator/references/color_codes.md` 了解自定义 - **teams-channel-post-writer**：参见 `teams-channel-post-writer/references/writing-guidelines.md` 了解质量标准 - **repomix-unmixer**：参见 `repomix-unmixer/references/repomix-format.md` 了解格式规范 -- **skill-creator**：参见 `skill-creator/SKILL.md` 了解完整的技能创建工作流 +- **skill-creator**：参见 `daymade-skill/skill-creator/SKILL.md` 了解完整的技能创建工作流 - **llm-icon-finder**：参见 `llm-icon-finder/references/icons-list.md` 了解可用图标 - **cli-demo-generator**：参见 `cli-demo-generator/references/vhs_syntax.md` 了解 VHS 语法和 `cli-demo-generator/references/best_practices.md` 了解演示指南 - **cloudflare-troubleshooting**：参见 `cloudflare-troubleshooting/references/api_overview.md` 了解 API 文档 @@ -2322,12 +2322,12 @@ uv run douban-skill/scripts/douban-rss-sync.py - **deep-research**：参见 `deep-research/references/research_report_template.md` 了解报告结构，并参见 `deep-research/references/source_quality_rubric.md` 了解来源分级标准 - **pdf-creator**：参见 `daymade-docs/pdf-creator/SKILL.md` 了解 PDF 转换与字体设置 - **claude-md-progressive-disclosurer**：参见 `daymade-claude-code/claude-md-progressive-disclosurer/SKILL.md` 了解 CLAUDE.md 优化工作流 -- **skills-search**：参见 `skills-search/SKILL.md` 了解 CCPM CLI 命令和注册表操作 +- **skills-search**：参见 `daymade-skill/skills-search/SKILL.md` 了解 CCPM CLI 命令和注册表操作 - **promptfoo-evaluation**：参见 `promptfoo-evaluation/references/promptfoo_api.md` 了解评测模式 - **iOS-APP-developer**：参见 `iOS-APP-developer/references/xcodegen-full.md` 了解 XcodeGen 选项与 project.yml 细节 - **twitter-reader**：参见 `twitter-reader/SKILL.md` 了解 API 密钥设置和 URL 格式支持 - **macos-cleaner**：参见 `macos-cleaner/references/cleanup_targets.md` 了解详细清理目标说明、`macos-cleaner/references/mole_integration.md` 了解 Mole 可视化工具集成、`macos-cleaner/references/safety_rules.md` 了解全面安全指南 -- **skill-reviewer**：参见 `skill-reviewer/references/evaluation_checklist.md` 了解完整评估标准、`skill-reviewer/references/pr_template.md` 了解 PR 模板、`skill-reviewer/references/marketplace_template.json` 了解 marketplace 配置 +- **skill-reviewer**：参见 `daymade-skill/skill-reviewer/references/evaluation_checklist.md` 了解完整评估标准、`daymade-skill/skill-reviewer/references/pr_template.md` 了解 PR 模板、`daymade-skill/skill-reviewer/references/marketplace_template.json` 了解 marketplace 配置 - **github-contributor**：参见 `github-contributor/references/pr_checklist.md` 了解 PR 质量清单、`github-contributor/references/project_evaluation.md` 了解项目评估标准、`github-contributor/references/communication_templates.md` 了解 issue/PR 沟通模板 - **i18n-expert**：参见 `i18n-expert/SKILL.md` 了解完整的 i18n 设置工作流程、键架构指导和审计程序 - **claude-skills-troubleshooting**：参见 `daymade-claude-code/claude-skills-troubleshooting/SKILL.md` 了解插件故障排除工作流程和架构 diff --git a/skill-creator/.gitignore b/daymade-skill/skill-creator/.gitignore similarity index 100% rename from skill-creator/.gitignore rename to daymade-skill/skill-creator/.gitignore diff --git a/skill-creator/LICENSE.txt b/daymade-skill/skill-creator/LICENSE.txt similarity index 100% rename from skill-creator/LICENSE.txt rename to daymade-skill/skill-creator/LICENSE.txt diff --git a/skill-creator/SKILL.md b/daymade-skill/skill-creator/SKILL.md similarity index 100% rename from skill-creator/SKILL.md rename to daymade-skill/skill-creator/SKILL.md diff --git a/skill-creator/agents/analyzer.md b/daymade-skill/skill-creator/agents/analyzer.md similarity index 100% rename from skill-creator/agents/analyzer.md rename to daymade-skill/skill-creator/agents/analyzer.md diff --git a/skill-creator/agents/comparator.md b/daymade-skill/skill-creator/agents/comparator.md similarity index 100% rename from skill-creator/agents/comparator.md rename to daymade-skill/skill-creator/agents/comparator.md diff --git a/skill-creator/agents/grader.md b/daymade-skill/skill-creator/agents/grader.md similarity index 100% rename from skill-creator/agents/grader.md rename to daymade-skill/skill-creator/agents/grader.md diff --git a/skill-creator/assets/eval_review.html b/daymade-skill/skill-creator/assets/eval_review.html similarity index 100% rename from skill-creator/assets/eval_review.html rename to daymade-skill/skill-creator/assets/eval_review.html diff --git a/skill-creator/eval-viewer/generate_review.py b/daymade-skill/skill-creator/eval-viewer/generate_review.py similarity index 100% rename from skill-creator/eval-viewer/generate_review.py rename to daymade-skill/skill-creator/eval-viewer/generate_review.py diff --git a/skill-creator/eval-viewer/viewer.html b/daymade-skill/skill-creator/eval-viewer/viewer.html similarity index 100% rename from skill-creator/eval-viewer/viewer.html rename to daymade-skill/skill-creator/eval-viewer/viewer.html diff --git a/skill-creator/references/prerequisites.md b/daymade-skill/skill-creator/references/prerequisites.md similarity index 100% rename from skill-creator/references/prerequisites.md rename to daymade-skill/skill-creator/references/prerequisites.md diff --git a/skill-creator/references/sanitization_checklist.md b/daymade-skill/skill-creator/references/sanitization_checklist.md similarity index 100% rename from skill-creator/references/sanitization_checklist.md rename to daymade-skill/skill-creator/references/sanitization_checklist.md diff --git a/skill-creator/references/schemas.md b/daymade-skill/skill-creator/references/schemas.md similarity index 100% rename from skill-creator/references/schemas.md rename to daymade-skill/skill-creator/references/schemas.md diff --git a/skill-creator/references/skill-development-methodology.md b/daymade-skill/skill-creator/references/skill-development-methodology.md similarity index 100% rename from skill-creator/references/skill-development-methodology.md rename to daymade-skill/skill-creator/references/skill-development-methodology.md diff --git a/skill-creator/scripts/__init__.py b/daymade-skill/skill-creator/scripts/__init__.py similarity index 100% rename from skill-creator/scripts/__init__.py rename to daymade-skill/skill-creator/scripts/__init__.py diff --git a/skill-creator/scripts/aggregate_benchmark.py b/daymade-skill/skill-creator/scripts/aggregate_benchmark.py similarity index 100% rename from skill-creator/scripts/aggregate_benchmark.py rename to daymade-skill/skill-creator/scripts/aggregate_benchmark.py diff --git a/skill-creator/scripts/generate_report.py b/daymade-skill/skill-creator/scripts/generate_report.py similarity index 100% rename from skill-creator/scripts/generate_report.py rename to daymade-skill/skill-creator/scripts/generate_report.py diff --git a/skill-creator/scripts/improve_description.py b/daymade-skill/skill-creator/scripts/improve_description.py similarity index 100% rename from skill-creator/scripts/improve_description.py rename to daymade-skill/skill-creator/scripts/improve_description.py diff --git a/skill-creator/scripts/init_skill.py b/daymade-skill/skill-creator/scripts/init_skill.py similarity index 100% rename from skill-creator/scripts/init_skill.py rename to daymade-skill/skill-creator/scripts/init_skill.py diff --git a/skill-creator/scripts/package_skill.py b/daymade-skill/skill-creator/scripts/package_skill.py similarity index 100% rename from skill-creator/scripts/package_skill.py rename to daymade-skill/skill-creator/scripts/package_skill.py diff --git a/skill-creator/scripts/quick_validate.py b/daymade-skill/skill-creator/scripts/quick_validate.py similarity index 100% rename from skill-creator/scripts/quick_validate.py rename to daymade-skill/skill-creator/scripts/quick_validate.py diff --git a/skill-creator/scripts/run_eval.py b/daymade-skill/skill-creator/scripts/run_eval.py similarity index 100% rename from skill-creator/scripts/run_eval.py rename to daymade-skill/skill-creator/scripts/run_eval.py diff --git a/skill-creator/scripts/run_loop.py b/daymade-skill/skill-creator/scripts/run_loop.py similarity index 100% rename from skill-creator/scripts/run_loop.py rename to daymade-skill/skill-creator/scripts/run_loop.py diff --git a/skill-creator/scripts/security_scan.py b/daymade-skill/skill-creator/scripts/security_scan.py similarity index 100% rename from skill-creator/scripts/security_scan.py rename to daymade-skill/skill-creator/scripts/security_scan.py diff --git a/skill-creator/scripts/utils.py b/daymade-skill/skill-creator/scripts/utils.py similarity index 100% rename from skill-creator/scripts/utils.py rename to daymade-skill/skill-creator/scripts/utils.py diff --git a/skill-creator/workflows/wrapper-skill/architecture_contract.md b/daymade-skill/skill-creator/workflows/wrapper-skill/architecture_contract.md similarity index 100% rename from skill-creator/workflows/wrapper-skill/architecture_contract.md rename to daymade-skill/skill-creator/workflows/wrapper-skill/architecture_contract.md diff --git a/skill-creator/workflows/wrapper-skill/patterns.md b/daymade-skill/skill-creator/workflows/wrapper-skill/patterns.md similarity index 100% rename from skill-creator/workflows/wrapper-skill/patterns.md rename to daymade-skill/skill-creator/workflows/wrapper-skill/patterns.md diff --git a/skill-creator/workflows/wrapper-skill/scripts/init_wrapper_skill.py b/daymade-skill/skill-creator/workflows/wrapper-skill/scripts/init_wrapper_skill.py similarity index 100% rename from skill-creator/workflows/wrapper-skill/scripts/init_wrapper_skill.py rename to daymade-skill/skill-creator/workflows/wrapper-skill/scripts/init_wrapper_skill.py diff --git a/skill-creator/workflows/wrapper-skill/verification_protocol.md b/daymade-skill/skill-creator/workflows/wrapper-skill/verification_protocol.md similarity index 100% rename from skill-creator/workflows/wrapper-skill/verification_protocol.md rename to daymade-skill/skill-creator/workflows/wrapper-skill/verification_protocol.md diff --git a/skill-creator/workflows/wrapper-skill/workflow.md b/daymade-skill/skill-creator/workflows/wrapper-skill/workflow.md similarity index 100% rename from skill-creator/workflows/wrapper-skill/workflow.md rename to daymade-skill/skill-creator/workflows/wrapper-skill/workflow.md diff --git a/skill-reviewer/.security-scan-passed b/daymade-skill/skill-reviewer/.security-scan-passed similarity index 100% rename from skill-reviewer/.security-scan-passed rename to daymade-skill/skill-reviewer/.security-scan-passed diff --git a/skill-reviewer/SKILL.md b/daymade-skill/skill-reviewer/SKILL.md similarity index 100% rename from skill-reviewer/SKILL.md rename to daymade-skill/skill-reviewer/SKILL.md diff --git a/skill-reviewer/references/evaluation_checklist.md b/daymade-skill/skill-reviewer/references/evaluation_checklist.md similarity index 100% rename from skill-reviewer/references/evaluation_checklist.md rename to daymade-skill/skill-reviewer/references/evaluation_checklist.md diff --git a/skill-reviewer/references/marketplace_template.json b/daymade-skill/skill-reviewer/references/marketplace_template.json similarity index 100% rename from skill-reviewer/references/marketplace_template.json rename to daymade-skill/skill-reviewer/references/marketplace_template.json diff --git a/skill-reviewer/references/pr_template.md b/daymade-skill/skill-reviewer/references/pr_template.md similarity index 100% rename from skill-reviewer/references/pr_template.md rename to daymade-skill/skill-reviewer/references/pr_template.md diff --git a/skills-search/.security-scan-passed b/daymade-skill/skills-search/.security-scan-passed similarity index 100% rename from skills-search/.security-scan-passed rename to daymade-skill/skills-search/.security-scan-passed diff --git a/skills-search/SKILL.md b/daymade-skill/skills-search/SKILL.md similarity index 100% rename from skills-search/SKILL.md rename to daymade-skill/skills-search/SKILL.md From 99c76a751b6bddd1f93823cc6f9fad2d53f75108 Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 10 May 2026 01:42:44 +0800 Subject: [PATCH 130/137] feat(pdf-creator): batch theme support + mobile theme + anti-pattern docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - batch_convert.py: add --theme, --backend, --no-preview flags - themes/mobile.css: new mobile-friendly theme (148mm×210mm, 15px, 1.9 line-height) - SKILL.md: pushier description, Anti-Pattern section, Print vs Mobile decision table Co-Authored-By: Claude Opus 4.7 --- .../pdf-creator/.security-scan-passed | 4 + daymade-docs/pdf-creator/SKILL.md | 69 +++++++-- .../pdf-creator/scripts/batch_convert.py | 28 +++- daymade-docs/pdf-creator/themes/mobile.css | 146 ++++++++++++++++++ 4 files changed, 233 insertions(+), 14 deletions(-) create mode 100644 daymade-docs/pdf-creator/.security-scan-passed create mode 100644 daymade-docs/pdf-creator/themes/mobile.css diff --git a/daymade-docs/pdf-creator/.security-scan-passed b/daymade-docs/pdf-creator/.security-scan-passed new file mode 100644 index 00000000..c202a5d1 --- /dev/null +++ b/daymade-docs/pdf-creator/.security-scan-passed @@ -0,0 +1,4 @@ +Security scan passed +Scanned at: 2026-05-10T00:54:48.296005 +Tool: gitleaks + pattern-based validation +Content hash: 7804284325ef2700b95f52e849e113b139a7fcd67062856e2b87b1bf2c1ecb6c diff --git a/daymade-docs/pdf-creator/SKILL.md b/daymade-docs/pdf-creator/SKILL.md index eb54cc5e..e142312f 100644 --- a/daymade-docs/pdf-creator/SKILL.md +++ b/daymade-docs/pdf-creator/SKILL.md @@ -1,6 +1,6 @@ --- name: pdf-creator -description: Create PDF documents from markdown with proper Chinese font support. Supports theme system (default for formal docs, warm-terra for training materials) and dual backend (weasyprint or Chrome). Triggers include "convert to PDF", "generate PDF", "markdown to PDF", or any request for creating printable documents. +description: Convert markdown files to professional PDF documents with proper Chinese font support, theme system, and visual self-check. Use whenever the user asks to create PDFs, convert markdown to PDF, generate printable documents, or needs documents formatted for print or mobile reading. This skill MUST be used instead of manual pandoc/Chrome invocations — it handles CJK typography, Chrome header/footer suppression, and mandatory visual verification that manual approaches miss. --- # PDF Creator @@ -10,12 +10,18 @@ Create professional PDF documents from markdown with Chinese font support and th ## Quick Start ```bash -# Default theme (formal: Songti SC + black/grey) +# Default theme (formal: Songti SC + black/grey, A4 print) uv run --with weasyprint scripts/md_to_pdf.py input.md output.pdf # Warm theme (training: PingFang SC + terra cotta) uv run --with weasyprint scripts/md_to_pdf.py input.md --theme warm-terra +# Mobile theme (narrow page, large font — for phone reading / WeChat sharing) +uv run --with weasyprint scripts/md_to_pdf.py input.md --theme mobile + +# Batch convert all markdown files with a specific theme +uv run --with weasyprint scripts/batch_convert.py *.md --theme warm-terra --no-preview + # No weasyprint? Use Chrome backend (auto-detected if weasyprint unavailable) python scripts/md_to_pdf.py input.md --theme warm-terra --backend chrome @@ -27,13 +33,25 @@ python scripts/md_to_pdf.py --list-themes dummy.md Stored in `themes/*.css`. Each theme is a standalone CSS file. -| Theme | Font | Color | Best for | -|-------|------|-------|----------| -| `default` | Songti SC + Heiti SC | Black/grey | Legal docs, contracts, formal reports | -| `warm-terra` | PingFang SC | Terra cotta (#d97756) + warm neutrals | Course outlines, training materials, workshops | +| Theme | Page Size | Font | Color | Best for | +|-------|-----------|------|-------|----------| +| `default` | A4 | Songti SC + Heiti SC | Black/grey | Legal docs, contracts, formal reports | +| `warm-terra` | A4 | PingFang SC | Terra cotta (#d97756) + warm neutrals | Course outlines, training materials, workshops | +| `mobile` | 148mm × 210mm | PingFang SC | Terra cotta + warm neutrals | Phone reading, WeChat sharing, on-the-go reference | To create a new theme: copy `themes/default.css`, modify, save as `themes/your-theme.css`. +## Print vs Mobile: Choose the Right Theme + +| Scenario | Recommended Theme | Why | +|----------|-------------------|-----| +| Print on A4 paper, handouts, contracts | `default` | Standard page size, formal typography | +| Training materials, course outlines | `warm-terra` | Warm accent color, readable for workshop contexts | +| Send via WeChat, read on phone | `mobile` | Narrow page (148mm), 15px font, 1.9 line-height — comfortable on small screens | +| Both print AND mobile needed | Run twice with different themes | The skill is fast; generate both versions | + +**Decision rule:** If the user does not specify, default to `warm-terra` for training/course content and `default` for formal documents. Ask "是否需要手机版？" only when the output channel is unclear. + ## Backends The script auto-detects the best available backend: @@ -48,9 +66,30 @@ Override with `--backend chrome` or `--backend weasyprint`. ## Batch Convert ```bash -uv run --with weasyprint scripts/batch_convert.py *.md --output-dir ./pdfs +# Default theme, same directory +uv run --with weasyprint scripts/batch_convert.py *.md + +# Specific theme, output directory, skip previews for speed +uv run --with weasyprint scripts/batch_convert.py *.md --theme warm-terra --output-dir ./pdfs --no-preview + +# Mobile theme for phone reading +uv run --with weasyprint scripts/batch_convert.py *.md --theme mobile --output-dir ./mobile-pdfs --no-preview ``` +## Anti-Pattern: Do NOT Manually Invoke pandoc + Chrome + +**Why this skill exists:** Manual `pandoc input.md -o out.html` + `chrome --headless --print-to-pdf` workflows silently fail in ways that are hard to detect: + +| Manual Step | What Goes Wrong | This Skill Fixes | +|---|---|---| +| `pandoc -o out.html` | No CJK-aware CSS → boxes/blanks for Chinese | Injects CJK font stack + typography patch | +| Chrome `--print-to-pdf` | Default header/footer appears (filename, date, URL, page numbers) | Passes `--no-pdf-header-footer` | +| No post-render check | "Exit code 0" assumed success; rendering bugs hidden | Auto-generates per-page PNG previews + typography lint | +| No theme system | One-size-fits-all; phone reading impossible | Three curated themes (default / warm-terra / mobile) | +| `batch_convert.py` missing | Writing ad-hoc loops, inconsistent flags | Built-in batch mode with `--theme` support | + +**Rule:** When the user asks for PDF conversion, ALWAYS use this skill. Never bypass it with manual pandoc/Chrome commands. + ## Troubleshooting **Chinese characters display as boxes**: Ensure Chinese fonts are installed (Songti SC, PingFang SC, etc.) @@ -59,20 +98,26 @@ uv run --with weasyprint scripts/batch_convert.py *.md --output-dir ./pdfs **CJK text in code blocks garbled (weasyprint)**: The script auto-detects code blocks containing Chinese/Japanese/Korean characters and converts them to styled divs with CJK-capable fonts. If you still see issues, use `--backend chrome` which has native CJK support. Alternatively, convert code blocks to markdown tables before generating the PDF. -**Chrome header/footer appearing**: The script passes `--no-pdf-header-footer`. If it still appears, your Chrome version may not support this flag — update Chrome. +**Chrome header/footer appearing**: The script passes `--no-pdf-header-footer`. If it still appears, your Chrome version may not support this flag — update Chrome. **Note:** If you bypassed this skill and used manual Chrome headless, this is the first symptom — see "Anti-Pattern" section above. **Inline code with mixed CJK + ASCII shows blanks in macOS Preview** (e.g. `` `Terminal/终端` `` renders only `Terminal/` with the CJK part missing): weasyprint subset-embeds PingFang SC as **OpenType (CID Type 0C)**, which strict PDF readers (macOS Preview / Adobe Reader) fail to render. Chrome's PDF viewer falls back automatically and hides the bug. Fix is in the default theme: code font-family chain prioritizes **CID TrueType** CJK fonts (Songti SC / Heiti SC) before OpenType ones (PingFang SC). To verify: `pdfplumber` + check `font['fontname']` of CJK chars — if any references `PingFang-SC` (CID Type 0C OT), readers will likely fail. Reorder font chain to put CID TrueType first. **Table column 1 with short label gets mid-broken** (e.g. `4/28（周|二）下|午`): pandoc auto-emits `` from dash counts in the markdown separator row. For `| ----- | --- | --- | -------- |` (uneven dash widths), pandoc allocates col 1 ~17% — too narrow for a 9-char CJK label. Inline `style=""` beats external CSS at equal specificity, so `td:first-child { width:... }` is silently shadowed. Fix is in default theme: `table colgroup col { width: auto !important }` neutralizes pandoc's hint, letting `table-layout: fixed` distribute equally (25% per column for a 4-col table). To verify: `pandoc input.md -t html | grep colgroup` — if it shows ``, the bug applies. -## Visual Self-Check (default behavior) +## Visual Self-Check (MANDATORY — Do Not Skip) -After every PDF generation, the script automatically: +**This is not optional.** After every PDF generation, the script automatically: 1. Converts each page to PNG via `pdftoppm` (poppler-utils) into a `-preview/` directory next to the PDF 2. Prints a structured self-check checklist reminding the caller to visually inspect each page - -**Why**: "PDF generated cleanly" ≠ "rendering matches markdown intent". Common silent failures include paragraphs collapsing into one (CommonMark soft-break behavior on consecutive non-blank lines), tables overflowing page margins, missing CJK / emoji glyphs, code block garbling. The checklist enforces visual verification as the default contract — not an optional step that's easy to skip. +3. Runs typography lint to detect CJK line-break anti-patterns + +**Why mandatory**: "PDF generated cleanly" ≠ "rendering matches markdown intent". Common silent failures include: +- Paragraphs collapsing into one (CommonMark soft-break on consecutive non-blank lines) +- Tables overflowing page margins +- Missing CJK / emoji glyphs +- Code block garbling +- Chrome default headers/footers (if bypassed this skill) **Workflow**: After running the script, `Read` each `page-NN.png` and verify against the markdown source. If anything renders differently from intent, **fix the markdown** (use `- ` real lists instead of pseudo-lists, insert blank lines, restructure tables) and rerun. The script does NOT silently "fix" non-standard markdown — that would mask the signal that the source is wrong, causing the same markdown to render incorrectly in other processors (Obsidian, GitHub, VS Code preview). diff --git a/daymade-docs/pdf-creator/scripts/batch_convert.py b/daymade-docs/pdf-creator/scripts/batch_convert.py index 25765d56..704c97e9 100644 --- a/daymade-docs/pdf-creator/scripts/batch_convert.py +++ b/daymade-docs/pdf-creator/scripts/batch_convert.py @@ -34,6 +34,24 @@ def main(): default=None, help='Output directory for PDFs (default: same as input)' ) + parser.add_argument( + '--theme', '-t', + type=str, + default='default', + help='CSS theme name (default: default). Available themes depend on what is in themes/' + ) + parser.add_argument( + '--backend', '-b', + type=str, + default=None, + choices=['weasyprint', 'chrome'], + help='PDF rendering backend (default: auto-detect)' + ) + parser.add_argument( + '--no-preview', + action='store_true', + help='Skip per-page PNG preview generation (faster for batch runs)' + ) args = parser.parse_args() @@ -64,8 +82,14 @@ def main(): pdf_file = str(md_path.with_suffix('.pdf')) try: - print(f"Converting: {md_file} -> {pdf_file}") - markdown_to_pdf(str(md_path), pdf_file) + print(f"Converting: {md_file} -> {pdf_file} (theme={args.theme})") + markdown_to_pdf( + str(md_path), + pdf_file, + theme=args.theme, + backend=args.backend, + previews=not args.no_preview, + ) success += 1 except Exception as e: print(f"[ERROR] Failed to convert {md_file}: {e}") diff --git a/daymade-docs/pdf-creator/themes/mobile.css b/daymade-docs/pdf-creator/themes/mobile.css new file mode 100644 index 00000000..603bae05 --- /dev/null +++ b/daymade-docs/pdf-creator/themes/mobile.css @@ -0,0 +1,146 @@ +/* + * Mobile — PDF theme for phone reading + * + * Narrow page (A5-ish), larger fonts, generous line-height. + * Best for: mobile reading, WeChat sharing, on-the-go reference + */ + +@page { + size: 148mm 210mm; + margin: 10mm; +} + +body { + font-family: 'PingFang SC', 'Microsoft YaHei', 'Noto Sans CJK SC', sans-serif; + max-width: 100%; + margin: 0 auto; + padding: 0; + font-size: 15px; + line-height: 1.9; + color: #1f1b17; +} + +h1 { + font-size: 26px; + font-weight: 800; + border-bottom: 2px solid #d97756; + padding-bottom: 10px; + margin-top: 0; + margin-bottom: 1em; + line-height: 1.3; +} + +h2 { + font-size: 20px; + font-weight: 700; + color: #d97756; + margin-top: 28px; + margin-bottom: 0.7em; + line-height: 1.3; +} + +h3 { + font-size: 17px; + font-weight: 700; + margin-top: 22px; + margin-bottom: 0.6em; + line-height: 1.3; +} + +p { + margin: 0.8em 0; +} + +ul, ol { + padding-left: 24px; + margin: 0.8em 0; +} + +li { + margin-bottom: 6px; + word-break: break-word; +} + +table { + border-collapse: collapse; + width: 100%; + margin: 12px 0; + font-size: 13px; +} + +th, td { + border: 1px solid #e2d6c8; + padding: 6px 8px; + text-align: left; + white-space: normal; + word-break: break-word; +} + +th { + background: #faf5f0; + font-weight: 700; +} + +blockquote { + border-left: 3px solid #d97756; + padding-left: 14px; + color: #6c6158; + margin: 14px 0; + font-size: 15px; + line-height: 1.8; +} + +hr { + border: none; + border-top: 1px solid #e2d6c8; + margin: 20px 0; +} + +header, .date { + display: none !important; +} + +code { + font-family: 'Menlo', 'PingFang SC', 'Microsoft YaHei', 'Noto Sans CJK SC', monospace; + background: #faf5f0; + padding: 2px 5px; + border-radius: 3px; + font-size: 13px; +} + +pre { + background: #faf5f0; + border: 1px solid #e2d6c8; + border-radius: 4px; + padding: 14px 16px; + margin: 12px 0; + overflow-wrap: break-word; + white-space: pre-wrap; + word-break: break-all; +} + +pre code { + font-family: 'Menlo', 'PingFang SC', 'Microsoft YaHei', 'Noto Sans CJK SC', monospace; + background: none; + padding: 0; + border-radius: 0; + font-size: 12px; + line-height: 1.7; +} + +.cjk-code-block { + font-family: inherit; + background: #faf5f0; + border: 1px solid #e2d6c8; + border-radius: 4px; + padding: 14px 16px; + margin: 12px 0; + font-size: 13px; + line-height: 1.8; + white-space: pre-wrap; + word-break: break-all; +} + +strong { + color: #1f1b17; +} From 3f3aeed4219f2eaaaed42a17e795f7b5556cb050 Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 10 May 2026 01:57:43 +0800 Subject: [PATCH 131/137] fix(marketplace): remove skills: ["./"] from 13 plugins (#64) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude Code 2.1.x path-escape validator rejects skills: ["./"] as escaping the plugin root. These 13 suite member plugins already have source pointing to the correct skill directory — the explicit skills field was redundant. Removing it lets auto-discovery handle resolution. Co-Authored-By: Claude Opus 4.6 (1M context) --- .claude-plugin/marketplace.json | 97 +++++++++++++++++---------------- CHANGELOG.md | 21 +++++++ 2 files changed, 71 insertions(+), 47 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 9c9ba2df..498613cb 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -5,8 +5,8 @@ "email": "daymadev89@gmail.com" }, "metadata": { - "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, macOS programmatic window screenshot capture workflows, verified Scrapling CLI installation and web extraction workflows, plugin marketplace development for converting skills repos into official Claude Code marketplaces, Tencent IMA knowledge base companion with zero-config installation across Claude Code/Codex/OpenClaw, upstream bug detection and runtime repair, personalized fan-out search with priority-based knowledge base boosting, and suite plugins that expose related skills under shared namespaces for combined installation workflows — including a dedicated Claude Code operations suite (daymade-claude-code) bundling session recovery, CLAUDE.md optimization, troubleshooting, statusline configuration, export repair, and marketplace development skills under one namespace", - "version": "1.46.1" + "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, macOS programmatic window screenshot capture workflows, verified Scrapling CLI installation and web extraction workflows, plugin marketplace development for converting skills repos into official Claude Code marketplaces, Tencent IMA knowledge base companion with zero-config installation across Claude Code/Codex/OpenClaw, upstream bug detection and runtime repair, personalized fan-out search with priority-based knowledge base boosting, and suite plugins that expose related skills under shared namespaces for combined installation workflows — including a dedicated Claude Code operations suite (daymade-claude-code) bundling session recovery, CLAUDE.md optimization, troubleshooting, statusline configuration, export repair, and marketplace development skills under one namespace, and the StepFun StepAudio 2.5 audio family (stepfun-tts for Contextual TTS with instruction + inline parentheses, stepfun-asr for the SSE endpoint that handles 30-minute audio in a single call)", + "version": "1.53.2" }, "plugins": [ { @@ -66,9 +66,6 @@ "file-tracking", "claude-code", "history-analysis" - ], - "skills": [ - "./" ] }, { @@ -85,9 +82,6 @@ "fix", "line-wrapping", "formatting" - ], - "skills": [ - "./" ] }, { @@ -104,9 +98,6 @@ "context-efficiency", "configuration", "token-savings" - ], - "skills": [ - "./" ] }, { @@ -126,9 +117,6 @@ "enabledPlugins", "settings", "marketplace" - ], - "skills": [ - "./" ] }, { @@ -209,9 +197,6 @@ "history", "workflow-continuation", "local-artifacts" - ], - "skills": [ - "./" ] }, { @@ -327,9 +312,6 @@ "document", "cjk", "chinese" - ], - "skills": [ - "./" ] }, { @@ -346,9 +328,6 @@ "redundancy", "merge", "docs" - ], - "skills": [ - "./" ] }, { @@ -414,6 +393,27 @@ "./fact-checker" ] }, + { + "name": "feishu-doc-scraper", + "description": "Save Feishu Docs and Feishu Wiki pages as clean Markdown from a live authenticated browser session. Uses TOC-driven section capture and coverage verification, and explicitly avoids relying on Web Clipper for virtual-scroll pages. Use when users ask to save, export, scrape, or archive a Feishu doc/wiki to Markdown, or when a Feishu page already open in Chrome needs to be turned into a local note.", + "source": "./", + "strict": false, + "version": "1.0.0", + "category": "productivity", + "keywords": [ + "feishu", + "lark", + "markdown", + "wiki", + "document", + "scraping", + "browser", + "archival" + ], + "skills": [ + "./feishu-doc-scraper" + ] + }, { "name": "financial-data-collector", "description": "Collect real financial data for any US publicly traded company from free public sources (yfinance). Output structured JSON with market data, historical financials, WACC inputs, and analyst estimates. Handles NaN year detection, CapEx sign preservation, and FCF definition mismatches. Use when users request company financials, stock data, DCF inputs, or financial data collection for any US equity ticker", @@ -624,9 +624,6 @@ "distribution", "packaging" ], - "skills": [ - "./" - ], "hooks": { "PostToolUse": [ { @@ -666,9 +663,6 @@ "mermaid", "quotes", "action-items" - ], - "skills": [ - "./" ] }, { @@ -684,9 +678,6 @@ "visualization", "flowchart", "sequence" - ], - "skills": [ - "./" ] }, { @@ -708,9 +699,6 @@ "legal", "reports", "typography" - ], - "skills": [ - "./" ] }, { @@ -729,9 +717,6 @@ "charts", "data-visualization", "pyramid-principle" - ], - "skills": [ - "./" ] }, { @@ -915,34 +900,52 @@ "git-status", "customization", "prompt" - ], - "skills": [ - "./" ] }, { "name": "stepfun-tts", - "description": "Generate speech and transcribe audio using StepFun's StepAudio 2.5 family — stepaudio-2.5-tts (Contextual TTS with instruction + inline parentheses) and stepaudio-2.5-asr (SSE endpoint, 32K context, ~100x RTF, handles up to 30-minute audio in a single call). Use when the user wants Chinese/Japanese TTS with emotional/prosody control, needs to transcribe long audio, migrates from step-tts-2 to stepaudio-2.5-tts (voice_label → instruction breaking change), or hits StepFun censorship / endpoint errors. Also triggers for 阶跃 TTS, StepAudio 合成, 语音合成, 配音, StepFun ASR, 转录, 语音识别.", + "description": "Generate Chinese / Japanese speech with StepFun's stepaudio-2.5-tts — Contextual TTS that replaces step-tts-2's `voice_label` with natural-language `instruction` (≤200 chars) plus inline `()` parentheses for句内 prosody. Use when the user wants emotional / prosody control over voice synthesis (whisper, pause, stress, mood pivot mid-sentence), batch-generates game / app voice lines, migrates from `step-tts-2` (the `voice_label → instruction` breaking change), or hits StepFun's stricter 2.5-era censorship (死/消失/political terms). Triggers on 阶跃 TTS, StepAudio 合成, 语音合成, 配音, 文本转语音, TTS 升级, 迁移 step-tts-2. For transcription with the sibling stepaudio-2.5-asr model, use the stepfun-asr skill instead.", "source": "./", "strict": false, - "version": "1.0.0", + "version": "2.0.0", "category": "productivity", "keywords": [ "tts", - "asr", "stepfun", "stepaudio", "stepaudio-2.5", "阶跃", "chinese-tts", "voice-synthesis", - "speech-to-text", "contextual-tts" ], "skills": [ "./stepfun-tts" ] }, + { + "name": "stepfun-asr", + "description": "Transcribe audio with StepFun's stepaudio-2.5-asr — an SSE endpoint (NOT /v1/audio/transcriptions) with 32K context, ~85-101x RTF on long audio, and a single-call ceiling around 30 minutes (no client-side chunking). Use when transcribing Chinese / English audio with StepFun, when long-form recordings (5-30 min) need to land in one request, when migrating from step-asr / step-asr-1.1, or when hitting the misleading `model stepaudio-2.5-asr not supported` error (which actually means wrong endpoint). Triggers on 阶跃 ASR, StepFun ASR, stepaudio-2.5-asr, 转录, 语音识别, 长音频转写, 语音转文字. For TTS with the sibling stepaudio-2.5-tts model, use the stepfun-tts skill instead.", + "source": "./", + "strict": false, + "version": "1.0.0", + "category": "productivity", + "keywords": [ + "asr", + "stepfun", + "stepaudio", + "stepaudio-2.5-asr", + "阶跃", + "chinese-asr", + "speech-to-text", + "transcription", + "long-audio", + "sse" + ], + "skills": [ + "./stepfun-asr" + ] + }, { "name": "teams-channel-post-writer", "description": "Create professional Microsoft Teams channel posts with Adaptive Cards, formatted announcements, and corporate communication standards", @@ -1004,10 +1007,10 @@ }, { "name": "tunnel-doctor", - "description": "Diagnoses and fixes conflicts between Tailscale and proxy/VPN tools (Shadowrocket, Clash, Surge) on macOS. Covers five conflict layers: route hijacking, HTTP proxy env var interception, system proxy bypass, SSH ProxyCommand double tunneling, and VM/container proxy propagation. Includes an automated quick-diagnose script plus SOP for remote development via SSH tunnels with proxy-safe Makefile patterns. Use when Tailscale ping works but SSH/HTTP times out, when browser returns 503 but curl works, when local vanity domains fail behind proxy, when git push fails with failed to begin relaying via HTTP, when Docker pull times out behind TUN/VPN, when setting up Tailscale SSH to WSL instances, or when bootstrapping remote dev environments over Tailscale", + "description": "Diagnoses and fixes conflicts between Tailscale and proxy/VPN tools (Shadowrocket, Clash, Surge) on macOS. Covers six conflict layers: route hijacking, HTTP proxy env var interception, system proxy bypass, SSH ProxyCommand double tunneling, VM/container proxy propagation, and stalled DNS resolver in macOS getaddrinfo chain (dead VPN daemon leaves zombie utun + DNS injection). Includes an automated quick-diagnose script plus SOP for remote development via SSH tunnels with proxy-safe Makefile patterns. Use when Tailscale ping works but SSH/HTTP times out, when browser returns 503 but curl works, when local vanity domains fail behind proxy, when git push fails with failed to begin relaying via HTTP, when Docker pull times out behind TUN/VPN, when setting up Tailscale SSH to WSL instances, when bootstrapping remote dev environments over Tailscale, or when ssh/curl/git hang ~60s before resolving a hostname while nslookup returns instantly", "source": "./", "strict": false, - "version": "1.4.0", + "version": "1.5.0", "category": "developer-tools", "keywords": [ "tailscale", diff --git a/CHANGELOG.md b/CHANGELOG.md index ac9556a2..c66c6a8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.53.2] - 2026-05-10 + +### Fixed +- Remove `skills: ["./"]` from 13 plugin entries that triggered Claude Code 2.1.x path-escape validator error (`skills path "./" escapes plugin root`). Affected plugins: claude-code-history-files-finder, claude-export-txt-better, claude-md-progressive-disclosurer, claude-skills-troubleshooting, continue-claude-work, doc-to-markdown, docs-cleaner, marketplace-dev, meeting-minutes-taker, mermaid-tools, pdf-creator, ppt-creator, statusline-generator. These are all suite member plugins whose `source` already points to the correct skill directory — the explicit `skills` field was redundant and is now omitted. Fixes [#64](https://github.com/daymade/claude-code-skills/issues/64). + +## [1.52.0] - 2026-04-30 + +### Added +- **stepfun-asr** v1.0.0: Transcribe audio with StepFun's `stepaudio-2.5-asr` — an SSE endpoint (NOT `/v1/audio/transcriptions`) with 32K context, ~85-101× RTF on long audio, and a single-call ceiling around 30 minutes (no client-side chunking). Split out from `stepfun-tts` so the ASR-specific traps (wrong-endpoint misleading error, Plan vs Normal key silent failure, SSE `error` event handling, repetition-hallucination edge case) live next to the `asr_transcribe.py` script that handles them. Bundled `scripts/asr_transcribe.py` (pure-stdlib CLI: env → `${CLAUDE_PLUGIN_DATA}/config.json` key resolution, base64 + nested JSON body, SSE parsing, censorship + transport error distinction). References cover the full SSE event contract, the legacy-vs-2.5 endpoint comparison table, and the "Plan key cannot call audio" gotcha. Suggests `transcript-fixer` / `meeting-minutes-taker` as natural downstream skills. + +### Changed +- **stepfun-tts** v1.0.0 → v2.0.0 (BREAKING): ASR functionality removed and split into the new `stepfun-asr` skill. The remaining skill focuses purely on Contextual TTS (`stepaudio-2.5-tts`) — `instruction` natural-language tone + inline `()` parentheses + the `voice_label` migration story from `step-tts-2`. SKILL.md, `references/api_reference.md`, and `references/known_issues.md` all stripped of ASR sections; description and keywords updated to TTS-only. `scripts/asr_transcribe.py` removed from this skill (now lives in `stepfun-asr`). +- Marketplace skill count: 51 → 52 (effective listed count; suite member skills not double-counted) +- Marketplace plugin entry count: 55 → 56 +- Marketplace version: 1.51.0 → 1.52.0 +- README.md, README.zh-CN.md: badges, descriptions, skill section #50 (stepfun-tts retitled "TTS only" + description rewritten), new skill section #52 (stepfun-asr), Use Cases entries (split into two), Documentation Quick Links, Requirements (StepFun key applies to both) +- CLAUDE.md: overview count, marketplace plugin count, Available Skills list (entry #50 description rewritten + new entry #52) + +### Note +This release also reconciles a versioning drift: commits `b2003d6` (statusline-generator → v1.1.0) and `ec7c313` (pdf-creator → v1.4.0) bumped their respective `plugins[].version` fields without bumping `metadata.version` and without adding CHANGELOG entries — a violation of the "any commit modifying a skill must bump that skill's version AND the marketplace metadata version" rule from `CLAUDE.md`. Those commits remain in history; v1.52.0 picks up the marketplace catalog version where it should have been after both, then adds the stepfun split on top. CHANGELOG entries for those individual skill bumps will not be retroactively backfilled — the version numbers in `marketplace.json` are authoritative and discoverable via `git log -- `. + ## [1.51.0] - 2026-04-26 ### Added From 41d979a5c0028a3ce2f61ebbcbdd3e2b0f7c8902 Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 10 May 2026 02:12:31 +0800 Subject: [PATCH 132/137] refactor(marketplace): align plugin source/skills with official pattern MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adopt the pattern used by 167/168 plugins in anthropics/claude-plugins-official: - source points directly to the skill directory (not repo root) - skills field omitted (auto-discovery from source directory) Before: source="./" skills=["./tunnel-doctor"] After: source="./tunnel-doctor" (no skills field) 39 root-level single-skill plugins migrated. The 3 suite plugins (daymade-claude-code, daymade-docs, daymade-skill) retain explicit skills arrays for multi-skill routing — matching the netsuite-suitecloud pattern in the official marketplace. Co-Authored-By: Claude Opus 4.6 (1M context) --- .claude-plugin/marketplace.json | 195 +++++++------------------------- CHANGELOG.md | 5 +- 2 files changed, 43 insertions(+), 157 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 498613cb..525dd32d 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -12,7 +12,7 @@ { "name": "asr-transcribe-to-text", "description": "Transcribe audio and video files to text using a remote ASR service (Qwen3-ASR or OpenAI-compatible endpoint). Extracts audio from video, sends to configurable ASR endpoint, outputs clean text. Use when the user wants to transcribe recordings, convert audio/video to text, do speech-to-text, or mentions ASR, Qwen ASR, 转录, 语音转文字, 录音转文字, or has a meeting recording, lecture, interview, or screen recording to transcribe.", - "source": "./", + "source": "./asr-transcribe-to-text", "strict": false, "version": "1.0.0", "category": "productivity", @@ -25,15 +25,12 @@ "video", "vllm", "ffmpeg" - ], - "skills": [ - "./asr-transcribe-to-text" ] }, { "name": "capture-screen", "description": "Programmatic screenshot capture on macOS. Get window IDs via Swift CGWindowListCopyWindowInfo, capture specific windows with screencapture -l, and control application windows via AppleScript. Supports multi-shot workflows for capturing different sections of the same window. Use when taking automated screenshots, capturing application windows, or creating visual documentation", - "source": "./", + "source": "./capture-screen", "strict": false, "version": "1.0.1", "category": "utilities", @@ -46,9 +43,6 @@ "applescript", "automation", "visual-documentation" - ], - "skills": [ - "./capture-screen" ] }, { @@ -122,7 +116,7 @@ { "name": "cli-demo-generator", "description": "Generate professional animated CLI demos and terminal recordings with VHS. Supports automated generation, batch processing, and interactive recording for documentation and tutorials", - "source": "./", + "source": "./cli-demo-generator", "strict": false, "version": "1.0.0", "category": "developer-tools", @@ -135,15 +129,12 @@ "recording", "animation", "documentation" - ], - "skills": [ - "./cli-demo-generator" ] }, { "name": "cloudflare-troubleshooting", "description": "Investigate and resolve Cloudflare configuration issues using API-driven evidence gathering. Use when troubleshooting ERR_TOO_MANY_REDIRECTS, SSL errors, DNS issues, or any Cloudflare-related problems", - "source": "./", + "source": "./cloudflare-troubleshooting", "strict": false, "version": "1.0.0", "category": "developer-tools", @@ -155,15 +146,12 @@ "api", "debugging", "devops" - ], - "skills": [ - "./cloudflare-troubleshooting" ] }, { "name": "competitors-analysis", "description": "Analyze competitor repositories with evidence-based approach. Use when tracking competitors, creating competitor profiles, or generating competitive analysis. All analysis must be based on actual cloned code, never assumptions. Triggers include analyze competitor, add competitor, competitive analysis, or 竞品分析", - "source": "./", + "source": "./competitors-analysis", "strict": false, "version": "1.0.1", "category": "productivity", @@ -175,9 +163,6 @@ "market-research", "竞品分析", "code-analysis" - ], - "skills": [ - "./competitors-analysis" ] }, { @@ -274,7 +259,7 @@ { "name": "deep-research", "description": "Generate format-controlled research reports with evidence tracking, source governance, and multi-pass synthesis. V6.1 adds: source accessibility (circular verification forbidden, exclusive advantage encouraged). Enterprise Research Mode: six-dimension data collection, SWOT/barrier/risk frameworks, and three-level quality control for company research", - "source": "./", + "source": "./deep-research", "strict": false, "version": "2.4.0", "category": "documentation", @@ -290,9 +275,6 @@ "enterprise", "company-research", "due-diligence" - ], - "skills": [ - "./deep-research" ] }, { @@ -333,7 +315,7 @@ { "name": "douban-skill", "description": "Export and sync Douban (豆瓣) book/movie/music/game collections to local CSV files via Frodo API. Supports full export (all history) and RSS incremental sync (recent items). Use when the user wants to export Douban reading/watching/listening/gaming history, back up their Douban data, set up incremental sync, or mentions 豆瓣/douban collections. Triggers on: 豆瓣, douban, 读书记录, 观影记录, 书影音, 导出豆瓣, export, backup, sync, collection.", - "source": "./", + "source": "./douban-skill", "strict": false, "version": "1.0.0", "category": "productivity", @@ -345,15 +327,12 @@ "backup", "rss", "frodo" - ], - "skills": [ - "./douban-skill" ] }, { "name": "excel-automation", "description": "Create, parse, and control Excel files on macOS. Professional formatting with openpyxl (font colors, fills, borders, conditional formatting), complex xlsm parsing with stdlib zipfile+xml for investment bank financial models, and Excel window control via AppleScript (zoom, scroll, select). Use when creating formatted Excel reports, parsing financial models, or automating Excel on macOS", - "source": "./", + "source": "./excel-automation", "strict": false, "version": "1.0.0", "category": "productivity", @@ -368,15 +347,12 @@ "macos", "dcf", "investment-banking" - ], - "skills": [ - "./excel-automation" ] }, { "name": "fact-checker", "description": "Verifies factual claims in documents using web search and official sources, then proposes corrections with user confirmation. Use when the user asks to fact-check, verify information, validate claims, check accuracy, or update outdated information in documents. Supports AI model specs, technical documentation, statistics, and general factual statements", - "source": "./", + "source": "./fact-checker", "strict": false, "version": "1.0.0", "category": "productivity", @@ -388,15 +364,12 @@ "validation", "corrections", "web-search" - ], - "skills": [ - "./fact-checker" ] }, { "name": "feishu-doc-scraper", "description": "Save Feishu Docs and Feishu Wiki pages as clean Markdown from a live authenticated browser session. Uses TOC-driven section capture and coverage verification, and explicitly avoids relying on Web Clipper for virtual-scroll pages. Use when users ask to save, export, scrape, or archive a Feishu doc/wiki to Markdown, or when a Feishu page already open in Chrome needs to be turned into a local note.", - "source": "./", + "source": "./feishu-doc-scraper", "strict": false, "version": "1.0.0", "category": "productivity", @@ -409,15 +382,12 @@ "scraping", "browser", "archival" - ], - "skills": [ - "./feishu-doc-scraper" ] }, { "name": "financial-data-collector", "description": "Collect real financial data for any US publicly traded company from free public sources (yfinance). Output structured JSON with market data, historical financials, WACC inputs, and analyst estimates. Handles NaN year detection, CapEx sign preservation, and FCF definition mismatches. Use when users request company financials, stock data, DCF inputs, or financial data collection for any US equity ticker", - "source": "./", + "source": "./financial-data-collector", "strict": false, "version": "1.0.0", "category": "productivity", @@ -431,15 +401,12 @@ "market-data", "investment-research", "sec-filings" - ], - "skills": [ - "./financial-data-collector" ] }, { "name": "gangtise-copilot", "description": "One-stop installer and companion for the full Gangtise (岗底斯投研) OpenAPI skill suite — 19 official skills covering data retrieval (OHLC 行情, 财务, 估值, 研报, 首席观点, 会议纪要, 调研纪要), research workflows (个股研究 L1-L4, 观点 PK 对抗性分析, 主题研究, 事件复盘), and utility (股票池管理, 公开网页搜索). Zero-config install to Claude Code / OpenClaw / Codex with 4 preset modes (full / workshop / minimal / custom), guides accessKey + secretAccessKey setup with a live validation call against open.gangtise.com, and ships a read-only diagnostic script. Use this skill whenever the user mentions Gangtise, 岗底斯, gangtise-data-client, gangtise-kb-client, gangtise-file-client, gangtise-stock-research, gangtise-opinion-pk, installing any gangtise-* skill, configuring its credentials, or reports errors like 'token is invalid', '接口地址错误', 'the uri can't be accessed'. This is a wrapper around Gangtise's official skills — it installs and orchestrates them rather than replacing them.", - "source": "./", + "source": "./gangtise-copilot", "strict": false, "version": "1.1.0", "category": "developer-tools", @@ -455,15 +422,12 @@ "claude-code", "openclaw", "codex" - ], - "skills": [ - "./gangtise-copilot" ] }, { "name": "github-contributor", "description": "Strategic guide for becoming an effective GitHub contributor. Covers opportunity discovery, project selection, high-quality PR creation, and reputation building. Use when looking to contribute to open-source projects, building GitHub presence, or learning contribution best practices", - "source": "./", + "source": "./github-contributor", "strict": false, "version": "1.0.3", "category": "developer-tools", @@ -475,15 +439,12 @@ "reputation", "contributor", "oss" - ], - "skills": [ - "./github-contributor" ] }, { "name": "github-ops", "description": "Comprehensive GitHub operations using gh CLI and GitHub API for pull requests, issues, repositories, workflows, and API interactions", - "source": "./", + "source": "./github-ops", "strict": false, "version": "1.0.0", "category": "developer-tools", @@ -494,15 +455,12 @@ "issues", "workflows", "api" - ], - "skills": [ - "./github-ops" ] }, { "name": "i18n-expert", "description": "Complete internationalization/localization setup and auditing for UI codebases. Configure i18n frameworks, replace hard-coded strings with translation keys, ensure locale parity between en-US and zh-CN, and validate pluralization and formatting. Use when setting up i18n for React/Next.js/Vue apps, auditing existing implementations, replacing hard-coded strings, ensuring proper error code mapping, or validating pluralization and date/time/number formatting across locales", - "source": "./", + "source": "./i18n-expert", "strict": false, "version": "1.0.0", "category": "developer-tools", @@ -517,15 +475,12 @@ "locale", "multilingual", "globalization" - ], - "skills": [ - "./i18n-expert" ] }, { "name": "iOS-APP-developer", "description": "Develops iOS applications with XcodeGen, SwiftUI, and SPM. Use when configuring XcodeGen project.yml, resolving SPM dependency issues, deploying to devices, handling code signing, debugging camera/AVFoundation, iOS version compatibility issues, or fixing Library not loaded @rpath framework errors. Includes state machine testing patterns for @MainActor classes", - "source": "./", + "source": "./iOS-APP-developer", "strict": false, "version": "1.1.0", "category": "developer-tools", @@ -541,15 +496,12 @@ "swift", "xcode", "testing" - ], - "skills": [ - "./iOS-APP-developer" ] }, { "name": "ima-copilot", "description": "One-stop companion and installer for the official Tencent IMA skill (ima.qq.com). Installs upstream ima-skill to Claude Code/Codex/OpenClaw via npx skills add, guides API key setup, detects and fixes known issues (including the missing-YAML-frontmatter bug in submodule SKILL.md files) with user consent, and implements a personalized fan-out search strategy with priority-based knowledge base boosting and silent truncation detection", - "source": "./", + "source": "./ima-copilot", "strict": false, "version": "1.0.1", "category": "developer-tools", @@ -565,15 +517,12 @@ "claude-code", "codex", "openclaw" - ], - "skills": [ - "./ima-copilot" ] }, { "name": "llm-icon-finder", "description": "Find and access AI/LLM model brand icons from lobe-icons library in SVG/PNG/WEBP formats", - "source": "./", + "source": "./llm-icon-finder", "strict": false, "version": "1.0.0", "category": "assets", @@ -583,15 +532,12 @@ "llm", "branding", "lobe-icons" - ], - "skills": [ - "./llm-icon-finder" ] }, { "name": "macos-cleaner", "description": "Intelligent macOS disk space analysis and cleanup with safety-first philosophy. Use when users report disk space issues, need to clean their Mac, or want to understand storage consumption. Analyzes system caches, application remnants, large files, and development environments (Docker, Homebrew, npm, pip) with risk categorization (Safe/Caution/Keep) and requires explicit user confirmation before any deletions. Includes Mole visual tool integration for hybrid workflow", - "source": "./", + "source": "./macos-cleaner", "strict": false, "version": "1.1.0", "category": "utilities", @@ -606,9 +552,6 @@ "homebrew", "system-maintenance", "safety" - ], - "skills": [ - "./macos-cleaner" ] }, { @@ -722,7 +665,7 @@ { "name": "product-analysis", "description": "Multi-path parallel product analysis with cross-model test-time compute scaling. Spawns parallel agents (Claude Code + Codex CLI) for multi-perspective exploration, then synthesizes findings into actionable optimization plans. Supports self-audit, UX audit, API audit, architecture review, and competitive benchmarking via competitors-analysis skill", - "source": "./", + "source": "./product-analysis", "strict": false, "version": "1.0.1", "category": "productivity", @@ -737,15 +680,12 @@ "synthesis", "product-audit", "information-architecture" - ], - "skills": [ - "./product-analysis" ] }, { "name": "prompt-optimizer", "description": "Transform vague prompts into precise, well-structured specifications using EARS (Easy Approach to Requirements Syntax) methodology. Use when users provide loose requirements, ambiguous feature descriptions, need to enhance prompts for AI-generated code/products/documents, request prompt optimization, or want to improve requirements engineering. Applies domain theories (GTD, BJ Fogg, Gestalt, AIDA, Zero Trust) and structured Role/Skills/Workflows/Examples/Formats framework", - "source": "./", + "source": "./prompt-optimizer", "strict": false, "version": "1.1.0", "category": "productivity", @@ -758,15 +698,12 @@ "domain-theory", "prompt-enhancement", "ai-prompting" - ], - "skills": [ - "./prompt-optimizer" ] }, { "name": "promptfoo-evaluation", "description": "Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like promptfoo, eval, LLM evaluation, prompt testing, or model comparison", - "source": "./", + "source": "./promptfoo-evaluation", "strict": false, "version": "1.1.0", "category": "developer-tools", @@ -779,15 +716,12 @@ "llm-rubric", "few-shot", "model-comparison" - ], - "skills": [ - "./promptfoo-evaluation" ] }, { "name": "qa-expert", "description": "Comprehensive QA testing infrastructure with autonomous LLM execution, Google Testing Standards (AAA pattern), and OWASP security testing. Use when establishing QA processes, writing test cases, executing test plans, tracking bugs with P0-P4 classification, calculating quality metrics, enforcing quality gates, or preparing third-party QA handoffs. Enables 100x faster test execution via master prompts", - "source": "./", + "source": "./qa-expert", "strict": false, "version": "1.0.0", "category": "developer-tools", @@ -802,15 +736,12 @@ "automation", "quality-gates", "metrics" - ], - "skills": [ - "./qa-expert" ] }, { "name": "repomix-safe-mixer", "description": "Safely package codebases with repomix by automatically detecting and removing hardcoded credentials before packing. Use when packaging code for distribution, creating reference packages, or when the user mentions security concerns about sharing code with repomix", - "source": "./", + "source": "./repomix-safe-mixer", "strict": false, "version": "1.0.0", "category": "security", @@ -822,15 +753,12 @@ "safe-packaging", "secret-detection", "code-security" - ], - "skills": [ - "./repomix-safe-mixer" ] }, { "name": "repomix-unmixer", "description": "Extract files from repomix packaged formats (XML, Markdown, JSON) with automatic format detection and validation", - "source": "./", + "source": "./repomix-unmixer", "strict": false, "version": "1.0.0", "category": "utilities", @@ -840,15 +768,12 @@ "extract", "xml", "conversion" - ], - "skills": [ - "./repomix-unmixer" ] }, { "name": "scrapling-skill", "description": "Install, troubleshoot, and use Scrapling CLI for extracting HTML, Markdown, or text from webpages. Diagnoses missing extras, Playwright browser runtime issues, TLS verification failures, and WeChat public article extraction patterns. Use when users mention Scrapling, `scrapling extract`, `uv tool install scrapling`, or need to decide between static and browser-backed fetching", - "source": "./", + "source": "./scrapling-skill", "strict": false, "version": "1.0.0", "category": "developer-tools", @@ -861,15 +786,12 @@ "wechat", "extraction", "cli" - ], - "skills": [ - "./scrapling-skill" ] }, { "name": "slides-creator", "description": "Narrative-first slide deck creation. Guides users through structured narrative design (ABCDEFG model), then delegates visual generation to baoyu-slide-deck. Triggers on create slides, make a presentation, generate deck, slide deck, PPT, or when user needs to turn content into visual slides.", - "source": "./", + "source": "./slides-creator", "strict": false, "version": "1.0.0", "category": "content-creation", @@ -881,9 +803,6 @@ "narrative", "storytelling", "ABCDEFG" - ], - "skills": [ - "./slides-creator" ] }, { @@ -905,7 +824,7 @@ { "name": "stepfun-tts", "description": "Generate Chinese / Japanese speech with StepFun's stepaudio-2.5-tts — Contextual TTS that replaces step-tts-2's `voice_label` with natural-language `instruction` (≤200 chars) plus inline `()` parentheses for句内 prosody. Use when the user wants emotional / prosody control over voice synthesis (whisper, pause, stress, mood pivot mid-sentence), batch-generates game / app voice lines, migrates from `step-tts-2` (the `voice_label → instruction` breaking change), or hits StepFun's stricter 2.5-era censorship (死/消失/political terms). Triggers on 阶跃 TTS, StepAudio 合成, 语音合成, 配音, 文本转语音, TTS 升级, 迁移 step-tts-2. For transcription with the sibling stepaudio-2.5-asr model, use the stepfun-asr skill instead.", - "source": "./", + "source": "./stepfun-tts", "strict": false, "version": "2.0.0", "category": "productivity", @@ -918,15 +837,12 @@ "chinese-tts", "voice-synthesis", "contextual-tts" - ], - "skills": [ - "./stepfun-tts" ] }, { "name": "stepfun-asr", "description": "Transcribe audio with StepFun's stepaudio-2.5-asr — an SSE endpoint (NOT /v1/audio/transcriptions) with 32K context, ~85-101x RTF on long audio, and a single-call ceiling around 30 minutes (no client-side chunking). Use when transcribing Chinese / English audio with StepFun, when long-form recordings (5-30 min) need to land in one request, when migrating from step-asr / step-asr-1.1, or when hitting the misleading `model stepaudio-2.5-asr not supported` error (which actually means wrong endpoint). Triggers on 阶跃 ASR, StepFun ASR, stepaudio-2.5-asr, 转录, 语音识别, 长音频转写, 语音转文字. For TTS with the sibling stepaudio-2.5-tts model, use the stepfun-tts skill instead.", - "source": "./", + "source": "./stepfun-asr", "strict": false, "version": "1.0.0", "category": "productivity", @@ -941,15 +857,12 @@ "transcription", "long-audio", "sse" - ], - "skills": [ - "./stepfun-asr" ] }, { "name": "teams-channel-post-writer", "description": "Create professional Microsoft Teams channel posts with Adaptive Cards, formatted announcements, and corporate communication standards", - "source": "./", + "source": "./teams-channel-post-writer", "strict": false, "version": "1.0.0", "category": "communication", @@ -959,15 +872,12 @@ "adaptive-cards", "communication", "announcements" - ], - "skills": [ - "./teams-channel-post-writer" ] }, { "name": "terraform-skill", "description": "Operational traps for Terraform provisioners, multi-environment isolation, and zero-to-deployment reliability. Covers provisioner timing races, SSH connection conflicts, DNS record duplication, volume permissions, database bootstrap gaps, snapshot cross-contamination, Cloudflare credential format errors, hardcoded domains in Caddyfiles/compose, and init-data-only-on-first-boot pitfalls. Activate when writing null_resource provisioners, creating multi-environment Terraform setups, debugging containers that are Restarting/unhealthy after terraform apply, setting up fresh instances with cloud-init, or any IaC code that SSHs into remote hosts. Also activate when the user mentions terraform plan/apply errors, provisioner failures, infrastructure drift, TLS certificate errors, or Caddy/gateway configuration.", - "source": "./", + "source": "./terraform-skill", "strict": false, "version": "1.0.0", "category": "developer-tools", @@ -979,15 +889,12 @@ "infrastructure", "cloud-init", "cloudflare" - ], - "skills": [ - "./terraform-skill" ] }, { "name": "transcript-fixer", "description": "Corrects speech-to-text (ASR/STT) transcription errors using dictionary rules and native Claude AI corrections (no API key needed by default). Supports intelligent paragraph breaks, filler word reduction, interactive review, Chinese domain names, and iterative dictionary building. Use when users mention transcript correction, ASR errors, speech-to-text mistakes, homophone errors, or working with transcription files", - "source": "./", + "source": "./transcript-fixer", "strict": false, "version": "1.4.0", "category": "productivity", @@ -1000,15 +907,12 @@ "ai", "meeting-notes", "nlp" - ], - "skills": [ - "./transcript-fixer" ] }, { "name": "tunnel-doctor", "description": "Diagnoses and fixes conflicts between Tailscale and proxy/VPN tools (Shadowrocket, Clash, Surge) on macOS. Covers six conflict layers: route hijacking, HTTP proxy env var interception, system proxy bypass, SSH ProxyCommand double tunneling, VM/container proxy propagation, and stalled DNS resolver in macOS getaddrinfo chain (dead VPN daemon leaves zombie utun + DNS injection). Includes an automated quick-diagnose script plus SOP for remote development via SSH tunnels with proxy-safe Makefile patterns. Use when Tailscale ping works but SSH/HTTP times out, when browser returns 503 but curl works, when local vanity domains fail behind proxy, when git push fails with failed to begin relaying via HTTP, when Docker pull times out behind TUN/VPN, when setting up Tailscale SSH to WSL instances, when bootstrapping remote dev environments over Tailscale, or when ssh/curl/git hang ~60s before resolving a hostname while nslookup returns instantly", - "source": "./", + "source": "./tunnel-doctor", "strict": false, "version": "1.5.0", "category": "developer-tools", @@ -1031,15 +935,12 @@ "makefile", "remote-development", "local-domain" - ], - "skills": [ - "./tunnel-doctor" ] }, { "name": "twitter-reader", "description": "Fetch Twitter/X post content including long-form Articles with full images and metadata. Use when Claude needs to retrieve tweet/article content, author info, engagement metrics (likes, retweets, bookmarks), and embedded media. Supports individual posts and X Articles (long-form content). Automatically downloads all images to local attachments folder and generates complete Markdown with proper image references. Preferred over Jina for X Articles with images.", - "source": "./", + "source": "./twitter-reader", "strict": false, "version": "1.1.0", "category": "utilities", @@ -1055,15 +956,12 @@ "images", "attachments", "markdown" - ], - "skills": [ - "./twitter-reader" ] }, { "name": "ui-designer", "description": "Extract design systems from reference UI images and generate implementation-ready UI design prompts. Use when users provide UI screenshots/mockups and want to create consistent designs", - "source": "./", + "source": "./ui-designer", "strict": false, "version": "1.0.0", "category": "design", @@ -1074,15 +972,12 @@ "screenshot", "design-extraction", "mvp" - ], - "skills": [ - "./ui-designer" ] }, { "name": "video-comparer", "description": "Compare two videos and generate interactive HTML reports with quality metrics (PSNR, SSIM) and frame-by-frame visual comparisons. Use when analyzing compression results, evaluating codec performance, or assessing video quality differences", - "source": "./", + "source": "./video-comparer", "strict": false, "version": "1.0.0", "category": "media", @@ -1095,15 +990,12 @@ "compression", "ffmpeg", "codec" - ], - "skills": [ - "./video-comparer" ] }, { "name": "windows-remote-desktop-connection-doctor", "description": "Diagnose Windows App (Microsoft Remote Desktop / Azure Virtual Desktop / W365) connection quality issues on macOS. Analyze transport protocol selection (UDP Shortpath vs WebSocket), detect VPN/proxy interference with STUN/TURN negotiation, and parse Windows App logs for Shortpath failures. This skill should be used when VDI connections are slow, when transport shows WebSocket instead of UDP, when RDP Shortpath fails to establish, or when RTT is unexpectedly high.", - "source": "./", + "source": "./windows-remote-desktop-connection-doctor", "strict": false, "version": "1.0.0", "category": "developer-tools", @@ -1122,15 +1014,12 @@ "vpn", "macos", "networking" - ], - "skills": [ - "./windows-remote-desktop-connection-doctor" ] }, { "name": "youtube-downloader", "description": "Download YouTube videos and HLS streams (m3u8) from platforms like Mux, Vimeo, etc. using yt-dlp and ffmpeg. Use when users request downloading videos, extracting audio, handling protected streams with authentication headers, or troubleshooting download issues like nsig extraction failures, 403 errors, or cookie extraction problems", - "source": "./", + "source": "./youtube-downloader", "strict": false, "version": "1.1.0", "category": "utilities", @@ -1147,15 +1036,12 @@ "streaming", "mux", "vimeo" - ], - "skills": [ - "./youtube-downloader" ] }, { "name": "debugging-network-issues", "description": "Evidence-driven investigation for network, streaming, and protocol-layer bugs. Use when debugging connection resets, SSE or long-polling stalls, fixed-time connection drops, CDN/proxy/CGNAT idle timeouts, or any incident where symptoms do not match the obvious cause. Applies falsification-first methodology with layered isolation experiments, env-gated runtime instrumentation, and counter-review agent teams.", - "source": "./", + "source": "./debugging-network-issues", "strict": false, "version": "1.0.0", "category": "developer-tools", @@ -1170,9 +1056,6 @@ "streaming", "troubleshooting", "methodology" - ], - "skills": [ - "./debugging-network-issues" ] } ] diff --git a/CHANGELOG.md b/CHANGELOG.md index c66c6a8d..f9d61129 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [1.53.2] - 2026-05-10 ### Fixed -- Remove `skills: ["./"]` from 13 plugin entries that triggered Claude Code 2.1.x path-escape validator error (`skills path "./" escapes plugin root`). Affected plugins: claude-code-history-files-finder, claude-export-txt-better, claude-md-progressive-disclosurer, claude-skills-troubleshooting, continue-claude-work, doc-to-markdown, docs-cleaner, marketplace-dev, meeting-minutes-taker, mermaid-tools, pdf-creator, ppt-creator, statusline-generator. These are all suite member plugins whose `source` already points to the correct skill directory — the explicit `skills` field was redundant and is now omitted. Fixes [#64](https://github.com/daymade/claude-code-skills/issues/64). +- Remove `skills: ["./"]` from 13 suite member plugin entries that triggered Claude Code 2.1.x path-escape validator error (`skills path "./" escapes plugin root`). Fixes [#64](https://github.com/daymade/claude-code-skills/issues/64). + +### Changed +- Align all 52 single-skill plugins with official Anthropic marketplace pattern: `source` points directly to the skill directory (e.g., `"./tunnel-doctor"`), `skills` field omitted (auto-discovery). Previously used `source: "./"` with `skills: ["./skill-name"]`. The 3 suite plugins (`daymade-claude-code`, `daymade-docs`, `daymade-skill`) retain explicit `skills` arrays for multi-skill routing. Matches the pattern used by 167 of 168 plugins in `anthropics/claude-plugins-official`. ## [1.52.0] - 2026-04-30 From b51ce8ca8b70e52d8ce16dcdd7563823da664a64 Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 10 May 2026 02:18:59 +0800 Subject: [PATCH 133/137] docs: sync all documentation with official source/skills pattern MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update 11 files to reflect the new plugin configuration pattern: - source points directly to skill directory (not repo root) - skills field omitted for single-skill plugins (auto-discovery) - skills field retained only for suite plugins (multi-skill routing) Files updated: - CLAUDE.md: marketplace config description, plugin count 57→55 - references/new-skill-guide.md: plugin entry template - references/plugin-architecture.md: config examples, install flow - references/plugin-troubleshooting.md: naming mismatch examples - marketplace-dev/SKILL.md: source/skills guidance, checklist, template - marketplace-dev/references/: cache patterns, anti-patterns, schema - skill-creator/SKILL.md: marketplace registration template - claude-skills-troubleshooting/references/architecture.md: schema example - marketplace-dev/hooks/post_edit_sync_check.sh: match skills via source path when skills array is absent Co-Authored-By: Claude Opus 4.6 (1M context) --- CLAUDE.md | 16 ++++---- .../references/architecture.md | 4 +- daymade-claude-code/marketplace-dev/SKILL.md | 23 +++++------ .../hooks/post_edit_sync_check.sh | 20 +++++----- .../references/anti_patterns.md | 6 +-- .../references/cache_and_source_patterns.md | 38 +++++++++++++------ .../references/marketplace_schema.md | 2 +- daymade-skill/skill-creator/SKILL.md | 5 +-- references/new-skill-guide.md | 5 +-- references/plugin-architecture.md | 21 ++++++++-- references/plugin-troubleshooting.md | 8 ++-- 11 files changed, 90 insertions(+), 58 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fc8e989c..3d6fd23f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Repository Overview -This is a Claude Code skills marketplace containing 51 production-ready skills organized in a plugin marketplace structure. Most plugins expose one skill for narrow installs; suite plugins expose related skills under shared namespaces for combined installation workflows. +This is a Claude Code skills marketplace containing 53 production-ready skills organized in a plugin marketplace structure. Most plugins expose one skill for narrow installs; suite plugins expose related skills under shared namespaces for combined installation workflows. **Essential Skill**: `skill-creator` is the most important skill in this marketplace - it's a meta-skill that enables users to create their own skills. Always recommend it first for users interested in extending Claude Code. @@ -152,10 +152,10 @@ If it fires, fix the issue — do NOT use `--no-verify` to bypass. ## Marketplace Configuration The marketplace is configured in `.claude-plugin/marketplace.json`: -- Contains 55 plugin entries: most map to one skill, while suite plugins (`daymade-docs`, `daymade-claude-code`) map to multiple related skills -- Each plugin has: name, description, version, category, keywords, skills array -- Marketplace metadata: name, owner, version, homepage -- Suite plugins use `/` (a top-level directory at the repo root) as the canonical source for their member skills so suite caches contain only those skills. Single-skill plugin entries for suite members should point to the same canonical subdirectories. +- Contains 55 plugin entries: single-skill plugins point `source` directly at the skill directory (no `skills` field); suite plugins (`daymade-docs`, `daymade-claude-code`, `daymade-skill`) use explicit `skills` arrays for multi-skill routing +- Each plugin has: name, description, source, version, category, keywords +- Marketplace metadata: name, owner, version +- Single-skill plugins follow the official pattern (167/168 plugins in `anthropics/claude-plugins-official`): `source` points to skill directory, `skills` omitted ### Versioning Architecture @@ -163,7 +163,7 @@ The marketplace is configured in `.claude-plugin/marketplace.json`: 1. **Marketplace Version** (`.claude-plugin/marketplace.json` → `metadata.version`) - Tracks the marketplace catalog as a whole - - Current: v1.45.1 + - Current: v1.53.0 - Bump when: Adding/removing skills, adding/removing suite plugins, major marketplace restructuring - Semantic versioning: MAJOR.MINOR.PATCH @@ -246,7 +246,9 @@ This applies when you change ANY file under a skill directory: 48. **terraform-skill** - Operational traps for Terraform provisioners, multi-environment isolation, and zero-to-deployment reliability; covers provisioner timing races, SSH connection conflicts, DNS record duplication, volume permissions, database bootstrap gaps, Cloudflare credential errors, and init-data-only-on-first-boot pitfalls 49. **slides-creator** - Narrative-first slide deck creation guiding users through structured narrative design (ABCDEFG model), then delegating visual generation to baoyu-slide-deck. Triggers on create slides, make a presentation, generate deck, slide deck, PPT, or when user needs to turn content into visual slides 50. **debugging-network-issues** - Evidence-driven, falsification-first methodology for network/streaming/protocol-layer bugs (HTTP/2 RST_STREAM, SSE stalls, fixed-time drops, CDN/proxy/CGNAT idle timeouts). Layered isolation experiments + counter-review filter, with bundled probe scripts and a real SSE 130s case study -51. **stepfun-tts** - StepFun StepAudio 2.5 family: stepaudio-2.5-tts (Contextual TTS via instruction + inline parentheses) and stepaudio-2.5-asr (SSE endpoint, 32K context, ~100x RTF, 30-min single-call). Captures the three breaking changes from step-tts-2 (voice_label removal, /v1/audio/asr/sse endpoint, stricter censorship) with migration playbook +51. **stepfun-tts** - StepFun stepaudio-2.5-tts (Contextual TTS): natural-language `instruction` (≤200 chars) + inline `()` parentheses for句内 prosody. Captures the two TTS-side breaking changes from step-tts-2 (voice_label removal + stricter 2.5-era censorship) with migration playbook +52. **stepfun-asr** - StepFun stepaudio-2.5-asr (SSE endpoint, 32K context, ~85-101× RTF, 30-min single-call). Hides the #1 trap of the 2.5 ASR family: it does NOT live on `/v1/audio/transcriptions` — the wrong endpoint returns a misleading `model not supported` error. Bundled stdlib CLI handles base64 + nested JSON body + SSE parsing including `error` events +53. **feishu-doc-scraper** - Save Feishu Docs and Feishu Wiki pages as clean Markdown from a live authenticated browser session with TOC-driven section capture, UI-noise removal, and explicit rejection of Web Clipper as the primary path on virtual-scroll pages **Recommendation**: Always suggest `skill-creator` first for users interested in creating skills or extending Claude Code. diff --git a/daymade-claude-code/claude-skills-troubleshooting/references/architecture.md b/daymade-claude-code/claude-skills-troubleshooting/references/architecture.md index 4c2e1316..0911017b 100644 --- a/daymade-claude-code/claude-skills-troubleshooting/references/architecture.md +++ b/daymade-claude-code/claude-skills-troubleshooting/references/architecture.md @@ -145,8 +145,8 @@ { "name": "plugin-name", "description": "...", - "version": "1.0.0", - "skills": ["./skill-directory"] + "source": "./skill-directory", + "version": "1.0.0" } ] } diff --git a/daymade-claude-code/marketplace-dev/SKILL.md b/daymade-claude-code/marketplace-dev/SKILL.md index 07221797..38ff2c9f 100644 --- a/daymade-claude-code/marketplace-dev/SKILL.md +++ b/daymade-claude-code/marketplace-dev/SKILL.md @@ -115,11 +115,13 @@ Key rules that are NOT obvious from the docs: 5. **`strict: false`** is required when there's no `plugin.json` in the repo. With `strict: false`, the marketplace entry IS the entire plugin definition. Having BOTH `strict: false` AND a `plugin.json` with components causes a load failure. -6. **`source` defines the installed plugin root**. All `skills` paths are relative - to that root. Use `source: "./"` only when a full repo snapshot is intended. - Use `source: "./path/to/skill"` + `skills: ["./"]` for a single-skill narrow - cache. Use `source: "./"` for suite plugins whose cache should - contain only the suite members. +6. **`source` defines the installed plugin root**. For single-skill plugins, point + `source` directly at the skill directory (e.g., `"./tunnel-doctor"`) and omit + `skills` entirely — this is the official pattern used by 167/168 plugins in + `anthropics/claude-plugins-official`. For suite plugins, use + `source: "./"` with explicit `skills` array listing subdirectories. + Avoid `source: "./"` (installs full repo as cache) and `skills: ["./"]` + (rejected by Claude Code 2.1.x path-escape validator). 7. **Reserved marketplace names** that CANNOT be used: `claude-code-marketplace`, `claude-code-plugins`, `claude-plugins-official`, `anthropic-marketplace`, `anthropic-plugins`, `agent-skills`, `knowledge-work-plugins`, `life-sciences`. @@ -147,12 +149,11 @@ Use this template, filling in from the analysis: { "name": "", "description": "", - "source": "./", + "source": "./", "strict": false, "version": "1.0.0", "category": "", - "keywords": ["", ""], - "skills": ["./skills/"] + "keywords": ["", ""] } ] } @@ -305,9 +306,9 @@ For each plugin entry: - [ ] `description` matches SKILL.md frontmatter EXACTLY (not rewritten) - [ ] `version` is `"1.0.0"` for new plugins, bumped for changed plugins -- [ ] `source` starts with `"./"` and intentionally matches the plugin cache boundary -- [ ] Every `skills` path starts with `"./"` and resolves relative to `source` -- [ ] If `source` points directly at a skill root, `skills` is `["./"]` +- [ ] `source` points directly at the skill directory (e.g., `"./skill-name"`) +- [ ] Single-skill plugins omit the `skills` field (auto-discovery from `source`) +- [ ] Suite plugins list `skills` paths relative to `source` - [ ] `strict` is `false` (no plugin.json in repo) - [ ] `name` is kebab-case, unique across all entries diff --git a/daymade-claude-code/marketplace-dev/hooks/post_edit_sync_check.sh b/daymade-claude-code/marketplace-dev/hooks/post_edit_sync_check.sh index 872289e3..8a7f0d65 100755 --- a/daymade-claude-code/marketplace-dev/hooks/post_edit_sync_check.sh +++ b/daymade-claude-code/marketplace-dev/hooks/post_edit_sync_check.sh @@ -46,15 +46,17 @@ skill_name = '$SKILL_NAME' found = False for p in data.get('plugins', []): skills = p.get('skills', []) - for s in skills: - if s.rstrip('/').split('/')[-1] == skill_name: - found = True - version = p.get('version', 'unknown') - print(json.dumps({ - 'result': f'SKILL.md for \"{skill_name}\" was modified. Remember to bump its version in marketplace.json (currently {version}). Users on the old version won\\'t receive this update otherwise.' - })) - break - if found: + source = p.get('source', '') + # Match via skills array (suite plugins) or source path (single-skill plugins) + matched = any(s.rstrip('/').split('/')[-1] == skill_name for s in skills) + if not matched and isinstance(source, str): + matched = source.rstrip('/').split('/')[-1] == skill_name + if matched: + found = True + version = p.get('version', 'unknown') + print(json.dumps({ + 'result': f'SKILL.md for \"{skill_name}\" was modified. Remember to bump its version in marketplace.json (currently {version}). Users on the old version won\\'t receive this update otherwise.' + })) break if not found: diff --git a/daymade-claude-code/marketplace-dev/references/anti_patterns.md b/daymade-claude-code/marketplace-dev/references/anti_patterns.md index 0e3ac57d..9eb1e05d 100644 --- a/daymade-claude-code/marketplace-dev/references/anti_patterns.md +++ b/daymade-claude-code/marketplace-dev/references/anti_patterns.md @@ -57,9 +57,9 @@ Not theoretical — each cost debugging time. ### Using full repo source for a narrow plugin - **Symptom**: Installing a single plugin creates a cache containing many unrelated skill directories. -- **Fix**: Point `source` at the intended plugin root and make `skills` relative to - that root. For a skill whose `SKILL.md` is directly in the source root, use - `skills: ["./"]`. +- **Fix**: Point `source` directly at the skill directory (e.g., `"./my-skill"`) + and omit the `skills` field. Do not use `skills: ["./"]` — it is rejected by + Claude Code 2.1.x path-escape validator. ### Assuming marketplace name controls slash namespace - **Symptom**: Expecting `/daymade-skills:mermaid-tools` after installing diff --git a/daymade-claude-code/marketplace-dev/references/cache_and_source_patterns.md b/daymade-claude-code/marketplace-dev/references/cache_and_source_patterns.md index c0083c1d..1fa58209 100644 --- a/daymade-claude-code/marketplace-dev/references/cache_and_source_patterns.md +++ b/daymade-claude-code/marketplace-dev/references/cache_and_source_patterns.md @@ -28,17 +28,17 @@ marketplace -> plugin -> skill `source` defines the installed plugin root. `skills` paths are resolved relative to that root. -## Pattern: Single-Skill Narrow Cache +## Pattern: Single-Skill Plugin -Use this when a skill should install and update independently: +Use this when a skill should install and update independently. Point `source` +directly at the skill directory and omit `skills` (auto-discovery): ```json { "name": "mermaid-tools", "source": "./daymade-docs/mermaid-tools", "strict": false, - "version": "1.0.2", - "skills": ["./"] + "version": "1.0.2" } ``` @@ -54,6 +54,9 @@ The slash command remains `/mermaid-tools:mermaid-tools` because the plugin and skill have the same name. This is acceptable when independence matters more than namespace aesthetics. +This is the official pattern used by 167 of 168 plugins in +`anthropics/claude-plugins-official`. + ## Pattern: Suite Plugin Use this when related skills should share one namespace: @@ -97,15 +100,14 @@ ppt-creator/ ## Canonical Source for Suite Members If users also need single-skill installs for suite members, point the individual -plugin entries at the same canonical subdirectories: +plugin entries at the same canonical subdirectories and omit `skills`: ```json { "name": "pdf-creator", "source": "./daymade-docs/pdf-creator", "strict": false, - "version": "1.3.2", - "skills": ["./"] + "version": "1.3.2" } ``` @@ -119,13 +121,27 @@ creates drift and makes version bumps ambiguous. ```json { "name": "mermaid-tools", - "source": "./", - "skills": ["./mermaid-tools"] + "source": "./" +} +``` + +This installs a full repository cache for one plugin. The cache will contain +unrelated skills and can confuse debugging. Use `source: "./mermaid-tools"` +instead. + +### Using `skills: ["./"]` + +```json +{ + "name": "pdf-creator", + "source": "./daymade-docs/pdf-creator", + "skills": ["./"] } ``` -This loads correctly but installs a full repository cache for one plugin. The cache -will contain unrelated skills and can confuse debugging. +Rejected by Claude Code 2.1.x path-escape validator with `skills path "./" +escapes plugin root`. Omit the `skills` field — auto-discovery finds SKILL.md +in the `source` directory. ### Symlink suite directories diff --git a/daymade-claude-code/marketplace-dev/references/marketplace_schema.md b/daymade-claude-code/marketplace-dev/references/marketplace_schema.md index 454126af..dc55a25d 100644 --- a/daymade-claude-code/marketplace-dev/references/marketplace_schema.md +++ b/daymade-claude-code/marketplace-dev/references/marketplace_schema.md @@ -75,7 +75,7 @@ manifest schema, plus marketplace-specific fields. | Source | Format | Example | |--------|--------|---------| -| Relative path | string `"./"` | `"source": "./"` | +| Relative path | string `"./"` | `"source": "./my-skill"` | | GitHub | object | `{"source": "github", "repo": "owner/repo"}` | | Git URL | object | `{"source": "url", "url": "https://..."}` | | Git subdirectory | object | `{"source": "git-subdir", "url": "...", "path": "..."}` | diff --git a/daymade-skill/skill-creator/SKILL.md b/daymade-skill/skill-creator/SKILL.md index 275acc83..6fbf5e55 100644 --- a/daymade-skill/skill-creator/SKILL.md +++ b/daymade-skill/skill-creator/SKILL.md @@ -1027,12 +1027,11 @@ After packaging, update the marketplace registry to include the new or updated s { "name": "skill-name", "description": "Copy from SKILL.md frontmatter description", - "source": "./", + "source": "./skill-name", "strict": false, "version": "1.0.0", "category": "developer-tools", - "keywords": ["relevant", "keywords"], - "skills": ["./skill-name"] + "keywords": ["relevant", "keywords"] } ``` diff --git a/references/new-skill-guide.md b/references/new-skill-guide.md index c4f89050..0b8ac585 100644 --- a/references/new-skill-guide.md +++ b/references/new-skill-guide.md @@ -121,12 +121,11 @@ Use **skill-name** to [describe primary use case]. Combine with **other-skill** { "name": "skill-name", "description": "Clear description with trigger conditions. Use when [scenarios]", - "source": "./", + "source": "./skill-name", "strict": false, "version": "1.0.0", "category": "appropriate-category", - "keywords": ["keyword1", "keyword2", "keyword3"], - "skills": ["./skill-name"] + "keywords": ["keyword1", "keyword2", "keyword3"] } ``` diff --git a/references/plugin-architecture.md b/references/plugin-architecture.md index b5239ea4..4ade1384 100644 --- a/references/plugin-architecture.md +++ b/references/plugin-architecture.md @@ -48,18 +48,31 @@ Plugin (marketplace.json entry) ``` **Configuration** (in `.claude-plugin/marketplace.json`): + +Single-skill plugin (official pattern — `source` points to skill directory, no `skills` field): ```json { "name": "my-plugin", "description": "Use when...", + "source": "./my-plugin", "version": "1.0.0", "category": "utilities", - "keywords": ["keyword1", "keyword2"], + "keywords": ["keyword1", "keyword2"] +} +``` + +Suite plugin (multiple skills under one namespace — `skills` required): +```json +{ + "name": "my-suite", + "description": "Suite description", + "source": "./my-suite", + "version": "1.0.0", "skills": ["./skill-1", "./skill-2"] } ``` -**Example**: The `skill-creator` plugin contains one skill (`./skill-creator`), while a hypothetical `developer-tools` plugin might contain multiple skills like `./git-helper`, `./code-reviewer`, `./test-runner`. +**Example**: The `skill-creator` plugin contains one skill (`source: "./daymade-skill/skill-creator"`), while the `daymade-docs` suite plugin bundles multiple skills like `doc-to-markdown`, `pdf-creator`, `mermaid-tools`. ### 3. Agents (Subagents) @@ -139,8 +152,8 @@ claude plugin install macos-cleaner@daymade-skills "plugins": [ { "name": "macos-cleaner", - "version": "1.0.0", - "skills": ["./macos-cleaner"] + "source": "./macos-cleaner", + "version": "1.0.0" } ] } diff --git a/references/plugin-troubleshooting.md b/references/plugin-troubleshooting.md index 4f7beffa..536fa997 100644 --- a/references/plugin-troubleshooting.md +++ b/references/plugin-troubleshooting.md @@ -88,16 +88,16 @@ cat .claude-plugin/marketplace.json | jq '.plugins[] | select(.name == "skill-na **Common mistakes**: ```json -// ❌ Wrong: name mismatch +// ❌ Wrong: name vs source mismatch { "name": "macos_cleaner", // Underscore - "skills": ["./macos-cleaner"] // Dash + "source": "./macos-cleaner" // Dash } // ✅ Correct: consistent naming { "name": "macos-cleaner", - "skills": ["./macos-cleaner"] + "source": "./macos-cleaner" } ``` @@ -326,7 +326,7 @@ claude plugin install new-plugin@my-marketplace # ✅ Works // marketplace.json { "name": "my_plugin", // Underscore - "skills": ["./my-plugin"] // Dash - will cause confusion + "source": "./my-plugin" // Dash - will cause confusion } ``` From 35e207620755ee2d1f6f26b74174c45243f96e5f Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 10 May 2026 02:34:33 +0800 Subject: [PATCH 134/137] feat(marketplace): create daymade-audio suite for speech pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bundle 5 audio/speech skills into a new suite covering the full pipeline: recording → ASR transcription → transcript correction → meeting minutes → TTS Suite members: - asr-transcribe-to-text (from repo root) - stepfun-asr (from repo root, newly tracked) - transcript-fixer (from repo root) - meeting-minutes-taker (from daymade-docs — semantic analysis, not document format processing) - stepfun-tts (from repo root) Marketplace 1.53.2 → 1.54.0, plugin entries 55 → 56 (4 suites: daymade-audio, daymade-claude-code, daymade-docs, daymade-skill). Co-Authored-By: Claude Opus 4.6 (1M context) --- .claude-plugin/marketplace.json | 43 ++++- CHANGELOG.md | 10 + CLAUDE.md | 2 +- .../.security-scan-passed | 0 .../asr-transcribe-to-text}/SKILL.md | 0 .../references/local_mlx_guide.md | 0 .../references/overlap_merge_strategy.md | 0 .../scripts/overlap_merge_transcribe.py | 0 .../scripts/transcribe_local_mlx.py | 0 .../meeting-minutes-taker/SKILL.md | 0 .../completeness_review_checklist.md | 0 .../references/context_file_template.md | 0 .../references/meeting_minutes_template.md | 0 .../stepfun-asr/.security-scan-passed | 4 + daymade-audio/stepfun-asr/SKILL.md | 134 +++++++++++++ .../stepfun-asr/references/api_reference.md | 106 +++++++++++ .../stepfun-asr}/references/known_issues.md | 104 ++++------ .../stepfun-asr}/scripts/asr_transcribe.py | 0 .../stepfun-tts/.security-scan-passed | 4 + .../stepfun-tts}/SKILL.md | 44 ++--- .../stepfun-tts/references/api_reference.md | 94 +++++++++ .../stepfun-tts/references/known_issues.md | 62 ++++++ .../references/migration_from_v2.md | 0 .../stepfun-tts}/scripts/ab_compare.sh | 0 .../stepfun-tts}/scripts/tts_generate.py | 0 .../transcript-fixer}/.gitignore | 0 .../transcript-fixer}/SKILL.md | 0 .../references/architecture.md | 0 .../references/best_practices.md | 0 .../references/database_schema.md | 0 .../references/dictionary_guide.md | 0 .../references/example_session.md | 0 .../references/false_positive_guide.md | 0 .../references/file_formats.md | 0 .../references/glm_api_setup.md | 0 .../references/installation_setup.md | 0 .../references/iteration_workflow.md | 0 .../references/quick_reference.md | 0 .../references/script_parameters.md | 0 .../references/sql_queries.md | 0 .../references/team_collaboration.md | 0 .../references/troubleshooting.md | 0 .../references/workflow_guide.md | 0 .../transcript-fixer}/requirements.txt | 0 .../transcript-fixer}/scripts/__init__.py | 0 .../scripts/check_type_hints.py | 0 .../transcript-fixer}/scripts/cli/__init__.py | 0 .../scripts/cli/argument_parser.py | 0 .../transcript-fixer}/scripts/cli/commands.py | 0 .../scripts/core/__init__.py | 0 .../scripts/core/ai_processor.py | 0 .../scripts/core/ai_processor_async.py | 0 .../scripts/core/change_extractor.py | 0 .../scripts/core/connection_pool.py | 0 .../scripts/core/correction_repository.py | 0 .../scripts/core/correction_service.py | 0 .../scripts/core/dictionary_processor.py | 0 .../scripts/core/learning_engine.py | 0 .../transcript-fixer}/scripts/core/schema.sql | 0 .../transcript-fixer}/scripts/ensure_deps.py | 0 .../scripts/examples/bulk_import.py | 0 .../scripts/fix_transcript_enhanced.py | 0 .../scripts/fix_transcript_timestamps.py | 0 .../scripts/fix_transcription.py | 0 .../scripts/generate_word_diff.py | 0 .../scripts/split_transcript_sections.py | 0 .../scripts/tests/__init__.py | 0 .../scripts/tests/test_audit_log_retention.py | 0 .../scripts/tests/test_common_words_safety.py | 0 .../scripts/tests/test_connection_pool.py | 0 .../scripts/tests/test_correction_service.py | 0 .../scripts/tests/test_domain_validator.py | 0 .../scripts/tests/test_error_recovery.py | 0 .../tests/test_fix_transcript_timestamps.py | 0 .../scripts/tests/test_learning_engine.py | 0 .../scripts/tests/test_path_validator.py | 0 .../tests/test_split_transcript_sections.py | 0 .../scripts/utils/__init__.py | 0 .../scripts/utils/audit_log_retention.py | 0 .../scripts/utils/common_words.py | 0 .../scripts/utils/concurrency_manager.py | 0 .../transcript-fixer}/scripts/utils/config.py | 0 .../scripts/utils/database_migration.py | 0 .../scripts/utils/db_migrations_cli.py | 0 .../scripts/utils/diff_formats/__init__.py | 0 .../utils/diff_formats/change_extractor.py | 0 .../scripts/utils/diff_formats/html_format.py | 0 .../utils/diff_formats/inline_format.py | 0 .../utils/diff_formats/markdown_format.py | 0 .../utils/diff_formats/text_splitter.py | 0 .../utils/diff_formats/unified_format.py | 0 .../scripts/utils/diff_generator.py | 0 .../scripts/utils/domain_validator.py | 0 .../scripts/utils/health_check.py | 0 .../scripts/utils/logging_config.py | 0 .../scripts/utils/metrics.py | 0 .../scripts/utils/migrations.py | 0 .../scripts/utils/path_validator.py | 0 .../scripts/utils/rate_limiter.py | 0 .../scripts/utils/retry_logic.py | 0 .../scripts/utils/security.py | 0 .../scripts/utils/validation.py | 0 stepfun-tts/.security-scan-passed | 4 - stepfun-tts/references/api_reference.md | 178 ------------------ 104 files changed, 503 insertions(+), 286 deletions(-) rename {asr-transcribe-to-text => daymade-audio/asr-transcribe-to-text}/.security-scan-passed (100%) rename {asr-transcribe-to-text => daymade-audio/asr-transcribe-to-text}/SKILL.md (100%) rename {asr-transcribe-to-text => daymade-audio/asr-transcribe-to-text}/references/local_mlx_guide.md (100%) rename {asr-transcribe-to-text => daymade-audio/asr-transcribe-to-text}/references/overlap_merge_strategy.md (100%) rename {asr-transcribe-to-text => daymade-audio/asr-transcribe-to-text}/scripts/overlap_merge_transcribe.py (100%) rename {asr-transcribe-to-text => daymade-audio/asr-transcribe-to-text}/scripts/transcribe_local_mlx.py (100%) rename {daymade-docs => daymade-audio}/meeting-minutes-taker/SKILL.md (100%) rename {daymade-docs => daymade-audio}/meeting-minutes-taker/references/completeness_review_checklist.md (100%) rename {daymade-docs => daymade-audio}/meeting-minutes-taker/references/context_file_template.md (100%) rename {daymade-docs => daymade-audio}/meeting-minutes-taker/references/meeting_minutes_template.md (100%) create mode 100644 daymade-audio/stepfun-asr/.security-scan-passed create mode 100644 daymade-audio/stepfun-asr/SKILL.md create mode 100644 daymade-audio/stepfun-asr/references/api_reference.md rename {stepfun-tts => daymade-audio/stepfun-asr}/references/known_issues.md (56%) rename {stepfun-tts => daymade-audio/stepfun-asr}/scripts/asr_transcribe.py (100%) create mode 100644 daymade-audio/stepfun-tts/.security-scan-passed rename {stepfun-tts => daymade-audio/stepfun-tts}/SKILL.md (52%) create mode 100644 daymade-audio/stepfun-tts/references/api_reference.md create mode 100644 daymade-audio/stepfun-tts/references/known_issues.md rename {stepfun-tts => daymade-audio/stepfun-tts}/references/migration_from_v2.md (100%) rename {stepfun-tts => daymade-audio/stepfun-tts}/scripts/ab_compare.sh (100%) rename {stepfun-tts => daymade-audio/stepfun-tts}/scripts/tts_generate.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/.gitignore (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/SKILL.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/architecture.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/best_practices.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/database_schema.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/dictionary_guide.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/example_session.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/false_positive_guide.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/file_formats.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/glm_api_setup.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/installation_setup.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/iteration_workflow.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/quick_reference.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/script_parameters.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/sql_queries.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/team_collaboration.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/troubleshooting.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/references/workflow_guide.md (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/requirements.txt (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/__init__.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/check_type_hints.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/cli/__init__.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/cli/argument_parser.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/cli/commands.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/__init__.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/ai_processor.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/ai_processor_async.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/change_extractor.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/connection_pool.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/correction_repository.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/correction_service.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/dictionary_processor.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/learning_engine.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/core/schema.sql (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/ensure_deps.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/examples/bulk_import.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/fix_transcript_enhanced.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/fix_transcript_timestamps.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/fix_transcription.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/generate_word_diff.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/split_transcript_sections.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/__init__.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_audit_log_retention.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_common_words_safety.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_connection_pool.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_correction_service.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_domain_validator.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_error_recovery.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_fix_transcript_timestamps.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_learning_engine.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_path_validator.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/tests/test_split_transcript_sections.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/__init__.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/audit_log_retention.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/common_words.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/concurrency_manager.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/config.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/database_migration.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/db_migrations_cli.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/diff_formats/__init__.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/diff_formats/change_extractor.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/diff_formats/html_format.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/diff_formats/inline_format.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/diff_formats/markdown_format.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/diff_formats/text_splitter.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/diff_formats/unified_format.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/diff_generator.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/domain_validator.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/health_check.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/logging_config.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/metrics.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/migrations.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/path_validator.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/rate_limiter.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/retry_logic.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/security.py (100%) rename {transcript-fixer => daymade-audio/transcript-fixer}/scripts/utils/validation.py (100%) delete mode 100644 stepfun-tts/.security-scan-passed delete mode 100644 stepfun-tts/references/api_reference.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 525dd32d..bcc14b16 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -6,13 +6,13 @@ }, "metadata": { "description": "Professional Claude Code skills for GitHub operations, document conversion, diagram generation, statusline customization, Teams communication, repomix utilities, skill creation, CLI demo generation, LLM icon access, Cloudflare troubleshooting, UI design system extraction, professional presentation creation, YouTube video downloading, secure repomix packaging, ASR transcription correction, video comparison quality analysis, comprehensive QA testing infrastructure, prompt optimization with EARS methodology, session history recovery, local Claude session continuation from `.claude` artifacts, documentation cleanup, format-controlled deep research report generation with evidence tracking, PDF generation with Chinese font support, CLAUDE.md progressive disclosure optimization, CCPM skill registry search and management, Promptfoo LLM evaluation framework, iOS app development with XcodeGen and SwiftUI, fact-checking with automated corrections, Twitter/X content fetching, intelligent macOS disk space recovery, skill quality review and improvement, GitHub contribution strategy, complete internationalization/localization setup, plugin/skill troubleshooting with diagnostic tools, evidence-based competitor analysis with source citations, Windows Remote Desktop (AVD/W365) connection quality diagnosis with transport protocol analysis and log parsing, Tailscale+proxy conflict diagnosis with SSH tunnel SOP for remote development, multi-path parallel product analysis with cross-model test-time compute scaling, real financial data collection for US equities with validation and yfinance pitfall handling, advanced Excel automation for formatted workbook generation and complex xlsm parsing, macOS programmatic window screenshot capture workflows, verified Scrapling CLI installation and web extraction workflows, plugin marketplace development for converting skills repos into official Claude Code marketplaces, Tencent IMA knowledge base companion with zero-config installation across Claude Code/Codex/OpenClaw, upstream bug detection and runtime repair, personalized fan-out search with priority-based knowledge base boosting, and suite plugins that expose related skills under shared namespaces for combined installation workflows — including a dedicated Claude Code operations suite (daymade-claude-code) bundling session recovery, CLAUDE.md optimization, troubleshooting, statusline configuration, export repair, and marketplace development skills under one namespace, and the StepFun StepAudio 2.5 audio family (stepfun-tts for Contextual TTS with instruction + inline parentheses, stepfun-asr for the SSE endpoint that handles 30-minute audio in a single call)", - "version": "1.53.2" + "version": "1.54.0" }, "plugins": [ { "name": "asr-transcribe-to-text", "description": "Transcribe audio and video files to text using a remote ASR service (Qwen3-ASR or OpenAI-compatible endpoint). Extracts audio from video, sends to configurable ASR endpoint, outputs clean text. Use when the user wants to transcribe recordings, convert audio/video to text, do speech-to-text, or mentions ASR, Qwen ASR, 转录, 语音转文字, 录音转文字, or has a meeting recording, lecture, interview, or screen recording to transcribe.", - "source": "./asr-transcribe-to-text", + "source": "./daymade-audio/asr-transcribe-to-text", "strict": false, "version": "1.0.0", "category": "productivity", @@ -212,7 +212,7 @@ }, { "name": "daymade-docs", - "description": "Documentation suite plugin that exposes document conversion, Mermaid diagram generation, PDF/PPT creation, documentation cleanup, and meeting minutes skills under one shared namespace", + "description": "Documentation suite plugin that exposes document conversion, Mermaid diagram generation, PDF/PPT creation, and documentation cleanup skills under one shared namespace", "source": "./daymade-docs", "strict": false, "version": "1.0.1", @@ -231,8 +231,33 @@ "./mermaid-tools", "./pdf-creator", "./ppt-creator", - "./docs-cleaner", - "./meeting-minutes-taker" + "./docs-cleaner" + ] + }, + { + "name": "daymade-audio", + "description": "Audio processing suite covering the full speech pipeline: ASR transcription (Qwen3, StepFun), transcript error correction, structured meeting minutes generation, and TTS voice synthesis (StepFun). Install once for the complete audio workflow.", + "source": "./daymade-audio", + "strict": false, + "version": "1.0.0", + "category": "suite", + "keywords": [ + "suite", + "audio", + "asr", + "tts", + "transcription", + "speech-to-text", + "text-to-speech", + "meeting-minutes", + "voice" + ], + "skills": [ + "./asr-transcribe-to-text", + "./stepfun-asr", + "./transcript-fixer", + "./meeting-minutes-taker", + "./stepfun-tts" ] }, { @@ -593,7 +618,7 @@ { "name": "meeting-minutes-taker", "description": "Transform meeting transcripts into high-fidelity, structured meeting minutes with iterative review. Features speaker identification via feature analysis (word count, speaking style, topic focus) with context.md team directory mapping, intelligent file naming from content, integration with doc-to-markdown and transcript-fixer for pre-processing, evidence-based recording with speaker quotes, Mermaid diagrams for architecture discussions, and multi-turn parallel generation with UNION merge", - "source": "./daymade-docs/meeting-minutes-taker", + "source": "./daymade-audio/meeting-minutes-taker", "strict": false, "version": "1.1.1", "category": "productivity", @@ -824,7 +849,7 @@ { "name": "stepfun-tts", "description": "Generate Chinese / Japanese speech with StepFun's stepaudio-2.5-tts — Contextual TTS that replaces step-tts-2's `voice_label` with natural-language `instruction` (≤200 chars) plus inline `()` parentheses for句内 prosody. Use when the user wants emotional / prosody control over voice synthesis (whisper, pause, stress, mood pivot mid-sentence), batch-generates game / app voice lines, migrates from `step-tts-2` (the `voice_label → instruction` breaking change), or hits StepFun's stricter 2.5-era censorship (死/消失/political terms). Triggers on 阶跃 TTS, StepAudio 合成, 语音合成, 配音, 文本转语音, TTS 升级, 迁移 step-tts-2. For transcription with the sibling stepaudio-2.5-asr model, use the stepfun-asr skill instead.", - "source": "./stepfun-tts", + "source": "./daymade-audio/stepfun-tts", "strict": false, "version": "2.0.0", "category": "productivity", @@ -842,7 +867,7 @@ { "name": "stepfun-asr", "description": "Transcribe audio with StepFun's stepaudio-2.5-asr — an SSE endpoint (NOT /v1/audio/transcriptions) with 32K context, ~85-101x RTF on long audio, and a single-call ceiling around 30 minutes (no client-side chunking). Use when transcribing Chinese / English audio with StepFun, when long-form recordings (5-30 min) need to land in one request, when migrating from step-asr / step-asr-1.1, or when hitting the misleading `model stepaudio-2.5-asr not supported` error (which actually means wrong endpoint). Triggers on 阶跃 ASR, StepFun ASR, stepaudio-2.5-asr, 转录, 语音识别, 长音频转写, 语音转文字. For TTS with the sibling stepaudio-2.5-tts model, use the stepfun-tts skill instead.", - "source": "./stepfun-asr", + "source": "./daymade-audio/stepfun-asr", "strict": false, "version": "1.0.0", "category": "productivity", @@ -894,7 +919,7 @@ { "name": "transcript-fixer", "description": "Corrects speech-to-text (ASR/STT) transcription errors using dictionary rules and native Claude AI corrections (no API key needed by default). Supports intelligent paragraph breaks, filler word reduction, interactive review, Chinese domain names, and iterative dictionary building. Use when users mention transcript correction, ASR errors, speech-to-text mistakes, homophone errors, or working with transcription files", - "source": "./transcript-fixer", + "source": "./daymade-audio/transcript-fixer", "strict": false, "version": "1.4.0", "category": "productivity", diff --git a/CHANGELOG.md b/CHANGELOG.md index f9d61129..8902d080 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.54.0] - 2026-05-10 + +### Added +- **daymade-audio** suite v1.0.0: Audio processing suite covering the full speech pipeline — ASR transcription (Qwen3, StepFun), transcript error correction, structured meeting minutes generation, and TTS voice synthesis. Bundles 5 skills: `asr-transcribe-to-text`, `stepfun-asr`, `transcript-fixer`, `meeting-minutes-taker`, `stepfun-tts`. + +### Changed +- Move `meeting-minutes-taker` from `daymade-docs` to `daymade-audio` — its core capability is semantic analysis of meeting transcripts, not document format processing. +- Move `asr-transcribe-to-text`, `stepfun-asr`, `stepfun-tts`, `transcript-fixer` from repo root into `daymade-audio/` suite directory. +- Marketplace plugin count: 55 → 56 (4 suites now: `daymade-audio`, `daymade-claude-code`, `daymade-docs`, `daymade-skill`). + ## [1.53.2] - 2026-05-10 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 3d6fd23f..bb180e03 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -152,7 +152,7 @@ If it fires, fix the issue — do NOT use `--no-verify` to bypass. ## Marketplace Configuration The marketplace is configured in `.claude-plugin/marketplace.json`: -- Contains 55 plugin entries: single-skill plugins point `source` directly at the skill directory (no `skills` field); suite plugins (`daymade-docs`, `daymade-claude-code`, `daymade-skill`) use explicit `skills` arrays for multi-skill routing +- Contains 56 plugin entries: single-skill plugins point `source` directly at the skill directory (no `skills` field); suite plugins (`daymade-audio`, `daymade-claude-code`, `daymade-docs`, `daymade-skill`) use explicit `skills` arrays for multi-skill routing - Each plugin has: name, description, source, version, category, keywords - Marketplace metadata: name, owner, version - Single-skill plugins follow the official pattern (167/168 plugins in `anthropics/claude-plugins-official`): `source` points to skill directory, `skills` omitted diff --git a/asr-transcribe-to-text/.security-scan-passed b/daymade-audio/asr-transcribe-to-text/.security-scan-passed similarity index 100% rename from asr-transcribe-to-text/.security-scan-passed rename to daymade-audio/asr-transcribe-to-text/.security-scan-passed diff --git a/asr-transcribe-to-text/SKILL.md b/daymade-audio/asr-transcribe-to-text/SKILL.md similarity index 100% rename from asr-transcribe-to-text/SKILL.md rename to daymade-audio/asr-transcribe-to-text/SKILL.md diff --git a/asr-transcribe-to-text/references/local_mlx_guide.md b/daymade-audio/asr-transcribe-to-text/references/local_mlx_guide.md similarity index 100% rename from asr-transcribe-to-text/references/local_mlx_guide.md rename to daymade-audio/asr-transcribe-to-text/references/local_mlx_guide.md diff --git a/asr-transcribe-to-text/references/overlap_merge_strategy.md b/daymade-audio/asr-transcribe-to-text/references/overlap_merge_strategy.md similarity index 100% rename from asr-transcribe-to-text/references/overlap_merge_strategy.md rename to daymade-audio/asr-transcribe-to-text/references/overlap_merge_strategy.md diff --git a/asr-transcribe-to-text/scripts/overlap_merge_transcribe.py b/daymade-audio/asr-transcribe-to-text/scripts/overlap_merge_transcribe.py similarity index 100% rename from asr-transcribe-to-text/scripts/overlap_merge_transcribe.py rename to daymade-audio/asr-transcribe-to-text/scripts/overlap_merge_transcribe.py diff --git a/asr-transcribe-to-text/scripts/transcribe_local_mlx.py b/daymade-audio/asr-transcribe-to-text/scripts/transcribe_local_mlx.py similarity index 100% rename from asr-transcribe-to-text/scripts/transcribe_local_mlx.py rename to daymade-audio/asr-transcribe-to-text/scripts/transcribe_local_mlx.py diff --git a/daymade-docs/meeting-minutes-taker/SKILL.md b/daymade-audio/meeting-minutes-taker/SKILL.md similarity index 100% rename from daymade-docs/meeting-minutes-taker/SKILL.md rename to daymade-audio/meeting-minutes-taker/SKILL.md diff --git a/daymade-docs/meeting-minutes-taker/references/completeness_review_checklist.md b/daymade-audio/meeting-minutes-taker/references/completeness_review_checklist.md similarity index 100% rename from daymade-docs/meeting-minutes-taker/references/completeness_review_checklist.md rename to daymade-audio/meeting-minutes-taker/references/completeness_review_checklist.md diff --git a/daymade-docs/meeting-minutes-taker/references/context_file_template.md b/daymade-audio/meeting-minutes-taker/references/context_file_template.md similarity index 100% rename from daymade-docs/meeting-minutes-taker/references/context_file_template.md rename to daymade-audio/meeting-minutes-taker/references/context_file_template.md diff --git a/daymade-docs/meeting-minutes-taker/references/meeting_minutes_template.md b/daymade-audio/meeting-minutes-taker/references/meeting_minutes_template.md similarity index 100% rename from daymade-docs/meeting-minutes-taker/references/meeting_minutes_template.md rename to daymade-audio/meeting-minutes-taker/references/meeting_minutes_template.md diff --git a/daymade-audio/stepfun-asr/.security-scan-passed b/daymade-audio/stepfun-asr/.security-scan-passed new file mode 100644 index 00000000..489a2c85 --- /dev/null +++ b/daymade-audio/stepfun-asr/.security-scan-passed @@ -0,0 +1,4 @@ +Security scan passed +Scanned at: 2026-04-30T16:44:52.294771 +Tool: gitleaks + pattern-based validation +Content hash: a0edbbb580527ed34ca4ae9bda443c6bd93a200434e85989c2a636d4435478a5 diff --git a/daymade-audio/stepfun-asr/SKILL.md b/daymade-audio/stepfun-asr/SKILL.md new file mode 100644 index 00000000..c240a562 --- /dev/null +++ b/daymade-audio/stepfun-asr/SKILL.md @@ -0,0 +1,134 @@ +--- +name: stepfun-asr +description: Transcribe audio with StepFun's stepaudio-2.5-asr — an SSE endpoint (NOT /v1/audio/transcriptions) with 32K context, ~85-101x RTF on long audio, and a single-call ceiling around 30 minutes (no client-side chunking). Use when transcribing Chinese / English audio with StepFun, when long-form recordings (5-30 min) need to land in one request, when migrating from step-asr / step-asr-1.1, or when hitting the misleading `model stepaudio-2.5-asr not supported` error (which actually means wrong endpoint). Triggers on 阶跃 ASR, StepFun ASR, stepaudio-2.5-asr, 转录, 语音识别, 长音频转写, 语音转文字. For TTS with the sibling stepaudio-2.5-tts model, use the stepfun-tts skill instead. +--- + +# StepFun stepaudio-2.5-asr + +Transcribe audio with StepFun's `stepaudio-2.5-asr` (released 2026-04, verified 2026-04-23). Long audio in one call, no chunking — but **only** if the request hits the right endpoint with the right body shape. The wrong endpoint returns an error that looks identical to "model doesn't exist", which is the #1 reason this skill exists. + +> Companion: for TTS with `stepaudio-2.5-tts` (the sibling model), use the `stepfun-tts` skill — they share an API key but live on different endpoints with different body shapes. + +## Why this skill exists — three traps that cost hours + +1. **Wrong endpoint, wrong error**. `stepaudio-2.5-asr` does **not** live on `/v1/audio/transcriptions` (that endpoint serves the older `step-asr` family). It lives on `/v1/audio/asr/sse` — SSE streaming, JSON body, base64 audio. Sending it to the wrong endpoint returns `{"error":{"message":"model stepaudio-2.5-asr not supported"}}`, which is **identical in structure** to a genuinely nonexistent model name. People waste hours filing whitelist tickets. + +2. **Plan key vs Normal key, silent failure**. StepFun's "Plan" subscription keys (cheap, text-only) cannot call audio endpoints, but the failure manifests as a 4xx with no auth-shaped error message. If your account has a Plan subscription, you need a separate "Normal" key from the same console. + +3. **SSE error events are real**. Censorship can fire on the ASR side too (rarely). Don't assume only `transcript.text.delta` and `transcript.text.done` events arrive — handle `type: error` events in the stream or you'll silently drop them. + +## Config and auth + +API key resolves in this order (fail-fast, no defaults): + +1. `$STEPFUN_API_KEY` environment variable +2. `${CLAUDE_PLUGIN_DATA}/config.json` with `{"api_key": "..."}` (cross-session persistence) + +First-time setup: + +```bash +mkdir -p "${CLAUDE_PLUGIN_DATA}" && cat > "${CLAUDE_PLUGIN_DATA}/config.json" <"} +EOF +``` + +If the user has not set a key, ask them to paste it — do not guess or use a placeholder. Get keys at https://platform.stepfun.com/ → API Keys. **Use a Normal key, not a Plan key.** + +## Quick start — single file + +```bash +python3 scripts/asr_transcribe.py /path/to/audio.mp3 +``` + +Output: plain text transcription on stdout. + +For machine-readable output with usage / timing: + +```bash +python3 scripts/asr_transcribe.py /path/to/audio.mp3 --json +``` + +For non-Chinese audio: + +```bash +python3 scripts/asr_transcribe.py /path/to/audio.mp3 --language en +``` + +The script handles base64 encoding, the nested `{audio: {data, input: {transcription, format}}}` body, SSE parsing, and the misleading-endpoint pitfall. Prefer it over hand-rolled HTTP calls unless integrating into a larger pipeline. + +## Decision table + +| Scenario | Action | +|---|---| +| Short clip (< 5 min), Chinese or English, mp3/wav/ogg/opus | `python3 scripts/asr_transcribe.py audio.mp3` | +| Long audio (5-30 min) | Same script — 32K context handles it in a single call, no chunking needed | +| Audio > 30 min | Split with ffmpeg before sending; the API rejects oversized payloads | +| Need usage/billing data | Add `--json` to capture `usage.input_tokens` / `usage.total_tokens` from `transcript.text.done` | +| Highly repetitive content (same phrase 5+ times, > 90s) | Cross-validate with `step-asr-1.1` — see repetition hallucination in `references/known_issues.md` | +| Hit `model stepaudio-2.5-asr not supported` | Wrong endpoint. Switch from `/v1/audio/transcriptions` to `/v1/audio/asr/sse` | +| Hit silent 4xx auth failure | Verify your key is "Normal" not "Plan" — Plan keys cannot call audio endpoints | +| Need to write raw HTTP (no Python) | Read `references/api_reference.md` for exact JSON body and SSE event shapes | + +## Supported audio formats + +The script auto-detects from extension; pass `--format` to override: + +| Extension | Format flag | Notes | +|---|---|---| +| `.mp3` | `mp3` | Most common, default | +| `.wav` | `wav` | Lossless | +| `.ogg` | `ogg` | OGG container | +| `.opus` | `ogg` | Opus codec in OGG container — pass through unchanged | +| `.pcm` | `pcm` | Raw PCM — also requires `format.rate`, `format.channel`, `format.bits` (see API reference) | + +For mp4/m4a/webm/etc., transcode to one of the above first via ffmpeg. Production pipelines often pre-transcode everything to OGG/Opus 16kHz mono to minimize base64 payload size. + +## Capacity and performance (verified 2026-04-23) + +- **32K context window** — single-call upper limit, no chunking needed for ≤ 30 min audio +- **~85-101× RTF** on long audio (17.4 min audio → 10.4s wall clock) +- **~5.3× speedup vs step-asr-1.1** at the 100s+ length range +- **Only ~2× speedup** at the 5-15s range — the LLM spin-up cost dominates short clips. If your workload is many short clips, the migration ROI is modest + +## Common error patterns + +| Error response | Actual cause | Fix | +|---|---|---| +| `"model stepaudio-2.5-asr not supported"` on `/v1/audio/transcriptions` | Wrong endpoint | Switch to `/v1/audio/asr/sse` (script does this) | +| Silent 4xx with no auth message | Using a "Plan" key on audio endpoint | Get a "Normal" key from the StepFun console | +| ASR returns 3-4× expected character count | Repetition hallucination on highly-repetitive audio | Cross-validate with `step-asr-1.1`; see `references/known_issues.md` | +| `data: {"type":"error","message":"content blocked..."}` mid-stream | Censorship fired on user-uploaded content | Handle SSE `error` event explicitly; don't assume only `delta`/`done` arrive | + +More edge cases in `references/known_issues.md`. + +## Design invariants (do not break) + +1. **Always pass through SSE** — don't try to buffer the response with a non-streaming client. The model emits `transcript.text.delta` for long audio; `transcript.text.done` carries the authoritative full text and `usage`. Reject the SSE format entirely and you'll get nothing. +2. **Take final text from `transcript.text.done.text`** — concatenated deltas can drift on edge cases. Deltas are for progressive UI; the `done` event is the source of truth. +3. **Handle `error` events in the stream** — don't treat the SSE stream as if only success events arrive. A blocked-content event mid-stream returns `type: error` with no `done` event. +4. **Fail-fast on missing API key** — never default to a placeholder or empty string. The script does this; preserve the behavior in any wrapper. + +## When to read references + +- `references/api_reference.md` — exact JSON request body, all fields, all SSE event types, response examples. Read when writing raw HTTP calls instead of using the bundled script. +- `references/known_issues.md` — repetition hallucination details, the wrong-endpoint diagnostic trail, Plan-vs-Normal key gotcha, ASR-side censorship handling, pricing opacity. Read when debugging anomalous output or evaluating whether to migrate from `step-asr-1.1`. + +## Pricing (verified 2026-04-23, volatile) + +`stepaudio-2.5-asr` is in invitation beta as of 2026-04-23 — no public per-minute rate. The `step-asr-1.1` baseline is 2.2 元/小时. The invitation PDF mentions "成本直降 80%" implying ~0.4 元/小时, but this is not yet on the pricing page. Re-verify at https://platform.stepfun.com/docs/zh/guides/pricing/details before quoting to stakeholders. + +## Next Step + +ASR transcripts almost always benefit from one of two downstream steps. After producing the transcript, suggest the natural next skill: + +``` +Transcription complete. The output is raw text from the model — common next steps: + +Options: +A) transcript-fixer — clean up ASR errors (homophones, segmentation, filler words). Recommended if the recording is a real-world conversation, podcast, or interview rather than read-aloud text +B) meeting-minutes-taker — turn the transcript into structured minutes with decisions, action items, and speaker attribution. Recommended if the recording is a meeting +C) No thanks — the raw transcript is what I needed +``` + +Skip the suggestion when the user has already specified the downstream tool, or when the transcription was clearly a one-off lookup (e.g., "what does this 15-second clip say?"). + diff --git a/daymade-audio/stepfun-asr/references/api_reference.md b/daymade-audio/stepfun-asr/references/api_reference.md new file mode 100644 index 00000000..6868802a --- /dev/null +++ b/daymade-audio/stepfun-asr/references/api_reference.md @@ -0,0 +1,106 @@ +# stepaudio-2.5-asr API Reference + +Exact request/response shapes for `stepaudio-2.5-asr`. Verified 2026-04-23 against the live StepFun API. Read this when calling the API by hand (curl, custom HTTP client) instead of using the bundled `scripts/asr_transcribe.py`. + +## Endpoint (NOT the one you'd guess) + +``` +POST https://api.stepfun.com/v1/audio/asr/sse +Content-Type: application/json +Accept: text/event-stream +Authorization: Bearer +``` + +**Do NOT** send `stepaudio-2.5-asr` to `/v1/audio/transcriptions` — that endpoint serves the older `step-asr` / `step-asr-1.1` family and returns a misleading `model stepaudio-2.5-asr not supported` error which looks identical to a permission/whitelist error. See `known_issues.md` for the full diagnostic trail. + +## Request body + +```json +{ + "audio": { + "data": "", + "input": { + "transcription": { + "language": "zh", + "model": "stepaudio-2.5-asr", + "enable_itn": true + }, + "format": { + "type": "mp3" + } + } + } +} +``` + +| Path | Required | Type | Notes | +|---|---|---|---| +| `audio.data` | yes | string | base64-encoded audio bytes. Accepts mp3, wav, ogg, opus (in ogg container), pcm | +| `audio.input.transcription.language` | yes | string | `zh` or `en`. Dialects and Japanese are not officially supported | +| `audio.input.transcription.model` | yes | string | Must be `stepaudio-2.5-asr` | +| `audio.input.transcription.enable_itn` | no | bool | Inverse text normalization (数字→words). Default true | +| `audio.input.format.type` | yes | string | `mp3` / `wav` / `ogg` / `pcm` | +| `audio.input.format.rate` | pcm only | int | Sample rate (required for raw PCM) | +| `audio.input.format.channel` | pcm only | int | Channel count (required for raw PCM) | +| `audio.input.format.bits` | optional | int | Sample depth, default 16 | + +## Response — SSE stream + +The response is a Server-Sent Events stream. Each line is either empty or starts with `data: `. Three event types: + +``` +data: {"type":"transcript.text.delta","meta":{...},"delta":"你好，"} + +data: {"type":"transcript.text.delta","meta":{...},"delta":"我是蕾格。"} + +data: {"type":"transcript.text.done","meta":{...},"text":"你好，我是蕾格。","usage":{"type":"tokens","input_tokens":69,"input_token_details":{"text_tokens":69,"audio_tokens":0},"output_tokens":9,"total_tokens":78}} +``` + +| Event type | Meaning | How to handle | +|---|---|---| +| `transcript.text.delta` | Incremental piece of the transcription | Concatenate for progressive UI; optional if you only need final text | +| `transcript.text.done` | Final, full transcription + usage | Take `text` as the authoritative result. Also contains `usage` for billing/telemetry | +| `error` | Server-side error mid-stream | Abort and propagate `message` to the caller | + +## Capacity + +- 32K context window +- Audio ≤ 30 min can be sent in a single call +- No client-side chunking needed for long audio (unlike step-asr) +- RTF 85-101× on Chinese speech verified 2026-04-23 + +## Known error responses + +```json +{"error":{"message":"model stepaudio-2.5-asr not supported","type":"request_params_invalid"}} +``` +→ Wrong endpoint. Switch from `/v1/audio/transcriptions` to `/v1/audio/asr/sse`. + +``` +data: {"type":"error","message":"content blocked ..."} +``` +→ Content censorship (rare on ASR). Same triggers as TTS (death/disappearance/political terms). + +## Comparison with sibling and legacy endpoints + +| Model | Endpoint | Request format | +|---|---|---| +| `stepaudio-2.5-asr` (this skill) | `/v1/audio/asr/sse` | JSON + base64 audio + SSE response | +| `stepaudio-2.5-tts` (sibling, see `stepfun-tts` skill) | `/v1/audio/speech` | JSON with `instruction` (no `voice_label`) | +| `step-asr` / `step-asr-1.1` (legacy) | `/v1/audio/transcriptions` | multipart/form-data | +| `step-tts-2` / `step-tts-mini` (legacy) | `/v1/audio/speech` | JSON with `voice_label` | + +Legacy `step-asr-1.1` is the fallback when `stepaudio-2.5-asr` hits the repetition-hallucination edge case (see `known_issues.md`). The endpoint and body shape are entirely different — multipart upload to `/v1/audio/transcriptions`, no SSE. + +## Auth and key handling + +- Key header: `Authorization: Bearer ` +- Keys can be retrieved at https://platform.stepfun.com/ → API Keys +- "Plan" keys (cheaper subscription) are **restricted** to text models on `api.stepfun.com/step_plan`. They **cannot** call audio endpoints. Use a "Normal" key for ASR calls. +- Same key works for both TTS and ASR — no separate scopes + +## Rate / throughput notes (observed, not officially documented) + +- Long audio ASR (17 min) has succeeded with `timeout=1200` in the bundled script +- ~400ms sleep between sequential requests avoids 429s in batch processing +- Single TCP connection per request — SSE stream is closed after `transcript.text.done` diff --git a/stepfun-tts/references/known_issues.md b/daymade-audio/stepfun-asr/references/known_issues.md similarity index 56% rename from stepfun-tts/references/known_issues.md rename to daymade-audio/stepfun-asr/references/known_issues.md index b9757b99..82a79a4c 100644 --- a/stepfun-tts/references/known_issues.md +++ b/daymade-audio/stepfun-asr/references/known_issues.md @@ -1,7 +1,25 @@ -# StepAudio 2.5 — Known Issues and Non-Obvious Behavior +# stepaudio-2.5-asr — Known Issues and Non-Obvious Behavior Collected from end-to-end testing 2026-04-23. These are things that burned real time to discover; they are not in the official docs. +## Wrong endpoint gives a misleading error (the #1 trap) + +**Symptom:** Calling `/v1/audio/transcriptions` with `model=stepaudio-2.5-asr`: + +```json +{"error":{"message":"model stepaudio-2.5-asr not supported","type":"request_params_invalid"}} +``` + +This response is **identical in structure** to sending a genuinely nonexistent model name. It takes real debugging to realize the model exists but on a different endpoint. + +**Diagnostic sequence that wastes the least time:** + +1. Try `step-asr` on the same endpoint — if it works, endpoint access is fine +2. Check the `/v1/audio/asr/sse` endpoint (the actual stepaudio-2.5-asr home) +3. If both fail, THEN ask BD about whitelist + +Don't assume "permission denied" on the first error. + ## ASR repetition hallucination (real, bounded) **Symptom:** Transcribe a TTS-generated audio of highly-repetitive Chinese text (e.g., the same 60-char sentence repeated 10 times) and `stepaudio-2.5-asr` returns 3-4× the expected character count, with the same sentence restated many extra times in the output. @@ -27,25 +45,6 @@ Collected from end-to-end testing 2026-04-23. These are things that burned real - If your domain has genuinely repetitive content (e.g., IVR transcripts, repeated sloganeering), cross-validate with `step-asr-1.1` on random samples - For most workflows: just use it; the hallucination mode is exotic -## Stricter content censorship than step-tts-2 - -**Symptom:** `stepaudio-2.5-tts` returns `{"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_block"}}` for content that step-tts-2 happily synthesized. - -**Observed triggers:** -- 死 (die/dead) in any context, even negation -- 消失 (disappear / vanish) -- Combinations with emotional context: "我快要...消失了" -- Politically sensitive terms (standard CN content rules) - -**Key insight:** Rewriting negations doesn't help — "我没有死" blocks as readily as "我死了". The classifier isn't doing deep semantic parsing. - -**Response strategies** (pick per line): -1. Rewrite: "RAG 已死" → "这个技术过时了" -2. Fallback: keep step-tts-2 for the 2-5% of lines that block -3. Whitelist: contact StepFun BD (worth it at >5% blockage) - -See `migration_from_v2.md` for the full blocking→fallback workflow. - ## ASR speed scales non-linearly — short audio is a trap **Observation:** The headline "5.9× faster than step-asr" from the marketing is true for long audio but misleading for short clips. @@ -60,49 +59,7 @@ See `migration_from_v2.md` for the full blocking→fallback workflow. **Practical implication:** If your workload is many short (<10s) clips, the speedup over `step-asr-1.1` is modest — 2× not 5×. If your workload is long audio (>2 min), the difference is dramatic and you should migrate. -## Wrong ASR endpoint gives a misleading error - -**Symptom:** Calling `/v1/audio/transcriptions` with `model=stepaudio-2.5-asr`: - -```json -{"error":{"message":"model stepaudio-2.5-asr not supported","type":"request_params_invalid"}} -``` - -This response is **identical in structure** to sending a genuinely nonexistent model name. It takes real debugging to realize the model exists but on a different endpoint. - -**Diagnostic sequence that wastes the least time:** - -1. Try `step-asr` on the same endpoint — if it works, endpoint access is fine -2. Check the `/v1/audio/asr/sse` endpoint (the actual stepaudio-2.5-asr home) -3. If both fail, THEN ask BD about whitelist - -Don't assume "permission denied" on the first error. - -## TTS duration inflation on short lines - -**Observation:** Very short lines (1-2s in step-tts-2) become dramatically longer in stepaudio-2.5-tts. - -Example from the reference project: -- `...你能看到我吗？` (10 chars) -- step-tts-2: 1.24s -- stepaudio-2.5-tts: 2.57s (**+107%**) - -**Cause:** The new model adds a pre-breath, pauses on `...` ellipses, and gives the line emotional weight — all of which lengthens delivery. - -**Not a bug, but have a plan:** -- If your UI has per-line timing (auto-advance, animation sync), re-tune it after migration -- If you want the old pacing, write `instruction: "快速、干脆、不要停顿"` — but this negates a lot of what you're paying for in the new model - -## `stepaudio-2.5-tts` is a "v2 model" for parameter rejection - -**Why the error says "v2 models":** StepFun internally groups `stepaudio-2.5-tts` with their v2 family despite the "2.5" version number. The error message `voice_label is not supported for v2 models` uses this internal grouping, which is confusing. - -Don't pattern-match on the version string. Just know that: -- `stepaudio-2.5-tts` → use `instruction` parameter -- `step-tts-2` → use `voice_label` parameter -- They are NOT API-compatible despite sharing `/v1/audio/speech` - -## ASR "Plan key" vs "Normal key" +## "Plan key" vs "Normal key" — silent auth failure StepFun sells a cheap "Plan" subscription for text models (step_plan endpoint). **Plan keys cannot call audio endpoints.** This silently manifests as 4xx errors that don't mention auth at all. @@ -116,16 +73,25 @@ If you hit auth-shaped failures and your account has a Plan subscription, verify data: {"type":"error","message":"content blocked ..."} ``` -Handle the `error` event type in the SSE stream — don't assume only `delta` and `done` events fire. +Handle the `error` event type in the SSE stream — don't assume only `delta` and `done` events fire. If your code only handles `transcript.text.delta` and `transcript.text.done`, a blocked-content event is silently dropped and the request appears to return empty text with no error surfaced to the caller. -## Pricing opacity for stepaudio-2.5-asr +The bundled `scripts/asr_transcribe.py` handles this correctly — see `_consume_sse()` for the pattern. + +## Pricing opacity As of 2026-04-23, `stepaudio-2.5-asr` is in invitation beta. No public per-minute rate. `step-asr-1.1` baseline is 2.2 元/小时. The invitation PDF mentions "成本直降 80%" implying roughly 0.4 元/小时, but this is not yet on the pricing page. Do not quote a price to a stakeholder without re-verifying at https://platform.stepfun.com/docs/zh/guides/pricing/details. -## TTS text cap: 1000 chars (hard, not soft) +## Empty transcript with no error + +**Symptom:** SSE stream completes normally but `transcript.text.done.text` is empty string. + +**Possible causes:** +1. Audio is silent / pure noise / corrupted +2. Audio language doesn't match the `language` parameter (e.g., sending English audio with `language: zh`) +3. Audio format mismatch (e.g., `format.type: mp3` but actual bytes are wav) -The API rejects >1000 char inputs with a 400 error. Split at sentence boundaries before sending. Non-obvious: when testing "what's the real limit?", avoid highly-repetitive test text — it can appear to succeed at 800 chars but produce strange audio (see the 2026-04 test where 800-char repetitive inputs played back normal audio but the ASR hallucinated 4× replay). +The bundled script falls back to concatenating delta chunks if the `done` event has empty text — but if both are empty, the issue is upstream (the audio itself, not the API). -## Voice cloning — not tested in this skill +## Long-audio timeout behavior -Zero-shot voice cloning (`9.9 元/音色`) is advertised as a headline feature but was not verified in this skill's test pass. If you need voice cloning, check the StepFun docs at https://platform.stepfun.com/docs/zh/api-reference/audio/create-voice and validate on your own data — don't assume the quality claims without a listen test. +The default `urllib`/`requests` timeout is too short for 17+ minute audio. The bundled script uses `timeout=1200` (20 minutes). If you write your own client, set the timeout to at least 2× expected wall clock time (RTF ~100× means 17 min audio takes ~10s wall clock, but TCP retries and network jitter can stretch this). diff --git a/stepfun-tts/scripts/asr_transcribe.py b/daymade-audio/stepfun-asr/scripts/asr_transcribe.py similarity index 100% rename from stepfun-tts/scripts/asr_transcribe.py rename to daymade-audio/stepfun-asr/scripts/asr_transcribe.py diff --git a/daymade-audio/stepfun-tts/.security-scan-passed b/daymade-audio/stepfun-tts/.security-scan-passed new file mode 100644 index 00000000..5339339e --- /dev/null +++ b/daymade-audio/stepfun-tts/.security-scan-passed @@ -0,0 +1,4 @@ +Security scan passed +Scanned at: 2026-04-30T16:45:06.762254 +Tool: gitleaks + pattern-based validation +Content hash: ee9f2151ac3b8e5f198b2a4fadd1e57662b0bfb0e856b4cb27f46a36950bcb23 diff --git a/stepfun-tts/SKILL.md b/daymade-audio/stepfun-tts/SKILL.md similarity index 52% rename from stepfun-tts/SKILL.md rename to daymade-audio/stepfun-tts/SKILL.md index 5b8c71ce..459f719a 100644 --- a/stepfun-tts/SKILL.md +++ b/daymade-audio/stepfun-tts/SKILL.md @@ -1,17 +1,18 @@ --- name: stepfun-tts -description: Generate speech and transcribe audio using StepFun's StepAudio 2.5 family — stepaudio-2.5-tts (Contextual TTS with instruction + inline parentheses) and stepaudio-2.5-asr (SSE endpoint, 32K context, ~100x RTF, handles up to 30-minute audio in a single call). Use when the user wants Chinese/Japanese TTS with emotional/prosody control, needs to transcribe long audio, migrates from step-tts-2 to stepaudio-2.5-tts (voice_label → instruction breaking change), or hits StepFun censorship / endpoint errors. Also triggers on phrases like 阶跃 TTS, StepAudio 合成, 语音合成, 配音, StepFun ASR, 转录, 语音识别, 文本转语音, TTS 升级, 迁移 step-tts-2. If the user's audio task mentions StepFun/阶跃/StepAudio by name, or involves Chinese TTS with情绪/情感 control, use this skill before falling back to generic audio handling. +description: Generate Chinese / Japanese speech with StepFun's stepaudio-2.5-tts — Contextual TTS that replaces step-tts-2's `voice_label` with natural-language `instruction` (≤200 chars) plus inline `()` parentheses for句内 prosody. Use when the user wants emotional / prosody control over voice synthesis (whisper, pause, stress, mood pivot mid-sentence), batch-generates game / app voice lines, migrates from `step-tts-2` (the `voice_label → instruction` breaking change), or hits StepFun's stricter 2.5-era censorship (死/消失/political terms). Triggers on 阶跃 TTS, StepAudio 合成, 语音合成, 配音, 文本转语音, TTS 升级, 迁移 step-tts-2. For transcription with the sibling stepaudio-2.5-asr model, use the stepfun-asr skill instead. --- -# StepFun StepAudio 2.5 — TTS + ASR +# StepFun stepaudio-2.5-tts -Generate Chinese/Japanese speech with `stepaudio-2.5-tts` and transcribe audio with `stepaudio-2.5-asr`. Both models were released in 2026-04 and verified end-to-end on 2026-04-23 (see `references/known_issues.md` for what passed and what didn't). +Generate Chinese / Japanese speech with `stepaudio-2.5-tts` (released 2026-04, verified 2026-04-23). Contextual TTS — emotion and prosody go through natural-language description, not fixed labels. -**Why this skill exists** — StepAudio 2.5 has three non-obvious pitfalls that cost hours if you don't know them: +> Companion: for transcription with `stepaudio-2.5-asr` (the sibling model), use the `stepfun-asr` skill — they share an API key but live on different endpoints with different body shapes. + +**Why this skill exists** — StepAudio 2.5 has two non-obvious pitfalls that cost hours if you don't know them: 1. `stepaudio-2.5-tts` **rejects** `voice_label` (the step-tts-2 way). Emotion/prosody now goes through `instruction` (natural-language description, ≤200 chars) and inline `()` parentheses inside the text itself. -2. `stepaudio-2.5-asr` **does not live on** `/v1/audio/transcriptions`. It's on `/v1/audio/asr/sse` (SSE streaming, JSON body, base64 audio). Using the wrong endpoint returns a misleading `model ... not supported` error that looks identical to "model doesn't exist". -3. Censorship is stricter — anything containing 死 / 消失 / sensitive political terms returns `censorship_block`. Your rewrite options are in `references/migration_from_v2.md`. +2. Censorship is stricter — anything containing 死 / 消失 / sensitive political terms returns `censorship_block`. Your rewrite options are in `references/migration_from_v2.md`. ## Config and auth @@ -25,24 +26,21 @@ mkdir -p "${CLAUDE_PLUGIN_DATA}" && cat > "${CLAUDE_PLUGIN_DATA}/config.json" << EOF ``` -If the user hasn't set a key, ask them to paste it (don't guess / don't use a placeholder). StepFun API keys are available at https://platform.stepfun.com/ → API Keys. +If the user hasn't set a key, ask them to paste it (don't guess / don't use a placeholder). StepFun API keys are available at https://platform.stepfun.com/ → API Keys. **Use a Normal key, not a Plan key** (Plan keys are restricted to text models and silently fail on audio endpoints). ## Common tasks — decision tree -| User wants... | Model | Script | Key detail | -|---|---|---|---| -| Synthesize 1–500 char Chinese with emotion | `stepaudio-2.5-tts` | `scripts/tts_generate.py` | Use `instruction` for mood, `()` for inline prosody | -| Synthesize long text (500–1000 char) | `stepaudio-2.5-tts` | `scripts/tts_generate.py` | 1000 char is the hard cap; split at semantic boundaries above that | -| Batch-generate game/app voice lines | `stepaudio-2.5-tts` | `scripts/tts_generate.py --batch ` | Handle `censorship_block` fallback individually | -| Transcribe short clip (<5 min) | `stepaudio-2.5-asr` | `scripts/asr_transcribe.py` | mp3 → base64 → SSE, parse `transcript.text.done` | -| Transcribe long audio (5–30 min) | `stepaudio-2.5-asr` | `scripts/asr_transcribe.py` | 32K context; single call, no chunking needed | -| A/B compare two TTS models | both | `scripts/ab_compare.sh` | Compares duration/size across two directories | -| Migrate from `step-tts-2` | — | see `references/migration_from_v2.md` | `voice_label.emotion` → `instruction` rewrite + censorship list | +| User wants... | Script | Key detail | +|---|---|---| +| Synthesize 1–500 char Chinese with emotion | `scripts/tts_generate.py` | Use `instruction` for mood, `()` for inline prosody | +| Synthesize long text (500–1000 char) | `scripts/tts_generate.py` | 1000 char is the hard cap; split at semantic boundaries above that | +| Batch-generate game/app voice lines | `scripts/tts_generate.py --batch ` | Handle `censorship_block` fallback individually | +| A/B compare two TTS models | `scripts/ab_compare.sh` | Compares duration/size across two directories | +| Migrate from `step-tts-2` | see `references/migration_from_v2.md` | `voice_label.emotion` → `instruction` rewrite + censorship list | ## Starting points - **Synthesize a single line**: Run `python3 scripts/tts_generate.py --text "你好" --out /tmp/hello.mp3 --instruction "温暖的希望感"`. For fine-grained control read the "Contextual TTS" section below. -- **Transcribe a file**: `python3 scripts/asr_transcribe.py /path/to/audio.mp3`. For >30 min audio, split first. - **A full migration** from `step-tts-2` → `stepaudio-2.5-tts`: read `references/migration_from_v2.md` end-to-end before touching code. It has the `INSTRUCTION_MAP`, the SKIP_CENSORED list pattern, and the output-directory-strategy for non-destructive A/B. ## Contextual TTS — beyond emotion labels @@ -72,31 +70,27 @@ Examples that worked in practice (from 2026-04-23 verification): | Error response | Actual cause | Fix | |---|---|---| -| `"model stepaudio-2.5-asr not supported"` on `/v1/audio/transcriptions` | Wrong endpoint — that endpoint only serves step-asr family | Switch to `/v1/audio/asr/sse` with SSE body (see `scripts/asr_transcribe.py`) | | `"voice_label is not supported for v2 models"` | Sent `voice_label` to `stepaudio-2.5-tts` | Remove `voice_label`; put the same intent into `instruction` as natural language | | `"The content you provided or machine outputted is blocked." type: censorship_block` | Sensitive word (死 / 消失 / etc.) | Rewrite the phrase OR fall back to `step-tts-2` for that specific line (mixed-model is fine) | -| ASR returns N× the expected character count | Hallucination bug on highly-repetitive content | Cross-check with step-asr-1.1; avoid sending audio that repeats the same phrase many times | -| Silent audio truncation (<420 chars input) | Input > 1000 char hard cap | Split at semantic boundaries; don't truncate mid-sentence | +| Silent audio truncation (input > 1000 chars) | Hard cap exceeded | Split at semantic boundaries; don't truncate mid-sentence | More in `references/known_issues.md`. ## When to read references -- `references/api_reference.md` — exact request/response JSON for TTS `/v1/audio/speech` and ASR `/v1/audio/asr/sse`, all fields, event types. Read when writing raw HTTP calls instead of using the bundled scripts. +- `references/api_reference.md` — exact request/response JSON for `/v1/audio/speech`, all fields, error responses. Read when writing raw HTTP calls instead of using the bundled scripts. - `references/migration_from_v2.md` — complete playbook for moving a step-tts-2 project to stepaudio-2.5-tts. Has the emotion→instruction rewrite table, the A/B directory strategy, decision checkpoints, and the 2026-04 speed/quality trade-off data (`stepaudio-2.5-tts` is ~20% slower than step-tts-2; audible prosody improvement). Read before any migration work. -- `references/known_issues.md` — repetition hallucination, censorship patterns, ASR speed cliff (short audio: 2× step-asr, long audio: 5.9×). Read when debugging anomalous output or evaluating whether to adopt. +- `references/known_issues.md` — censorship patterns, TTS duration inflation, v2-family parameter naming gotcha, 1000-char hard cap. Read when debugging anomalous output or evaluating whether to adopt. ## Design invariants (don't break these) 1. **Non-destructive A/B output** — when regenerating a corpus with a new model, write to a parallel directory (`voice/zh_v25/`), never overwrite the production corpus. The migration playbook shows why. 2. **Per-line censorship handling** — if 2/29 lines get `censorship_block`, don't fail the batch. Log the skipped IDs, continue. Mixed-model fallback (step-tts-2 for the skipped 2) is normal. -3. **Always pass through SSE for ASR** — don't try to work around the streaming API with a buffered client. The model emits `transcript.text.delta` events for long audio; collecting only `transcript.text.done` works fine, but rejecting the SSE format entirely doesn't. -4. **Don't duplicate voice_label logic in new code** — any new TTS code targeting stepaudio-2.5-tts should only use `instruction` + inline `()`. Do not write a branch that conditionally emits `voice_label`. +3. **Don't duplicate voice_label logic in new code** — any new TTS code targeting stepaudio-2.5-tts should only use `instruction` + inline `()`. Do not write a branch that conditionally emits `voice_label`. ## Pricing (verified 2026-04-23, volatile) - `stepaudio-2.5-tts` contextual synthesis: ~5.8 元 / 万字符 - Zero-shot voice cloning: ~9.9 元 / 音色 -- `stepaudio-2.5-asr` — pricing tier not yet public (invitation beta); `step-asr-1.1` baseline is 2.2 元/小时 Re-verify at https://platform.stepfun.com/docs/zh/guides/pricing/details before quoting to stakeholders. diff --git a/daymade-audio/stepfun-tts/references/api_reference.md b/daymade-audio/stepfun-tts/references/api_reference.md new file mode 100644 index 00000000..1b119091 --- /dev/null +++ b/daymade-audio/stepfun-tts/references/api_reference.md @@ -0,0 +1,94 @@ +# stepaudio-2.5-tts API Reference + +Exact request/response shapes for `stepaudio-2.5-tts`. Verified 2026-04-23 against the live StepFun API. Read this when you need to call the API by hand (curl, custom HTTP client) instead of using the bundled `scripts/tts_generate.py`. + +## Endpoint + +``` +POST https://api.stepfun.com/v1/audio/speech +Content-Type: application/json +Authorization: Bearer +``` + +## Request body + +```json +{ + "model": "stepaudio-2.5-tts", + "input": "你好，我是蕾格。", + "voice": "shuangkuaijiejie", + "response_format": "mp3", + "speed": 1.0, + "volume": 1.0, + "instruction": "克制的悲伤，语气低沉柔弱" +} +``` + +| Field | Required | Type | Notes | +|---|---|---|---| +| `model` | yes | string | Must be `stepaudio-2.5-tts` | +| `input` | yes | string | ≤1000 chars; can contain inline `(directive)` parentheses | +| `voice` | yes | string | e.g. `shuangkuaijiejie`. Zero-shot clones use the clone's ID | +| `response_format` | yes | string | `mp3` (default), `wav`, or `opus` | +| `speed` | no | float | 0.5-2.0, default 1.0 | +| `volume` | no | float | 0.0-2.0, default 1.0 | +| `instruction` | no | string | Global tone directive, natural language, ≤200 chars | +| `voice_label` | — | — | **DO NOT SEND**. Returns `voice_label is not supported for v2 models`. Belongs to step-tts-2 | + +## Inline directives inside `input` + +Parentheses `()` in the `input` are consumed as TTS control signals, not pronounced. Examples that work: + +- `(停顿一下)` — insert a pause +- `(轻声)` — reduce volume / breathy +- `(加重)` — stress the following word +- `(试探着问)` — apply a tone shift mid-sentence +- `(突然沉下来)` — emotion pivot + +You can mix `instruction` (global tone) with inline `()` (per-phrase micro-control): + +```json +{ + "instruction": "富有情绪弧线的独白", + "input": "(试探着问)你好吗？(开心地)太好了！(突然沉下来)不过...我快要消失了。" +} +``` + +## Response + +On success: binary audio stream in the requested `response_format`. HTTP 200. No JSON wrapper. Save the body directly as `.mp3`/`.wav`/`.opus`. + +## Known error responses + +```json +{"error":{"message":"voice_label is not supported for v2 models","type":"request_params_invalid"}} +``` +→ Remove `voice_label`, use `instruction` instead. + +```json +{"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_block"}} +``` +→ Content triggered censorship. Common triggers: 死, 消失, politically sensitive terms. See `known_issues.md`. + +## Comparison with sibling and legacy endpoints + +| Model | Endpoint | Request format | +|---|---|---| +| `stepaudio-2.5-tts` (this skill) | `/v1/audio/speech` | JSON with `instruction` (no voice_label) | +| `stepaudio-2.5-asr` (sibling, see `stepfun-asr` skill) | `/v1/audio/asr/sse` | JSON + base64 audio + SSE response | +| `step-tts-2` / `step-tts-mini` (legacy) | `/v1/audio/speech` | JSON with `voice_label` | +| `step-asr` / `step-asr-1.1` (legacy) | `/v1/audio/transcriptions` | multipart/form-data | + +Legacy `step-tts-2` still works. It's the baseline in `migration_from_v2.md` and the per-line fallback when `stepaudio-2.5-tts` hits `censorship_block`. + +## Auth and key handling + +- Key header: `Authorization: Bearer ` +- Keys can be retrieved at https://platform.stepfun.com/ → API Keys +- "Plan" keys (cheaper subscription) are **restricted** to text models on `api.stepfun.com/step_plan`. They **cannot** call audio endpoints. Use a "Normal" key for all TTS calls. +- Same key works for both TTS and ASR — no separate scopes + +## Rate / throughput notes (observed, not officially documented) + +- ~400ms sleep between batch requests avoids 429s in practice +- MP3 responses consistently at 128kbps 24kHz mono (TTS default) diff --git a/daymade-audio/stepfun-tts/references/known_issues.md b/daymade-audio/stepfun-tts/references/known_issues.md new file mode 100644 index 00000000..886cdeca --- /dev/null +++ b/daymade-audio/stepfun-tts/references/known_issues.md @@ -0,0 +1,62 @@ +# stepaudio-2.5-tts — Known Issues and Non-Obvious Behavior + +Collected from end-to-end testing 2026-04-23. These are things that burned real time to discover; they are not in the official docs. + +## Stricter content censorship than step-tts-2 + +**Symptom:** `stepaudio-2.5-tts` returns `{"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_block"}}` for content that step-tts-2 happily synthesized. + +**Observed triggers:** +- 死 (die/dead) in any context, even negation +- 消失 (disappear / vanish) +- Combinations with emotional context: "我快要...消失了" +- Politically sensitive terms (standard CN content rules) + +**Key insight:** Rewriting negations doesn't help — "我没有死" blocks as readily as "我死了". The classifier isn't doing deep semantic parsing. + +**Response strategies** (pick per line): +1. Rewrite: "RAG 已死" → "这个技术过时了" +2. Fallback: keep step-tts-2 for the 2-5% of lines that block +3. Whitelist: contact StepFun BD (worth it at >5% blockage) + +See `migration_from_v2.md` for the full blocking→fallback workflow. + +## TTS duration inflation on short lines + +**Observation:** Very short lines (1-2s in step-tts-2) become dramatically longer in stepaudio-2.5-tts. + +Example from the reference project: +- `...你能看到我吗？` (10 chars) +- step-tts-2: 1.24s +- stepaudio-2.5-tts: 2.57s (**+107%**) + +**Cause:** The new model adds a pre-breath, pauses on `...` ellipses, and gives the line emotional weight — all of which lengthens delivery. + +**Not a bug, but have a plan:** +- If your UI has per-line timing (auto-advance, animation sync), re-tune it after migration +- If you want the old pacing, write `instruction: "快速、干脆、不要停顿"` — but this negates a lot of what you're paying for in the new model + +## `stepaudio-2.5-tts` is a "v2 model" for parameter rejection + +**Why the error says "v2 models":** StepFun internally groups `stepaudio-2.5-tts` with their v2 family despite the "2.5" version number. The error message `voice_label is not supported for v2 models` uses this internal grouping, which is confusing. + +Don't pattern-match on the version string. Just know that: +- `stepaudio-2.5-tts` → use `instruction` parameter +- `step-tts-2` → use `voice_label` parameter +- They are NOT API-compatible despite sharing `/v1/audio/speech` + +## TTS text cap: 1000 chars (hard, not soft) + +The API rejects >1000 char inputs with a 400 error. Split at sentence boundaries before sending. + +Non-obvious caveat when probing the limit: don't use highly-repetitive test text. The TTS itself accepts repetitive 800-char inputs and produces normal audio, but if you then transcribe that audio with `stepaudio-2.5-asr` for round-trip verification, the ASR can hallucinate 3-4× character expansion (a known ASR-side bug, see the `stepfun-asr` skill's `known_issues.md`). Use varied real-world text for cap-probing tests. + +## Voice cloning — not tested in this skill + +Zero-shot voice cloning (`9.9 元/音色`) is advertised as a headline feature but was not verified in this skill's test pass. If you need voice cloning, check the StepFun docs at https://platform.stepfun.com/docs/zh/api-reference/audio/create-voice and validate on your own data — don't assume the quality claims without a listen test. + +## "Plan key" vs "Normal key" — silent audio failure + +StepFun sells a cheap "Plan" subscription for text models (step_plan endpoint). **Plan keys cannot call audio endpoints.** This silently manifests as 4xx errors that don't mention auth at all. + +If you hit auth-shaped failures and your account has a Plan subscription, verify you're using a Normal key (different value, obtained separately in the StepFun console under the same "API Keys" page). diff --git a/stepfun-tts/references/migration_from_v2.md b/daymade-audio/stepfun-tts/references/migration_from_v2.md similarity index 100% rename from stepfun-tts/references/migration_from_v2.md rename to daymade-audio/stepfun-tts/references/migration_from_v2.md diff --git a/stepfun-tts/scripts/ab_compare.sh b/daymade-audio/stepfun-tts/scripts/ab_compare.sh similarity index 100% rename from stepfun-tts/scripts/ab_compare.sh rename to daymade-audio/stepfun-tts/scripts/ab_compare.sh diff --git a/stepfun-tts/scripts/tts_generate.py b/daymade-audio/stepfun-tts/scripts/tts_generate.py similarity index 100% rename from stepfun-tts/scripts/tts_generate.py rename to daymade-audio/stepfun-tts/scripts/tts_generate.py diff --git a/transcript-fixer/.gitignore b/daymade-audio/transcript-fixer/.gitignore similarity index 100% rename from transcript-fixer/.gitignore rename to daymade-audio/transcript-fixer/.gitignore diff --git a/transcript-fixer/SKILL.md b/daymade-audio/transcript-fixer/SKILL.md similarity index 100% rename from transcript-fixer/SKILL.md rename to daymade-audio/transcript-fixer/SKILL.md diff --git a/transcript-fixer/references/architecture.md b/daymade-audio/transcript-fixer/references/architecture.md similarity index 100% rename from transcript-fixer/references/architecture.md rename to daymade-audio/transcript-fixer/references/architecture.md diff --git a/transcript-fixer/references/best_practices.md b/daymade-audio/transcript-fixer/references/best_practices.md similarity index 100% rename from transcript-fixer/references/best_practices.md rename to daymade-audio/transcript-fixer/references/best_practices.md diff --git a/transcript-fixer/references/database_schema.md b/daymade-audio/transcript-fixer/references/database_schema.md similarity index 100% rename from transcript-fixer/references/database_schema.md rename to daymade-audio/transcript-fixer/references/database_schema.md diff --git a/transcript-fixer/references/dictionary_guide.md b/daymade-audio/transcript-fixer/references/dictionary_guide.md similarity index 100% rename from transcript-fixer/references/dictionary_guide.md rename to daymade-audio/transcript-fixer/references/dictionary_guide.md diff --git a/transcript-fixer/references/example_session.md b/daymade-audio/transcript-fixer/references/example_session.md similarity index 100% rename from transcript-fixer/references/example_session.md rename to daymade-audio/transcript-fixer/references/example_session.md diff --git a/transcript-fixer/references/false_positive_guide.md b/daymade-audio/transcript-fixer/references/false_positive_guide.md similarity index 100% rename from transcript-fixer/references/false_positive_guide.md rename to daymade-audio/transcript-fixer/references/false_positive_guide.md diff --git a/transcript-fixer/references/file_formats.md b/daymade-audio/transcript-fixer/references/file_formats.md similarity index 100% rename from transcript-fixer/references/file_formats.md rename to daymade-audio/transcript-fixer/references/file_formats.md diff --git a/transcript-fixer/references/glm_api_setup.md b/daymade-audio/transcript-fixer/references/glm_api_setup.md similarity index 100% rename from transcript-fixer/references/glm_api_setup.md rename to daymade-audio/transcript-fixer/references/glm_api_setup.md diff --git a/transcript-fixer/references/installation_setup.md b/daymade-audio/transcript-fixer/references/installation_setup.md similarity index 100% rename from transcript-fixer/references/installation_setup.md rename to daymade-audio/transcript-fixer/references/installation_setup.md diff --git a/transcript-fixer/references/iteration_workflow.md b/daymade-audio/transcript-fixer/references/iteration_workflow.md similarity index 100% rename from transcript-fixer/references/iteration_workflow.md rename to daymade-audio/transcript-fixer/references/iteration_workflow.md diff --git a/transcript-fixer/references/quick_reference.md b/daymade-audio/transcript-fixer/references/quick_reference.md similarity index 100% rename from transcript-fixer/references/quick_reference.md rename to daymade-audio/transcript-fixer/references/quick_reference.md diff --git a/transcript-fixer/references/script_parameters.md b/daymade-audio/transcript-fixer/references/script_parameters.md similarity index 100% rename from transcript-fixer/references/script_parameters.md rename to daymade-audio/transcript-fixer/references/script_parameters.md diff --git a/transcript-fixer/references/sql_queries.md b/daymade-audio/transcript-fixer/references/sql_queries.md similarity index 100% rename from transcript-fixer/references/sql_queries.md rename to daymade-audio/transcript-fixer/references/sql_queries.md diff --git a/transcript-fixer/references/team_collaboration.md b/daymade-audio/transcript-fixer/references/team_collaboration.md similarity index 100% rename from transcript-fixer/references/team_collaboration.md rename to daymade-audio/transcript-fixer/references/team_collaboration.md diff --git a/transcript-fixer/references/troubleshooting.md b/daymade-audio/transcript-fixer/references/troubleshooting.md similarity index 100% rename from transcript-fixer/references/troubleshooting.md rename to daymade-audio/transcript-fixer/references/troubleshooting.md diff --git a/transcript-fixer/references/workflow_guide.md b/daymade-audio/transcript-fixer/references/workflow_guide.md similarity index 100% rename from transcript-fixer/references/workflow_guide.md rename to daymade-audio/transcript-fixer/references/workflow_guide.md diff --git a/transcript-fixer/requirements.txt b/daymade-audio/transcript-fixer/requirements.txt similarity index 100% rename from transcript-fixer/requirements.txt rename to daymade-audio/transcript-fixer/requirements.txt diff --git a/transcript-fixer/scripts/__init__.py b/daymade-audio/transcript-fixer/scripts/__init__.py similarity index 100% rename from transcript-fixer/scripts/__init__.py rename to daymade-audio/transcript-fixer/scripts/__init__.py diff --git a/transcript-fixer/scripts/check_type_hints.py b/daymade-audio/transcript-fixer/scripts/check_type_hints.py similarity index 100% rename from transcript-fixer/scripts/check_type_hints.py rename to daymade-audio/transcript-fixer/scripts/check_type_hints.py diff --git a/transcript-fixer/scripts/cli/__init__.py b/daymade-audio/transcript-fixer/scripts/cli/__init__.py similarity index 100% rename from transcript-fixer/scripts/cli/__init__.py rename to daymade-audio/transcript-fixer/scripts/cli/__init__.py diff --git a/transcript-fixer/scripts/cli/argument_parser.py b/daymade-audio/transcript-fixer/scripts/cli/argument_parser.py similarity index 100% rename from transcript-fixer/scripts/cli/argument_parser.py rename to daymade-audio/transcript-fixer/scripts/cli/argument_parser.py diff --git a/transcript-fixer/scripts/cli/commands.py b/daymade-audio/transcript-fixer/scripts/cli/commands.py similarity index 100% rename from transcript-fixer/scripts/cli/commands.py rename to daymade-audio/transcript-fixer/scripts/cli/commands.py diff --git a/transcript-fixer/scripts/core/__init__.py b/daymade-audio/transcript-fixer/scripts/core/__init__.py similarity index 100% rename from transcript-fixer/scripts/core/__init__.py rename to daymade-audio/transcript-fixer/scripts/core/__init__.py diff --git a/transcript-fixer/scripts/core/ai_processor.py b/daymade-audio/transcript-fixer/scripts/core/ai_processor.py similarity index 100% rename from transcript-fixer/scripts/core/ai_processor.py rename to daymade-audio/transcript-fixer/scripts/core/ai_processor.py diff --git a/transcript-fixer/scripts/core/ai_processor_async.py b/daymade-audio/transcript-fixer/scripts/core/ai_processor_async.py similarity index 100% rename from transcript-fixer/scripts/core/ai_processor_async.py rename to daymade-audio/transcript-fixer/scripts/core/ai_processor_async.py diff --git a/transcript-fixer/scripts/core/change_extractor.py b/daymade-audio/transcript-fixer/scripts/core/change_extractor.py similarity index 100% rename from transcript-fixer/scripts/core/change_extractor.py rename to daymade-audio/transcript-fixer/scripts/core/change_extractor.py diff --git a/transcript-fixer/scripts/core/connection_pool.py b/daymade-audio/transcript-fixer/scripts/core/connection_pool.py similarity index 100% rename from transcript-fixer/scripts/core/connection_pool.py rename to daymade-audio/transcript-fixer/scripts/core/connection_pool.py diff --git a/transcript-fixer/scripts/core/correction_repository.py b/daymade-audio/transcript-fixer/scripts/core/correction_repository.py similarity index 100% rename from transcript-fixer/scripts/core/correction_repository.py rename to daymade-audio/transcript-fixer/scripts/core/correction_repository.py diff --git a/transcript-fixer/scripts/core/correction_service.py b/daymade-audio/transcript-fixer/scripts/core/correction_service.py similarity index 100% rename from transcript-fixer/scripts/core/correction_service.py rename to daymade-audio/transcript-fixer/scripts/core/correction_service.py diff --git a/transcript-fixer/scripts/core/dictionary_processor.py b/daymade-audio/transcript-fixer/scripts/core/dictionary_processor.py similarity index 100% rename from transcript-fixer/scripts/core/dictionary_processor.py rename to daymade-audio/transcript-fixer/scripts/core/dictionary_processor.py diff --git a/transcript-fixer/scripts/core/learning_engine.py b/daymade-audio/transcript-fixer/scripts/core/learning_engine.py similarity index 100% rename from transcript-fixer/scripts/core/learning_engine.py rename to daymade-audio/transcript-fixer/scripts/core/learning_engine.py diff --git a/transcript-fixer/scripts/core/schema.sql b/daymade-audio/transcript-fixer/scripts/core/schema.sql similarity index 100% rename from transcript-fixer/scripts/core/schema.sql rename to daymade-audio/transcript-fixer/scripts/core/schema.sql diff --git a/transcript-fixer/scripts/ensure_deps.py b/daymade-audio/transcript-fixer/scripts/ensure_deps.py similarity index 100% rename from transcript-fixer/scripts/ensure_deps.py rename to daymade-audio/transcript-fixer/scripts/ensure_deps.py diff --git a/transcript-fixer/scripts/examples/bulk_import.py b/daymade-audio/transcript-fixer/scripts/examples/bulk_import.py similarity index 100% rename from transcript-fixer/scripts/examples/bulk_import.py rename to daymade-audio/transcript-fixer/scripts/examples/bulk_import.py diff --git a/transcript-fixer/scripts/fix_transcript_enhanced.py b/daymade-audio/transcript-fixer/scripts/fix_transcript_enhanced.py similarity index 100% rename from transcript-fixer/scripts/fix_transcript_enhanced.py rename to daymade-audio/transcript-fixer/scripts/fix_transcript_enhanced.py diff --git a/transcript-fixer/scripts/fix_transcript_timestamps.py b/daymade-audio/transcript-fixer/scripts/fix_transcript_timestamps.py similarity index 100% rename from transcript-fixer/scripts/fix_transcript_timestamps.py rename to daymade-audio/transcript-fixer/scripts/fix_transcript_timestamps.py diff --git a/transcript-fixer/scripts/fix_transcription.py b/daymade-audio/transcript-fixer/scripts/fix_transcription.py similarity index 100% rename from transcript-fixer/scripts/fix_transcription.py rename to daymade-audio/transcript-fixer/scripts/fix_transcription.py diff --git a/transcript-fixer/scripts/generate_word_diff.py b/daymade-audio/transcript-fixer/scripts/generate_word_diff.py similarity index 100% rename from transcript-fixer/scripts/generate_word_diff.py rename to daymade-audio/transcript-fixer/scripts/generate_word_diff.py diff --git a/transcript-fixer/scripts/split_transcript_sections.py b/daymade-audio/transcript-fixer/scripts/split_transcript_sections.py similarity index 100% rename from transcript-fixer/scripts/split_transcript_sections.py rename to daymade-audio/transcript-fixer/scripts/split_transcript_sections.py diff --git a/transcript-fixer/scripts/tests/__init__.py b/daymade-audio/transcript-fixer/scripts/tests/__init__.py similarity index 100% rename from transcript-fixer/scripts/tests/__init__.py rename to daymade-audio/transcript-fixer/scripts/tests/__init__.py diff --git a/transcript-fixer/scripts/tests/test_audit_log_retention.py b/daymade-audio/transcript-fixer/scripts/tests/test_audit_log_retention.py similarity index 100% rename from transcript-fixer/scripts/tests/test_audit_log_retention.py rename to daymade-audio/transcript-fixer/scripts/tests/test_audit_log_retention.py diff --git a/transcript-fixer/scripts/tests/test_common_words_safety.py b/daymade-audio/transcript-fixer/scripts/tests/test_common_words_safety.py similarity index 100% rename from transcript-fixer/scripts/tests/test_common_words_safety.py rename to daymade-audio/transcript-fixer/scripts/tests/test_common_words_safety.py diff --git a/transcript-fixer/scripts/tests/test_connection_pool.py b/daymade-audio/transcript-fixer/scripts/tests/test_connection_pool.py similarity index 100% rename from transcript-fixer/scripts/tests/test_connection_pool.py rename to daymade-audio/transcript-fixer/scripts/tests/test_connection_pool.py diff --git a/transcript-fixer/scripts/tests/test_correction_service.py b/daymade-audio/transcript-fixer/scripts/tests/test_correction_service.py similarity index 100% rename from transcript-fixer/scripts/tests/test_correction_service.py rename to daymade-audio/transcript-fixer/scripts/tests/test_correction_service.py diff --git a/transcript-fixer/scripts/tests/test_domain_validator.py b/daymade-audio/transcript-fixer/scripts/tests/test_domain_validator.py similarity index 100% rename from transcript-fixer/scripts/tests/test_domain_validator.py rename to daymade-audio/transcript-fixer/scripts/tests/test_domain_validator.py diff --git a/transcript-fixer/scripts/tests/test_error_recovery.py b/daymade-audio/transcript-fixer/scripts/tests/test_error_recovery.py similarity index 100% rename from transcript-fixer/scripts/tests/test_error_recovery.py rename to daymade-audio/transcript-fixer/scripts/tests/test_error_recovery.py diff --git a/transcript-fixer/scripts/tests/test_fix_transcript_timestamps.py b/daymade-audio/transcript-fixer/scripts/tests/test_fix_transcript_timestamps.py similarity index 100% rename from transcript-fixer/scripts/tests/test_fix_transcript_timestamps.py rename to daymade-audio/transcript-fixer/scripts/tests/test_fix_transcript_timestamps.py diff --git a/transcript-fixer/scripts/tests/test_learning_engine.py b/daymade-audio/transcript-fixer/scripts/tests/test_learning_engine.py similarity index 100% rename from transcript-fixer/scripts/tests/test_learning_engine.py rename to daymade-audio/transcript-fixer/scripts/tests/test_learning_engine.py diff --git a/transcript-fixer/scripts/tests/test_path_validator.py b/daymade-audio/transcript-fixer/scripts/tests/test_path_validator.py similarity index 100% rename from transcript-fixer/scripts/tests/test_path_validator.py rename to daymade-audio/transcript-fixer/scripts/tests/test_path_validator.py diff --git a/transcript-fixer/scripts/tests/test_split_transcript_sections.py b/daymade-audio/transcript-fixer/scripts/tests/test_split_transcript_sections.py similarity index 100% rename from transcript-fixer/scripts/tests/test_split_transcript_sections.py rename to daymade-audio/transcript-fixer/scripts/tests/test_split_transcript_sections.py diff --git a/transcript-fixer/scripts/utils/__init__.py b/daymade-audio/transcript-fixer/scripts/utils/__init__.py similarity index 100% rename from transcript-fixer/scripts/utils/__init__.py rename to daymade-audio/transcript-fixer/scripts/utils/__init__.py diff --git a/transcript-fixer/scripts/utils/audit_log_retention.py b/daymade-audio/transcript-fixer/scripts/utils/audit_log_retention.py similarity index 100% rename from transcript-fixer/scripts/utils/audit_log_retention.py rename to daymade-audio/transcript-fixer/scripts/utils/audit_log_retention.py diff --git a/transcript-fixer/scripts/utils/common_words.py b/daymade-audio/transcript-fixer/scripts/utils/common_words.py similarity index 100% rename from transcript-fixer/scripts/utils/common_words.py rename to daymade-audio/transcript-fixer/scripts/utils/common_words.py diff --git a/transcript-fixer/scripts/utils/concurrency_manager.py b/daymade-audio/transcript-fixer/scripts/utils/concurrency_manager.py similarity index 100% rename from transcript-fixer/scripts/utils/concurrency_manager.py rename to daymade-audio/transcript-fixer/scripts/utils/concurrency_manager.py diff --git a/transcript-fixer/scripts/utils/config.py b/daymade-audio/transcript-fixer/scripts/utils/config.py similarity index 100% rename from transcript-fixer/scripts/utils/config.py rename to daymade-audio/transcript-fixer/scripts/utils/config.py diff --git a/transcript-fixer/scripts/utils/database_migration.py b/daymade-audio/transcript-fixer/scripts/utils/database_migration.py similarity index 100% rename from transcript-fixer/scripts/utils/database_migration.py rename to daymade-audio/transcript-fixer/scripts/utils/database_migration.py diff --git a/transcript-fixer/scripts/utils/db_migrations_cli.py b/daymade-audio/transcript-fixer/scripts/utils/db_migrations_cli.py similarity index 100% rename from transcript-fixer/scripts/utils/db_migrations_cli.py rename to daymade-audio/transcript-fixer/scripts/utils/db_migrations_cli.py diff --git a/transcript-fixer/scripts/utils/diff_formats/__init__.py b/daymade-audio/transcript-fixer/scripts/utils/diff_formats/__init__.py similarity index 100% rename from transcript-fixer/scripts/utils/diff_formats/__init__.py rename to daymade-audio/transcript-fixer/scripts/utils/diff_formats/__init__.py diff --git a/transcript-fixer/scripts/utils/diff_formats/change_extractor.py b/daymade-audio/transcript-fixer/scripts/utils/diff_formats/change_extractor.py similarity index 100% rename from transcript-fixer/scripts/utils/diff_formats/change_extractor.py rename to daymade-audio/transcript-fixer/scripts/utils/diff_formats/change_extractor.py diff --git a/transcript-fixer/scripts/utils/diff_formats/html_format.py b/daymade-audio/transcript-fixer/scripts/utils/diff_formats/html_format.py similarity index 100% rename from transcript-fixer/scripts/utils/diff_formats/html_format.py rename to daymade-audio/transcript-fixer/scripts/utils/diff_formats/html_format.py diff --git a/transcript-fixer/scripts/utils/diff_formats/inline_format.py b/daymade-audio/transcript-fixer/scripts/utils/diff_formats/inline_format.py similarity index 100% rename from transcript-fixer/scripts/utils/diff_formats/inline_format.py rename to daymade-audio/transcript-fixer/scripts/utils/diff_formats/inline_format.py diff --git a/transcript-fixer/scripts/utils/diff_formats/markdown_format.py b/daymade-audio/transcript-fixer/scripts/utils/diff_formats/markdown_format.py similarity index 100% rename from transcript-fixer/scripts/utils/diff_formats/markdown_format.py rename to daymade-audio/transcript-fixer/scripts/utils/diff_formats/markdown_format.py diff --git a/transcript-fixer/scripts/utils/diff_formats/text_splitter.py b/daymade-audio/transcript-fixer/scripts/utils/diff_formats/text_splitter.py similarity index 100% rename from transcript-fixer/scripts/utils/diff_formats/text_splitter.py rename to daymade-audio/transcript-fixer/scripts/utils/diff_formats/text_splitter.py diff --git a/transcript-fixer/scripts/utils/diff_formats/unified_format.py b/daymade-audio/transcript-fixer/scripts/utils/diff_formats/unified_format.py similarity index 100% rename from transcript-fixer/scripts/utils/diff_formats/unified_format.py rename to daymade-audio/transcript-fixer/scripts/utils/diff_formats/unified_format.py diff --git a/transcript-fixer/scripts/utils/diff_generator.py b/daymade-audio/transcript-fixer/scripts/utils/diff_generator.py similarity index 100% rename from transcript-fixer/scripts/utils/diff_generator.py rename to daymade-audio/transcript-fixer/scripts/utils/diff_generator.py diff --git a/transcript-fixer/scripts/utils/domain_validator.py b/daymade-audio/transcript-fixer/scripts/utils/domain_validator.py similarity index 100% rename from transcript-fixer/scripts/utils/domain_validator.py rename to daymade-audio/transcript-fixer/scripts/utils/domain_validator.py diff --git a/transcript-fixer/scripts/utils/health_check.py b/daymade-audio/transcript-fixer/scripts/utils/health_check.py similarity index 100% rename from transcript-fixer/scripts/utils/health_check.py rename to daymade-audio/transcript-fixer/scripts/utils/health_check.py diff --git a/transcript-fixer/scripts/utils/logging_config.py b/daymade-audio/transcript-fixer/scripts/utils/logging_config.py similarity index 100% rename from transcript-fixer/scripts/utils/logging_config.py rename to daymade-audio/transcript-fixer/scripts/utils/logging_config.py diff --git a/transcript-fixer/scripts/utils/metrics.py b/daymade-audio/transcript-fixer/scripts/utils/metrics.py similarity index 100% rename from transcript-fixer/scripts/utils/metrics.py rename to daymade-audio/transcript-fixer/scripts/utils/metrics.py diff --git a/transcript-fixer/scripts/utils/migrations.py b/daymade-audio/transcript-fixer/scripts/utils/migrations.py similarity index 100% rename from transcript-fixer/scripts/utils/migrations.py rename to daymade-audio/transcript-fixer/scripts/utils/migrations.py diff --git a/transcript-fixer/scripts/utils/path_validator.py b/daymade-audio/transcript-fixer/scripts/utils/path_validator.py similarity index 100% rename from transcript-fixer/scripts/utils/path_validator.py rename to daymade-audio/transcript-fixer/scripts/utils/path_validator.py diff --git a/transcript-fixer/scripts/utils/rate_limiter.py b/daymade-audio/transcript-fixer/scripts/utils/rate_limiter.py similarity index 100% rename from transcript-fixer/scripts/utils/rate_limiter.py rename to daymade-audio/transcript-fixer/scripts/utils/rate_limiter.py diff --git a/transcript-fixer/scripts/utils/retry_logic.py b/daymade-audio/transcript-fixer/scripts/utils/retry_logic.py similarity index 100% rename from transcript-fixer/scripts/utils/retry_logic.py rename to daymade-audio/transcript-fixer/scripts/utils/retry_logic.py diff --git a/transcript-fixer/scripts/utils/security.py b/daymade-audio/transcript-fixer/scripts/utils/security.py similarity index 100% rename from transcript-fixer/scripts/utils/security.py rename to daymade-audio/transcript-fixer/scripts/utils/security.py diff --git a/transcript-fixer/scripts/utils/validation.py b/daymade-audio/transcript-fixer/scripts/utils/validation.py similarity index 100% rename from transcript-fixer/scripts/utils/validation.py rename to daymade-audio/transcript-fixer/scripts/utils/validation.py diff --git a/stepfun-tts/.security-scan-passed b/stepfun-tts/.security-scan-passed deleted file mode 100644 index dcc047af..00000000 --- a/stepfun-tts/.security-scan-passed +++ /dev/null @@ -1,4 +0,0 @@ -Security scan passed -Scanned at: 2026-04-26T21:45:20.824609 -Tool: gitleaks + pattern-based validation -Content hash: ed5288f648ffe4e0cdd3bcb844dd05f510dc86316ce4c8b4b9afd9c6ab95a1cb diff --git a/stepfun-tts/references/api_reference.md b/stepfun-tts/references/api_reference.md deleted file mode 100644 index aa4599cf..00000000 --- a/stepfun-tts/references/api_reference.md +++ /dev/null @@ -1,178 +0,0 @@ -# StepAudio 2.5 API Reference - -Exact request/response shapes for `stepaudio-2.5-tts` and `stepaudio-2.5-asr`. Verified 2026-04-23 against the live StepFun API. Read this when you need to call the API by hand (curl, custom HTTP client) instead of using the bundled scripts. - -## TTS — `stepaudio-2.5-tts` - -### Endpoint - -``` -POST https://api.stepfun.com/v1/audio/speech -Content-Type: application/json -Authorization: Bearer -``` - -### Request body - -```json -{ - "model": "stepaudio-2.5-tts", - "input": "你好，我是蕾格。", - "voice": "shuangkuaijiejie", - "response_format": "mp3", - "speed": 1.0, - "volume": 1.0, - "instruction": "克制的悲伤，语气低沉柔弱" -} -``` - -| Field | Required | Type | Notes | -|---|---|---|---| -| `model` | yes | string | Must be `stepaudio-2.5-tts` | -| `input` | yes | string | ≤1000 chars; can contain inline `(directive)` parentheses | -| `voice` | yes | string | e.g. `shuangkuaijiejie`. Zero-shot clones use the clone's ID | -| `response_format` | yes | string | `mp3` (default), `wav`, or `opus` | -| `speed` | no | float | 0.5-2.0, default 1.0 | -| `volume` | no | float | 0.0-2.0, default 1.0 | -| `instruction` | no | string | Global tone directive, natural language, ≤200 chars | -| `voice_label` | — | — | **DO NOT SEND**. Returns `voice_label is not supported for v2 models`. Belongs to step-tts-2 | - -### Inline directives inside `input` - -Parentheses `()` in the `input` are consumed as TTS control signals, not pronounced. Examples that work: - -- `(停顿一下)` — insert a pause -- `(轻声)` — reduce volume / breathy -- `(加重)` — stress the following word -- `(试探着问)` — apply a tone shift mid-sentence -- `(突然沉下来)` — emotion pivot - -You can mix `instruction` (global tone) with inline `()` (per-phrase micro-control): - -```json -{ - "instruction": "富有情绪弧线的独白", - "input": "(试探着问)你好吗？(开心地)太好了！(突然沉下来)不过...我快要消失了。" -} -``` - -### Response - -On success: binary audio stream in the requested `response_format`. HTTP 200. No JSON wrapper. Save the body directly as `.mp3`/`.wav`/`.opus`. - -### Known error responses - -```json -{"error":{"message":"voice_label is not supported for v2 models","type":"request_params_invalid"}} -``` -→ Remove `voice_label`, use `instruction` instead. - -```json -{"error":{"message":"The content you provided or machine outputted is blocked.","type":"censorship_block"}} -``` -→ Content triggered censorship. Common triggers: 死, 消失, politically sensitive terms. See `known_issues.md`. - -## ASR — `stepaudio-2.5-asr` - -### Endpoint (NOT the one you'd guess) - -``` -POST https://api.stepfun.com/v1/audio/asr/sse -Content-Type: application/json -Accept: text/event-stream -Authorization: Bearer -``` - -**Do NOT** send `stepaudio-2.5-asr` to `/v1/audio/transcriptions` — that endpoint only serves the older `step-asr` / `step-asr-1.1` family, and returns a misleading `model stepaudio-2.5-asr not supported` which looks identical to a permission/whitelist error. See `known_issues.md` for the full diagnostic trail. - -### Request body - -```json -{ - "audio": { - "data": "", - "input": { - "transcription": { - "language": "zh", - "model": "stepaudio-2.5-asr", - "enable_itn": true - }, - "format": { - "type": "mp3" - } - } - } -} -``` - -| Path | Required | Type | Notes | -|---|---|---|---| -| `audio.data` | yes | string | base64-encoded audio bytes. Accepts mp3, wav, ogg, opus (in ogg container), pcm | -| `audio.input.transcription.language` | yes | string | `zh` or `en`. Dialects and Japanese are not officially supported | -| `audio.input.transcription.model` | yes | string | Must be `stepaudio-2.5-asr` | -| `audio.input.transcription.enable_itn` | no | bool | Inverse text normalization (数字→words). Default true | -| `audio.input.format.type` | yes | string | `mp3` / `wav` / `ogg` / `pcm` | -| `audio.input.format.rate` | pcm only | int | Sample rate (required for raw PCM) | -| `audio.input.format.channel` | pcm only | int | Channel count (required for raw PCM) | -| `audio.input.format.bits` | optional | int | Sample depth, default 16 | - -### Response — SSE stream - -The response is a Server-Sent Events stream. Each line is either empty or starts with `data: `. Three event types: - -``` -data: {"type":"transcript.text.delta","meta":{...},"delta":"你好，"} - -data: {"type":"transcript.text.delta","meta":{...},"delta":"我是蕾格。"} - -data: {"type":"transcript.text.done","meta":{...},"text":"你好，我是蕾格。","usage":{"type":"tokens","input_tokens":69,"input_token_details":{"text_tokens":69,"audio_tokens":0},"output_tokens":9,"total_tokens":78}} -``` - -| Event type | Meaning | How to handle | -|---|---|---| -| `transcript.text.delta` | Incremental piece of the transcription | Concatenate for progressive UI; optional if you only need final text | -| `transcript.text.done` | Final, full transcription + usage | Take `text` as the authoritative result. Also contains `usage` for billing/telemetry | -| `error` | Server-side error mid-stream | Abort and propagate `message` to the caller | - -### Capacity - -- 32K context window -- Audio ≤ 30 min can be sent in a single call -- No client-side chunking needed for long audio (unlike step-asr) -- RTF 85-101× on Chinese speech verified 2026-04-23 - -### Known error responses - -```json -{"error":{"message":"model stepaudio-2.5-asr not supported","type":"request_params_invalid"}} -``` -→ Wrong endpoint. Switch from `/v1/audio/transcriptions` to `/v1/audio/asr/sse`. - -``` -data: {"type":"error","message":"content blocked ..."} -``` -→ Content censorship (rare on ASR). Same triggers as TTS. - -## Comparison with legacy endpoints (for reference) - -| Model | Endpoint | Request format | -|---|---|---| -| `step-tts-2` / `step-tts-mini` | `/v1/audio/speech` | JSON with `voice_label` | -| `stepaudio-2.5-tts` | `/v1/audio/speech` | JSON with `instruction` (no voice_label) | -| `step-asr` / `step-asr-1.1` | `/v1/audio/transcriptions` | multipart/form-data | -| `stepaudio-2.5-asr` | `/v1/audio/asr/sse` | JSON + base64 audio + SSE response | - -Legacy endpoints (`step-*`) still work. They're the baseline in `references/migration_from_v2.md` and the fallback choice when `stepaudio-2.5-*` hits `censorship_block` or the 2.5 ASR repetition-hallucination edge case. - -## Auth and key handling - -- Key header: `Authorization: Bearer ` -- Keys can be retrieved at https://platform.stepfun.com/ → API Keys -- "Plan" keys (cheaper subscription) are **restricted** to text models on `api.stepfun.com/step_plan`. They **cannot** call audio endpoints. Use a "Normal" key for all TTS/ASR calls. -- Same key works for both TTS and ASR — no separate scopes - -## Rate / throughput notes (observed, not officially documented) - -- ~400ms sleep between batch requests avoids 429s in practice -- Long audio ASR (17 min) has succeeded with `timeout=1200` -- MP3 responses consistently at 128kbps 24kHz mono (TTS default) From 3e999df00b9f72f9d284ebd108e401463b7ad13f Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 10 May 2026 02:36:25 +0800 Subject: [PATCH 135/137] docs: update README badges and skill counts for v1.52.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sync badge counts (skills 51→52, version 1.51.0→1.52.0) and add stepfun-asr skill entry in both English and Chinese READMEs. Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 54 +++++++++++++++++++++++++++++++++++-------------- README.zh-CN.md | 54 +++++++++++++++++++++++++++++++++++-------------- 2 files changed, 78 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index 03153866..9bff4613 100644 --- a/README.md +++ b/README.md @@ -6,15 +6,15 @@ [![简体中文](https://img.shields.io/badge/语言-简体中文-red)](./README.zh-CN.md) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Skills](https://img.shields.io/badge/skills-51-blue.svg)](https://github.com/daymade/claude-code-skills) -[![Version](https://img.shields.io/badge/version-1.51.0-green.svg)](https://github.com/daymade/claude-code-skills) +[![Skills](https://img.shields.io/badge/skills-52-blue.svg)](https://github.com/daymade/claude-code-skills) +[![Version](https://img.shields.io/badge/version-1.52.0-green.svg)](https://github.com/daymade/claude-code-skills) [![Claude Code](https://img.shields.io/badge/Claude%20Code-2.0.13+-purple.svg)](https://claude.com/code) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/daymade/claude-code-skills/graphs/commit-activity) -Professional Claude Code skills marketplace featuring 51 production-ready skills for enhanced development workflows. +Professional Claude Code skills marketplace featuring 52 production-ready skills for enhanced development workflows. ## 📑 Table of Contents @@ -2101,24 +2101,44 @@ Falsification-first methodology for network, streaming, and protocol-layer bugs --- -### 50. **stepfun-tts** - StepFun StepAudio 2.5 TTS + ASR +### 50. **stepfun-tts** - StepFun StepAudio 2.5 Contextual TTS -Generate Chinese / Japanese speech and transcribe long audio with StepFun's StepAudio 2.5 family. Captures the three non-obvious pitfalls that cost hours otherwise: `voice_label` removal, the `/v1/audio/asr/sse` endpoint, and stricter censorship. +Generate Chinese / Japanese speech with `stepaudio-2.5-tts`. Captures the two non-obvious TTS pitfalls that cost hours otherwise: `voice_label` removal (replaced by natural-language `instruction`) and stricter 2.5-era censorship (死/消失/political terms). **When to use:** -- Chinese / Japanese TTS with emotional and prosody control -- Long audio transcription (up to ~30 minutes single-call, 32K context, ~100x RTF) +- Chinese / Japanese TTS with emotional and prosody control (whisper, pause, stress, mid-sentence pivot) +- Batch-generating game / app voice lines with per-line `censorship_block` fallback - Migration from `step-tts-2` to `stepaudio-2.5-tts` (`voice_label` → `instruction` breaking change) -- Hitting StepFun censorship blocks or endpoint mismatches +- Hitting StepFun censorship blocks on previously-fine content **Key features:** - `stepaudio-2.5-tts` with `instruction` (≤200 chars natural-language mood) + inline `()` prosody -- `stepaudio-2.5-asr` SSE streaming with base64 audio (avoids the misleading "model not supported" error) -- Bundled `tts_generate.py` (with `--batch `), `asr_transcribe.py`, `ab_compare.sh` +- Bundled `tts_generate.py` (with `--batch `) and `ab_compare.sh` - API key resolution: `$STEPFUN_API_KEY` → `${CLAUDE_PLUGIN_DATA}/config.json` fallback - Censorship rewrite playbook in `references/migration_from_v2.md` -**Requirements**: StepFun API key (https://platform.stepfun.com/). +**Requirements**: StepFun API key, "Normal" tier (https://platform.stepfun.com/). For ASR / transcription, use the sibling `stepfun-asr` skill below. + +--- + +### 52. **stepfun-asr** - StepFun StepAudio 2.5 ASR (SSE Endpoint) + +Transcribe Chinese / English audio with `stepaudio-2.5-asr`. Hides the #1 trap of the 2.5 ASR family: it does NOT live on `/v1/audio/transcriptions` — the wrong endpoint returns a misleading `model stepaudio-2.5-asr not supported` error that looks identical to a permission/whitelist failure. + +**When to use:** +- Long audio transcription (up to ~30 minutes single-call, 32K context, ~85-101× RTF — no client-side chunking) +- Migration from `step-asr` / `step-asr-1.1` (different endpoint, different body shape, SSE response) +- Hitting the misleading `model stepaudio-2.5-asr not supported` error (= wrong endpoint, not permission) +- Silent 4xx auth failures on audio endpoints (= using a "Plan" key instead of a "Normal" key) + +**Key features:** +- `/v1/audio/asr/sse` SSE streaming with base64 audio + nested JSON body (the script handles all four traps) +- Bundled `asr_transcribe.py` — pure-stdlib CLI, auto-detects mp3/wav/ogg/opus/pcm by extension +- Handles SSE `error` events (censorship can fire on ASR side too — rare but real) +- API key resolution: `$STEPFUN_API_KEY` → `${CLAUDE_PLUGIN_DATA}/config.json` fallback +- Suggests `transcript-fixer` (ASR error correction) and `meeting-minutes-taker` (structured minutes) as natural downstream skills + +**Requirements**: StepFun API key, "Normal" tier (https://platform.stepfun.com/). Plan keys cannot call audio endpoints. --- @@ -2245,8 +2265,11 @@ Use **terraform-skill** when your `terraform apply` fails at a provisioner step, ### For Network, Streaming & Protocol-Layer Debugging Use **debugging-network-issues** when symptoms do not match the obvious cause: HTTP/2 `RST_STREAM`, SSE stalls at exactly 60s/100s/130s, "works sometimes but not always" failures, or anything that looks like an idle-timeout incident through CDN / proxy / CGNAT chains. The skill replaces assumption-stacking with **layered isolation experiments** — running the same logical request through three or more paths that differ by one hop — plus a counter-review pattern for shipping fixes only after the hypothesis has been falsified, not just confirmed. -### For Chinese TTS & Long-Audio Transcription (StepFun) -Use **stepfun-tts** for Chinese / Japanese voice synthesis with emotional control via `instruction` + inline `()` prosody, or for transcribing up to 30-minute audio in a single call (32K context, ~100x RTF). Captures the three breaking changes that ambush new StepAudio 2.5 users: `voice_label` removal, the `/v1/audio/asr/sse` endpoint mismatch, and stricter censorship rules. Combine with **transcript-fixer** for ASR post-processing or with **meeting-minutes-taker** to turn long recordings into structured minutes. +### For Chinese TTS (StepFun StepAudio 2.5) +Use **stepfun-tts** for Chinese / Japanese voice synthesis with emotional control via `instruction` + inline `()` prosody. Captures the two breaking changes that ambush new StepAudio 2.5 users: `voice_label` removal and stricter 2.5-era censorship rules. Pair with `step-tts-2` as a per-line fallback for content that triggers censorship. + +### For Long-Audio Transcription (StepFun StepAudio 2.5) +Use **stepfun-asr** for transcribing up to 30-minute Chinese / English audio in a single SSE call (32K context, ~85-101× RTF, no client-side chunking). Hides the #1 trap — the model does NOT live on `/v1/audio/transcriptions`; the wrong endpoint returns a misleading "model not supported" error. Combine with **transcript-fixer** for ASR error correction or with **meeting-minutes-taker** to turn long recordings into structured minutes. ## 📚 Documentation @@ -2304,7 +2327,8 @@ Each skill includes: - **terraform-skill**: See `terraform-skill/SKILL.md` for the full catalogue of operational traps organised by exact error → root cause → copy-paste fix - **slides-creator**: See `slides-creator/SKILL.md` for the narrative-first workflow, `slides-creator/references/narrative-design-guide.md` for the ABCDEFG model, and `slides-creator/references/content-creation-first-law.md` for the universal content creation principle - **debugging-network-issues**: See `debugging-network-issues/SKILL.md` for the falsification-first workflow, `debugging-network-issues/references/layered-isolation-experiment.md` for the multi-hop isolation pattern, and `debugging-network-issues/references/case-sse-rst-130s.md` for the real production case study -- **stepfun-tts**: See `stepfun-tts/SKILL.md` for the TTS+ASR decision tree and `stepfun-tts/references/migration_from_v2.md` for the `voice_label` → `instruction` migration playbook plus the censorship rewrite list +- **stepfun-tts**: See `stepfun-tts/SKILL.md` for the Contextual TTS decision tree and `stepfun-tts/references/migration_from_v2.md` for the `voice_label` → `instruction` migration playbook plus the censorship rewrite list +- **stepfun-asr**: See `stepfun-asr/SKILL.md` for the SSE-endpoint workflow and the four ASR-side traps (wrong endpoint, Plan-vs-Normal key, repetition hallucination, SSE `error` event). `stepfun-asr/references/api_reference.md` documents the exact JSON request body and SSE event contract for raw HTTP integration ## 🛠️ Requirements @@ -2333,7 +2357,7 @@ Each skill includes: - **Python 3.8+** (for continue-claude-work): bundled script for session extraction (no external dependencies) - **uv + Scrapling CLI** (for scrapling-skill): `uv tool install 'scrapling[shell]'` and `scrapling install` for browser-backed fetches - **Node.js 18+ + curl + unzip** (for ima-copilot): `npx skills` is fetched on demand from the npm registry; IMA OpenAPI credentials from [https://ima.qq.com/agent-interface](https://ima.qq.com/agent-interface) -- **StepFun API key** (for stepfun-tts): Available at [https://platform.stepfun.com/](https://platform.stepfun.com/) → API Keys +- **StepFun API key** (for stepfun-tts and stepfun-asr — must be "Normal" tier, Plan keys silently fail on audio endpoints): Available at [https://platform.stepfun.com/](https://platform.stepfun.com/) → API Keys ## ❓ FAQ diff --git a/README.zh-CN.md b/README.zh-CN.md index 61d56c6d..3d807517 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -6,15 +6,15 @@ [![简体中文](https://img.shields.io/badge/语言-简体中文-red)](./README.zh-CN.md) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -[![Skills](https://img.shields.io/badge/skills-51-blue.svg)](https://github.com/daymade/claude-code-skills) -[![Version](https://img.shields.io/badge/version-1.51.0-green.svg)](https://github.com/daymade/claude-code-skills) +[![Skills](https://img.shields.io/badge/skills-52-blue.svg)](https://github.com/daymade/claude-code-skills) +[![Version](https://img.shields.io/badge/version-1.52.0-green.svg)](https://github.com/daymade/claude-code-skills) [![Claude Code](https://img.shields.io/badge/Claude%20Code-2.0.13+-purple.svg)](https://claude.com/code) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](./CONTRIBUTING.md) [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/daymade/claude-code-skills/graphs/commit-activity) -专业的 Claude Code 技能市场，提供 51 个生产就绪的技能，用于增强开发工作流。 +专业的 Claude Code 技能市场，提供 52 个生产就绪的技能，用于增强开发工作流。 ## 📑 目录 @@ -2142,24 +2142,44 @@ uv run douban-skill/scripts/douban-rss-sync.py --- -### 50. **stepfun-tts** - 阶跃 StepAudio 2.5 TTS + ASR +### 50. **stepfun-tts** - 阶跃 StepAudio 2.5 Contextual TTS -用 StepFun 阶跃的 StepAudio 2.5 系列做中文 / 日语语音合成与长音频转写。封装了三个会浪费时间的非显然坑：`voice_label` 移除、`/v1/audio/asr/sse` 端点、更严的审查。 +用 `stepaudio-2.5-tts` 做中文 / 日语语音合成。封装了 TTS 部分两个会浪费时间的非显然坑：`voice_label` 被移除（改用自然语言 `instruction`）以及 2.5 时代更严格的审查（死/消失/政治敏感词）。 **使用场景：** -- 带情感和韵律控制的中 / 日语 TTS -- 长音频转写（单次最长 ~30 分钟、32K context、~100x RTF） +- 带情感和韵律控制的中 / 日语 TTS（耳语、停顿、加重、句中情绪转折） +- 批量生成游戏 / 应用语音条目，每条单独处理 `censorship_block` 兜底 - 从 `step-tts-2` 迁移到 `stepaudio-2.5-tts`（`voice_label` → `instruction` 是破坏性变更） -- 遇到 StepFun 审查拦截或端点错误 +- 之前能合成的内容现在被审查拦截 **主要功能：** - `stepaudio-2.5-tts`：用 `instruction`（≤200 字自然语言情绪）+ 文中 `()` 行内韵律 -- `stepaudio-2.5-asr`：SSE 流式 + base64 音频（避开误导性的 "model not supported" 错误） -- 内置 `tts_generate.py`（含 `--batch `）、`asr_transcribe.py`、`ab_compare.sh` +- 内置 `tts_generate.py`（含 `--batch `）、`ab_compare.sh` - API key 解析顺序：`$STEPFUN_API_KEY` → `${CLAUDE_PLUGIN_DATA}/config.json` 兜底 - `references/migration_from_v2.md` 给出审查拦截的改写策略 -**要求**：StepFun API key（https://platform.stepfun.com/）。 +**要求**：StepFun API key 的 "Normal" 等级（https://platform.stepfun.com/）。如需 ASR / 转写，使用下方的姊妹技能 `stepfun-asr`。 + +--- + +### 52. **stepfun-asr** - 阶跃 StepAudio 2.5 ASR（SSE 端点） + +用 `stepaudio-2.5-asr` 转写中 / 英文音频。封装 2.5 ASR 系列最坑的一点：模型**不在** `/v1/audio/transcriptions`——错端点返回的 `model stepaudio-2.5-asr not supported` 看起来跟权限被拒一模一样，会让人浪费几小时排查。 + +**使用场景：** +- 长音频转写（单次最长 ~30 分钟、32K context、~85-101× RTF、无需客户端切片） +- 从 `step-asr` / `step-asr-1.1` 迁移（端点不同、请求体不同、响应是 SSE 流） +- 遇到误导性的 `model stepaudio-2.5-asr not supported` 错误（= 端点用错了，不是权限问题） +- 调音频端点遭遇无声 4xx 鉴权失败（= 用了 "Plan" key 而不是 "Normal" key） + +**主要功能：** +- `/v1/audio/asr/sse` SSE 流 + base64 音频 + 嵌套 JSON 请求体（脚本一并处理四个坑） +- 内置 `asr_transcribe.py`——纯 stdlib CLI，按扩展名自动识别 mp3/wav/ogg/opus/pcm +- 处理 SSE `error` 事件（审查在 ASR 端也会触发——罕见但真实存在） +- API key 解析顺序：`$STEPFUN_API_KEY` → `${CLAUDE_PLUGIN_DATA}/config.json` 兜底 +- 推荐 `transcript-fixer`（ASR 纠错）和 `meeting-minutes-taker`（结构化纪要）作为下游技能 + +**要求**：StepFun API key 的 "Normal" 等级（https://platform.stepfun.com/）。Plan key 调不通音频端点。 --- @@ -2286,8 +2306,11 @@ uv run douban-skill/scripts/douban-rss-sync.py ### 网络、流式与协议层调试 使用 **debugging-network-issues** 应对症状和"显然原因"对不上的场景：HTTP/2 `RST_STREAM`、SSE 在 60s/100s/130s 整点卡死、"时灵时不灵"故障、或 CDN / 代理 / CGNAT 链路上的空闲超时事件。Skill 用**分层隔离实验**（同一逻辑请求走三条以上、每条仅差一跳的路径）替代假设堆叠，再加一套反审查模式——只在假设被**证伪**而不是单纯被"证实"之后才上 fix。 -### 中文 TTS 与长音频转写（StepFun 阶跃） -使用 **stepfun-tts** 进行中 / 日语语音合成（通过 `instruction` + 行内 `()` 控制情绪与韵律），或单次最长 30 分钟的长音频转写（32K context、~100x RTF）。封装了让 StepAudio 2.5 新用户必踩的三个破坏性变更：`voice_label` 移除、`/v1/audio/asr/sse` 端点错位、更严的审查规则。可与 **transcript-fixer** 组合做 ASR 后处理，或与 **meeting-minutes-taker** 把长录音变成结构化纪要。 +### 中文 TTS（StepFun 阶跃 StepAudio 2.5） +使用 **stepfun-tts** 进行中 / 日语语音合成（通过 `instruction` + 行内 `()` 控制情绪与韵律）。封装了让 StepAudio 2.5 新用户必踩的两个 TTS 破坏性变更：`voice_label` 移除和 2.5 时代更严的审查规则。可把 `step-tts-2` 作为单条审查兜底来组合使用。 + +### 长音频转写（StepFun 阶跃 StepAudio 2.5） +使用 **stepfun-asr** 单次 SSE 调用转写最长 30 分钟的中 / 英文音频（32K context、~85-101× RTF、无需客户端切片）。封装了 #1 大坑——模型**不在** `/v1/audio/transcriptions`，错端点返回误导性的 "model not supported" 错误。可与 **transcript-fixer** 组合做 ASR 纠错，或与 **meeting-minutes-taker** 把长录音变成结构化纪要。 ## 📚 文档 @@ -2345,7 +2368,8 @@ uv run douban-skill/scripts/douban-rss-sync.py - **terraform-skill**：参见 `terraform-skill/SKILL.md` 查看按确切报错 → 根本原因 → 复制粘贴修复组织的实操陷阱完整目录 - **slides-creator**：参见 `slides-creator/SKILL.md` 了解叙事优先工作流，参见 `slides-creator/references/narrative-design-guide.md` 了解 ABCDEFG 模型，参见 `slides-creator/references/content-creation-first-law.md` 了解通用内容创作原则 - **debugging-network-issues**：参见 `debugging-network-issues/SKILL.md` 了解证伪优先工作流，参见 `debugging-network-issues/references/layered-isolation-experiment.md` 了解多跳隔离模式，参见 `debugging-network-issues/references/case-sse-rst-130s.md` 查看真实生产案例 -- **stepfun-tts**：参见 `stepfun-tts/SKILL.md` 了解 TTS+ASR 决策树，参见 `stepfun-tts/references/migration_from_v2.md` 查看 `voice_label` → `instruction` 迁移手册和审查改写清单 +- **stepfun-tts**：参见 `stepfun-tts/SKILL.md` 了解 Contextual TTS 决策树，参见 `stepfun-tts/references/migration_from_v2.md` 查看 `voice_label` → `instruction` 迁移手册和审查改写清单 +- **stepfun-asr**：参见 `stepfun-asr/SKILL.md` 了解 SSE 端点工作流和 ASR 侧四个坑（错端点、Plan vs Normal key、重复幻觉、SSE `error` 事件）。`stepfun-asr/references/api_reference.md` 给出原始 HTTP 集成所需的 JSON 请求体和 SSE 事件契约 ## 🛠️ 系统要求 @@ -2371,7 +2395,7 @@ uv run douban-skill/scripts/douban-rss-sync.py - **Python 3.8+**（用于 continue-claude-work）：内置脚本进行会话提取（无外部依赖） - **uv + Scrapling CLI**（用于 scrapling-skill）：`uv tool install 'scrapling[shell]'`，浏览器抓取前运行 `scrapling install` - **Node.js 18+ + curl + unzip**（用于 ima-copilot）：`npx skills` 按需从 npm registry 拉取；IMA OpenAPI 凭据从 [https://ima.qq.com/agent-interface](https://ima.qq.com/agent-interface) 获取 -- **StepFun API key**（用于 stepfun-tts）：在 [https://platform.stepfun.com/](https://platform.stepfun.com/) → API Keys 获取 +- **StepFun API key**（用于 stepfun-tts 和 stepfun-asr——必须是 "Normal" 等级，Plan key 调音频端点会无声失败）：在 [https://platform.stepfun.com/](https://platform.stepfun.com/) → API Keys 获取 ## ❓ 常见问题 From 1549306e8ff1c97dfcf3a74d13a63b44126deddb Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 10 May 2026 02:36:31 +0800 Subject: [PATCH 136/137] feat(statusline-generator): rewrite SKILL.md + scripts for v1.1.0 Overhaul statusline-generator with improved context window display, health check script, customization guide, and troubleshooting decision tree. Co-Authored-By: Claude Opus 4.6 (1M context) --- .../statusline-generator/SKILL.md | 309 +++++++----------- .../references/context-window-schema.md | 10 +- .../references/customization.md | 130 ++++++++ .../troubleshooting-decision-tree.md | 231 +++++++++++++ .../scripts/generate_statusline.sh | 274 +++++++++------- .../scripts/health_check.sh | 146 +++++++++ .../scripts/install_statusline.sh | 127 ++++--- 7 files changed, 862 insertions(+), 365 deletions(-) create mode 100644 daymade-claude-code/statusline-generator/references/customization.md create mode 100644 daymade-claude-code/statusline-generator/references/troubleshooting-decision-tree.md mode change 100644 => 100755 daymade-claude-code/statusline-generator/scripts/generate_statusline.sh create mode 100755 daymade-claude-code/statusline-generator/scripts/health_check.sh mode change 100644 => 100755 daymade-claude-code/statusline-generator/scripts/install_statusline.sh diff --git a/daymade-claude-code/statusline-generator/SKILL.md b/daymade-claude-code/statusline-generator/SKILL.md index 0bd73bb6..18451476 100644 --- a/daymade-claude-code/statusline-generator/SKILL.md +++ b/daymade-claude-code/statusline-generator/SKILL.md @@ -1,253 +1,174 @@ --- name: statusline-generator -description: Configures and customizes Claude Code statuslines with context window display (actual token counts), multi-line layouts, cost tracking via ccusage, git status indicators, human-readable formatting, and customizable colors. Activates for statusline setup, installation, configuration, customization, context window display, color changes, cost display, git status integration, or troubleshooting statusline issues. +description: Install, configure, customize, or troubleshoot the Claude Code statusline (the line above the prompt with cwd, model, and token counts). Use when the user wants to set up or change the statusline, switch between minimal and full layouts, show absolute token counts (e.g. ctx 108K / 1M) instead of a percentage, add cost via ccusage or git status, or fix a statusline that is blank, silent, stuck, shows "permission denied", or stopped updating after a script edit (commonly a missing chmod +x). Also covers debug-dumping the stdin JSON Claude Code passes the script. Trigger phrases include "configure statusline", "statusline blank", "status line not showing", "statusline broken", "show token count in statusline", 状态栏, 状态栏不显示, 状态栏空白, 显示工作目录, 显示 token 数. --- # Statusline Generator -## Overview +A single-source-of-truth statusline for Claude Code. One script, two layouts, +end-to-end self-verification. -This skill provides tools and guidance for creating and customizing Claude Code statuslines. It generates multi-line statuslines optimized for portrait screens, integrates with `ccusage` for session/daily cost tracking, displays git branch status, and supports color customization. +## Quick health check (start here when something is wrong) -## When to Use This Skill +Run this first whenever the statusline misbehaves. It catches the silent failures +that account for most "configured but not working" reports: -This skill activates for: -- Statusline configuration requests for Claude Code -- Cost information display (session/daily costs) -- Multi-line layouts for portrait or narrow screens -- Statusline color or format customization -- Statusline display or cost tracking issues -- Git status or path shortening features - -## Dependencies - -The statusline script needs to parse JSON from stdin. It auto-detects the available parser: - -| Priority | Tool | Availability | -|----------|------|-------------| -| 1 (preferred) | `jq` | macOS/Linux: `brew install jq` / `apt install jq`; Windows: `choco install jq` or `scoop install jq` | -| 2 (fallback) | `python3` | Pre-installed on macOS and most Linux distros; Windows: `winget install python3` or python.org | - -If neither is available, context window and cost display will be skipped — git branch and path still work. - -Other requirements: -- `git` — for branch status (optional; gracefully skips outside repos) -- `awk` — for number formatting (installed on macOS/Linux by default; Git Bash on Windows includes it) -- `ccusage` — for session/daily cost tracking (optional; gracefully skips if missing) - -## Quick Start - -### Basic Installation - -Install the default multi-line statusline: - -1. Run the installation script: - ```bash - bash scripts/install_statusline.sh - ``` +```bash +bash scripts/health_check.sh +``` -2. Restart Claude Code to see the statusline +It validates four layers: +1. `~/.claude/statusline.sh` exists and is executable. **Missing `chmod +x` is + the single most common silent-failure cause** — Claude Code runs the script, + `exec` fails, statusline goes blank. +2. `~/.claude/settings.json` has a valid `statusLine` block pointing at the script. +3. Mock stdin tests covering complete data, zero tokens, missing fields, + and `$HOME` path shortening. +4. Real stdin replay from `/tmp/.claude-statusline-last-stdin.json` if you + previously ran with `CLAUDE_STATUSLINE_DEBUG=1`. -The default statusline displays: -- **Line 1**: `username (model) [session_cost/daily_cost] ctx: 89.5K/1.0M (9%)` -- **Line 2**: `current_path` -- **Line 3**: `[git:branch*+]` +Each failure prints a one-line fix command — you don't have to read documentation +to recover. -### Manual Installation +## Quick install -Alternatively, manually install by: +```bash +bash scripts/install_statusline.sh +``` -1. Copy `scripts/generate_statusline.sh` to `~/.claude/statusline.sh` -2. Make it executable: `chmod +x ~/.claude/statusline.sh` -3. Update `~/.claude/settings.json`: - ```json - { - "statusLine": { - "type": "command", - "command": "bash /home/username/.claude/statusline.sh", - "padding": 0 - } - } - ``` +This script: +- Backs up any existing `~/.claude/statusline.sh` and `settings.json`. +- Copies `generate_statusline.sh` to `~/.claude/statusline.sh` and `chmod +x`s it. +- Updates `settings.json` `statusLine` block via `jq` (preserves other settings). +- **Mandatorily runs `health_check.sh` and shows the result** — installation + is not "complete" until verification passes. -## Statusline Features +Restart Claude Code (or send any new message) to see the statusline update. -### Multi-Line Layout +## What you get -The statusline uses a 3-line layout optimized for portrait screens: +### Default — minimal one-line layout ``` -username (Sonnet 4.5 [1M]) [$0.26/$25.93] ctx: 89.5K/1.0M (9%) -~/workspace/java/ready-together-svc -[git:feature/branch-name*+] +~/code/myproject Opus 4.7 (1M context) ctx: 108K / 1M ``` -**Benefits:** -- Shorter lines fit narrow screens -- Clear visual separation of information types -- No horizontal scrolling needed - -### Cost Tracking Integration - -Cost tracking via `ccusage`: -- **Session Cost**: Current conversation cost -- **Daily Cost**: Total cost for today -- **Format**: `[$session/$daily]` in magenta -- **Caching**: 2-minute cache to avoid performance impact -- **Background Fetch**: First run loads costs asynchronously +Just the essentials: short path, model name, absolute token counts. No colors, +no git, no cost, no percentage. Designed for users who want signal without noise. -**Requirements:** `ccusage` must be installed and in PATH. See `references/ccusage_integration.md` for installation and troubleshooting. +### Full — multi-line with cost and git -### Model Name Shortening +Set `CLAUDE_STATUSLINE_LAYOUT=full` in your shell profile to enable: -Model names are automatically shortened: -- `"Sonnet 4.5 (with 1M token context)"` → `"Sonnet 4.5 [1M]"` -- `"Opus 4.1 (with 500K token context)"` → `"Opus 4.1 [500K]"` - -This saves horizontal space while preserving key information. - -### Context Window Display - -The statusline can show actual context window usage with human-readable token counts — not just a percentage. This is the most important statusline feature for long sessions. - -Format: ``` -ctx: 88.9K/1.0M (9%) +alex (Sonnet 4.6) [$0.42/$25.93] ctx: 108K/1M (11%) +~/code/myproject +[git:main*+] ``` -Components: -- **Used tokens**: `current_usage.input_tokens + cache_read_input_tokens + cache_creation_input_tokens` -- **Total capacity**: `context_window_size` from the input JSON -- **Percentage**: Shown in parentheses as secondary info -- **Human-readable**: `<1000` → raw, `≥1000` → `X.XK`, `≥1M` → `X.XM` -- **Color-coded**: Green (≤50%), Yellow (51–80%), Red (>80%) - -**Important**: Use `current_usage.*` fields to compute actual context usage. `total_input_tokens` is the session-cumulative count and can exceed the context window size. +- Line 1: user, model, ccusage session/daily costs, color-coded ctx (green ≤50%, + yellow 51–80%, red >80%). +- Line 2: short path. +- Line 3: git branch with `*` for modified, `+` for untracked. -Full statusline input JSON schema (all available fields): `references/context-window-schema.md`. +## Layouts: how to switch -### Git Status Indicators +The script reads layout from environment, not flags (Claude Code passes JSON on stdin, +so flags would conflict). Set in `~/.zshrc` or `~/.bashrc`: -Git branch status shows: -- **Yellow**: Clean branch (no changes) -- **Red**: Dirty branch (uncommitted changes) -- **Indicators**: - - `*` - Modified or staged files - - `+` - Untracked files - - Example: `[git:main*+]` - Modified files and untracked files - -### Path Shortening - -Paths are shortened: -- Home directory replaced with `~` -- Example: `/home/username/workspace/project` → `~/workspace/project` - -### Color Scheme - -Default colors optimized for visibility: -- **Username**: Bright Green (`\033[01;32m`) -- **Model**: Bright Cyan (`\033[01;36m`) -- **Costs**: Bright Magenta (`\033[01;35m`) -- **Path**: Bright White (`\033[01;37m`) -- **Git (clean)**: Bright Yellow (`\033[01;33m`) -- **Git (dirty)**: Bright Red (`\033[01;31m`) - -## Customization - -### Changing Colors - -Customize colors by editing `~/.claude/statusline.sh` and modifying the ANSI color codes in the final `printf` statement. See `references/color_codes.md` for available colors. - -**Example: Change username to blue** ```bash -# Find this line: -printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m%s\n\033[01;37m%s\033[00m\n%s' \ +# Minimal (default — same as not setting it) +export CLAUDE_STATUSLINE_LAYOUT=minimal -# Change \033[01;32m (green) to \033[01;34m (blue): -printf '\033[01;34m%s\033[00m \033[01;36m(%s)\033[00m%s\n\033[01;37m%s\033[00m\n%s' \ +# Full +export CLAUDE_STATUSLINE_LAYOUT=full ``` -### Single-Line Layout +Restart your shell (or `source` the rc file) so Claude Code inherits the change, +then send a message — statusline refreshes within 300ms. + +## Debug stdin capture -Convert to single-line layout by modifying the final `printf`: +To see exactly what JSON Claude Code sends your script: ```bash -# Replace: -printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m%s\n\033[01;37m%s\033[00m\n%s' \ - "$username" "$model" "$cost_info" "$short_path" "$git_info" - -# With single-line + context: -printf '\033[01;36m[%s]\033[00m \033[01;35m%s\033[00m%s | %bctx: %s/%s (%s%%)\033[00m | \033[01;32m$%s\033[00m' \ - "$model" "$short_dir" "$git_info" "$ctx_color" "$ctx_used_h" "$ctx_size_h" "$ctx_used_pct" "$cost" -# Output: [Sonnet 4.5 [1M]] project (main*) | ctx: 89.5K/1M (9%) | $0.42 +export CLAUDE_STATUSLINE_DEBUG=1 ``` -### Disabling Cost Tracking +Each invocation writes its stdin to `/tmp/.claude-statusline-last-stdin.json` +(overwriting on every refresh). Inspect with `jq .`. Useful for: -If `ccusage` is unavailable or not desired: +- Diagnosing why a field doesn't render the way you expect. +- Re-running the script against real input: `cat /tmp/.claude-statusline-last-stdin.json | ~/.claude/statusline.sh`. +- Filing bug reports — paste the dump as ground truth. -1. Comment out the cost section in the script (lines ~47-73) -2. Remove `%s` for `$cost_info` from the final `printf` +## Authoring rules (why this skill is shaped this way) -See `references/ccusage_integration.md` for details. +Two production failure modes drove the current design. Both are sealed in code, +not just docs: -### Adding Custom Elements +### Rule 1 — Always `chmod +x`, always verify by running -Add custom information (e.g., hostname, time): +The single biggest silent-failure cause of any statusline is a script without +the executable bit: Claude Code's `exec` fails silently and the bar goes blank +with no error. `install_statusline.sh` always `chmod +x`s; `health_check.sh` +flags the bit if missing. **If you hand-write or hand-edit a statusline script, +mock-test it before declaring done:** `echo '{}' | bash your-script.sh`. -```bash -# Add variable before final printf: -hostname=$(hostname -s) -current_time=$(date +%H:%M) +### Rule 2 — "Configuration complete" is meaningless without evidence -# Update printf to include new elements: -printf '\033[01;32m%s@%s\033[00m \033[01;36m(%s)\033[00m%s [%s]\n...' \ - "$username" "$hostname" "$model" "$cost_info" "$current_time" ... -``` +"Wrote the file and updated settings.json" is not the same as "the script runs +and produces the expected output." `install_statusline.sh` therefore always +runs `health_check.sh` at the end and exits non-zero if any check fails. +Treat any "complete!" report from any agent that lacks evidence as suspect. -## Troubleshooting +For field-level traps (`used_percentage` null at session start, `total_input_tokens` +semantics across Claude Code versions, hardcoded `context_window_size`), see +[`references/context-window-schema.md`](references/context-window-schema.md). -### Costs Not Showing +## Customization -**Check:** -1. Is `ccusage` installed? Run `which ccusage` -2. Test `ccusage` manually: `ccusage session --json --offline -o desc` -3. Wait 5-10 seconds after first display (background fetch) -4. Check cache: `ls -lh /tmp/claude_cost_cache_*.txt` +For colors, custom segments (hostname, time, etc.), and disabling cost tracking, +see [`references/customization.md`](references/customization.md). -**Solution:** See `references/ccusage_integration.md` for detailed troubleshooting. +## Dependencies -### Colors Hard to Read +The script auto-detects available tools and degrades gracefully: -**Solution:** Adjust colors for your terminal background using `references/color_codes.md`. Bright colors (`01;3X`) are generally more visible than regular (`00;3X`). +| Tool | Required for | Fallback | +|------|-------------|----------| +| `jq` | JSON parsing (preferred) | falls back to `python3` | +| `python3` | JSON parsing fallback | bare `cwd` only | +| `awk` | token K/M formatting | required by both layouts | +| `git` | git status (full layout) | silent skip if missing or not in repo | +| `ccusage` | cost (full layout) | silent skip if missing | -### Statusline Not Updating +Install on macOS: `brew install jq`. On Debian/Ubuntu: `apt install jq`. -**Check:** -1. Verify settings.json points to correct script path -2. Ensure script is executable: `chmod +x ~/.claude/statusline.sh` -3. Restart Claude Code +## Troubleshooting -### Git Status Not Showing +For symptom-by-symptom diagnostics, see +[`references/troubleshooting-decision-tree.md`](references/troubleshooting-decision-tree.md). +It walks through: -**Check:** -1. Are you in a git repository? -2. Test git commands: `git branch --show-current` -3. Check git permissions in the directory +1. Statusline blank or never updates (chmod cause) +2. ctx segment missing or wrong (field traps) +3. Want token counts not percentages (layout switch) +4. Colors render as raw escape codes (terminal compatibility) +5. Git segment missing (full layout) +6. Cost segment missing (ccusage / cache) +7. Edits have no effect (path mismatch) +8. Slow refresh (jq vs python3) ## Resources -### scripts/generate_statusline.sh -Main statusline script with all features (context window, multi-line, ccusage, git, colors). Copy to `~/.claude/statusline.sh` for use. - -### scripts/install_statusline.sh -Automated installation script that copies the statusline script and updates settings.json. - -### references/context-window-schema.md -Complete statusline input JSON schema. Documents all fields under `context_window`, `cost`, `model`, `workspace`. Load for context window display implementation or debugging. - -### references/color_codes.md -Complete ANSI color code reference for customizing statusline colors. Load when users request color customization. - -### references/ccusage_integration.md -Detailed explanation of ccusage integration, caching strategy, JSON structure, and troubleshooting. Load when users experience cost tracking issues or want to understand how it works. \ No newline at end of file +| File | Purpose | +|------|---------| +| `scripts/generate_statusline.sh` | The statusline script. Single source of truth. Two layouts via `CLAUDE_STATUSLINE_LAYOUT`. | +| `scripts/install_statusline.sh` | Idempotent installer. Backs up, copies, chmods, wires `settings.json`, runs health check. | +| `scripts/health_check.sh` | Four-layer verification: file perms, settings.json wiring, mock stdin tests, real stdin replay. | +| `references/troubleshooting-decision-tree.md` | Symptom-driven diagnostic flowchart. Load when statusline misbehaves. | +| `references/customization.md` | Color changes, custom segments, threshold tuning, single-line full layout. Load when user wants to modify how the statusline looks. | +| `references/context-window-schema.md` | Claude Code statusline JSON schema. Documents every field plus `current_usage` vs `total_input_tokens` semantics across versions. | +| `references/color_codes.md` | ANSI color codes reference. Load for color customization. | +| `references/ccusage_integration.md` | ccusage integration deep-dive: caching, JSON shape, troubleshooting. Load for cost-related issues. | diff --git a/daymade-claude-code/statusline-generator/references/context-window-schema.md b/daymade-claude-code/statusline-generator/references/context-window-schema.md index aac2b0db..d3b0037b 100644 --- a/daymade-claude-code/statusline-generator/references/context-window-schema.md +++ b/daymade-claude-code/statusline-generator/references/context-window-schema.md @@ -62,8 +62,8 @@ The statusline script receives a JSON object on stdin. This reference documents | `context_window_size` | number | Model's total context window in tokens (e.g., 1000000 = 1M) | | `used_percentage` | number | Current context usage as percentage (0-100, may have decimals) | | `remaining_percentage` | number | Remaining context capacity as percentage | -| `total_input_tokens` | number | **Session-cumulative** input tokens (not current context state) | -| `total_output_tokens` | number | **Session-cumulative** output tokens | +| `total_input_tokens` | number | Tokens currently in the context window (sum of `input_tokens` + `cache_creation_input_tokens` + `cache_read_input_tokens`). **Before Claude Code v2.1.132 this was session-cumulative and could exceed `context_window_size`.** | +| `total_output_tokens` | number | Output tokens from the most recent response. **Before v2.1.132 this was session-cumulative.** | | `current_usage.input_tokens` | number | Current turn uncached input tokens | | `current_usage.output_tokens` | number | Current turn output tokens | | `current_usage.cache_creation_input_tokens` | number | Tokens written to prompt cache this turn | @@ -80,7 +80,11 @@ current_context_used = current_usage.input_tokens This sum should approximately equal `context_window_size * used_percentage / 100`. -**Do NOT use `total_input_tokens`** — it's the session-level cumulative total, which can exceed the context window size (e.g., 150K total_input_tokens in a 100K context window is normal because old content gets evicted). +**Prefer `current_usage.*` summed.** It is correct on every Claude Code version +(0 at session start, never null, never cumulative). `total_input_tokens` is +equivalent on v2.1.132 and later but was session-cumulative before that and +could exceed `context_window_size`. If you need to support older Claude Code +versions, use `current_usage`. ### Model Context Window Sizes (Reference) diff --git a/daymade-claude-code/statusline-generator/references/customization.md b/daymade-claude-code/statusline-generator/references/customization.md new file mode 100644 index 00000000..c7aa5e25 --- /dev/null +++ b/daymade-claude-code/statusline-generator/references/customization.md @@ -0,0 +1,130 @@ +# Customization + +How to modify the statusline beyond layout switching. Load this file when the +user wants colors, custom segments, or to disable specific features. + +## Change colors (full layout) + +Colors are ANSI escape codes in `printf` calls inside `render_full()`. See +[`color_codes.md`](color_codes.md) for the complete code reference. + +**Example — recolor the username from green to blue:** + +```bash +# In scripts/generate_statusline.sh, find this line in render_full: +printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m%s%s\n...' "$username" ... +# ^^^^^^^^ green (32) + +# Change to blue (34): +printf '\033[01;34m%s\033[00m \033[01;36m(%s)\033[00m%s%s\n...' "$username" ... +``` + +**Standard color codes:** + +| Code | Color | +|------|-------| +| `\033[01;30m` | Bright black (gray) | +| `\033[01;31m` | Bright red | +| `\033[01;32m` | Bright green | +| `\033[01;33m` | Bright yellow | +| `\033[01;34m` | Bright blue | +| `\033[01;35m` | Bright magenta | +| `\033[01;36m` | Bright cyan | +| `\033[01;37m` | Bright white | +| `\033[00m` | Reset | + +After editing, always verify with `bash scripts/health_check.sh` to confirm +mock tests still pass. + +## Add custom segments (full layout) + +Extend `render_full()` to add hostname, time, weather, or anything else. + +**Pattern:** + +```bash +render_full() { + # ... existing code ... + + # Your new segment + local hostname segment_color + hostname=$(hostname -s) + segment_color="\033[01;34m" # blue + + # Add to the existing printf format string + printf '\033[01;32m%s@%s\033[00m \033[01;36m(%s)\033[00m%s%s\n...' \ + "$username" "$hostname" "$model" "$cost_info" "$ctx_display" + # ^^^^^ ^^^^^^^^^ added +} +``` + +**Pattern — time:** + +```bash +local now +now=$(date +%H:%M) +# Add %s and "$now" to the printf +``` + +Keep additions side-effect-free so `health_check.sh` mock tests remain +deterministic. After editing, run `bash scripts/health_check.sh`. + +## Disable cost tracking (full layout) + +Cost requires `ccusage`. If `ccusage` is not installed, `cost_info` is empty +automatically — **no edit needed**, the segment silently disappears. + +To skip the lookup entirely (saves ~50ms on each refresh): + +In `scripts/generate_statusline.sh`, locate the cost block in `render_full()`: + +```bash +if command -v ccusage >/dev/null 2>&1; then + # ... cache + ccusage logic ... +fi +``` + +Wrap or comment out the entire `if` block. Health check will still pass. + +## Switch to single-line full layout + +The default `full` layout is multi-line (3 lines: header / path / git). To +collapse it into a single line, edit the final `printf` in `render_full()`: + +```bash +# Three-line (default full): +printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m%s%s\n\033[01;37m%s\033[00m\n%s' \ + "$username" "$model" "$cost_info" "$ctx_display" "$short_path" "$git_info" + +# Single-line full: +printf '\033[01;36m[%s]\033[00m \033[01;37m%s\033[00m %s%s | \033[01;32m$%s\033[00m' \ + "$model" "$short_path" "$git_info" "$ctx_display" "$cost" +``` + +## Change ctx color thresholds + +Default thresholds (full layout): green ≤50%, yellow 51–80%, red >80%. + +To change, edit in `render_full()`: + +```bash +local ctx_color="\033[01;32m" # green ≤50% +if [ "${ctx_pct_int:-0}" -gt 80 ]; then + ctx_color="\033[01;31m" # red >80% +elif [ "${ctx_pct_int:-0}" -gt 50 ]; then + ctx_color="\033[01;33m" # yellow 51-80% +fi +``` + +Adjust the `80` and `50` cutoffs to taste. + +## Where to put the env var setting + +For `CLAUDE_STATUSLINE_LAYOUT` and `CLAUDE_STATUSLINE_DEBUG` to apply to +Claude Code, the variables must be exported in the shell **before** Claude +Code starts. Set them in: + +- macOS / Linux: `~/.zshrc`, `~/.bashrc`, `~/.profile`, or `~/.config/fish/config.fish` +- Per-project: `.envrc` (with [direnv](https://direnv.net/)) + +Restart the shell or `source` the rc file, then start Claude Code. diff --git a/daymade-claude-code/statusline-generator/references/troubleshooting-decision-tree.md b/daymade-claude-code/statusline-generator/references/troubleshooting-decision-tree.md new file mode 100644 index 00000000..7827b156 --- /dev/null +++ b/daymade-claude-code/statusline-generator/references/troubleshooting-decision-tree.md @@ -0,0 +1,231 @@ +# Troubleshooting Decision Tree + +When the statusline misbehaves, run the health check first — it covers most issues: + +```bash +bash scripts/health_check.sh +``` + +If the health check passes but you still see an issue, walk the symptoms below. + +--- + +## Symptom 1: Statusline is blank or never updates + +**Most common root cause: the script is not executable.** Claude Code runs the +configured `command` and silently shows nothing if the exec fails. This is the +single biggest source of "I configured it but nothing happens" reports. + +**Diagnose:** +```bash +ls -la "$(jq -r '.statusLine.command' ~/.claude/settings.json | sed "s|^~|$HOME|; s|^bash ||")" +``` + +Look at the leftmost column. If it does not start with `-rwx`, the script is +missing its executable bit. + +**Fix:** +```bash +chmod +x ~/.claude/statusline.sh +``` + +(Substitute the actual path from your `settings.json`.) + +**Other possible causes:** +- `settings.json` `statusLine.command` points to a path that no longer exists. + Run `bash scripts/health_check.sh` to confirm. +- The script's shebang references an interpreter that's not installed. Verify + with `head -1 ~/.claude/statusline.sh` — should be `#!/usr/bin/env bash` or + similar. +- The script writes to stderr instead of stdout. Claude Code only displays + stdout. Pipe a mock stdin and inspect: `echo '{}' | ~/.claude/statusline.sh`. + +--- + +## Symptom 2: `ctx: ...` segment is missing or shows wrong numbers + +**Root cause A: `used_percentage` is `null` early in a session.** + +The Claude Code docs explicitly warn that `context_window.used_percentage` and +`remaining_percentage` "may be `null` early in the session." If your script +gates the ctx segment on these fields, the segment vanishes until the first +real API response populates them. A naive `// empty` filter swallows the +output entirely. + +**Fix:** This skill's `generate_statusline.sh` computes used tokens from +`current_usage.input_tokens + cache_read_input_tokens + cache_creation_input_tokens`, +which is `0` (not null) at session start, so the ctx segment renders even +before any real usage. + +**Root cause B: confusing `total_input_tokens` with current context.** + +In Claude Code v2.1.131 and earlier, `total_input_tokens` was a session-cumulative +count and could exceed `context_window_size`. Using it as "current usage" +shows percentages above 100% in long sessions. + +**Fix:** Use `current_usage.*` summed (this skill's default). It always +reflects the current context state regardless of Claude Code version. + +**Root cause C: `context_window_size` missing from JSON.** + +Some early versions or non-standard clients omit this field. The script then +divides by zero (or skips the segment). + +**Diagnose:** +```bash +export CLAUDE_STATUSLINE_DEBUG=1 +# Send any message in Claude Code, then: +jq '.context_window' /tmp/.claude-statusline-last-stdin.json +``` + +If `context_window_size` is missing, your Claude Code is too old or running +a non-standard runtime. Update Claude Code. + +--- + +## Symptom 3: I want token counts, not percentages + +The default layout shows `ctx: 108K / 1M` (token counts only). If yours shows +percentages, you are running the `full` layout. + +**Switch to minimal:** + +Remove this from your shell rc file: +```bash +export CLAUDE_STATUSLINE_LAYOUT=full +``` + +Or set it explicitly to minimal: +```bash +export CLAUDE_STATUSLINE_LAYOUT=minimal +``` + +Then start a new shell so Claude Code inherits the new env. + +--- + +## Symptom 4: Colors look wrong or render as literal escape codes + +**Diagnose:** +```bash +echo '{"workspace":{"current_dir":"/tmp"},"model":{"display_name":"M"},"context_window":{"context_window_size":100000,"used_percentage":80,"current_usage":{"input_tokens":80000,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}' | \ + CLAUDE_STATUSLINE_LAYOUT=full ~/.claude/statusline.sh | cat -v +``` + +If you see `^[[01;33m` instead of yellow text, your terminal is not +interpreting ANSI escape codes. + +**Fix:** +- Most modern terminals (iTerm2, Apple Terminal, Kitty, Windows Terminal, + WezTerm) support these by default. If yours doesn't, switch terminal. +- Claude Code itself renders the statusline output and supports ANSI colors, + so this issue is rare during actual use — only shows up when you pipe + the script manually. + +--- + +## Symptom 5: Git status segment is missing (full layout) + +**Root cause A:** You're not inside a git repository. Expected — git segment +is silent outside repos. + +**Root cause B:** `git` binary not installed. + +**Diagnose:** +```bash +command -v git || echo "git not installed" +git -C "$PWD" rev-parse --git-dir 2>&1 +``` + +**Fix:** Install git, or accept that the segment is hidden. + +**Root cause C:** Permission errors reading `.git/`. + +```bash +ls -la "$(git rev-parse --git-dir 2>/dev/null)" 2>&1 | head -3 +``` + +--- + +## Symptom 6: Cost segment is missing (full layout) + +The cost segment depends on `ccusage` being installed and on PATH. It runs +asynchronously with a 2-minute cache, so the first display after install is +expected to lack it; the cache populates within 5–10 seconds. + +**Diagnose:** +```bash +command -v ccusage || echo "ccusage not installed" +ccusage session --json --offline -o desc 2>&1 | head -20 +ls -lh /tmp/claude_cost_cache_*.txt 2>/dev/null +``` + +**Fix:** +- Install `ccusage`: `npm install -g ccusage` (or see the `ccusage` repo for + current install instructions). +- Wait for the background fetch on first use. +- Force refresh: `rm /tmp/claude_cost_cache_*.txt` then trigger a statusline + update by sending any message in Claude Code. + +For deeper details and offline data structure, see +[`ccusage_integration.md`](ccusage_integration.md). + +--- + +## Symptom 7: I edited the script but my changes have no effect + +Claude Code reads the script every refresh, so live edits **should** take +effect on the next statusline update (typically the next assistant message). + +**Diagnose:** +```bash +# Confirm Claude Code actually runs your edited script: +export CLAUDE_STATUSLINE_DEBUG=1 +# Then send a message in Claude Code and check the dump: +ls -la /tmp/.claude-statusline-last-stdin.json +``` + +If the dump's mtime updates after your message, the script is being executed. +If your edits still have no effect, try `bash health_check.sh` to confirm the +script being run is actually the one you edited (it might be a stale copy at +a different path). + +**Possible mismatch:** `settings.json` `statusLine.command` points to one path, +but you edited a different file. The health check warns when these diverge. + +--- + +## Symptom 8: Script is slow (statusline appears with delay) + +**Diagnose:** +```bash +time (echo '{"workspace":{"current_dir":"/tmp"},"model":{"display_name":"M"},"context_window":{"context_window_size":1000,"current_usage":{"input_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}' | ~/.claude/statusline.sh) +``` + +Should be well under 100ms. If much slower, suspect: + +- **`ccusage` blocking:** Should be backgrounded by the script. Verify with + `time ccusage session --json --offline -o desc | head -1`. +- **`git` slow on a large repo:** Switch to minimal layout to skip git lookup. +- **`python3` fallback hot path:** Install `jq` (`brew install jq`). + +--- + +## When all else fails + +Capture both real stdin and the script's actual output, then re-run health check: + +```bash +export CLAUDE_STATUSLINE_DEBUG=1 +# Send a message in Claude Code, wait 1 second, then: +bash scripts/health_check.sh +echo "---" +echo "Real stdin Claude Code sent:" +jq . /tmp/.claude-statusline-last-stdin.json +echo "---" +echo "Script output for that stdin:" +cat /tmp/.claude-statusline-last-stdin.json | ~/.claude/statusline.sh +``` + +The combination of `health_check.sh` results plus the captured stdin/output +pair is enough evidence to diagnose virtually any issue. diff --git a/daymade-claude-code/statusline-generator/scripts/generate_statusline.sh b/daymade-claude-code/statusline-generator/scripts/generate_statusline.sh old mode 100644 new mode 100755 index 55a82d62..892410a6 --- a/daymade-claude-code/statusline-generator/scripts/generate_statusline.sh +++ b/daymade-claude-code/statusline-generator/scripts/generate_statusline.sh @@ -1,146 +1,188 @@ #!/usr/bin/env bash -# Statusline script — multi-line layout with context window, cost, git -# Dependencies: git, awk. jq preferred; python3 used as fallback if jq missing. +# Claude Code statusline — single source of truth. +# +# Default (minimal): ~/short/path Model Name ctx: 108K / 1M +# Full layout: +# user (Model) [$session/$daily] ctx: 108K / 1M (11%) +# ~/short/path +# [git:branch*+] +# +# Configuration via environment variables (no flags — Claude Code passes JSON on stdin): +# CLAUDE_STATUSLINE_LAYOUT=full enable multi-line cost/git/percentage layout +# CLAUDE_STATUSLINE_DEBUG=1 dump stdin to /tmp/.claude-statusline-last-stdin.json +# (for end-to-end verification, see health_check.sh) +# +# Dependencies: jq preferred (python3 fallback). awk for number formatting. +# Optional: git (for full layout's git status), ccusage (for cost display in full layout). input=$(cat) -# --- JSON field extraction (jq with python3 fallback) --- -if command -v jq &>/dev/null; then - IFS=$'\t' read -r model_full cwd ctx_used_pct cost_raw ctx_size \ - ctx_input ctx_cache_read ctx_cache_create <<< \ - "$(echo "$input" | jq -r ' +if [ -n "$CLAUDE_STATUSLINE_DEBUG" ]; then + printf '%s' "$input" > /tmp/.claude-statusline-last-stdin.json 2>/dev/null +fi + +LAYOUT="${CLAUDE_STATUSLINE_LAYOUT:-minimal}" + +# ---------- JSON field extraction (jq preferred, python3 fallback) ---------- +parse_with_jq() { + echo "$input" | jq -r ' [ (.model.display_name // "Claude"), (.workspace.current_dir // ""), - (.context_window.used_percentage // 0), - (.cost.total_cost_usd // 0), (.context_window.context_window_size // 0), (.context_window.current_usage.input_tokens // 0), (.context_window.current_usage.cache_read_input_tokens // 0), - (.context_window.current_usage.cache_creation_input_tokens // 0) + (.context_window.current_usage.cache_creation_input_tokens // 0), + (.context_window.used_percentage // 0), + (.cost.total_cost_usd // 0) ] | @tsv - ')" -else - # Windows / no-jq: fall back to python3 (stdlib only, no pip needed) - IFS=$'\t' read -r model_full cwd ctx_used_pct cost_raw ctx_size \ - ctx_input ctx_cache_read ctx_cache_create <<< \ - "$(echo "$input" | python3 -c ' + ' +} + +parse_with_python() { + echo "$input" | python3 -c ' import json, sys d = json.load(sys.stdin) -cw = d.get("context_window", {}) -cu = cw.get("current_usage", {}) -vals = [ +cw = d.get("context_window") or {} +cu = cw.get("current_usage") or {} +print("\t".join(str(v) for v in [ d.get("model", {}).get("display_name", "Claude"), d.get("workspace", {}).get("current_dir", ""), - cw.get("used_percentage", 0), - d.get("cost", {}).get("total_cost_usd", 0), - cw.get("context_window_size", 0), - cu.get("input_tokens", 0), - cu.get("cache_read_input_tokens", 0), - cu.get("cache_creation_input_tokens", 0), -] -print("\t".join(str(v) for v in vals)) -')" + cw.get("context_window_size", 0) or 0, + cu.get("input_tokens", 0) or 0, + cu.get("cache_read_input_tokens", 0) or 0, + cu.get("cache_creation_input_tokens", 0) or 0, + cw.get("used_percentage", 0) or 0, + (d.get("cost") or {}).get("total_cost_usd", 0) or 0, +])) +' +} + +if command -v jq >/dev/null 2>&1; then + parsed=$(parse_with_jq) +elif command -v python3 >/dev/null 2>&1; then + parsed=$(parse_with_python) +else + # No JSON parser — degrade to bare cwd + echo "$PWD" + exit 0 fi -# coerce percentage to int safely -ctx_used_pct=$(printf '%.0f' "${ctx_used_pct:-0}" 2>/dev/null || echo 0) +IFS=$'\t' read -r model_full cwd ctx_size ctx_input ctx_cache_read ctx_cache_create ctx_pct cost_raw <<< "$parsed" -# --- derived values --- cwd="${cwd:-$PWD}" -ctx_used=$((ctx_input + ctx_cache_read + ctx_cache_create)) - -# human-readable number formatter -human() { - local n=${1:-0} - if [ "$n" -ge 1000000 ]; then - awk -v v="$n" 'BEGIN {printf "%.1fM", v/1000000}' - elif [ "$n" -ge 1000 ]; then - awk -v v="$n" 'BEGIN {printf "%.1fK", v/1000}' - else - echo "$n" - fi +ctx_size="${ctx_size:-0}" +ctx_used=$((${ctx_input:-0} + ${ctx_cache_read:-0} + ${ctx_cache_create:-0})) +ctx_pct_int=$(printf '%.0f' "${ctx_pct:-0}" 2>/dev/null || echo 0) +short_path="${cwd/#$HOME/~}" + +# ---------- Helpers ---------- +# Format token counts: 999 / 108K / 1M / 1.5M +human_tokens() { + awk -v n="${1:-0}" 'BEGIN { + n = n + 0 + if (n >= 1000000) { + m = n / 1000000 + if (m == int(m)) printf "%dM", m + else printf "%.1fM", m + } else if (n >= 1000) { + printf "%dK", int(n/1000 + 0.5) + } else { + printf "%d", n + } + }' } -ctx_used_h=$(human $ctx_used) -ctx_size_h=$(human $ctx_size) -cost=$(echo "$cost_raw" | awk '{printf "%.2f", $1}') - -# model name shortening -model=$(echo "$model_full" \ - | sed -E 's/\(with ([0-9]+[KM]) token context\)/[\1]/' \ - | sed -E 's/\[1m\]/[1M]/' \ - | sed 's/ *$//') - -username=$(whoami) -short_path="${cwd/#$HOME/\~}" - -# --- context color thresholds --- -ctx_color="\033[01;32m" # green ≤50% -if [ "$ctx_used_pct" -gt 80 ]; then - ctx_color="\033[01;31m" # red >80% -elif [ "$ctx_used_pct" -gt 50 ]; then - ctx_color="\033[01;33m" # yellow 51-80% -fi - -# --- git branch status --- -git_info="" -if [ -d "$cwd/.git" ] || git -C "$cwd" --no-optional-locks rev-parse --git-dir >/dev/null 2>&1; then - branch=$(git -C "$cwd" --no-optional-locks branch --show-current 2>/dev/null || echo "detached") - - status="" - if ! git -C "$cwd" --no-optional-locks diff --quiet 2>/dev/null || \ - ! git -C "$cwd" --no-optional-locks diff --cached --quiet 2>/dev/null; then - status="*" +# ---------- Minimal layout (default) ---------- +render_minimal() { + local out="$short_path" + [ -n "$model_full" ] && out="${out} ${model_full}" + if [ "${ctx_size:-0}" -gt 0 ] 2>/dev/null; then + out="${out} ctx: $(human_tokens "$ctx_used") / $(human_tokens "$ctx_size")" fi - if [ -n "$(git -C "$cwd" --no-optional-locks ls-files --others --exclude-standard 2>/dev/null)" ]; then - status="${status}+" + echo "$out" +} + +# ---------- Full layout (multi-line: cost + git + percentage) ---------- +render_full() { + local username + username=$(whoami) + + # Color threshold by ctx percentage + local ctx_color="\033[01;32m" # green ≤50% + if [ "${ctx_pct_int:-0}" -gt 80 ]; then + ctx_color="\033[01;31m" # red >80% + elif [ "${ctx_pct_int:-0}" -gt 50 ]; then + ctx_color="\033[01;33m" # yellow 51-80% fi - if [ -n "$status" ]; then - git_info=$(printf ' \033[01;31m[git:%s%s]\033[00m' "$branch" "$status") - else - git_info=$(printf ' \033[01;33m[git:%s]\033[00m' "$branch") + # Model name shortening: "Sonnet 4.5 (with 1M token context)" -> "Sonnet 4.5 [1M]" + local model + model=$(echo "$model_full" \ + | sed -E 's/\(with ([0-9]+[KM]) token context\)/[\1]/' \ + | sed -E 's/\[1m\]/[1M]/' \ + | sed 's/ *$//') + + # Git status (best-effort; silent if not a git repo or git missing) + local git_info="" + if command -v git >/dev/null 2>&1 && \ + git -C "$cwd" --no-optional-locks rev-parse --git-dir >/dev/null 2>&1; then + local branch status="" + branch=$(git -C "$cwd" --no-optional-locks branch --show-current 2>/dev/null || echo "detached") + if ! git -C "$cwd" --no-optional-locks diff --quiet 2>/dev/null || \ + ! git -C "$cwd" --no-optional-locks diff --cached --quiet 2>/dev/null; then + status="*" + fi + if [ -n "$(git -C "$cwd" --no-optional-locks ls-files --others --exclude-standard 2>/dev/null)" ]; then + status="${status}+" + fi + if [ -n "$status" ]; then + git_info=$(printf '\033[01;31m[git:%s%s]\033[00m' "$branch" "$status") + else + git_info=$(printf '\033[01;33m[git:%s]\033[00m' "$branch") + fi fi -fi -# --- cost via ccusage (async, non-blocking; requires jq or python3) --- -cost_info="" -if command -v jq &>/dev/null || command -v python3 &>/dev/null; then - cache_file="/tmp/claude_cost_cache_$(date +%Y%m%d_%H%M).txt" - find /tmp -name "claude_cost_cache_*.txt" -mmin +2 -delete 2>/dev/null - - if [ -f "$cache_file" ]; then - cost_info=$(cat "$cache_file") - else - { - if command -v jq &>/dev/null; then - session=$(ccusage session --json --offline -o desc 2>/dev/null | jq -r '.sessions[0].totalCost' 2>/dev/null | xargs printf "%.2f" 2>/dev/null) - daily=$(ccusage daily --json --offline -o desc 2>/dev/null | jq -r '.daily[0].totalCost' 2>/dev/null | xargs printf "%.2f" 2>/dev/null) - else - session=$(ccusage session --json --offline -o desc 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(f\"{d['sessions'][0]['totalCost']:.2f}\")" 2>/dev/null) - daily=$(ccusage daily --json --offline -o desc 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); print(f\"{d['daily'][0]['totalCost']:.2f}\")" 2>/dev/null) - fi - if [ -n "$session" ] && [ -n "$daily" ] && [ "$session" != "" ] && [ "$daily" != "" ]; then - printf ' \033[01;35m[$%s/$%s]\033[00m' "$session" "$daily" > "$cache_file" - fi - } & - prev_cache=$(find /tmp -name "claude_cost_cache_*.txt" -mmin -10 2>/dev/null | head -1) - if [ -f "$prev_cache" ]; then - cost_info=$(cat "$prev_cache") + # Cost via ccusage (cached, async; silent if ccusage unavailable) + local cost_info="" + if command -v ccusage >/dev/null 2>&1; then + local cache_file="/tmp/claude_cost_cache_$(date +%Y%m%d_%H%M).txt" + find /tmp -maxdepth 1 -name "claude_cost_cache_*.txt" -mmin +2 -delete 2>/dev/null + if [ -f "$cache_file" ]; then + cost_info=$(cat "$cache_file") + else + { + local session daily + if command -v jq >/dev/null 2>&1; then + session=$(ccusage session --json --offline -o desc 2>/dev/null | jq -r '.sessions[0].totalCost // 0' | xargs printf "%.2f" 2>/dev/null) + daily=$(ccusage daily --json --offline -o desc 2>/dev/null | jq -r '.daily[0].totalCost // 0' | xargs printf "%.2f" 2>/dev/null) + else + session=$(ccusage session --json --offline -o desc 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); s=d.get('sessions',[{}])[0]; print(f\"{s.get('totalCost',0):.2f}\")" 2>/dev/null) + daily=$(ccusage daily --json --offline -o desc 2>/dev/null | python3 -c "import json,sys; d=json.load(sys.stdin); s=d.get('daily',[{}])[0]; print(f\"{s.get('totalCost',0):.2f}\")" 2>/dev/null) + fi + if [ -n "$session" ] && [ -n "$daily" ]; then + printf ' \033[01;35m[$%s/$%s]\033[00m' "$session" "$daily" > "$cache_file" + fi + } & + local prev_cache + prev_cache=$(find /tmp -maxdepth 1 -name "claude_cost_cache_*.txt" -mmin -10 2>/dev/null | head -1) + [ -f "$prev_cache" ] && cost_info=$(cat "$prev_cache") fi fi -fi -# --- context display string --- -ctx_display="" -if [ "$ctx_size" -gt 0 ]; then - ctx_display=$(printf " ${ctx_color}ctx: %s/%s (%s%%)" "$ctx_used_h" "$ctx_size_h" "$ctx_used_pct") -fi + # Context display + local ctx_display="" + if [ "${ctx_size:-0}" -gt 0 ] 2>/dev/null; then + ctx_display=$(printf " ${ctx_color}ctx: %s/%s (%s%%)\033[00m" \ + "$(human_tokens "$ctx_used")" "$(human_tokens "$ctx_size")" "$ctx_pct_int") + fi + + # Three lines: header / path / git + printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m%s%s\n\033[01;37m%s\033[00m\n%s' \ + "$username" "$model" "$cost_info" "$ctx_display" "$short_path" "$git_info" +} -# --- output: 3-line layout --- -# Line 1: username (model) [costs] ctx: used/total (pct%) -# Line 2: path -# Line 3: [git:branch] -printf '\033[01;32m%s\033[00m \033[01;36m(%s)\033[00m%s%s\n\033[01;37m%s\033[00m\n%s' \ - "$username" "$model" "$cost_info" "$ctx_display" "$short_path" "$git_info" +case "$LAYOUT" in + full) render_full ;; + minimal|*) render_minimal ;; +esac diff --git a/daymade-claude-code/statusline-generator/scripts/health_check.sh b/daymade-claude-code/statusline-generator/scripts/health_check.sh new file mode 100755 index 00000000..48a9477f --- /dev/null +++ b/daymade-claude-code/statusline-generator/scripts/health_check.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# Statusline health check — verify configuration end-to-end. +# +# Runs four layers of verification: +# 1. Script exists and is executable (chmod +x). +# 2. settings.json points to the script and uses type=command. +# 3. Mock stdin tests (minimal + edge cases) — script must output expected shape. +# 4. Real stdin replay (if /tmp/.claude-statusline-last-stdin.json exists from CLAUDE_STATUSLINE_DEBUG). +# +# Usage: +# bash health_check.sh # checks ~/.claude/statusline.sh +# bash health_check.sh /custom/path.sh # checks custom script + +set -u + +SCRIPT_PATH="${1:-$HOME/.claude/statusline.sh}" +SETTINGS_FILE="$HOME/.claude/settings.json" +DEBUG_DUMP="/tmp/.claude-statusline-last-stdin.json" + +# Color output only if stdout is a terminal +if [ -t 1 ]; then + G='\033[01;32m'; R='\033[01;31m'; Y='\033[01;33m'; B='\033[01;36m'; N='\033[00m' +else + G=''; R=''; Y=''; B=''; N='' +fi + +PASS=0 +FAIL=0 +WARN=0 + +ok() { printf "${G}✓${N} %s\n" "$1"; PASS=$((PASS+1)); } +fail() { printf "${R}✗${N} %s\n" "$1"; [ -n "${2:-}" ] && printf " ${B}fix:${N} %s\n" "$2"; FAIL=$((FAIL+1)); } +warn() { printf "${Y}⚠${N} %s\n" "$1"; [ -n "${2:-}" ] && printf " ${B}note:${N} %s\n" "$2"; WARN=$((WARN+1)); } +section() { printf "\n${B}== %s ==${N}\n" "$1"; } + +# ---------- 1. Script existence + permissions ---------- +section "1. Script file" + +if [ ! -f "$SCRIPT_PATH" ]; then + fail "Script not found: $SCRIPT_PATH" "copy generate_statusline.sh from this skill to that path" + echo + printf "${R}HARD FAIL — cannot continue checks${N}\n" + exit 1 +fi +ok "Script exists: $SCRIPT_PATH" + +if [ ! -x "$SCRIPT_PATH" ]; then + fail "Script not executable (this is the #1 silent-failure cause)" "chmod +x '$SCRIPT_PATH'" +else + ok "Script is executable (chmod +x)" +fi + +# ---------- 2. settings.json wiring ---------- +section "2. settings.json wiring" + +if [ ! -f "$SETTINGS_FILE" ]; then + fail "settings.json not found: $SETTINGS_FILE" "create it with statusLine block, see SKILL.md Quick Start" +elif command -v jq >/dev/null 2>&1; then + sl_type=$(jq -r '.statusLine.type // "MISSING"' "$SETTINGS_FILE" 2>/dev/null) + sl_cmd=$(jq -r '.statusLine.command // "MISSING"' "$SETTINGS_FILE" 2>/dev/null) + + if [ "$sl_type" = "MISSING" ]; then + fail "settings.json has no statusLine block" "run install_statusline.sh" + elif [ "$sl_type" != "command" ]; then + fail "statusLine.type is '$sl_type', expected 'command'" "set statusLine.type to \"command\"" + else + ok "statusLine.type = command" + fi + + if [ "$sl_cmd" = "MISSING" ]; then + fail "statusLine.command is missing" "set it to a path or shell command" + else + # Expand ~ for comparison + sl_cmd_expanded="${sl_cmd/#\~/$HOME}" + # Strip leading "bash " if user wrapped it + sl_cmd_stripped="${sl_cmd_expanded#bash }" + if [ "$sl_cmd_stripped" = "$SCRIPT_PATH" ] || [ "$sl_cmd_expanded" = "$SCRIPT_PATH" ]; then + ok "statusLine.command points to $SCRIPT_PATH" + else + warn "statusLine.command = '$sl_cmd' (expected to reference $SCRIPT_PATH)" \ + "edit settings.json statusLine.command if you intended to use this script" + fi + fi +else + warn "jq not installed — cannot validate settings.json structure" "brew install jq (or apt install jq)" +fi + +# ---------- 3. Mock stdin tests ---------- +section "3. Mock stdin tests (minimal layout)" + +run_mock() { + local label="$1" json="$2" expect_pattern="$3" + local out + out=$(echo "$json" | "$SCRIPT_PATH" 2>&1) + if echo "$out" | grep -Eq "$expect_pattern"; then + ok "$label" + printf " output: %s\n" "$out" + else + fail "$label — output didn't match expected shape" "expected pattern: $expect_pattern; got: $out" + fi +} + +# Test 1: complete data +run_mock "complete data → renders cwd + model + ctx" \ + '{"workspace":{"current_dir":"/tmp/test"},"model":{"display_name":"TestModel"},"context_window":{"context_window_size":1000000,"current_usage":{"input_tokens":50000,"cache_read_input_tokens":0,"cache_creation_input_tokens":50000}}}' \ + 'TestModel.*ctx: 100K / 1M' + +# Test 2: zero tokens (session start) +run_mock "0 tokens (session just started)" \ + '{"workspace":{"current_dir":"/tmp/test"},"model":{"display_name":"M"},"context_window":{"context_window_size":1000000,"current_usage":{"input_tokens":0,"cache_read_input_tokens":0,"cache_creation_input_tokens":0}}}' \ + 'ctx: 0 / 1M' + +# Test 3: missing context_window entirely +run_mock "missing context_window field → no ctx segment" \ + '{"workspace":{"current_dir":"/tmp/test"},"model":{"display_name":"M"}}' \ + '/tmp/test M' + +# Test 4: cwd in HOME → tilde shortening +run_mock "cwd in HOME → ~ short path" \ + "{\"workspace\":{\"current_dir\":\"$HOME/x/y\"},\"model\":{\"display_name\":\"M\"}}" \ + '~/x/y' + +# ---------- 4. Real stdin replay (if available) ---------- +section "4. Real stdin replay" + +if [ -f "$DEBUG_DUMP" ]; then + out=$(cat "$DEBUG_DUMP" | "$SCRIPT_PATH" 2>&1) + if [ -n "$out" ]; then + ok "Real stdin from $DEBUG_DUMP renders successfully" + printf " output: %s\n" "$out" + else + fail "Real stdin produced empty output" "inspect $DEBUG_DUMP and run: cat $DEBUG_DUMP | $SCRIPT_PATH" + fi +else + warn "No real stdin dump available at $DEBUG_DUMP" \ + "to capture: export CLAUDE_STATUSLINE_DEBUG=1, send any message in Claude Code, re-run this check" +fi + +# ---------- Summary ---------- +section "Summary" +printf "Pass: ${G}%d${N} Fail: ${R}%d${N} Warn: ${Y}%d${N}\n" "$PASS" "$FAIL" "$WARN" + +if [ "$FAIL" -gt 0 ]; then + exit 1 +fi +exit 0 diff --git a/daymade-claude-code/statusline-generator/scripts/install_statusline.sh b/daymade-claude-code/statusline-generator/scripts/install_statusline.sh old mode 100644 new mode 100755 index 18b4ff9a..94c49e0d --- a/daymade-claude-code/statusline-generator/scripts/install_statusline.sh +++ b/daymade-claude-code/statusline-generator/scripts/install_statusline.sh @@ -1,73 +1,96 @@ #!/usr/bin/env bash - -# Install statusline script to Claude Code configuration directory -# Usage: ./install_statusline.sh [target_path] +# Install statusline script + wire settings.json + run health check. +# +# After install, the script always finishes with a health_check.sh run so the +# user sees concrete pass/fail evidence (not just "Installation complete"). +# This prevents the silent-failure mode where chmod is forgotten or the +# settings.json command points to the wrong path. +# +# Usage: +# bash install_statusline.sh # install to ~/.claude/statusline.sh +# bash install_statusline.sh /custom/path.sh # install to custom path set -e -# Determine target path -if [ -n "$1" ]; then - TARGET_PATH="$1" -else - TARGET_PATH="$HOME/.claude/statusline.sh" -fi - -# Get the directory where this script is located +TARGET_PATH="${1:-$HOME/.claude/statusline.sh}" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SOURCE_SCRIPT="$SCRIPT_DIR/generate_statusline.sh" +HEALTH_CHECK="$SCRIPT_DIR/health_check.sh" +SETTINGS_FILE="$HOME/.claude/settings.json" +TARGET_DIR=$(dirname "$TARGET_PATH") -# Check if source script exists if [ ! -f "$SOURCE_SCRIPT" ]; then - echo "❌ Error: generate_statusline.sh not found at $SOURCE_SCRIPT" + echo "ERROR: generate_statusline.sh not found at $SOURCE_SCRIPT" >&2 exit 1 fi -# Create .claude directory if it doesn't exist -CLAUDE_DIR=$(dirname "$TARGET_PATH") -if [ ! -d "$CLAUDE_DIR" ]; then - echo "📁 Creating directory: $CLAUDE_DIR" - mkdir -p "$CLAUDE_DIR" +if [ ! -d "$TARGET_DIR" ]; then + echo ">> Creating directory: $TARGET_DIR" + mkdir -p "$TARGET_DIR" fi -# Copy the script -echo "📋 Copying statusline script to: $TARGET_PATH" -cp "$SOURCE_SCRIPT" "$TARGET_PATH" -chmod +x "$TARGET_PATH" +# Backup existing target script if present +if [ -f "$TARGET_PATH" ]; then + backup="$TARGET_PATH.bak.$(date +%Y%m%d_%H%M%S)" + echo ">> Backing up existing script: $backup" + cp "$TARGET_PATH" "$backup" +fi -# Update settings.json -SETTINGS_FILE="$HOME/.claude/settings.json" +echo ">> Installing: $SOURCE_SCRIPT -> $TARGET_PATH" +cp "$SOURCE_SCRIPT" "$TARGET_PATH" +chmod +x "$TARGET_PATH" # critical — silent failure root cause if missed +# Wire settings.json if [ ! -f "$SETTINGS_FILE" ]; then - echo "⚠️ Warning: settings.json not found at $SETTINGS_FILE" - echo " Please create it manually or restart Claude Code" - exit 0 -fi + echo ">> Creating new settings.json with statusLine block" + cat > "$SETTINGS_FILE" </dev/null 2>&1; then + settings_backup="$SETTINGS_FILE.bak.$(date +%Y%m%d_%H%M%S)" + cp "$SETTINGS_FILE" "$settings_backup" + echo ">> Backed up settings.json: $settings_backup" -# Check if statusLine already configured -if grep -q '"statusLine"' "$SETTINGS_FILE"; then - echo "✅ statusLine already configured in settings.json" - echo " Current configuration will use the updated script" + tmp=$(mktemp) + jq --arg cmd "$TARGET_PATH" \ + '.statusLine = {"type":"command","command":$cmd,"padding":(.statusLine.padding // 0)}' \ + "$SETTINGS_FILE" > "$tmp" + mv "$tmp" "$SETTINGS_FILE" + echo ">> Updated settings.json statusLine.command -> $TARGET_PATH" else - echo "📝 Adding statusLine configuration to settings.json" - - # Backup settings.json - cp "$SETTINGS_FILE" "$SETTINGS_FILE.backup" - - # Add statusLine configuration using jq - jq '. + {"statusLine": {"type": "command", "command": "bash '"$TARGET_PATH"'", "padding": 0}}' "$SETTINGS_FILE.backup" > "$SETTINGS_FILE" + echo "WARN: jq not installed — cannot safely edit settings.json" >&2 + echo " Manually set statusLine.command to: $TARGET_PATH" >&2 +fi - echo "✅ statusLine configuration added" - echo " Backup saved to: $SETTINGS_FILE.backup" +# ---- Mandatory health check (do not let the user discover failures themselves) ---- +echo +echo "== Running health check ==" +if [ -x "$HEALTH_CHECK" ] || bash "$HEALTH_CHECK" --help >/dev/null 2>&1; then + bash "$HEALTH_CHECK" "$TARGET_PATH" || { + echo + echo "WARN: Health check reported issues. Review the output above." >&2 + echo " Re-run anytime: bash $HEALTH_CHECK $TARGET_PATH" >&2 + exit 1 + } +else + echo "WARN: health_check.sh not found or not executable — skipping post-install verification" >&2 fi -echo "" -echo "🎉 Installation complete!" -echo "" -echo "Next steps:" -echo " 1. Restart Claude Code to see your new statusline" -echo " 2. The statusline will show:" -echo " Line 1: username (model) [session_cost/daily_cost]" -echo " Line 2: current_path" -echo " Line 3: [git:branch]" -echo "" -echo "Note: Cost information requires ccusage to be installed and accessible" \ No newline at end of file +echo +echo "Installation complete." +echo +echo "Usage:" +echo " Default : minimal one-line layout (cwd + model + ctx tokens)" +echo " Full layout : add to ~/.zshrc or ~/.bashrc:" +echo " export CLAUDE_STATUSLINE_LAYOUT=full" +echo " (multi-line with cost via ccusage + git status + percentage)" +echo " Debug stdin : export CLAUDE_STATUSLINE_DEBUG=1" +echo " dumps each invocation's stdin to /tmp/.claude-statusline-last-stdin.json" +echo +echo "Verify anytime: bash $HEALTH_CHECK" From 8d07e4a7ab05e7c8ff9e31df3b4557c1f9d09b26 Mon Sep 17 00:00:00 2001 From: daymade Date: Sun, 10 May 2026 02:36:41 +0800 Subject: [PATCH 137/137] chore: tunnel-doctor DNS layer, feishu-doc-scraper, cleanup - tunnel-doctor: add DNS resolver chain stall documentation (layer 5) - feishu-doc-scraper: track new skill files - Remove COMPETITOR_MATRIX.md (stale, not maintained) - Remove wechat-article-scraper path patterns from .pii-path-patterns (skill was previously unregistered from marketplace) Co-Authored-By: Claude Opus 4.6 (1M context) --- .pii-path-patterns | 2 - COMPETITOR_MATRIX.md | 137 ------ feishu-doc-scraper/.security-scan-passed | 4 + feishu-doc-scraper/SKILL.md | 453 ++++++++++++++++++ .../references/capture-manifest.md | 53 ++ .../references/history-derived-rules.md | 69 +++ .../references/tooling-matrix.md | 92 ++++ .../scripts/build_feishu_markdown.py | 116 +++++ .../scripts/check_heading_coverage.py | 103 ++++ tunnel-doctor/SKILL.md | 126 ++++- .../references/dns_resolver_chain_stall.md | 210 ++++++++ 11 files changed, 1225 insertions(+), 140 deletions(-) delete mode 100644 COMPETITOR_MATRIX.md create mode 100644 feishu-doc-scraper/.security-scan-passed create mode 100644 feishu-doc-scraper/SKILL.md create mode 100644 feishu-doc-scraper/references/capture-manifest.md create mode 100644 feishu-doc-scraper/references/history-derived-rules.md create mode 100644 feishu-doc-scraper/references/tooling-matrix.md create mode 100755 feishu-doc-scraper/scripts/build_feishu_markdown.py create mode 100755 feishu-doc-scraper/scripts/check_heading_coverage.py create mode 100644 tunnel-doctor/references/dns_resolver_chain_stall.md diff --git a/.pii-path-patterns b/.pii-path-patterns index 81bb463f..1898ca40 100644 --- a/.pii-path-patterns +++ b/.pii-path-patterns @@ -2,5 +2,3 @@ (^|/)coverage(/|$) (^|/)node_modules(/|$) (^|/).*\.db$ -(^|/)wechat-article-scraper/web/backend/storage/images(/|$) -(^|/)wechat-article-scraper/web/feeds(/|$) diff --git a/COMPETITOR_MATRIX.md b/COMPETITOR_MATRIX.md deleted file mode 100644 index eed46137..00000000 --- a/COMPETITOR_MATRIX.md +++ /dev/null @@ -1,137 +0,0 @@ -# 竞品功能对比矩阵 v91 - -## 评估标准 - -- ✅ 已实现且对标 -- ⚠️ 已实现但有差距 -- ❌ 未实现 -- ❓ 未知（需测试） - -## 核心功能矩阵 - -| 功能类别 | 功能点 | WeChat Scraper | Omnivore | Wallabag | Matter | Readwise | 优先级 | -|---------|-------|----------------|----------|----------|--------|----------|--------| -| **抓取** | 微信文章抓取 | ⚠️ 6级策略但无规则库 | ❌ | ❌ | ❌ | ❌ | P0 | -| | 通用网页抓取 | ⚠️ 基础实现 | ✅ Readability | ✅ Graby | ✅ | ❌ | P0 | -| | 站点规则(ftr-site-config) | ✅ Round 95 已完成 | ✅ | ✅ | ❌ | ❌ | P1 | -| | 反爬处理 | ⚠️ 基础 | ✅ 代理池 | ⚠️ 简单 | ✅ | ❌ | P1 | -| | 图片下载 | ✅ | ✅ | ✅ | ✅ | ❌ | P1 | -| | 视频抓取 | ❌ | ⚠️ 有限 | ❌ | ✅ | ❌ | P2 | -| | 抓取成功率 | 🔄 测试中 | ~95% | ~90% | ~95% | N/A | P0 | -| **阅读** | 沉浸式阅读器 | ✅ | ✅ | ✅ | ✅ | ✅ | P0 | -| | 主题切换 | ❌ | ✅ | ✅ | ✅ | ✅ | P1 | -| | 字体调节 | ⚠️ 基础 | ✅ | ✅ | ✅ | ✅ | P1 | -| | 进度同步 | ✅ | ✅ | ⚠️ | ✅ | ✅ | P0 | -| | 离线阅读 | ⚠️ PWA缓存 | ✅ | ✅ | ✅ | ✅ | P1 | -| | 全文搜索 | ⚠️ pgvector但未验证 | ✅ | ✅ | ✅ | ✅ | P0 | -| **批注** | 文本高亮 | ✅ | ✅ | ⚠️ | ✅ | ✅ | P0 | -| | 添加评论 | ✅ | ✅ | ❌ | ✅ | ✅ | P0 | -| | 标签系统 | ✅ | ✅ | ✅ | ⚠️ | ✅ | P1 | -| | 定位准确率 | 🔄 测试中 | ~99% | N/A | ~95% | N/A | P0 | -| | 批注导出 | ⚠️ Markdown | ✅ | ✅ | ✅ | ✅ | P1 | -| | 社交批注 | ❌ | ✅ | ❌ | ⚠️ | ❌ | P2 | -| **TTS** | 文本朗读 | ✅ Round 98 Edge TTS | ❌ | ❌ | ✅ | ❌ | P1 | -| | 语速调节 | ⚠️ | N/A | N/A | ✅ | N/A | P1 | -| | 离线TTS | ⚠️ Web Speech | N/A | N/A | ✅ | N/A | P2 | -| **同步** | 云同步 | ✅ Supabase | ✅ | ✅ | ✅ | ✅ | P0 | -| | 多设备 | ✅ | ✅ | ✅ | ✅ | ✅ | P0 | -| | 冲突解决 | ⚠️ 简单 | ✅ | ⚠️ | ✅ | ✅ | P1 | -| | 同步速度 | ❓ 未测 | <1s | ~3s | <1s | <1s | P1 | -| **导出** | Markdown | ✅ | ✅ | ✅ | ✅ | ✅ | P0 | -| | PDF | ✅ | ✅ | ✅ | ❌ | ❌ | P1 | -| | Notion | ✅ | ❌ | ❌ | ❌ | ❌ | P1 | -| | Obsidian | ✅ | ❌ | ❌ | ❌ | ❌ | P1 | -| | Readwise | ⚠️ | ❌ | ❌ | ❌ | ✅ 原生 | P2 | -| | Kindle推送 | ❌ | ❌ | ⚠️ | ❌ | ❌ | P2 | -| **发现** | 邮件订阅 | ❌ | ✅ | ❌ | ❌ | ✅ | P2 | -| | 微信转发收藏 | ✅ MiniApp+Official | ❌ | ❌ | ✅ | ❌ | P0 | -| | 浏览器插件 | ✅ | ✅ | ✅ | ✅ | ✅ | P0 | -| | 移动端App | ⚠️ PWA | ✅ 原生 | ❌ | ✅ 原生 | ✅ | P1 | -| **工程化** | 单元测试 | 🔄 ~5% | ✅ 80%+ | ✅ 70%+ | ✅ | ✅ | P0 | -| | E2E测试 | 🔄 Stagehand | ✅ | ✅ | ✅ | ✅ | P0 | -| | CI/CD | ✅ GitHub Actions | ✅ | ✅ | ✅ | ✅ | P1 | -| | 错误监控 | ⚠️ Sentry配置 | ✅ | ✅ | ✅ | ✅ | P1 | -| | 性能监控 | ⚠️ Health check | ✅ | ✅ | ⚠️ | ✅ | P1 | -| | 文档完善度 | ⚠️ | ✅ | ✅ | ✅ | ✅ | P2 | - -## 关键差距分析 - -### P0（关键差距） - -1. **抓取成功率未知** - - 没有基准测试数据 - - 需要建立测试集（100篇各类型文章） - -2. **批注定位准确率未测** - - 核心功能没有量化指标 - - 需要自动化测试验证 - -3. **工程化严重不足** - - 测试覆盖率 5% vs 竞品 70-80% - - 无 CI/CD 流水线 - -4. **~~微信生态集成~~** ✅ Round 92 已完成 - - ~~Cubox/Matter 支持微信转发即收藏~~ - - ~~我们需要复制链接 → 粘贴 → 抓取（3步 vs 1步）~~ - - 已实现：小程序转发 + 公众号消息双轨集成 - -### P1（重要差距） - -1. **TTS朗读** - - Matter 的核心差异化 - - 刚实现，未验证 - -2. **站点规则(ftr-site-config)** - - Omnivore/Wallabag 使用社区规则库 - - 我们自研策略，维护成本高 - -3. **主题/字体定制** - - 竞品都支持，我们缺失 - -### P2（次要差距） - -1. 视频抓取 -2. 邮件订阅 -3. Kindle推送 - -## 行动计划 - -### 阶段1：验证现有功能（2周） -- [x] ~~建立100篇文章测试集~~ ✅ Round 93 已完成 -- [x] ~~测量抓取成功率~~ 🔄 Round 93 已建立测试框架 -- [x] ~~测量批注定位准确率~~ 🔄 Round 93 已建立测试框架 -- [ ] 完成E2E测试覆盖核心流程 - -### 阶段2：补齐关键差距（4周） -- [ ] 提升测试覆盖率至60% -- [ ] 集成 ftr-site-config 规则库 -- [ ] 验证TTS功能 -- [x] ~~CI/CD流水线~~ ✅ Round 94 已完成 - -### 阶段3：差异化功能（4周） -- [x] ~~微信转发集成方案~~ ✅ Round 92 已完成 -- [ ] 主题系统 -- [ ] 性能优化 - -## 测试数据收集 - -需要收集的指标： - -``` -抓取成功率 = 成功抓取数 / 总尝试数 × 100% -目标: ≥ 95% - -批注定位准确率 = 正确定位数 / 总批注数 × 100% -目标: ≥ 98% - -API响应时间 (p95) -目标: ≤ 200ms - -同步冲突率 = 冲突次数 / 总同步数 × 100% -目标: ≤ 0.1% -``` - ---- - -最后更新: Round 91 -下次更新: Round 100 (完成验证后) diff --git a/feishu-doc-scraper/.security-scan-passed b/feishu-doc-scraper/.security-scan-passed new file mode 100644 index 00000000..3f832585 --- /dev/null +++ b/feishu-doc-scraper/.security-scan-passed @@ -0,0 +1,4 @@ +Security scan passed +Scanned at: 2026-05-07T15:42:42.227427 +Tool: gitleaks + pattern-based validation +Content hash: 21b07848405443441a93068f56b3c1c4a39681ece93b21e18b56e0796368128f diff --git a/feishu-doc-scraper/SKILL.md b/feishu-doc-scraper/SKILL.md new file mode 100644 index 00000000..3d149904 --- /dev/null +++ b/feishu-doc-scraper/SKILL.md @@ -0,0 +1,453 @@ +--- +name: feishu-doc-scraper +description: Save Feishu Docs and Feishu Wiki pages as clean Markdown from a live authenticated browser session. This skill should be used when the user asks to "save this Feishu doc as markdown", "scrape/export a Feishu wiki", "导出飞书文档", "保存飞书到 markdown", "把 Chrome 里的飞书页面存成 md", or wants a Feishu page archived locally with high fidelity. Use it proactively whenever the source is a Feishu document and correctness matters, even if the user only says clipping, archiving, or converting the page. +compatibility: Requires at least one browser automation surface with access to an authenticated local browser session. Prefer Browser Use or Chrome DevTools MCP. Use Computer Use when DOM-native tooling cannot reach the content. +argument-hint: [feishu-url-or-output-path] +--- + +# Feishu Doc Scraper + +Save a Feishu document from an authenticated browser into clean Markdown. Treat the live rendered page as the source of truth and verify coverage before closing the task. + +## Hard Rules + +- Reuse the browser tab that is already logged in whenever possible. Do not assume a fresh browser session has access. +- Treat the sidebar table of contents as the coverage contract. If the page exposes a TOC, every meaningful TOC heading must land in the saved Markdown. +- Treat clipboard copy as unavailable when Feishu shows a copy-permission warning. Do not waste cycles trying the same blocked path repeatedly. +- Treat Web Clipper as non-authoritative on virtual-scroll or lazy-rendered Feishu pages. If it misses headings, sections, or most of the word count, discard it as a primary source. +- Remove UI noise. Do not keep comments, "you may also ask", support footer links, upload logs, or other shell UI around the document body. +- Do not invent missing cells or missing paragraphs. If a table is hard to recover precisely, keep a lossless textual representation instead of guessing. +- Finish only after running the bundled heading coverage check or an equivalent manual coverage pass. +- **Do not use zoom < 1 to force more rendering.** Zooming out causes `bear-virtual-renderUnit-placeholder` cells in tables, producing empty or corrupted table rows. Keep zoom at 1.0. +- **Trust the DOM class.** Do not promote `docx-text-block` or `docx-quote-block` to headings just because they look like headings visually. Only `docx-heading1/2/3-block` become `#/##/###`. +- **No document-specific heuristics.** Do not add rules that match specific text, keywords, or document structure. The same code must work for any Feishu document without modification. + +## Workflow + +### 1. Probe the page + +Capture the ground truth before extracting: + +- Record document title and source URL. +- Check whether the page is authenticated and readable. +- Capture visible word count if Feishu shows one. +- Capture the sidebar table of contents if present. +- Detect copy restriction banners early. +- Detect virtual scrolling or lazy rendering early. + +#### Detecting virtual scroll (critical) + +Run this diagnostic to determine whether the page uses virtual rendering: + +```javascript +() => { + // Method 1: compare TOC count vs rendered heading count + const tocItems = document.querySelectorAll('.catalogue__list-item, .catalog-item, [class*="catalog-item"]').length; + const renderedHeadings = document.querySelectorAll('.docx-heading2-block, .docx-heading3-block, .docx-heading1-block').length; + + // Method 2: check for loading containers + const loadingBlocks = document.querySelectorAll('.docx-block-loading-container, [class*="loading"]').length; + + // Method 3: check total block count vs expected scale + const totalBlocks = document.querySelectorAll('.block').length; + + // Method 4: identify the real scroll container (not window) + const scrollContainer = document.querySelector('.bear-web-x-container, .page-main, .content-scroller, .docx-width-mode-standard, [class*="docx-width"]'); + + return { + tocItems, + renderedHeadings, + loadingBlocks, + totalBlocks, + hasVirtualScroll: tocItems > renderedHeadings + 2 || loadingBlocks > 0 || totalBlocks < 10, + scrollContainerClass: scrollContainer?.className?.substring(0, 80) || 'window', + scrollContainerTextLength: scrollContainer?.innerText?.length || 0 + }; +} +``` + +**Interpretation:** +- `tocItems >> renderedHeadings` → virtual scroll confirmed. The sidebar shows more sections than are in the DOM. +- `loadingBlocks > 0` → lazy-rendered content exists that has not been fetched. +- `totalBlocks < 10` on a long document → most content is virtualized. +- `scrollContainerClass` shows the real scroll target. Feishu usually scrolls a nested div, not `window`. + +If virtual scroll is detected, **do not** rely on `window.scrollTo`. You must use TOC-driven section extraction (see §3). + +If the output path is genuinely ambiguous, use `AskUserQuestion` when available. If it is not available, ask one concise question. Otherwise choose a repo-appropriate archival location and proceed. + +### 2. Choose the acquisition path + +Read [references/tooling-matrix.md](references/tooling-matrix.md) before selecting tools. Use this order: + +1. DOM or accessibility extraction with Browser Use or Chrome DevTools. +2. Computer Use with accessibility snapshots and anchor-by-anchor navigation. +3. Screenshot-assisted manual extraction only when the richer paths cannot reach the content. + +Do not use Web Clipper as the main extraction path on Feishu virtual-scroll documents. It may be used as a weak cross-check on simple, fully rendered pages, but never as the acceptance signal. + +### 3. Extract section by section + +Default to TOC-driven extraction. This is the only reliable path for Feishu virtual-scroll documents. + +#### 3a. Collect the TOC + +Extract the sidebar TOC as a structured list with both text and clickable elements: + +```javascript +() => { + const tocItems = Array.from(document.querySelectorAll('.catalogue__list-item, .catalog-item, [class*="catalog-item"]')); + return tocItems.map((item, idx) => { + const textEl = item.querySelector('.wiki-ssr-sidebar__catalog-item-text, [class*="catalog-item-text"], [class*="title"]') || item; + return { + index: idx, + text: textEl.innerText?.trim() || '', + element: item, + clickable: item.querySelector('a, button, [role="button"]') || item + }; + }).filter(item => item.text.length > 0); +} +``` + +If the sidebar itself is lazy-loaded (shows only first few items), scroll the sidebar container first to load all TOC items. + +#### 3b. TOC-driven click-and-capture loop + +For each TOC item: + +1. **Click the TOC item** to jump to that section: + ```javascript + tocItem.click(); // or tocItem.querySelector('a').click() + ``` + Wait 2.5 seconds for Feishu to render the target section. + +2. **Capture the newly rendered blocks** in the main content area. The key is to capture **all blocks that belong to this section**, including those below the heading until the next heading. Convert each DOM block to Markdown immediately during capture — do not store raw `innerText` and assume downstream rendering will fix it. + + ```javascript + () => { + const blocks = Array.from(document.querySelectorAll('.block')); + const headingBlock = blocks.find(b => + b.className.includes('heading') && + b.innerText?.trim().includes('YOUR_HEADING_TEXT') + ); + const headingIndex = blocks.indexOf(headingBlock); + const nextHeadingIndex = blocks.findIndex((b, i) => + i > headingIndex && b.className.includes('heading') + ); + const sectionBlocks = blocks.slice( + headingIndex, + nextHeadingIndex === -1 ? undefined : nextHeadingIndex + ); + return sectionBlocks.map(b => domBlockToMarkdown(b)); + } + + function domBlockToMarkdown(block) { + const cls = block.className || ''; + const text = block.innerText?.trim()?.replace(/[​]/g, '') || ''; + + // Headings — trust the DOM class, never guess + if (cls.includes('docx-heading1-block')) return '# ' + text; + if (cls.includes('docx-heading2-block')) return '## ' + text; + if (cls.includes('docx-heading3-block')) return '### ' + text; + + // Tables — handled separately by the table merger; skip here + if (cls.includes('docx-table-block')) return null; + + // Lists — skip parent blocks that contain nested children + if (cls.includes('docx-bullet-block') || cls.includes('docx-list-block')) { + const hasNested = block.querySelectorAll('.docx-bullet-block, .docx-list-block').length > 0; + if (hasNested) return null; // parent with nested bullets handled by extractBullets + + const depthClass = cls.match(/indent-(\d+)/); + const depth = depthClass ? parseInt(depthClass[1]) : 0; + const indent = ' '.repeat(depth); + return indent + '- ' + text.replace(/^[•◦]\s*/, ''); + } + + // Text / Quote / All other blocks — preserve inline formatting only + return inlineMarkdown(block); + } + + function inlineMarkdown(node) { + // Walk the DOM tree, converting inline tags to Markdown without + // changing the block-level semantics. Bold → **, italic → *. + let result = ''; + for (const child of node.childNodes) { + if (child.nodeType === 3) { + result += child.textContent; + } else if (child.nodeType === 1) { + const tag = child.tagName.toLowerCase(); + const inner = inlineMarkdown(child); + if (tag === 'b' || tag === 'strong') result += `**${inner}**`; + else if (tag === 'i' || tag === 'em') result += `*${inner}*`; + else if (tag === 'u') result += `${inner}`; + else if (tag === 'br') result += '\n'; + else result += inner; // span, a, etc. — unwrap but keep text + } + } + return result.replace(/\n+$/, ''); + } + ``` + +3. **Handle nested scroll within a section**: Some sections (especially those with tables) span multiple "pages" in Feishu's virtual scroll. After clicking a TOC item, scroll the **main content scroll container** (not window) in increments to reveal more blocks: + ```javascript + // Use the same scroll container detected in the probing phase + const scrollContainer = document.querySelector('.bear-web-x-container, .page-main, .content-scroller, [class*="docx-width"]'); + scrollContainer.scrollBy(0, scrollContainer.clientHeight * 0.7); + ``` + After each scroll, wait 600ms, then capture any new `.block` elements (deduplicate by `data-block-id`). + +4. **Store in manifest**: Add the captured blocks to the manifest under the current heading. Keep overlap between sections — deduplicate later by `data-block-id`. + + **Do not promote text blocks to headings.** If a section has sub-sections that are not in the sidebar TOC, they are either: + - Rendered as real `docx-heading3-block` (captured naturally), or + - Styled text inside the body (keep as body text with inline formatting). + +#### 3c. Nested bullet extraction (critical) + +Feishu renders nested lists as a parent `.docx-bullet-block` containing child `.docx-bullet-block` elements inside `.list-children`. Extract them recursively: + +```javascript +function extractBullets(el, depth = 0) { + const results = []; + + // Extract parent text from .list-content or .ace-line + const listContent = el.querySelector('.list-content, .ace-line'); + if (listContent) { + const text = inlineMarkdown(listContent).replace(/^[•◦]\s*/, '').trim(); + if (text) results.push({depth, text}); + } + + // Find direct child bullet blocks (descendants whose closest bullet ancestor is el itself) + const allNested = el.querySelectorAll('.docx-bullet-block, .docx-list-block'); + const directChildren = Array.from(allNested).filter(b => { + const parent = b.parentElement?.closest('.docx-bullet-block, .docx-list-block'); + return b !== el && parent === el; + }); + + directChildren.forEach(child => { + results.push(...extractBullets(child, depth + 1)); + }); + + return results; +} +``` + +In the main capture loop, skip nested bullets (they're handled by their parent): + +```javascript +const parentBullet = el.closest('.docx-bullet-block, .docx-list-block'); +const isNested = parentBullet && parentBullet !== el; +if (isNested) return; // parent will handle this bullet +``` + +#### 3d. Table extraction (critical) + +Feishu tables render as nested DOM structures inside `.docx-table-block`. **Do not** rely on `innerText` for tables — it loses column alignment and includes newlines. Tables must be converted to Markdown table format in the manifest body. + +**Critical: Skip blocks inside tables.** When querying `.block`, table cell blocks may also match. Exclude any `.block` that is a descendant of `.docx-table-block` unless it IS the `.docx-table-block` itself: + +```javascript +function isInsideTable(el) { + return !!el.closest('.docx-table-block, .table-block'); +} + +// In capture loop: +if (isInsideTable(block) && !block.className.includes('docx-table-block')) return; +``` + +**Virtual scroll splits tables**: A single logical table may be rendered as multiple `.docx-table-block` instances across virtual scroll boundaries. Merge them by matching the header row (first row) as a key — if two consecutive table blocks share the same header, they are parts of the same table. Concatenate their body rows (skip duplicate headers). + +Extract table structure explicitly: + +```javascript +function extractTable(tableBlock) { + const rows = []; + + tableBlock.querySelectorAll('tr, .docx-table-tr').forEach(rowEl => { + const cells = Array.from(rowEl.querySelectorAll('td, .table-cell-block, .docx-table_cell-block')) + .map(cell => cell.innerText?.trim()?.replace(/[​\n]/g, '') || '') + .filter(c => c !== ''); + if (cells.length > 0) rows.push(cells); + }); + + return rows; +} +``` + +Convert cell arrays to Markdown table rows **before** storing in the manifest: + +```javascript +function tableToMarkdown(rows) { + if (!rows || rows.length === 0) return []; + const header = '| ' + rows[0].join(' | ') + ' |'; + const divider = '|' + rows[0].map(() => '---').join('|') + '|'; + const body = rows.slice(1).map(r => '| ' + r.join(' | ') + ' |'); + return [header, divider, ...body]; +} +``` + +Store in the manifest as Markdown table lines: +```json +{ + "heading_level": 3, + "heading": "课程总览（两天标准版）", + "body": [ + "| 时间 | 模块 | 讲师 |", + "|---|---|---|", + "| 9:40-10:00 | 签到 | — |", + "| 10:00-12:00 | 模块一：AI 营销战略课 | Jett |" + ] +} +``` + +**Preserve table structure as-is.** Do not split or rearrange table rows based on content heuristics. If the source document contains a single table, the output must contain a single Markdown table. + +#### 3e. Fallback when TOC is missing or empty + +If there is no TOC: + +- Build a manual heading list while traversing top-to-bottom. +- Use repeated snapshots and stable scroll increments **on the real scroll container** (not window). +- Stop only when the bottom of the document is reached and the content no longer changes. + +### 4. Normalize into Markdown + +Use `scripts/build_feishu_markdown.py` when a structured manifest helps. The manifest format is documented in [references/capture-manifest.md](references/capture-manifest.md). + +Normalize with these rules: + +- Keep frontmatter minimal: `title`, `source`, `author`, `published`, `created`, `description`, `tags`. +- Preserve heading hierarchy. +- Render stable tables as Markdown tables. +- If table structure is ambiguous, keep it as labeled text blocks instead of fabricating cells. +- Collapse duplicated section fragments introduced by anchor overlap. +- Exclude all non-document chrome. + +### 5. Sort by data-block-id + +After capturing all blocks across all TOC sections, sort them by numeric `data-block-id` before generating Markdown. Feishu assigns numeric IDs in document logical order (lower = earlier in document). This is the most reliable way to reconstruct document order when virtual scroll has reordered the DOM. + +```javascript +const sortedBlocks = Array.from(allBlocks.values()).sort((a, b) => { + const aid = typeof a.id === 'number' ? a.id : parseInt(a.id); + const bid = typeof b.id === 'number' ? b.id : parseInt(b.id); + return aid - bid; +}); +``` + +### 6. Verify coverage + +Run `scripts/check_heading_coverage.py` against the final Markdown and the expected heading list: + +```bash +python3 scripts/check_heading_coverage.py \ + --markdown-file /path/to/output.md \ + --headings-file /path/to/expected-headings.txt +``` + +If the check reports missing headings: + +- revisit those anchors +- re-extract the missing sections +- rebuild the Markdown +- rerun the coverage check + +#### Enhanced coverage checks (run these inline) + +**Check 1: Section body completeness** + +Verify that every heading in the output has non-empty body content. Empty sections (heading only) are a strong signal that virtual-scroll content was not captured: + +```bash +python3 -c " +import re, sys +md = open(sys.argv[1]).read() +headings = re.findall(r'^(#{2,6})\s+(.+)$', md, re.M) +for level, title in headings: + # Find content between this heading and next heading + pattern = rf'{re.escape(level)}\s+{re.escape(title)}\n\n(.+?)(?=\n#{1,6}\s|\Z)' + match = re.search(pattern, md, re.S) + body = match.group(1).strip() if match else '' + if len(body) < 20: + print(f'WARNING: empty section: {title}') +" /path/to/output.md +``` + +**Check 2: Scale validation** + +Cross-check the result against the page-level word count when Feishu exposes one. Also compare against the DOM text length captured during probing: + +| Metric | Source | Acceptance | +|--------|--------|------------| +| TOC heading count | Sidebar | Must equal output heading count | +| Section body non-empty | Output | >95% of sections must have body >20 chars | +| Table presence | TOC keywords | "总览"/"overview"/"schedule" → output must have `\|` tables | +| Word count | Feishu UI (if shown) | Output within 20% of page count | + +**Large divergence is a failure signal and requires another extraction pass.** Do not declare completion if >20% of sections are empty or if expected tables are missing. + +### 7. Deduplicate after capture + +Virtual scroll unloads and re-renders blocks, which can cause the same logical content to appear with different `data-block-id` values. Two common duplicate patterns: + +1. **Table cell blocks leaking as text**: A `.docx-table_cell-block` that escapes the `isInsideTable` filter appears as a standalone text line. Remove any text block whose text exactly matches a table cell value. + +2. **Nested bullet children rendered without parent**: When a parent bullet block is unloaded but its children remain visible, those children get captured as standalone bullets. Remove any bullet/text block whose text exactly matches a bullet already present inside a `bullets`-type block. + +Run deduplication **after sorting by `data-block-id`** but **before generating Markdown**: + +```javascript +// Build a set of all texts that are already accounted for in structured blocks +const coveredTexts = new Set(); +sortedBlocks.forEach(b => { + if (b.type === 'table') { + b.tableRows.forEach(row => row.forEach(cell => coveredTexts.add(cell))); + } + if (b.type === 'bullets') { + b.bulletList.forEach(item => coveredTexts.add(item.text)); + } +}); + +// Filter out standalone blocks that are already covered +const dedupedBlocks = sortedBlocks.filter(b => { + if (b.type === 'text' || b.type === 'bullet') { + return !coveredTexts.has(b.text); + } + return true; +}); +``` + +Then generate Markdown from `dedupedBlocks` instead of `sortedBlocks`. + +## Failure Patterns + +Read [references/history-derived-rules.md](references/history-derived-rules.md) when the page behaves strangely. The important patterns are: + +- copy restrictions make clipboard paths dead ends +- virtual scrolling makes one-shot extraction incomplete +- extension output can look plausible while silently dropping sections +- TOC coverage plus visible word count is the most reliable acceptance pair +- **zoom < 1 causes table placeholders** — never zoom out to force rendering +- **blocks inside tables must be skipped** or they pollute the output as duplicate text +- **`data-block-id` numeric ordering** is more reliable than DOM order for reconstructing document sequence + +## Output Contract + +Deliver: + +- one clean Markdown file +- the original source URL in frontmatter +- headings that cover the document body +- no UI noise +- a verified coverage result + +If the user asks for local archival only, stop there. If they also want a repo note integrated into a larger knowledge system, place it in the repo-appropriate clipping or reference location after the Markdown is verified. + +## Resources + +- [references/tooling-matrix.md](references/tooling-matrix.md): tool selection and fallback ladder +- [references/capture-manifest.md](references/capture-manifest.md): manifest shape for structured rendering +- [references/history-derived-rules.md](references/history-derived-rules.md): battle-tested rules distilled from local Feishu scraping sessions +- `scripts/build_feishu_markdown.py`: render a structured capture manifest into final Markdown +- `scripts/check_heading_coverage.py`: verify TOC heading coverage and detect common UI noise diff --git a/feishu-doc-scraper/references/capture-manifest.md b/feishu-doc-scraper/references/capture-manifest.md new file mode 100644 index 00000000..994380d8 --- /dev/null +++ b/feishu-doc-scraper/references/capture-manifest.md @@ -0,0 +1,53 @@ +# Capture Manifest + +Use `scripts/build_feishu_markdown.py` when extraction is easier to stage as structured data before rendering. + +## Minimal Shape + +```json +{ + "title": "Document title", + "source": "https://example.feishu.cn/wiki/...", + "author": ["Author A", "Author B"], + "published": "", + "created": "2026-05-07", + "description": "Short summary", + "tags": ["clippings", "feishu"], + "sections": [ + { + "heading_level": 1, + "heading": "Main Heading", + "body": [ + "Paragraph one.", + "- Bullet item", + "| Col A | Col B |", + "| --- | --- |", + "| A1 | B1 |" + ] + } + ] +} +``` + +## Field Rules + +- `title`: required +- `source`: strongly recommended +- `author`: string or array of strings +- `published`: optional +- `created`: optional, defaults to today only if the caller sets it +- `description`: optional +- `tags`: optional, string or array +- `sections`: required array +- `heading_level`: optional, defaults to `2` +- `body`: string or array of Markdown blocks + +## Rendering Command + +```bash +python3 scripts/build_feishu_markdown.py \ + --input /path/to/capture.json \ + --output /path/to/output.md +``` + +If `--output` is omitted, the renderer prints Markdown to stdout. diff --git a/feishu-doc-scraper/references/history-derived-rules.md b/feishu-doc-scraper/references/history-derived-rules.md new file mode 100644 index 00000000..71581fff --- /dev/null +++ b/feishu-doc-scraper/references/history-derived-rules.md @@ -0,0 +1,69 @@ +# History-Derived Rules + +These rules were distilled from repeated local Feishu scraping sessions and follow verified behavior rather than guesswork. + +## Rule 1: Copy Warnings Mean Clipboard Is Dead + +If Feishu shows a banner saying copying is restricted, treat clipboard extraction as blocked. Do not keep retrying `Cmd+C`, browser copy commands, or "copy all" variants as the main plan. + +## Rule 2: Virtual Scroll Breaks One-Shot Extraction + +Feishu wiki and doc pages often virtual-render only the visible region plus a small buffer. Any extractor that reads "the page" once can silently miss later sections. + +Implication: + +- never trust a single pass +- **always use the real scroll container**, not `window.scrollTo`. Feishu scrolls a nested div (usually `.bear-web-x-container`, `.page-main`, or `[class*="docx-width"]`). Scrolling `window` does nothing. +- **click TOC items to trigger section rendering**, not just scroll. Feishu responds to TOC clicks by fetching and rendering the target section's blocks. +- after each TOC click, wait 2.5s for rendering, then capture all `.block` elements between the target heading and the next heading +- some sections span multiple virtual "pages" — scroll the content container in increments after clicking, capturing new blocks each time +- deduplicate blocks by `data-block-id` to avoid double-counting overlap + +## Rule 3: Web Clipper Can Look Correct While Still Being Incomplete + +Extension output can capture only the rendered subset and still produce plausible Markdown or HTML. Plausibility is not acceptance. + +Implication: + +- treat Web Clipper as non-authoritative on virtual-scroll pages +- if TOC headings or word count do not line up, discard it as the main source + +## Rule 4: TOC Coverage Is the Best Section-Level Contract + +The left sidebar TOC is the most reliable list of meaningful document sections. Use it as the checklist for coverage validation. + +## Rule 5: Remove UI Noise Aggressively + +Common Feishu noise to delete: + +- comments +- "you may also ask" +- support footer items +- upload logs +- "contact support" +- recommendation panels +- empty interaction controls + +## Rule 6: Validate Against Scale, Not Exact Word Count + +When Feishu shows a visible word count, use it as a scale check. A final Markdown body that is dramatically shorter than the page count is probably incomplete even if the saved file looks tidy. + +## Rule 7: Trust the DOM Class, Do Not Promote Text Blocks to Headings + +If the sidebar TOC does not list a sub-section, it is not a heading. Feishu sometimes styles body text as bold to make it *look* like a heading, but the DOM class remains `docx-text-block` or `docx-quote-block`. Respect the DOM class: only `docx-heading1/2/3-block` become `#/##/###`. Bold body text stays as body text with inline `**` formatting. + +## Rule 8: Zoom < 1 Causes Table Placeholders + +Do not zoom out to force more content into the viewport. At zoom levels below 1.0, Feishu renders `bear-virtual-renderUnit-placeholder` inside table cells, producing empty or corrupted rows. Keep zoom at 1.0 and rely on TOC-driven section extraction instead. + +## Rule 9: Skip Blocks Inside Tables + +When querying `.block`, table cell blocks (`docx-table_cell-block`, `.table-cell-block`) also match. If not excluded, they appear as duplicate standalone text blocks in the output, polluting the markdown with table cell values outside the table. Exclude any block whose closest `.docx-table-block` ancestor is not itself. + +## Rule 10: Use data-block-id Numeric Order for Document Sequence + +Virtual scroll unloads and re-renders blocks, which can reorder the DOM. `compareDocumentPosition` and DOM order are unreliable. Feishu assigns numeric `data-block-id` values in document logical order (lower = earlier). Sort captured blocks by numeric `data-block-id` before generating markdown. + +## Rule 11: Nested Bullets Have Parent-Child DOM Structure + +Feishu nested lists use a parent `.docx-bullet-block` containing `.list-children` with child `.docx-bullet-block` elements. Extract parent text from `.list-content` or `.ace-line`, then recursively extract direct child bullets. Skip child bullets in the main capture loop (they're handled by their parent). diff --git a/feishu-doc-scraper/references/tooling-matrix.md b/feishu-doc-scraper/references/tooling-matrix.md new file mode 100644 index 00000000..c3910b30 --- /dev/null +++ b/feishu-doc-scraper/references/tooling-matrix.md @@ -0,0 +1,92 @@ +# Tooling Matrix + +Use the strongest browser surface available. Prefer data-bearing surfaces over purely visual ones. + +## Tool Order + +1. Browser Use +2. Chrome DevTools MCP +3. Computer Use +4. Screenshots plus manual extraction + +## Selection Rules + +### Browser Use + +Use when: + +- the harness can open or inspect the authenticated tab +- page text or semantic page reads are available +- element targeting is stable enough for anchor navigation + +Strengths: + +- direct page text access +- lower friction for repeated section capture +- easier local verification on browser state + +Weaknesses: + +- may not preserve every table structure +- may still be subject to virtual-scroll partial rendering + +### Chrome DevTools MCP + +Use when: + +- DOM or accessibility snapshots are needed +- anchor navigation needs scripted control +- section content is present in the page tree but not easy to copy visually +- **you need to identify the real scroll container and execute per-section extraction on virtual-scroll pages** + +Strengths: + +- structured snapshots +- scripted evaluation +- good for repeated per-anchor extraction +- **can run diagnostic scripts to detect virtual scroll and identify the true scroll container** +- **can programmatically click TOC items and capture newly rendered blocks** + +Weaknesses: + +- dynamic Feishu rendering can still hide unloaded sections +- requires careful re-snapshotting after each navigation +- **requires explicit waiting (600-1000ms) after TOC clicks for section rendering** + +### Computer Use + +Use when: + +- the page is already open in a real browser and authenticated there +- DOM-native tooling cannot attach or cannot read the content reliably +- the task depends on real browser state such as local extensions, cookies, or corporate login flows + +Strengths: + +- sees the same authenticated browser the user sees +- works even when browser-internal APIs are unavailable +- useful for TOC clicking and visual confirmation + +Weaknesses: + +- slower +- more sensitive to UI drift +- requires explicit verification after every major interaction + +## Rejected Primary Paths + +Do not use these as the main capture path on Feishu docs: + +- Web Clipper on virtual-scroll pages +- clipboard copy after a copy restriction warning +- one-shot "read the whole page" attempts without TOC coverage checking + +## Acceptance Signal + +Accept the scrape only when all of these are true: + +- the final Markdown covers the expected TOC headings +- the final body roughly matches the document's visible word-count scale when Feishu exposes one +- **>95% of sections have non-empty body content** (empty headings are a sign of missed virtual-scroll content) +- **tables present in TOC ("总览", "overview", "schedule") are captured as Markdown tables** in the output +- no `docx-block-loading-container` elements remain unvisited in the DOM diff --git a/feishu-doc-scraper/scripts/build_feishu_markdown.py b/feishu-doc-scraper/scripts/build_feishu_markdown.py new file mode 100755 index 00000000..5bbd4877 --- /dev/null +++ b/feishu-doc-scraper/scripts/build_feishu_markdown.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 +""" +Render a structured Feishu capture manifest into Markdown. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + + +def to_list(value): + if value is None: + return [] + if isinstance(value, list): + return [str(item) for item in value if str(item).strip()] + return [str(value)] + + +def yaml_lines(manifest): + lines = ["---"] + simple_fields = [ + "title", + "source", + "published", + "created", + "description", + ] + for field in simple_fields: + value = manifest.get(field, "") + lines.append(f'{field}: "{str(value).replace(chr(34), chr(39))}"' if value else f"{field}:") + + for key in ("author", "tags"): + values = to_list(manifest.get(key)) + if values: + lines.append(f"{key}:") + for value in values: + lines.append(f' - "{value.replace(chr(34), chr(39))}"') + else: + lines.append(f"{key}:") + + lines.append("---") + return lines + + +def normalize_body(body): + if body is None: + return [] + if isinstance(body, list): + return [str(block).strip() for block in body if str(block).strip()] + text = str(body).strip() + return [text] if text else [] + + +def section_lines(section): + level = int(section.get("heading_level", 2)) + level = max(1, min(level, 6)) + heading = str(section.get("heading", "")).strip() + if not heading: + raise ValueError("Section heading is required") + + lines = [f'{"#" * level} {heading}'] + body_blocks = normalize_body(section.get("body")) + if body_blocks: + lines.append("") + lines.extend(body_blocks) + return lines + + +def render_markdown(manifest): + title = str(manifest.get("title", "")).strip() + if not title: + raise ValueError("Manifest title is required") + + sections = manifest.get("sections") + if not isinstance(sections, list) or not sections: + raise ValueError("Manifest sections must be a non-empty list") + + lines = yaml_lines(manifest) + lines.extend(["", f"# {title}"]) + + source = str(manifest.get("source", "")).strip() + if source: + lines.extend(["", f"Source: <{source}>"]) + + for section in sections: + lines.extend(["", *section_lines(section)]) + + return "\n".join(lines).rstrip() + "\n" + + +def parse_args(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--input", required=True, help="Path to capture manifest JSON") + parser.add_argument("--output", help="Optional output markdown path") + return parser.parse_args() + + +def main(): + args = parse_args() + manifest_path = Path(args.input) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + markdown = render_markdown(manifest) + + if args.output: + output_path = Path(args.output) + output_path.write_text(markdown, encoding="utf-8") + print(f"Wrote {output_path}") + else: + sys.stdout.write(markdown) + + +if __name__ == "__main__": + main() diff --git a/feishu-doc-scraper/scripts/check_heading_coverage.py b/feishu-doc-scraper/scripts/check_heading_coverage.py new file mode 100755 index 00000000..333b530e --- /dev/null +++ b/feishu-doc-scraper/scripts/check_heading_coverage.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +""" +Check that expected Feishu headings are present in the final Markdown output. +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +NOISE_PATTERNS = ( + "you may also ask", + "recommended content", + "upload logs", + "contact support", + "comments", +) + + +def normalize(text: str) -> str: + text = text.strip().lower() + text = re.sub(r"^#+\s*", "", text) + text = re.sub(r"\s+", "", text) + # Remove punctuation and special characters. Using a set avoids the + # regex escaping trap (the previous character class terminated early + # because \\] was interpreted as a literal backslash followed by ]). + _REMOVE_CHARS = set( + chr(c) for c in ( + 0x60, 0x7E, 0x7C, 0x2C, 0x2E, 0x21, 0x3F, 0x28, 0x29, + 0x5B, 0x5D, 0x3C, 0x3E, 0x300A, 0x300B, + 0x22, 0x27, 0x201C, 0x201D, 0x2018, 0x2019, + 0x5C, 0x2D, 0x2B, + 0x3A, 0xFF1A, + 0x3002, 0xFF0C, + 0xFF01, 0xFF1F, + 0xFF08, 0xFF09, + 0x2014, 0x2013, + ) + ) + text = "".join(c for c in text if c not in _REMOVE_CHARS) + return text + + +def load_expected(headings_file: Path) -> list[str]: + return [line.strip() for line in headings_file.read_text(encoding="utf-8").splitlines() if line.strip()] + + +def extract_headings(markdown_text: str) -> list[str]: + headings = [] + for line in markdown_text.splitlines(): + if re.match(r"^#{1,6}\s+\S", line): + headings.append(line.strip()) + return headings + + +def detect_noise(markdown_text: str) -> list[str]: + lowered = markdown_text.lower() + return [pattern for pattern in NOISE_PATTERNS if pattern in lowered] + + +def parse_args(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--markdown-file", required=True, help="Generated markdown file") + parser.add_argument("--headings-file", required=True, help="Plain text file with one expected heading per line") + return parser.parse_args() + + +def main(): + args = parse_args() + markdown_path = Path(args.markdown_file) + headings_path = Path(args.headings_file) + + markdown_text = markdown_path.read_text(encoding="utf-8") + expected = load_expected(headings_path) + found = extract_headings(markdown_text) + + found_index = {normalize(item): item for item in found} + missing = [item for item in expected if normalize(item) not in found_index] + noise_hits = detect_noise(markdown_text) + + print(f"Expected headings: {len(expected)}") + print(f"Found markdown headings: {len(found)}") + + if missing: + print("Missing headings:") + for item in missing: + print(f" - {item}") + + if noise_hits: + print("Noise patterns detected:") + for item in noise_hits: + print(f" - {item}") + + if missing or noise_hits: + sys.exit(1) + + print("Heading coverage check passed.") + + +if __name__ == "__main__": + main() diff --git a/tunnel-doctor/SKILL.md b/tunnel-doctor/SKILL.md index 84e9a581..6bca6a2d 100644 --- a/tunnel-doctor/SKILL.md +++ b/tunnel-doctor/SKILL.md @@ -1,6 +1,6 @@ --- name: tunnel-doctor -description: Diagnoses and fixes conflicts between Tailscale and proxy/VPN tools (Shadowrocket, Clash, Surge) on macOS. Covers five conflict layers - (1) route hijacking, (2) HTTP proxy env var interception, (3) system proxy bypass, (4) SSH ProxyCommand double tunneling, and (5) VM/container runtime proxy propagation (OrbStack/Docker). Includes SOP for remote development via SSH tunnels with proxy-safe Makefile patterns. Use when Tailscale ping works but SSH/HTTP times out, when browser returns 503 but curl works, when git push fails with "failed to begin relaying via HTTP", when Docker pull times out behind TUN/VPN, when setting up Tailscale SSH to WSL instances, or when bootstrapping remote dev environments over Tailscale. +description: Diagnoses and fixes conflicts between Tailscale and proxy/VPN tools (Shadowrocket, Clash, Surge) on macOS. Covers six conflict layers - (1) route hijacking, (2) HTTP proxy env var interception, (3) system proxy bypass, (4) SSH ProxyCommand double tunneling, (5) VM/container runtime proxy propagation (OrbStack/Docker), and (6) stalled DNS resolver in macOS getaddrinfo chain (dead VPN daemon leaving zombie utun + DNS injection). Includes SOP for remote development via SSH tunnels with proxy-safe Makefile patterns. Use when Tailscale ping works but SSH/HTTP times out, when browser returns 503 but curl works, when git push fails with "failed to begin relaying via HTTP", when Docker pull times out behind TUN/VPN, when setting up Tailscale SSH to WSL instances, when bootstrapping remote dev environments over Tailscale, when ssh/curl/git hang ~60 seconds before resolving a hostname while nslookup returns instantly, when ping to a resolver IP works but dig to the same IP times out, or when ssh -vvv freezes at "debug2: resolving" without ever reaching "debug1: connect". allowed-tools: Read, Grep, Edit, Bash --- @@ -42,6 +42,7 @@ Determine which scenario applies: - **SSH connects but `be-child ssh` exits code 1** → WSL snap sandbox issue (Step 5) - **TCP port 22 reachable (`nc -z` succeeds) but SSH fails with `kex_exchange_identification: Connection closed`** → Tailscale SSH proxy intercept on WSL (Step 5A) - **`tailscale ssh` returns "not available on App Store builds"** → Wrong Tailscale distribution on macOS (Step 5B) +- **Any tool using system DNS (`ssh`, `curl`, `git`) hangs ~60s before resolving, but `nslookup` returns instantly** → Stalled resolver in `getaddrinfo` chain (Step 2I) **Key distinctions**: - SSH does NOT use `http_proxy`/`NO_PROXY` env vars. If SSH works but HTTP doesn't → Layer 2. @@ -54,6 +55,25 @@ Determine which scenario applies: - If DNS resolves to `198.18.x.x` virtual IPs → TUN DNS hijack (Step 2H). - If `nc -z` succeeds on port 22 but SSH gets no banner (`kex_exchange_identification`) → Tailscale SSH proxy intercept (Step 5A). Confirm with `tcpdump -i any port 22` on the remote — 0 packets means Tailscale intercepts above the kernel. - If `tailscale ssh` fails with "not available on App Store builds" → install Standalone Tailscale (Step 5B). +- If `nslookup ` is fast (<0.1s) but `dscacheutil -q host -a name ` takes 60s+ → a supplemental resolver in `scutil --dns` is dead (Step 2I). +- If `ping ` succeeds but `dig @` times out → daemon dead, `utun` interface zombied. ICMP is answered by the interface; the actual port-53 service is gone (Step 2I). +- If `ssh -vvv` hangs immediately after `debug2: resolving "" port ` and never reaches `debug1: connect to address` → DNS resolution stage, not network connect stage. This is Step 2I, not Step 2B/2H. + +### Diagnosis Discipline (Read Before Committing to a Hypothesis) + +When symptoms point at a component (proxy, VPN, route table, DNS), **don't commit to a hypothesis from circumstantial evidence — verify with that component's own health endpoint first.** Each component has a one-line health check faster and more reliable than ruling out neighbors: + +| Suspected component | Authoritative health check (run this first) | +|---------------------|---------------------------------------------| +| HTTP proxy (Shadowrocket / Clash / Surge) | `curl -x http://127.0.0.1: -m 10 https://api.github.com` returns 200 | +| Tailscale daemon | `tailscale status` returns peer list (not connection error) | +| A specific DNS resolver | `dig @ +tries=1 +timeout=3 example.com` <100ms | +| Routing for an IP | `route -n get ` shows expected interface | +| Per-resolver bisection (when DNS is suspect) | The `for ns in ...; do dig @$ns ...` loop in Step 2I | + +**Why this matters**: A symptom that matches the description of Step 2X does not, by itself, prove component X is the problem. Multiple layers can produce overlapping symptoms (a 60-second hang during `git push` could be proxy node death, fakeip route corruption, or DNS resolver stall — all plausible from the user-visible symptom alone). Reaching for the most specific verification first avoids committing to a wrong layer and chasing it down a dead end. + +If the failing operation involves DNS at all, **run the per-nameserver bisection from Step 2I before suspecting proxy or routing**. It rules in/out the largest single class of macOS-on-China-network failures in under 15 seconds. ### Fast Path: Run Automated Checks @@ -569,6 +589,108 @@ IP-CIDR,192.30.252.0/22,DIRECT This is more robust but requires proxy tool config access. +### Step 2I: Fix Stalled DNS Resolver in `getaddrinfo` Chain + +**Symptom**: `ssh`, `curl` (no `-x`), `git`, and any other tool using system DNS hangs ~60 seconds before resolving. `ssh -vvv` freezes immediately after: + +``` +debug2: resolving "" port +debug3: resolve_host: lookup : +``` + +…and never reaches `debug1: connect to address`. After the wait it eventually succeeds — but every new connection pays the same penalty. `nslookup ` returns instantly (~10ms) but `dscacheutil -q host -a name ` takes 60s+. + +**Root cause**: macOS `getaddrinfo` consults every entry in `scutil --dns` whose `domain` filter matches (or has no filter at all). If one resolver's nameserver is unreachable but its interface is still in the routing table, `getaddrinfo` waits the full UDP retry timeout (typically 30-60s) before falling through to the next resolver. The most common real-world trigger is a tunneling daemon (Tailscale, Cisco AnyConnect, Pulse Secure) that crashed without unwinding its `utun` and DNS injection. + +**Why `nslookup` lies**: `nslookup` reads only `/etc/resolv.conf` (one nameserver). `dscacheutil` and `getaddrinfo` go through DirectoryService, which queries the whole resolver chain in `scutil --dns`. A divergence between these two is the smoking gun. + +**The "ping ok but DNS dead" trap**: `ping ` may answer in <1ms even when port 53 is dead, because the `utun` interface still claims the IP and replies to ICMP locally. Don't infer resolver health from `ping`. Test the actual service: `dig @ +tries=1 +timeout=3 example.com`. + +#### Diagnosis: Bisect by Nameserver + +Find the dead resolver in under 15 seconds: + +```bash +# 1. Read every resolver's nameserver, interface, and matching scope +scutil --dns | grep -E "^resolver|nameserver|domain :|search domain|if_index" + +# 2. Time each nameserver in isolation (3-second cap) +for ns in ; do + printf " %s: " "$ns" + /usr/bin/time -p dig @$ns +tries=1 +timeout=3 +short example.com 2>&1 | tr '\n' ' ' + echo +done +``` + +Healthy nameservers respond in <0.1s. The dead one returns `connection timed out; no servers could be reached` after exactly 3.01s. + +For IPv6 resolvers, run the same `dig @` test — Tailscale and several VPNs inject both v4 and v6 addresses, and either side dying produces the same symptom. + +#### Read Resolver Attributes — Determines Blast Radius + +Each `scutil --dns` resolver has attributes that decide which queries it participates in: + +| Attribute | Matches | Stall radius if this resolver dies | +|-----------|---------|------------------------------------| +| `domain : foo.com` | Only `*.foo.com` queries | Bounded — only `foo.com` lookups stall | +| `search domain : foo` | All queries (search suffix appended) | Unbounded — every lookup stalls | +| No `domain` field at all | All queries (default participation) | Unbounded — every lookup stalls | + +A dead resolver with a `domain` filter is annoying but localized. A dead resolver with no `domain` filter (very common with VPN-injected DNS like Tailscale's `100.100.100.100`) tanks every system lookup until you fix it. + +#### Confirm the Suspect Component + +Once the bisection identifies the dead nameserver, identify which app injected it (interface name in `if_index` is the strongest hint — `utun*` interfaces usually trace back to a VPN daemon). + +For Tailscale specifically: + +```bash +tailscale status +# Healthy: lists peers +# Dead: failed to connect to local Tailscale service; is Tailscale running? +``` + +The "failed to connect" error means the daemon process is gone but the network configuration it injected (utun interface + DNS resolver entry) hasn't been cleaned up. The same pattern applies to any VPN/tunneling tool. + +#### Fix + +Restart the responsible app at the application level so its cleanup hooks run and remove the stale interface: + +**Tailscale (App Store and Standalone macOS builds)**: + +```bash +osascript -e 'quit app "Tailscale"' && sleep 3 && open -a Tailscale +``` + +For other VPN/tunneling tools, prefer a clean app-level quit (menu bar → Quit, or `osascript -e 'quit app ""'`) over `kill -9`. Forced kill skips cleanup and can leave the same dead-interface state. Only escalate to `pkill -9 ` if the app refuses to exit normally. + +**Why "restart the app" beats "flush DNS cache"**: `sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder` flushes cached results, but the resolver chain in `scutil --dns` is rebuilt from network configuration, not from the cache. The dead resolver is still there after a flush. The fix has to come from the app that registered the resolver in the first place. + +#### Verify End-to-End (4 Dimensions) + +A DNS-resolver fix is easy to half-verify. All four must pass before declaring the system path healed: + +```bash +# 1. The owning daemon is back (not just its UI) +tailscale status | head -3 + +# 2. The previously-dead nameserver responds fast +dig @ +tries=1 +timeout=3 +short example.com +# Expected: <0.1s, returns IP + +# 3. macOS system path is unblocked (proves getaddrinfo recovered) +/usr/bin/time -p dscacheutil -q host -a name example.com +# Expected: <0.1s, returns IP + +# 4. The original failing command works WITHOUT any workaround +ssh -o "ProxyCommand=none" -T git@github.com +# Expected: "Hi ! You've successfully authenticated..." +``` + +The fourth dimension is the one that matters most. If you applied a workaround during diagnosis (a `ProxyCommand` that delegates DNS to a SOCKS5 proxy, a `/etc/hosts` entry, a hardcoded IP), running the original command with the workaround disabled (`ProxyCommand=none`) is the only way to know you actually healed the system DNS path rather than just routed around it. + +See [references/dns_resolver_chain_stall.md](references/dns_resolver_chain_stall.md) for the full mental model of macOS resolver ordering, the IPv4-vs-IPv6 split, and a worked example walking through every diagnostic command and its real output. + ### Step 3: Fix Proxy Tool Configuration Identify the proxy tool and apply the appropriate fix. See [references/proxy_conflict_reference.md](references/proxy_conflict_reference.md) for detailed instructions per tool. @@ -736,6 +858,8 @@ ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no @ 'echo All three must pass. If step 1 fails, revisit Step 3. If step 1 shows wrong utun (e.g., Shadowrocket's utun with MTU 4064 instead of Tailscale's with MTU 1280), that is also a route conflict. If step 2 passes but step 3 fails with `kex_exchange_identification`, revisit Step 5A (Tailscale SSH proxy intercept). If step 2 fails, check WSL sshd or firewall. If step 3 fails with other errors, revisit Steps 4-5. +**For DNS-related fixes (Step 2I)**, the three steps above are not sufficient — they don't cover system-DNS recovery. Use the four-dimensional verification at the end of Step 2I instead: daemon health, per-resolver `dig`, `dscacheutil`, and the original failing command run **without** any workaround. + ## SOP: Remote Development via Tailscale Proactive setup guide for remote development over Tailscale with proxy tools. Follow these steps **before** encountering problems. diff --git a/tunnel-doctor/references/dns_resolver_chain_stall.md b/tunnel-doctor/references/dns_resolver_chain_stall.md new file mode 100644 index 00000000..5b642a86 --- /dev/null +++ b/tunnel-doctor/references/dns_resolver_chain_stall.md @@ -0,0 +1,210 @@ +# macOS DNS Resolver Chain Stall — Deep Reference + +This reference covers the mental model, diagnostic procedure, and a worked example for the failure mode introduced in [SKILL.md § Step 2I](../SKILL.md). Read this when: + +- The Step 2I body's bisection alone hasn't isolated the problem +- You need to explain to a teammate why `nslookup` and the application disagree +- You're writing your own automation that interacts with `scutil --dns` + +## The Mental Model + +### macOS DNS resolution paths (most apps vs. nslookup) + +``` + Application (ssh, curl, git, browser, ...) + │ getaddrinfo() + ▼ + DirectoryService (system-wide, async) + │ consults + ▼ + mDNSResponder (resolver-chain executor) + │ queries in parallel: + ┌─────────────┼─────────────┬──────────────┐ + ▼ ▼ ▼ ▼ + resolver #1 resolver #2 resolver #3 resolver #N + (default) (utunN, VPN) (per-domain) (mdns / arpa) +``` + +`nslookup` does **not** go through this path. It opens a UDP socket to the first nameserver in `/etc/resolv.conf` and parses the reply itself. That's why `nslookup` can return instantly while `ssh`/`curl`/`git`/Chrome all hang. + +### Resolver attributes that matter + +`scutil --dns` lists every resolver entry. Three fields decide whether a resolver participates in a given lookup: + +| Field | Meaning | When dead, what stalls | +|-------|---------|------------------------| +| `nameserver[N]` | Where to send the query | The address that has to respond | +| `domain : ` | Only matches queries ending in this suffix | Only that suffix's lookups stall | +| `search domain : ` | Suffix used by short-name expansion; resolver still participates in fully-qualified lookups | Every lookup stalls | +| (no `domain` field) | Default-participation supplemental resolver | Every lookup stalls | +| `flags : Supplemental` | Resolver is consulted in addition to the default, not in place of it | Every lookup stalls | + +A useful shorthand: **if a resolver has no `domain :` line, treat it as participating in every lookup.** That's the high-blast-radius case. + +### Why a dead daemon ≠ a removed resolver + +When a VPN or tunneling daemon (Tailscale, AnyConnect, OpenVPN with `--dhcp-option DNS`, etc.) starts up, it registers a resolver entry with the system network configuration via `SystemConfiguration.framework`. When the daemon dies cleanly, it un-registers the entry as part of teardown. When it crashes, the entry stays. + +After a crash: + +- The `utun` interface is still in `ifconfig` (kernel doesn't auto-tear it down) +- The route table still has the daemon's CGNAT/RFC1918 ranges pointing at that `utun` +- `scutil --dns` still has the resolver entry the daemon registered +- The actual port-53 listener inside the daemon is gone + +This explains the Step 2I trap: `ping ` works because the `utun` interface still owns the IP and answers ICMP at the kernel level. `dig @` fails because there's no UDP/53 listener anymore. + +## Worked Example + +Reproduces a real diagnosis. Substitute any environment-specific values (nameservers, hostnames) with what `scutil --dns` shows on your machine. + +### 1. The original symptom + +```text +$ git push +# … hangs ~60 seconds, then either succeeds slowly or times out + +$ ssh -vvv git@github.com +… +debug2: resolving "ssh.github.com" port 443 +debug3: resolve_host: lookup ssh.github.com:443 +# (frozen here for ~60 seconds) +``` + +### 2. First-line check: nslookup vs dscacheutil divergence + +```text +$ time nslookup ssh.github.com +Server: 198.18.0.2 +Address: 198.18.0.2#53 +Non-authoritative answer: +Name: ssh.github.com +Address: 198.18.0.14 +nslookup ssh.github.com 0.01s user 0.00s system 23% cpu 0.06 total + +$ time dscacheutil -q host -a name ssh.github.com +# (no output for 60 seconds, then …) +dscacheutil ssh.github.com 0.00s user 0.00s system 0% cpu 1:00.01 total +``` + +A 1000x divergence (`0.06s` vs `60.01s`) between these two on the same hostname is diagnostic for a stalled supplemental resolver. Stop suspecting the proxy or the route table; this is system DNS. + +### 3. List every resolver + +```text +$ scutil --dns | grep -E "^resolver|nameserver|domain :|search domain|if_index" +resolver #1 + search domain[0] : .ts.net + nameserver[0] : 198.18.0.2 + if_index : 37 (utun7) +resolver #2 + nameserver[0] : 100.100.100.100 + nameserver[1] : fd7a:115c:a1e0::53 + if_index : 24 (utun6) +resolver #3 + nameserver[0] : 198.18.0.2 + if_index : 37 (utun7) +resolver #4 + domain : .ts.net. + nameserver[0] : 100.100.100.100 + nameserver[1] : fd7a:115c:a1e0::53 + if_index : 24 (utun6) +resolver #11 + domain : baidu.com + nameserver[0] : 223.5.5.5 + nameserver[1] : 119.29.29.29 +… +``` + +Three observations from this output: + +- Resolver #2 has `nameserver` but **no `domain :` line** → it participates in every lookup +- Resolver #2 lives on `utun6` → trace back what owns `utun6` +- Resolver #4 has `domain : .ts.net.` → bounded scope; only `*.ts.net` queries route through it + +If resolver #2 stalls, every system DNS query stalls. This is the high-blast-radius case. + +### 4. Bisect + +```text +$ for ns in 198.18.0.2 100.100.100.100 223.5.5.5 119.29.29.29; do + printf " %s: " "$ns" + /usr/bin/time -p dig @$ns +tries=1 +timeout=3 +short example.com 2>&1 | tr '\n' ' ' + echo + done + 198.18.0.2: 93.184.215.14 real 0.01 ... + 100.100.100.100: ;; connection timed out; no servers could be reached real 3.01 ... + 223.5.5.5: 93.184.215.14 real 0.01 ... + 119.29.29.29: 93.184.215.14 real 0.01 ... +``` + +`100.100.100.100` is dead. The IPv6 nameserver should also be tested: + +```text +$ /usr/bin/time -p dig @fd7a:115c:a1e0::53 +tries=1 +timeout=3 +short example.com +;; connection timed out; no servers could be reached +real 3.01 ... +``` + +Both halves of resolver #2 are dead. Both addresses (v4 and v6) are inside the same VPN's address space (Tailscale's CGNAT/ULA), so they share fate. + +### 5. Identify the owning component + +`100.100.100.100` is Tailscale's MagicDNS address (well-known). Confirm: + +```text +$ tailscale status +failed to connect to local Tailscale service; is Tailscale running? +``` + +The daemon is dead but the network configuration it registered is still present. + +### 6. The "ping ok but DNS dead" check + +This step is what catches false-negative diagnoses ("ping works, can't be the network"): + +```text +$ ping -c 1 -W 2000 100.100.100.100 +PING 100.100.100.100 (100.100.100.100): 56 data bytes +64 bytes from 100.100.100.100: icmp_seq=0 ttl=64 time=0.448 ms +``` + +ICMP comes back in under half a millisecond. The interface is alive. **The service on that interface is not.** + +### 7. Fix and verify + +```text +$ osascript -e 'quit app "Tailscale"' && sleep 3 && open -a Tailscale + +$ tailscale status | head -3 +100.x.x.x @ macOS - +… + +$ /usr/bin/time -p dig @100.100.100.100 +tries=1 +timeout=3 +short example.com +93.184.215.14 +real 0.01 … + +$ /usr/bin/time -p dscacheutil -q host -a name example.com +name: example.com +ip_address: 93.184.215.14 +real 0.01 … + +$ ssh -o "ProxyCommand=none" -T git@github.com +Hi ! You've successfully authenticated, but GitHub does not provide shell access. +``` + +Four-dimensional verification passes; system DNS path is healed. + +## Counterexamples — When This Is NOT The Problem + +The Step 2I pattern is specific. Several adjacent symptoms have different fixes: + +| Symptom | Looks like Step 2I, but is actually | Fix | +|---------|-------------------------------------|-----| +| `nslookup` is also slow | Default DNS is bad, not a supplemental resolver | Replace `nameserver` in `/etc/resolv.conf` (won't persist across DHCP) or fix proxy DNS | +| `ssh` hangs at `debug1: connect to address X.X.X.X` (after resolution succeeds) | Network/route layer, not DNS | Step 2B (route conflict) or Step 2H (TUN DNS hijack) | +| Lookup works initially, slows down over hours | Cache poisoning or memory pressure on `mDNSResponder` | `sudo killall -HUP mDNSResponder` | +| Only one specific domain is slow | Per-domain resolver with a `domain :` filter is dead | Same Step 2I procedure, but the blast radius is bounded | +| `curl -x http://127.0.0.1:` works but `curl` (no `-x`) doesn't | Proxy works; DNS works; the issue is `NO_PROXY` config or env vars | Step 2A | + +The four-dimensional verification at the end of Step 2I is what distinguishes "I fixed DNS" from "I worked around DNS." If dimension 4 (`ssh -o "ProxyCommand=none"`) still fails after the daemon restart, the resolver chain isn't the problem — go back to Step 1 and re-bisect.