-
Notifications
You must be signed in to change notification settings - Fork 43
Add missing scripts referenced in the guide (#11) #17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # Scripts | ||
|
|
||
| Runnable companions to the guide. The parts describe what each script does and | ||
| why; the source lives here so you can `git clone` once and run them instead of | ||
| copy-pasting out of Markdown. | ||
|
|
||
| All scripts are zero-dependency: the `.mjs` tools need only Node.js (ES | ||
| modules), and the `.sh` wrappers need only `bash` plus the CLIs called out in | ||
| their comments (`openclaw`, `git`, `jq`, `awk`, `bc`). | ||
|
|
||
| ## Vault graph tools — [Part 9](../part9-vault-memory.md) | ||
|
|
||
| Run these from your workspace root (or set `OPENCLAW_WORKSPACE` to point at it). | ||
|
|
||
| | Script | Purpose | | ||
| |--------|---------| | ||
| | [`vault-graph/graph-indexer.mjs`](./vault-graph/graph-indexer.mjs) | Scans every `.md` file, parses `[[wiki-links]]`, writes a JSON adjacency graph to `vault/06_system/graph-index.json`. | | ||
| | [`vault-graph/graph-search.mjs`](./vault-graph/graph-search.mjs) | Traverses the graph — finds a file plus its direct and 2nd-degree connections. | | ||
| | [`vault-graph/auto-capture.mjs`](./vault-graph/auto-capture.mjs) | Turns an insight into a claim-named note in `vault/00_inbox/` and links it to related MOCs. | | ||
| | [`vault-graph/process-inbox.mjs`](./vault-graph/process-inbox.mjs) | Reviews inbox notes and suggests (or moves them to) the right vault folder. | | ||
| | [`vault-graph/update-mocs.mjs`](./vault-graph/update-mocs.mjs) | Health check — finds broken wiki-links, stale items, and orphaned notes. | | ||
|
|
||
| ```bash | ||
| # typical first run | ||
| mkdir -p vault/{00_inbox,01_thinking,02_reference,03_creating,04_published,05_archive,06_system} | ||
| node scripts/vault-graph/graph-indexer.mjs | ||
| node scripts/vault-graph/graph-search.mjs "memory" | ||
| ``` | ||
|
|
||
| > Note: `vault-graph/auto-capture.mjs` (Part 9) is **not** the same as the | ||
| > `auto-capture` hook in [`hooks/auto-capture/`](../hooks/auto-capture/) (Part | ||
| > 11). The Part 9 script files a note you hand it; the Part 11 hook extracts | ||
| > notes from a session transcript automatically. | ||
|
|
||
| ## Autonomy wrappers | ||
|
|
||
| | Script | Part | Purpose | | ||
| |--------|------|---------| | ||
| | [`ralph.sh`](./ralph.sh) | [Part 30](../part30-ralph-loop-in-openclaw.md) | Runs the Ralph loop — a fresh OpenClaw session per iteration until `PRD.json` is `done` or the iteration / wall-clock / USD budget is exhausted. Needs `PRD.json` and `loop-prompt.md` in the project root. | | ||
| | [`fan-out.sh`](./fan-out.sh) | [Part 15](../part15-infrastructure-hardening.md) | Spawns one agent per `*.md` task prompt across N git worktrees, then waits on all of them and reports failures. | | ||
|
|
||
| ```bash | ||
| scripts/ralph.sh /path/to/project | ||
| scripts/fan-out.sh ./tasks | ||
| ``` | ||
|
|
||
| ## Referenced elsewhere (not bundled here) | ||
|
|
||
| A few scripts mentioned in the guide intentionally live outside this repo — | ||
| their source isn't reproduced in the Markdown, so committing a copy here would | ||
| just go stale. Here's where each one actually comes from: | ||
|
|
||
| | Path in the guide | Part | Where it lives | | ||
| |-------------------|------|----------------| | ||
| | `scripts/memory-bridge/memory-query.js`, `preflight-context.js` | [Part 13](../part13-memory-bridge.md) | Standalone Memory Bridge tool — copy the two scripts into `scripts/memory-bridge/` in your own workspace as described in Part 13. | | ||
| | `lightrag-watcher.py` | [Part 21](../part21-realtime-knowledge-sync.md) | Your real-time sync watcher — Part 21 shows the config block to edit; deploy it under your own `scripts/`. | | ||
| | `scripts/qwen_embed_server_v3.py` | [Part 10](../part10-state-of-the-art-embeddings.md) | Ships inside your local OpenClaw install (installed by the one-shot prompt in Part 17), not in this guide repo. | |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/usr/bin/env bash | ||
| # scripts/fan-out.sh <tasks-dir> | ||
| set -euo pipefail | ||
| tasks_dir="$(realpath "${1:?pass a directory containing one *.md task-prompt per agent}")" | ||
| base_repo="$(pwd)" | ||
|
|
||
| mkdir -p "$base_repo/.worktrees" | ||
| declare -a pids=() | ||
|
|
||
| for task in "$tasks_dir"/*.md; do | ||
| abs_task="$(realpath "$task")" # pin before we cd into the worktree | ||
| name=$(basename "$task" .md) | ||
| wt="$base_repo/.worktrees/$name" | ||
| branch="agent/$name" | ||
|
|
||
| git worktree add "$wt" -b "$branch" >/dev/null | ||
| ( | ||
| cd "$wt" | ||
| openclaw run --prompt-file "$abs_task" --ephemeral --output json \ | ||
| > "$base_repo/.worktrees/$name.log" 2>&1 | ||
| ) & | ||
| pids+=($!) | ||
| echo "[fan-out] spawned $name (pid ${pids[-1]}) in $wt" | ||
| done | ||
|
|
||
| # Don't let one failing agent orphan the rest: track failures but keep waiting. | ||
| set +e | ||
| failures=0 | ||
| for pid in "${pids[@]}"; do | ||
| wait "$pid" || ((failures++)) | ||
| done | ||
| set -e | ||
| echo "[fan-out] all agents done ($failures failed of ${#pids[@]}). Branches: agent/*" | ||
| [[ $failures -eq 0 ]] || exit 1 |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| #!/usr/bin/env bash | ||
| # scripts/ralph.sh — run the Ralph loop until PRD is done or budget is exhausted | ||
| set -euo pipefail | ||
|
|
||
| PROJECT_ROOT="${1:-$(pwd)}" | ||
| cd "$PROJECT_ROOT" | ||
|
|
||
| [ ! -f PRD.json ] && { echo "PRD.json missing"; exit 1; } | ||
| [ ! -f loop-prompt.md ] && { echo "loop-prompt.md missing"; exit 1; } | ||
|
|
||
| max_iter=$(jq -r '.budget.max_iterations // 40' PRD.json) | ||
| max_usd=$(jq -r '.budget.max_usd // 20' PRD.json) | ||
| max_hours=$(jq -r '.budget.max_wall_hours // 4' PRD.json) | ||
|
|
||
| start=$(date +%s) | ||
| total_usd=0.00 | ||
| i=0 | ||
|
|
||
| while : ; do | ||
| i=$((i+1)) | ||
| status=$(jq -r '.status' PRD.json) | ||
| [ "$status" = "done" ] && { echo "[ralph] PRD.done — exiting at iter $i"; break; } | ||
|
|
||
| elapsed=$(( ($(date +%s) - start) / 3600 )) | ||
| [ "$elapsed" -ge "$max_hours" ] && { echo "[ralph] wall-clock budget exhausted"; break; } | ||
| [ "$i" -gt "$max_iter" ] && { echo "[ralph] iteration budget exhausted"; break; } | ||
|
|
||
| echo "[ralph] === iter $i (elapsed ${elapsed}h, spent \$$total_usd) ===" | ||
|
|
||
| # Fresh OpenClaw session per iteration. --ephemeral keeps it out of your main history. | ||
| out_json=$(openclaw run \ | ||
| --prompt-file loop-prompt.md \ | ||
| --ephemeral \ | ||
| --output json) | ||
|
|
||
| iter_usd=$(echo "$out_json" | jq -r '.usage.cost_usd // 0') | ||
| total_usd=$(echo "$total_usd $iter_usd" | awk '{print $1 + $2}') | ||
|
|
||
| (( $(echo "$total_usd >= $max_usd" | bc -l) )) && { echo "[ralph] USD budget exhausted"; break; } | ||
|
|
||
| # Let the dreaming scheduler consolidate anything learned before the next iteration. | ||
| sleep 2 | ||
| done | ||
|
|
||
| # Final summary. | ||
| echo | ||
| echo "[ralph] final status=$(jq -r '.status' PRD.json) iterations=$i spent=\$${total_usd}" | ||
| jq '.tasks[] | {id, status}' PRD.json |
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,101 @@ | ||||||||||||||||||||||||
| #!/usr/bin/env node | ||||||||||||||||||||||||
| /** | ||||||||||||||||||||||||
| * Auto-Capture: Converts insights into claim-named vault notes. | ||||||||||||||||||||||||
| * | ||||||||||||||||||||||||
| * Usage: | ||||||||||||||||||||||||
| * node auto-capture.mjs --claim "nemotron mamba wont train on windows" --body "details..." | ||||||||||||||||||||||||
| * node auto-capture.mjs --file summary.txt | ||||||||||||||||||||||||
| * echo "insight text" | node auto-capture.mjs | ||||||||||||||||||||||||
| */ | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| import fs from 'fs'; | ||||||||||||||||||||||||
| import path from 'path'; | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| const VAULT = process.env.OPENCLAW_WORKSPACE | ||||||||||||||||||||||||
| ? path.join(process.env.OPENCLAW_WORKSPACE, 'vault') | ||||||||||||||||||||||||
| : path.resolve('vault'); | ||||||||||||||||||||||||
| const INBOX = path.join(VAULT, '00_inbox'); | ||||||||||||||||||||||||
| const THINKING = path.join(VAULT, '01_thinking'); | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| function toClaimName(text) { | ||||||||||||||||||||||||
| let claim = text.split(/[.!?\n]/)[0].trim(); | ||||||||||||||||||||||||
| if (claim.length > 80) claim = claim.substring(0, 80); | ||||||||||||||||||||||||
| return claim.toLowerCase() | ||||||||||||||||||||||||
| .replace(/[^a-z0-9\s-]/g, '') | ||||||||||||||||||||||||
| .replace(/\s+/g, '-') | ||||||||||||||||||||||||
| .replace(/-+/g, '-') | ||||||||||||||||||||||||
| .replace(/^-|-$/g, '') | ||||||||||||||||||||||||
| .substring(0, 60) + '.md'; | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| function findRelatedMOCs(text) { | ||||||||||||||||||||||||
| if (!fs.existsSync(THINKING)) return []; | ||||||||||||||||||||||||
| const mocs = fs.readdirSync(THINKING).filter(f => f.endsWith('.md') && f !== 'README.md'); | ||||||||||||||||||||||||
| const textLower = text.toLowerCase(); | ||||||||||||||||||||||||
| return mocs.filter(moc => { | ||||||||||||||||||||||||
| const keywords = moc.replace('.md', '').split('-').filter(w => w.length > 3); | ||||||||||||||||||||||||
| return keywords.filter(kw => textLower.includes(kw)).length >= 2; | ||||||||||||||||||||||||
| }).map(m => m.replace('.md', '')); | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| function buildNote(claim, body, relatedMOCs) { | ||||||||||||||||||||||||
| const date = new Date().toISOString().split('T')[0]; | ||||||||||||||||||||||||
| let note = `# ${claim}\n\n## Key Facts\n\n${body}\n\n`; | ||||||||||||||||||||||||
| if (relatedMOCs.length > 0) { | ||||||||||||||||||||||||
| note += `## Connected Topics\n\n`; | ||||||||||||||||||||||||
| for (const moc of relatedMOCs) note += `- [[${moc}]]\n`; | ||||||||||||||||||||||||
| note += `\n`; | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| note += `## Agent Notes\n\n- [ ] Review and verify this capture\n`; | ||||||||||||||||||||||||
| note += `- [ ] Link to additional related notes\n`; | ||||||||||||||||||||||||
| note += `- [ ] Move from inbox to appropriate vault folder\n`; | ||||||||||||||||||||||||
| note += `\n_Captured: ${date}_\n`; | ||||||||||||||||||||||||
| return note; | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| function updateMOCs(filename, relatedMOCs) { | ||||||||||||||||||||||||
| const linkName = filename.replace('.md', ''); | ||||||||||||||||||||||||
| for (const moc of relatedMOCs) { | ||||||||||||||||||||||||
| const mocPath = path.join(THINKING, moc + '.md'); | ||||||||||||||||||||||||
| if (!fs.existsSync(mocPath)) continue; | ||||||||||||||||||||||||
| let content = fs.readFileSync(mocPath, 'utf8'); | ||||||||||||||||||||||||
| if (!content.includes(`[[${linkName}]]`)) { | ||||||||||||||||||||||||
| const idx = content.indexOf('## Agent Notes'); | ||||||||||||||||||||||||
| if (idx !== -1) { | ||||||||||||||||||||||||
| const insertPoint = content.indexOf('\n', idx) + 1; | ||||||||||||||||||||||||
| content = content.substring(0, insertPoint) | ||||||||||||||||||||||||
| + `\n- [ ] New capture linked: [[${linkName}]]\n` | ||||||||||||||||||||||||
| + content.substring(insertPoint); | ||||||||||||||||||||||||
| fs.writeFileSync(mocPath, content, 'utf8'); | ||||||||||||||||||||||||
| console.log(` Updated MOC: ${moc}.md`); | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| async function main() { | ||||||||||||||||||||||||
| const args = process.argv.slice(2); | ||||||||||||||||||||||||
| let claim = '', body = ''; | ||||||||||||||||||||||||
| if (args.includes('--claim')) claim = args[args.indexOf('--claim') + 1] || ''; | ||||||||||||||||||||||||
| if (args.includes('--body')) body = args[args.indexOf('--body') + 1] || ''; | ||||||||||||||||||||||||
| if (args.includes('--file')) { | ||||||||||||||||||||||||
| const f = args[args.indexOf('--file') + 1]; | ||||||||||||||||||||||||
| if (f && fs.existsSync(f)) { body = fs.readFileSync(f, 'utf8'); if (!claim) claim = body.split('\n')[0]; } | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| if (!claim && !body) { | ||||||||||||||||||||||||
| console.log('Usage: node auto-capture.mjs --claim "insight" --body "details..."'); | ||||||||||||||||||||||||
| process.exit(1); | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| const filename = toClaimName(claim); | ||||||||||||||||||||||||
|
Comment on lines
+85
to
+89
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔴 When
Suggested change
Was this helpful? React with 👍 or 👎 to provide feedback.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed real bug. Note this is pre-existing — the script was copied verbatim out of Part 9's code block, so the same defect lives in the guide prose too. Since this PR is purely "commit the inline scripts as files," I left the logic untouched to keep the file and the Part 9 block identical. Happy to fix it in a follow-up (or here) — but the fix should land in both |
||||||||||||||||||||||||
| const filepath = path.join(INBOX, filename); | ||||||||||||||||||||||||
| const relatedMOCs = findRelatedMOCs(claim + ' ' + body); | ||||||||||||||||||||||||
| if (!fs.existsSync(INBOX)) fs.mkdirSync(INBOX, { recursive: true }); | ||||||||||||||||||||||||
| fs.writeFileSync(filepath, buildNote(claim, body, relatedMOCs), 'utf8'); | ||||||||||||||||||||||||
| console.log(`✅ Captured: ${filename}`); | ||||||||||||||||||||||||
| if (relatedMOCs.length > 0) { | ||||||||||||||||||||||||
| console.log(` Related MOCs: ${relatedMOCs.join(', ')}`); | ||||||||||||||||||||||||
| updateMOCs(filename, relatedMOCs); | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
| } | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| main().catch(console.error); | ||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴
updateMOCscorrupts MOC file when## Agent Noteshas no trailing newlineIf the
## Agent Notesheading is the last line of a MOC file (no trailing newline),content.indexOf('\n', idx)atauto-capture.mjs:65returns-1. Adding 1 givesinsertPoint = 0, socontent.substring(0, 0)is the empty string. The result is the new link line prepended to the beginning of the file, with the entire original content appended after it — corrupting the MOC structure.Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same situation as the other finding — real bug, but pre-existing and copied verbatim from Part 9, so I kept the file identical to the guide block in this "extract scripts to files" PR. The suggested guard (
const nlIdx = content.indexOf('\n', idx); const insertPoint = nlIdx !== -1 ? nlIdx + 1 : content.length;) is correct. If you want it fixed, I'll patch bothauto-capture.mjsand the Part 9 code block together so they stay in sync.