Skip to content

coolthor/musubi

Repository files navigation

Musubi 結び

Surface the connections your search engine misses.

A deterministic knowledge graph for markdown notes — no LLM, no lock-in, no server.

English | 繁體中文

Musubi demo knowledge graph

$ musubi neighbors "vllm"

◇ [inference] Ollama vs vLLM: Same Model, 30% Speed Gap on GPU
  id=12  degree=5  concepts=15

  · w=7   [inference] Quantization Format Cheat Sheet: FP8, NVFP4, GGUF, AWQ, GPTQ
      shared: gotcha, gpu, vllm, inference
  · w=5   [devops] Docker GPU OOM: Why Your Container Crashes After 2 Hours
      shared: gpu, vram, vllm, inference
  · w=3   [inference] MoE vs Dense: Why Bandwidth Decides Everything
      shared: gpu, tok/s, throughput

You wrote a debugging note about FP8 KV cache six weeks ago. Today you're fixing a different quantization issue. Keyword search won't find the old note because the vocabulary is different. Musubi links them through shared concepts — quantization, kv cache, vllm — so the connection surfaces automatically.


What it does

Musubi scans a directory of markdown files, extracts technical concepts from each document, and builds a weighted graph of concept co-occurrence. The result is a JSON graph you can query instantly from the command line:

Command What you get
musubi init Interactive setup — try the demo or point at your notes
musubi build --source <dir> Build the graph from your notes directory
musubi stats "How big is my knowledge graph? What are the hubs?"
musubi neighbors <doc> "What other notes are related to this one?"
musubi cold "Which notes have gone stale and lost connections?"
musubi search <query> Search + graph-expanded neighbors + quality badges (confidence + staleness)
musubi benchmark Measure how many tokens musubi saves you (details)

No servers. No databases. No API keys. Just markdown files and a CLI.


Why not...

Tool What it does Why musubi is different
Obsidian Graph View Visual graph of [[wikilinks]] Locked to Obsidian. Edges are manual (you write the links). No concept extraction, no cold detection, no CLI. Musubi works with any markdown files.
LightRAG LLM-powered entity extraction → knowledge graph → RAG Requires an LLM for every build ($$). Server setup. Designed for RAG pipelines, not personal note management. Musubi builds are deterministic, free, and take 24 seconds.
Logseq / Roam Graph-based note apps with bidirectional linking Proprietary formats. Manual linking. No automatic concept discovery. Musubi reads plain .md files — your existing ones, unmodified.
Neo4j + scripts Full graph database Requires a running Neo4j server. Overkill for personal notes. Musubi is a single pip install — the graph is a JSON file.
Vector search alone Find similar documents by embedding distance Similarity ≠ community. Two Python files are "similar" (both are code) but not intellectually related. Concept co-occurrence captures topical relationships, not surface-level text similarity.

Musubi's position: it's a companion, not a replacement. Your markdown files stay where they are. The graph is a pure derivative you can delete and rebuild anytime. Zero lock-in.


Numbers

The included demo notes (20 docs across 5 domains) build instantly. Real-world corpora of 400+ docs build in under 30 seconds:

Metric Demo (20 docs) Production (400 docs)
Concept edges 17 13,915
Isolated nodes 7 (35%) 0 (0%)
Build time 0.7s 24s
Graph file size 8 KB 1 MB
Read-side query < 100ms < 200ms

The 35% isolation rate in the demo drops to 0% in production because larger corpora have more concept overlap — and with qmd mode, the embedding fallback catches the rest.

The default concept dictionary ships 180+ generic tech terms. You add your own domain vocabulary in a plain text file (see Customizing concepts).


Install

Requires Python 3.11+. No other external tools needed.

One-liner (if you have uv):

uvx --from git+https://github.com/coolthor/musubi musubi init

This downloads, installs, and launches the setup wizard in one shot.

Or install permanently:

git clone https://github.com/coolthor/musubi.git
cd musubi
uv tool install .           # or: pip install .

musubi init                 # interactive setup wizard

The setup wizard checks your environment, lets you try the bundled demo or point at your own notes, and walks you through custom concepts and cron setup — all in under a minute.

Optional: qmd integration

If you use @tobilu/qmd (an on-device hybrid search CLI for markdown), musubi can read its SQLite index directly:

  • musubi build (no --source) reads from the qmd index, including embeddings for a better isolation fallback
  • musubi search delegates keyword search to qmd search --json and expands results with graph neighbors

qmd is not required. Without it, use musubi build --source <dir>.


Quick start

# Build from a directory of markdown files
musubi build --source ~/my-notes/

# Explore
musubi stats                         # graph overview
musubi neighbors "some-doc-slug"     # what's connected to this?
musubi cold --limit 20               # what's gone stale?

Example output: musubi stats

◇ Musubi — Graph Stats

  version:   0.1.0
  nodes:     20
  edges:     17
    · concept: 17
  isolated:  7 (35.0%)
  avg deg:   1.7
  hub node:  deg=5  [inference] Ollama vs vLLM: Same Model, 30% Speed Gap on GPU

  collections:
    inference          5
    agents             4
    devops             4
    webdev             4
    general            3

  top concepts:
    vllm                      5 docs
    agent                     4 docs
    inference                 4 docs
    gpu                       4 docs
    git                       4 docs

Example output: musubi neighbors

◇ [inference] Ollama vs vLLM: Same Model, 30% Speed Gap on GPU
  id=12  degree=5  concepts=15

  · w=7   [inference] Quantization Format Cheat Sheet: FP8, NVFP4, GGUF, AWQ, GPTQ
      shared: gotcha, gpu, vllm, inference
  · w=5   [devops] Docker GPU OOM: Why Your Container Crashes After 2 Hours
      shared: gpu, vram, vllm, inference
  · w=3   [inference] MoE vs Dense: Why Bandwidth Decides Everything
      shared: gpu, tok/s, throughput

Notice how Docker GPU OOM (a devops note) surfaces as a neighbor of a benchmark note — they share gpu, vram, vllm, and inference. Keyword search for "benchmark" would never return a devops debugging note.

Example output: musubi cold

◇ Cold nodes (top 5)
  score = 0.5·(1/deg) + 0.2·(1/concepts) + 0.3·(days/180)

  score   deg   days   label
  0.567   0     0      ❄ [webdev] React Server Components: Three Gotchas
  0.567   0     0      ❄ [devops] PostgreSQL Autovacuum Deadlock
  0.550   0     0      ❄ [agents] Tool Calling Patterns: When the Model Gets It Wrong
  0.540   0     0      ❄ [agents] Agent Memory: File-Based vs Database-Backed
  0.525   0     0      ❄ [webdev] Next.js i18n: hreflang x-default Splits Rankings

The ❄ marker flags isolated nodes (degree = 0). These docs don't share enough concepts with anything else — either they need richer vocabulary, or they're genuinely standalone topics.


Memory quality: confidence + staleness

Graph search can surface the right document and still mislead you — because the document itself was wrong, or because the code it described has since changed. Musubi adds two lightweight signals so future-you (or your LLM assistant) can tell a verified finding from an unverified hypothesis, and can see when a note points at code that has moved on.

Confidence tag

Add a confidence: field to any note's frontmatter:

---
title: Redis connection churn — root cause
confidence: verified
verified_by: "production logs + /metrics reproduce"
---

Three levels: verified (you checked), hypothesis (best guess so far), superseded (later work proved it wrong — add superseded_by: path/to/correction.md so readers can follow the trail). Untagged notes stay untagged — no forced migration.

Staleness detection

Musubi scans every note for referenced filesystem paths (/Users/..., ~/... style). At search time, it stats those paths and compares their mtime to the note's own mtime. If a referenced file is newer than the note, the note is flagged ⚠ stale — the source code has moved on, the note may not have.

◇ Musubi search: redis singleton churn

  ★ 1.000  [bps-api] Redis Connection Churn ⚠ stale (1 src newer)
            [verified ✓]  docs/experience-2026-03-01-...md
  + 0.671  [bps-api] Service Layer Performance Audit (Codex)

Missing files, directories, and /dev/null are filtered out — only regular-file mtimes count.

Why it matters

One bad note can poison days of work. An earlier incident: a "35-step success" hypothesis was written without verifying the exit status, then trusted the next day as fact — a full day was lost chasing its false premise. With confidence: hypothesis, the next search would have shown the caveat next to the title. With staleness, a note that referenced the trajectory file would have flagged stale once the file was overwritten.

Both signals are visible in musubi search, musubi neighbors, and the MCP tool output. Untagged notes behave exactly as before.


How it works

                    musubi build
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
   ┌────────────┐ ┌────────────┐ ┌────────────┐
   │ ~/notes/   │ │ qmd sqlite │ │ concepts   │
   │ *.md files │ │ (optional) │ │ dictionary │
   └─────┬──────┘ └─────┬──────┘ └─────┬──────┘
         │              │              │
         └──────────────┼──────────────┘
                        ▼
              ┌──────────────────┐
              │ concept          │
              │ extraction       │  regex matching against
              │ per document     │  180+ built-in terms
              └────────┬─────────┘  + your custom terms
                       ▼
              ┌──────────────────┐
              │ co-occurrence    │  if doc A and doc B both
              │ edge building    │  mention "vllm" + "kv cache"
              │                  │  → edge(A,B, weight=2)
              └────────┬─────────┘
                       ▼
              ┌──────────────────┐
              │ embedding        │  (qmd mode only)
              │ fallback for     │  cosine sim > 0.5 for
              │ isolated nodes   │  docs with 0 concept edges
              └────────┬─────────┘
                       ▼
              ┌──────────────────┐
              │ graph.json       │  NetworkX node_link_data
              │ (~1 MB)          │  stored at $MUSUBI_GRAPH_PATH
              └──────────────────┘
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       stats      neighbors      cold
       search        path

Key design decisions

  • Concept extraction is deterministic. No LLM in the build loop. Same input → same graph, every time. No API costs, no hallucinated entities, no rate limits.
  • Read path imports nothing heavy. musubi stats and musubi neighbors load the JSON file and answer from memory. No networkx, no numpy on the read path. Fast cold starts.
  • Edges are interpretable. Every edge carries its shared_concepts list, so you can always see why two docs are connected — not just that they are.

Customizing concepts

The default dictionary covers generic tech terms. Add your own domain vocabulary in a plain text file:

# ~/.config/musubi/concepts.txt
# One term per line. # for comments.

# your product names
my-internal-tool
secretproduct

# your domain (e.g. finance)
bull put spread
delta hedging
iv rank

# your hardware
dgx spark
gb10

Then rebuild: musubi build --source ~/notes/

The default + custom concepts are merged at build time. Your custom file is never committed to the musubi repo.


Configuration

All paths are overridable via environment variables. Sensible XDG defaults mean you don't need to set anything:

Variable Default What it is
MUSUBI_GRAPH_PATH $XDG_DATA_HOME/musubi/graph.json Serialized graph
MUSUBI_QMD_DB $XDG_CACHE_HOME/qmd/index.sqlite qmd SQLite index
MUSUBI_QMD_BIN qmd on PATH qmd CLI binary
MUSUBI_CONCEPTS_FILE $XDG_CONFIG_HOME/musubi/concepts.txt Custom concept list
MUSUBI_LOG_DIR $XDG_STATE_HOME/musubi/ Build logs
MUSUBI_WATCH_DIRS (none) Colon-separated dirs to watch for staleness

Try it yourself (demo)

The repo includes 20 demo notes across 5 domains. Build and explore in 30 seconds:

git clone https://github.com/coolthor/musubi.git
cd musubi
uv tool install .
musubi build --source examples/demo-notes/
musubi stats
musubi neighbors "vllm"
musubi cold

The interactive graph visualization can be opened in any browser after building.


Use it on your own notes

Done with the demo? Here's how to switch to your actual notes.

Step 1: Point at your notes directory

musubi build --source ~/path/to/your/notes/

Musubi recursively finds all *.md and *.mdx files. Subdirectories become collections (shown as [inference], [devops], etc. in output). If all files are at the root level, they go into a default collection.

Works with any markdown notes system — Obsidian vaults, Logseq directories, Dendron workspaces, or just a folder of .md files.

Step 2: Check what you got

musubi stats       # how many nodes, edges, isolated docs?
musubi cold        # anything gone stale?

If musubi cold shows a lot of ❄ isolated nodes, your notes probably use domain-specific vocabulary that isn't in the default concept dictionary. That's normal — go to Step 3.

Step 3: Add your vocabulary (optional but recommended)

Create a concepts file with terms specific to your work:

mkdir -p ~/.config/musubi

cat > ~/.config/musubi/concepts.txt << 'EOF'
# My domain vocabulary
# One term per line. Lines starting with # are comments.

# example: if you work in finance
black-scholes
portfolio rebalancing
sharpe ratio

# example: your internal tools
my-internal-api
staging-cluster
deployment-pipeline

# example: your tech stack
fastapi
celery
rabbitmq
EOF

Then rebuild:

musubi build --source ~/path/to/your/notes/
musubi stats    # edges and avg degree should increase
musubi cold     # fewer isolated nodes

Tip: Look at the top concepts in musubi stats. If the top concepts are too generic (like "the", "api", "code"), they're acting as stopwords and connecting everything to everything. Remove overly generic terms from your concepts file. If important domain terms are missing, add them.

Step 4: You're done — the graph stays fresh automatically

Musubi auto-rebuilds when it detects staleness (see Keeping the graph fresh). No cron job needed.

Directory structure tips

# Good: subdirectories become meaningful collections
~/notes/
├── projects/          → [projects] collection
│   ├── api-redesign.md
│   └── mobile-app.md
├── debugging/         → [debugging] collection
│   ├── postgres-oom.md
│   └── redis-timeout.md
└── learning/          → [learning] collection
    ├── rust-ownership.md
    └── k8s-networking.md

# Also fine: flat directory → all in [default] collection
~/notes/
├── api-redesign.md
├── postgres-oom.md
└── rust-ownership.md

Common questions

Q: How many notes do I need for the graph to be useful? A: 30+. Below that, most notes are isolated. Above 100, you start getting rich cross-domain connections that surprise you.

Q: Can I point at multiple directories? A: Not yet in a single command. Workaround: symlink them under one parent directory, or use qmd which supports multiple named collections.

Q: Will it modify my markdown files? A: No. Musubi is strictly read-only. It reads your files, builds a separate graph.json, and never touches the source.


Keeping the graph fresh

Musubi auto-rebuilds the graph when it detects staleness — no cron job or manual rebuild needed.

Every time you run musubi neighbors, musubi search, musubi cold, or musubi stats, musubi checks three conditions:

  1. Graph age — older than 24 hours?
  2. Source index — was the qmd sqlite re-indexed since the last build?
  3. Watched directories — any .md file newer than the graph?

If any condition is true, musubi rebuilds automatically before returning results:

$ musubi neighbors "vllm"
  ↻ graph is stale (3d old), auto-rebuilding...
  ✓ rebuilt: 402 docs, 16596 edges

◇ [ai-muninn] dgx-spark-nemotron-120b-vllm
  ...

Watching extra directories

Set MUSUBI_WATCH_DIRS to monitor directories outside the qmd index — for example, an auto-memory directory where your AI assistant writes notes:

# in ~/.zshrc or ~/.bashrc
export MUSUBI_WATCH_DIRS="$HOME/.claude/memory:$HOME/extra-notes"

Colon-separated, just like $PATH. Any .md file in those directories that's newer than the graph triggers a rebuild on the next query.

Manual rebuild

You can still force a rebuild anytime:

musubi build                        # from qmd index
musubi build --source ~/notes/      # from a directory

Measure your token savings

Musubi itself uses zero LLM tokens — the graph is built with deterministic regex matching, not LLM entity extraction. The harder question is: does using musubi save tokens when you use an LLM to search your notes? The honest answer is "sometimes, with caveats" — the benchmark lets you check whether it holds for your own corpus.

# Preview the 10 test tasks (no API calls, no cost)
musubi benchmark --dry-run

# Full run — compares grep-only vs musubi-augmented retrieval
# Requires ANTHROPIC_API_KEY. Cost: ~$0.15 per run.
export ANTHROPIC_API_KEY=sk-ant-...
musubi benchmark

# Run against your own corpus + custom task list
musubi benchmark --tasks my-tasks.json --notes ~/my-notes/

The benchmark runs each task twice — once with only grep + cat (baseline), once with musubi search + musubi neighbors — and measures total tokens consumed. Results go to experiments/token-saving/results/summary.md.

What it tests

Task type Count What it measures
Direct lookup 3 Find a note when you know the topic
Cross-domain 3 Find notes across different collections — musubi's main advantage
Exploration 2 "Show me related notes" — only graph can do this
Health check 2 "What's stale?" — only musubi can answer

Preconditions (read before trusting any number)

Token savings are not unconditional. What actually matters:

  1. Corpus size. On the 20-doc demo corpus, baseline grep beats musubi on total tokens — when every file is two grep -l hops away, keyword search is simply more efficient. A real-corpus run on ~50 well-curated notes flipped this to a ~36% saving for musubi. Below ~30 docs, don't expect wins.
  2. Task mix. Musubi dominates health_check (staleness / orphan detection — grep literally cannot answer these) and does well on exploration. On pure direct_lookup of a known topic, grep is often still cheaper, because musubi search + read_file is two steps where grep -l + cat is two steps but the grep output is tinier. If your workflow is mostly direct lookup, savings are modest or negative.
  3. Baseline choice. The benchmark compares against naïve grep, which is the realistic fallback for someone without musubi. It is not a comparison against qmd alone. Since musubi builds on qmd, some of the observed savings come from qmd's FTS/vector search being better than grep, not from the graph layer itself. We don't try to decompose those — the thing musubi replaces for an end user is "grep my notes," so that's the baseline we report.
  4. LLM behavior is unstable. We've run three A/B attempts at "optimizations" (compact output, qmd-snippet passthrough, embedding hybrid) that looked like wins on paper and regressed in live runs. Run-to-run variance is real. Trust medians across a few runs, not a single number.

Why we test instead of claim

We don't put a single "saves X% tokens" headline in the README because the honest answer depends on your corpus, your task mix, and today's model behavior. Run the benchmark on your own notes with --tasks my-tasks.json --notes ~/my-notes/ and treat the output as a measurement of your setup, not a universal claim.


Using with Claude Code

Two integration levels — pick the one that fits:

Option A: Auto-hook (recommended)

Add a PreToolUse hook that runs musubi search automatically before every web search. Claude sees your internal knowledge before hitting the web — no way to forget.

1. Save the hook script:

mkdir -p ~/.claude/hooks

cat > ~/.claude/hooks/musubi-presearch.sh << 'HOOKEOF'
#!/bin/bash
# Runs musubi search before WebSearch/WebFetch so Claude sees
# internal knowledge before searching externally.
MUSUBI=$(command -v musubi 2>/dev/null)
[ -z "$MUSUBI" ] && exit 0
[ ! -f "$HOME/.local/share/musubi/graph.json" ] && exit 0

QUERY=$(echo "$TOOL_INPUT" | python3 -c "
import sys, json
try:
    d = json.load(sys.stdin)
    print(d.get('query', ''))
except:
    pass
" 2>/dev/null)
[ -z "$QUERY" ] && exit 0

RESULT=$("$MUSUBI" search "$QUERY" --limit 3 2>/dev/null)
if [ -n "$RESULT" ] && ! echo "$RESULT" | grep -q "No results\|not found"; then
    echo "🔗 musubi (internal knowledge):"
    echo "$RESULT"
fi
HOOKEOF

chmod +x ~/.claude/hooks/musubi-presearch.sh

2. Add to ~/.claude/settings.json (inside hooks.PreToolUse):

{
  "matcher": "WebSearch",
  "hooks": [{
    "type": "command",
    "command": "bash ~/.claude/hooks/musubi-presearch.sh",
    "statusMessage": "checking internal knowledge..."
  }]
}

Now every WebSearch shows a 🔗 musubi block first — Claude automatically knows what you already have before searching the internet.

Option B: CLAUDE.md instructions (simpler)

If you don't want hooks, add this to your CLAUDE.md instead:

### Retrieval workflow

1. Before searching the web, check internal knowledge:
   `musubi search "<topic>"` — shows keyword hits + graph neighbors
2. If a relevant doc is found, expand context:
   `musubi neighbors "<filename>" --limit 3`
3. Only then search externally for what's actually new.

This relies on Claude following instructions (usually works, but not guaranteed — Option A is foolproof).

No MCP server needed

Both options work through the shell. Claude Code calls musubi the same way it calls grep or git. No server process, no configuration beyond the hook or CLAUDE.md text.


Prior art

  • LightRAG — LLM-powered knowledge graph + RAG. Musubi borrows the "treat notes as a graph" idea but replaces LLM extraction with deterministic concept matching.
  • Obsidian graph view — the visual intuition. Musubi gives you the same structure as a CLI tool on plain markdown files.
  • Louvain community detection, betweenness centrality — the graph theory behind musubi cold and community identification.

License

MIT. See LICENSE.

About

Musubi (結び) — a knowledge-graph companion for flat-file markdown note systems. Ties your notes together.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors