fix(rag): skip hidden and junk directories when indexing

[StressTestor]

Summary

index_personal_documents walks the target directory with no pruning, and DEFAULT_FILE_EXTENSIONS includes .js, .json, .css, .html. Point RAG at a real-world folder and it indexes tool internals: an Obsidian vault's .obsidian/ (2.6MB of minified plugin JS in the reporter's case), a repo's .git/, node_modules/, __pycache__/. Indexing time multiplies and retrieval fills with junk chunks.

This PR prunes hidden directories and a small junk list (node_modules, __pycache__, venv) from the walk in place, and skips hidden files. The explicitly passed root is exempt, so a user who deliberately indexes a hidden directory still gets its contents. Plus 3 hermetic regression tests (same VectorRAG.__new__ pattern as test_rag_remove_directory_scope.py); all 3 fail on dev without the fix.

Target branch

• This PR targets dev, not main.

Linked Issue

Fixes #5559

Type of Change

• Bug fix (non-breaking — fixes a confirmed issue)

Checklist

• I searched open issues and open PRs — this is not a duplicate.
• This PR targets dev
• My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
• I actually ran the app (uvicorn app:app + a local ChromaDB at localhost:8100) and verified the change works end-to-end.

How to Test

1. python -m pytest tests/test_rag_index_hidden_dirs.py: 3 tests covering hidden dirs, hidden files, junk dirs, and the explicit-hidden-root exemption. They fail on dev without the fix.

2. End-to-end (what i ran): start ChromaDB and the app, then create a directory under data/personal_docs/ containing real content plus junk dirs:

   scratch-vault/
     readme.md                      <- real
     notes/meeting.md               <- real
     .obsidian/plugins/main.js      <- junk
     .git/config.json               <- junk
     node_modules/lib/index.js      <- junk
     __pycache__/cached.py          <- junk
     venv/lib/site.py               <- junk
   

3. POST /api/personal/add_directory with that path. Response on this branch: indexed_count: 2, failed_count: 0.

4. Confirm in the vector store that only the two real files landed. Querying the odysseus_rag_fastembed collection's metadatas after the call returns exactly notes/meeting.md and readme.md as sources, none of the junk paths. On dev the same tree also embeds the .obsidian/, .git/, node_modules/, __pycache__/, and venv/ files.

Also ran: full python -m pytest (4673 passed, 3 skipped; the 5 failures are the docker-socket tests, which fail identically on clean dev on this machine, macOS with no docker socket) and python -m compileall app.py core routes src services scripts tests.

Scope notes

• Chunks already indexed from junk dirs are not cleaned up by this change. Remove and re-add the directory to rebuild it clean.
• Hidden files are now skipped with no opt-out. The issue marks include_hidden as optional; adding it means plumbing a flag through the route, PersonalDocsManager, RAGManager, and the UI, so i left it out to keep this small. Can follow up if wanted.
• The junk list is deliberately minimal. dist/, build/, target/ could join it, but those names collide with legitimate content folders more often.

[RaresKeY]

I checked the latest head and found one author-actionable issue that should be addressed before merge.

Findings

issue (RAG indexing): Apply the same pruning policy to the personal-doc index

• Problem: The new filter only covers VectorRAG.index_personal_documents(). After a successful add of a newly tracked directory, the route calls PersonalDocsManager.add_directory(..., index=False), which still refreshes through an unpruned load_personal_index() walk. Hidden and junk supported files are therefore still read, chunked, retained in the manager's in-memory index, and exposed by personal-doc listings; persisted tracked directories are rebuilt into that manager index the same way on startup.

• Impact: Repositories and vaults can still incur broad traversal, content-read, and memory/listing cost from entries such as .obsidian/config.json and node_modules/**/README.md or package.json. The vector embedding path is fixed, but the linked end-to-end indexing problem remains partially present, and PersonalDocsManager's compatibility keyword fallback can still return the retained chunks.

• Ask: Reuse the same hidden/junk traversal policy in load_personal_index() through a shared constant/helper, preserve the explicit-hidden-root behavior, and add a manager/list regression proving both index representations omit the same entries.

• Location: src/rag_vector.py:506, routes/personal_routes.py:207, src/personal_docs.py:96

[StressTestor]

pushed efed441. the fix was only half applied: the keyword index (load_personal_index, which refresh_index builds from) was still walking the tree unpruned, so .obsidian/node_modules content stayed in keyword retrieval and the file listing.

single-sourced the prune policy in src/index_walk and wired both the vector and keyword walkers to it so they can't drift again. also made the junk-dir match case-insensitive, since Node_Modules on a case-insensitive filesystem slipped the old exact-match set. added keyword-path regressions covering hidden/junk dirs, hidden files, junk nested below the top level, and the case-insensitive case.

[StressTestor]

@RaresKeY heads up on the CodeQL path-injection High here. it's pre-existing (the os.walk predates this PR) and a false positive: the add_directory route already confines the path with realpath + commonpath under PERSONAL_DIR before it reaches os.walk, CodeQL just doesn't count commonpath as a barrier. the other callers are local by design (stdio mcp, cli). looks safe to dismiss.

[RaresKeY]

LGTM. The latest changes consistently filter hidden and junk directories across both vector and keyword indexing. Focused tests pass, and I found no actionable issues.