fix(personal): run directory indexing off the event loop

[StressTestor]

Summary

POST /api/personal/add_directory is an async def handler that called the fully synchronous rag.index_personal_documents inline, so the entire indexing job (os.walk, file reads, per-chunk embedding, Chroma inserts) ran on the event loop. Every other request queued behind it: indexing a real directory froze the UI and API for 25+ minutes with no sign of life (the reporter's users concluded the app crashed).

This PR moves the blocking section into the threadpool via fastapi.concurrency.run_in_threadpool. personal_docs_manager.add_directory stays inside the moved section because it triggers refresh_index(), which re-extracts text across tracked directories, also blocking work. A module-level lock serializes index jobs: without it the threadpool move would let concurrent requests run parallel jobs that race PersonalDocsManager's unsynchronized list mutations and plain open('w') file writes. Jobs previously serialized on the blocked loop anyway, so one-at-a-time is behavior parity, minus the freeze.

Target branch

• This PR targets dev, not main.

Linked Issue

Fixes #5558

Type of Change

• Bug fix (non-breaking — fixes a confirmed issue)

Checklist

• I searched open issues and open PRs — this is not a duplicate.
• This PR targets dev
• My changes are limited to the scope described above — no unrelated refactors or whitespace changes mixed in.
• I actually ran the app (uvicorn app:app + a local ChromaDB at localhost:8100) and verified the change works end-to-end.

How to Test

1. python -m pytest tests/test_add_directory_event_loop.py: 4 tests. The thread-identity test asserts indexing and bookkeeping run off the event loop thread (fails on dev, where both run on it); the serialization test asserts concurrent requests never run parallel index jobs; the other two pin response shape and the error path.
2. End-to-end (what i ran): start ChromaDB and the app, seed data/personal_docs/bigvault/ with 150 markdown files, POST /api/personal/add_directory, and while the job is running, time a GET /api/personal from a second terminal.
3. Results on this branch: the GET answers in 0.002s while indexing is in flight. Same probe on unpatched dev: the GET takes 3.2s, because it queues behind the whole remaining index job (with the reporter's 25-minute job, that GET waits minutes).
4. The index job itself completes identically on both: indexed_count: 150, failed_count: 0.

Also ran: full python -m pytest (4674 passed, 3 skipped; the 5 failures are the docker-socket tests, which fail identically on clean dev on this machine, macOS with no docker socket) and python -m compileall app.py core routes src services scripts tests.

Scope notes

• This is the issue's option 2 (threadpool), not option 3 (background job with id + progress). The request still blocks the client for the duration, but the app stays responsive for everyone. Option 3 is a feature-sized change (new endpoints + Knowledge UI work); happy to take it as a follow-up if wanted.
• A long index job still holds the HTTP request open, so aggressive proxy timeouts can drop the response while the job completes server-side. Pre-existing behavior, unchanged here.
• The lock covers add_directory jobs only. Now that the app stays responsive during indexing, an admin hitting POST /reload or DELETE /remove_directory mid-job could race the indexing thread's bookkeeping, a window that could not occur before because the blocked loop prevented any dispatch. Admin-only and narrow; the clean fix is a lock inside PersonalDocsManager itself, which i kept out of this PR to keep the diff reviewable. Can follow up.

[RaresKeY]

I checked the latest head against current dev. The single-request offload works, but its admission boundary still permits inconsistent index state and broad request stalls under overlap.

Findings

issue (concurrency): Admit and serialize directory mutations before offloading

• Problem: _index_job_lock is acquired only after an add has consumed a shared AnyIO worker token, and remove does not take it at all. One add/remove interleaving returned 200 for both requests but left tracking and vectors in a state no complete serial order can produce; separately, 40 queued adds borrowed all 40 default worker tokens and prevented an unrelated synchronous endpoint from starting.

• Impact: A successful removal can leave the directory re-tracked or its content partially searchable, while a burst or retry loop of authorized add requests can recreate the multi-minute application hang for synchronous routes and dependencies sharing the worker pool.

• Ask: Acquire asynchronous serialization admission before offloading an add, and run remove's complete tracking/vector transition through the same gate. Audit reload/upload/file/direct mutation paths against that boundary, keep admitted blocking work off the event loop, and add same-loop add-vs-remove plus worker-capacity regressions.

• Location: routes/personal_routes.py:220

Validation

• All exact-head required GitHub checks are green, including pytest and compileall.
• Focused and adjacent personal/RAG tests passed, and compile/diff checks are clean.
• Independent secretless route probes reproduced both impossible add/remove final states and the 40-token worker-capacity exhaustion.
• Not run: live Chroma/embedding concurrency, a real 25-minute directory load, multiple server processes, or real client-disconnect/proxy behavior.

[StressTestor]

pushed 13c7f6d. moved the job lock to an asyncio.Lock acquired in the handler before offloading, and routed add, remove, and reload through it. that closes both halves: remove/reload are now serialized against an in-flight add (the inconsistent-state path), and a queued request parks on the event loop instead of pinning a threadpool worker (the starvation). remove and reload also run their blocking work off the loop now.

added add-vs-remove and add-vs-reload serialization regressions, and converted the existing add-vs-add test to the same driver. they run async via ASGITransport because an asyncio.Lock deadlocks the sync TestClient portal (same reason test_notes_fail_closed_auth.py uses it). left the upload/delete sibling handlers and the partial-index-on-bookkeeping-failure window as separate follow-ups to keep this scoped.