Speed up project load for knowledge bases with many files. `GET…#396
Merged
Conversation
* perf(open-knowledge): serve /api/pages from in-memory index Cache page title and icon on the markdown file index, derived from the content the watcher already reads for its change-detection hash, so GET /api/pages serves them from memory instead of re-reading and re-parsing every file on every request. Previously handlePages did a synchronous readFileSync plus two frontmatter parses per file per request. On cold load that ran as a second full-directory read pass concurrently with the watcher seed walk, and it re-ran in full on every window focus and file-change refresh. For knowledge bases with many files this dominated project load time. Titles and icons now stay current through create, edit, and rename events; entries built without enrichment fall back to a one-off disk read, so behavior is unchanged. * test(open-knowledge): assert rename re-derives cached icon Addresses pr-review: the rename arm of updateFileIndex is a structurally distinct path (delete old entry, set new from derivePageMeta), so pin that it re-derives both title and icon from the new content rather than carrying a stale value forward. * docs(open-knowledge): restore markdownIndexView JSDoc displaced by derivePageMeta Address pr-review Minor: inserting derivePageMeta left its JSDoc above the helper and orphaned markdownIndexView's doc comment. Reorder so each comment sits above its own function. * test(open-knowledge): cover conflict-arm + ENOENT fallback; trim fallback comment Address local review on the /api/pages cache: - pin that the conflict disk-event arm enriches title/icon (it shares the create/update case in updateFileIndex but is a distinct trigger). - add a bare-entry ENOENT test: a missing file with no cached title falls back to the docName (the safe-default branch). - trim the handlePages fallback comment to the structural why (drop the process/test-artifact phrasing per comment discipline). GitOrigin-RevId: e5d94d7d9b2cc7809e0209d44f69dc0e09eb8730
![inkeep-internal-ci[bot]](https://avatars.githubusercontent.com/in/2806322?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated approval from agents-private public-mirror-sync (run: https://github.com/inkeep/agents-private/actions/runs/28469768924). Source of truth is the monorepo; direct edits on inkeep/open-knowledge are overwritten on next sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Speed up project load for knowledge bases with many files.
GET /api/pagespreviously re-read and re-parsed every markdown file from disk on every request — including the redundant full-directory pass that ran concurrently with the watcher's seed walk on cold load, and a fresh full re-read on every window focus / file-change refresh. Page titles and icons are now cached on the in-memory file index (derived from the content the watcher already reads for its change-detection hash, so no extra disk reads), and/api/pagesserves them straight from memory. Titles/icons stay current through create, edit, and rename events. Behavior is unchanged; only the cost of listing pages drops from O(files) disk reads per request to an in-memory scan.