Fix slow library loading and repeated blocked-thread warnings - #604
Open
PaoloElMarinelo wants to merge 1 commit into
Open
Fix slow library loading and repeated blocked-thread warnings#604PaoloElMarinelo wants to merge 1 commit into
PaoloElMarinelo wants to merge 1 commit into
Conversation
The local library screen could take minutes to load and spam the console with `io.vertx.core.VertxException: Thread blocked` warnings when the library contained many packs, especially with a large unofficial metadata database. Three compounding issues in the packs() request path: - DatabaseMetadataService re-read and re-parsed the entire unofficial metadata database file from disk on every single pack lookup, and rewrote the whole file on every refresh, instead of caching it in memory like the official database already was. - LibraryService.packs() read the most recent pack file of every group a second time to refresh its metadata, bypassing the existing short-lived pack cache and duplicating disk reads. - ArchiveStoryPackReader.readMetadata used a sequential zip stream, which has to inflate every asset entry just to skip past it, even though only story.json and thumbnail.png are needed. Switched to random-access reads via the zip's central directory. Fixes marian-m12l#603
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The local library screen could take minutes to load and spam the console with
io.vertx.core.VertxException: Thread blockedwarnings when the library contained many packs, especially with a large unofficial metadata database.Three compounding issues in the
packs()request path:DatabaseMetadataServicere-read and re-parsed the entire unofficial metadata database file from disk on every single pack lookup, and rewrote the whole file on every refresh, instead of caching it in memory like the official database already was.LibraryService.packs()read the most recent pack file of every group a second time to refresh its metadata, bypassing the existing short-lived pack cache and duplicating disk reads.ArchiveStoryPackReader.readMetadataused a sequential zip stream, which has to inflate every asset entry just to skip past it, even though onlystory.jsonandthumbnail.pngare needed. Switched to random-access reads via the zip's central directory.Fixes #603
Test plan
Verified against a real local library (249 pack files, ~46 unofficial database entries, 18.5 MB
unofficial.json) by building the full distribution and measuringGET /api/library/packs:Thread blockedwarnings (60s-260s+ per block), library load in the multiple-minutes rangemvn clean installbuild passes