Skip to content

⚡ Bolt: Optimize JSON file repository save_items batching #83

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ivangegovdve-sudo wants to merge 1 commit into
base: main
Choose a base branch
from
Open

⚡ Bolt: Optimize JSON file repository save_items batching #83

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .jules/bolt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,7 @@
## 2024-03-16 - Batch JSON File I/O Operations
**Learning:** When using JSON file-backed repositories, iterating over items sequentially and calling `save_item` or `record_attempt` inside a loop leads to N+1 file read/write operations. This creates a significant performance bottleneck, especially when importing progress snapshots with numerous items and attempts.
**Action:** Prefer batch processing methods (e.g., `save_items`, `record_attempts`) so file-backed adapters can load storage once, update it in memory, and write it back in a single pass.

## 2025-03-28 - In-place JSON Dictionary Updates
**Learning:** Fully deserializing and reserializing all stored records in JSON file-backed adapters (like `JsonFilePracticeRepository`) during batch operations (`save_items`) creates a significant O(N) CPU/memory bottleneck.
**Action:** When saving multiple records, avoid full domain conversions (`_item_from_dict` / `_item_to_dict` loops) for unchanged items. Instead, update the raw dictionary entries in-place before re-saving the file.
27 changes: 18 additions & 9 deletions src/python_learning_orchestrated/adapters/json_file_practice_repository.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,15 +46,24 @@ def save_items(self, items: list[LearningItem]) -> None:

storage = self._load_storage()
raw_items = storage.get("items", [])
existing_items = []
if isinstance(raw_items, list):
existing_items = [
_item_from_dict(entry) for entry in raw_items if isinstance(entry, dict)
]
by_id = {existing.id: existing for existing in existing_items}
for item in items:
by_id[item.id] = item
storage["items"] = [_item_to_dict(entry) for entry in by_id.values()]
if not isinstance(raw_items, list):
raw_items = []

# Avoid O(N) CPU/memory penalties by doing in-place updates of raw entries
items_by_id = {str(item.id): _item_to_dict(item) for item in items}

updated_raw_items = []
for entry in raw_items:
if isinstance(entry, dict):
entry_id = str(entry.get("id"))
if entry_id in items_by_id:
updated_raw_items.append(items_by_id.pop(entry_id))
else:
updated_raw_items.append(entry)

updated_raw_items.extend(items_by_id.values())
Comment on lines +52 to +64

@gemini-code-assist gemini-code-assist Bot Mar 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

This new implementation has a subtle change in behavior compared to the old one. If the JSON file contains items with duplicate ids, the old implementation would effectively de-duplicate them by ID, keeping the last one. This new implementation replaces only the first occurrence it finds and keeps subsequent duplicates. This can lead to unexpected data corruption if duplicate IDs exist in the storage file.

The proposed implementation below is not only simpler but also correctly preserves the de-duplication behavior of the original code by using a dictionary to merge existing and new items. This ensures that items are correctly updated or added while maintaining data integrity, and it still provides the desired performance optimization.

Suggested change
# Avoid O(N) CPU/memory penalties by doing in-place updates of raw entries
items_by_id = {str(item.id): _item_to_dict(item) for item in items}
updated_raw_items = []
for entry in raw_items:
if isinstance(entry, dict):
entry_id = str(entry.get("id"))
if entry_id in items_by_id:
updated_raw_items.append(items_by_id.pop(entry_id))
else:
updated_raw_items.append(entry)
updated_raw_items.extend(items_by_id.values())
# Create a dictionary of existing raw items by ID. This also de-duplicates
# items, preserving the behavior of the old implementation.
existing_items_by_id = {
str(entry.get("id")): entry
for entry in raw_items
if isinstance(entry, dict)
}
# Convert new/updated items to dicts and update the existing items
new_items_by_id = {str(item.id): _item_to_dict(item) for item in items}
existing_items_by_id.update(new_items_by_id)
updated_raw_items = list(existing_items_by_id.values())


storage["items"] = updated_raw_items
self._save_storage(storage)

def list_attempts(self) -> list[Attempt]:
Expand Down
Loading