Skip to content

Move Parquet saving into a separate item list collector#1127

Open
mdekstrand wants to merge 6 commits into
lenskit:mainfrom
mdekstrand:feature/result-collector
Open

Move Parquet saving into a separate item list collector#1127
mdekstrand wants to merge 6 commits into
lenskit:mainfrom
mdekstrand:feature/result-collector

Conversation

@mdekstrand

Copy link
Copy Markdown
Member

This adds ParquetItemListCollector that saves item lists as we go to a Parquet file, and does some smaller type improvements and refactors to the data code.

@mdekstrand mdekstrand added this to the 2026.3 milestone Jun 30, 2026
@mdekstrand mdekstrand added the data Data management support. label Jun 30, 2026
@mdekstrand mdekstrand self-assigned this Jun 30, 2026
@mdekstrand mdekstrand moved this from Backlog to In Progress in LensKit Development Jun 30, 2026
@codecov

codecov Bot commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 20.20202% with 79 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.83%. Comparing base (a3282e8) to head (b41a6e1).

Files with missing lines Patch % Lines
src/lenskit/data/_collection/_parquet.py 17.77% 37 Missing ⚠️
src/lenskit/batch/_runner.py 0.00% 18 Missing ⚠️
src/lenskit/batch/_results.py 0.00% 9 Missing ⚠️
src/lenskit/data/_collection/_base.py 38.46% 8 Missing ⚠️
src/lenskit/batch/_queries.py 0.00% 6 Missing ⚠️
src/lenskit/batch/__init__.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1127       +/-   ##
===========================================
- Coverage   89.53%   17.83%   -71.71%     
===========================================
  Files         256      209       -47     
  Lines       17167    14419     -2748     
===========================================
- Hits        15370     2571    -12799     
- Misses       1797    11848    +10051     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data Data management support.

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant