Add CV results-reporting framework + small real experiment script#25
Open
sainag7 wants to merge 8 commits into
Open
Add CV results-reporting framework + small real experiment script#25sainag7 wants to merge 8 commits into
sainag7 wants to merge 8 commits into
Conversation
sainag7
commented
Jun 22, 2026
sainag7
commented
Collaborator
- task_id logging through nested_lnso_cv (optional, backward compatible) so predictions/intervals can be broken down per subject and per task
- smoke_test_cross_val: thread trial id as task_id, persist subject_id_mapping
- plotting.results_io: load result JSON (single/multi-model) into tidy DataFrames
- plotting.result_figures: confusion matrix, per-subject accuracy, subject/task heatmap, hyperparameter sweep (1D/2D/3D), reliability + uncertainty (matplotlib + scikit-learn only)
- plotting.report: CLI -> <=4 multi-panel figures + mean+/-std metrics table (CSV/LaTeX/Markdown)
- run_experiment: small real nested-LNSO DREAMER run, reuses smoke data prep
- tests for results_io and result_figures; gitignore generated outputs
- task_id logging through nested_lnso_cv (optional, backward compatible) so predictions/intervals can be broken down per subject and per task - smoke_test_cross_val: thread trial id as task_id, persist subject_id_mapping - plotting.results_io: load result JSON (single/multi-model) into tidy DataFrames - plotting.result_figures: confusion matrix, per-subject accuracy, subject/task heatmap, hyperparameter sweep (1D/2D/3D), reliability + uncertainty (matplotlib + scikit-learn only) - plotting.report: CLI -> <=4 multi-panel figures + mean+/-std metrics table (CSV/LaTeX/Markdown) - run_experiment: small real nested-LNSO DREAMER run, reuses smoke data prep - tests for results_io and result_figures; gitignore generated outputs
There was a problem hiding this comment.
Pull request overview
This PR adds a results reporting pipeline for nested_lnso_cv runs, including JSON→DataFrame loading utilities, reusable matplotlib-based figure panels, and a CLI that generates a compact figure set plus a metrics table. It also threads an optional task_id through cross-validation logs to enable per-subject/per-task breakdowns, and adds a small “real experiment” runner script for DREAMER.
Changes:
- Add
eegproc.plotting.results_ioto load single- or multi-model CV JSON results into tidy DataFrames for downstream plotting/reporting. - Add
eegproc.plotting.result_figures+eegproc.plotting.reportto generate standard report figures/tables (with CLI entrypoint), plus tests for both. - Extend
nested_lnso_cvlogging to optionally includetask_id, and propagate session/trial IDs from DREAMER smoke/experiment scripts.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| src/tests/test_results_io.py | Adds unit tests covering single vs multi-model JSON shapes and optional task/mapping fields. |
| src/tests/test_result_figures.py | Adds headless plotting/report tests that render and save figures/tables to disk. |
| src/eegproc/plotting/results_io.py | Introduces JSON→tidy-DataFrames loader (load_results) and helper utilities. |
| src/eegproc/plotting/result_figures.py | Adds plotting primitives (confusion matrix, subject/task heatmap, sweeps, reliability/uncertainty). |
| src/eegproc/plotting/report.py | Adds report composition + CLI to write multi-panel figures and metrics tables. |
| src/eegproc/plotting/init.py | Re-exports new reporting/plotting APIs from the plotting package. |
| src/eegproc/deep_learning/smoke_test_cross_val.py | Adds task/session IDs to the prepared dataset and persists subject-id mapping into results JSON. |
| src/eegproc/deep_learning/run_experiment.py | Adds a small “real” nested-LNSO DREAMER experiment script writing results to experiment_outputs/. |
| src/eegproc/deep_learning/cross_val.py | Adds optional task_id_array plumbed through prediction + variational interval logs. |
| .gitignore | Ignores generated reporting artifacts (experiment_outputs/, figures*/). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+215
to
+217
| fig = _subject_figure(tables, model, metric) | ||
| _save(fig, out_dir, f"{prefix}_fig2_subjects", formats, dpi) | ||
| written.append(out_dir / f"{prefix}_fig2_subjects.{formats[0]}") |
Comment on lines
+222
to
+224
| fig = _hyperparameter_figure(tables, model, params, metric) | ||
| _save(fig, out_dir, f"{prefix}_fig3_hyperparameters", formats, dpi) | ||
| written.append(out_dir / f"{prefix}_fig3_hyperparameters.{formats[0]}") |
Comment on lines
+230
to
+232
| fig = _reliability_figure(tables, model) | ||
| _save(fig, out_dir, f"{prefix}_fig4_reliability", formats, dpi) | ||
| written.append(out_dir / f"{prefix}_fig4_reliability.{formats[0]}") |
Comment on lines
+246
to
+249
| def main() -> None: | ||
| import matplotlib | ||
|
|
||
| matplotlib.use("Agg") # CLI writes files only; force a headless backend. |
VitorInserra
requested changes
Jun 26, 2026
Collaborator
There was a problem hiding this comment.
Looks great! Very good work.
-
Could you please go over the copilot suggestions and see if they should be changed.
-
One small bug I found is that for the per subject bar chart it is not in order.
-
Could you also change the bar chart to a line chart.
-
Lastly, can you add functions to plot many models' accuracies?
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sai Nagamalla <shreyas.nagamalla@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sai Nagamalla <shreyas.nagamalla@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sai Nagamalla <shreyas.nagamalla@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sai Nagamalla <shreyas.nagamalla@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sai Nagamalla <shreyas.nagamalla@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Sai Nagamalla <shreyas.nagamalla@gmail.com>
2. Added function for model accuracy 3. Changed per object bar chart to be in order
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.