Skip to content

Core port: storage, importer, reports, CLI and desktop UI (Phases 2–7)#2

Merged
kestrel99 merged 22 commits into
masterfrom
feat/core-storage
Jun 11, 2026
Merged

Core port: storage, importer, reports, CLI and desktop UI (Phases 2–7)#2
kestrel99 merged 22 commits into
masterfrom
feat/core-storage

Conversation

@kestrel99

Copy link
Copy Markdown
Owner

Summary

Builds out the .NET 10 / Avalonia port of Census across Phases 2–7: a clean domain model, SQLite storage, the NONMEM XML importer, report/export/archive, the CLI, the desktop UI replicating the original Census, and the release pipeline.

What's included

Domain & storage

  • Pure Census.Domain records (Run, Estimation, Parameter, FileArtifact).
  • SQLite store via Dapper with versioned, idempotent migrations (0001_initial, 0002_timing); cascade deletes; DeleteRun.

Importer (Census.Import)

  • NONMEM 7.2+ XML, namespace-prefix tolerant; validated against 276 real reference files (NM 7.3/7.4/7.5/7.6).
  • Theta/omega/sigma, SE, shrinkage (incl. flat SUBPOP1 layout), warnings.
  • Run/estimation timings; Comment from $PROBLEM title.
  • Condition number computed from the correlation matrix via a Jacobi eigensolver (NM 7.3 leaves the XML eigenvalue fields empty).

Reports / archive / CLI

  • CSV (CsvHelper), HTML & LaTeX (Scriban) writers; ZIP archive with JSON manifest.
  • census CLI: import, import-folder, export-run, compare, archive.

Desktop UI (Census.App, Avalonia)

  • Three-pane layout matching the original: icon toolbar, left filter tree, run grid, and estimation grid + parameter tabs.
  • Project management, recent files, run detail with theta/omega/sigma, import/export/archive/delete actions.
  • Settings dialog incl. user-configurable grid font size; native Windows fonts; Light theme.

Release pipeline

  • CI builds/tests on Windows/macOS/Linux; on tag, publishes self-contained single-file artifacts per RID, generates a GitHub Release with SHA-256 checksums. Code-signing stub in place (SignPath pending).

Tests

43 xUnit tests pass (importer incl. real-file corpus, storage round-trip, condition-number eigensolver). Release build clean.

Follow-ups (not in this PR)

  • PsN import (placeholder tab).
  • Legacy .lst listing parser (if needed).
  • Compare/Diagnostics/Matrices views (currently stubs/placeholders).
  • Code signing once SignPath is approved.
  • Re-import existing projects to populate newly computed fields (timings, condition numbers).

🤖 Generated with Claude Code

kestrel99 and others added 22 commits June 10, 2026 08:35
Adds Census.Storage with a forward-only embedded-SQL migration runner
(Migrator) and a Dapper-backed SqliteProjectStore (Create/Open/SaveRun/GetRuns).
Schema v1 covers runs, estimations, parameters, warnings, and file_artifacts
with cascade deletes and indexed foreign keys. Round-trip and idempotency
tests pass on all three platforms.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implements NonmemXmlImporter.Import() with XDocument/LINQ-to-XML parsing
robust to both NONMEM 7.2 (bare element names) and 7.6+ (nm: namespace
prefix) output. Extracts estimations, theta/omega/sigma parameters with
standard errors and shrinkage, condition number, OFV, warnings, and
ObsRecs/Individuals from problem_options. MD5 computed on source file.
Fixture run27.xml covers the NONMEM 7.6 format; inline XML tests cover
the 7.2 legacy etashrink fallback, termination warnings, and simulation
runs. 14/14 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Real NONMEM 7.3-7.5 output writes etashrink/etashrinksd as a single
SUBPOP1 row with one col per eta, not one row per eta. ParseMatrix was
only extracting shrinkage for the first eta in this format.

Add FlattenShrinkageValues helper that detects the layout and returns
an indexed list that ParseMatrix can use without further branching.

Add a test for the flat shrinkage format and RealWorldImporterTests that
validates all 276 NM 7.3/7.4/7.5/7.6 IQ reference XML files on this
machine parse without exceptions and report expected field coverage.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tWriter

Adds ParameterRow (internal record struct with RSE/CI computed fields),
CsvReportWriter (IReportWriter/CSV via CsvHelper, G6 invariant formatting),
and 5 xUnit tests covering header, theta values, omega shrinkage, no-estimations,
and null-SE edge case. All 25 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds HtmlReportWriter (IReportWriter) that renders runs to a complete HTML5
document via an embedded Scriban template, with snapshot and assertion tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…chive, compare)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…d run list

Replace the scaffold placeholder with a real application shell: DockPanel
layout with menu, toolbar, status bar, and a DataGrid showing run summary
columns (RunNo, Parent, Method, OFV, dOFV, ConditionNumber, Obs, Ind, Key,
Warnings). Add IDialogService/AvaloniaDialogService for file pickers,
ISettingsService/JsonSettingsService for persistent settings, AppSettings
model, RunSummaryViewModel, and wire up MainWindowViewModel with commands
for New/Open/Close/Exit project lifecycle.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add ImportRun, ImportFolder, ExportRun, and ArchiveRun relay commands to
MainWindowViewModel with CanExecute guards, and wire toolbar buttons and
File menu items in MainWindow.axaml.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a release job that runs after all publish jobs succeed on a tag,
downloads the four platform artifacts, renames them with RID suffixes,
and creates a GitHub Release with auto-generated notes.

Signing stubs remain as TODOs pending certificate provisioning.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Generates checksums.txt alongside binaries in each GitHub Release so
users can verify downloads independently of code signing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replicates the classic Census interface:
- Icon toolbar (New/Open Database, Import Run/Folder, Delete Run,
  Compare, Diagnostics, Report, Preferences, Quit) using Segoe Fluent
  Icons glyphs.
- Left sidebar with vertical "Project Overview / Run Files" tab strip,
  a "Filter" header, an "All runs" navigation tree, and an Expanded
  toggle that selects runs and drives the grid.
- Run list columns Run / Flag (status circle) / Comment / OFV / dOFV
  (red when non-zero) / Parent / Cond No / Warnings.
- Bottom split: "Estimation" steps grid on the left driving the
  parameter tabs (Theta/Omega/Sigma/Matrices/Miscellaneous/PsN) on the
  right; selecting a step updates the parameter tables.
- "Connected – <path>" status bar; Light theme variant to match the
  original look.

Adds IProjectStore.DeleteRun (cascades via FK) and wires Delete Run,
Compare and Diagnostics commands into the view model.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…PROBLEM

Importer now reads per-estimation estimation_elapsed_time and
covariance_elapsed_time, run-level start_datetime/stop_datetime/
total_cputime, and problem-level post_elapsed_time/finaloutput_elapsed_time.
Run.Comment defaults to the $PROBLEM title (problem_title) when no PsN
run name is present.

Schema migration 0002 adds the timing columns; SaveRun/GetRuns persist
and load them. The Estimation grid gains Est Time and Cov Time columns,
and the run-level timings are appended as labelled rows.

Tests: importer test for timings + problem-title comment; storage
round-trip extended to cover all new fields.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Top run grid gains Start, Stop and Total CPU columns (run-level
  values that apply to the whole run).
- Estimation grid gains Post Time and Final Out columns (per-problem
  values shown on each $EST step row); the previously appended
  run-level timing rows are removed.
- Estimation grid stays one row per $EST step; multi-$PROBLEM files
  remain out of scope (importer reads the first problem only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Grid cell and column-header font size now comes from a GridFontSize
  dynamic resource (default 11, one level smaller than the 12 used
  elsewhere). Configurable in Preferences and applied live.
- Persisted via AppSettings.GridFontSize and applied at startup.
- Override Fluent TabItemHeaderFontSize so bottom-right tab headings
  match the rest of the UI (12) instead of the oversized default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Real NONMEM 7.3 output leaves covariance_status eigenvalues zeroed and
has no <eigenvalues> element, so condition numbers were never populated.
The importer now derives the condition number (ratio of largest to
smallest eigenvalue of the correlation matrix) via a Jacobi eigensolver,
falling back through: explicit <eigenvalues> vector, populated
covariance_status, then the <correlation> matrix. Fixed parameters
(zero SE) are excluded, and non-positive-definite matrices yield no value.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add analytical 3x3 equicorrelation case (eigenvalues 2.0, 0.5 -> cond 4.0)
  to validate the Jacobi solver beyond the 2x2 case.
- Strengthen the real-world importer test to assert condition numbers now
  populate from the correlation matrix and are all positive and finite.

Note: none of the 276 reference runs printed eigenvalues (PRINT=E off),
so NONMEM reported no condition number to diff against; values are
validated analytically and for positivity/finiteness instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@kestrel99 kestrel99 merged commit de661b0 into master Jun 11, 2026
5 checks passed
@kestrel99 kestrel99 deleted the feat/core-storage branch June 11, 2026 15:05
kestrel99 added a commit that referenced this pull request Jun 15, 2026
PORTING_PROGRESS_REVIEW.md gap #2: the CLI/UI/API text promised "XML or
listing" and PsN import, but the only importer is NonmemXmlImporter (.xml).
Behaviour was already XML-only via CanImport; this aligns the promises.

- IRunImporter doc: state .xml is the only format today; listing/PsN planned.
- CLI: import/import-folder descriptions and the <source> argument help now
  say XML output, not "XML or listing".
- GUI import picker: drop the misleading "All Files" filter (XML-only).
- GUI ImportRun: guard with CanImport and report parse failures via status
  text instead of throwing unhandled.

dotnet test: 60 passed, 0 failed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant