V_epsilon: implement Brainstorm E observation tier + shared shape library#62
Merged
Conversation
Replaces the Option-A dimension-named scalar observation classes with Brainstorm E identity-named (property-named) classes; the typed composites become the shared shape library each property class pins as its value type. A is removed outright (not deprecated) per "no one is using A" in early dev. Scalar observations (isa scalar_observation; value pins a typed composite): body_weight_observation (mass), organ_volume_observation (volume), body_length_observation (length), age_observation (duration), core_temperature_observation (temperature), heart_rate_observation / respiration_rate_observation (frequency), blood_pressure_observation (pressure), litter_size_observation / cell_count_observation (count), body_condition_observation / behavioral_score_observation (score), concentration_observation (concentration), membrane_potential_observation (voltage), generic_scalar_observation (escape hatch, kept). Removed: mass_/length_/duration_/temperature_/pressure_/count_/score_/ frequency_/volume_observation. Categorical observations: categorical_observation is now an abstract genus; concrete classes are named by property (full E): developmental_stage_observation (pinned root UBERON:0000105), health_status_observation, behavioral_phenotype_observation, pigmentation_observation, estrous_stage_observation, generic_categorical_observation (escape hatch, free ontology_term). Series/expression families unchanged (already E-compatible, file-backed); no opaque_/standalone_/daq_ leaves, per Standalone_Removal_Decision. Updates index.json registrations, topics.json observation group, and the V_epsilon_SPEC.md observation section. All 96 schema tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
Promotes the typed value-composites to named abstract shape classes that own the `value` field, and rewires both tiers to inherit them as a second superclass (resolving E parent open issue #7): scalar_mass, scalar_length, scalar_duration, scalar_volume, scalar_temperature, scalar_pressure, scalar_frequency, scalar_voltage, scalar_current, scalar_concentration, scalar_count, scalar_score, generic_scalar, categorical_concept. Each scalar observation/manipulation now reads e.g. body_weight_observation isa scalar_observation, scalar_mass temperature_manipulation isa scalar_manipulation, scalar_temperature so the same shape class is shared by the read (observation) and write (manipulation) of that shape, `value` is defined once, and isa-by-shape sweeps both tiers. Categorical property classes inherit categorical_concept (developmental_stage and generic override the value binding). Uses the existing mixin pattern (cf. epochid in epoch_relative_reference); no meta-schema change. Updates index.json, topics.json (new shape_library group), and V_epsilon_SPEC.md. All 96 schema tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
…ation Brainstorm E Part 2: a series is a higher cardinality of the same observation, not a new class. Prototype (contained to one class): - core_temperature_observation overrides `value` to an ARRAY of the temperature composite (mustBeScalar: false); a single reading is the length-1 case. - adds `sample_time`, an array of duration offsets from the anchoring time_reference, element-aligned with value (value[i] at time_reference + sample_time[i]). This keeps the depends_on time_reference meaning "one moment, many frames" and puts the per-sample axis in embedded data. - worked example: examples/core_temperature_observation_series.json (a 3-point cooling curve). - tests/test_series_cardinality.py: asserts the array-of-structures shape and ships the len(value)==len(sample_time) check (D3) that JSON Schema cannot express on a single field. The array-of-composite shape (not a bare numeric matrix) preserves the query model's existential [*] search; same-sample value/time correlation is the documented limit. No meta-schema change (mustBeScalar is already an independent boolean). Full design + rollout: Series_As_Cardinality_Proposal.md (ndi-next-steps). All 103 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
ruff F401 — the new test module uses plain assertions, no pytest API. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
…table_row→obs) #1 of the migration work — textual conversion specifications for the E split: - treatment.md: rewritten from the conservative class-preserving conversion to the E SPLIT — treatment dispatches by ontology branch to injection / bath / procedural_manipulation / temperature_manipulation (and other scalar_manip) / environmental_manipulation, with focal-vs-ambient and structure as data (target_structure), numeric_value → typed value or companion observation, the Dab string_value→target_structure edge case, and out-of-tier routing for non-manipulation rows. Mostly 1→1 with branch dispatch. - ontology_table_row.md (new): the 1→N observation-tier split — each row → a scalar/categorical observation property class (or generic escape hatch), with subject-intrinsic and relational rows routed out of tier. Flags the required 1→N migrator engine capability. - _index.md: registers both split specs. These are the spec the migration code (#2, DID-matlab) implements; they are version-/engine-agnostic. Schema tests unaffected (103 pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
…ative_reference
Two independent additions:
1. subject_statement / subject_assertion (Subject_Statement_Decision):
- subject_statement (abstract, ← base): owns the subject_id anchor;
timeless (no time_reference).
- subject_assertion (concrete): timeless asserted facts —
asserted_property + value + source (species/strain/sex/genotype).
- subject_interaction re-parented under subject_statement; it adds the
required time_reference_# (subject_id now inherited). Subclasses tighten
(timed) the timeless parent; leaves still resolve subject_id +
time_reference (verified by inheritance flatten).
2. session_relative_reference (← time_reference): an ORDINAL anchor relative
to the session — dep session_id → session, relation enum {before, after,
at_start_of, at_end_of, concurrent_with, during}, no metric offset. For
interactions with neither a device clock nor a wall-clock date (e.g. an
awake behavioral test "at the end of the session"). This is the ordinal,
session-anchored sibling of event_relative (which is metric and targets a
subject_interaction, neither of which fits the no-clock/no-date case).
Updates index.json, topics.json (new subject_statement group; session_relative
added to time_reference group), and V_epsilon_SPEC.md. All 103 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
…he split specs treatment.md and ontology_table_row.md now describe the implemented anchor: each migrated interaction emits/links a session_relative_reference (relation=during, -> session via base.session_id) instead of a synthesized empty time_reference. Notes the 1->2 (treatment) and 1->N+1 (table, shared anchor) cardinality and that the 1->N engine capability has landed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
…aq-less anchor)
session_extent (<- base): depends_on session_id -> session; start/end timestamps
(UTC, at least one present), grounding enum {declared, epoch_derived,
reference_derived, developmental}, is_approximate. Lets a curator say
'the session began/ended at UTC x' for daq-less sessions; the same record is
derived from epochs when they exist. session_relative_reference resolves
through it (at_start_of->start, at_end_of->end, during->[start,end]), grounding
ordinal claims onto UTC. All 103 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
…allback Discovery showed the dominant form is parallel char fields (names/ variable_names/ontology_nodes/data), not a rows array; migrator falls back to migrating unchanged as ontology_table_row pending the per-row split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
…-column observations) Update the spec note from 'follow-up' to 'implemented': the parallel char-field layout (names/variable_names/ontology_nodes/data) splits one row into per-column observations; unrecognised layouts still fall back to migrating unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
The dominant V_epsilon corpus quarantine (~47K docs: 41,095 JH + 6,205 Dab) was "Required field categorical_concept.value is missing". Root cause: categorical_concept (the shape mixin) declared a required `value` in its own block, but the overrider classes (developmental_stage_ observation, generic_categorical_observation) re-declared their own `value`, so a migrated doc filled the concrete block and left the inherited categorical_concept.value empty. Resolve by flattening: categorical_concept.value now declares placement: concrete_class, so `value` lives in each categorical observation's OWN block and categorical_concept contributes no block. The two re-declared `value` fields are removed (one value per concrete class, no same-block name collision). developmental_stage_observation's UBERON:0000105 root is now a documented convention rather than a schema-enforced binding (the trade-off of the flatten approach). Docs updated to match: from_did_v1/ontology_table_row.md and V_epsilon_SPEC.md. All 103 schema tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
subject_group is deprecated in V_epsilon and folds into the subject tier (subject with is_group: true; membership becomes group_assignment events assembled in the NDI layer). Add the conversion doc and index row; the migrator lands in DID-matlab. No schema JSON change (subject.is_group and group_assignment already exist). 103 schema tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
Step 2 of Series_As_Cardinality_Proposal.md (accepted). Every scalar shape mixin's `value` is now an array of its composite (mustBeScalar: false) -- a single reading is the length-1 case, a curve is length N of the same class -- and the scalar_observation / scalar_manipulation genera carry a parallel, element-aligned `sample_time` array (optional; absent => single reading at the anchor). core_temperature_observation no longer overrides value/sample_time; it inherits the cardinality form from scalar_temperature / scalar_observation like every other scalar property class. This also clears the JH corpus quarantine "core_temperature_observation.value missing" (691): value now lives in the inherited scalar_temperature block that the migrator already fills, and mustBeScalar:false is strictly more permissive, so a single reading validates as the length-1 case (same scalar-struct-is-length-1 convention target_structure already uses -- no migrator change needed). Updates: 13 scalar mixins, scalar_observation/scalar_manipulation, core_temperature_observation (override removed), the worked example (value -> scalar_temperature block, sample_time -> scalar_observation block), test_series_cardinality.py (assert the mixin-level array shape), and V_epsilon_SPEC.md. 104 schema tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
…atment_transfer folds Document the three remaining deprecated-class folds (the migrators land in DID-matlab): treatment_drug -> injection(kind:drug), virus_injection -> injection(kind:virus, dilution in the mixture concentration), and treatment_transfer -> biological_transfer (recipient->subject, donor carried). Target classes already exist in draft; no schema JSON change. 104 schema tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
Investigated the did_v1 document set: the subject_group body is empty and no v1 document depends on a subject_group, so group membership is not recorded anywhere in v1. There is therefore no source to migrate into group_assignment events -- the subject_group -> subject(is_group) fold is the complete migration, and group_assignment is a forward-looking class for newly authored data, not a migration target. Update the conversion doc so this is a recorded decision rather than a dangling TODO. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
Move the 69 family-A migration classes from draft/ to stable/ and set maturity_level: "stable": the subject_interaction spine, the time_reference family, the scalar/categorical observation tiers + shape mixins, the manipulation tiers, and the annotation/event classes (group_assignment, placement, derivation, session_extent, interaction_purpose, stimulus_approach). These run clean against the discovery corpora (0 quarantine), so they are frozen for cutover. Left in draft/ (40 classes): the genomics/omics + file-backed dataseries data-format families (expression_*, sequence_read_data_*, reference_sequence_data_* / reference_annotation_data_*, dataseries_* / timeseries_* / imageseries_*, instrument) -- a separate effort not exercised by the physiology corpora. index.json tier/maturity/path updated for all 69; test_series_cardinality resolves classes across tiers; SPEC tier-status note added. The assembled epsilon-schemas set (stable + draft) is unchanged, so corpus validation is unaffected. 104 schema tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ABEeDJ83r9Hq9PaiSije8o
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Brainstorm E (identity-named tiers + a shared data-shape library) in
V_epsilon, replacing the Option-A observation tier. A is removed outright (not deprecated) — no one is using A in early development.See the developed catalog this implements:
ndi-next-steps→Summer 2026/1_Ingestion/20260615/Brainstorm_E_Class_Catalog.md.What changed
Shape library (new — abstract mixin classes,
← base)The typed value-composites are promoted to named classes that own the
valuefield and are mixed into the identity classes as a second superclass (resolves the E parent's open issue #7):scalar_mass,scalar_length,scalar_duration,scalar_volume,scalar_temperature,scalar_pressure,scalar_frequency,scalar_voltage,scalar_current,scalar_concentration,scalar_count,scalar_score,generic_scalar,categorical_concept.The same shape class is inherited by the observation that reads it and the manipulation that imposes it — e.g.
scalar_temperature← bothcore_temperature_observationandtemperature_manipulation— sovalueis defined once andisa scalar_temperaturesweeps both tiers. Uses the existing mixin pattern (cf.epochidinepoch_relative_reference); no meta-schema change.Observation tier — named by property (E)
Scalar dimension classes replaced by property classes (
class isa scalar_observation, scalar_<unit>):body_weight_observation,organ_volume_observation,body_length_observation,age_observation,core_temperature_observation,heart_rate_observation,respiration_rate_observation,blood_pressure_observation,litter_size_observation,cell_count_observation,body_condition_observation,behavioral_score_observation,concentration_observation,membrane_potential_observation,generic_scalar_observation.categorical_observationbecomes an abstract genus; concrete property classes inheritcategorical_concept:developmental_stage_observation(pinned rootUBERON:0000105),health_status_observation,behavioral_phenotype_observation,pigmentation_observation,estrous_stage_observation,generic_categorical_observation.Removed A classes:
mass_/length_/duration_/temperature_/pressure_/count_/score_/frequency_/volume_observation.Series/expression families unchanged (already E-compatible, file-backed); no
opaque_/standalone_/daq_leaves, perStandalone_Removal_Decision.Plumbing
index.jsonregistrations,topics.json(newshape_librarygroup + refreshedobservationsgroup), and theV_epsilon_SPEC.mdobservation section.Validation
All 96 schema tests pass locally. Field flattening verified: e.g.
body_weight_observationresolvesvalue(←scalar_mass) +measured_property/target_structure(←observation) +subject_id/time_reference(←subject_interaction).Deferred (not in this PR)
valueas array + paralleltime_reference) — needs a meta-schema array change.bindingconstraint for now.🤖 Generated with Claude Code
Generated by Claude Code