From 4a7b37b4ebc2e840bf6d56089ab29cb9aa42dc94 Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 19:33:20 -0400 Subject: [PATCH 01/20] feat: add interaction-cycle CartPole challenges --- README.md | 8 +- site/src/content/docs/core/architecture.mdx | 20 +- site/src/content/docs/core/embodiment.mdx | 17 +- .../content/docs/core/interaction-cycle.mdx | 75 +++ site/src/content/docs/core/task-tour.mdx | 40 ++ .../src/content/docs/core/tools-artifacts.mdx | 1 + src/BrainlessLab.jl | 45 ++ src/api/Highlevel.jl | 121 ++++- src/core/Interfaces.jl | 41 ++ src/envs/CartPoleVariants.jl | 74 ++- src/envs/PlankCartPole.jl | 450 ++++++++++++++++++ src/run/ComponentCatalog.jl | 42 ++ src/run/EmbodimentConfig.jl | 9 +- src/run/Evaluation.jl | 232 +++++++++ src/tasks/Tasks.jl | 110 +++++ src/world/Embodiment.jl | 147 ++++-- src/world/Ensemble.jl | 79 ++- src/world/Interaction.jl | 287 +++++++++++ test/runtests.jl | 2 + test/test_component_catalog.jl | 4 + test/test_interaction_cycle.jl | 79 +++ test/test_plank_cartpole.jl | 139 ++++++ 22 files changed, 1936 insertions(+), 86 deletions(-) create mode 100644 site/src/content/docs/core/interaction-cycle.mdx create mode 100644 src/envs/PlankCartPole.jl create mode 100644 src/run/Evaluation.jl create mode 100644 src/world/Interaction.jl create mode 100644 test/test_interaction_cycle.jl create mode 100644 test/test_plank_cartpole.jl diff --git a/README.md b/README.md index 44f0339..fa3154e 100644 --- a/README.md +++ b/README.md @@ -61,10 +61,16 @@ NodeModel → Reservoir → AbstractBody → Agent → Ensemble{Environment} ``` `AbstractBody` is the public body boundary. `Embodiment` is the generic concrete -composition of geometry, sensors, encoders, actuators, dynamics, optional physiology, +composition of geometry, sensors, encoders, readouts, actuators, dynamics, optional physiology, stable ports, and runtime state. An `Ensemble` of one and an ensemble of many use the same synchronous lifecycle. +`FixedRateCycle` explicitly separates a world step from native neural frames. This supports +held inputs, temporal spike encoders, mean or instant reduction, and categorical voting +without putting task-specific timing branches into the simulation loop. Four experimental +Plank CartPole challenge profiles use this seam; Tracking and Pong remain the initial core +benchmark tasks. + `ObjectWorld` demonstrates composition of physical components, objects, fields, spectral appearance, and typed effects. It is not a calibrated benchmark. The established tracking and Pong tasks are the first core task contracts. diff --git a/site/src/content/docs/core/architecture.mdx b/site/src/content/docs/core/architecture.mdx index 78b4d4c..f5b274c 100644 --- a/site/src/content/docs/core/architecture.mdx +++ b/site/src/content/docs/core/architecture.mdx @@ -24,7 +24,7 @@ runs an ensemble of one and an ensemble of many. - **`Reservoir`** stores a population of nodes, wiring, and dynamic state. - **`AbstractBody`** is the dispatch boundary between a reservoir and a world. - **`Embodiment`** is the generic concrete body composition. -- **`Agent`** pairs one reservoir with one body. +- **`Agent`** pairs one reservoir with one body and one interaction cycle. - **`Environment`** owns external state and relations. - **`Ensemble`** advances one or more agents synchronously. - **`TaskSpec`** supplies setup, rollout defaults, and optional score metadata. @@ -39,6 +39,7 @@ runs an ensemble of one and an ensemble of many. | physical footprint | geometry component | | raw physical sample | sensor plus world sampling method | | raw sample to receptor channels | encoder | +| neural-frame outputs to one effector signal | readout | | effector values to bounded command | actuator | | command to motion | dynamics | | optional internal regulation, feedback, effects, and viability | physiology | @@ -55,9 +56,11 @@ the effect came from food. ```text prepare world → sample every body from the same pre-action state - → sample sensors and encode receptor ports - → append any physiology feedback receptors - → step each reservoir and read its effectors + → sample sensors once + → encode one or more native neural frames + → append any physiology feedback receptors to each frame + → step each reservoir for each neural frame + → reduce neural-frame outputs through the body's readout → decode typed commands → apply the full command frame → collect world effects @@ -68,6 +71,15 @@ prepare world No agent sees another agent's already-applied action from the same tick. This invariant matters for causal timing and replay. +`FixedRateCycle(K)` makes the world-to-neural clock explicit. The world is sampled once, +an encoder may distribute that observation over `K` neural frames, and the body emits one +command after its readout finishes. A reservoir may still own internal integration below +one neural frame. These are separate clocks; neither belongs in the task-specific world. + +The default `MeanReadout` preserves the prior held-input behavior. `InstantReadout` selects +the final neural frame. `VotingReadout` makes categorical frame voting explicit, including +its deterministic first-index tie rule. See [Interaction cycles](/core/interaction-cycle/). + ## Ports join the body and reservoir The body declares named receptor and effector ports: diff --git a/site/src/content/docs/core/embodiment.mdx b/site/src/content/docs/core/embodiment.mdx index e36bc40..9287982 100644 --- a/site/src/content/docs/core/embodiment.mdx +++ b/site/src/content/docs/core/embodiment.mdx @@ -18,7 +18,7 @@ An `Embodiment` composes: ```text geometry sensors → encoders → receptor ports -effector ports → actuators → typed commands → dynamics +neural-frame outputs → readout → effector ports → actuators → typed commands → dynamics optional physiology → feedback receptors, effect interpretation, viability traits + runtime state ``` @@ -36,8 +36,9 @@ n_effectors(body) ## Sensing and action stay separate A sensor declares and samples raw physical values. An encoder converts identified sensor -samples into reservoir receptor channels. An actuator converts normalized effector values -into a bounded command. A dynamics component integrates that command into motion. +samples into one or more reservoir receptor frames. A readout reduces neural-frame outputs +to one effector signal. An actuator converts that signal into a bounded typed command. A +dynamics component integrates that command into motion. The main path is: @@ -45,8 +46,9 @@ The main path is: world sample → sensor sample → encoder - → receptor vector - → reservoir + → receptor frame(s) + → reservoir frame(s) + → readout → effector vector → actuator → typed command @@ -54,12 +56,13 @@ world sample ``` The built-in physical component families include spectral cameras, mounted field probes, -bilateral contrast encoding, wheel or forward-turn actuation, and planar force/yaw -actuation. Query the catalog instead of assuming parameter names: +bilateral contrast encoding, mean/instant/voting readouts, wheel or forward-turn actuation, +and planar force/yaw actuation. Query the catalog instead of assuming parameter names: ```julia components(family=:sensor) component_info(:sensor, :spectral_camera) +component_info(:readout, :voting) readiness() ``` diff --git a/site/src/content/docs/core/interaction-cycle.mdx b/site/src/content/docs/core/interaction-cycle.mdx new file mode 100644 index 0000000..b2b5d4a --- /dev/null +++ b/site/src/content/docs/core/interaction-cycle.mdx @@ -0,0 +1,75 @@ +--- +title: Interaction cycles +description: One timing contract for held inputs, temporal encoders, neural-frame readouts, and synchronous action. +--- + +An interaction cycle defines how one world observation becomes one world action. It does +not define the task's trial count, reset policy, or statistical aggregation. + +```text +world time t + → sample world and sensors once + → begin encoder state + → for neural frame 1:K + encode receptor frame + step reservoir once + observe neural output in readout + → finish readout + → decode one typed actuator command + → apply all agents' commands synchronously +world time t + 1 +``` + +The three clocks are deliberately separate: + +| Clock | Owner | Example | +| --- | --- | --- | +| world step | environment and ensemble | one CartPole physics update | +| neural frame | `InteractionCycle` | 24 input/readout frames before that update | +| reservoir-internal substep | reservoir implementation | compartment integration within one frame | + +## The default contract + +`FixedRateCycle(K)` runs exactly `K` native reservoir frames per world step. The world is +sampled only once. A memoryless encoder repeats that sample; a temporal encoder may spread +it across frames. + +An embodiment owns exactly one readout in the standard runtime: + +- `MeanReadout` averages neural outputs, then applies the declared output projection; +- `InstantReadout` uses the final neural output; +- `VotingReadout` projects every frame, votes for a categorical effector, and resolves ties + by first index. + +The default cycle and `MeanReadout` preserve existing one-frame and held-input behavior. +Temporal behavior is therefore opt-in and visible in the resolved simulation configuration. + +## Why readout belongs to embodiment + +The reservoir produces neural state. The readout states how a body observes that state over +the interaction interval. The actuator then turns the resulting effector signal into a +bounded command. Keeping these distinct avoids mixing sensory encoding, temporal reduction, +and motor physics in one "decoder" object. + +Embodiment TOML may select a registered readout: + +```toml +[[components]] +id = "neural_readout" +family = "readout" +kind = "mean" +[components.parameters] +``` + +Custom temporal encoders extend `begin_encoding!` and `encode_frame!`. Custom readouts +extend `begin_readout!`, `observe_frame!`, and `finish_readout!`. Validate port widths and +cycle requirements before a rollout begins. + +## Evaluation is a different layer + +`EvaluationProtocol` governs complete trials: count, horizon, warmup, reset rule, whether a +constructed design is fixed, and aggregation. It does not alter the interaction cycle. +This distinction lets the same embodied agent run one diagnostic simulation, a held-out +benchmark, a sweep, or evolution without changing its sensorimotor semantics. + +

Source: src/world/Interaction.jl, src/world/Ensemble.jl, src/world/Embodiment.jl, src/run/Evaluation.jl.

diff --git a/site/src/content/docs/core/task-tour.mdx b/site/src/content/docs/core/task-tour.mdx index ac6004f..9dc291b 100644 --- a/site/src/content/docs/core/task-tour.mdx +++ b/site/src/content/docs/core/task-tour.mdx @@ -86,6 +86,46 @@ number of independent runs before treating the mean as reliable. These tasks use direct vector adapters. Their worlds already produce the reservoir receptor vector and consume a task-specific effector vector. +## Experimental CartPole challenge ladder + +Four additional task profiles reproduce the interface ladder proposed by Plank and +colleagues for neuromorphic systems: + +| Task | Observations | Actions | Encoder | Target raw fitness | +| --- | --- | --- | --- | ---: | +| `:cartpole_plank_easy` | position and velocity of cart and pole | left, right | Spike-FF-2 | 14,250 | +| `:cartpole_plank_medium` | position and velocity of cart and pole | no-op, left, right, with 75% no-op pressure | Spike-FF-2 | 12,000 | +| `:cartpole_plank_hard` | cart position and pole angle | no-op, left, right | Spike-FF-2 | 9,000 | +| `:cartpole_plank_hardest` | cart position and pole angle | left, right | Argyle-4 | 6,000 | + +All four run a 15,000-step mission with 24 native neural frames per world step and a +frame-voting readout. They are tagged `:challenge` and `:experimental`; they are deliberately +outside the Tracking/Pong core aggregate. The useful first result may simply be a clear +failure boundary for an otherwise capable design. + +For a quick integration check, use `simulate`. For the declared repeated-start contract, +use `evaluate_plank_cartpole`, which retains the complete initial-condition set and every raw +trial: + +```julia +result = evaluate_plank_cartpole( + :cartpole_plank_easy; + node=:falandays, + build_seed=11, + trial_seed=20_000, +) +``` + +The evaluation holds one constructed topology fixed and fully restores neural, plastic, and +body state between starts. Do not average the four levels into a generic competence score. +The Spike-FF-2 schedule is checked against the authors' public example. The present +Argyle-4 implementation follows the paper's adjacent-bin, nine-spike description, but is +marked as a BrainlessLab schedule until a source fixture establishes trajectory-level +conformance. + +Sources: [published benchmark paper](https://www.mdpi.com/2079-9268/15/1/5) and +[TENNLab CartPole example](https://github.com/TENNLab-UTK/framework-open/blob/main/markdown/cartpole_example.md). + ## Population tasks `:torus` and `:forage` use the established situated adapter. diff --git a/site/src/content/docs/core/tools-artifacts.mdx b/site/src/content/docs/core/tools-artifacts.mdx index 9bba463..c93a6ed 100644 --- a/site/src/content/docs/core/tools-artifacts.mdx +++ b/site/src/content/docs/core/tools-artifacts.mdx @@ -72,6 +72,7 @@ evidence status before opening any sealed output. | `sweep/run.jl ablate` | apply registered mechanism interventions | | `bench/` | compare a declared model roster on a declared task grid | | `evolve` and training tools | select fixed genomes on development tasks | +| `evaluate_plank_cartpole` | run the experimental repeated-start CartPole challenge contract | Examples: diff --git a/src/BrainlessLab.jl b/src/BrainlessLab.jl index 70e3b4f..2287dd3 100644 --- a/src/BrainlessLab.jl +++ b/src/BrainlessLab.jl @@ -36,6 +36,7 @@ include("world/BilateralSensing.jl") include("world/Body.jl") include("world/Motor.jl") include("world/PhysicalComponents.jl") +include("world/Interaction.jl") include("world/Embodiment.jl") include("world/SectorVision.jl") include("world/Homeostasis.jl") @@ -56,6 +57,7 @@ include("nodes/HomeostaticFlowV2.jl") include("envs/WallBox.jl") include("envs/Envs.jl") include("envs/CartPoleVariants.jl") +include("envs/PlankCartPole.jl") include("tasks/Scoring.jl") include("tasks/Tasks.jl") include("world/Environments.jl") @@ -86,6 +88,7 @@ include("drivers/Fixed.jl") include("drivers/Plastic.jl") include("run/EmbodimentConfig.jl") include("run/ComponentCatalog.jl") +include("run/Evaluation.jl") include("world/ObjectWorld.jl") include("tasks/ShoalForage.jl") include("analysis/ShoalForage.jl") @@ -128,6 +131,8 @@ export step!, rawspec, sample!, encode!, + begin_encoding!, + encode_frame!, encoder_sources, sense!, decode!, @@ -281,6 +286,18 @@ export TaskWorld, CartPoleHardEnv, CartPoleLongEnv, CartPoleSwingupEnv, + PlankCartPoleLevel, + PLANK_CARTPOLE_LEVELS, + PLANK_CARTPOLE_MISSION_STEPS, + PLANK_CARTPOLE_NEURAL_FRAMES, + PLANK_CARTPOLE_EVAL_EPISODES, + plank_cartpole_level, + SpikeFF2Encoder, + Argyle4Encoder, + PlankCartPoleEnv, + PlankCartPoleSetup, + plank_cartpole_fitness, + set_plank_cartpole_state!, cartpole_balancer, cartpole_swingup_controller, distance_last, @@ -309,6 +326,11 @@ export TaskSpec, CARTPOLE_HARD_TASK, CARTPOLE_SWINGUP_TASK, CARTPOLE_LONG_TASK, + PLANK_CARTPOLE_PROTOCOL, + CARTPOLE_PLANK_EASY_TASK, + CARTPOLE_PLANK_MEDIUM_TASK, + CARTPOLE_PLANK_HARD_TASK, + CARTPOLE_PLANK_HARDEST_TASK, TORUS_TASK, FORAGE_TASK, FORAGE_FLOOR_ANCHOR, @@ -394,6 +416,19 @@ export Agent, KinematicMotor, readout, readout_policy, + readout_components, + primary_readout, + begin_readout!, + observe_frame!, + finish_readout!, + InteractionCycle, + FixedRateCycle, + neural_frames, + default_interaction_cycle, + AbstractReadout, + MeanReadout, + InstantReadout, + VotingReadout, AbstractSensor, AbstractEncoder, IdentityEncoder, @@ -606,6 +641,7 @@ export SimResult, simulate, variants, tasks, + task_info, branching_ratio, branching_ratio_mr, branching_ratio_mr_windowed, @@ -667,6 +703,11 @@ export RunConfig, write_embodiment_config, materialize_embodiment, materialize_blueprint, + EvaluationProtocol, + EvaluationResult, + PLANK_CARTPOLE_EVALUATION, + plank_cartpole_initial_conditions, + evaluate_plank_cartpole, MountedFieldProbe, sample_field_probe!, BilateralContrastEncoder, @@ -782,6 +823,10 @@ register_task!(:cartpole, CARTPOLE_TASK) register_task!(:cartpole_hard, CARTPOLE_HARD_TASK) register_task!(:cartpole_swingup, CARTPOLE_SWINGUP_TASK) register_task!(:cartpole_long, CARTPOLE_LONG_TASK) +register_task!(:cartpole_plank_easy, CARTPOLE_PLANK_EASY_TASK) +register_task!(:cartpole_plank_medium, CARTPOLE_PLANK_MEDIUM_TASK) +register_task!(:cartpole_plank_hard, CARTPOLE_PLANK_HARD_TASK) +register_task!(:cartpole_plank_hardest, CARTPOLE_PLANK_HARDEST_TASK) register_task!(:torus, TORUS_TASK) register_task!(:forage, FORAGE_TASK) register_task!(:shoal_forage, SHOAL_FORAGE_TASK) diff --git a/src/api/Highlevel.jl b/src/api/Highlevel.jl index 1ba7c8d..df55bd2 100644 --- a/src/api/Highlevel.jl +++ b/src/api/Highlevel.jl @@ -87,11 +87,38 @@ Return the registered high-level node variant symbols. variants() = sort!(collect(keys(NODES))) """ - tasks() + tasks(; tag=nothing, status=nothing) -Return the registered high-level task symbols. +Return registered high-level task symbols, optionally filtered by one declared +task tag and/or stability status. Registration remains distinct from benchmark +participation: tags select active sets without hiding other tasks. """ -tasks() = sort!(collect(keys(TASKS))) +function tasks(; tag=nothing, status=nothing) + tag === nothing && status === nothing && return sort!(collect(keys(TASKS))) + tag_ = tag === nothing ? nothing : Symbol(tag) + status_ = status === nothing ? nothing : Symbol(status) + found = Symbol[] + for (name, task) in TASKS + task isa TaskSpec || continue + tag_ === nothing || tag_ in task.tags || continue + status_ === nothing || task.status === status_ || continue + push!(found, name) + end + return sort!(found) +end + +"""Return typed discovery metadata for one registered task.""" +function task_info(task::Union{Symbol,AbstractString,TaskSpec}) + spec = task isa TaskSpec ? task : resolve_task(Symbol(task)) + return ( + name=spec.name, + status=spec.status, + tags=spec.tags, + protocol=spec.protocol, + interaction_cycle=spec.interaction_cycle, + score_key=spec.score_key, + ) +end function _record_symbols(record) record === nothing && return Symbol[] @@ -753,9 +780,13 @@ function _validate_agent_ports(reservoir::Reservoir, body::AbstractBody) return nothing end -function _make_agent(reservoir::Reservoir, body::AbstractBody) +function _make_agent( + reservoir::Reservoir, + body::AbstractBody; + cycle::Union{Nothing,InteractionCycle}=nothing, +) _validate_agent_ports(reservoir, body) - return Agent(reservoir, body) + return Agent(reservoir, body; cycle=cycle) end function _setup_for_node_count( @@ -838,7 +869,11 @@ function _make_ensemble( node_kwargs=body_node_options, ablation=ablation, ) - agents[i] = _make_agent(reservoir, bodies[i]) + agents[i] = _make_agent( + reservoir, + bodies[i]; + cycle=task_spec.interaction_cycle, + ) end recorder = Recorder(enabled=record, every=every, compute_every=_spectral_compute_every(spectral_every)) @@ -1026,6 +1061,22 @@ _encoder_component_config(encoder::IdentityEncoder) = ( sources=encoder.source_ids, ) _encoder_component_config(encoder::SituatedEncoder) = (kind=:situated,) +_encoder_component_config(encoder::SpikeFF2Encoder) = ( + kind=:spike_ff_2, + scales=encoder.scales, + port_ids=encoder.port_ids, + sources=encoder.source_ids, + neural_frames=PLANK_CARTPOLE_NEURAL_FRAMES, +) +_encoder_component_config(encoder::Argyle4Encoder) = ( + kind=:argyle_4, + minima=encoder.minima, + maxima=encoder.maxima, + port_ids=encoder.port_ids, + sources=encoder.source_ids, + spike_schedule=_ARGYLE_9_FRAME_SCHEDULE, + implementation=:brainlesslab_v1, +) _encoder_component_config(encoder::AbstractEncoder) = ( kind=:custom, type=_config_type(encoder), @@ -1114,12 +1165,31 @@ _physiology_config(physiology::RegulatedPhysiology) = ( ) _physiology_config(physiology) = (kind=:custom, type=_config_type(physiology)) +_readout_config(readout_component::MeanReadout) = ( + kind=:mean, + policy=_motor_config(readout_policy(readout_component)), +) +_readout_config(readout_component::InstantReadout) = ( + kind=:instant, + policy=_motor_config(readout_policy(readout_component)), +) +_readout_config(readout_component::VotingReadout) = ( + kind=:voting, + tie_break=:lowest_index, + policy=_motor_config(readout_policy(readout_component)), +) +_readout_config(readout_component::AbstractReadout) = ( + kind=:custom, + type=_config_type(readout_component), +) + function _body_config(body::Embodiment) return ( kind=:embodiment, geometry=_geometry_config(body.geometry), sensors=Tuple(_sensor_component_config(sensor) for sensor in body.sensors), encoders=Tuple(_encoder_component_config(encoder) for encoder in body.encoders), + readouts=Tuple(_readout_config(readout_component) for readout_component in body.readouts), actuators=Tuple(_actuator_component_config(actuator) for actuator in body.actuators), dynamics=_dynamics_config(body.dynamics), physiology=_physiology_config(body.physiology), @@ -1130,6 +1200,15 @@ function _body_config(body::Embodiment) ) end +_interaction_cycle_config(cycle::FixedRateCycle) = ( + kind=:fixed_rate, + neural_frames=neural_frames(cycle), +) +_interaction_cycle_config(cycle::InteractionCycle) = ( + kind=:custom, + type=_config_type(cycle), +) + _body_config(body::AbstractBody) = ( kind=:custom, type=_config_type(body), @@ -1277,6 +1356,29 @@ function _environment_config(world::TaskWorld) ) end +function _environment_config(world::PlankCartPoleEnv) + return ( + kind=:task, + world=:plank_cartpole, + level=world.level.name, + observations=world.level.observation_indices, + actions=world.level.actions, + encoder=world.level.encoder, + target_fitness=world.level.target_fitness, + activity_threshold=world.level.activity_threshold, + mission_steps=PLANK_CARTPOLE_MISSION_STEPS, + tau=world.tau, + gravity=world.gravity, + force=world.force_mag, + pole_length=world.pole_length, + pole_mass=world.pole_mass, + cart_mass=world.cart_mass, + maximum_cart_position=world.max_x, + maximum_pole_angle=world.max_theta, + initial_ranges=world.initial_ranges, + ) +end + function _situated_config_value(m::SituatedEnvironment, name::Symbol) name === :n_agents && return length(m.positions) name === :visual_coupling && return m.visual_coupling @@ -1392,6 +1494,7 @@ function _simulation_config( slot=slot, body=_body_config(body_at_slot(c, slot)), network=network_snapshot(agent_at_slot(c, slot).reservoir), + interaction_cycle=_interaction_cycle_config(agent_at_slot(c, slot).cycle), ) for slot in 1:nagents(c) ) @@ -1411,6 +1514,12 @@ function _simulation_config( ablation=ablation, ablation_notes=Tuple(ablation_notes), interventions=interventions === nothing ? () : Tuple(interventions), + task_contract=task_spec isa TaskSpec ? ( + name=task_spec.name, + status=task_spec.status, + tags=task_spec.tags, + protocol=task_spec.protocol, + ) : nothing, outcome_contract=task_spec isa TaskSpec && task_spec.score_key !== nothing ? ( key=task_spec.score_key, floor=score_floor(task_spec), diff --git a/src/core/Interfaces.jl b/src/core/Interfaces.jl index e80ed8c..1b53a1c 100644 --- a/src/core/Interfaces.jl +++ b/src/core/Interfaces.jl @@ -332,6 +332,23 @@ Encode a raw percept into receptor values for a reservoir or body. """ function encode! end +""" + begin_encoding!(encoder_or_body, samples, cycle) + +Prepare one world-step observation for frame-wise receptor encoding. The +environment is sampled once per world step; an encoder may then expose a +different receptor vector on each neural frame without resampling the world. +""" +function begin_encoding! end + +""" + encode_frame!(encoder_or_body, state, frame, cycle) + +Encode one native neural frame from the state returned by +[`begin_encoding!`](@ref). +""" +function encode_frame! end + """ encoder_sources(encoder) @@ -399,6 +416,30 @@ Return the reservoir readout policy carried by an `AbstractBody`. """ function readout_policy end +""" + readout_components(body) + +Return the readout components that reduce neural-frame outputs into effector +signals. The standard runtime currently requires exactly one readout per body. +""" +function readout_components end + +""" + primary_readout(body) + +Return the single readout used by the standard interaction cycle. +""" +function primary_readout end + +"""Reset a readout accumulator at the start of one world step.""" +function begin_readout! end + +"""Observe one reservoir output during a neural frame.""" +function observe_frame! end + +"""Finish a readout reduction and return the effector signal.""" +function finish_readout! end + """ metrics(object, args...) diff --git a/src/envs/CartPoleVariants.jl b/src/envs/CartPoleVariants.jl index 4c92313..dce18ee 100644 --- a/src/envs/CartPoleVariants.jl +++ b/src/envs/CartPoleVariants.jl @@ -152,28 +152,64 @@ function sense(env::CartPoleVariantEnv) return sensors end -function _cartpole_step_state!(env::CartPoleVariantEnv, force::Real) - x = env.state[1] - x_dot = env.state[2] - theta = env.state[3] - theta_dot = env.state[4] +function _integrate_cartpole_state!( + state::Vector{Float64}, + force::Real; + tau::Real, + gravity::Real, + pole_length::Real, + pole_mass::Real, + total_mass::Real, + integrator::Symbol=:semi_implicit_euler, +) + length(state) == 4 || throw(DimensionMismatch( + "CartPole state must contain x, x_dot, theta, and theta_dot", + )) + x = state[1] + x_dot = state[2] + theta = state[3] + theta_dot = state[4] costheta = cos(theta) sintheta = sin(theta) - temp = (Float64(force) + env.pole_mass * env.pole_length * theta_dot^2 * sintheta) / env.total_mass - thetaacc = (env.gravity * sintheta - costheta * temp) / - (env.pole_length * ((4.0 / 3.0) - env.pole_mass * costheta^2 / env.total_mass)) - xacc = temp - env.pole_mass * env.pole_length * thetaacc * costheta / env.total_mass - - x_dot += env.tau * xacc - x += env.tau * x_dot - theta_dot += env.tau * thetaacc - theta += env.tau * theta_dot - - env.state[1] = Float64(x) - env.state[2] = Float64(x_dot) - env.state[3] = Float64(theta) - env.state[4] = Float64(theta_dot) + temp = (Float64(force) + pole_mass * pole_length * theta_dot^2 * sintheta) / total_mass + thetaacc = (gravity * sintheta - costheta * temp) / + (pole_length * ((4.0 / 3.0) - pole_mass * costheta^2 / total_mass)) + xacc = temp - pole_mass * pole_length * thetaacc * costheta / total_mass + + if integrator === :euler + x += tau * x_dot + x_dot += tau * xacc + theta += tau * theta_dot + theta_dot += tau * thetaacc + elseif integrator === :semi_implicit_euler + x_dot += tau * xacc + x += tau * x_dot + theta_dot += tau * thetaacc + theta += tau * theta_dot + else + throw(ArgumentError( + "unsupported CartPole integrator $(repr(integrator)); expected :euler or :semi_implicit_euler", + )) + end + + state[1] = Float64(x) + state[2] = Float64(x_dot) + state[3] = Float64(theta) + state[4] = Float64(theta_dot) + return state +end + +function _cartpole_step_state!(env::CartPoleVariantEnv, force::Real) + _integrate_cartpole_state!( + env.state, + force; + tau=env.tau, + gravity=env.gravity, + pole_length=env.pole_length, + pole_mass=env.pole_mass, + total_mass=env.total_mass, + ) return env end diff --git a/src/envs/PlankCartPole.jl b/src/envs/PlankCartPole.jl new file mode 100644 index 0000000..cb55880 --- /dev/null +++ b/src/envs/PlankCartPole.jl @@ -0,0 +1,450 @@ +const PLANK_CARTPOLE_MISSION_STEPS = 15_000 +const PLANK_CARTPOLE_NEURAL_FRAMES = 24 +const PLANK_CARTPOLE_EVAL_EPISODES = 1_000 + +"""Frozen task-interface definition for one Plank CartPole challenge level.""" +struct PlankCartPoleLevel + name::Symbol + observation_indices::Tuple{Vararg{Int}} + actions::Tuple{Vararg{Symbol}} + encoder::Symbol + target_fitness::Float64 + activity_threshold::Union{Nothing,Float64} +end + +const PLANK_CARTPOLE_LEVELS = ( + easy=PlankCartPoleLevel( + :easy, + (1, 2, 3, 4), + (:left, :right), + :spike_ff_2, + 14_250.0, + nothing, + ), + medium=PlankCartPoleLevel( + :medium, + (1, 2, 3, 4), + (:noop, :left, :right), + :spike_ff_2, + 12_000.0, + 0.75, + ), + hard=PlankCartPoleLevel( + :hard, + (1, 3), + (:noop, :left, :right), + :spike_ff_2, + 9_000.0, + nothing, + ), + hardest=PlankCartPoleLevel( + :hardest, + (1, 3), + (:left, :right), + :argyle_4, + 6_000.0, + nothing, + ), +) + +function plank_cartpole_level(level::Union{Symbol,AbstractString,PlankCartPoleLevel}) + level isa PlankCartPoleLevel && return level + name = Symbol(level) + hasproperty(PLANK_CARTPOLE_LEVELS, name) || throw(ArgumentError( + "unknown Plank CartPole level :$(name); use :easy, :medium, :hard, or :hardest", + )) + return getproperty(PLANK_CARTPOLE_LEVELS, name) +end + +const _PLANK_CARTPOLE_SCALES = (2.4, 2.0, 0.2095, 2.0) + +"""Two-bin flip-flop temporal spike encoder used by Easy, Medium, and Hard.""" +struct SpikeFF2Encoder{N,P<:Tuple,S<:Tuple} <: AbstractEncoder + scales::NTuple{N,Float64} + port_ids::P + source_ids::S +end + +function SpikeFF2Encoder(scales; prefix::Symbol=:cartpole, sources=()) + scales_ = Tuple(Float64(scale) for scale in scales) + all(scale -> isfinite(scale) && scale > 0.0, scales_) || throw(ArgumentError( + "SpikeFF2Encoder scales must be finite and positive", + )) + n = length(scales_) + ids = ntuple(2n) do index + observation = cld(index, 2) + sign = isodd(index) ? :negative : :positive + Symbol(prefix, :_, observation, :_, sign) + end + source_ids = Tuple(Symbol(source) for source in sources) + return SpikeFF2Encoder{n,typeof(ids),typeof(source_ids)}( + scales_, + ids, + source_ids, + ) +end + +encoder_sources(encoder::SpikeFF2Encoder) = + isempty(encoder.source_ids) ? nothing : encoder.source_ids +n_receptors(encoder::SpikeFF2Encoder) = length(encoder.port_ids) +portspec(encoder::SpikeFF2Encoder) = PortSpec( + n_receptors(encoder), + 0, + Port{NoPlacement}[Port(id) for id in encoder.port_ids], + Port{NoPlacement}[], +) + +mutable struct SpikeFF2State + counts::Vector{Int} + bins::Vector{Int} + frame::Vector{Float64} +end + +function begin_encoding!(encoder::SpikeFF2Encoder, samples, cycle::FixedRateCycle) + neural_frames(cycle) == PLANK_CARTPOLE_NEURAL_FRAMES || throw(ArgumentError( + "SpikeFF2Encoder protocol requires $(PLANK_CARTPOLE_NEURAL_FRAMES) neural frames, " * + "got $(neural_frames(cycle))", + )) + values = _component_float_vector(samples) + length(values) == length(encoder.scales) || throw(DimensionMismatch( + "SpikeFF2Encoder expected $(length(encoder.scales)) observations, got $(length(values))", + )) + counts = Vector{Int}(undef, length(values)) + bins = Vector{Int}(undef, length(values)) + @inbounds for index in eachindex(values) + value = Float64(values[index]) + isfinite(value) || throw(ArgumentError("CartPole observation $(index) must be finite")) + counts[index] = clamp(ceil(Int, 8.0 * abs(value) / encoder.scales[index]), 0, 8) + bins[index] = 2index - (value <= 0.0 ? 1 : 0) + end + return SpikeFF2State(counts, bins, zeros(n_receptors(encoder))) +end + +function encode_frame!( + ::SpikeFF2Encoder, + state::SpikeFF2State, + frame::Integer, + cycle::FixedRateCycle, +) + 1 <= frame <= neural_frames(cycle) || throw(BoundsError(1:neural_frames(cycle), frame)) + fill!(state.frame, 0.0) + # Authors' processor schedule uses Apply_Spike times 0, 3, ..., 21 followed + # by RUN 24. Julia frame 1 represents processor time 0. + slot = rem(frame - 1, 3) == 0 ? div(frame - 1, 3) + 1 : 0 + if 1 <= slot <= 8 + @inbounds for index in eachindex(state.counts) + slot <= state.counts[index] && (state.frame[state.bins[index]] = 1.0) + end + end + return state.frame +end + +"""Four-bin adjacent population encoder with nine conserved spikes per value.""" +struct Argyle4Encoder{N,P<:Tuple,S<:Tuple} <: AbstractEncoder + minima::NTuple{N,Float64} + maxima::NTuple{N,Float64} + port_ids::P + source_ids::S +end + +function Argyle4Encoder(scales; prefix::Symbol=:cartpole, sources=()) + scales_ = Tuple(Float64(scale) for scale in scales) + all(scale -> isfinite(scale) && scale > 0.0, scales_) || throw(ArgumentError( + "Argyle4Encoder scales must be finite and positive", + )) + n = length(scales_) + ids = ntuple(4n) do index + observation = cld(index, 4) + bin = mod1(index, 4) + Symbol(prefix, :_, observation, :_bin_, bin) + end + source_ids = Tuple(Symbol(source) for source in sources) + return Argyle4Encoder{n,typeof(ids),typeof(source_ids)}( + ntuple(index -> -scales_[index], n), + scales_, + ids, + source_ids, + ) +end + +encoder_sources(encoder::Argyle4Encoder) = + isempty(encoder.source_ids) ? nothing : encoder.source_ids +n_receptors(encoder::Argyle4Encoder) = length(encoder.port_ids) +portspec(encoder::Argyle4Encoder) = PortSpec( + n_receptors(encoder), + 0, + Port{NoPlacement}[Port(id) for id in encoder.port_ids], + Port{NoPlacement}[], +) + +mutable struct Argyle4State + first_bins::Vector{Int} + first_counts::Vector{Int} + second_bins::Vector{Int} + second_counts::Vector{Int} + frame::Vector{Float64} +end + +function begin_encoding!(encoder::Argyle4Encoder, samples, cycle::FixedRateCycle) + neural_frames(cycle) == PLANK_CARTPOLE_NEURAL_FRAMES || throw(ArgumentError( + "Argyle4Encoder protocol requires $(PLANK_CARTPOLE_NEURAL_FRAMES) neural frames, " * + "got $(neural_frames(cycle))", + )) + values = _component_float_vector(samples) + length(values) == length(encoder.minima) || throw(DimensionMismatch( + "Argyle4Encoder expected $(length(encoder.minima)) observations, got $(length(values))", + )) + first_bins = Vector{Int}(undef, length(values)) + first_counts = Vector{Int}(undef, length(values)) + second_bins = Vector{Int}(undef, length(values)) + second_counts = Vector{Int}(undef, length(values)) + @inbounds for index in eachindex(values) + value = Float64(values[index]) + isfinite(value) || throw(ArgumentError("CartPole observation $(index) must be finite")) + p = clamp( + (value - encoder.minima[index]) / + (encoder.maxima[index] - encoder.minima[index]), + 0.0, + 1.0, + ) + position = 3.0 * p + local_first = min(floor(Int, position) + 1, 4) + local_second = min(local_first + 1, 4) + second_count = local_first == local_second ? 0 : round(Int, 9.0 * (position - floor(position))) + first_bins[index] = 4(index - 1) + local_first + second_bins[index] = 4(index - 1) + local_second + second_counts[index] = clamp(second_count, 0, 9) + first_counts[index] = 9 - second_counts[index] + end + return Argyle4State( + first_bins, + first_counts, + second_bins, + second_counts, + zeros(n_receptors(encoder)), + ) +end + +const _ARGYLE_9_FRAME_SCHEDULE = (1, 4, 7, 10, 13, 15, 18, 21, 24) + +function encode_frame!( + ::Argyle4Encoder, + state::Argyle4State, + frame::Integer, + cycle::FixedRateCycle, +) + 1 <= frame <= neural_frames(cycle) || throw(BoundsError(1:neural_frames(cycle), frame)) + fill!(state.frame, 0.0) + slot = findfirst(==(Int(frame)), _ARGYLE_9_FRAME_SCHEDULE) + slot === nothing && return state.frame + @inbounds for index in eachindex(state.first_counts) + slot <= state.first_counts[index] && (state.frame[state.first_bins[index]] = 1.0) + slot <= state.second_counts[index] && (state.frame[state.second_bins[index]] = 1.0) + end + return state.frame +end + +mutable struct PlankCartPoleEnv{R} <: TaskWorld + rng::R + level::PlankCartPoleLevel + tau::Float64 + gravity::Float64 + force_mag::Float64 + pole_length::Float64 + pole_mass::Float64 + cart_mass::Float64 + total_mass::Float64 + max_x::Float64 + max_theta::Float64 + initial_ranges::NTuple{4,NTuple{2,Float64}} + state::Vector{Float64} + step_count::Int + noop_count::Int + done::Bool +end + +function PlankCartPoleEnv(; + rng=Random.default_rng(), + level=:easy, + initial_ranges=((-1.2, 1.2), (-0.05, 0.05), (-0.10475, 0.10475), (-0.05, 0.05)), +) + level_ = plank_cartpole_level(level) + ranges = Tuple((Float64(range[1]), Float64(range[2])) for range in initial_ranges) + length(ranges) == 4 || throw(DimensionMismatch("CartPole initial_ranges must contain four ranges")) + all(range -> range[1] <= range[2], ranges) || throw(ArgumentError( + "CartPole initial ranges must be ordered", + )) + state = [_cartpole_sample(rng, range) for range in ranges] + return PlankCartPoleEnv( + rng, + level_, + 0.02, + 9.8, + 10.0, + 0.5, + 0.1, + 1.0, + 1.1, + 2.4, + 0.2095, + ranges, + state, + 0, + 0, + false, + ) +end + +PlankCartPoleEnv(seed::Integer; kwargs...) = + PlankCartPoleEnv(; rng=MersenneTwister(seed), kwargs...) + +n_receptors(environment::PlankCartPoleEnv) = + environment.level.encoder === :argyle_4 ? 4length(environment.level.observation_indices) : + 2length(environment.level.observation_indices) +n_effectors(environment::PlankCartPoleEnv) = length(environment.level.actions) +default_ticks(::PlankCartPoleEnv) = PLANK_CARTPOLE_MISSION_STEPS +default_window(::PlankCartPoleEnv) = PLANK_CARTPOLE_MISSION_STEPS + +function sense(environment::PlankCartPoleEnv) + environment.done && return zeros(length(environment.level.observation_indices)) + return Float64[environment.state[index] for index in environment.level.observation_indices] +end + +function _plank_cartpole_action(environment::PlankCartPoleEnv, effectors) + values = _bounded_effectors(effectors, n_effectors(environment)) + _, winner = findmax(values) + return environment.level.actions[winner] +end + +function step!(environment::PlankCartPoleEnv, effectors) + environment.done && return environment + action = _plank_cartpole_action(environment, effectors) + action === :noop && (environment.noop_count += 1) + force = action === :left ? -environment.force_mag : + action === :right ? environment.force_mag : 0.0 + _integrate_cartpole_state!( + environment.state, + force; + tau=environment.tau, + gravity=environment.gravity, + pole_length=environment.pole_length, + pole_mass=environment.pole_mass, + total_mass=environment.total_mass, + # The source protocol executes Gym/Gymnasium CartPole's default + # explicit-Euler path. Keep this distinct from the legacy BrainlessLab + # variants, which predate these challenge profiles. + integrator=:euler, + ) + environment.step_count += 1 + environment.done = + abs(environment.state[1]) > environment.max_x || + abs(environment.state[3]) > environment.max_theta || + environment.step_count >= PLANK_CARTPOLE_MISSION_STEPS + return environment +end + +function reset!(environment::PlankCartPoleEnv) + @inbounds for index in eachindex(environment.state) + environment.state[index] = _cartpole_sample(environment.rng, environment.initial_ranges[index]) + end + environment.step_count = 0 + environment.noop_count = 0 + environment.done = false + return environment +end + +"""Install one explicit four-value test state and clear episode counters.""" +function set_plank_cartpole_state!(environment::PlankCartPoleEnv, state) + values = Tuple(Float64(value) for value in state) + length(values) == 4 || throw(DimensionMismatch( + "Plank CartPole state must contain x, x_dot, theta, and theta_dot", + )) + all(isfinite, values) || throw(ArgumentError("Plank CartPole state values must be finite")) + copyto!(environment.state, values) + environment.step_count = 0 + environment.noop_count = 0 + environment.done = false + return environment +end + +function plank_cartpole_fitness(environment::PlankCartPoleEnv) + threshold = environment.level.activity_threshold + threshold === nothing && return Float64(environment.step_count) + environment.step_count == 0 && return 0.0 + noop_fraction = environment.noop_count / environment.step_count + return noop_fraction > threshold ? + Float64(environment.step_count) : + Float64(environment.noop_count / threshold) +end + +function metrics(environment::PlankCartPoleEnv, window::Integer=PLANK_CARTPOLE_MISSION_STEPS) + fitness = plank_cartpole_fitness(environment) + return ( + name="cartpole_plank_$(environment.level.name)", + score=fitness, + fitness=fitness, + mission_fraction=environment.step_count / PLANK_CARTPOLE_MISSION_STEPS, + steps_balanced=environment.step_count, + noop_count=environment.noop_count, + noop_fraction=environment.step_count == 0 ? 0.0 : environment.noop_count / environment.step_count, + target_fitness=environment.level.target_fitness, + achieved=fitness >= environment.level.target_fitness, + fell=environment.step_count < PLANK_CARTPOLE_MISSION_STEPS, + xy_path=nothing, + ) +end + +scene(environment::PlankCartPoleEnv) = ( + kind=:cartpole, + x=environment.state[1], + theta=environment.state[3], + max_x=environment.max_x, + pole_length=environment.pole_length, +) + +"""Task setup that freezes the level's sensory, temporal, and motor interface.""" +struct PlankCartPoleSetup + level::PlankCartPoleLevel +end + +function (setup::PlankCartPoleSetup)(; + seed=0, + rng=nothing, + body=nothing, + n_nodes=nothing, + kwargs..., +) + body === nothing || throw(ArgumentError( + "Plank CartPole benchmark profiles freeze their embodiment; register a separate " * + "experimental task to change sensors, encoding, readout, or actuators", + )) + rng_ = rng === nothing ? MersenneTwister(Int(seed)) : rng + environment = PlankCartPoleEnv(; rng=rng_, level=setup.level, kwargs...) + indices = setup.level.observation_indices + scales = Tuple(_PLANK_CARTPOLE_SCALES[index] for index in indices) + sensor = DirectRelaySensor(length(indices)) + encoder = setup.level.encoder === :argyle_4 ? + Argyle4Encoder(scales; sources=(:cartpole_state,)) : + SpikeFF2Encoder(scales; sources=(:cartpole_state,)) + embodiment = Embodiment(; + sensors=(sensor,), + encoders=(encoder,), + readouts=(VotingReadout(),), + actuators=(DirectRelayActuator(setup.level.actions),), + traits=( + benchmark=:plank_cartpole, + level=setup.level.name, + interface_frozen=true, + ), + component_ids=( + geometry=:direct_geometry, + sensors=(:cartpole_state,), + encoders=(:cartpole_spike_encoder,), + readouts=(:winner_take_all,), + actuators=(:cartpole_action,), + dynamics=:environment_dynamics, + physiology=:physiology, + ), + ) + return TaskSetup(environment, [embodiment]) +end diff --git a/src/run/ComponentCatalog.jl b/src/run/ComponentCatalog.jl index fff7aa8..4970dd7 100644 --- a/src/run/ComponentCatalog.jl +++ b/src/run/ComponentCatalog.jl @@ -446,6 +446,21 @@ function _resolve_identity_encoder(config::ComponentConfig) return IdentityEncoder(port_ids; sources=source_ids) end +function _resolve_mean_readout(config::ComponentConfig) + _component_parameters(config; allowed=()) + return MeanReadout() +end + +function _resolve_instant_readout(config::ComponentConfig) + _component_parameters(config; allowed=()) + return InstantReadout() +end + +function _resolve_voting_readout(config::ComponentConfig) + _component_parameters(config; allowed=()) + return VotingReadout() +end + const _CAMERA_CHANNEL_CENTRES_NM = Dict( :ultraviolet => 365.0, @@ -877,6 +892,33 @@ function _register_builtin_component_catalog!() docs_path=core_docs, core_tests=(:core_differential_robot_roundtrip, :core_identity_encoder_composition), ), + _builtin_component_descriptor( + :readout, :mean, _resolve_mean_readout; + capabilities=(:config_materialization, :temporal_reduction, :graded_output), + parameters=(required=(), optional=()), + conformance=:mean_readout_contract, + conformance_path="test/test_interaction_cycle.jl", + example_path=robot_example, + readiness=:core, + docs_path=core_docs, + core_tests=(:core_differential_robot_roundtrip,), + ), + _builtin_component_descriptor( + :readout, :instant, _resolve_instant_readout; + capabilities=(:config_materialization, :temporal_reduction, :final_frame_output), + parameters=(required=(), optional=()), + conformance=:instant_readout_contract, + conformance_path="test/test_interaction_cycle.jl", + example_path="test/test_interaction_cycle.jl", + ), + _builtin_component_descriptor( + :readout, :voting, _resolve_voting_readout; + capabilities=(:config_materialization, :temporal_reduction, :categorical_output), + parameters=(required=(), optional=()), + conformance=:voting_readout_contract, + conformance_path="test/test_interaction_cycle.jl", + example_path="test/test_interaction_cycle.jl", + ), _builtin_component_descriptor( :actuator, :forward_turn, _resolve_forward_turn; capabilities=(:config_materialization, :effector_decode), diff --git a/src/run/EmbodimentConfig.jl b/src/run/EmbodimentConfig.jl index 05e64bc..de37141 100644 --- a/src/run/EmbodimentConfig.jl +++ b/src/run/EmbodimentConfig.jl @@ -5,7 +5,7 @@ const _EMBODIMENT_TOP_LEVEL_KEYS = Set(("schema_version", "name", "extends", "co const _EMBODIMENT_COMPONENT_KEYS = Set(("id", "family", "kind", "parameters")) const _EMBODIMENT_LEGACY_KEYS = Set(("ven", "body", "morphology", "motor")) const _COMPOSABLE_EMBODIMENT_FAMILIES = - (:geometry, :sensor, :encoder, :actuator, :dynamics, :physiology) + (:geometry, :sensor, :encoder, :readout, :actuator, :dynamics, :physiology) """Runtime-independent configuration for one named embodiment component.""" struct ComponentConfig{P<:NamedTuple} @@ -23,7 +23,7 @@ struct ComponentConfig{P<:NamedTuple} isempty(String(kind)) && throw(ArgumentError("component kind must not be empty")) family in (:body, :morphology, :motor, :ven) && throw(ArgumentError( "legacy component family :$(family) is unsupported; use generic families such as " * - ":geometry, :sensor, :encoder, :actuator, :dynamics, or :physiology", + ":geometry, :sensor, :encoder, :readout, :actuator, :dynamics, or :physiology", )) kind === :ven && throw(ArgumentError( "legacy component kind :ven is unsupported; compose the required generic components explicitly", @@ -380,12 +380,14 @@ function _validate_standard_embodiment_structure( geometry = _structure_family_components(subject, :geometry) sensors = _structure_family_components(subject, :sensor) + readouts = _structure_family_components(subject, :readout) actuators = _structure_family_components(subject, :actuator) dynamics = _structure_family_components(subject, :dynamics) physiology = _structure_family_components(subject, :physiology) for (family, components) in ( (:geometry, geometry), + (:readout, readouts), (:dynamics, dynamics), (:physiology, physiology), ) @@ -449,6 +451,7 @@ function _compose_embodiment(blueprint::EmbodimentBlueprint) physiology = _single_component(blueprint, :physiology, NoPhysiology()) sensors = _family_components(blueprint, :sensor) actuators = _family_components(blueprint, :actuator) + readouts = _family_components(blueprint, :readout) encoder_components = _family_components(blueprint, :encoder) isempty(sensors) && throw(ArgumentError("embodiment :$(blueprint.name) requires at least one sensor")) isempty(actuators) && throw(ArgumentError("embodiment :$(blueprint.name) requires at least one actuator")) @@ -469,6 +472,7 @@ function _compose_embodiment(blueprint::EmbodimentBlueprint) geometry=geometry.value, sensors=Tuple(component.value for component in sensors), encoders=Tuple(component.value for component in encoders), + readouts=isempty(readouts) ? nothing : Tuple(component.value for component in readouts), actuators=Tuple(component.value for component in actuators), dynamics=dynamics.value, physiology=physiology.value, @@ -478,6 +482,7 @@ function _compose_embodiment(blueprint::EmbodimentBlueprint) geometry=geometry.id, sensors=Tuple(component.id for component in sensors), encoders=Tuple(component.id for component in encoders), + readouts=isempty(readouts) ? (:readout_1,) : Tuple(component.id for component in readouts), actuators=Tuple(component.id for component in actuators), dynamics=dynamics.id, physiology=physiology.id, diff --git a/src/run/Evaluation.jl b/src/run/Evaluation.jl new file mode 100644 index 0000000..a280dd2 --- /dev/null +++ b/src/run/Evaluation.jl @@ -0,0 +1,232 @@ +""" + EvaluationProtocol + +Protocol-level sampling and reset rules for repeated evaluation. This is kept +separate from [`InteractionCycle`](@ref): the interaction cycle governs clocks +inside one world step, while this contract governs trials around a complete +task rollout. +""" +struct EvaluationProtocol + trials::Int + horizon::Int + warmup::Int + reset::Symbol + design_scope::Symbol + aggregation::Symbol + + function EvaluationProtocol(; + trials::Integer, + horizon::Integer, + warmup::Integer=0, + reset::Symbol=:full, + design_scope::Symbol=:fixed, + aggregation::Symbol=:mean, + ) + trials_ = Int(trials) + horizon_ = Int(horizon) + warmup_ = Int(warmup) + trials_ >= 1 || throw(ArgumentError("evaluation trials must be positive")) + horizon_ >= 1 || throw(ArgumentError("evaluation horizon must be positive")) + 0 <= warmup_ < horizon_ || throw(ArgumentError( + "evaluation warmup must lie in 0:(horizon - 1)", + )) + reset in (:full, :dynamic, :none) || throw(ArgumentError( + "evaluation reset must be :full, :dynamic, or :none", + )) + design_scope in (:fixed, :per_trial) || throw(ArgumentError( + "evaluation design_scope must be :fixed or :per_trial", + )) + aggregation in (:mean, :median, :raw) || throw(ArgumentError( + "evaluation aggregation must be :mean, :median, or :raw", + )) + return new(trials_, horizon_, warmup_, reset, design_scope, aggregation) + end +end + +const PLANK_CARTPOLE_EVALUATION = EvaluationProtocol( + trials=PLANK_CARTPOLE_EVAL_EPISODES, + horizon=PLANK_CARTPOLE_MISSION_STEPS, + reset=:full, + design_scope=:fixed, + aggregation=:mean, +) + +"""Raw, auditable result of one repeated evaluation protocol.""" +struct EvaluationResult{S,P,M} + task::Symbol + node::Symbol + build_seed::Int + trial_seed::Int + initial_conditions::Vector{S} + trials::Vector{NamedTuple} + protocol::P + summary::M +end + +function _evaluation_mean(values) + isempty(values) && return NaN + return sum(Float64(value) for value in values) / length(values) +end + +function _evaluation_median(values) + isempty(values) && return NaN + ordered = sort!(Float64.(collect(values))) + middle = length(ordered) ÷ 2 + return isodd(length(ordered)) ? ordered[middle + 1] : + (ordered[middle] + ordered[middle + 1]) / 2 +end + +""" + plank_cartpole_initial_conditions(seed, trials; ranges=...) + +Generate and return the complete held test set before evaluation. The explicit +states are retained in `EvaluationResult`, so a seed is not treated as a +portable replay guarantee. +""" +function plank_cartpole_initial_conditions( + seed::Integer, + trials::Integer; + ranges=((-1.2, 1.2), (-0.05, 0.05), (-0.10475, 0.10475), (-0.05, 0.05)), +) + count = Int(trials) + count >= 1 || throw(ArgumentError("CartPole evaluation trials must be positive")) + ranges_ = Tuple((Float64(range[1]), Float64(range[2])) for range in ranges) + length(ranges_) == 4 || throw(DimensionMismatch( + "CartPole initial-condition ranges must contain four ranges", + )) + all(range -> range[1] <= range[2], ranges_) || throw(ArgumentError( + "CartPole initial-condition ranges must be ordered", + )) + rng = MersenneTwister(Int(seed)) + return NTuple{4,Float64}[ + ntuple(index -> _cartpole_sample(rng, ranges_[index]), 4) + for _ in 1:count + ] +end + +function _require_plank_cartpole_task(task) + spec = _task_spec(task) + spec.setup isa PlankCartPoleSetup || throw(ArgumentError( + "evaluate_plank_cartpole requires one of the four :cartpole_plank_* task profiles", + )) + return spec +end + +function _reset_plank_trial!(ensemble::Ensemble, initial_condition, protocol::EvaluationProtocol) + environment = ensemble.environment + environment isa PlankCartPoleEnv || throw(ArgumentError( + "Plank CartPole evaluation received environment $(typeof(environment))", + )) + if protocol.reset === :full + foreach_group(ensemble) do group + for agent in group_agents(group) + reset!(agent) + end + end + elseif protocol.reset === :dynamic + foreach_group(ensemble) do group + for agent in group_agents(group) + reset!(agent.body) + end + end + end + set_plank_cartpole_state!(environment, initial_condition) + ensemble.t = 0 + return ensemble +end + +function _plank_trial!(ensemble::Ensemble, protocol::EvaluationProtocol) + environment = ensemble.environment + while !environment.done && environment.step_count < protocol.horizon + step!(ensemble) + end + outcome = metrics(environment, protocol.horizon) + return ( + fitness=Float64(outcome.fitness), + steps=Int(outcome.steps_balanced), + noop_fraction=Float64(outcome.noop_fraction), + achieved=Bool(outcome.achieved), + ) +end + +""" + evaluate_plank_cartpole(task; node=:falandays, protocol=PLANK_CARTPOLE_EVALUATION, + build_seed=0, trial_seed=10_000, kwargs...) + +Evaluate one fixed node construction across an explicit held set of CartPole +initial conditions. Each trial fully resets reservoir dynamics, plastic state, +and embodiment state while retaining the same constructed topology. Raw trials +and initial conditions are returned; the four challenge levels are never +collapsed into a cross-task aggregate. +""" +function evaluate_plank_cartpole( + task; + node=:falandays, + protocol::EvaluationProtocol=PLANK_CARTPOLE_EVALUATION, + build_seed::Integer=0, + trial_seed::Integer=10_000, + initial_conditions=nothing, + kwargs..., +) + spec = _require_plank_cartpole_task(task) + protocol.design_scope === :fixed || throw(ArgumentError( + "the current Plank CartPole evaluator requires design_scope=:fixed", + )) + protocol.warmup == 0 || throw(ArgumentError( + "the Plank CartPole protocol has no unscored warmup; use warmup=0", + )) + protocol.horizon <= PLANK_CARTPOLE_MISSION_STEPS || throw(ArgumentError( + "Plank CartPole horizon cannot exceed $(PLANK_CARTPOLE_MISSION_STEPS)", + )) + + states = initial_conditions === nothing ? + plank_cartpole_initial_conditions(trial_seed, protocol.trials) : + NTuple{4,Float64}[Tuple(Float64(value) for value in state) for state in initial_conditions] + length(states) == protocol.trials || throw(DimensionMismatch( + "evaluation protocol declares $(protocol.trials) trials but received $(length(states)) initial conditions", + )) + + node_ = Symbol(node) + setup = _build_ensemble( + spec, + node_; + ticks=protocol.horizon, + seed=Int(build_seed), + record=(), + kwargs..., + ) + ensemble = setup.ensemble + ensemble.recorder = nothing + trials = Vector{NamedTuple}(undef, protocol.trials) + for index in eachindex(states) + _reset_plank_trial!(ensemble, states[index], protocol) + trials[index] = _plank_trial!(ensemble, protocol) + end + + fitness = Float64[trial.fitness for trial in trials] + achieved = count(trial -> trial.achieved, trials) + aggregate = protocol.aggregation === :mean ? _evaluation_mean(fitness) : + protocol.aggregation === :median ? _evaluation_median(fitness) : nothing + summary = ( + n=length(trials), + mean_fitness=_evaluation_mean(fitness), + median_fitness=_evaluation_median(fitness), + minimum_fitness=minimum(fitness), + maximum_fitness=maximum(fitness), + target_fitness=spec.setup.level.target_fitness, + achieved=achieved, + achieved_fraction=achieved / length(trials), + aggregation=protocol.aggregation, + aggregate=aggregate, + ) + return EvaluationResult( + spec.name, + node_, + Int(build_seed), + Int(trial_seed), + states, + trials, + protocol, + summary, + ) +end diff --git a/src/tasks/Tasks.jl b/src/tasks/Tasks.jl index 1280f58..9436150 100644 --- a/src/tasks/Tasks.jl +++ b/src/tasks/Tasks.jl @@ -89,6 +89,10 @@ struct TaskSpec{S,E} <: AbstractTask n_effectors::Union{Nothing,Int} default_ticks::Int default_window::Int + interaction_cycle::Union{Nothing,InteractionCycle} + status::Symbol + tags::Tuple{Vararg{Symbol}} + protocol::NamedTuple floor::ScoreAnchor ceiling::ScoreAnchor score_key::Union{Nothing,Symbol} @@ -102,6 +106,10 @@ function TaskSpec( n_effectors::Integer=n_effectors(env_type), default_ticks::Integer=default_ticks(env_type), default_window::Integer=default_window(env_type), + interaction_cycle::Union{Nothing,InteractionCycle}=nothing, + status::Symbol=:stable, + tags=(), + protocol::NamedTuple=NamedTuple(), floor=nothing, ceiling=nothing, score_floor=nothing, @@ -117,6 +125,10 @@ function TaskSpec( n_effectors=n_effectors, default_ticks=default_ticks, default_window=default_window, + interaction_cycle=interaction_cycle, + status=status, + tags=tags, + protocol=protocol, floor=floor, ceiling=ceiling, score_floor=score_floor, @@ -134,6 +146,10 @@ function TaskSpec( n_effectors=nothing, default_ticks::Integer=1000, default_window::Integer=default_ticks, + interaction_cycle::Union{Nothing,InteractionCycle}=nothing, + status::Symbol=:stable, + tags=(), + protocol::NamedTuple=NamedTuple(), floor=nothing, ceiling=nothing, score_floor=nothing, @@ -142,6 +158,14 @@ function TaskSpec( descriptor_keys=Symbol[], ) where {S} task_name = Symbol(name) + status in (:reference, :stable, :experimental, :control, :alias) || + throw(ArgumentError( + "task :$(task_name) status must be :reference, :stable, :experimental, :control, or :alias", + )) + tags_ = Tuple(Symbol(tag) for tag in tags) + length(unique(tags_)) == length(tags_) || throw(ArgumentError( + "task :$(task_name) tags must be unique; got $(tags_)", + )) floor_anchor = _task_anchor( task_name, :floor, @@ -164,6 +188,10 @@ function TaskSpec( n_effectors === nothing ? nothing : Int(n_effectors), Int(default_ticks), Int(default_window), + interaction_cycle, + status, + tags_, + protocol, floor_anchor, ceiling_anchor, score_key, @@ -227,6 +255,8 @@ end const WALL_TASK = TaskSpec( :wall, WallEnv; + status=:experimental, + tags=(:extended,), floor=null_anchor(0.763125, "null=null_random, score_key=nav_score, seeds 0:7, git d420563, 2026-07-04"), ceiling=analytic(1.0; note="nav_score max = collision-free navigation while moving (a true analytic optimum); untrained falandays ref measured ~0.013 << null 0.763, so the analytic optimum is the honest ceiling, not a reference agent"), score_key=:nav_score, @@ -236,6 +266,8 @@ const WALL_TASK = TaskSpec( const TRACKING_TASK = TaskSpec( :tracking, TrackingEnv; + status=:reference, + tags=(:benchmark, :qualification, :core), floor=analytic(0.0; note="E[cos]=0 chance"), ceiling=analytic(1.0; note="perfect heading alignment"), score_key=:track_score, @@ -244,6 +276,8 @@ const TRACKING_TASK = TaskSpec( const PONG_TASK = TaskSpec( :pong, PongEnv; + status=:reference, + tags=(:benchmark, :qualification, :core), floor=null_anchor(0.3561507936507936, "null=null_random, score_key=hit_rate, seeds 0:7, git d420563, 2026-07-04"), ceiling=analytic(1.0; note="hit_rate max = intercept every ball (a true analytic optimum); no trained reference agent exists yet, so a reference-agent ceiling is a TODO(reference-genome)"), score_key=:hit_rate, @@ -252,6 +286,8 @@ const PONG_TASK = TaskSpec( const PONG_HITRATE_TASK = TaskSpec( :pong_hitrate, PongEnv; + status=:alias, + tags=(:alias,), floor=null_anchor(0.3561507936507936, "null=null_random, score_key=hit_rate, seeds 0:7, git d420563, 2026-07-04"), ceiling=analytic(1.0; note="hit_rate max = intercept every ball (a true analytic optimum); no trained reference agent exists yet, so a reference-agent ceiling is a TODO(reference-genome)"), score_key=:hit_rate, @@ -260,6 +296,8 @@ const PONG_HITRATE_TASK = TaskSpec( const CARTPOLE_TASK = TaskSpec( :cartpole, CartPoleEnv; + status=:experimental, + tags=(:extended, :control), floor=analytic(0.0; note="minimum balanced fraction"), ceiling=analytic(1.0; note="full episode balanced"), ) @@ -267,6 +305,8 @@ const CARTPOLE_TASK = TaskSpec( const CARTPOLE_HARD_TASK = TaskSpec( :cartpole_hard, CartPoleHardEnv; + status=:experimental, + tags=(:extended, :legacy_cartpole_variant), floor=analytic(0.0; note="minimum balanced fraction"), ceiling=analytic(1.0; note="full window balanced"), ) @@ -274,6 +314,8 @@ const CARTPOLE_HARD_TASK = TaskSpec( const CARTPOLE_SWINGUP_TASK = TaskSpec( :cartpole_swingup, CartPoleSwingupEnv; + status=:experimental, + tags=(:extended, :legacy_cartpole_variant), floor=null_anchor(0.1569039231621101, "null=null_random, score_key=mean_uprightness, seeds 0:7, git d420563, 2026-07-04"), ceiling=analytic(1.0; note="perfect uprightness"), score_key=:mean_uprightness, @@ -282,10 +324,78 @@ const CARTPOLE_SWINGUP_TASK = TaskSpec( const CARTPOLE_LONG_TASK = TaskSpec( :cartpole_long, CartPoleLongEnv; + status=:experimental, + tags=(:extended, :legacy_cartpole_variant), floor=analytic(0.0; note="minimum balanced fraction"), ceiling=analytic(1.0; note="full window balanced"), ) +const PLANK_CARTPOLE_PROTOCOL = ( + family=:plank_cartpole_2025, + source_doi="10.3390/jlpea15010005", + source_repository="TENNLab-UTK/framework-open", + source_path="markdown/cartpole_example.md", + mission_steps=PLANK_CARTPOLE_MISSION_STEPS, + neural_frames=PLANK_CARTPOLE_NEURAL_FRAMES, + evaluation_episodes=PLANK_CARTPOLE_EVAL_EPISODES, + reset_policy=:restore_all_dynamic_and_plastic_state, + topology_policy=:fixed_within_outer_block, + aggregation=:mean_raw_fitness, + cross_task_aggregate=false, + conformance=( + dynamics=:gym_default_explicit_euler, + spike_ff_2=:authors_source_example_matched, + argyle_4=:paper_specified_adjacent_bins_brainlesslab_schedule_v1, + voting=:authors_source_lower_index_tie, + ), +) + +function _plank_cartpole_task(level_name::Symbol) + level = plank_cartpole_level(level_name) + receptor_count = level.encoder === :argyle_4 ? + 4length(level.observation_indices) : + 2length(level.observation_indices) + return TaskSpec( + Symbol(:cartpole_plank_, level.name), + PlankCartPoleSetup(level); + env_type=PlankCartPoleEnv, + n_receptors=receptor_count, + n_effectors=length(level.actions), + default_ticks=PLANK_CARTPOLE_MISSION_STEPS, + default_window=PLANK_CARTPOLE_MISSION_STEPS, + interaction_cycle=FixedRateCycle(PLANK_CARTPOLE_NEURAL_FRAMES), + status=:experimental, + tags=(:challenge, :experimental, :extended, :plank_cartpole), + protocol=(; + PLANK_CARTPOLE_PROTOCOL..., + level=level.name, + observations=level.observation_indices, + actions=level.actions, + encoder=level.encoder, + target_fitness=level.target_fitness, + activity_threshold=level.activity_threshold, + ), + floor=analytic(0.0; note="minimum raw CartPole fitness"), + ceiling=analytic( + PLANK_CARTPOLE_MISSION_STEPS; + note="maximum mission time before the Medium activity adjustment", + ), + score_key=:fitness, + descriptor_keys=( + :mission_fraction, + :steps_balanced, + :noop_fraction, + :target_fitness, + :achieved, + ), + ) +end + +const CARTPOLE_PLANK_EASY_TASK = _plank_cartpole_task(:easy) +const CARTPOLE_PLANK_MEDIUM_TASK = _plank_cartpole_task(:medium) +const CARTPOLE_PLANK_HARD_TASK = _plank_cartpole_task(:hard) +const CARTPOLE_PLANK_HARDEST_TASK = _plank_cartpole_task(:hardest) + const FORAGE_FLOOR_ANCHOR = null_anchor(0.4556865216303779, "null=null_random, score_key=forage_score, seeds 0:7, git d420563, 2026-07-04") const FORAGE_CEILING_ANCHOR = diff --git a/src/world/Embodiment.jl b/src/world/Embodiment.jl index 69ac8e2..25d2202 100644 --- a/src/world/Embodiment.jl +++ b/src/world/Embodiment.jl @@ -588,10 +588,11 @@ end The single composed body type. Biological and robotic embodiments differ only in their component values; no morphology or organism subclass is required. """ -struct Embodiment{G<:AbstractGeometry,S<:Tuple,E<:Tuple,A<:Tuple,D<:AbstractDynamics,P<:AbstractPhysiology,T,St} <: AbstractBody +struct Embodiment{G<:AbstractGeometry,S<:Tuple,E<:Tuple,R<:Tuple,A<:Tuple,D<:AbstractDynamics,P<:AbstractPhysiology,T,St} <: AbstractBody geometry::G sensors::S encoders::E + readouts::R actuators::A dynamics::D physiology::P @@ -603,6 +604,7 @@ function Embodiment(; geometry::AbstractGeometry=NoGeometry(), sensors=(DirectRelaySensor(0),), encoders=(IdentityEncoder(0; prefix=:direct),), + readouts=nothing, actuators=(DirectRelayActuator(1),), dynamics::AbstractDynamics=NoDynamics(), physiology::AbstractPhysiology=NoPhysiology(), @@ -613,19 +615,26 @@ function Embodiment(; sensors_ = Tuple(sensors) encoders_ = Tuple(encoders) actuators_ = Tuple(actuators) + readouts_ = readouts === nothing ? _default_readouts(actuators_) : Tuple(readouts) isempty(sensors_) && throw(ArgumentError("Embodiment requires at least one sensor component")) isempty(encoders_) && throw(ArgumentError("Embodiment requires at least one encoder component")) + length(readouts_) == 1 || throw(ArgumentError( + "the standard Embodiment runtime currently requires exactly one readout component", + )) isempty(actuators_) && throw(ArgumentError("Embodiment requires at least one actuator component")) all(sensor -> sensor isa AbstractSensor, sensors_) || throw(ArgumentError("Embodiment sensors must all subtype AbstractSensor")) all(encoder -> encoder isa AbstractEncoder, encoders_) || throw(ArgumentError("Embodiment encoders must all subtype AbstractEncoder")) + all(readout -> readout isa AbstractReadout, readouts_) || + throw(ArgumentError("Embodiment readouts must all subtype AbstractReadout")) all(actuator -> actuator isa AbstractActuator, actuators_) || throw(ArgumentError("Embodiment actuators must all subtype AbstractActuator")) ids = _normalize_component_ids( component_ids, length(sensors_), length(encoders_), + length(readouts_), length(actuators_), ) encoders_, encoder_ids = _complete_encoder_components( @@ -634,7 +643,7 @@ function Embodiment(; ids = _normalize_component_ids((; ids..., encoders=encoder_ids, - ), length(sensors_), length(encoders_), length(actuators_)) + ), length(sensors_), length(encoders_), length(readouts_), length(actuators_)) encoder_groups = _encoder_groups(sensors_, encoders_, ids.sensors, ids.encoders) commands = Tuple(command_buffer(actuator) for actuator in actuators_) port_spec = _embodiment_portspec(ids, encoder_groups, actuators_, physiology) @@ -647,10 +656,11 @@ function Embodiment(; receptor_buffer, state, ) - return Embodiment{typeof(geometry),typeof(sensors_),typeof(encoders_),typeof(actuators_),typeof(dynamics),typeof(physiology),typeof(traits),typeof(state_)}( + return Embodiment{typeof(geometry),typeof(sensors_),typeof(encoders_),typeof(readouts_),typeof(actuators_),typeof(dynamics),typeof(physiology),typeof(traits),typeof(state_)}( geometry, sensors_, encoders_, + readouts_, actuators_, dynamics, physiology, @@ -659,6 +669,11 @@ function Embodiment(; ) end +function _default_readouts(actuators::Tuple) + policy = length(actuators) == 1 ? readout_policy(only(actuators)) : PASSTHROUGH_MOTOR + return (MeanReadout(policy),) +end + function _sensor_identity_port_ids(sensor_id::Symbol, sensor::AbstractSensor) spec = applicable(portspec, sensor) ? portspec(sensor) : nothing width = _raw_width(sensor) @@ -723,30 +738,52 @@ end _default_ids(prefix::Symbol, count::Int) = ntuple(i -> Symbol(prefix, :_, i), count) -function _normalize_component_ids(component_ids, nsensors::Int, nencoders::Int, nactuators::Int) +function _normalize_component_ids( + component_ids, + nsensors::Int, + nencoders::Int, + nreadouts::Int, + nactuators::Int, +) defaults = ( geometry=:geometry, sensors=_default_ids(:sensor, nsensors), encoders=_default_ids(:encoder, nencoders), + readouts=_default_ids(:readout, nreadouts), actuators=_default_ids(:actuator, nactuators), dynamics=:dynamics, physiology=:physiology, ) ids = component_ids === nothing ? defaults : component_ids + legacy_fields = (:geometry, :sensors, :encoders, :actuators, :dynamics, :physiology) + if propertynames(ids) == legacy_fields + ids = ( + geometry=ids.geometry, + sensors=ids.sensors, + encoders=ids.encoders, + readouts=defaults.readouts, + actuators=ids.actuators, + dynamics=ids.dynamics, + physiology=ids.physiology, + ) + end required = propertynames(defaults) propertynames(ids) == required || throw(ArgumentError( "component_ids must have fields $(required), got $(propertynames(ids))", )) sensors = Tuple(Symbol(id) for id in ids.sensors) encoders = Tuple(Symbol(id) for id in ids.encoders) + readouts = Tuple(Symbol(id) for id in ids.readouts) actuators = Tuple(Symbol(id) for id in ids.actuators) length(sensors) == nsensors || throw(DimensionMismatch("sensor component ID count does not match sensors")) length(encoders) == nencoders || throw(DimensionMismatch("encoder component ID count does not match encoders")) + length(readouts) == nreadouts || throw(DimensionMismatch("readout component ID count does not match readouts")) length(actuators) == nactuators || throw(DimensionMismatch("actuator component ID count does not match actuators")) normalized = ( geometry=Symbol(ids.geometry), sensors=sensors, encoders=encoders, + readouts=readouts, actuators=actuators, dynamics=Symbol(ids.dynamics), physiology=Symbol(ids.physiology), @@ -755,6 +792,7 @@ function _normalize_component_ids(component_ids, nsensors::Int, nencoders::Int, normalized.geometry, normalized.sensors..., normalized.encoders..., + normalized.readouts..., normalized.actuators..., normalized.dynamics, normalized.physiology, @@ -812,6 +850,7 @@ end sensor_components(body::Embodiment) = body.sensors encoder_components(body::Embodiment) = body.encoders +readout_components(body::Embodiment) = body.readouts actuator_components(body::Embodiment) = body.actuators function component_slots(body::Embodiment) ids = body.state.ids @@ -819,6 +858,7 @@ function component_slots(body::Embodiment) geometry=ComponentSlot(ids.geometry, body.geometry), sensors=Tuple(ComponentSlot(id, value) for (id, value) in zip(ids.sensors, body.sensors)), encoders=Tuple(ComponentSlot(id, value) for (id, value) in zip(ids.encoders, body.encoders)), + readouts=Tuple(ComponentSlot(id, value) for (id, value) in zip(ids.readouts, body.readouts)), actuators=Tuple(ComponentSlot(id, value) for (id, value) in zip(ids.actuators, body.actuators)), dynamics=ComponentSlot(ids.dynamics, body.dynamics), physiology=ComponentSlot(ids.physiology, body.physiology), @@ -829,7 +869,8 @@ situated_sensor(body::AbstractBody) = throw(ArgumentError( )) situated_sensor(body::Embodiment) = only(body.sensors) primary_actuator(body::Embodiment) = only(body.actuators) -readout_policy(body::Embodiment) = readout_policy(primary_actuator(body)) +primary_readout(body::Embodiment) = only(body.readouts) +readout_policy(body::Embodiment) = readout_policy(primary_readout(body)) _namespaced_port(component_id::Symbol, port) = Port{Any}(Symbol(component_id, :__, port.id), port.placement) @@ -958,18 +999,18 @@ function _flatten_raw_samples(samples::Tuple) return reduce(vcat, (_component_float_vector(sample) for sample in samples); init=Float64[]) end -function _encode_group(encoder::SituatedEncoder, sensors::Tuple, samples::Tuple) +function _encoder_input(encoder::SituatedEncoder, sensors::Tuple, samples::Tuple) length(sensors) == length(samples) == 1 || throw(DimensionMismatch( "SituatedEncoder consumes exactly one SituatedSensorLayout", )) only(sensors) isa SituatedSensorLayout || throw(ArgumentError( "SituatedEncoder requires a SituatedSensorLayout sensor", )) - return encode!(encoder, only(samples)) + return only(samples) end -function _encode_group(encoder::AbstractBilateralEncoder, sensors::Tuple, samples::Tuple) +function _encoder_input(encoder::AbstractBilateralEncoder, sensors::Tuple, samples::Tuple) length(sensors) == length(samples) == 2 || throw(DimensionMismatch( "bilateral encoders consume exactly two sensor sample vectors", )) @@ -983,48 +1024,93 @@ function _encode_group(encoder::AbstractBilateralEncoder, sensors::Tuple, sample paired[2channel - 1] = left[channel] paired[2channel] = right[channel] end - return encode!(encoder, paired) + return paired end -function _encode_group(encoder::AbstractEncoder, sensors::Tuple, samples::Tuple) +function _encoder_input(encoder::AbstractEncoder, sensors::Tuple, samples::Tuple) expected = sum(_raw_width, sensors; init=0) raw = _flatten_raw_samples(samples) length(raw) == expected || throw(DimensionMismatch( "raw sensor group declared width $(expected), got $(length(raw)) samples", )) - return encode!(encoder, raw) + return raw +end + +function _encode_group(encoder::AbstractEncoder, sensors::Tuple, samples::Tuple) + return encode!(encoder, _encoder_input(encoder, sensors, samples)) +end + +struct EmbodimentEncodingState{G} + groups::G + feedback_width::Int end -function _encoded_sensors!(output::Vector{Float64}, body::Embodiment, percept) +function begin_encoding!(body::Embodiment, percept, cycle::FixedRateCycle) samples = _sensor_samples(body, percept) - offset = 0 - for (_, encoder, sensors, indices) in _encoder_groups(body) + groups = map(_encoder_groups(body)) do (_, encoder, sensors, indices) selected_samples = Tuple(samples[index] for index in indices) - encoded = _component_float_vector(_encode_group(encoder, sensors, selected_samples)) + input = _encoder_input(encoder, sensors, selected_samples) + begin_encoding!(encoder, input, cycle) + end + sensor_width = sum( + n_receptors(_encoder_portspec(group[2], group[3])) + for group in _encoder_groups(body); + init=0, + ) + feedback_width = length(body.state.receptor_buffer) - sensor_width + feedback_width >= 0 || throw(DimensionMismatch( + "Embodiment encoders expose $(sensor_width) receptors, but its cached port " * + "contract has width $(length(body.state.receptor_buffer))", + )) + feedback = @view body.state.receptor_buffer[(sensor_width + 1):end] + physiology_feedback!(feedback, body.physiology) + return EmbodimentEncodingState(groups, feedback_width) +end + +function _encoded_sensor_frame!( + output::Vector{Float64}, + body::Embodiment, + encoding_states::Tuple, + frame::Integer, + cycle::FixedRateCycle, +) + length(encoding_states) == length(_encoder_groups(body)) || throw(DimensionMismatch( + "Embodiment has $(length(_encoder_groups(body))) encoder groups but received " * + "$(length(encoding_states)) encoding states", + )) + offset = 0 + for ((_, encoder, _, _), encoding_state) in zip(_encoder_groups(body), encoding_states) + encoded = _component_float_vector(encode_frame!(encoder, encoding_state, frame, cycle)) copyto!(output, offset + 1, encoded, 1, length(encoded)) offset += length(encoded) end return offset end -function sense!(body::Embodiment, percept) +function encode_frame!( + body::Embodiment, + state::EmbodimentEncodingState, + frame::Integer, + cycle::FixedRateCycle, +) + 1 <= frame <= neural_frames(cycle) || throw(BoundsError(1:neural_frames(cycle), frame)) output = body.state.receptor_buffer - base_width = _encoded_sensors!(output, body, percept) + base_width = _encoded_sensor_frame!(output, body, state.groups, frame, cycle) feedback_width = length(output) - base_width - feedback_width >= 0 || throw(DimensionMismatch( - "Embodiment encoded $(base_width) sensor receptors, but its cached port " * - "contract has width $(length(output))", - )) - feedback = @view output[(base_width + 1):length(output)] - physiology_feedback!(feedback, body.physiology) - base_width + length(feedback) == length(output) || throw(DimensionMismatch( - "Embodiment encoded $(base_width) sensor receptors and $(length(feedback)) " * - "physiology receptors, but its cached port contract has width $(length(output))", + feedback_width == state.feedback_width || throw(DimensionMismatch( + "Embodiment frame requires $(feedback_width) physiology receptors, but the " * + "held feedback state has $(state.feedback_width)", )) physiology_alive(body.physiology) || fill!(output, 0.0) return output end +function sense!(body::Embodiment, percept) + cycle = FixedRateCycle(1) + state = begin_encoding!(body, percept, cycle) + return encode_frame!(body, state, 1, cycle) +end + function decode!(body::Embodiment, values) length(values) == n_effectors(body) || throw(DimensionMismatch( "Embodiment expected $(n_effectors(body)) effectors, got $(length(values))", @@ -1051,7 +1137,13 @@ function update!(body::Embodiment, effects=()) return nothing end function reset!(body::Embodiment) - for component in (body.sensors..., body.encoders..., body.actuators..., body.dynamics) + for component in ( + body.sensors..., + body.encoders..., + body.readouts..., + body.actuators..., + body.dynamics, + ) applicable(reset!, component) && reset!(component) end applicable(reset!, body.state.user) && reset!(body.state.user) @@ -1070,6 +1162,7 @@ function _component_states(body::Embodiment) slots.geometry, slots.sensors..., slots.encoders..., + slots.readouts..., slots.actuators..., slots.dynamics, slots.physiology, diff --git a/src/world/Ensemble.jl b/src/world/Ensemble.jl index 0ee5652..f639da5 100644 --- a/src/world/Ensemble.jl +++ b/src/world/Ensemble.jl @@ -1,6 +1,31 @@ -struct Agent{R<:Reservoir,B<:AbstractBody} +struct Agent{R<:Reservoir,B<:AbstractBody,C<:InteractionCycle,S<:InteractionState} reservoir::R body::B + cycle::C + interaction::S +end + +function Agent( + reservoir::Reservoir, + body::AbstractBody; + cycle::Union{Nothing,InteractionCycle}=nothing, +) + cycle_ = cycle === nothing ? default_interaction_cycle(reservoir) : cycle + cycle_ isa FixedRateCycle || throw(ArgumentError( + "the standard runtime currently supports FixedRateCycle, got $(typeof(cycle_))", + )) + interaction = InteractionState(primary_readout(body), reservoir, body) + return Agent(reservoir, body, cycle_, interaction) +end + +Agent(reservoir::Reservoir, body::AbstractBody, cycle::InteractionCycle) = + Agent(reservoir, body; cycle=cycle) + +function reset!(agent::Agent) + reset!(agent.reservoir) + applicable(reset!, agent.body) && reset!(agent.body) + begin_interaction!(agent.interaction, primary_readout(agent.body), agent.cycle) + return agent end """Stable identity for an ensemble entity, independent of its world slot.""" @@ -623,6 +648,29 @@ end # (mirrors NoisyInput's transparent effectors/getproperty forwarding). readout(m::Motor, w::NoisyInput, spikes) = readout(m, getfield(w, :inner), spikes) +function _run_interaction!(agent::Agent, percept) + reservoir = agent.reservoir + body = agent.body + cycle = agent.cycle + readout_component = primary_readout(body) + state = agent.interaction + encoding_state = begin_encoding!(body, percept, cycle) + begin_interaction!(state, readout_component, cycle) + + @inbounds for frame in 1:neural_frames(cycle) + receptors = encode_frame!(body, encoding_state, frame, cycle) + observe_receptors!(state, receptors) + neural_output = step!(reservoir, receptors) + observe_frame!(state.readout, readout_component, reservoir, neural_output, frame) + end + + receptor_mean = finish_receptors!(state, cycle) + effector_signal = finish_readout!(state.readout, readout_component, reservoir, cycle) + command = decode!(body, effector_signal) + neural_mean = recorded_neural_output(state.readout, cycle) + return receptor_mean, neural_mean, command +end + function _step_homogeneous!(c::Ensemble, store::HomogeneousStore) agents = store.agents bodies = _agent_bodies(store) @@ -632,32 +680,26 @@ function _step_homogeneous!(c::Ensemble, store::HomogeneousStore) length(percepts) == length(agents) || throw(DimensionMismatch("environment returned $(length(percepts)) percepts for $(length(agents)) agents")) - receptor_vectors = Vector{Vector{Float64}}(undef, length(agents)) - @inbounds for i in eachindex(agents) - body = agents[i].body - receptor_vectors[i] = alive(body) ? - _receptor_vector(sense!(body, percepts[i])) : - zeros(Float64, n_receptors(body)) - end - remember_receptors!(c.environment, receptor_vectors) - spikes = Vector{Vector{Float64}}(undef, length(agents)) + receptor_vectors = Vector{Vector{Float64}}(undef, length(agents)) rates = Vector{Float64}(undef, length(agents)) Es = _homogeneous_command_storage(bodies) @inbounds for i in eachindex(agents) agent = agents[i] if alive(agent.body) - s = step_window!(agent.reservoir, receptor_vectors[i]) - E = readout(readout_policy(agent.body), agent.reservoir, s) - Es[i] = decode!(agent.body, E) + receptors, s, command = _run_interaction!(agent, percepts[i]) + receptor_vectors[i] = _receptor_vector(receptors) + Es[i] = command else + receptor_vectors[i] = zeros(Float64, n_receptors(agent.body)) s = zeros(Float64, n_nodes(agent.reservoir)) Es[i] = _inactive_command(agent.body) end spikes[i] = s rates[i] = _spike_rate(s) end + remember_receptors!(c.environment, receptor_vectors) effects = apply_commands!(c.environment, bodies, Es) @inbounds for i in eachindex(bodies) @@ -685,15 +727,12 @@ function _step_agent_group!( agents = group.agents @inbounds for (local_index, slot) in enumerate(group.slots) agent = agents[local_index] - receptors = alive(agent.body) ? - _receptor_vector(sense!(agent.body, percepts[slot])) : - zeros(Float64, n_receptors(agent.body)) - receptor_vectors[slot] = receptors if alive(agent.body) - s = step_window!(agent.reservoir, receptors) - output = readout(readout_policy(agent.body), agent.reservoir, s) - commands[slot] = decode!(agent.body, output) + receptors, s, command = _run_interaction!(agent, percepts[slot]) + receptor_vectors[slot] = _receptor_vector(receptors) + commands[slot] = command else + receptor_vectors[slot] = zeros(Float64, n_receptors(agent.body)) s = zeros(Float64, n_nodes(agent.reservoir)) commands[slot] = _inactive_command(agent.body) end diff --git a/src/world/Interaction.jl b/src/world/Interaction.jl new file mode 100644 index 0000000..b577758 --- /dev/null +++ b/src/world/Interaction.jl @@ -0,0 +1,287 @@ +""" + InteractionCycle + +Abstract timing contract between one world step and the native neural frames +used to sense, update a reservoir, and form one command. Reservoir-internal +integration remains owned by the reservoir and is not represented here. +""" +abstract type InteractionCycle end + +""" + FixedRateCycle(neural_frames=1) + +Run exactly `neural_frames` native reservoir updates for every world step. The +world is sampled once, frame encoders may distribute that sample through time, +and the selected readout reduces the frame outputs to one effector signal. +""" +struct FixedRateCycle <: InteractionCycle + neural_frames::Int + + function FixedRateCycle(neural_frames::Integer=1) + frames = Int(neural_frames) + frames >= 1 || throw(ArgumentError( + "FixedRateCycle neural_frames must be positive, got $(frames)", + )) + return new(frames) + end +end + +neural_frames(cycle::FixedRateCycle) = cycle.neural_frames + +"""Default legacy-compatible cycle for a reservoir.""" +function default_interaction_cycle(reservoir::Reservoir) + frames = windowing(reservoir) isa IntrinsicWindow ? 1 : temporal_window(reservoir) + return FixedRateCycle(frames) +end + +"""Abstract neural-output reduction carried by an embodiment.""" +abstract type AbstractReadout end + +""" + MeanReadout(policy=PASSTHROUGH_MOTOR) + +Mean-reduce native neural outputs over the interaction cycle, then project the +result through the legacy reservoir readout policy. This exactly represents the +previous held-input temporal-window behavior and is the default compatibility +readout. +""" +struct MeanReadout{P} <: AbstractReadout + policy::P +end + +"""Use only the final native neural output in the cycle.""" +struct InstantReadout{P} <: AbstractReadout + policy::P +end + +""" + VotingReadout(policy=PASSTHROUGH_MOTOR) + +Project each neural frame to effectors, award one vote to the first maximal +effector, and emit a one-hot signal for the first maximal vote total. The stable +first-index tie rule matches the Plank CartPole protocol. +""" +struct VotingReadout{P} <: AbstractReadout + policy::P +end + +MeanReadout() = MeanReadout(PASSTHROUGH_MOTOR) +InstantReadout() = InstantReadout(PASSTHROUGH_MOTOR) +VotingReadout() = VotingReadout(PASSTHROUGH_MOTOR) + +readout_policy(readout::AbstractReadout) = readout.policy + +mutable struct MeanReadoutState + neural_sum::Vector{Float64} + neural_mean::Vector{Float64} +end + +mutable struct InstantReadoutState + neural_sum::Vector{Float64} + neural_mean::Vector{Float64} + neural_last::Vector{Float64} +end + +mutable struct VotingReadoutState + neural_sum::Vector{Float64} + neural_mean::Vector{Float64} + votes::Vector{Int} + signal::Vector{Float64} +end + +function readout_state(::MeanReadout, reservoir::Reservoir) + return MeanReadoutState(zeros(n_nodes(reservoir)), zeros(n_nodes(reservoir))) +end + +function readout_state(::InstantReadout, reservoir::Reservoir) + return InstantReadoutState( + zeros(n_nodes(reservoir)), + zeros(n_nodes(reservoir)), + zeros(n_nodes(reservoir)), + ) +end + +function readout_state(::VotingReadout, reservoir::Reservoir) + return VotingReadoutState( + zeros(n_nodes(reservoir)), + zeros(n_nodes(reservoir)), + zeros(Int, n_effectors(reservoir)), + zeros(n_effectors(reservoir)), + ) +end + +function begin_readout!(state::MeanReadoutState, ::MeanReadout, ::FixedRateCycle) + fill!(state.neural_sum, 0.0) + fill!(state.neural_mean, 0.0) + return state +end + +function begin_readout!(state::InstantReadoutState, ::InstantReadout, ::FixedRateCycle) + fill!(state.neural_sum, 0.0) + fill!(state.neural_mean, 0.0) + fill!(state.neural_last, 0.0) + return state +end + +function begin_readout!(state::VotingReadoutState, ::VotingReadout, ::FixedRateCycle) + fill!(state.neural_sum, 0.0) + fill!(state.neural_mean, 0.0) + fill!(state.votes, 0) + fill!(state.signal, 0.0) + return state +end + +function _require_neural_width(state_values, neural_output) + length(neural_output) == length(state_values) || throw(DimensionMismatch( + "readout expected $(length(state_values)) neural outputs, got $(length(neural_output))", + )) + return neural_output +end + +function observe_frame!( + state::MeanReadoutState, + ::MeanReadout, + ::Reservoir, + neural_output, + frame::Integer, +) + _require_neural_width(state.neural_sum, neural_output) + @inbounds for index in eachindex(state.neural_sum, neural_output) + state.neural_sum[index] += Float64(neural_output[index]) + end + return state +end + +function observe_frame!( + state::InstantReadoutState, + ::InstantReadout, + ::Reservoir, + neural_output, + frame::Integer, +) + _require_neural_width(state.neural_last, neural_output) + @inbounds for index in eachindex(state.neural_last, neural_output) + value = Float64(neural_output[index]) + state.neural_sum[index] += value + state.neural_last[index] = value + end + return state +end + +function observe_frame!( + state::VotingReadoutState, + readout_component::VotingReadout, + reservoir::Reservoir, + neural_output, + frame::Integer, +) + _require_neural_width(state.neural_sum, neural_output) + @inbounds for index in eachindex(state.neural_sum, neural_output) + state.neural_sum[index] += Float64(neural_output[index]) + end + projected = readout(readout_component.policy, reservoir, neural_output) + length(projected) == length(state.votes) || throw(DimensionMismatch( + "voting readout expected $(length(state.votes)) effectors, got $(length(projected))", + )) + _, winner = findmax(projected) + state.votes[winner] += 1 + return state +end + +function finish_readout!( + state::MeanReadoutState, + readout_component::MeanReadout, + reservoir::Reservoir, + cycle::FixedRateCycle, +) + scale = inv(Float64(neural_frames(cycle))) + @inbounds for index in eachindex(state.neural_mean, state.neural_sum) + state.neural_mean[index] = state.neural_sum[index] * scale + end + return readout(readout_component.policy, reservoir, state.neural_mean) +end + +function finish_readout!( + state::InstantReadoutState, + readout_component::InstantReadout, + reservoir::Reservoir, + cycle::FixedRateCycle, +) + scale = inv(Float64(neural_frames(cycle))) + @inbounds for index in eachindex(state.neural_mean, state.neural_sum) + state.neural_mean[index] = state.neural_sum[index] * scale + end + return readout(readout_component.policy, reservoir, state.neural_last) +end + +function finish_readout!( + state::VotingReadoutState, + ::VotingReadout, + ::Reservoir, + cycle::FixedRateCycle, +) + scale = inv(Float64(neural_frames(cycle))) + @inbounds for index in eachindex(state.neural_mean, state.neural_sum) + state.neural_mean[index] = state.neural_sum[index] * scale + end + fill!(state.signal, 0.0) + _, winner = findmax(state.votes) + state.signal[winner] = 1.0 + return state.signal +end + +recorded_neural_output(state::MeanReadoutState, ::FixedRateCycle) = state.neural_mean +recorded_neural_output(state::InstantReadoutState, ::FixedRateCycle) = state.neural_mean +recorded_neural_output(state::VotingReadoutState, ::FixedRateCycle) = state.neural_mean + +"""Per-agent reusable buffers for one interaction cycle.""" +mutable struct InteractionState{R} + readout::R + receptor_sum::Vector{Float64} + receptor_mean::Vector{Float64} +end + +function InteractionState(readout_component::AbstractReadout, reservoir::Reservoir, body::AbstractBody) + return InteractionState( + readout_state(readout_component, reservoir), + zeros(n_receptors(body)), + zeros(n_receptors(body)), + ) +end + +function begin_interaction!(state::InteractionState, readout_component, cycle::FixedRateCycle) + fill!(state.receptor_sum, 0.0) + fill!(state.receptor_mean, 0.0) + begin_readout!(state.readout, readout_component, cycle) + return state +end + +function observe_receptors!(state::InteractionState, receptors) + length(receptors) == length(state.receptor_sum) || throw(DimensionMismatch( + "interaction expected $(length(state.receptor_sum)) receptors, got $(length(receptors))", + )) + @inbounds for index in eachindex(state.receptor_sum, receptors) + state.receptor_sum[index] += Float64(receptors[index]) + end + return state +end + +function finish_receptors!(state::InteractionState, cycle::FixedRateCycle) + scale = inv(Float64(neural_frames(cycle))) + @inbounds for index in eachindex(state.receptor_mean, state.receptor_sum) + state.receptor_mean[index] = state.receptor_sum[index] * scale + end + return state.receptor_mean +end + +# Memoryless encoders and legacy bodies encode once and hold that result across +# the cycle. Temporal encoders specialize these methods. +begin_encoding!(encoder::AbstractEncoder, samples, ::FixedRateCycle) = encode!(encoder, samples) +encode_frame!(::AbstractEncoder, state, ::Integer, ::FixedRateCycle) = state + +begin_encoding!(body::AbstractBody, percept, ::FixedRateCycle) = sense!(body, percept) +encode_frame!(::AbstractBody, state, ::Integer, ::FixedRateCycle) = state + +# Compatibility for custom bodies that still expose only a Motor policy. +readout_components(body::AbstractBody) = (MeanReadout(readout_policy(body)),) +primary_readout(body::AbstractBody) = only(readout_components(body)) diff --git a/test/runtests.jl b/test/runtests.jl index e2d573c..8e5049f 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -17,6 +17,7 @@ include("test_paper_constants.jl") include("test_authors_parity.jl") include("test_falandays.jl") include("test_window.jl") +include("test_interaction_cycle.jl") include("test_noisy_input.jl") include("test_dendritic.jl") include("test_sorn.jl") @@ -31,6 +32,7 @@ include("test_tracking_env_params.jl") include("test_core_task_controls.jl") include("test_core_calibration.jl") include("test_cartpole_variants.jl") +include("test_plank_cartpole.jl") include("test_morphology.jl") include("test_homeostasis.jl") include("test_sensor.jl") diff --git a/test/test_component_catalog.jl b/test/test_component_catalog.jl index 671e485..c8db2ee 100644 --- a/test/test_component_catalog.jl +++ b/test/test_component_catalog.jl @@ -8,6 +8,9 @@ (:sensor, :field_probe), (:encoder, :identity), (:encoder, :bilateral_contrast), + (:readout, :mean), + (:readout, :instant), + (:readout, :voting), (:actuator, :forward_turn), (:actuator, :antagonistic_turn), (:actuator, :differential_drive), @@ -24,6 +27,7 @@ (:physiology, :none), (:sensor, :spectral_camera), (:encoder, :identity), + (:readout, :mean), (:actuator, :differential_drive), (:dynamics, :differential_drive), )) diff --git a/test/test_interaction_cycle.jl b/test/test_interaction_cycle.jl new file mode 100644 index 0000000..5adc1a0 --- /dev/null +++ b/test/test_interaction_cycle.jl @@ -0,0 +1,79 @@ +using BrainlessLab +using Test + +import BrainlessLab: effectors, n_effectors, n_nodes, n_receptors, reset!, step! + +mutable struct _CycleCounterReservoir <: Reservoir + steps::Int + nr::Int + ne::Int +end + +n_receptors(reservoir::_CycleCounterReservoir) = reservoir.nr +n_effectors(reservoir::_CycleCounterReservoir) = reservoir.ne +n_nodes(::_CycleCounterReservoir) = 2 +function step!(reservoir::_CycleCounterReservoir, receptors) + reservoir.steps += 1 + return Float64[reservoir.steps, sum(receptors)] +end +effectors(reservoir::_CycleCounterReservoir, neural_output) = + Float64[neural_output[index] for index in 1:reservoir.ne] +reset!(reservoir::_CycleCounterReservoir) = (reservoir.steps = 0; reservoir) + +@testset "FixedRateCycle owns world-to-neural timing" begin + @test neural_frames(FixedRateCycle()) == 1 + @test neural_frames(FixedRateCycle(24)) == 24 + @test_throws ArgumentError FixedRateCycle(0) + + body = direct_embodiment(2, 2; readouts=(MeanReadout(),)) + reservoir = _CycleCounterReservoir(0, 2, 2) + agent = Agent(reservoir, body; cycle=FixedRateCycle(4)) + # The interaction helper is tested directly because WallEnv's native + # observation width is unrelated to this minimal contract reservoir. + receptors, neural, command = BrainlessLab._run_interaction!(agent, [0.25, 0.75]) + @test reservoir.steps == 4 + @test receptors == [0.25, 0.75] + @test neural == [2.5, 1.0] + @test command_values(command) == [1.0, 1.0] +end + +@testset "default cycle preserves held-input window semantics" begin + receptors = [0.3, 0.7, 0.1, 0.5] + windowed = BrainlessLab._falandays_native(20, 4, 2; seed=5, substeps=3) + manual = BrainlessLab._falandays_native(20, 4, 2; seed=5, substeps=1) + agent = Agent(windowed, direct_embodiment(4, 2)) + held, neural_mean, _ = BrainlessLab._run_interaction!(agent, receptors) + manual_outputs = [step!(manual, receptors) for _ in 1:3] + @test held ≈ receptors + @test neural_mean ≈ sum(manual_outputs) ./ 3 +end + +@testset "readout reductions are explicit and deterministic" begin + reservoir = _CycleCounterReservoir(0, 1, 2) + cycle = FixedRateCycle(3) + + mean_readout = MeanReadout() + mean_state = BrainlessLab.readout_state(mean_readout, reservoir) + begin_readout!(mean_state, mean_readout, cycle) + for (frame, output) in enumerate(([1.0, 0.0], [0.0, 1.0], [1.0, 1.0])) + observe_frame!(mean_state, mean_readout, reservoir, output, frame) + end + @test finish_readout!(mean_state, mean_readout, reservoir, cycle) ≈ [2 / 3, 2 / 3] + + voting = VotingReadout() + voting_state = BrainlessLab.readout_state(voting, reservoir) + begin_readout!(voting_state, voting, cycle) + for (frame, output) in enumerate(([0.5, 0.5], [0.0, 1.0], [1.0, 0.0])) + observe_frame!(voting_state, voting, reservoir, output, frame) + end + # Both the frame-one tie and the final vote tie choose the lower index. + @test finish_readout!(voting_state, voting, reservoir, cycle) == [1.0, 0.0] +end + +@testset "Embodiment owns its readout component" begin + body = direct_embodiment(2, 2; readouts=(InstantReadout(),)) + @test only(readout_components(body)) isa InstantReadout + @test primary_readout(body) === only(body.readouts) + @test component_id(only(component_slots(body).readouts)) === :readout_1 + @test readout_policy(body) === BrainlessLab.PASSTHROUGH_MOTOR +end diff --git a/test/test_plank_cartpole.jl b/test/test_plank_cartpole.jl new file mode 100644 index 0000000..9212f8c --- /dev/null +++ b/test/test_plank_cartpole.jl @@ -0,0 +1,139 @@ +using BrainlessLab +using Random +using Test + +@testset "Plank Spike-FF-2 source example" begin + encoder = SpikeFF2Encoder((2.4, 2.0, 0.209, 2.0)) + cycle = FixedRateCycle(24) + state = begin_encoding!(encoder, [0.852, -0.007, 0.018, -0.659], cycle) + totals = zeros(Int, 8) + for frame in 1:24 + totals .+= Int.(encode_frame!(encoder, state, frame, cycle)) + end + # Authors' documented example: +x=3, -dx=1, +theta=1, -dtheta=3. + @test totals == [0, 3, 1, 0, 0, 1, 3, 0] + @test sum(totals) == 8 +end + +@testset "Argyle-4 conserves nine spikes per observation" begin + encoder = Argyle4Encoder((2.4, 0.2095)) + cycle = FixedRateCycle(24) + state = begin_encoding!(encoder, [0.0, 0.10475], cycle) + totals = zeros(Int, 8) + for frame in 1:24 + totals .+= Int.(encode_frame!(encoder, state, frame, cycle)) + end + @test sum(@view totals[1:4]) == 9 + @test sum(@view totals[5:8]) == 9 + @test count(>(0), @view totals[1:4]) <= 2 + @test count(>(0), @view totals[5:8]) <= 2 +end + +@testset "four Plank CartPole levels are challenge-tagged profiles" begin + expected = Set(( + :cartpole_plank_easy, + :cartpole_plank_medium, + :cartpole_plank_hard, + :cartpole_plank_hardest, + )) + @test Set(tasks(tag=:challenge)) == expected + @test Set(tasks(tag=:qualification)) == Set((:tracking, :pong)) + @test Set(tasks(tag=:benchmark)) == Set((:tracking, :pong)) + @test :pong_hitrate in tasks(tag=:alias) + @test :wall in tasks(tag=:extended) + @test :wall ∉ tasks(tag=:qualification) + + for task in expected + info = task_info(task) + @test info.status === :experimental + @test info.interaction_cycle == FixedRateCycle(24) + @test info.protocol.evaluation_episodes == 1000 + @test info.protocol.cross_task_aggregate === false + + setup = setup_task(resolve_task(task); seed=11) + @test setup.environment isa PlankCartPoleEnv + body = only(setup.bodies) + @test primary_readout(body) isa VotingReadout + @test n_receptors(body) == resolve_task(task).n_receptors + @test n_effectors(body) == resolve_task(task).n_effectors + end +end + +@testset "Plank CartPole action and fitness contracts" begin + dynamics = PlankCartPoleEnv(rng=MersenneTwister(7), level=:easy) + set_plank_cartpole_state!(dynamics, (0.1, 0.2, 0.05, -0.1)) + previous = copy(dynamics.state) + step!(dynamics, [1.0, 0.0]) + # Gym's default explicit Euler updates positions from the old velocities. + @test dynamics.state[1] ≈ previous[1] + dynamics.tau * previous[2] + @test dynamics.state[3] ≈ previous[3] + dynamics.tau * previous[4] + + medium = PlankCartPoleEnv(rng=MersenneTwister(1), level=:medium) + initial = copy(medium.state) + step!(medium, [1.0, 0.0, 0.0]) + @test medium.noop_count == 1 + @test medium.step_count == 1 + @test medium.state != initial + @test plank_cartpole_fitness(medium) == 1.0 + + medium.step_count = 100 + medium.noop_count = 70 + @test plank_cartpole_fitness(medium) ≈ 70 / 0.75 + medium.noop_count = 80 + @test plank_cartpole_fitness(medium) == 100.0 + + easy = PlankCartPoleEnv(rng=MersenneTwister(2), level=:easy) + @test BrainlessLab._plank_cartpole_action(easy, [0.5, 0.5]) === :left +end + +@testset "Plank profiles execute through the standard simulation path" begin + for task in tasks(tag=:challenge) + result = simulate( + task; + node=:falandays, + n_nodes=20, + ticks=3, + seed=3, + record=Symbol[], + ) + @test result isa SimResult + @test isfinite(result.metrics.fitness) + @test result.config.agents[1].body.traits.interface_frozen + end +end + +@testset "Plank evaluation freezes design and records the test set" begin + protocol = EvaluationProtocol( + trials=3, + horizon=5, + reset=:full, + design_scope=:fixed, + aggregation=:mean, + ) + states = plank_cartpole_initial_conditions(71, 3) + @test states == plank_cartpole_initial_conditions(71, 3) + @test states != plank_cartpole_initial_conditions(72, 3) + + result = evaluate_plank_cartpole( + :cartpole_plank_easy; + node=:null_random, + protocol=protocol, + build_seed=4, + trial_seed=71, + n_nodes=8, + ) + @test result isa EvaluationResult + @test result.initial_conditions == states + @test length(result.trials) == 3 + @test result.summary.n == 3 + @test result.summary.target_fitness == 14_250.0 + @test all(trial -> trial.steps <= 5, result.trials) + @test result.protocol.design_scope === :fixed + + @test_throws ArgumentError evaluate_plank_cartpole( + :tracking; + node=:null_random, + protocol=protocol, + n_nodes=8, + ) +end From d456fab969da40be528c1143c049c2048bb71b23 Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 22:24:27 -0400 Subject: [PATCH 02/20] Narrow Plank CartPole to experimental tasks --- README.md | 4 ++-- site/src/content/docs/core/task-tour.mdx | 8 ++++---- site/src/content/docs/core/tools-artifacts.mdx | 2 +- src/envs/PlankCartPole.jl | 8 ++++---- src/run/Evaluation.jl | 2 +- src/tasks/Tasks.jl | 2 +- test/test_plank_cartpole.jl | 7 ++++--- 7 files changed, 17 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index fa3154e..d139eb9 100644 --- a/README.md +++ b/README.md @@ -68,8 +68,8 @@ synchronous lifecycle. `FixedRateCycle` explicitly separates a world step from native neural frames. This supports held inputs, temporal spike encoders, mean or instant reduction, and categorical voting without putting task-specific timing branches into the simulation loop. Four experimental -Plank CartPole challenge profiles use this seam; Tracking and Pong remain the initial core -benchmark tasks. +Plank CartPole task profiles use this seam as an experimental proving ground; +Tracking and Pong remain the initial core benchmark tasks. `ObjectWorld` demonstrates composition of physical components, objects, fields, spectral appearance, and typed effects. It is not a calibrated benchmark. The established tracking diff --git a/site/src/content/docs/core/task-tour.mdx b/site/src/content/docs/core/task-tour.mdx index 9dc291b..f6f3620 100644 --- a/site/src/content/docs/core/task-tour.mdx +++ b/site/src/content/docs/core/task-tour.mdx @@ -86,7 +86,7 @@ number of independent runs before treating the mean as reliable. These tasks use direct vector adapters. Their worlds already produce the reservoir receptor vector and consume a task-specific effector vector. -## Experimental CartPole challenge ladder +## Experimental Plank CartPole levels Four additional task profiles reproduce the interface ladder proposed by Plank and colleagues for neuromorphic systems: @@ -99,9 +99,9 @@ colleagues for neuromorphic systems: | `:cartpole_plank_hardest` | cart position and pole angle | left, right | Argyle-4 | 6,000 | All four run a 15,000-step mission with 24 native neural frames per world step and a -frame-voting readout. They are tagged `:challenge` and `:experimental`; they are deliberately -outside the Tracking/Pong core aggregate. The useful first result may simply be a clear -failure boundary for an otherwise capable design. +frame-voting readout. They are tagged `:experimental` and `:plank_cartpole`; they are deliberately +outside the Tracking/Pong core benchmark. The useful first result may simply be a clear +performance boundary for an otherwise capable design. For a quick integration check, use `simulate`. For the declared repeated-start contract, use `evaluate_plank_cartpole`, which retains the complete initial-condition set and every raw diff --git a/site/src/content/docs/core/tools-artifacts.mdx b/site/src/content/docs/core/tools-artifacts.mdx index c93a6ed..864ef7b 100644 --- a/site/src/content/docs/core/tools-artifacts.mdx +++ b/site/src/content/docs/core/tools-artifacts.mdx @@ -72,7 +72,7 @@ evidence status before opening any sealed output. | `sweep/run.jl ablate` | apply registered mechanism interventions | | `bench/` | compare a declared model roster on a declared task grid | | `evolve` and training tools | select fixed genomes on development tasks | -| `evaluate_plank_cartpole` | run the experimental repeated-start CartPole challenge contract | +| `evaluate_plank_cartpole` | run the experimental repeated-start Plank CartPole protocol | Examples: diff --git a/src/envs/PlankCartPole.jl b/src/envs/PlankCartPole.jl index cb55880..ff75766 100644 --- a/src/envs/PlankCartPole.jl +++ b/src/envs/PlankCartPole.jl @@ -2,7 +2,7 @@ const PLANK_CARTPOLE_MISSION_STEPS = 15_000 const PLANK_CARTPOLE_NEURAL_FRAMES = 24 const PLANK_CARTPOLE_EVAL_EPISODES = 1_000 -"""Frozen task-interface definition for one Plank CartPole challenge level.""" +"""Frozen task-interface definition for one experimental Plank CartPole level.""" struct PlankCartPoleLevel name::Symbol observation_indices::Tuple{Vararg{Int}} @@ -332,7 +332,7 @@ function step!(environment::PlankCartPoleEnv, effectors) total_mass=environment.total_mass, # The source protocol executes Gym/Gymnasium CartPole's default # explicit-Euler path. Keep this distinct from the legacy BrainlessLab - # variants, which predate these challenge profiles. + # variants, which predate these task profiles. integrator=:euler, ) environment.step_count += 1 @@ -415,7 +415,7 @@ function (setup::PlankCartPoleSetup)(; kwargs..., ) body === nothing || throw(ArgumentError( - "Plank CartPole benchmark profiles freeze their embodiment; register a separate " * + "Plank CartPole task profiles freeze their embodiment; register a separate " * "experimental task to change sensors, encoding, readout, or actuators", )) rng_ = rng === nothing ? MersenneTwister(Int(seed)) : rng @@ -432,7 +432,7 @@ function (setup::PlankCartPoleSetup)(; readouts=(VotingReadout(),), actuators=(DirectRelayActuator(setup.level.actions),), traits=( - benchmark=:plank_cartpole, + family=:plank_cartpole, level=setup.level.name, interface_frozen=true, ), diff --git a/src/run/Evaluation.jl b/src/run/Evaluation.jl index a280dd2..bc42841 100644 --- a/src/run/Evaluation.jl +++ b/src/run/Evaluation.jl @@ -156,7 +156,7 @@ end Evaluate one fixed node construction across an explicit held set of CartPole initial conditions. Each trial fully resets reservoir dynamics, plastic state, and embodiment state while retaining the same constructed topology. Raw trials -and initial conditions are returned; the four challenge levels are never +and initial conditions are returned; the four Plank levels are never collapsed into a cross-task aggregate. """ function evaluate_plank_cartpole( diff --git a/src/tasks/Tasks.jl b/src/tasks/Tasks.jl index 9436150..4c82f6c 100644 --- a/src/tasks/Tasks.jl +++ b/src/tasks/Tasks.jl @@ -365,7 +365,7 @@ function _plank_cartpole_task(level_name::Symbol) default_window=PLANK_CARTPOLE_MISSION_STEPS, interaction_cycle=FixedRateCycle(PLANK_CARTPOLE_NEURAL_FRAMES), status=:experimental, - tags=(:challenge, :experimental, :extended, :plank_cartpole), + tags=(:experimental, :plank_cartpole), protocol=(; PLANK_CARTPOLE_PROTOCOL..., level=level.name, diff --git a/test/test_plank_cartpole.jl b/test/test_plank_cartpole.jl index 9212f8c..9b64d66 100644 --- a/test/test_plank_cartpole.jl +++ b/test/test_plank_cartpole.jl @@ -29,14 +29,14 @@ end @test count(>(0), @view totals[5:8]) <= 2 end -@testset "four Plank CartPole levels are challenge-tagged profiles" begin +@testset "four Plank CartPole levels are experimental task profiles" begin expected = Set(( :cartpole_plank_easy, :cartpole_plank_medium, :cartpole_plank_hard, :cartpole_plank_hardest, )) - @test Set(tasks(tag=:challenge)) == expected + @test Set(tasks(tag=:plank_cartpole)) == expected @test Set(tasks(tag=:qualification)) == Set((:tracking, :pong)) @test Set(tasks(tag=:benchmark)) == Set((:tracking, :pong)) @test :pong_hitrate in tasks(tag=:alias) @@ -46,6 +46,7 @@ end for task in expected info = task_info(task) @test info.status === :experimental + @test info.tags == (:experimental, :plank_cartpole) @test info.interaction_cycle == FixedRateCycle(24) @test info.protocol.evaluation_episodes == 1000 @test info.protocol.cross_task_aggregate === false @@ -87,7 +88,7 @@ end end @testset "Plank profiles execute through the standard simulation path" begin - for task in tasks(tag=:challenge) + for task in tasks(tag=:plank_cartpole) result = simulate( task; node=:falandays, From 5cb2dbef0ee42cb555a7e72ee2df38de405ebeb1 Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 22:49:41 -0400 Subject: [PATCH 03/20] Prepare experimental v0.1.1 baseline --- .github/workflows/ci.yml | 91 + .gitignore | 1 - CHANGELOG.md | 16 + CITATION.cff | 10 +- Manifest.toml | 4 +- Project.toml | 12 +- README.md | 12 +- bench/Manifest.toml | 1662 +++++++++++++++++ bench/Project.toml | 12 + bench/configs/smoke.toml | 12 +- configs/ci_sweep.toml | 20 + examples/templates/new_project/Project.toml | 3 + profile/Manifest.toml | 6 +- profile/Profile.jl | 7 +- profile/Project.toml | 13 + .../src/content/docs/core/getting-started.mdx | 6 +- test/runtests.jl | 5 + 17 files changed, 1866 insertions(+), 26 deletions(-) create mode 100644 .github/workflows/ci.yml create mode 100644 CHANGELOG.md create mode 100644 bench/Manifest.toml create mode 100644 configs/ci_sweep.toml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..b63fd21 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,91 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ci-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + package: + name: Julia ${{ matrix.julia-version }} / ${{ matrix.os }} + runs-on: ${{ matrix.os }} + timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + os: [ubuntu-22.04, macos-14, windows-2022] + julia-version: ['1.10.11', '1'] + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + - uses: julia-actions/setup-julia@fa02766e078afaaf09b14210362cee14137e6a32 # v3 + with: + version: ${{ matrix.julia-version }} + - uses: julia-actions/cache@a45e8fa8be21c18a06b7177052533149e61e9b38 # v3 + - name: Instantiate the pinned environment + run: julia --project=. -e 'using Pkg; Pkg.instantiate(); Pkg.precompile()' + - uses: julia-actions/julia-runtest@6e050c8013b833b1195105ff2fce9cd802f53271 # v1 + + compat-floor: + name: Manifest-free resolution / Julia 1.10.11 + runs-on: ubuntu-22.04 + timeout-minutes: 45 + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + - uses: julia-actions/setup-julia@fa02766e078afaaf09b14210362cee14137e6a32 # v3 + with: + version: '1.10.11' + - uses: julia-actions/cache@a45e8fa8be21c18a06b7177052533149e61e9b38 # v3 + - name: Resolve and test without the checked-in manifest + run: julia --project=. -e 'using Pkg; rm("Manifest.toml"; force=true); Pkg.resolve(); Pkg.instantiate(); Pkg.test()' + + tool-smoke: + name: Tool and template smoke tests + runs-on: ubuntu-22.04 + timeout-minutes: 45 + env: + BRAINLESSLAB_AUTOTHREADS: '0' + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + - uses: julia-actions/setup-julia@fa02766e078afaaf09b14210362cee14137e6a32 # v3 + with: + version: '1.10.11' + - uses: julia-actions/cache@a45e8fa8be21c18a06b7177052533149e61e9b38 # v3 + - name: Instantiate benchmark environment + run: julia --project=bench -e 'using Pkg; Pkg.instantiate()' + - name: Benchmark statistics and execution smoke + run: | + julia --project=bench bench/test_stats.jl + julia --project=bench -e 'include("bench/Benchmark.jl"); using .Benchmark; cfg = Benchmark.read_bench_config("bench/configs/smoke.toml"); out = Benchmark.run_benchmark(cfg; out_root=mktempdir()); @assert isfile(joinpath(out.dir, "summary.csv"))' + - name: Instantiate profile environment + run: julia --project=profile -e 'using Pkg; Pkg.instantiate()' + - name: Profile execution smoke + run: julia --project=profile -e 'include("profile/Profile.jl"); using .NodeProfile; out = NodeProfile.node_profile(:falandays; tasks=(:tracking,), n_seeds=1, canonical_N=Dict(:tracking => 12), gifs=false, out_root=mktempdir()); @assert isfile(out.metrics)' + - name: Sweep execution smoke + run: julia --project=. -e 'using BrainlessLab; out = run_sweep("configs/ci_sweep.toml"; root=mktempdir()); @assert isfile(out.results)' + - name: Instantiate and execute project template + run: | + julia --project=examples/templates/new_project -e 'using Pkg; Pkg.develop(path=pwd()); Pkg.instantiate()' + julia --project=examples/templates/new_project examples/templates/new_project/run.jl --ticks 20 --n-nodes 12 --out "${{ runner.temp }}/brainlesslab-template-smoke" + + documentation: + name: Locked documentation build + runs-on: ubuntu-22.04 + timeout-minutes: 20 + defaults: + run: + working-directory: site + steps: + - uses: actions/checkout@d23441a48e516b6c34aea4fa41551a30e30af803 # v6 + - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2 + with: + bun-version: 1.3.11 + - run: bun install --frozen-lockfile + - run: bun run build diff --git a/.gitignore b/.gitignore index 8794d02..35fbee3 100644 --- a/.gitignore +++ b/.gitignore @@ -14,7 +14,6 @@ examples/output/ /experiments/runs/ /experiments/results/homeostatic_needs_v2/ /bench/genomes/ -/bench/Manifest.toml /bench/output/ /examples/templates/new_project/output/ /examples/templates/new_project/Manifest.toml diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..b38369a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,16 @@ +# Changelog + +## 0.1.1 — 2026-07-22 + +BrainlessLab 0.1.1 is an experimental research preview. + +- Align package and citation metadata on version 0.1.1. +- Add reproducible package, compatibility-floor, tool-smoke, and documentation CI. +- Add package-quality checks without weakening numerical conformance or calibration tests. +- Clarify repository installation and the pre-1.0 stability boundary. + +## 0.1.0 — 2026-07-04 + +The public tag named `v0.1.0` was created from code whose `Project.toml` still reported +version `0.0.1`. That historical tag remains immutable; 0.1.1 is the first release in +which the package and citation versions are aligned. diff --git a/CITATION.cff b/CITATION.cff index a9d0628..bbb5647 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -1,11 +1,11 @@ cff-version: 1.2.0 message: >- If you use BrainlessLab, please cite this software, and also cite the - original Falandays et al. work that the :falandays_base model reimplements. + original Falandays et al. work that the :falandays model reimplements. title: BrainlessLab.jl abstract: >- - An extensible Julia lab for "brainless" cognition — behaviour that emerges - from collectives of simple neuron-like nodes — built around an authors-faithful + An experimental Julia research platform for behaviour that emerges from + collectives of simple neuron-like nodes, built around an authors-faithful reimplementation of the Falandays et al. homeostatic spiking reservoir. type: software authors: @@ -17,8 +17,8 @@ authors: family-names: Jackson - given-names: William family-names: O'Hearn -version: 0.0.1 -date-released: "2026-07-04" +version: 0.1.1 +date-released: "2026-07-22" license: MIT repository-code: "https://github.com/btgaskin/brainless-lab" url: "https://brainless-lab.pages.dev" diff --git a/Manifest.toml b/Manifest.toml index d275535..cb4be76 100644 --- a/Manifest.toml +++ b/Manifest.toml @@ -2,7 +2,7 @@ julia_version = "1.12.6" manifest_format = "2.0" -project_hash = "e5f54b75d015108e62f7964a038c6db733ff2e1f" +project_hash = "1936d7250a908cf710d7a57233db3835c77344fd" [[deps.ArgTools]] uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" @@ -20,7 +20,7 @@ version = "1.11.0" deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "StaticArrays", "Statistics", "TOML"] path = "." uuid = "d12add44-1e3e-4161-9a99-c2121a2f0f38" -version = "0.0.1" +version = "0.1.1" [deps.BrainlessLab.extensions] BrainlessLabMakieExt = "Makie" diff --git a/Project.toml b/Project.toml index 0433b4b..9f460e0 100644 --- a/Project.toml +++ b/Project.toml @@ -1,7 +1,7 @@ name = "BrainlessLab" uuid = "d12add44-1e3e-4161-9a99-c2121a2f0f38" authors = ["btgaskin "] -version = "0.0.1" +version = "0.1.1" [deps] Dates = "ade2ca70-3891-5945-98fb-dc099432e06a" @@ -19,18 +19,26 @@ Makie = "ee78f7c6-11fb-53f2-987a-cfe4a2b5a57a" BrainlessLabMakieExt = "Makie" [compat] +Aqua = "0.8" CairoMakie = "0.12, 0.13, 0.14, 0.15" +Dates = "1.10" JLD2 = "0.5" +LinearAlgebra = "1.10" Makie = "0.21, 0.22, 0.23, 0.24" NPZ = "0.4" +Random = "1.10" StaticArrays = "1" +Statistics = "1.10" +TOML = "1" +Test = "1.10" julia = "1.10" [extras] +Aqua = "4c88cf16-eb10-579e-8560-4a9242c79595" CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0" Makie = "ee78f7c6-11fb-53f2-987a-cfe4a2b5a57a" NPZ = "15e1cf62-19b3-5cfa-8e77-841668bca605" Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" [targets] -test = ["CairoMakie", "Makie", "NPZ", "Test"] +test = ["Aqua", "CairoMakie", "Makie", "NPZ", "Test"] diff --git a/README.md b/README.md index d139eb9..215d4d1 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # BrainlessLab.jl +[![CI](https://github.com/btgaskin/brainless-lab/actions/workflows/ci.yml/badge.svg)](https://github.com/btgaskin/brainless-lab/actions/workflows/ci.yml) +

BrainlessLab

@@ -11,9 +13,10 @@ Diverse Intelligences Summer Institute 2026

-BrainlessLab is an extensible Julia lab for neural reservoirs in closed sensorimotor -loops. It provides tasks, generic embodiment, single-agent and population worlds, -recording, analysis, batch tools, and evidence-aware experiment workflows. +BrainlessLab v0.1.1 is an **experimental research preview** for neural reservoirs in +closed sensorimotor loops. It provides tasks, generic embodiment, single-agent and +population worlds, recording, analysis, batch tools, and evidence-aware experiment +workflows. APIs and artifact layouts may change before 1.0. The canonical baseline is `node=:falandays`: an authors-faithful implementation of the tested Falandays homeostatic spiking reservoir. It adapts neural activity online and has no @@ -22,7 +25,8 @@ studies are experimental unless their documentation states a narrower validated ## Quickstart -Install Julia, clone the repository, and use the pinned project: +BrainlessLab is not yet registered in Julia General. Install Julia 1.10 or newer, +clone the repository, and use its pinned project: ```bash git clone https://github.com/btgaskin/brainless-lab.git diff --git a/bench/Manifest.toml b/bench/Manifest.toml new file mode 100644 index 0000000..b509ca1 --- /dev/null +++ b/bench/Manifest.toml @@ -0,0 +1,1662 @@ +# This file is machine-generated - editing it directly is not advised + +julia_version = "1.12.6" +manifest_format = "2.0" +project_hash = "63394ceac60298dd43be0560b12adaef165476e3" + +[[deps.AbstractFFTs]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "d92ad398961a3ed262d8bf04a1a2b8340f915fef" +uuid = "621f4979-c628-5d54-868e-fcf4e3e8185c" +version = "1.5.0" +weakdeps = ["ChainRulesCore", "Test"] + + [deps.AbstractFFTs.extensions] + AbstractFFTsChainRulesCoreExt = "ChainRulesCore" + AbstractFFTsTestExt = "Test" + +[[deps.AbstractTrees]] +git-tree-sha1 = "2d9c9a55f9c93e8887ad391fbae72f8ef55e1177" +uuid = "1520ce14-60c1-5f80-bbc7-55ef81b5835c" +version = "0.4.5" + +[[deps.Accessors]] +deps = ["CompositionsBase", "ConstructionBase", "Dates", "InverseFunctions", "MacroTools"] +git-tree-sha1 = "7063ad1083578215c7c4bf410368150abe8d5524" +uuid = "7d9f7c33-5ae7-4f3b-8dc6-eff91059b697" +version = "0.1.45" + + [deps.Accessors.extensions] + AxisKeysExt = "AxisKeys" + IntervalSetsExt = "IntervalSets" + LinearAlgebraExt = "LinearAlgebra" + StaticArraysExt = "StaticArrays" + StructArraysExt = "StructArrays" + TestExt = "Test" + UnitfulExt = "Unitful" + + [deps.Accessors.weakdeps] + AxisKeys = "94b1ba4f-4ee9-5380-92f1-94cde586c3c5" + IntervalSets = "8197267c-284f-5f27-9208-e0e47529a953" + LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + StaticArrays = "90137ffa-7385-5640-81b9-e52037218182" + StructArrays = "09ab397b-f2b6-538f-b94a-2f83cf4a842a" + Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d" + +[[deps.Adapt]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "daa72978cd7a624246e894a4f4f067706d4e17e2" +uuid = "79e6a3ab-5dfb-504d-930d-738a2a938a0e" +version = "4.7.0" +weakdeps = ["SparseArrays", "StaticArrays"] + + [deps.Adapt.extensions] + AdaptSparseArraysExt = "SparseArrays" + AdaptStaticArraysExt = "StaticArrays" + +[[deps.AdaptivePredicates]] +git-tree-sha1 = "7e651ea8d262d2d74ce75fdf47c4d63c07dba7a6" +uuid = "35492f91-a3bd-45ad-95db-fcad7dcfedb7" +version = "1.2.0" + +[[deps.AliasTables]] +deps = ["PtrArrays", "Random"] +git-tree-sha1 = "9876e1e164b144ca45e9e3198d0b689cadfed9ff" +uuid = "66dad0bd-aa9a-41b7-9441-69ab47430ed8" +version = "1.1.3" + +[[deps.Animations]] +deps = ["Colors"] +git-tree-sha1 = "e092fa223bf66a3c41f9c022bd074d916dc303e7" +uuid = "27a7e980-b3e6-11e9-2bcd-0b925532e340" +version = "0.4.2" + +[[deps.ArgTools]] +uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" +version = "1.1.2" + +[[deps.Artifacts]] +uuid = "56f22d72-fd6d-98f1-02f0-08ddc0907c33" +version = "1.11.0" + +[[deps.Automa]] +deps = ["PrecompileTools", "TranscodingStreams"] +git-tree-sha1 = "94eab0b3ccdcac361188cc661daf69d4433c1818" +uuid = "67c07d97-cdcb-5c2c-af73-a7f9c32a568b" +version = "1.2.0" + +[[deps.AxisAlgorithms]] +deps = ["LinearAlgebra", "Random", "SparseArrays", "WoodburyMatrices"] +git-tree-sha1 = "01b8ccb13d68535d73d2b0c23e39bd23155fb712" +uuid = "13072b0f-2c55-5437-9ae7-d433b7a33950" +version = "1.1.0" + +[[deps.AxisArrays]] +deps = ["Dates", "IntervalSets", "IterTools", "RangeArrays"] +git-tree-sha1 = "4126b08903b777c88edf1754288144a0492c05ad" +uuid = "39de3d68-74b9-583c-8d2d-e117c070f3a9" +version = "0.4.8" + +[[deps.Base64]] +uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" +version = "1.11.0" + +[[deps.BaseDirs]] +git-tree-sha1 = "8c290a1b223deaeea9aea44b235d24546da8eb98" +uuid = "18cc8868-cbac-4acf-b575-c8ff214dc66f" +version = "1.4.0" + +[[deps.BrainlessLab]] +deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "StaticArrays", "Statistics", "TOML"] +path = ".." +uuid = "d12add44-1e3e-4161-9a99-c2121a2f0f38" +version = "0.1.1" +weakdeps = ["Makie"] + + [deps.BrainlessLab.extensions] + BrainlessLabMakieExt = "Makie" + +[[deps.Bzip2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "1b96ea4a01afe0ea4090c5c8039690672dd13f2e" +uuid = "6e34b625-4abd-537c-b88f-471c36dfa7a0" +version = "1.0.9+0" + +[[deps.CEnum]] +git-tree-sha1 = "389ad5c84de1ae7cf0e28e381131c98ea87d54fc" +uuid = "fa961155-64e5-5f13-b03f-caf6b980ea82" +version = "0.5.0" + +[[deps.CRC32c]] +uuid = "8bf52ea8-c179-5cab-976a-9e18b702a9bc" +version = "1.11.0" + +[[deps.CRlibm]] +deps = ["CRlibm_jll"] +git-tree-sha1 = "66188d9d103b92b6cd705214242e27f5737a1e5e" +uuid = "96374032-68de-5a5b-8d9e-752f78720389" +version = "1.0.2" + +[[deps.CRlibm_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "e329286945d0cfc04456972ea732551869af1cfc" +uuid = "4e9b3aee-d8a1-5a3d-ad8b-7d824db253f0" +version = "1.0.1+0" + +[[deps.Cairo]] +deps = ["Cairo_jll", "Colors", "Glib_jll", "Graphics", "Libdl", "Pango_jll"] +git-tree-sha1 = "71aa551c5c33f1a4415867fe06b7844faadb0ae9" +uuid = "159f3aea-2a34-519c-b102-8c37f9878175" +version = "1.1.1" + +[[deps.CairoMakie]] +deps = ["CRC32c", "Cairo", "Cairo_jll", "Colors", "FileIO", "FreeType", "GeometryBasics", "LinearAlgebra", "Makie", "PrecompileTools"] +git-tree-sha1 = "47142129b1777e21da58cff265050b10d8560588" +uuid = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0" +version = "0.15.13" + +[[deps.Cairo_jll]] +deps = ["Artifacts", "Bzip2_jll", "CompilerSupportLibraries_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "JLLWrappers", "Libdl", "Pixman_jll", "Xorg_libXext_jll", "Xorg_libXrender_jll", "Zlib_jll", "libpng_jll"] +git-tree-sha1 = "1fa950ebc3e37eccd51c6a8fe1f92f7d86263522" +uuid = "83423d85-b0ee-5818-9007-b63ccbeb887a" +version = "1.18.7+0" + +[[deps.ChainRulesCore]] +deps = ["Compat", "LinearAlgebra"] +git-tree-sha1 = "12177ad6b3cad7fd50c8b3825ce24a99ad61c18f" +uuid = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" +version = "1.26.1" +weakdeps = ["SparseArrays"] + + [deps.ChainRulesCore.extensions] + ChainRulesCoreSparseArraysExt = "SparseArrays" + +[[deps.CodecZstd]] +deps = ["TranscodingStreams", "Zstd_jll"] +git-tree-sha1 = "da54a6cd93c54950c15adf1d336cfd7d71f51a56" +uuid = "6b39b394-51ab-5f42-8807-6242bab2b4c2" +version = "0.8.7" + +[[deps.ColorBrewer]] +deps = ["Colors", "JSON"] +git-tree-sha1 = "07da79661b919001e6863b81fc572497daa58349" +uuid = "a2cac450-b92f-5266-8821-25eda20663c8" +version = "0.4.2" + +[[deps.ColorSchemes]] +deps = ["ColorTypes", "ColorVectorSpace", "Colors", "FixedPointNumbers", "PrecompileTools", "Random"] +git-tree-sha1 = "b0fd3f56fa442f81e0a47815c92245acfaaa4e34" +uuid = "35d6a980-a343-548e-a6ea-1d62b119f2f4" +version = "3.31.0" + +[[deps.ColorTypes]] +deps = ["FixedPointNumbers", "Random"] +git-tree-sha1 = "67e11ee83a43eb71ddc950302c53bf33f0690dfe" +uuid = "3da002f7-5984-5a60-b8a6-cbb66c0b333f" +version = "0.12.1" +weakdeps = ["StyledStrings"] + + [deps.ColorTypes.extensions] + StyledStringsExt = "StyledStrings" + +[[deps.ColorVectorSpace]] +deps = ["ColorTypes", "FixedPointNumbers", "LinearAlgebra", "Requires", "Statistics", "TensorCore"] +git-tree-sha1 = "8b3b6f87ce8f65a2b4f857528fd8d70086cd72b1" +uuid = "c3611d14-8923-5661-9e6a-0046d554d3a4" +version = "0.11.0" +weakdeps = ["SpecialFunctions"] + + [deps.ColorVectorSpace.extensions] + SpecialFunctionsExt = "SpecialFunctions" + +[[deps.Colors]] +deps = ["ColorTypes", "FixedPointNumbers", "Reexport"] +git-tree-sha1 = "37ea44092930b1811e666c3bc38065d7d87fcc74" +uuid = "5ae59095-9a9b-59fe-a467-6f913c188581" +version = "0.13.1" + +[[deps.CommonSolve]] +git-tree-sha1 = "eeaad7cef88554c2fa56b5a3f71cfd5cb708c662" +uuid = "38540f10-b2f7-11e9-35d8-d573e4eb0ff2" +version = "0.2.11" + +[[deps.Compat]] +deps = ["TOML", "UUIDs"] +git-tree-sha1 = "9d8a54ce4b17aa5bdce0ea5c34bc5e7c340d16ad" +uuid = "34da2185-b29b-5c13-b0c7-acf172513d20" +version = "4.18.1" +weakdeps = ["Dates", "LinearAlgebra"] + + [deps.Compat.extensions] + CompatLinearAlgebraExt = "LinearAlgebra" + +[[deps.CompilerSupportLibraries_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "e66e0078-7015-5450-92f7-15fbd957f2ae" +version = "1.3.0+1" + +[[deps.CompositionsBase]] +git-tree-sha1 = "802bb88cd69dfd1509f6670416bd4434015693ad" +uuid = "a33af91c-f02d-484b-be07-31d278c5ca2b" +version = "0.1.2" +weakdeps = ["InverseFunctions"] + + [deps.CompositionsBase.extensions] + CompositionsBaseInverseFunctionsExt = "InverseFunctions" + +[[deps.ComputePipeline]] +deps = ["Observables", "Preferences"] +git-tree-sha1 = "7bc84b769c1d384315e7b5c4ac03a6c303e6cf35" +uuid = "95dc2771-c249-4cd0-9c9f-1f3b4330693c" +version = "0.1.8" + +[[deps.ConstructionBase]] +git-tree-sha1 = "b4b092499347b18a015186eae3042f72267106cb" +uuid = "187b0558-2788-49d3-abe0-74a17ed4e7c9" +version = "1.6.0" +weakdeps = ["IntervalSets", "LinearAlgebra", "StaticArrays"] + + [deps.ConstructionBase.extensions] + ConstructionBaseIntervalSetsExt = "IntervalSets" + ConstructionBaseLinearAlgebraExt = "LinearAlgebra" + ConstructionBaseStaticArraysExt = "StaticArrays" + +[[deps.Contour]] +git-tree-sha1 = "439e35b0b36e2e5881738abc8857bd92ad6ff9a8" +uuid = "d38c429a-6771-53c6-b99e-75d170b6e991" +version = "0.6.3" + +[[deps.CoreMath]] +deps = ["CoreMath_jll"] +git-tree-sha1 = "8c0480f92b1b1796239156a1b9b1bfb1b39499b4" +uuid = "b7a15901-be09-4a0e-87d2-2e66b0e09b5a" +version = "0.1.0" + +[[deps.CoreMath_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "a692a4c1dc59a4b8bc0b6403876eb3250fde2bc3" +uuid = "a38c48d9-6df1-5ac9-9223-b6ada3b5572b" +version = "0.1.0+0" + +[[deps.DataAPI]] +git-tree-sha1 = "abe83f3a2f1b857aac70ef8b269080af17764bbe" +uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" +version = "1.16.0" + +[[deps.DataStructures]] +deps = ["OrderedCollections"] +git-tree-sha1 = "b0bc6d2cad1fed8b7fd59a1551a991cb3d2809e6" +uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8" +version = "0.19.6" + +[[deps.DataValueInterfaces]] +git-tree-sha1 = "bfc1187b79289637fa0ef6d4436ebdfe6905cbd6" +uuid = "e2d170a0-9d28-54be-80f0-106bbe20a464" +version = "1.0.0" + +[[deps.Dates]] +deps = ["Printf"] +uuid = "ade2ca70-3891-5945-98fb-dc099432e06a" +version = "1.11.0" + +[[deps.DelaunayTriangulation]] +deps = ["AdaptivePredicates", "EnumX", "ExactPredicates", "Random"] +git-tree-sha1 = "c55f5a9fd67bdbc8e089b5a3111fe4292986a8e8" +uuid = "927a84f5-c5f4-47a5-9785-b46e178433df" +version = "1.6.6" + +[[deps.Distributed]] +deps = ["Random", "Serialization", "Sockets"] +uuid = "8ba89e20-285c-5b6f-9357-94700520ee1b" +version = "1.11.0" + +[[deps.Distributions]] +deps = ["AliasTables", "FillArrays", "LinearAlgebra", "PDMats", "Printf", "QuadGK", "Random", "Roots", "SpecialFunctions", "Statistics", "StatsAPI", "StatsBase", "StatsFuns"] +git-tree-sha1 = "cd3c5ac74cd3923c8945c6a81518c46abd0e73a3" +uuid = "31c24e10-a181-5473-b8eb-7969acd0382f" +version = "0.25.129" + + [deps.Distributions.extensions] + DistributionsChainRulesCoreExt = "ChainRulesCore" + DistributionsDensityInterfaceExt = "DensityInterface" + DistributionsSparseConnectivityTracerExt = "SparseConnectivityTracer" + DistributionsTestExt = "Test" + + [deps.Distributions.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + DensityInterface = "b429d917-457f-4dbc-8f4c-0cc954292b1d" + SparseConnectivityTracer = "9f842d2f-2579-4b1d-911e-f412cf18a3f5" + Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[[deps.DocStringExtensions]] +git-tree-sha1 = "7442a5dfe1ebb773c29cc2962a8980f47221d76c" +uuid = "ffbed154-4ef7-542d-bbb7-c09d3a79fcae" +version = "0.9.5" + +[[deps.Downloads]] +deps = ["ArgTools", "FileWatching", "LibCURL", "NetworkOptions"] +uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6" +version = "1.7.0" + +[[deps.EarCut_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "e3290f2d49e661fbd94046d7e3726ffcb2d41053" +uuid = "5ae413db-bbd1-5e63-b57d-d24a61df00f5" +version = "2.2.4+0" + +[[deps.EnumX]] +git-tree-sha1 = "c49898e8438c828577f04b92fc9368c388ac783c" +uuid = "4e289a0a-7415-4d19-859d-a7e5c4648b56" +version = "1.0.7" + +[[deps.ExactPredicates]] +deps = ["IntervalArithmetic", "Random", "StaticArrays"] +git-tree-sha1 = "83231673ea4d3d6008ac74dc5079e77ab2209d8f" +uuid = "429591f6-91af-11e9-00e2-59fbe8cec110" +version = "2.2.9" + +[[deps.Expat_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e6c4a6407a949e79a9d3f249bf49e6987c80e01f" +uuid = "2e619515-83b5-522b-bb60-26c02a35a201" +version = "2.8.2+0" + +[[deps.FFMPEG_jll]] +deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "JLLWrappers", "LAME_jll", "Libdl", "Ogg_jll", "OpenSSL_jll", "Opus_jll", "PCRE2_jll", "Zlib_jll", "libaom_jll", "libass_jll", "libfdk_aac_jll", "libva_jll", "libvorbis_jll", "x264_jll", "x265_jll"] +git-tree-sha1 = "7a58e45171b63ed4782f2d36fdee8713a469e6e0" +uuid = "b22a6f82-2f65-5046-a5b2-351ab43fb4e5" +version = "8.1.2+0" + +[[deps.FFTA]] +deps = ["AbstractFFTs", "DocStringExtensions", "LinearAlgebra", "MuladdMacro", "Primes", "Random", "Reexport"] +git-tree-sha1 = "65e55303b72f4a567a51b174dd2c47496efeb95a" +uuid = "b86e33f2-c0db-4aa1-a6e0-ab43e668529e" +version = "0.3.1" + +[[deps.FileIO]] +deps = ["Pkg", "Requires", "UUIDs"] +git-tree-sha1 = "6621fef488e496356c9c9625d0562c12a6070819" +uuid = "5789e2e9-d7fb-5bc7-8068-2c6fae9b9549" +version = "1.20.0" + + [deps.FileIO.extensions] + HTTPExt = "HTTP" + + [deps.FileIO.weakdeps] + HTTP = "cd3eb016-35fb-5094-929b-558a96fad6f3" + +[[deps.FilePaths]] +deps = ["FilePathsBase", "MacroTools", "Reexport"] +git-tree-sha1 = "a1b2fbfe98503f15b665ed45b3d149e5d8895e4c" +uuid = "8fc22ac5-c921-52a6-82fd-178b2807b824" +version = "0.9.0" + + [deps.FilePaths.extensions] + FilePathsGlobExt = "Glob" + FilePathsURIParserExt = "URIParser" + FilePathsURIsExt = "URIs" + + [deps.FilePaths.weakdeps] + Glob = "c27321d9-0574-5035-807b-f59d2c89b15c" + URIParser = "30578b45-9adc-5946-b283-645ec420af67" + URIs = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4" + +[[deps.FilePathsBase]] +deps = ["Compat", "Dates"] +git-tree-sha1 = "3bab2c5aa25e7840a4b065805c0cdfc01f3068d2" +uuid = "48062228-2e41-5def-b9a4-89aafe57970f" +version = "0.9.24" +weakdeps = ["Mmap", "Test"] + + [deps.FilePathsBase.extensions] + FilePathsBaseMmapExt = "Mmap" + FilePathsBaseTestExt = "Test" + +[[deps.FileWatching]] +uuid = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee" +version = "1.11.0" + +[[deps.FillArrays]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "5bad39456d9f0166184fce2248783dd9862645c1" +uuid = "1a297f60-69ca-5386-bcde-b61e274b549b" +version = "1.17.0" +weakdeps = ["PDMats", "SparseArrays", "StaticArrays", "Statistics"] + + [deps.FillArrays.extensions] + FillArraysPDMatsExt = "PDMats" + FillArraysSparseArraysExt = "SparseArrays" + FillArraysStaticArraysExt = "StaticArrays" + FillArraysStatisticsExt = "Statistics" + +[[deps.FixedPointNumbers]] +deps = ["Random", "Statistics"] +git-tree-sha1 = "59af96b98217c6ef4ae0dfe065ac7c20831d1a84" +uuid = "53c48c17-4a7d-5ca2-90c5-79b7896eea93" +version = "0.8.6" + +[[deps.Fontconfig_jll]] +deps = ["Artifacts", "Bzip2_jll", "Expat_jll", "FreeType2_jll", "JLLWrappers", "Libdl", "Libuuid_jll", "Zlib_jll"] +git-tree-sha1 = "f85dac9a96a01087df6e3a749840015a0ca3817d" +uuid = "a3f928ae-7b40-5064-980b-68af3947d34b" +version = "2.17.1+0" + +[[deps.Format]] +git-tree-sha1 = "9c68794ef81b08086aeb32eeaf33531668d5f5fc" +uuid = "1fa38f19-a742-5d3f-a2b9-30dd87b9d5f8" +version = "1.3.7" + +[[deps.FreeType]] +deps = ["CEnum", "FreeType2_jll"] +git-tree-sha1 = "907369da0f8e80728ab49c1c7e09327bf0d6d999" +uuid = "b38be410-82b0-50bf-ab77-7b57e271db43" +version = "4.1.1" + +[[deps.FreeType2_jll]] +deps = ["Artifacts", "Bzip2_jll", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "70329abc09b886fd2c5d94ad2d9527639c421e3e" +uuid = "d7e528f0-a631-5988-bf34-fe36492bcfd7" +version = "2.14.3+1" + +[[deps.FreeTypeAbstraction]] +deps = ["BaseDirs", "ColorVectorSpace", "Colors", "FreeType", "GeometryBasics", "Mmap"] +git-tree-sha1 = "4ebb930ef4a43817991ba35db6317a05e59abd11" +uuid = "663a7486-cb36-511b-a19d-713bb74d65c9" +version = "0.10.8" + +[[deps.FriBidi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "7a214fdac5ed5f59a22c2d9a885a16da1c74bbc7" +uuid = "559328eb-81f9-559d-9380-de523a88c83c" +version = "1.0.17+0" + +[[deps.Gamma]] +git-tree-sha1 = "86f86b6168a016ed88e4ae4e64577b98c3b59e8e" +uuid = "a0844989-3bd2-4988-8bea-c9407ab0941b" +version = "1.1.0" + +[[deps.GeometryBasics]] +deps = ["EarCut_jll", "LinearAlgebra", "PrecompileTools", "Random", "StaticArrays"] +git-tree-sha1 = "364685f5ffde25deb1bbcfd5bb278a5c6b7a9b37" +uuid = "5c1252a2-5f33-56bf-86c9-59e7332b4326" +version = "0.5.11" + + [deps.GeometryBasics.extensions] + ExtentsExt = "Extents" + GeometryBasicsGeoInterfaceExt = "GeoInterface" + IntervalSetsExt = "IntervalSets" + + [deps.GeometryBasics.weakdeps] + Extents = "411431e0-e8b7-467b-b5e0-f676ba4f2910" + GeoInterface = "cf35fbd7-0cd7-5166-be24-54bfbe79505f" + IntervalSets = "8197267c-284f-5f27-9208-e0e47529a953" + +[[deps.GettextRuntime_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl", "Libiconv_jll"] +git-tree-sha1 = "45288942190db7c5f760f59c04495064eedf9340" +uuid = "b0724c58-0f36-5564-988d-3bb0596ebc4a" +version = "0.22.4+0" + +[[deps.Giflib_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "6570366d757b50fabae9f4315ad74d2e40c0560a" +uuid = "59f7168a-df46-5410-90c8-f2779963d0ec" +version = "5.2.3+0" + +[[deps.Glib_jll]] +deps = ["Artifacts", "GettextRuntime_jll", "JLLWrappers", "Libdl", "Libffi_jll", "Libiconv_jll", "Libmount_jll", "PCRE2_jll", "Zlib_jll"] +git-tree-sha1 = "24f6def62397474a297bfcec22384101609142ed" +uuid = "7746bdde-850d-59dc-9ae8-88ece973131d" +version = "2.86.3+0" + +[[deps.Graphics]] +deps = ["Colors", "LinearAlgebra", "NaNMath"] +git-tree-sha1 = "a641238db938fff9b2f60d08ed9030387daf428c" +uuid = "a2bd30eb-e257-5431-a919-1863eab51364" +version = "1.1.3" + +[[deps.Graphite2_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "69ffb934a5c5b7e086a0b4fee3427db2556fba6e" +uuid = "3b182d85-2403-5c21-9c21-1e1f0cc25472" +version = "1.3.16+0" + +[[deps.GridLayoutBase]] +deps = ["GeometryBasics", "InteractiveUtils", "Observables"] +git-tree-sha1 = "93d5c27c8de51687a2c70ec0716e6e76f298416f" +uuid = "3955a311-db13-416c-9275-1d80ed98e5e9" +version = "0.11.2" + +[[deps.HarfBuzz_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "Glib_jll", "Graphite2_jll", "JLLWrappers", "Libdl", "Libffi_jll"] +git-tree-sha1 = "f923f9a774fcf3f5cb761bfa43aeadd689714813" +uuid = "2e76f6c2-a576-52d4-95c1-20adfe4de566" +version = "8.5.1+0" + +[[deps.HashArrayMappedTries]] +git-tree-sha1 = "2eaa69a7cab70a52b9687c8bf950a5a93ec895ae" +uuid = "076d061b-32b6-4027-95e0-9a2c6f6d7e74" +version = "0.2.0" + +[[deps.HypergeometricFunctions]] +deps = ["Gamma", "LinearAlgebra"] +git-tree-sha1 = "18d7deab5fb0440dc6a7b6993c5c27b25420de10" +uuid = "34004b35-14d8-5ef3-9330-4cdb6864b03a" +version = "0.3.29" + +[[deps.ImageAxes]] +deps = ["AxisArrays", "ImageBase", "ImageCore", "Reexport", "SimpleTraits"] +git-tree-sha1 = "e12629406c6c4442539436581041d372d69c55ba" +uuid = "2803e5a7-5153-5ecf-9a86-9b4c37f5f5ac" +version = "0.6.12" + +[[deps.ImageBase]] +deps = ["ImageCore", "Reexport"] +git-tree-sha1 = "eb49b82c172811fd2c86759fa0553a2221feb909" +uuid = "c817782e-172a-44cc-b673-b171935fbb9e" +version = "0.1.7" + +[[deps.ImageCore]] +deps = ["ColorVectorSpace", "Colors", "FixedPointNumbers", "MappedArrays", "MosaicViews", "OffsetArrays", "PaddedViews", "PrecompileTools", "Reexport"] +git-tree-sha1 = "8c193230235bbcee22c8066b0374f63b5683c2d3" +uuid = "a09fc81d-aa75-5fe9-8630-4744c3626534" +version = "0.10.5" + +[[deps.ImageIO]] +deps = ["FileIO", "IndirectArrays", "JpegTurbo", "LazyModules", "Netpbm", "OpenEXR", "PNGFiles", "QOI", "Sixel", "TiffImages", "UUIDs", "WebP"] +git-tree-sha1 = "696144904b76e1ca433b886b4e7edd067d76cbf7" +uuid = "82e4d734-157c-48bb-816b-45c225c6df19" +version = "0.6.9" + +[[deps.ImageMetadata]] +deps = ["AxisArrays", "ImageAxes", "ImageBase", "ImageCore"] +git-tree-sha1 = "2a81c3897be6fbcde0802a0ebe6796d0562f63ec" +uuid = "bc367c6b-8a6b-528e-b4bd-a4b897500b49" +version = "0.9.10" + +[[deps.Imath_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "dcc8d0cd653e55213df9b75ebc6fe4a8d3254c65" +uuid = "905a6f67-0a94-5f89-b386-d35d92009cd1" +version = "3.2.2+0" + +[[deps.IndirectArrays]] +git-tree-sha1 = "012e604e1c7458645cb8b436f8fba789a51b257f" +uuid = "9b13fd28-a010-5f03-acff-a1bbcff69959" +version = "1.0.0" + +[[deps.Inflate]] +git-tree-sha1 = "d1b1b796e47d94588b3757fe84fbf65a5ec4a80d" +uuid = "d25df0c9-e2be-5dd7-82c8-3ad0b3e990b9" +version = "0.1.5" + +[[deps.IntegerMathUtils]] +git-tree-sha1 = "4c1acff2dc6b6967e7e750633c50bc3b8d83e617" +uuid = "18e54dd8-cb9d-406c-a71d-865a43cbb235" +version = "0.1.3" + +[[deps.InteractiveUtils]] +deps = ["Markdown"] +uuid = "b77e0a4c-d291-57a0-90e8-8db25a27a240" +version = "1.11.0" + +[[deps.Interpolations]] +deps = ["Adapt", "AxisAlgorithms", "ChainRulesCore", "LinearAlgebra", "OffsetArrays", "Random", "Ratios", "SharedArrays", "SparseArrays", "StaticArrays", "WoodburyMatrices"] +git-tree-sha1 = "48922d06068130f87e43edef52382e6a94305ae6" +uuid = "a98d9a8b-a2ab-59e6-89dd-64a1c18fca59" +version = "0.16.3" + + [deps.Interpolations.extensions] + InterpolationsForwardDiffExt = "ForwardDiff" + InterpolationsUnitfulExt = "Unitful" + + [deps.Interpolations.weakdeps] + ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210" + Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d" + +[[deps.IntervalArithmetic]] +deps = ["CRlibm", "CoreMath", "MacroTools", "OpenBLASConsistentFPCSR_jll", "Printf", "Random", "RoundingEmulator"] +git-tree-sha1 = "c3ee408ae340565f41699e3a3fa1053698c7626e" +uuid = "d1acc4aa-44c8-5952-acd4-ba5d80a2a253" +version = "1.0.10" + + [deps.IntervalArithmetic.extensions] + IntervalArithmeticArblibExt = "Arblib" + IntervalArithmeticDiffRulesExt = "DiffRules" + IntervalArithmeticForwardDiffExt = "ForwardDiff" + IntervalArithmeticIntervalSetsExt = "IntervalSets" + IntervalArithmeticIrrationalConstantsExt = "IrrationalConstants" + IntervalArithmeticLinearAlgebraExt = "LinearAlgebra" + IntervalArithmeticRecipesBaseExt = "RecipesBase" + IntervalArithmeticSparseArraysExt = "SparseArrays" + + [deps.IntervalArithmetic.weakdeps] + Arblib = "fb37089c-8514-4489-9461-98f9c8763369" + DiffRules = "b552c78f-8df3-52c6-915a-8e097449b14b" + ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210" + IntervalSets = "8197267c-284f-5f27-9208-e0e47529a953" + IrrationalConstants = "92d709cd-6900-40b7-9082-c6be49f344b6" + LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + RecipesBase = "3cdcf5f2-1ef4-517c-9805-6587b60abb01" + SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" + +[[deps.IntervalSets]] +git-tree-sha1 = "79d6bd28c8d9bccc2229784f1bd637689b256377" +uuid = "8197267c-284f-5f27-9208-e0e47529a953" +version = "0.7.14" + + [deps.IntervalSets.extensions] + IntervalSetsRandomExt = "Random" + IntervalSetsRecipesBaseExt = "RecipesBase" + IntervalSetsStatisticsExt = "Statistics" + + [deps.IntervalSets.weakdeps] + Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" + RecipesBase = "3cdcf5f2-1ef4-517c-9805-6587b60abb01" + Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" + +[[deps.InverseFunctions]] +git-tree-sha1 = "a779299d77cd080bf77b97535acecd73e1c5e5cb" +uuid = "3587e190-3f89-42d0-90ee-14403ec27112" +version = "0.1.17" +weakdeps = ["Dates", "Test"] + + [deps.InverseFunctions.extensions] + InverseFunctionsDatesExt = "Dates" + InverseFunctionsTestExt = "Test" + +[[deps.IrrationalConstants]] +git-tree-sha1 = "b2d91fe939cae05960e760110b328288867b5758" +uuid = "92d709cd-6900-40b7-9082-c6be49f344b6" +version = "0.2.6" + +[[deps.Isoband]] +deps = ["isoband_jll"] +git-tree-sha1 = "f9b6d97355599074dc867318950adaa6f9946137" +uuid = "f1662d9f-8043-43de-a69a-05efc1cc6ff4" +version = "0.1.1" + +[[deps.IterTools]] +git-tree-sha1 = "42d5f897009e7ff2cf88db414a389e5ed1bdd023" +uuid = "c8e1da08-722c-5040-9ed9-7db0dc04731e" +version = "1.10.0" + +[[deps.IteratorInterfaceExtensions]] +git-tree-sha1 = "a3f24677c21f5bbe9d2a714f95dcd58337fb2856" +uuid = "82899510-4779-5014-852e-03e436cf321d" +version = "1.0.0" + +[[deps.JLD2]] +deps = ["FileIO", "MacroTools", "Mmap", "OrderedCollections", "PrecompileTools", "ScopedValues", "TranscodingStreams"] +git-tree-sha1 = "d97791feefda45729613fafeccc4fbef3f539151" +uuid = "033835bb-8acc-5ee8-8aae-3f567f8a3819" +version = "0.5.15" + + [deps.JLD2.extensions] + UnPackExt = "UnPack" + + [deps.JLD2.weakdeps] + UnPack = "3a884ed6-31ef-47d7-9d2a-63182c4928ed" + +[[deps.JLLWrappers]] +deps = ["Artifacts", "Preferences"] +git-tree-sha1 = "7204148362dafe5fe6a273f855b8ccbe4df8173e" +uuid = "692b3bcd-3c85-4b1f-b108-f13ce0eb3210" +version = "1.8.0" + +[[deps.JSON]] +deps = ["Dates", "Logging", "Parsers", "PrecompileTools", "StructUtils", "UUIDs", "Unicode"] +git-tree-sha1 = "c89d196f5ffb64bfbf80985b699ea913b0d2c211" +uuid = "682c06a0-de6a-54ab-a142-c8b1cf79cde6" +version = "1.6.1" + + [deps.JSON.extensions] + JSONArrowExt = ["ArrowTypes"] + + [deps.JSON.weakdeps] + ArrowTypes = "31f734f8-188a-4ce0-8406-c8a06bd891cd" + +[[deps.JpegTurbo]] +deps = ["CEnum", "FileIO", "ImageCore", "JpegTurbo_jll", "TOML"] +git-tree-sha1 = "9496de8fb52c224a2e3f9ff403947674517317d9" +uuid = "b835a17e-a41a-41e7-81f0-2f016b05efe0" +version = "0.1.6" + +[[deps.JpegTurbo_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "1dae3057da6f2b9c857afef03177bbdc7c4afe92" +uuid = "aacddb02-875f-59d6-b918-886e6ef4fbf8" +version = "3.2.0+0" + +[[deps.JuliaSyntaxHighlighting]] +deps = ["StyledStrings"] +uuid = "ac6e5ff7-fb65-4e79-a425-ec3bc9c03011" +version = "1.12.0" + +[[deps.KernelDensity]] +deps = ["Distributions", "DocStringExtensions", "FFTA", "Interpolations", "StatsBase"] +git-tree-sha1 = "9eda8292dd3268b3b7ec9df21bbfac24e177ec52" +uuid = "5ab0869b-81aa-558d-bb23-cbf5423bbe9b" +version = "0.6.12" + +[[deps.LAME_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "059aabebaa7c82ccb853dd4a0ee9d17796f7e1bc" +uuid = "c1c5ebd0-6772-5130-a774-d5fcae4a789d" +version = "3.100.3+0" + +[[deps.LERC_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "17b94ecafcfa45e8360a4fc9ca6b583b049e4e37" +uuid = "88015f11-f218-50d7-93a8-a6af411a945d" +version = "4.1.0+0" + +[[deps.LLVMOpenMP_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "b7970cef8ae1c990ba0c09cd8bdc1145e006632f" +uuid = "1d63c593-3942-5779-bab2-d838dc0a180e" +version = "22.1.7+0" + +[[deps.LaTeXStrings]] +git-tree-sha1 = "dda21b8cbd6a6c40d9d02a73230f9d70fed6918c" +uuid = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" +version = "1.4.0" + +[[deps.LazyModules]] +git-tree-sha1 = "a560dd966b386ac9ae60bdd3a3d3a326062d3c3e" +uuid = "8cdb02fc-e678-4876-92c5-9defec4f444e" +version = "0.3.1" + +[[deps.LibCURL]] +deps = ["LibCURL_jll", "MozillaCACerts_jll"] +uuid = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21" +version = "0.6.4" + +[[deps.LibCURL_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "OpenSSL_jll", "Zlib_jll", "nghttp2_jll"] +uuid = "deac9b47-8bc7-5906-a0fe-35ac56dc84c0" +version = "8.15.0+0" + +[[deps.LibGit2]] +deps = ["LibGit2_jll", "NetworkOptions", "Printf", "SHA"] +uuid = "76f85450-5226-5b5a-8eaa-529ad045b433" +version = "1.11.0" + +[[deps.LibGit2_jll]] +deps = ["Artifacts", "LibSSH2_jll", "Libdl", "OpenSSL_jll"] +uuid = "e37daf67-58a4-590a-8e99-b0245dd2ffc5" +version = "1.9.0+0" + +[[deps.LibSSH2_jll]] +deps = ["Artifacts", "Libdl", "OpenSSL_jll"] +uuid = "29816b5a-b9ab-546f-933c-edad1886dfa8" +version = "1.11.3+1" + +[[deps.Libdl]] +uuid = "8f399da3-3557-5675-b5ff-fb832c97cbdb" +version = "1.11.0" + +[[deps.Libffi_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "c8da7e6a91781c41a863611c7e966098d783c57a" +uuid = "e9f186c6-92d2-5b65-8a66-fee21dc1b490" +version = "3.4.7+0" + +[[deps.Libglvnd_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll", "Xorg_libXext_jll"] +git-tree-sha1 = "d36c21b9e7c172a44a10484125024495e2625ac0" +uuid = "7e76a0d4-f3c7-5321-8279-8d96eeed0f29" +version = "1.7.1+1" + +[[deps.Libiconv_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "be484f5c92fad0bd8acfef35fe017900b0b73809" +uuid = "94ce4f54-9a6c-5748-9c1c-f9c7231a4531" +version = "1.18.0+0" + +[[deps.Libmount_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "cc3ad4faf30015a3e8094c9b5b7f19e85bdf2386" +uuid = "4b2f31a3-9ecc-558c-b454-b3730dcb73e9" +version = "2.42.0+0" + +[[deps.Libtiff_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "LERC_jll", "Libdl", "XZ_jll", "Zlib_jll", "Zstd_jll"] +git-tree-sha1 = "aebd334d06cee9f24cea70bd19a39749daf73881" +uuid = "89763e89-9b03-5906-acba-b20f662cd828" +version = "4.7.3+0" + +[[deps.Libuuid_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "d620582b1f0cbe2c72dd1d5bd195a9ce73370ab1" +uuid = "38a345b3-de98-5d2b-a5d3-14cd9215e700" +version = "2.42.0+0" + +[[deps.LinearAlgebra]] +deps = ["Libdl", "OpenBLAS_jll", "libblastrampoline_jll"] +uuid = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" +version = "1.12.0" + +[[deps.LogExpFunctions]] +deps = ["DocStringExtensions", "IrrationalConstants", "LinearAlgebra"] +git-tree-sha1 = "bba2d9aa057d8f126415de240573e86a8f39d2a1" +uuid = "2ab3a3ac-af41-5b50-aa03-7779005ae688" +version = "1.0.1" + + [deps.LogExpFunctions.extensions] + LogExpFunctionsChainRulesCoreExt = "ChainRulesCore" + LogExpFunctionsChangesOfVariablesExt = "ChangesOfVariables" + LogExpFunctionsInverseFunctionsExt = "InverseFunctions" + + [deps.LogExpFunctions.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + ChangesOfVariables = "9e997f8a-9a97-42d5-a9f1-ce6bfc15e2c0" + InverseFunctions = "3587e190-3f89-42d0-90ee-14403ec27112" + +[[deps.Logging]] +uuid = "56ddb016-857b-54e1-b83d-db4d58db5568" +version = "1.11.0" + +[[deps.MacroTools]] +git-tree-sha1 = "1e0228a030642014fe5cfe68c2c0a818f9e3f522" +uuid = "1914dd2f-81c6-5fcd-8719-6d5c9610ff09" +version = "0.5.16" + +[[deps.Makie]] +deps = ["Animations", "Base64", "CRC32c", "ColorBrewer", "ColorSchemes", "ColorTypes", "Colors", "ComputePipeline", "Contour", "Dates", "DelaunayTriangulation", "Distributions", "DocStringExtensions", "Downloads", "FFMPEG_jll", "FileIO", "FilePaths", "FixedPointNumbers", "Format", "FreeType", "FreeTypeAbstraction", "GeometryBasics", "GridLayoutBase", "ImageBase", "ImageIO", "InteractiveUtils", "Interpolations", "IntervalSets", "InverseFunctions", "Isoband", "KernelDensity", "LaTeXStrings", "LinearAlgebra", "MacroTools", "Markdown", "MathTeXEngine", "Observables", "OffsetArrays", "PNGFiles", "Packing", "Pkg", "PlotUtils", "PolygonOps", "PrecompileTools", "Printf", "REPL", "Random", "RelocatableFolders", "Scratch", "ShaderAbstractions", "SignedDistanceFields", "SparseArrays", "Statistics", "StatsBase", "StatsFuns", "StructArrays", "TriplotBase", "UnicodeFun", "Unitful"] +git-tree-sha1 = "f2c8715d05bf10f9d4dc354e69dee30b6be53239" +uuid = "ee78f7c6-11fb-53f2-987a-cfe4a2b5a57a" +version = "0.24.13" + + [deps.Makie.extensions] + MakieDynamicQuantitiesExt = "DynamicQuantities" + + [deps.Makie.weakdeps] + DynamicQuantities = "06fc5a27-2a28-4c7c-a15d-362465fb6821" + +[[deps.MappedArrays]] +git-tree-sha1 = "0ee4497a4e80dbd29c058fcee6493f5219556f40" +uuid = "dbb5928d-eab1-5f90-85c2-b9b0edb7c900" +version = "0.4.3" + +[[deps.Markdown]] +deps = ["Base64", "JuliaSyntaxHighlighting", "StyledStrings"] +uuid = "d6f4376e-aef5-505a-96c1-9c027394607a" +version = "1.11.0" + +[[deps.MathTeXEngine]] +deps = ["AbstractTrees", "Automa", "DataStructures", "FreeTypeAbstraction", "GeometryBasics", "LaTeXStrings", "REPL", "RelocatableFolders", "UnicodeFun"] +git-tree-sha1 = "aa1078778be5a8e5259ff04fbc3d258b3e78d464" +uuid = "0a4f8689-d25c-4efe-a92b-7142dfc1aa53" +version = "0.6.9" + +[[deps.Missings]] +deps = ["DataAPI"] +git-tree-sha1 = "ec4f7fbeab05d7747bdf98eb74d130a2a2ed298d" +uuid = "e1d29d7a-bbdc-5cf2-9ac0-f12de2c33e28" +version = "1.2.0" + +[[deps.Mmap]] +uuid = "a63ad114-7e13-5084-954f-fe012c677804" +version = "1.11.0" + +[[deps.MosaicViews]] +deps = ["MappedArrays", "OffsetArrays", "PaddedViews", "StackViews"] +git-tree-sha1 = "7b86a5d4d70a9f5cdf2dacb3cbe6d251d1a61dbe" +uuid = "e94cdb99-869f-56ef-bcf0-1ae2bcbe0389" +version = "0.3.4" + +[[deps.MozillaCACerts_jll]] +uuid = "14a3606d-f60d-562e-9121-12d972cd8159" +version = "2025.11.4" + +[[deps.MuladdMacro]] +deps = ["PrecompileTools"] +git-tree-sha1 = "e8dcbeef032ba2f9051a44ac22b4e54e3a1a0099" +uuid = "46d2c3a1-f734-5fdb-9937-b9b9aeba4221" +version = "0.2.6" + +[[deps.NaNMath]] +deps = ["OpenLibm_jll"] +git-tree-sha1 = "dbd2e8cd2c1c27f0b584f6661b4309609c5a685e" +uuid = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3" +version = "1.1.4" + +[[deps.Netpbm]] +deps = ["FileIO", "ImageCore", "ImageMetadata"] +git-tree-sha1 = "d92b107dbb887293622df7697a2223f9f8176fcd" +uuid = "f09324ee-3d7c-5217-9330-fc30815ba969" +version = "1.1.1" + +[[deps.NetworkOptions]] +uuid = "ca575930-c2e3-43a9-ace4-1e988b2c1908" +version = "1.3.0" + +[[deps.Observables]] +git-tree-sha1 = "7438a59546cf62428fc9d1bc94729146d37a7225" +uuid = "510215fc-4207-5dde-b226-833fc4488ee2" +version = "0.5.5" + +[[deps.OffsetArrays]] +git-tree-sha1 = "117432e406b5c023f665fa73dc26e79ec3630151" +uuid = "6fe1bfb0-de20-5000-8ca7-80f57d26f881" +version = "1.17.0" +weakdeps = ["Adapt"] + + [deps.OffsetArrays.extensions] + OffsetArraysAdaptExt = "Adapt" + +[[deps.Ogg_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "b6aa4566bb7ae78498a5e68943863fa8b5231b59" +uuid = "e7412a2a-1a6e-54c0-be00-318e2571c051" +version = "1.3.6+0" + +[[deps.OpenBLASConsistentFPCSR_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl"] +git-tree-sha1 = "dafdaa3ff15f20ff703d909d3a6f574a5b0586f3" +uuid = "6cdc7f73-28fd-5e50-80fb-958a8875b1af" +version = "0.3.33+1" + +[[deps.OpenBLAS_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] +uuid = "4536629a-c528-5b80-bd46-f80d51c5b363" +version = "0.3.29+0" + +[[deps.OpenEXR]] +deps = ["Colors", "FileIO", "OpenEXR_jll"] +git-tree-sha1 = "97db9e07fe2091882c765380ef58ec553074e9c7" +uuid = "52e1d378-f018-4a11-a4be-720524705ac7" +version = "0.3.3" + +[[deps.OpenEXR_jll]] +deps = ["Artifacts", "Imath_jll", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "0d621a4beb5e48d195f907c3c5b0bea285d9ff9d" +uuid = "18a262bb-aa17-5467-a713-aee519bc75cb" +version = "3.4.13+0" + +[[deps.OpenLibm_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "05823500-19ac-5b8b-9628-191a04bc5112" +version = "0.8.7+0" + +[[deps.OpenSSL_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "458c3c95-2e84-50aa-8efc-19380b2a3a95" +version = "3.5.4+0" + +[[deps.OpenSpecFun_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "Libdl"] +git-tree-sha1 = "1346c9208249809840c91b26703912dff463d335" +uuid = "efe28fd5-8261-553b-a9e1-b2916fc3738e" +version = "0.5.6+0" + +[[deps.Opus_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e2bb57a313a74b8104064b7efd01406c0a50d2ff" +uuid = "91d4177d-7536-5919-b921-800302f37372" +version = "1.6.1+0" + +[[deps.OrderedCollections]] +git-tree-sha1 = "94ba93778373a53bfd5a0caaf7d809c445292ff4" +uuid = "bac558e1-5e72-5ebc-8fee-abe8a469f55d" +version = "1.8.2" + +[[deps.PCRE2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "efcefdf7-47ab-520b-bdef-62a2eaa19f15" +version = "10.44.0+1" + +[[deps.PDMats]] +deps = ["LinearAlgebra", "SparseArrays", "SuiteSparse"] +git-tree-sha1 = "26766d4b5f1a410c218a19b85a672c6edb693c65" +uuid = "90014a1f-27ba-587c-ab20-58faa44d9150" +version = "0.11.40" +weakdeps = ["StatsBase"] + + [deps.PDMats.extensions] + StatsBaseExt = "StatsBase" + +[[deps.PNGFiles]] +deps = ["Base64", "CEnum", "ImageCore", "IndirectArrays", "OffsetArrays", "libpng_jll"] +git-tree-sha1 = "32b657a0d57c310a1a172bfc8c8cf68c5e674323" +uuid = "f57f5aa1-a3ce-4bc8-8ab9-96f992907883" +version = "0.4.5" + +[[deps.Packing]] +deps = ["GeometryBasics"] +git-tree-sha1 = "bc5bf2ea3d5351edf285a06b0016788a121ce92c" +uuid = "19eb6ba3-879d-56ad-ad62-d5c202156566" +version = "0.5.1" + +[[deps.PaddedViews]] +deps = ["OffsetArrays"] +git-tree-sha1 = "0fac6313486baae819364c52b4f483450a9d793f" +uuid = "5432bcbf-9aad-5242-b902-cca2824c8663" +version = "0.5.12" + +[[deps.Pango_jll]] +deps = ["Artifacts", "Cairo_jll", "Fontconfig_jll", "FreeType2_jll", "FriBidi_jll", "Glib_jll", "HarfBuzz_jll", "JLLWrappers", "Libdl"] +git-tree-sha1 = "58e5ed5e386e156bd93e86b305ebd21ac63d2d04" +uuid = "36c8627f-9965-5494-a995-c6b170f724f3" +version = "1.57.1+0" + +[[deps.Parsers]] +deps = ["Dates", "PrecompileTools", "UUIDs"] +git-tree-sha1 = "32a4e09c5f29402573d673901778a0e03b0807b9" +uuid = "69de0a69-1ddd-5017-9359-2bf0b02dc9f0" +version = "2.8.6" + +[[deps.Pixman_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "JLLWrappers", "LLVMOpenMP_jll", "Libdl"] +git-tree-sha1 = "e4a6721aa89e62e5d4217c0b21bd714263779dda" +uuid = "30392449-352a-5448-841d-b1acce4e97dc" +version = "0.46.4+0" + +[[deps.Pkg]] +deps = ["Artifacts", "Dates", "Downloads", "FileWatching", "LibGit2", "Libdl", "Logging", "Markdown", "Printf", "Random", "SHA", "TOML", "Tar", "UUIDs", "p7zip_jll"] +uuid = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f" +version = "1.12.1" +weakdeps = ["REPL"] + + [deps.Pkg.extensions] + REPLExt = "REPL" + +[[deps.PkgVersion]] +deps = ["Pkg"] +git-tree-sha1 = "f9501cc0430a26bc3d156ae1b5b0c1b47af4d6da" +uuid = "eebad327-c553-4316-9ea0-9fa01ccd7688" +version = "0.3.3" + +[[deps.PlotUtils]] +deps = ["ColorSchemes", "Colors", "Dates", "PrecompileTools", "Printf", "Random", "Reexport", "StableRNGs", "Statistics"] +git-tree-sha1 = "26ca162858917496748aad52bb5d3be4d26a228a" +uuid = "995b91a9-d308-5afd-9ec6-746e21dbc043" +version = "1.4.4" + +[[deps.PolygonOps]] +git-tree-sha1 = "77b3d3605fc1cd0b42d95eba87dfcd2bf67d5ff6" +uuid = "647866c9-e3ac-4575-94e7-e3d426903924" +version = "0.1.2" + +[[deps.PrecompileTools]] +deps = ["Preferences"] +git-tree-sha1 = "edbeefc7a4889f528644251bdb5fc9ab5348bc2c" +uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a" +version = "1.3.4" + +[[deps.Preferences]] +deps = ["TOML"] +git-tree-sha1 = "8b770b60760d4451834fe79dd483e318eee709c4" +uuid = "21216c6a-2e73-6563-6e65-726566657250" +version = "1.5.2" + +[[deps.Primes]] +deps = ["IntegerMathUtils"] +git-tree-sha1 = "25cdd1d20cd005b52fc12cb6be3f75faaf59bb9b" +uuid = "27ebfcd6-29c5-5fa9-bf4b-fb8fc14df3ae" +version = "0.5.7" + +[[deps.Printf]] +deps = ["Unicode"] +uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7" +version = "1.11.0" + +[[deps.ProgressMeter]] +deps = ["Distributed", "Printf"] +git-tree-sha1 = "fbb92c6c56b34e1a2c4c36058f68f332bec840e7" +uuid = "92933f4c-e287-5a05-a399-4b506db050ca" +version = "1.11.0" + +[[deps.PtrArrays]] +git-tree-sha1 = "4fbbafbc6251b883f4d2705356f3641f3652a7fe" +uuid = "43287f4e-b6f4-7ad1-bb20-aadabca52c3d" +version = "1.4.0" + +[[deps.QOI]] +deps = ["ColorTypes", "FileIO", "FixedPointNumbers"] +git-tree-sha1 = "472daaa816895cb7aee81658d4e7aec901fa1106" +uuid = "4b34888f-f399-49d4-9bb3-47ed5cae4e65" +version = "1.0.2" + +[[deps.QuadGK]] +deps = ["DataStructures", "LinearAlgebra"] +git-tree-sha1 = "5e8e8b0ab68215d7a2b14b9921a946fee794749e" +uuid = "1fd47b50-473d-5c70-9696-f719f8f3bcdc" +version = "2.11.3" + + [deps.QuadGK.extensions] + QuadGKEnzymeExt = "Enzyme" + + [deps.QuadGK.weakdeps] + Enzyme = "7da242da-08ed-463a-9acd-ee780be4f1d9" + +[[deps.REPL]] +deps = ["InteractiveUtils", "JuliaSyntaxHighlighting", "Markdown", "Sockets", "StyledStrings", "Unicode"] +uuid = "3fa0cd96-eef1-5676-8a61-b3b8758bbffb" +version = "1.11.0" + +[[deps.Random]] +deps = ["SHA"] +uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" +version = "1.11.0" + +[[deps.RangeArrays]] +git-tree-sha1 = "b9039e93773ddcfc828f12aadf7115b4b4d225f5" +uuid = "b3c3ace0-ae52-54e7-9d0b-2c1406fd6b9d" +version = "0.3.2" + +[[deps.Ratios]] +deps = ["Requires"] +git-tree-sha1 = "1342a47bf3260ee108163042310d26f2be5ec90b" +uuid = "c84ed2f1-dad5-54f0-aa8e-dbefe2724439" +version = "0.4.5" +weakdeps = ["FixedPointNumbers"] + + [deps.Ratios.extensions] + RatiosFixedPointNumbersExt = "FixedPointNumbers" + +[[deps.Reexport]] +git-tree-sha1 = "45e428421666073eab6f2da5c9d310d99bb12f9b" +uuid = "189a3867-3050-52da-a836-e630ba90ab69" +version = "1.2.2" + +[[deps.RelocatableFolders]] +deps = ["SHA", "Scratch"] +git-tree-sha1 = "ffdaf70d81cf6ff22c2b6e733c900c3321cab864" +uuid = "05181044-ff0b-4ac5-8273-598c1e38db00" +version = "1.0.1" + +[[deps.Requires]] +deps = ["UUIDs"] +git-tree-sha1 = "62389eeff14780bfe55195b7204c0d8738436d64" +uuid = "ae029012-a4dd-5104-9daa-d747884805df" +version = "1.3.1" + +[[deps.Rmath]] +deps = ["Random", "Rmath_jll"] +git-tree-sha1 = "5b3d50eb374cea306873b371d3f8d3915a018f0b" +uuid = "79098fc4-a85e-5d69-aa6a-4863f24498fa" +version = "0.9.0" + +[[deps.Rmath_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "58cdd8fb2201a6267e1db87ff148dd6c1dbd8ad8" +uuid = "f50d1b31-88e8-58de-be2c-1cc44531875f" +version = "0.5.1+0" + +[[deps.Roots]] +deps = ["Accessors", "CommonSolve", "Printf"] +git-tree-sha1 = "7fb25a964849d90a0446366cdefca822e0e84900" +uuid = "f2b01f46-fcfa-551c-844a-d8ac1e96c665" +version = "3.0.6" + + [deps.Roots.extensions] + RootsChainRulesCoreExt = "ChainRulesCore" + RootsForwardDiffExt = "ForwardDiff" + RootsIntervalRootFindingExt = "IntervalRootFinding" + RootsSymPyExt = "SymPy" + RootsSymPyPythonCallExt = "SymPyPythonCall" + RootsUnitfulExt = "Unitful" + + [deps.Roots.weakdeps] + ChainRulesCore = "d360d2e6-b24c-11e9-a2a3-2a2ae2dbcce4" + ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210" + IntervalRootFinding = "d2bf35a9-74e0-55ec-b149-d360ff49b807" + SymPy = "24249f21-da20-56a4-8eb1-6a02cf4ae2e6" + SymPyPythonCall = "bc8888f7-b21e-4b7c-a06a-5d9c9496438c" + Unitful = "1986cc42-f94f-5a68-af5c-568840ba703d" + +[[deps.RoundingEmulator]] +git-tree-sha1 = "40b9edad2e5287e05bd413a38f61a8ff55b9557b" +uuid = "5eaf0fd0-dfba-4ccb-bf02-d820a40db705" +version = "0.2.1" + +[[deps.SHA]] +uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce" +version = "0.7.0" + +[[deps.SIMD]] +deps = ["PrecompileTools"] +git-tree-sha1 = "e24dc23107d426a096d3eae6c165b921e74c18e4" +uuid = "fdea26ae-647d-5447-a871-4b548cad5224" +version = "3.7.2" + +[[deps.ScopedValues]] +deps = ["HashArrayMappedTries", "Logging"] +git-tree-sha1 = "67a144433c4ce877ee6d1ada69a124d6b1ecf7be" +uuid = "7e506255-f358-4e82-b7e4-beb19740aa63" +version = "1.6.2" + +[[deps.Scratch]] +deps = ["Dates"] +git-tree-sha1 = "9b81b8393e50b7d4e6d0a9f14e192294d3b7c109" +uuid = "6c6a2e73-6563-6170-7368-637461726353" +version = "1.3.0" + +[[deps.Serialization]] +uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b" +version = "1.11.0" + +[[deps.ShaderAbstractions]] +deps = ["ColorTypes", "FixedPointNumbers", "GeometryBasics", "LinearAlgebra", "Observables", "StaticArrays"] +git-tree-sha1 = "818554664a2e01fc3784becb2eb3a82326a604b6" +uuid = "65257c39-d410-5151-9873-9b3e5be5013e" +version = "0.5.0" + +[[deps.SharedArrays]] +deps = ["Distributed", "Mmap", "Random", "Serialization"] +uuid = "1a1011a3-84de-559e-8e89-a11a2f7dc383" +version = "1.11.0" + +[[deps.SignedDistanceFields]] +deps = ["Statistics"] +git-tree-sha1 = "3949ad92e1c9d2ff0cd4a1317d5ecbba682f4b92" +uuid = "73760f76-fbc4-59ce-8f25-708e95d2df96" +version = "0.4.1" + +[[deps.SimpleTraits]] +deps = ["InteractiveUtils", "MacroTools"] +git-tree-sha1 = "7ddb0b49c109481b046972c0e4ab02b2127d6a75" +uuid = "699a6c99-e7fa-54fc-8d76-47d257e15c1d" +version = "0.9.6" + +[[deps.Sixel]] +deps = ["Dates", "FileIO", "ImageCore", "IndirectArrays", "OffsetArrays", "REPL", "libsixel_jll"] +git-tree-sha1 = "0494aed9501e7fb65daba895fb7fd57cc38bc743" +uuid = "45858cf5-a6b0-47a3-bbea-62219f50df47" +version = "0.1.5" + +[[deps.Sockets]] +uuid = "6462fe0b-24de-5631-8697-dd941f90decc" +version = "1.11.0" + +[[deps.SortingAlgorithms]] +deps = ["DataStructures"] +git-tree-sha1 = "13cd91cc9be159e3f4d95b857fa2aa383b53772a" +uuid = "a2af1166-a08f-5f64-846c-94a0d3cef48c" +version = "1.2.3" + +[[deps.SparseArrays]] +deps = ["Libdl", "LinearAlgebra", "Random", "Serialization", "SuiteSparse_jll"] +uuid = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" +version = "1.12.0" + +[[deps.SpecialFunctions]] +deps = ["IrrationalConstants", "LogExpFunctions", "OpenLibm_jll", "OpenSpecFun_jll"] +git-tree-sha1 = "6547cbdd8ce32efba0d21c5a40fa96d1a3548f9f" +uuid = "276daf66-3868-5448-9aa4-cd146d93841b" +version = "2.8.0" +weakdeps = ["ChainRulesCore"] + + [deps.SpecialFunctions.extensions] + SpecialFunctionsChainRulesCoreExt = "ChainRulesCore" + +[[deps.StableRNGs]] +deps = ["Random"] +git-tree-sha1 = "4f96c596b8c8258cc7d3b19797854d368f243ddc" +uuid = "860ef19b-820b-49d6-a774-d7a799459cd3" +version = "1.0.4" + +[[deps.StackViews]] +deps = ["OffsetArrays"] +git-tree-sha1 = "be1cf4eb0ac528d96f5115b4ed80c26a8d8ae621" +uuid = "cae243ae-269e-4f55-b966-ac2d0dc13c15" +version = "0.1.2" + +[[deps.StaticArrays]] +deps = ["LinearAlgebra", "PrecompileTools", "Random", "StaticArraysCore"] +git-tree-sha1 = "246a8bb2e6667f832eea063c3a56aef96429a3db" +uuid = "90137ffa-7385-5640-81b9-e52037218182" +version = "1.9.18" +weakdeps = ["ChainRulesCore", "Statistics"] + + [deps.StaticArrays.extensions] + StaticArraysChainRulesCoreExt = "ChainRulesCore" + StaticArraysStatisticsExt = "Statistics" + +[[deps.StaticArraysCore]] +git-tree-sha1 = "6ab403037779dae8c514bad259f32a447262455a" +uuid = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" +version = "1.4.4" + +[[deps.Statistics]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "ae3bb1eb3bba077cd276bc5cfc337cc65c3075c0" +uuid = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" +version = "1.11.1" +weakdeps = ["SparseArrays"] + + [deps.Statistics.extensions] + SparseArraysExt = ["SparseArrays"] + +[[deps.StatsAPI]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "178ed29fd5b2a2cfc3bd31c13375ae925623ff36" +uuid = "82ae8749-77ed-4fe6-ae5f-f523153014b0" +version = "1.8.0" + +[[deps.StatsBase]] +deps = ["AliasTables", "DataAPI", "DataStructures", "IrrationalConstants", "LinearAlgebra", "LogExpFunctions", "Missings", "Printf", "Random", "SortingAlgorithms", "SparseArrays", "Statistics", "StatsAPI"] +git-tree-sha1 = "e4d7a1a0edc20af42689ea6f4f3587a2175d50ee" +uuid = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91" +version = "0.34.12" + +[[deps.StatsFuns]] +deps = ["HypergeometricFunctions", "IrrationalConstants", "LogExpFunctions", "Reexport", "Rmath", "SpecialFunctions"] +git-tree-sha1 = "770240df9a3b8888065046948f7a09b4e0f997d5" +uuid = "4c63d2b9-4356-54db-8cca-17b64c39e42c" +version = "2.2.0" +weakdeps = ["ChainRulesCore", "InverseFunctions"] + + [deps.StatsFuns.extensions] + StatsFunsChainRulesCoreExt = "ChainRulesCore" + StatsFunsInverseFunctionsExt = "InverseFunctions" + +[[deps.StructArrays]] +deps = ["ConstructionBase", "DataAPI", "Tables"] +git-tree-sha1 = "ad8002667372439f2e3611cfd14097e03fa4bccd" +uuid = "09ab397b-f2b6-538f-b94a-2f83cf4a842a" +version = "0.7.3" + + [deps.StructArrays.extensions] + StructArraysAdaptExt = "Adapt" + StructArraysGPUArraysCoreExt = ["GPUArraysCore", "KernelAbstractions"] + StructArraysLinearAlgebraExt = "LinearAlgebra" + StructArraysSparseArraysExt = "SparseArrays" + StructArraysStaticArraysExt = "StaticArrays" + + [deps.StructArrays.weakdeps] + Adapt = "79e6a3ab-5dfb-504d-930d-738a2a938a0e" + GPUArraysCore = "46192b85-c4d5-4398-a991-12ede77f4527" + KernelAbstractions = "63c18a36-062a-441e-b654-da1e3ab1ce7c" + LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" + SparseArrays = "2f01184e-e22b-5df5-ae63-d93ebab69eaf" + StaticArrays = "90137ffa-7385-5640-81b9-e52037218182" + +[[deps.StructUtils]] +deps = ["Dates", "UUIDs"] +git-tree-sha1 = "82bee338d650aa515f31866c460cb7e3bcef90b8" +uuid = "ec057cc2-7a8d-4b58-b3b3-92acb9f63b42" +version = "2.8.2" + + [deps.StructUtils.extensions] + StructUtilsMeasurementsExt = ["Measurements"] + StructUtilsStaticArraysCoreExt = ["StaticArraysCore"] + StructUtilsTablesExt = ["Tables"] + + [deps.StructUtils.weakdeps] + Measurements = "eff96d63-e80a-5855-80a2-b1b0885c5ab7" + StaticArraysCore = "1e83bf80-4336-4d27-bf5d-d5a4f845583c" + Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" + +[[deps.StyledStrings]] +uuid = "f489334b-da3d-4c2e-b8f0-e476e12c162b" +version = "1.11.0" + +[[deps.SuiteSparse]] +deps = ["Libdl", "LinearAlgebra", "Serialization", "SparseArrays"] +uuid = "4607b0f0-06f3-5cda-b6b1-a6196a1729e9" + +[[deps.SuiteSparse_jll]] +deps = ["Artifacts", "Libdl", "libblastrampoline_jll"] +uuid = "bea87d4a-7f5b-5778-9afe-8cc45184846c" +version = "7.8.3+2" + +[[deps.TOML]] +deps = ["Dates"] +uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76" +version = "1.0.3" + +[[deps.TableTraits]] +deps = ["IteratorInterfaceExtensions"] +git-tree-sha1 = "c06b2f539df1c6efa794486abfb6ed2022561a39" +uuid = "3783bdb8-4a98-5b6b-af9a-565f29a5fe9c" +version = "1.0.1" + +[[deps.Tables]] +deps = ["DataAPI", "DataValueInterfaces", "IteratorInterfaceExtensions", "OrderedCollections", "TableTraits"] +git-tree-sha1 = "0f38a06c83f0007bbab3cf911262841c9a0f07e0" +uuid = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" +version = "1.13.0" + +[[deps.Tar]] +deps = ["ArgTools", "SHA"] +uuid = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e" +version = "1.10.0" + +[[deps.TensorCore]] +deps = ["LinearAlgebra"] +git-tree-sha1 = "1feb45f88d133a655e001435632f019a9a1bcdb6" +uuid = "62fd8b95-f654-4bbd-a8a5-9c27f68ccd50" +version = "0.1.1" + +[[deps.Test]] +deps = ["InteractiveUtils", "Logging", "Random", "Serialization"] +uuid = "8dfed614-e22c-5e08-85e1-65c5234f0b40" +version = "1.11.0" + +[[deps.TiffImages]] +deps = ["CodecZstd", "ColorTypes", "DataStructures", "DocStringExtensions", "FileIO", "FixedPointNumbers", "IndirectArrays", "Inflate", "Mmap", "OffsetArrays", "PkgVersion", "PrecompileTools", "ProgressMeter", "SIMD", "UUIDs"] +git-tree-sha1 = "9ca5f1f2d42f80df4b8c9f6ab5a64f438bbd9976" +uuid = "731e570b-9d59-4bfa-96dc-6df516fadf69" +version = "0.11.9" + +[[deps.TranscodingStreams]] +git-tree-sha1 = "0c45878dcfdcfa8480052b6ab162cdd138781742" +uuid = "3bb67fe8-82b1-5028-8e26-92a6c54297fa" +version = "0.11.3" + +[[deps.TriplotBase]] +git-tree-sha1 = "4d4ed7f294cda19382ff7de4c137d24d16adc89b" +uuid = "981d1d27-644d-49a2-9326-4793e63143c3" +version = "0.1.0" + +[[deps.UUIDs]] +deps = ["Random", "SHA"] +uuid = "cf7118a7-6976-5b1a-9a39-7adc72f591a4" +version = "1.11.0" + +[[deps.Unicode]] +uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5" +version = "1.11.0" + +[[deps.UnicodeFun]] +deps = ["REPL"] +git-tree-sha1 = "53915e50200959667e78a92a418594b428dffddf" +uuid = "1cfade01-22cf-5700-b092-accc4b62d6e1" +version = "0.4.1" + +[[deps.Unitful]] +deps = ["Dates", "LinearAlgebra", "Random"] +git-tree-sha1 = "57e1b2c9de4bd6f40ecb9de4ac1797b81970d008" +uuid = "1986cc42-f94f-5a68-af5c-568840ba703d" +version = "1.28.0" + + [deps.Unitful.extensions] + ConstructionBaseUnitfulExt = "ConstructionBase" + ForwardDiffExt = "ForwardDiff" + InverseFunctionsUnitfulExt = "InverseFunctions" + LatexifyExt = ["Latexify", "LaTeXStrings"] + NaNMathExt = "NaNMath" + PrintfExt = "Printf" + + [deps.Unitful.weakdeps] + ConstructionBase = "187b0558-2788-49d3-abe0-74a17ed4e7c9" + ForwardDiff = "f6369f11-7733-5829-9624-2563aa707210" + InverseFunctions = "3587e190-3f89-42d0-90ee-14403ec27112" + LaTeXStrings = "b964fa9f-0449-5b57-a5c2-d3ea65f4040f" + Latexify = "23fbe1c1-3f47-55db-b15f-69d7ec21a316" + NaNMath = "77ba4419-2d1f-58cd-9bb1-8ffee604a2e3" + Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7" + +[[deps.WebP]] +deps = ["CEnum", "ColorTypes", "FileIO", "FixedPointNumbers", "ImageCore", "libwebp_jll"] +git-tree-sha1 = "aa1ca3c47f119fbdae8770c29820e5e6119b83f2" +uuid = "e3aaa7dc-3e4b-44e0-be63-ffb868ccd7c1" +version = "0.1.3" + +[[deps.WoodburyMatrices]] +deps = ["LinearAlgebra", "SparseArrays"] +git-tree-sha1 = "248a7031b3da79a127f14e5dc5f417e26f9f6db7" +uuid = "efce3f68-66dc-5838-9240-27a6d6f5f9b6" +version = "1.1.0" + +[[deps.XZ_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "b29c22e245d092b8b4e8d3c09ad7baa586d9f573" +uuid = "ffd25f8a-64ca-5728-b0f7-c24cf3aae800" +version = "5.8.3+0" + +[[deps.Xorg_libX11_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libxcb_jll", "Xorg_xtrans_jll"] +git-tree-sha1 = "808090ede1d41644447dd5cbafced4731c56bd2f" +uuid = "4f6342f7-b3d2-589e-9d20-edeb45f2b2bc" +version = "1.8.13+0" + +[[deps.Xorg_libXau_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "aa1261ebbac3ccc8d16558ae6799524c450ed16b" +uuid = "0c0b7dd1-d40b-584c-a123-a41640f87eec" +version = "1.0.13+0" + +[[deps.Xorg_libXdmcp_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "52858d64353db33a56e13c341d7bf44cd0d7b309" +uuid = "a3789734-cfe1-5b06-b2d0-1dd0d9d62d05" +version = "1.1.6+0" + +[[deps.Xorg_libXext_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "1a4a26870bf1e5d26cd585e38038d399d7e65706" +uuid = "1082639a-0dae-5f34-9b06-72781eeb8cb3" +version = "1.3.8+0" + +[[deps.Xorg_libXfixes_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "75e00946e43621e09d431d9b95818ee751e6b2ef" +uuid = "d091e8ba-531a-589c-9de9-94069b037ed8" +version = "6.0.2+0" + +[[deps.Xorg_libXrender_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll"] +git-tree-sha1 = "7ed9347888fac59a618302ee38216dd0379c480d" +uuid = "ea2f1a96-1ddc-540d-b46f-429655e07cfa" +version = "0.9.12+0" + +[[deps.Xorg_libpciaccess_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "58972370b81423fc546c56a60ed1a009450177c3" +uuid = "a65dc6b1-eb27-53a1-bb3e-dea574b5389e" +version = "0.19.0+0" + +[[deps.Xorg_libxcb_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libXau_jll", "Xorg_libXdmcp_jll"] +git-tree-sha1 = "bfcaf7ec088eaba362093393fe11aa141fa15422" +uuid = "c7cfdc94-dc32-55de-ac96-5a1b8d977c5b" +version = "1.17.1+0" + +[[deps.Xorg_xtrans_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "a63799ff68005991f9d9491b6e95bd3478d783cb" +uuid = "c5fb5394-a638-5e4d-96e5-b29de1b5cf10" +version = "1.6.0+0" + +[[deps.Zlib_jll]] +deps = ["Libdl"] +uuid = "83775a58-1f1d-513f-b197-d71354ab007a" +version = "1.3.1+2" + +[[deps.Zstd_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "446b23e73536f84e8037f5dce465e92275f6a308" +uuid = "3161d3a3-bdf6-5164-811a-617609db77b4" +version = "1.5.7+1" + +[[deps.isoband_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Pkg"] +git-tree-sha1 = "51b5eeb3f98367157a7a12a1fb0aa5328946c03c" +uuid = "9a68df92-36a6-505f-a73e-abb412b6bfb4" +version = "0.2.3+0" + +[[deps.libaom_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "850b06095ee71f0135d644ffd8a52850699581ed" +uuid = "a4ae2306-e953-59d6-aa16-d00cac43593b" +version = "3.13.3+0" + +[[deps.libass_jll]] +deps = ["Artifacts", "Bzip2_jll", "FreeType2_jll", "FriBidi_jll", "HarfBuzz_jll", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "125eedcb0a4a0bba65b657251ce1d27c8714e9d6" +uuid = "0ac62f75-1d6f-5e53-bd7c-93b484bb37c0" +version = "0.17.4+0" + +[[deps.libblastrampoline_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850b90-86db-534c-a0d3-1478176c7d93" +version = "5.15.0+0" + +[[deps.libdrm_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libpciaccess_jll"] +git-tree-sha1 = "28e57478e8a160d346a19c28b3fffb9273bcc9c2" +uuid = "8e53e030-5e6c-5a89-a30b-be5b7263a166" +version = "2.4.134+0" + +[[deps.libfdk_aac_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "646634dd19587a56ee2f1199563ec056c5f228df" +uuid = "f638f0a6-7fb0-5443-88ba-1cc74229b280" +version = "2.0.4+0" + +[[deps.libpng_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Zlib_jll"] +git-tree-sha1 = "e51150d5ab85cee6fc36726850f0e627ad2e4aba" +uuid = "b53b4c65-9356-5827-b1ea-8c7a1a84506f" +version = "1.6.58+0" + +[[deps.libsixel_jll]] +deps = ["Artifacts", "JLLWrappers", "JpegTurbo_jll", "Libdl", "libpng_jll"] +git-tree-sha1 = "c1733e347283df07689d71d61e14be986e49e47a" +uuid = "075b6546-f08a-558a-be8f-8157d0f608a5" +version = "1.10.5+0" + +[[deps.libva_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Xorg_libX11_jll", "Xorg_libXext_jll", "Xorg_libXfixes_jll", "libdrm_jll"] +git-tree-sha1 = "7dbf96baae3310fe2fa0df0ccbb3c6288d5816c9" +uuid = "9a156e7d-b971-5f62-b2c9-67348b8fb97c" +version = "2.23.0+0" + +[[deps.libvorbis_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl", "Ogg_jll"] +git-tree-sha1 = "11e1772e7f3cc987e9d3de991dd4f6b2602663a5" +uuid = "f27f6e37-5d2b-51aa-960f-b287f2bc3b7a" +version = "1.3.8+0" + +[[deps.libwebp_jll]] +deps = ["Artifacts", "Giflib_jll", "JLLWrappers", "JpegTurbo_jll", "Libdl", "Libglvnd_jll", "Libtiff_jll", "libpng_jll"] +git-tree-sha1 = "4e4282c4d846e11dce56d74fa8040130b7a95cb3" +uuid = "c5f90fcd-3b7e-5836-afba-fc50a0988cb2" +version = "1.6.0+0" + +[[deps.nghttp2_jll]] +deps = ["Artifacts", "Libdl"] +uuid = "8e850ede-7688-5339-a07c-302acd2aaf8d" +version = "1.64.0+1" + +[[deps.p7zip_jll]] +deps = ["Artifacts", "CompilerSupportLibraries_jll", "Libdl"] +uuid = "3f19e933-33d8-53b3-aaab-bd5110c3b7a0" +version = "17.7.0+0" + +[[deps.x264_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "14cc7083fc6dff3cc44f2bc435ee96d06ed79aa7" +uuid = "1270edf5-f2f9-52d2-97e9-ab00b5d0237a" +version = "10164.0.1+0" + +[[deps.x265_jll]] +deps = ["Artifacts", "JLLWrappers", "Libdl"] +git-tree-sha1 = "e7b67590c14d487e734dcb925924c5dc43ec85f3" +uuid = "dfaa095f-4041-5dcd-9319-2fabd8486b76" +version = "4.1.0+0" diff --git a/bench/Project.toml b/bench/Project.toml index e00dde7..d8d5815 100644 --- a/bench/Project.toml +++ b/bench/Project.toml @@ -6,3 +6,15 @@ JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819" Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76" +Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" + +[compat] +BrainlessLab = "0.1.1" +CairoMakie = "0.12, 0.13, 0.14, 0.15" +Dates = "1.10" +JLD2 = "0.5" +Random = "1.10" +Statistics = "1.10" +TOML = "1" +Test = "1.10" +julia = "1.10" diff --git a/bench/configs/smoke.toml b/bench/configs/smoke.toml index 2caf900..6b454f6 100644 --- a/bench/configs/smoke.toml +++ b/bench/configs/smoke.toml @@ -1,12 +1,12 @@ -neurons = ["falandays", "falandays_ablated", "compartmental_structured"] -tasks = ["wall", "tracking"] +neurons = ["falandays"] +tasks = ["tracking"] -n_trials = 5 -n_nodes = 60 -ticks = 150 +n_trials = 1 +n_nodes = 12 +ticks = 20 seed_base = 1000 baseline = "falandays" alpha = 0.05 -gifs = true +gifs = false [prep] diff --git a/configs/ci_sweep.toml b/configs/ci_sweep.toml new file mode 100644 index 0000000..60a98bd --- /dev/null +++ b/configs/ci_sweep.toml @@ -0,0 +1,20 @@ +[sweep] +id = "ci_sweep_smoke" +mode = "one_at_a_time" +seeds = [0] +max_cells = 1 +max_rollouts = 1 +threaded = false + +[baseline] +node = "falandays" +task = "tracking" +N = 12 +ticks = 20 +window = 20 + +[axes] +"node.leak" = [0.25] + +[analytics] +measures = ["liveness"] diff --git a/examples/templates/new_project/Project.toml b/examples/templates/new_project/Project.toml index 15b757c..292611a 100644 --- a/examples/templates/new_project/Project.toml +++ b/examples/templates/new_project/Project.toml @@ -6,7 +6,10 @@ version = "0.1.0" [deps] BrainlessLab = "d12add44-1e3e-4161-9a99-c2121a2f0f38" CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0" +Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" [compat] +BrainlessLab = "0.1.1" CairoMakie = "0.12, 0.13, 0.14, 0.15" +Random = "1.10" julia = "1.10" diff --git a/profile/Manifest.toml b/profile/Manifest.toml index 9a49021..50eafe6 100644 --- a/profile/Manifest.toml +++ b/profile/Manifest.toml @@ -2,7 +2,7 @@ julia_version = "1.12.6" manifest_format = "2.0" -project_hash = "9cf78c06b0aac5f1ae9e2713e0087acb7b5b5437" +project_hash = "c1e3b888cad7645bcf6748dd96b92f1d32d24207" [[deps.AbstractFFTs]] deps = ["LinearAlgebra"] @@ -111,10 +111,10 @@ uuid = "18cc8868-cbac-4acf-b575-c8ff214dc66f" version = "1.4.0" [[deps.BrainlessLab]] -deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "StaticArrays", "TOML"] +deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "StaticArrays", "Statistics", "TOML"] path = ".." uuid = "d12add44-1e3e-4161-9a99-c2121a2f0f38" -version = "0.0.1" +version = "0.1.1" weakdeps = ["Makie"] [deps.BrainlessLab.extensions] diff --git a/profile/Profile.jl b/profile/Profile.jl index f0ef90a..0347a32 100644 --- a/profile/Profile.jl +++ b/profile/Profile.jl @@ -527,7 +527,8 @@ end task_profile(node_sym, task; n_seeds=8, canonical_N=CANONICAL_N) Run `n_seeds` rollouts of `task` with `node_sym` at the task's canonical N, -recording `(:rate, :scene, :poses)`, over the task's default ticks. Returns a +recording task signals plus rate and scene channels over the task's default +ticks. Returns a NamedTuple with the seed-averaged branching-ratio series, mean/std sigma, mean/std score, the run parameters used (N, R, E, ticks), and `factor_data`: per task-scoped analysis registered for the task, the seed-1 σ(t) and @@ -561,7 +562,9 @@ function task_profile(node_sym::Symbol, task::Symbol; n_seeds::Integer=8, canoni seed1_target_error = nothing seed_results = BrainlessLab.parallel_map(1:Int(n_seeds)) do s - record_channels = s == 1 ? (:spikes, :rate, :scene, :poses, :acts, :targets) : (:spikes, :rate, :scene, :poses) + record_channels = s == 1 ? + (:spikes, :rate, :scene, :poses, :percepts, :acts, :targets) : + (:spikes, :rate, :scene, :poses) sim = simulate(task; node=node_sym, n_nodes=N, seed=s, record=record_channels) br = branching_ratio(sim) sigma_mr_value = try diff --git a/profile/Project.toml b/profile/Project.toml index 24d77e3..26fffe5 100644 --- a/profile/Project.toml +++ b/profile/Project.toml @@ -1,7 +1,20 @@ [deps] BrainlessLab = "d12add44-1e3e-4161-9a99-c2121a2f0f38" +Dates = "ade2ca70-3891-5945-98fb-dc099432e06a" CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0" +Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" Printf = "de0858da-6303-5e67-8744-51eddeeeb8d7" Base64 = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76" + +[compat] +Base64 = "1.10" +BrainlessLab = "0.1.1" +CairoMakie = "0.12, 0.13, 0.14, 0.15" +Dates = "1.10" +Printf = "1.10" +Random = "1.10" +Statistics = "1.10" +TOML = "1" +julia = "1.10" diff --git a/site/src/content/docs/core/getting-started.mdx b/site/src/content/docs/core/getting-started.mdx index 9aa9eaa..cb928bf 100644 --- a/site/src/content/docs/core/getting-started.mdx +++ b/site/src/content/docs/core/getting-started.mdx @@ -8,6 +8,9 @@ BrainlessLab studies neural reservoirs in closed sensorimotor loops. The canonic Start with the tracking task. Tracking has a clear observation, a clear action, and a continuous error signal. +The current release is an experimental research preview. BrainlessLab is not yet +registered in Julia General, and its APIs and artifact layouts may change before 1.0. + You can use the browser model for orientation. Use the Julia package for reproducible runs and recorded evidence. @@ -31,7 +34,8 @@ limitations. You do not need to name source files or Julia types. ### Run it locally Install Julia with the -[official Julia installer](https://julialang.org/install/). Then run: +[official Julia installer](https://julialang.org/install/). BrainlessLab requires Julia +1.10 or newer. Clone the repository and instantiate its pinned environment: ```bash git clone https://github.com/btgaskin/brainless-lab.git diff --git a/test/runtests.jl b/test/runtests.jl index 8e5049f..1cd368e 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -1,5 +1,10 @@ using BrainlessLab using Test +using Aqua + +@testset "Package quality" begin + Aqua.test_all(BrainlessLab) +end include("testutils.jl") include("test_components.jl") From 1748e5de356f112db1be83a5bc0bc69c4392a912 Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 22:49:39 -0400 Subject: [PATCH 04/20] feat: add typed specification kernel --- src/core/Specifications.jl | 538 +++++++++++++++++++++++++++++++++++ test/test_contract_kernel.jl | 186 ++++++++++++ 2 files changed, 724 insertions(+) create mode 100644 src/core/Specifications.jl create mode 100644 test/test_contract_kernel.jl diff --git a/src/core/Specifications.jl b/src/core/Specifications.jl new file mode 100644 index 0000000..ffbf7c8 --- /dev/null +++ b/src/core/Specifications.jl @@ -0,0 +1,538 @@ +const IMPLEMENTATION_STABILITIES = (:reference, :stable, :experimental, :control) +const CONSTRUCTION_SCOPES = (:evaluation, :block, :trial) +const RESET_POLICIES = (:full, :body_environment, :none) +const AGGREGATE_POLICIES = (:none, :mean, :median, :sum, :minimum, :maximum) +const EVOLUTION_SCALES = (:linear, :log, :integer) + +function _nonempty_symbol(value, label::AbstractString) + symbol = Symbol(value) + isempty(String(symbol)) && throw(ArgumentError("$(label) must not be empty")) + return symbol +end + +function _symbol_tuple(values, label::AbstractString) + source = values isa Union{Symbol,AbstractString} ? (values,) : values + result = Tuple(_nonempty_symbol(value, label) for value in source) + length(unique(result)) == length(result) || + throw(ArgumentError("$(label) must be unique")) + return result +end + +""" + Registry{K,V}(name=:registry) + +A small typed registry for resolved component descriptors. Registration rejects +duplicate keys; replacement is deliberately a separate concern so accidental +load-order changes cannot silently alter an experiment. +""" +struct Registry{K,V} + name::Symbol + entries::Dict{K,V} + + function Registry{K,V}(name::Union{Symbol,AbstractString}=:registry) where {K,V} + name_ = _nonempty_symbol(name, "registry name") + return new{K,V}(name_, Dict{K,V}()) + end +end + +Base.length(registry::Registry) = length(registry.entries) +Base.isempty(registry::Registry) = isempty(registry.entries) +Base.haskey(registry::Registry, key) = haskey(registry.entries, key) +Base.keys(registry::Registry) = keys(registry.entries) +Base.values(registry::Registry) = values(registry.entries) +Base.iterate(registry::Registry, state...) = iterate(registry.entries, state...) + +""" + register!(registry, key, value) + +Register `value` under `key`. Duplicate keys are always rejected. +""" +function register!(registry::Registry{K,V}, key::K, value::V) where {K,V} + haskey(registry.entries, key) && throw(ArgumentError( + "$(registry.name) registry key $(repr(key)) is already registered", + )) + registry.entries[key] = value + return value +end + +""" + resolve(registry, key) + +Resolve a registered key, reporting the sorted known keys when resolution +fails. +""" +function resolve(registry::Registry{K}, key::K) where {K} + haskey(registry.entries, key) && return registry.entries[key] + known = sort!(collect(keys(registry.entries)); by=string) + known_message = isempty(known) ? "none registered" : join(repr.(known), ", ") + throw(KeyError( + "Unknown $(registry.name) registry key $(repr(key)). Known keys: $(known_message).", + )) +end + +Base.getindex(registry::Registry{K}, key::K) where {K} = resolve(registry, key) + +""" + ImplementationSpec(key, implementation; kwargs...) + +Generic discovery metadata for a registered implementation. This descriptor +does not assume that the implementation is callable: tasks, bodies, analyses, +and immutable specifications can all be registered through the same contract. +""" +struct ImplementationSpec{I,T<:Tuple,C<:Tuple,M<:NamedTuple} + key::Symbol + implementation::I + label::String + description::String + origin::String + stability::Symbol + tags::T + capabilities::C + metadata::M +end + +function ImplementationSpec( + key::Union{Symbol,AbstractString}, + implementation; + label::AbstractString=string(key), + description::AbstractString="", + origin::AbstractString="BrainlessLab", + stability::Symbol=:experimental, + tags=(), + capabilities=(), + metadata::NamedTuple=NamedTuple(), +) + key_ = _nonempty_symbol(key, "implementation key") + isempty(strip(label)) && throw(ArgumentError("implementation label must not be empty")) + isempty(strip(origin)) && throw(ArgumentError("implementation origin must not be empty")) + stability in IMPLEMENTATION_STABILITIES || throw(ArgumentError( + "implementation stability must be one of " * + join(":" .* string.(IMPLEMENTATION_STABILITIES), ", "), + )) + tags_ = _symbol_tuple(tags, "implementation tags") + capabilities_ = _symbol_tuple(capabilities, "implementation capabilities") + return ImplementationSpec{ + typeof(implementation), + typeof(tags_), + typeof(capabilities_), + typeof(metadata), + }( + key_, + implementation, + String(label), + String(description), + String(origin), + stability, + tags_, + capabilities_, + metadata, + ) +end + +""" + EquationSpec(name, latex; kwargs...) + +Human-readable mathematical metadata suitable for generated reports. Variable +definitions are stored as unique `Symbol => description` pairs; references are +plain citation identifiers or URLs. +""" +struct EquationSpec{V<:Tuple,R<:Tuple} + name::Symbol + title::String + latex::String + description::String + variables::V + references::R +end + +function _equation_variables(variables) + source = variables isa Pair ? (variables,) : variables + result = Tuple(begin + variable isa Pair || throw(ArgumentError( + "equation variables must be pairs of symbol => description", + )) + name = _nonempty_symbol(first(variable), "equation variable") + description = String(last(variable)) + isempty(strip(description)) && throw(ArgumentError( + "equation variable descriptions must not be empty", + )) + name => description + end for variable in source) + names = first.(result) + length(unique(names)) == length(names) || + throw(ArgumentError("equation variables must be unique")) + return result +end + +function _equation_references(references) + source = references isa AbstractString ? (references,) : references + result = Tuple(String(reference) for reference in source) + all(reference -> !isempty(strip(reference)), result) || + throw(ArgumentError("equation references must not be empty")) + length(unique(result)) == length(result) || + throw(ArgumentError("equation references must be unique")) + return result +end + +function EquationSpec( + name::Union{Symbol,AbstractString}, + latex::AbstractString; + title::AbstractString=string(name), + description::AbstractString="", + variables=(), + references=(), +) + name_ = _nonempty_symbol(name, "equation name") + isempty(strip(title)) && throw(ArgumentError("equation title must not be empty")) + isempty(strip(latex)) && throw(ArgumentError("equation LaTeX must not be empty")) + variables_ = _equation_variables(variables) + references_ = _equation_references(references) + return EquationSpec{typeof(variables_),typeof(references_)}( + name_, + String(title), + String(latex), + String(description), + variables_, + references_, + ) +end + +function _parameter_value_valid(validator, value, name::Symbol) + validator === nothing && return value + applicable(validator, value) || throw(ArgumentError( + "validator for parameter :$(name) is not callable with $(typeof(value))", + )) + verdict = validator(value) + verdict isa Bool || throw(ArgumentError( + "validator for parameter :$(name) must return Bool, got $(typeof(verdict))", + )) + verdict || throw(ArgumentError( + "invalid value $(repr(value)) for parameter :$(name)", + )) + return value +end + +function _parameter_sweep(sweep, validator, name::Symbol) + sweep === nothing && return nothing + sweep isa Union{AbstractString,Symbol,Number} && throw(ArgumentError( + "sweep metadata for parameter :$(name) must be an iterable of candidate values", + )) + values = Tuple(sweep) + isempty(values) && throw(ArgumentError( + "sweep metadata for parameter :$(name) must not be empty", + )) + length(unique(values)) == length(values) || throw(ArgumentError( + "sweep metadata for parameter :$(name) must contain unique values", + )) + foreach(value -> _parameter_value_valid(validator, value, name), values) + return values +end + +function _categorical_evolution(evolve::NamedTuple, validator, name::Symbol) + propertynames(evolve) == (:values,) || throw(ArgumentError( + "categorical evolution metadata for parameter :$(name) must contain only :values", + )) + values = _parameter_sweep(evolve.values, validator, name) + values === nothing && throw(ArgumentError( + "categorical evolution metadata for parameter :$(name) requires candidate values", + )) + return (values=values,) +end + +function _bounded_evolution(evolve::NamedTuple, validator, name::Symbol, default) + allowed = (:lower, :upper, :scale, :mutation_scale) + unknown = setdiff(propertynames(evolve), allowed) + isempty(unknown) || throw(ArgumentError( + "unknown evolution metadata for parameter :$(name): " * + join(":" .* string.(unknown), ", "), + )) + hasproperty(evolve, :lower) && hasproperty(evolve, :upper) || throw(ArgumentError( + "bounded evolution metadata for parameter :$(name) requires :lower and :upper", + )) + lower = evolve.lower + upper = evolve.upper + lower isa Real && upper isa Real && default isa Real || throw(ArgumentError( + "bounded evolution metadata for parameter :$(name) requires numeric bounds and default", + )) + isfinite(lower) && isfinite(upper) || throw(ArgumentError( + "evolution bounds for parameter :$(name) must be finite", + )) + lower <= upper || throw(ArgumentError( + "evolution lower bound for parameter :$(name) exceeds its upper bound", + )) + lower <= default <= upper || throw(ArgumentError( + "default for parameter :$(name) lies outside its evolution bounds", + )) + _parameter_value_valid(validator, lower, name) + _parameter_value_valid(validator, upper, name) + + scale = hasproperty(evolve, :scale) ? Symbol(evolve.scale) : :linear + scale in EVOLUTION_SCALES || throw(ArgumentError( + "evolution scale for parameter :$(name) must be one of " * + join(":" .* string.(EVOLUTION_SCALES), ", "), + )) + if scale === :log + lower > zero(lower) || throw(ArgumentError( + "log-scaled evolution for parameter :$(name) requires a positive lower bound", + )) + elseif scale === :integer + all(value -> value isa Integer, (lower, default, upper)) || throw(ArgumentError( + "integer-scaled evolution for parameter :$(name) requires integer bounds and default", + )) + end + + mutation_scale = hasproperty(evolve, :mutation_scale) ? evolve.mutation_scale : nothing + if mutation_scale !== nothing + mutation_scale isa Real && isfinite(mutation_scale) && mutation_scale > 0 || + throw(ArgumentError( + "evolution mutation_scale for parameter :$(name) must be finite and positive", + )) + end + return ( + lower=lower, + upper=upper, + scale=scale, + mutation_scale=mutation_scale, + ) +end + +function _parameter_evolution(evolve, validator, name::Symbol, default) + evolve === nothing && return nothing + evolve isa NamedTuple || throw(ArgumentError( + "evolution metadata for parameter :$(name) must be a NamedTuple", + )) + hasproperty(evolve, :values) && + return _categorical_evolution(evolve, validator, name) + return _bounded_evolution(evolve, validator, name, default) +end + +""" + ParameterSpec(name, default; kwargs...) + +One configurable parameter and its cold-path research metadata. `owner` +identifies the component level that interprets the value; node count therefore +need not be owned by a node model. `sweep` is a finite candidate set. `evolve` +is either `(values=(...),)` or bounded metadata with `lower`, `upper`, and +optional `scale`/`mutation_scale`. +""" +struct ParameterSpec{T,V,S,E} + name::Symbol + owner::Symbol + default::T + validator::V + sweep::S + evolve::E + description::String + units::Union{Nothing,String} +end + +function ParameterSpec( + name::Union{Symbol,AbstractString}, + default; + owner::Union{Symbol,AbstractString}=:node, + validator=nothing, + sweep=nothing, + evolve=nothing, + description::AbstractString="", + units::Union{Nothing,AbstractString}=nothing, +) + name_ = _nonempty_symbol(name, "parameter name") + owner_ = _nonempty_symbol(owner, "parameter owner") + units_ = units === nothing ? nothing : String(units) + units_ !== nothing && isempty(strip(units_)) && + throw(ArgumentError("parameter units must not be empty")) + _parameter_value_valid(validator, default, name_) + sweep_ = _parameter_sweep(sweep, validator, name_) + evolve_ = _parameter_evolution(evolve, validator, name_, default) + return ParameterSpec{ + typeof(default), + typeof(validator), + typeof(sweep_), + typeof(evolve_), + }( + name_, + owner_, + default, + validator, + sweep_, + evolve_, + String(description), + units_, + ) +end + +"""Validate and return a candidate value for a parameter.""" +validate_parameter(spec::ParameterSpec, value) = + _parameter_value_valid(spec.validator, value, spec.name) + +sweepable(spec::ParameterSpec) = spec.sweep !== nothing +evolvable(spec::ParameterSpec) = spec.evolve !== nothing + +""" + SeedStreamSpec(name; description="") + +A declared independent random stream. Stream names become stable inputs to seed +derivation and must therefore be treated as part of an evaluation protocol. +""" +struct SeedStreamSpec + name::Symbol + description::String + + function SeedStreamSpec( + name::Union{Symbol,AbstractString}; + description::AbstractString="", + ) + name_ = _nonempty_symbol(name, "seed stream name") + return new(name_, String(description)) + end +end + +const DEFAULT_SEED_STREAMS = ( + SeedStreamSpec(:environment), + SeedStreamSpec(:node_construction), + SeedStreamSpec(:runtime), + SeedStreamSpec(:trial), + SeedStreamSpec(:optimizer), + SeedStreamSpec(:bootstrap), +) + +function _seed_streams(streams) + source = streams isa Union{Symbol,AbstractString,SeedStreamSpec} ? (streams,) : streams + result = Tuple( + stream isa SeedStreamSpec ? stream : SeedStreamSpec(stream) + for stream in source + ) + isempty(result) && throw(ArgumentError("evaluation must declare at least one seed stream")) + names = getfield.(result, :name) + length(unique(names)) == length(names) || + throw(ArgumentError("evaluation seed stream names must be unique")) + return result +end + +function _root_seed(seed::Integer) + seed >= 0 || throw(ArgumentError("evaluation root_seed must be non-negative")) + try + return UInt64(seed) + catch + throw(ArgumentError("evaluation root_seed must fit in UInt64")) + end +end + +""" + EvaluationSpec(; kwargs...) + +Outer replication protocol for a resolved composition. Construction scope, +reset policy, and aggregation are explicit so trials cannot silently share +state or change their inferential unit. +""" +struct EvaluationSpec{S<:Tuple} + blocks::Int + trials_per_block::Int + horizon::Int + warmup::Int + construction_scope::Symbol + reset::Symbol + root_seed::UInt64 + streams::S + aggregate::Symbol +end + +function EvaluationSpec(; + blocks::Integer=1, + trials_per_block::Integer=1, + horizon::Integer, + warmup::Integer=0, + construction_scope::Symbol=:trial, + reset::Symbol=:full, + root_seed::Integer=0, + streams=DEFAULT_SEED_STREAMS, + aggregate::Symbol=:mean, +) + blocks_ = Int(blocks) + trials_ = Int(trials_per_block) + horizon_ = Int(horizon) + warmup_ = Int(warmup) + blocks_ > 0 || throw(ArgumentError("evaluation blocks must be positive")) + trials_ > 0 || throw(ArgumentError("evaluation trials_per_block must be positive")) + horizon_ > 0 || throw(ArgumentError("evaluation horizon must be positive")) + warmup_ >= 0 || throw(ArgumentError("evaluation warmup must be non-negative")) + construction_scope in CONSTRUCTION_SCOPES || throw(ArgumentError( + "evaluation construction_scope must be one of " * + join(":" .* string.(CONSTRUCTION_SCOPES), ", "), + )) + reset in RESET_POLICIES || throw(ArgumentError( + "evaluation reset must be one of " * + join(":" .* string.(RESET_POLICIES), ", "), + )) + aggregate in AGGREGATE_POLICIES || throw(ArgumentError( + "evaluation aggregate must be one of " * + join(":" .* string.(AGGREGATE_POLICIES), ", "), + )) + streams_ = _seed_streams(streams) + return EvaluationSpec{typeof(streams_)}( + blocks_, + trials_, + horizon_, + warmup_, + construction_scope, + reset, + _root_seed(root_seed), + streams_, + aggregate, + ) +end + +seed_stream_names(spec::EvaluationSpec) = getfield.(spec.streams, :name) + +const _FNV64_OFFSET = UInt64(0xcbf29ce484222325) +const _FNV64_PRIME = UInt64(0x00000100000001b3) +const _SPLITMIX64_GAMMA = UInt64(0x9e3779b97f4a7c15) +const _SPLITMIX64_MIX1 = UInt64(0xbf58476d1ce4e5b9) +const _SPLITMIX64_MIX2 = UInt64(0x94d049bb133111eb) + +function _stable_symbol_word(name::Symbol) + value = _FNV64_OFFSET + for byte in codeunits(String(name)) + value = (value ⊻ UInt64(byte)) * _FNV64_PRIME + end + return value +end + +@inline function _splitmix64(value::UInt64) + mixed = value + _SPLITMIX64_GAMMA + mixed = (mixed ⊻ (mixed >> 30)) * _SPLITMIX64_MIX1 + mixed = (mixed ⊻ (mixed >> 27)) * _SPLITMIX64_MIX2 + return mixed ⊻ (mixed >> 31) +end + +""" + derive_seed(spec, stream, coordinates...) + +Derive a schedule-independent `UInt64` seed from the evaluation root, declared +stream name, and non-negative integer coordinates such as block and trial. +The algorithm uses stable UTF-8/FNV name encoding followed by SplitMix64; it +never calls Julia's version-dependent `hash`. +""" +function derive_seed( + spec::EvaluationSpec, + stream::Union{Symbol,AbstractString}, + coordinates::Integer..., +) + stream_ = Symbol(stream) + stream_ in seed_stream_names(spec) || throw(KeyError( + "Unknown evaluation seed stream :$(stream_). Declared streams: " * + join(":" .* string.(seed_stream_names(spec)), ", "), + )) + seed = _splitmix64(spec.root_seed ⊻ _stable_symbol_word(stream_)) + for (position, coordinate) in enumerate(coordinates) + coordinate >= 0 || throw(ArgumentError("seed coordinates must be non-negative")) + coordinate_ = try + UInt64(coordinate) + catch + throw(ArgumentError("seed coordinates must fit in UInt64")) + end + position_word = UInt64(position) * _SPLITMIX64_GAMMA + seed = _splitmix64(seed ⊻ coordinate_ ⊻ position_word) + end + return seed +end diff --git a/test/test_contract_kernel.jl b/test/test_contract_kernel.jl new file mode 100644 index 0000000..9d586f2 --- /dev/null +++ b/test/test_contract_kernel.jl @@ -0,0 +1,186 @@ +using Test + +module ContractKernel +include(joinpath(@__DIR__, "..", "src", "core", "Specifications.jl")) +end + +using .ContractKernel: + EquationSpec, + EvaluationSpec, + ImplementationSpec, + ParameterSpec, + Registry, + SeedStreamSpec, + derive_seed, + evolvable, + register!, + resolve, + seed_stream_names, + sweepable, + validate_parameter + +@testset "typed registry" begin + registry = Registry{Symbol,Int}(:nodes) + @test isempty(registry) + @test register!(registry, :a, 1) == 1 + @test registry[:a] == 1 + @test length(registry) == 1 + @test collect(keys(registry)) == [:a] + @test_throws ArgumentError register!(registry, :a, 2) + @test_throws KeyError resolve(registry, :missing) + @test_throws MethodError register!(registry, "b", 2) + @test_throws MethodError register!(registry, :b, 2.0) +end + +@testset "implementation and equation metadata" begin + implementation = ImplementationSpec( + :falandays, + identity; + label="Falandays reference", + origin="Falandays et al.", + stability=:reference, + tags=(:benchmark, :qualification), + capabilities=(:plasticity, :spiking), + metadata=(family=:homeostatic,), + ) + @test implementation.key === :falandays + @test implementation.tags == (:benchmark, :qualification) + @test implementation.metadata.family === :homeostatic + @test_throws ArgumentError ImplementationSpec( + :invalid, + identity; + stability=:unknown, + ) + @test_throws ArgumentError ImplementationSpec( + :invalid, + identity; + tags=(:duplicate, :duplicate), + ) + + equation = EquationSpec( + :activation, + raw"a_n(t) = \lambda a_n(t-1) + I_n(t)"; + title="Leaky activation", + variables=( + :a => "node activation", + :lambda => "leak coefficient", + ), + references=("Falandays2024",), + ) + @test equation.name === :activation + @test equation.variables[2] == (:lambda => "leak coefficient") + @test_throws ArgumentError EquationSpec(:empty, "") + @test_throws ArgumentError EquationSpec( + :duplicate, + "x"; + variables=(:x => "first", :x => "second"), + ) + @test EquationSpec( + :single_variable, + "x"; + variables=:x => "value", + ).variables == (:x => "value",) +end + +@testset "parameter metadata" begin + leak = ParameterSpec( + :leak, + 0.25; + owner=:node, + validator=value -> 0.0 <= value <= 1.0, + sweep=(0.1, 0.25, 0.5), + evolve=(lower=0.0, upper=1.0, scale=:linear, mutation_scale=0.05), + description="activation retained between ticks", + ) + @test leak.default == 0.25 + @test leak.sweep == (0.1, 0.25, 0.5) + @test leak.evolve.scale === :linear + @test sweepable(leak) + @test evolvable(leak) + @test validate_parameter(leak, 0.75) == 0.75 + @test_throws ArgumentError validate_parameter(leak, 1.1) + + connectivity = ParameterSpec( + :recurrent_connectivity, + :sparse; + owner=:reservoir, + validator=value -> value in (:sparse, :dense), + evolve=(values=(:sparse, :dense),), + ) + @test connectivity.owner === :reservoir + @test connectivity.evolve.values == (:sparse, :dense) + + @test_throws ArgumentError ParameterSpec( + :bad_default, + 2.0; + validator=value -> 0 <= value <= 1, + ) + @test_throws ArgumentError ParameterSpec( + :bad_sweep, + 0.5; + sweep=(0.2, 0.2), + ) + @test_throws ArgumentError ParameterSpec( + :bad_bounds, + 0.5; + evolve=(lower=0.6, upper=1.0), + ) + @test_throws ArgumentError ParameterSpec( + :bad_log, + 0.5; + evolve=(lower=0.0, upper=1.0, scale=:log), + ) + @test_throws ArgumentError ParameterSpec( + :bad_categories, + :a; + evolve=(values=nothing,), + ) + @test_throws ArgumentError ParameterSpec( + :bad_validator, + 0.5; + validator=value -> value, + ) +end + +@testset "evaluation and stable named streams" begin + evaluation = EvaluationSpec( + blocks=3, + trials_per_block=4, + horizon=7_200, + warmup=100, + construction_scope=:block, + reset=:body_environment, + root_seed=42, + streams=( + SeedStreamSpec(:environment), + SeedStreamSpec(:node_construction), + SeedStreamSpec(:bootstrap), + ), + aggregate=:median, + ) + @test evaluation.blocks == 3 + @test evaluation.root_seed == UInt64(42) + @test seed_stream_names(evaluation) == + (:environment, :node_construction, :bootstrap) + + environment_seed = derive_seed(evaluation, :environment, 1, 1) + @test environment_seed == derive_seed(evaluation, "environment", 1, 1) + @test environment_seed != derive_seed(evaluation, :environment, 1, 2) + @test environment_seed != derive_seed(evaluation, :node_construction, 1, 1) + @test environment_seed == UInt64(0x330373f0e97c4790) + + reordered = EvaluationSpec( + horizon=7_200, + root_seed=42, + streams=(:bootstrap, :environment, :node_construction), + ) + @test derive_seed(reordered, :environment, 1, 1) == environment_seed + + @test_throws ArgumentError EvaluationSpec(horizon=0) + @test_throws ArgumentError EvaluationSpec(horizon=10, construction_scope=:episode) + @test_throws ArgumentError EvaluationSpec(horizon=10, reset=:partial) + @test_throws ArgumentError EvaluationSpec(horizon=10, aggregate=:standard_error) + @test_throws ArgumentError EvaluationSpec(horizon=10, streams=(:trial, :trial)) + @test_throws KeyError derive_seed(evaluation, :optimizer, 1) + @test_throws ArgumentError derive_seed(evaluation, :environment, -1) +end From e5673f90cba039c1b91690cd83b4d8ce077aa50e Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 22:59:31 -0400 Subject: [PATCH 05/20] feat: add typed composition and evaluation contracts --- src/BrainlessLab.jl | 39 ++++ src/api/Composition.jl | 149 +++++++++++++++ src/core/Catalog.jl | 334 ++++++++++++++++++++++++++++++++++ src/core/Composition.jl | 321 ++++++++++++++++++++++++++++++++ src/core/Specifications.jl | 29 ++- test/runtests.jl | 2 + test/test_composition_spec.jl | 102 +++++++++++ test/test_contract_kernel.jl | 10 +- 8 files changed, 972 insertions(+), 14 deletions(-) create mode 100644 src/api/Composition.jl create mode 100644 src/core/Catalog.jl create mode 100644 src/core/Composition.jl create mode 100644 test/test_composition_spec.jl diff --git a/src/BrainlessLab.jl b/src/BrainlessLab.jl index 2287dd3..6724ef1 100644 --- a/src/BrainlessLab.jl +++ b/src/BrainlessLab.jl @@ -17,6 +17,7 @@ import Base: view include("core/Interfaces.jl") include("core/Traits.jl") include("core/Params.jl") +include("core/Specifications.jl") include("core/Registry.jl") include("core/Components.jl") include("core/Recorder.jl") @@ -60,11 +61,14 @@ include("envs/CartPoleVariants.jl") include("envs/PlankCartPole.jl") include("tasks/Scoring.jl") include("tasks/Tasks.jl") +include("core/Composition.jl") include("world/Environments.jl") include("world/Ensemble.jl") include("world/Metrics.jl") include("api/paper_config.jl") include("api/Highlevel.jl") +include("core/Catalog.jl") +include("api/Composition.jl") include("analysis/ActivityLevels.jl") include("analysis/Branching.jl") include("analysis/Avalanches.jl") @@ -637,6 +641,39 @@ export Recorder, export parallel_map, init_parallelism! +export Registry, + register!, + ImplementationSpec, + EquationSpec, + ParameterSpec, + validate_parameter, + sweepable, + evolvable, + SeedStreamSpec, + EvaluationSpec, + seed_stream_names, + derive_seed + +export NodeBuildContext, + NodeSpec, + node_parameter, + node_parameter_set, + resolve_parameters, + CompositionSpec, + ResolvedComposition, + RegistrySet, + DEFAULT_REGISTRY, + register_default!, + register_builtins!, + node_spec, + task_spec, + composition_spec, + nodes, + compositions, + default_composition, + resolve_composition, + falandays_node_spec + export SimResult, simulate, variants, @@ -902,4 +939,6 @@ register_ablation!(:disable_vision, DisableVision) register_optimizer!(:sepcma, SepCMA) +register_builtins!(DEFAULT_REGISTRY) + end diff --git a/src/api/Composition.jl b/src/api/Composition.jl new file mode 100644 index 0000000..07a36cb --- /dev/null +++ b/src/api/Composition.jl @@ -0,0 +1,149 @@ +function _composition_namedtuple(values::Dict{Symbol,Any}) + names = Tuple(sort!(collect(keys(values)); by=string)) + return NamedTuple{names}(Tuple(values[name] for name in names)) +end + +function _composition_body(resolved::ResolvedComposition) + resolved.body === nothing && return nothing + return _materialize_registered_body(resolved.body, resolved.body_options) +end + +function _composition_seed_ledger( + evaluation::EvaluationSpec, + block::Integer, + trial::Integer, + agent::Integer, +) + return ( + topology=derive_seed(evaluation, :topology, block, trial, agent), + node_state=derive_seed(evaluation, :node_state, block, trial, agent), + world=derive_seed(evaluation, :world, block, trial), + body=derive_seed(evaluation, :body, block, trial, agent), + task=derive_seed(evaluation, :task, block, trial), + mechanism=derive_seed(evaluation, :mechanism, block, trial, agent), + ) +end + +function _build_composition( + resolved::ResolvedComposition, + evaluation::EvaluationSpec; + block::Integer=1, + trial::Integer=1, + record=_DEFAULT_RECORD_CHANNELS, + every::Integer=1, +) + body = _composition_body(resolved) + task_options = _composition_namedtuple(resolved.task_options) + world_seed = _seed_to_int(derive_seed(evaluation, :world, block, trial)) + task_setup = _setup_for_node_count( + resolved.task, + resolved.n_nodes; + seed=world_seed, + body=body, + task_options..., + ) + bodies = task_setup.bodies + agents = Vector{Agent}(undef, length(bodies)) + ledgers = Vector{NamedTuple}(undef, length(bodies)) + @inbounds for slot in eachindex(bodies) + body_at_slot = bodies[slot] + layout = portspec(body_at_slot) + default_link_p = Float64(get(resolved.parameters, :link_p, 0.1)) + profile = receptor_link_profile(body_at_slot, default_link_p) + seeds = _composition_seed_ledger(evaluation, block, trial, slot) + context = NodeBuildContext( + resolved.n_nodes, + layout, + seeds; + receptor_profile=profile, + ) + reservoir = resolved.node.build(context, resolved.parameters) + reservoir isa Reservoir || throw(ArgumentError( + "node :$(resolved.node.id) returned $(typeof(reservoir)), not Reservoir", + )) + agents[slot] = _make_agent( + reservoir, + body_at_slot; + cycle=resolved.interaction_cycle, + ) + ledgers[slot] = seeds + end + recorder = Recorder(enabled=_record_symbols(record), every=Int(every)) + ensemble = Ensemble(agents, task_setup.environment; recorder=recorder) + return ( + ensemble=ensemble, + recorder=recorder, + seed_ledger=Tuple(ledgers), + ) +end + +""" + simulate(composition::CompositionSpec; registry=DEFAULT_REGISTRY, ...) + +Resolve and run one explicit composition. Task horizon, replication, resets, +and inferential aggregation belong to `EvaluationSpec`; this convenience method +runs one trial and accepts a temporary `ticks` override for interactive use. +""" +function simulate( + composition::CompositionSpec; + registry::RegistrySet=DEFAULT_REGISTRY, + ticks=nothing, + seed::Integer=0, + record=_DEFAULT_RECORD_CHANNELS, + every::Integer=1, + window=nothing, + metrics=nothing, +) + resolved = resolve_composition(composition, registry) + tick_count = ticks === nothing ? resolved.task.default_ticks : Int(ticks) + tick_count > 0 || throw(ArgumentError("simulation ticks must be positive")) + window_ = window === nothing ? min(tick_count, resolved.task.default_window) : Int(window) + 0 < window_ <= tick_count || throw(ArgumentError( + "simulation window must lie in 1:ticks", + )) + evaluation = EvaluationSpec(horizon=tick_count, root_seed=seed) + setup = _build_composition( + resolved, + evaluation; + block=1, + trial=1, + record=record, + every=every, + ) + outcome = rollout!(setup.ensemble, tick_count; window=window_, metrics=metrics) + base_config = _simulation_config( + setup.ensemble; + ticks=tick_count, + seed=Int(seed), + record=_record_symbols(record), + every=Int(every), + window=window_, + n_nodes=resolved.n_nodes, + ablation=:none, + ablation_notes=(), + interventions=nothing, + task_spec=resolved.task, + ) + config = merge( + base_config, + ( + composition=composition.id, + parameters=_composition_namedtuple(resolved.parameters), + seed_ledger=setup.seed_ledger, + ), + ) + return SimResult( + setup.recorder, + outcome, + resolved.task.name, + resolved.node.id, + config, + ) +end + +simulate( + composition::Union{Symbol,AbstractString}, + registry::RegistrySet; + kwargs..., +) = simulate(composition_spec(registry, composition); registry=registry, kwargs...) + diff --git a/src/core/Catalog.jl b/src/core/Catalog.jl new file mode 100644 index 0000000..4c276e0 --- /dev/null +++ b/src/core/Catalog.jl @@ -0,0 +1,334 @@ +_seed_to_int(seed::Integer) = Int(mod(UInt64(seed), UInt64(typemax(Int)))) + +function _context_seed(context::NodeBuildContext, name::Symbol) + hasproperty(context.seeds, name) || throw(KeyError( + "node build context has no :$(name) seed", + )) + return _seed_to_int(getproperty(context.seeds, name)) +end + +function _generic_node_builder(id::Symbol, constructor) + profile_keyword = node_receptor_profile_keyword(id) + return function (context::NodeBuildContext, values) + options = Dict{Symbol,Any}(values) + if context.receptor_profile !== nothing + profile_keyword === nothing && throw(ArgumentError( + "body requires a receptor profile but node :$(id) does not declare that capability", + )) + options[profile_keyword] = context.receptor_profile + end + options[:seed] = _context_seed(context, :topology) + keywords = (; (key => value for (key, value) in options)...) + reservoir = constructor( + context.n_nodes, + n_receptors(context.ports), + n_effectors(context.ports); + keywords..., + ) + reservoir isa Reservoir || throw(ArgumentError( + "node :$(id) builder returned $(typeof(reservoir)), not Reservoir", + )) + return reservoir + end +end + +function _falandays_parameters() + defaults = FalandaysParams() + nonnegative = value -> value isa Float64 && isfinite(value) && value >= 0.0 + positive = value -> value isa Float64 && isfinite(value) && value > 0.0 + return ( + ParameterSpec( + :leak, + defaults.leak; + validator=value -> value isa Float64 && isfinite(value) && 0.0 <= value <= 1.0, + sweep=(0.1, 0.25, 0.5, 0.75), + evolve=(lower=0.0, upper=1.0, scale=:linear, mutation_scale=0.05), + description="activation retained between updates", + ), + ParameterSpec( + :lrate_wmat, + defaults.lrate_wmat; + validator=nonnegative, + sweep=(0.05, 0.1, 0.35, 1.0), + evolve=(lower=1.0e-4, upper=2.0, scale=:log, mutation_scale=0.2), + description="local recurrent-weight homeostasis rate", + ), + ParameterSpec( + :lrate_targ, + defaults.lrate_targ; + validator=nonnegative, + sweep=(0.001, 0.01, 0.1), + evolve=(lower=1.0e-4, upper=0.5, scale=:log, mutation_scale=0.2), + description="target-activity adaptation rate", + ), + ParameterSpec( + :threshold_mult, + defaults.threshold_mult; + validator=positive, + sweep=(1.5, 2.0, 2.5), + evolve=(lower=0.100001, upper=8.0, scale=:log, mutation_scale=0.15), + description="target-to-spike-threshold multiplier", + ), + ParameterSpec( + :targ_min, + defaults.targ_min; + validator=positive, + sweep=(0.5, 1.0, 1.5), + evolve=(lower=0.100001, upper=5.0, scale=:log, mutation_scale=0.15), + description="minimum homeostatic target activity", + ), + ParameterSpec( + :input_weight, + defaults.input_weight; + validator=nonnegative, + sweep=(0.75, 1.875, 2.75, 4.0), + evolve=(lower=1.0e-4, upper=12.5, scale=:log, mutation_scale=0.2), + description="sensory input amplitude", + ), + ParameterSpec( + :weight_init_std, + defaults.weight_init_std; + validator=nonnegative, + sweep=(0.25, 0.5, 1.0, 2.0), + evolve=(lower=1.0e-4, upper=4.0, scale=:log, mutation_scale=0.2), + description="initial recurrent-weight scale", + ), + ParameterSpec( + :learn_on, + defaults.learn_on; + validator=value -> value isa Bool, + description="enable online homeostatic plasticity", + ), + ParameterSpec( + :link_p, + 0.1; + owner=:reservoir, + validator=value -> value isa Float64 && isfinite(value) && 0.0 <= value <= 1.0, + sweep=(0.05, 0.1, 0.2, 0.4), + description="recurrent connection probability", + ), + ParameterSpec( + :weight_init_mode, + :legacy_normal; + owner=:reservoir, + validator=value -> value in (:legacy_normal, :excitatory, :pong_mixed), + description="initial recurrent-weight sign regime", + ), + ParameterSpec( + :rectify, + true; + owner=:reservoir, + validator=value -> value isa Bool, + description="rectify activation before thresholding", + ), + ParameterSpec( + :topology, + :bernoulli; + owner=:reservoir, + validator=value -> value in (:bernoulli, :watts_strogatz), + description="recurrent connectivity family", + ), + ParameterSpec( + :repair_masks, + true; + owner=:reservoir, + validator=value -> value isa Bool, + description="repair empty input or output masks", + ), + ) +end + +function _falandays_equations() + return ( + EquationSpec( + :activation, + raw"a_n(t)=\lambda a_n(t-1)+\sum_r U_{rn}x_r(t)+\sum_j W_{jn}(t)s_j(t-1)"; + title="Locally driven activation", + description="Sensory and previous-step recurrent currents update each node locally.", + variables=( + :a => "node activation", + :lambda => "leak coefficient", + :x => "sensory input", + :U => "input weight", + :W => "recurrent weight", + :s => "previous-step spike", + ), + ), + EquationSpec( + :weight_homeostasis, + raw"W_{jn}\leftarrow W_{jn}-\eta_W\,s_j(t-1)\,\frac{a_n(t)-T_n(t)}{k_n}"; + title="Local weight homeostasis", + variables=( + :eta_W => "weight-learning rate", + :T => "target activity", + :k => "active incoming connections", + ), + ), + EquationSpec( + :target_homeostasis, + raw"T_n\leftarrow\max\left(T_n+\eta_T(a_n-T_n),T_{\min}\right)"; + title="Local target adaptation", + variables=( + :eta_T => "target-learning rate", + :T_min => "minimum target activity", + ), + ), + ) +end + +function _falandays_builder(context::NodeBuildContext, values) + params = FalandaysParams( + leak=values[:leak], + lrate_wmat=values[:lrate_wmat], + lrate_targ=values[:lrate_targ], + threshold_mult=values[:threshold_mult], + targ_min=values[:targ_min], + input_weight=values[:input_weight], + weight_init_std=values[:weight_init_std], + learn_on=values[:learn_on], + ) + options = Dict{Symbol,Any}( + :params => params, + :link_p => values[:link_p], + :weight_init_mode => values[:weight_init_mode], + :rectify => values[:rectify], + :topology => values[:topology], + :repair_masks => values[:repair_masks], + ) + context.receptor_profile === nothing || + (options[:input_link_p] = context.receptor_profile) + keywords = (; (key => value for (key, value) in options)...) + return _falandays_native( + context.n_nodes, + n_receptors(context.ports), + n_effectors(context.ports); + seed=_context_seed(context, :topology), + keywords..., + ) +end + +function falandays_node_spec() + return NodeSpec( + :falandays, + _falandays_builder; + genome_type=FalandaysParams, + stability=:reference, + tags=(:reference,), + capabilities=( + :spiking, + :online_plasticity, + :recurrent_weights, + :homeostatic_target, + :receptor_profile, + ), + parameters=_falandays_parameters(), + parameter_sets=Dict( + :sweep => (:leak, :lrate_wmat), + :evolve => ( + :leak, + :lrate_wmat, + :lrate_targ, + :threshold_mult, + :targ_min, + :input_weight, + :weight_init_std, + ), + :connectivity => (:link_p,), + ), + equations=_falandays_equations(), + default_analyses=( + :branching_ratio_mr, + :node_target_error, + :spectral_radius, + :fano_factor, + :participation_ratio, + ), + metadata=(source="Falandays et al. authors-derived Julia implementation",), + ) +end + +function _generic_registered_node_spec(id::Symbol, constructor) + genome = try + genome_type(id) + catch + nothing + end + capabilities = Symbol[] + genome === nothing || push!(capabilities, :evolvable) + node_receptor_profile_keyword(id) === nothing || push!(capabilities, :receptor_profile) + return NodeSpec( + id, + _generic_node_builder(id, constructor); + genome_type=genome, + stability=id === :null_random ? :control : :experimental, + tags=id === :null_random ? (:control,) : (:experimental,), + capabilities=Tuple(capabilities), + metadata=(adapter=:registered_constructor,), + ) +end + +function _falandays_reference_composition(task::Symbol) + config = falandays_paper_config(task) + return CompositionSpec( + Symbol("falandays_", task), + :falandays, + task; + n_nodes=config.nnodes, + parameters=Dict{Symbol,Any}( + :lrate_wmat => config.lrate_wmat, + :lrate_targ => config.lrate_targ, + :input_weight => config.input_amp, + :weight_init_mode => config.weight_init_mode, + :rectify => false, + :topology => :bernoulli, + :repair_masks => false, + ), + ) +end + +function register_builtins!(registry::RegistrySet) + register!(registry, falandays_node_spec()) + for (id, constructor) in sort!(collect(NODES); by=pair -> string(first(pair))) + id in (:falandays, :falandays_base, :falandays_ablated) && continue + register!(registry, _generic_registered_node_spec(id, constructor)) + end + + for (id, task) in sort!(collect(TASKS); by=pair -> string(first(pair))) + id === :pong_hitrate && continue + task isa TaskSpec || continue + register!(registry, task) + end + + for (id, implementation) in sort!(collect(BODIES); by=pair -> string(first(pair))) + register!(registry, :bodies, ImplementationSpec(id, implementation)) + end + for (id, implementation) in sort!(collect(DRIVES); by=pair -> string(first(pair))) + register!(registry, :drives, ImplementationSpec(id, implementation)) + end + for (id, implementation) in sort!(collect(MOTORS); by=pair -> string(first(pair))) + register!(registry, :motors, ImplementationSpec(id, implementation)) + end + for (id, implementation) in sort!(collect(SENSORS); by=pair -> string(first(pair))) + register!(registry, :sensors, ImplementationSpec(id, implementation)) + end + for (id, implementation) in sort!(collect(METRICS); by=pair -> string(first(pair))) + register!(registry, :metrics, ImplementationSpec(id, implementation)) + end + for (id, implementation) in sort!(collect(VIEWS); by=pair -> string(first(pair))) + register!(registry, :views, ImplementationSpec(id, implementation)) + end + for (id, implementation) in sort!(collect(OPTIMIZERS); by=pair -> string(first(pair))) + register!(registry, :optimizers, ImplementationSpec(id, implementation)) + end + for (id, implementation) in sort!(collect(ABLATIONS); by=pair -> string(first(pair))) + register!(registry, :ablations, ImplementationSpec(id, implementation)) + end + + for task in (:wall, :tracking, :pong) + register_default!(registry, _falandays_reference_composition(task)) + end + return registry +end + +const DEFAULT_REGISTRY = RegistrySet() + diff --git a/src/core/Composition.jl b/src/core/Composition.jl new file mode 100644 index 0000000..505800b --- /dev/null +++ b/src/core/Composition.jl @@ -0,0 +1,321 @@ +"""Cold-path context supplied to a registered node builder.""" +struct NodeBuildContext{P,S,R} + n_nodes::Int + ports::P + seeds::S + receptor_profile::R + + function NodeBuildContext( + n_nodes::Integer, + ports, + seeds; + receptor_profile=nothing, + ) + count = Int(n_nodes) + count > 0 || throw(ArgumentError("node build context requires a positive n_nodes")) + return new{typeof(ports),typeof(seeds),typeof(receptor_profile)}( + count, + ports, + seeds, + receptor_profile, + ) + end +end + +""" + NodeSpec + +Discoverable contract for one neural substrate. Node count and the task/body +ports are supplied by `NodeBuildContext`; the node owns only its mechanism and +declared parameter surface. +""" +struct NodeSpec{B,G,P,E,M} + id::Symbol + build::B + genome_type::G + stability::Symbol + tags::Tuple{Vararg{Symbol}} + capabilities::Tuple{Vararg{Symbol}} + parameters::P + parameter_sets::Dict{Symbol,Tuple{Vararg{Symbol}}} + equations::E + default_analyses::Tuple{Vararg{Symbol}} + metadata::M +end + +function NodeSpec( + id::Union{Symbol,AbstractString}, + build; + genome_type=nothing, + stability::Symbol=:experimental, + tags=(), + capabilities=(), + parameters=(), + parameter_sets=Dict{Symbol,Tuple{Vararg{Symbol}}}(), + equations=(), + default_analyses=(), + metadata::NamedTuple=NamedTuple(), +) + id_ = _nonempty_symbol(id, "node id") + stability in IMPLEMENTATION_STABILITIES || throw(ArgumentError( + "node :$(id_) has invalid stability :$(stability)", + )) + genome_type === nothing || + (genome_type isa Type && genome_type <: NodeModel) || + throw(ArgumentError("node :$(id_) genome_type must be a NodeModel type or nothing")) + tags_ = _symbol_tuple(tags, "node tags") + capabilities_ = _symbol_tuple(capabilities, "node capabilities") + parameters_ = Tuple(parameters) + all(parameter -> parameter isa ParameterSpec, parameters_) || throw(ArgumentError( + "node :$(id_) parameters must all be ParameterSpec values", + )) + parameter_names = Tuple(parameter.name for parameter in parameters_) + length(unique(parameter_names)) == length(parameter_names) || throw(ArgumentError( + "node :$(id_) parameter names must be unique", + )) + sets_ = Dict{Symbol,Tuple{Vararg{Symbol}}}() + for (set_name, members) in pairs(parameter_sets) + set_name_ = _nonempty_symbol(set_name, "parameter-set name") + members_ = _symbol_tuple(members, "parameter-set members") + unknown = setdiff(members_, parameter_names) + isempty(unknown) || throw(ArgumentError( + "node :$(id_) parameter set :$(set_name_) references unknown parameters $(unknown)", + )) + sets_[set_name_] = members_ + end + equations_ = Tuple(equations) + all(equation -> equation isa EquationSpec, equations_) || throw(ArgumentError( + "node :$(id_) equations must all be EquationSpec values", + )) + analyses_ = _symbol_tuple(default_analyses, "default analyses") + return NodeSpec{ + typeof(build), + typeof(genome_type), + typeof(parameters_), + typeof(equations_), + typeof(metadata), + }( + id_, + build, + genome_type, + stability, + tags_, + capabilities_, + parameters_, + sets_, + equations_, + analyses_, + metadata, + ) +end + +node_parameter(spec::NodeSpec, name::Union{Symbol,AbstractString}) = begin + name_ = Symbol(name) + index = findfirst(parameter -> parameter.name === name_, spec.parameters) + index === nothing && throw(KeyError("node :$(spec.id) has no parameter :$(name_)")) + spec.parameters[index] +end + +function node_parameter_set(spec::NodeSpec, name::Union{Symbol,AbstractString}) + name_ = Symbol(name) + haskey(spec.parameter_sets, name_) || throw(KeyError( + "node :$(spec.id) has no parameter set :$(name_)", + )) + return spec.parameter_sets[name_] +end + +function resolve_parameters(spec::NodeSpec, overrides=Dict{Symbol,Any}()) + override_dict = Dict{Symbol,Any}(Symbol(key) => value for (key, value) in pairs(overrides)) + known = Set(parameter.name for parameter in spec.parameters) + unknown = sort!(collect(setdiff(Set(keys(override_dict)), known)); by=string) + isempty(unknown) || throw(ArgumentError( + "node :$(spec.id) received unknown parameters $(unknown)", + )) + resolved = Dict{Symbol,Any}() + for parameter in spec.parameters + value = get(override_dict, parameter.name, parameter.default) + resolved[parameter.name] = validate_parameter(parameter, value) + end + return resolved +end + +"""One serializable, runnable node-task-body composition.""" +Base.@kwdef struct CompositionSpec + id::Symbol + node::Symbol + task::Symbol + body::Union{Nothing,Symbol}=nothing + n_agents::Union{Nothing,Int}=nothing + n_nodes::Int + parameters::Dict{Symbol,Any}=Dict{Symbol,Any}() + task_options::Dict{Symbol,Any}=Dict{Symbol,Any}() + body_options::Dict{Symbol,Any}=Dict{Symbol,Any}() + interaction_cycle::Union{Nothing,InteractionCycle}=nothing +end + +function CompositionSpec( + id::Union{Symbol,AbstractString}, + node::Union{Symbol,AbstractString}, + task::Union{Symbol,AbstractString}; + body=nothing, + n_agents=nothing, + n_nodes::Integer, + parameters=Dict{Symbol,Any}(), + task_options=Dict{Symbol,Any}(), + body_options=Dict{Symbol,Any}(), + interaction_cycle::Union{Nothing,InteractionCycle}=nothing, +) + id_ = _nonempty_symbol(id, "composition id") + node_ = _nonempty_symbol(node, "composition node") + task_ = _nonempty_symbol(task, "composition task") + count = Int(n_nodes) + count > 0 || throw(ArgumentError("composition :$(id_) requires positive n_nodes")) + agents = n_agents === nothing ? nothing : Int(n_agents) + agents === nothing || agents > 0 || throw(ArgumentError( + "composition :$(id_) requires positive n_agents when specified", + )) + body_ = body === nothing ? nothing : _nonempty_symbol(body, "composition body") + return CompositionSpec( + id_, + node_, + task_, + body_, + agents, + count, + Dict{Symbol,Any}(Symbol(key) => value for (key, value) in pairs(parameters)), + Dict{Symbol,Any}(Symbol(key) => value for (key, value) in pairs(task_options)), + Dict{Symbol,Any}(Symbol(key) => value for (key, value) in pairs(body_options)), + interaction_cycle, + ) +end + +struct ResolvedComposition{N,T,B,C} + id::Symbol + node::N + task::T + body::B + n_agents::Union{Nothing,Int} + n_nodes::Int + parameters::Dict{Symbol,Any} + task_options::Dict{Symbol,Any} + body_options::Dict{Symbol,Any} + interaction_cycle::C +end + +mutable struct RegistrySet + nodes::Registry{Symbol,NodeSpec} + tasks::Registry{Symbol,TaskSpec} + bodies::Registry{Symbol,ImplementationSpec} + drives::Registry{Symbol,ImplementationSpec} + motors::Registry{Symbol,ImplementationSpec} + sensors::Registry{Symbol,ImplementationSpec} + metrics::Registry{Symbol,ImplementationSpec} + analyses::Registry{Symbol,ImplementationSpec} + views::Registry{Symbol,ImplementationSpec} + optimizers::Registry{Symbol,ImplementationSpec} + ablations::Registry{Symbol,ImplementationSpec} + compositions::Registry{Symbol,CompositionSpec} + benchmarks::Registry{Symbol,Any} + experiments::Registry{Tuple{Symbol,VersionNumber},Any} + composition_defaults::Dict{Tuple{Symbol,Symbol},Symbol} +end + +function RegistrySet() + return RegistrySet( + Registry{Symbol,NodeSpec}(:nodes), + Registry{Symbol,TaskSpec}(:tasks), + Registry{Symbol,ImplementationSpec}(:bodies), + Registry{Symbol,ImplementationSpec}(:drives), + Registry{Symbol,ImplementationSpec}(:motors), + Registry{Symbol,ImplementationSpec}(:sensors), + Registry{Symbol,ImplementationSpec}(:metrics), + Registry{Symbol,ImplementationSpec}(:analyses), + Registry{Symbol,ImplementationSpec}(:views), + Registry{Symbol,ImplementationSpec}(:optimizers), + Registry{Symbol,ImplementationSpec}(:ablations), + Registry{Symbol,CompositionSpec}(:compositions), + Registry{Symbol,Any}(:benchmarks), + Registry{Tuple{Symbol,VersionNumber},Any}(:experiments), + Dict{Tuple{Symbol,Symbol},Symbol}(), + ) +end + +register!(registry::RegistrySet, spec::NodeSpec) = register!(registry.nodes, spec.id, spec) +register!(registry::RegistrySet, spec::TaskSpec) = register!(registry.tasks, spec.name, spec) +register!(registry::RegistrySet, spec::CompositionSpec) = + register!(registry.compositions, spec.id, spec) + +function register!(registry::RegistrySet, kind::Symbol, spec::ImplementationSpec) + kind in (:bodies, :drives, :motors, :sensors, :metrics, :analyses, :views, :optimizers, :ablations) || + throw(ArgumentError("unknown implementation registry :$(kind)")) + return register!(getfield(registry, kind), spec.key, spec) +end + +function register_default!(registry::RegistrySet, composition::CompositionSpec) + register!(registry, composition) + key = (composition.node, composition.task) + haskey(registry.composition_defaults, key) && throw(ArgumentError( + "default composition for node :$(composition.node) and task :$(composition.task) is already registered", + )) + registry.composition_defaults[key] = composition.id + return composition +end + +node_spec(registry::RegistrySet, id::Union{Symbol,AbstractString}) = + resolve(registry.nodes, Symbol(id)) +task_spec(registry::RegistrySet, id::Union{Symbol,AbstractString}) = + resolve(registry.tasks, Symbol(id)) +composition_spec(registry::RegistrySet, id::Union{Symbol,AbstractString}) = + resolve(registry.compositions, Symbol(id)) + +nodes(registry::RegistrySet) = sort!(collect(keys(registry.nodes)); by=string) +tasks(registry::RegistrySet) = sort!(collect(keys(registry.tasks)); by=string) +compositions(registry::RegistrySet) = sort!(collect(keys(registry.compositions)); by=string) + +function default_composition( + registry::RegistrySet, + node::Union{Symbol,AbstractString}, + task::Union{Symbol,AbstractString}, +) + key = (Symbol(node), Symbol(task)) + haskey(registry.composition_defaults, key) || throw(KeyError( + "no default composition for node :$(key[1]) and task :$(key[2])", + )) + return composition_spec(registry, registry.composition_defaults[key]) +end + +function _materialize_registered_body(spec::ImplementationSpec, options::Dict{Symbol,Any}) + implementation = spec.implementation + implementation isa AbstractBody && return deepcopy(implementation) + values = (; (key => value for (key, value) in options)...) + applicable(implementation; values...) || throw(ArgumentError( + "registered body :$(spec.key) does not accept its declared options", + )) + body = implementation(; values...) + body isa AbstractBody || throw(ArgumentError( + "registered body :$(spec.key) returned $(typeof(body)), not AbstractBody", + )) + return body +end + +function resolve_composition(spec::CompositionSpec, registry::RegistrySet) + node = node_spec(registry, spec.node) + task = task_spec(registry, spec.task) + body = spec.body === nothing ? nothing : resolve(registry.bodies, spec.body) + parameters = resolve_parameters(node, spec.parameters) + task_options = copy(spec.task_options) + spec.n_agents === nothing || (task_options[:n_agents] = spec.n_agents) + return ResolvedComposition( + spec.id, + node, + task, + body, + spec.n_agents, + spec.n_nodes, + parameters, + task_options, + copy(spec.body_options), + spec.interaction_cycle === nothing ? task.interaction_cycle : spec.interaction_cycle, + ) +end + diff --git a/src/core/Specifications.jl b/src/core/Specifications.jl index ffbf7c8..90a65c6 100644 --- a/src/core/Specifications.jl +++ b/src/core/Specifications.jl @@ -318,6 +318,7 @@ optional `scale`/`mutation_scale`. struct ParameterSpec{T,V,S,E} name::Symbol owner::Symbol + datatype::Type default::T validator::V sweep::S @@ -330,6 +331,7 @@ function ParameterSpec( name::Union{Symbol,AbstractString}, default; owner::Union{Symbol,AbstractString}=:node, + datatype::Type=typeof(default), validator=nothing, sweep=nothing, evolve=nothing, @@ -338,6 +340,9 @@ function ParameterSpec( ) name_ = _nonempty_symbol(name, "parameter name") owner_ = _nonempty_symbol(owner, "parameter owner") + default isa datatype || throw(ArgumentError( + "default for parameter :$(name_) must be a $(datatype), got $(typeof(default))", + )) units_ = units === nothing ? nothing : String(units) units_ !== nothing && isempty(strip(units_)) && throw(ArgumentError("parameter units must not be empty")) @@ -352,6 +357,7 @@ function ParameterSpec( }( name_, owner_, + datatype, default, validator, sweep_, @@ -362,8 +368,12 @@ function ParameterSpec( end """Validate and return a candidate value for a parameter.""" -validate_parameter(spec::ParameterSpec, value) = - _parameter_value_valid(spec.validator, value, spec.name) +function validate_parameter(spec::ParameterSpec, value) + value isa spec.datatype || throw(ArgumentError( + "parameter :$(spec.name) must be a $(spec.datatype), got $(typeof(value))", + )) + return _parameter_value_valid(spec.validator, value, spec.name) +end sweepable(spec::ParameterSpec) = spec.sweep !== nothing evolvable(spec::ParameterSpec) = spec.evolve !== nothing @@ -388,12 +398,12 @@ struct SeedStreamSpec end const DEFAULT_SEED_STREAMS = ( - SeedStreamSpec(:environment), - SeedStreamSpec(:node_construction), - SeedStreamSpec(:runtime), - SeedStreamSpec(:trial), - SeedStreamSpec(:optimizer), - SeedStreamSpec(:bootstrap), + SeedStreamSpec(:topology), + SeedStreamSpec(:node_state), + SeedStreamSpec(:world), + SeedStreamSpec(:body), + SeedStreamSpec(:task), + SeedStreamSpec(:mechanism), ) function _seed_streams(streams) @@ -456,6 +466,9 @@ function EvaluationSpec(; trials_ > 0 || throw(ArgumentError("evaluation trials_per_block must be positive")) horizon_ > 0 || throw(ArgumentError("evaluation horizon must be positive")) warmup_ >= 0 || throw(ArgumentError("evaluation warmup must be non-negative")) + warmup_ < horizon_ || throw(ArgumentError( + "evaluation warmup must be less than its horizon", + )) construction_scope in CONSTRUCTION_SCOPES || throw(ArgumentError( "evaluation construction_scope must be one of " * join(":" .* string.(CONSTRUCTION_SCOPES), ", "), diff --git a/test/runtests.jl b/test/runtests.jl index 1cd368e..54b6af7 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -38,6 +38,8 @@ include("test_core_task_controls.jl") include("test_core_calibration.jl") include("test_cartpole_variants.jl") include("test_plank_cartpole.jl") +include("test_contract_kernel.jl") +include("test_composition_spec.jl") include("test_morphology.jl") include("test_homeostasis.jl") include("test_sensor.jl") diff --git a/test/test_composition_spec.jl b/test/test_composition_spec.jl new file mode 100644 index 0000000..d077f05 --- /dev/null +++ b/test/test_composition_spec.jl @@ -0,0 +1,102 @@ +using BrainlessLab +using Test + +@testset "typed composition catalog" begin + registry = RegistrySet() + @test isempty(nodes(registry)) + register!(registry, falandays_node_spec()) + @test nodes(registry) == [:falandays] + @test_throws ArgumentError register!(registry, falandays_node_spec()) + + falandays = node_spec(DEFAULT_REGISTRY, :falandays) + @test falandays.stability === :reference + @test falandays.genome_type === FalandaysParams + @test node_parameter_set(falandays, :sweep) == (:leak, :lrate_wmat) + @test node_parameter_set(falandays, :evolve) == ( + :leak, + :lrate_wmat, + :lrate_targ, + :threshold_mult, + :targ_min, + :input_weight, + :weight_init_std, + ) + @test node_parameter(falandays, :link_p).owner === :reservoir + @test_throws KeyError node_parameter(falandays, :n_nodes) + + @test task_spec(DEFAULT_REGISTRY, :wall).status === :experimental + @test task_spec(DEFAULT_REGISTRY, :tracking).status === :reference + @test task_spec(DEFAULT_REGISTRY, :pong).status === :reference + @test :pong_hitrate ∉ tasks(DEFAULT_REGISTRY) + @test all(task -> task in tasks(DEFAULT_REGISTRY), ( + :cartpole_plank_easy, + :cartpole_plank_medium, + :cartpole_plank_hard, + :cartpole_plank_hardest, + )) + + tracking = default_composition(DEFAULT_REGISTRY, :falandays, :tracking) + pong = default_composition(DEFAULT_REGISTRY, :falandays, :pong) + wall = default_composition(DEFAULT_REGISTRY, :falandays, :wall) + @test tracking.n_nodes == 200 + @test tracking.parameters[:input_weight] == 0.75 + @test tracking.parameters[:lrate_targ] == 0.01 + @test tracking.parameters[:weight_init_mode] === :excitatory + @test pong.n_nodes == 500 + @test pong.parameters[:input_weight] == 2.75 + @test pong.parameters[:lrate_targ] == 0.1 + @test pong.parameters[:weight_init_mode] === :pong_mixed + @test wall.n_nodes == 200 + @test_throws KeyError default_composition( + DEFAULT_REGISTRY, + :falandays, + :cartpole_plank_easy, + ) + + bad = CompositionSpec( + :bad, + :falandays, + :tracking; + n_nodes=12, + parameters=Dict(:unknown => 1.0), + ) + @test_throws ArgumentError resolve_composition(bad, DEFAULT_REGISTRY) + + resolved = resolve_composition(tracking, DEFAULT_REGISTRY) + @test resolved.parameters[:leak] == FalandaysParams().leak + @test resolved.parameters[:lrate_wmat] == 1.0 + @test resolved.interaction_cycle === nothing +end + +@testset "CompositionSpec executes through named seed streams" begin + composition = CompositionSpec( + :tracking_smoke, + :falandays, + :tracking; + n_nodes=12, + parameters=Dict( + :input_weight => 0.75, + :lrate_wmat => 1.0, + :lrate_targ => 0.01, + :weight_init_mode => :excitatory, + :rectify => false, + :repair_masks => false, + ), + ) + first_run = simulate(composition; ticks=8, seed=19, record=()) + second_run = simulate(composition; ticks=8, seed=19, record=()) + @test first_run.metrics == second_run.metrics + @test first_run.config.composition === :tracking_smoke + @test first_run.config.n_nodes == 12 + @test first_run.config.seed_ledger == second_run.config.seed_ledger + @test propertynames(first_run.config.seed_ledger[1]) == ( + :topology, + :node_state, + :world, + :body, + :task, + :mechanism, + ) + @test task_outcome(first_run).key === :track_score +end + diff --git a/test/test_contract_kernel.jl b/test/test_contract_kernel.jl index 9d586f2..593cbe4 100644 --- a/test/test_contract_kernel.jl +++ b/test/test_contract_kernel.jl @@ -1,10 +1,5 @@ using Test - -module ContractKernel -include(joinpath(@__DIR__, "..", "src", "core", "Specifications.jl")) -end - -using .ContractKernel: +using BrainlessLab: EquationSpec, EvaluationSpec, ImplementationSpec, @@ -93,12 +88,14 @@ end description="activation retained between ticks", ) @test leak.default == 0.25 + @test leak.datatype === Float64 @test leak.sweep == (0.1, 0.25, 0.5) @test leak.evolve.scale === :linear @test sweepable(leak) @test evolvable(leak) @test validate_parameter(leak, 0.75) == 0.75 @test_throws ArgumentError validate_parameter(leak, 1.1) + @test_throws ArgumentError validate_parameter(leak, 1) connectivity = ParameterSpec( :recurrent_connectivity, @@ -177,6 +174,7 @@ end @test derive_seed(reordered, :environment, 1, 1) == environment_seed @test_throws ArgumentError EvaluationSpec(horizon=0) + @test_throws ArgumentError EvaluationSpec(horizon=10, warmup=10) @test_throws ArgumentError EvaluationSpec(horizon=10, construction_scope=:episode) @test_throws ArgumentError EvaluationSpec(horizon=10, reset=:partial) @test_throws ArgumentError EvaluationSpec(horizon=10, aggregate=:standard_error) From e6b8d0823f502f9177e7e06942c174d6d57a7e83 Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 23:02:44 -0400 Subject: [PATCH 06/20] feat: unify research operation plans --- src/BrainlessLab.jl | 19 ++ src/operations/Plans.jl | 327 +++++++++++++++++++++++++++++++++++ test/runtests.jl | 1 + test/test_operation_plans.jl | 87 ++++++++++ 4 files changed, 434 insertions(+) create mode 100644 src/operations/Plans.jl create mode 100644 test/test_operation_plans.jl diff --git a/src/BrainlessLab.jl b/src/BrainlessLab.jl index 6724ef1..ee7d0ab 100644 --- a/src/BrainlessLab.jl +++ b/src/BrainlessLab.jl @@ -69,6 +69,7 @@ include("api/paper_config.jl") include("api/Highlevel.jl") include("core/Catalog.jl") include("api/Composition.jl") +include("operations/Plans.jl") include("analysis/ActivityLevels.jl") include("analysis/Branching.jl") include("analysis/Avalanches.jl") @@ -674,6 +675,24 @@ export NodeBuildContext, resolve_composition, falandays_node_spec +export AbstractOperationPlan, + AbstractResolvedOperationPlan, + AbstractOperationResult, + EvaluationTarget, + ProfilePlan, + SweepAxis, + SweepPlan, + AblationSpec, + AblationPlan, + EvolutionPlan, + BenchmarkCasePlan, + BenchmarkPlan, + ExperimentSpec, + validate, + execute, + tables, + summary + export SimResult, simulate, variants, diff --git a/src/operations/Plans.jl b/src/operations/Plans.jl new file mode 100644 index 0000000..44a1a9d --- /dev/null +++ b/src/operations/Plans.jl @@ -0,0 +1,327 @@ +abstract type AbstractOperationPlan end +abstract type AbstractResolvedOperationPlan end +abstract type AbstractOperationResult end + +"""One named composition plus its complete outer evaluation protocol.""" +struct EvaluationTarget{C<:CompositionSpec,E<:EvaluationSpec} + id::Symbol + composition::C + evaluation::E + + function EvaluationTarget( + id::Union{Symbol,AbstractString}, + composition::C, + evaluation::E, + ) where {C<:CompositionSpec,E<:EvaluationSpec} + id_ = _nonempty_symbol(id, "evaluation target id") + return new{C,E}(id_, composition, evaluation) + end +end + +struct ProfilePlan{T<:EvaluationTarget} <: AbstractOperationPlan + id::Symbol + target::T + analyses::Tuple{Vararg{Symbol}} + record_every::Int +end + +function ProfilePlan( + id::Union{Symbol,AbstractString}, + target::EvaluationTarget; + analyses=(), + record_every::Integer=1, +) + id_ = _nonempty_symbol(id, "profile plan id") + analyses_ = _symbol_tuple(analyses, "profile analyses") + every = Int(record_every) + every > 0 || throw(ArgumentError("profile record_every must be positive")) + return ProfilePlan(id_, target, analyses_, every) +end + +struct SweepAxis{V<:Tuple} + parameter::Symbol + values::V + + function SweepAxis{V}(parameter::Symbol, values::V) where {V<:Tuple} + isempty(values) && throw(ArgumentError("sweep axis :$(parameter) must not be empty")) + length(unique(values)) == length(values) || throw(ArgumentError( + "sweep axis :$(parameter) values must be unique", + )) + return new{V}(parameter, values) + end +end + +function SweepAxis(parameter::Union{Symbol,AbstractString}, values) + parameter_ = _nonempty_symbol(parameter, "sweep parameter") + values_ = Tuple(values) + isempty(values_) && throw(ArgumentError("sweep axis :$(parameter_) must not be empty")) + length(unique(values_)) == length(values_) || throw(ArgumentError( + "sweep axis :$(parameter_) values must be unique", + )) + return SweepAxis{typeof(values_)}(parameter_, values_) +end + +struct SweepPlan{T<:EvaluationTarget,A<:Tuple} <: AbstractOperationPlan + id::Symbol + target::T + axes::A + mode::Symbol + max_rollouts::Int +end + +function SweepPlan( + id::Union{Symbol,AbstractString}, + target::EvaluationTarget; + axes=(), + mode::Symbol=:factorial, + max_rollouts::Integer=10_000, +) + id_ = _nonempty_symbol(id, "sweep plan id") + axes_ = Tuple(axes) + all(axis -> axis isa SweepAxis, axes_) || throw(ArgumentError( + "sweep axes must all be SweepAxis values", + )) + names = Tuple(axis.parameter for axis in axes_) + length(unique(names)) == length(names) || throw(ArgumentError( + "sweep axis parameters must be unique", + )) + mode in (:factorial, :one_at_a_time) || throw(ArgumentError( + "sweep mode must be :factorial or :one_at_a_time", + )) + limit = Int(max_rollouts) + limit > 0 || throw(ArgumentError("sweep max_rollouts must be positive")) + return SweepPlan(id_, target, axes_, mode, limit) +end + +"""A registered causal intervention, with explicit applicability metadata.""" +struct AblationSpec{A,M} + id::Symbol + apply::A + stage::Symbol + required_capabilities::Tuple{Vararg{Symbol}} + description::String + metadata::M +end + +function AblationSpec( + id::Union{Symbol,AbstractString}, + apply; + stage::Symbol=:composition, + required_capabilities=(), + description::AbstractString="", + metadata::NamedTuple=NamedTuple(), +) + id_ = _nonempty_symbol(id, "ablation id") + stage in (:composition, :reservoir, :task) || throw(ArgumentError( + "ablation :$(id_) stage must be :composition, :reservoir, or :task", + )) + capabilities = _symbol_tuple(required_capabilities, "ablation capabilities") + return AblationSpec{typeof(apply),typeof(metadata)}( + id_, + apply, + stage, + capabilities, + String(description), + metadata, + ) +end + +struct AblationPlan{T<:EvaluationTarget,A<:Tuple} <: AbstractOperationPlan + id::Symbol + target::T + ablations::A +end + + +function AblationPlan( + id::Union{Symbol,AbstractString}, + target::EvaluationTarget; + ablations, +) + id_ = _nonempty_symbol(id, "ablation plan id") + ablations_ = _symbol_tuple(ablations, "ablation cases") + isempty(ablations_) && throw(ArgumentError("ablation plan requires at least one case")) + return AblationPlan(id_, target, ablations_) +end + +struct EvolutionPlan{T<:EvaluationTarget,H<:Tuple} <: AbstractOperationPlan + id::Symbol + training::T + heldout_targets::H + optimizer::Symbol + parameter_set::Symbol + objective::Symbol + generations::Int + popsize::Int + sigma0::Float64 +end + +function EvolutionPlan( + id::Union{Symbol,AbstractString}, + training::EvaluationTarget; + heldout_targets=(), + optimizer::Union{Symbol,AbstractString}=:sepcma, + parameter_set::Union{Symbol,AbstractString}=:evolve, + objective::Union{Symbol,AbstractString}=:normalized_score, + generations::Integer=50, + popsize::Integer=64, + sigma0::Real=0.5, +) + id_ = _nonempty_symbol(id, "evolution plan id") + heldout = Tuple(heldout_targets) + all(target -> target isa EvaluationTarget, heldout) || throw(ArgumentError( + "heldout_targets must all be EvaluationTarget values", + )) + generations_ = Int(generations) + popsize_ = Int(popsize) + sigma = Float64(sigma0) + generations_ > 0 || throw(ArgumentError("evolution generations must be positive")) + popsize_ >= 2 || throw(ArgumentError("evolution popsize must be at least 2")) + isfinite(sigma) && sigma > 0 || throw(ArgumentError( + "evolution sigma0 must be finite and positive", + )) + return EvolutionPlan( + id_, + training, + heldout, + _nonempty_symbol(optimizer, "evolution optimizer"), + _nonempty_symbol(parameter_set, "evolution parameter set"), + _nonempty_symbol(objective, "evolution objective"), + generations_, + popsize_, + sigma, + ) +end + +struct BenchmarkCasePlan{C<:Tuple} + id::Symbol + conditions::C + baseline::Symbol +end + +function BenchmarkCasePlan( + id::Union{Symbol,AbstractString}, + conditions; + baseline::Union{Symbol,AbstractString}, +) + id_ = _nonempty_symbol(id, "benchmark case id") + conditions_ = Tuple(conditions) + isempty(conditions_) && throw(ArgumentError("benchmark case requires conditions")) + all(condition -> condition isa EvaluationTarget, conditions_) || throw(ArgumentError( + "benchmark conditions must all be EvaluationTarget values", + )) + names = Tuple(condition.id for condition in conditions_) + length(unique(names)) == length(names) || throw(ArgumentError( + "benchmark condition ids must be unique within a case", + )) + baseline_ = Symbol(baseline) + baseline_ in names || throw(ArgumentError( + "benchmark baseline :$(baseline_) is not a condition in case :$(id_)", + )) + return BenchmarkCasePlan(id_, conditions_, baseline_) +end + +struct BenchmarkPlan{C<:Tuple} <: AbstractOperationPlan + id::Symbol + cases::C +end + +function BenchmarkPlan(id::Union{Symbol,AbstractString}, cases) + id_ = _nonempty_symbol(id, "benchmark plan id") + cases_ = Tuple(cases) + isempty(cases_) && throw(ArgumentError("benchmark plan requires at least one case")) + all(case -> case isa BenchmarkCasePlan, cases_) || throw(ArgumentError( + "benchmark cases must all be BenchmarkCasePlan values", + )) + names = Tuple(case.id for case in cases_) + length(unique(names)) == length(names) || throw(ArgumentError( + "benchmark case ids must be unique", + )) + return BenchmarkPlan(id_, cases_) +end + +const EXPERIMENT_EVIDENCE_STATES = ( + :exploratory, + :tuned, + :frozen, + :confirmed, + :promoted, + :retired, +) + +"""A citable scientific protocol composed from named conditions and operations.""" +struct ExperimentSpec{C<:Tuple,O<:Tuple,M} + id::Symbol + version::VersionNumber + title::String + question::String + conditions::C + operations::O + evidence_state::Symbol + limitations::Tuple{Vararg{String}} + metadata::M +end + +function ExperimentSpec( + id::Union{Symbol,AbstractString}, + version::VersionNumber; + title::AbstractString, + question::AbstractString, + conditions, + operations, + evidence_state::Symbol=:exploratory, + limitations=(), + metadata::NamedTuple=NamedTuple(), +) + id_ = _nonempty_symbol(id, "experiment id") + isempty(strip(title)) && throw(ArgumentError("experiment title must not be empty")) + isempty(strip(question)) && throw(ArgumentError("experiment question must not be empty")) + conditions_ = Tuple(conditions) + operations_ = Tuple(operations) + isempty(conditions_) && throw(ArgumentError("experiment requires named conditions")) + isempty(operations_) && throw(ArgumentError("experiment requires operations")) + all(condition -> condition isa EvaluationTarget, conditions_) || throw(ArgumentError( + "experiment conditions must all be EvaluationTarget values", + )) + all(operation -> operation isa AbstractOperationPlan, operations_) || throw(ArgumentError( + "experiment operations must all be operation plans", + )) + condition_ids = Tuple(condition.id for condition in conditions_) + length(unique(condition_ids)) == length(condition_ids) || throw(ArgumentError( + "experiment condition ids must be unique", + )) + evidence_state in EXPERIMENT_EVIDENCE_STATES || throw(ArgumentError( + "invalid experiment evidence_state :$(evidence_state)", + )) + limitations_ = Tuple(String(limitation) for limitation in limitations) + return ExperimentSpec{ + typeof(conditions_), + typeof(operations_), + typeof(metadata), + }( + id_, + version, + String(title), + String(question), + conditions_, + operations_, + evidence_state, + limitations_, + metadata, + ) +end + +"""Validate a cold operation plan against one explicit registry set.""" +validate(plan::AbstractOperationPlan, registry::RegistrySet) = plan + +"""Resolve registry names and defaults without executing simulations.""" +resolve(plan::AbstractOperationPlan, registry::RegistrySet) = throw(MethodError(resolve, (plan, registry))) + +"""Execute an already-resolved operation plan.""" +execute(plan::AbstractResolvedOperationPlan) = throw(MethodError(execute, (plan,))) + +"""Return authoritative named tables for a typed operation result.""" +tables(result::AbstractOperationResult) = throw(MethodError(tables, (result,))) + +"""Return the compact derived summary for a typed operation result.""" +summary(result::AbstractOperationResult) = throw(MethodError(summary, (result,))) diff --git a/test/runtests.jl b/test/runtests.jl index 54b6af7..9451334 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -40,6 +40,7 @@ include("test_cartpole_variants.jl") include("test_plank_cartpole.jl") include("test_contract_kernel.jl") include("test_composition_spec.jl") +include("test_operation_plans.jl") include("test_morphology.jl") include("test_homeostasis.jl") include("test_sensor.jl") diff --git a/test/test_operation_plans.jl b/test/test_operation_plans.jl new file mode 100644 index 0000000..84bf0cc --- /dev/null +++ b/test/test_operation_plans.jl @@ -0,0 +1,87 @@ +using BrainlessLab +using Test + +function _plan_target(id, task; blocks=1, trials=2, horizon=20) + composition = default_composition(DEFAULT_REGISTRY, :falandays, task) + evaluation = EvaluationSpec( + blocks=blocks, + trials_per_block=trials, + horizon=horizon, + root_seed=17, + ) + return EvaluationTarget(id, composition, evaluation) +end + +@testset "one operation-plan schema" begin + tracking = _plan_target(:tracking, :tracking) + pong = _plan_target(:pong, :pong) + + profile = ProfilePlan(:profile_tracking, tracking; record_every=2) + @test profile isa AbstractOperationPlan + @test isempty(profile.analyses) + @test profile.record_every == 2 + @test_throws ArgumentError ProfilePlan(:bad, tracking; record_every=0) + + axis = SweepAxis(:leak, (0.1, 0.25, 0.5)) + sweep = SweepPlan(:sweep_tracking, tracking; axes=(axis,), max_rollouts=100) + @test sweep.mode === :factorial + @test sweep.axes[1].values == (0.1, 0.25, 0.5) + @test_throws ArgumentError SweepAxis(:leak, ()) + @test_throws ArgumentError SweepPlan(:bad, tracking; mode=:random) + + ablation = AblationPlan( + :ablate_tracking, + tracking; + ablations=(:freeze_plasticity, :clamp_target), + ) + @test ablation.ablations == (:freeze_plasticity, :clamp_target) + + evolution = EvolutionPlan( + :evolve_tracking, + tracking; + heldout_targets=(pong,), + generations=5, + popsize=24, + ) + @test evolution.parameter_set === :evolve + @test evolution.heldout_targets == (pong,) + @test_throws ArgumentError EvolutionPlan(:bad, tracking; popsize=1) + + tracking_case = BenchmarkCasePlan( + :tracking, + (tracking,); + baseline=:tracking, + ) + pong_case = BenchmarkCasePlan(:pong, (pong,); baseline=:pong) + benchmark = BenchmarkPlan(:core, (tracking_case, pong_case)) + @test Tuple(case.id for case in benchmark.cases) == (:tracking, :pong) + @test !hasproperty(benchmark, :aggregate) + @test_throws ArgumentError BenchmarkCasePlan( + :bad, + (tracking,); + baseline=:missing, + ) + + experiment = ExperimentSpec( + :falandays_cross_task, + v"1.0.0"; + title="Evolve one task, evaluate the other", + question="How does task-specific parameter evolution move performance across the core benchmark?", + conditions=(tracking, pong), + operations=(evolution, benchmark), + evidence_state=:exploratory, + limitations=("Parameter evolution only; node structure is fixed.",), + ) + @test experiment.version == v"1.0.0" + @test experiment.evidence_state === :exploratory + @test_throws ArgumentError ExperimentSpec( + :bad, + v"1.0.0"; + title="Bad", + question="Bad?", + conditions=(tracking,), + operations=(evolution,), + evidence_state=:certain, + ) +end + From 12fa93434d918d8460d34652852ffff8d16696d7 Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 23:05:55 -0400 Subject: [PATCH 07/20] feat: execute evaluations with named trial ledgers --- src/BrainlessLab.jl | 8 ++ src/api/Composition.jl | 30 +++++- src/operations/Evaluation.jl | 189 +++++++++++++++++++++++++++++++++++ test/test_operation_plans.jl | 65 +++++++++++- 4 files changed, 287 insertions(+), 5 deletions(-) create mode 100644 src/operations/Evaluation.jl diff --git a/src/BrainlessLab.jl b/src/BrainlessLab.jl index ee7d0ab..7097ffb 100644 --- a/src/BrainlessLab.jl +++ b/src/BrainlessLab.jl @@ -70,6 +70,7 @@ include("api/Highlevel.jl") include("core/Catalog.jl") include("api/Composition.jl") include("operations/Plans.jl") +include("operations/Evaluation.jl") include("analysis/ActivityLevels.jl") include("analysis/Branching.jl") include("analysis/Avalanches.jl") @@ -693,6 +694,13 @@ export AbstractOperationPlan, tables, summary +export EvaluationTrial, + EvaluationBatch, + evaluate, + realized_initial_state, + trial_row, + trial_table + export SimResult, simulate, variants, diff --git a/src/api/Composition.jl b/src/api/Composition.jl index 07a36cb..1639df9 100644 --- a/src/api/Composition.jl +++ b/src/api/Composition.jl @@ -13,10 +13,24 @@ function _composition_seed_ledger( block::Integer, trial::Integer, agent::Integer, + construction_block::Integer=block, + construction_trial::Integer=trial, ) return ( - topology=derive_seed(evaluation, :topology, block, trial, agent), - node_state=derive_seed(evaluation, :node_state, block, trial, agent), + topology=derive_seed( + evaluation, + :topology, + construction_block, + construction_trial, + agent, + ), + node_state=derive_seed( + evaluation, + :node_state, + construction_block, + construction_trial, + agent, + ), world=derive_seed(evaluation, :world, block, trial), body=derive_seed(evaluation, :body, block, trial, agent), task=derive_seed(evaluation, :task, block, trial), @@ -29,6 +43,8 @@ function _build_composition( evaluation::EvaluationSpec; block::Integer=1, trial::Integer=1, + construction_block::Integer=block, + construction_trial::Integer=trial, record=_DEFAULT_RECORD_CHANNELS, every::Integer=1, ) @@ -50,7 +66,14 @@ function _build_composition( layout = portspec(body_at_slot) default_link_p = Float64(get(resolved.parameters, :link_p, 0.1)) profile = receptor_link_profile(body_at_slot, default_link_p) - seeds = _composition_seed_ledger(evaluation, block, trial, slot) + seeds = _composition_seed_ledger( + evaluation, + block, + trial, + slot, + construction_block, + construction_trial, + ) context = NodeBuildContext( resolved.n_nodes, layout, @@ -146,4 +169,3 @@ simulate( registry::RegistrySet; kwargs..., ) = simulate(composition_spec(registry, composition); registry=registry, kwargs...) - diff --git a/src/operations/Evaluation.jl b/src/operations/Evaluation.jl new file mode 100644 index 0000000..6c1f2c3 --- /dev/null +++ b/src/operations/Evaluation.jl @@ -0,0 +1,189 @@ +"""Explicit initial state retained for audit when an environment exposes one.""" +realized_initial_state(::Environment) = nothing +realized_initial_state(environment::PlankCartPoleEnv) = Tuple(environment.state) + +struct EvaluationTrial{S<:SimResult,I,L} + condition::Symbol + block::Int + trial::Int + seeds::L + initial_state::I + simulation::S +end + +struct EvaluationBatch{T<:EvaluationTarget,R<:ResolvedComposition,E<:Tuple} + target::T + resolved::R + trials::E +end + +function _construction_coordinates( + evaluation::EvaluationSpec, + block::Integer, + trial::Integer, +) + evaluation.construction_scope === :evaluation && return (0, 0) + evaluation.construction_scope === :block && return (Int(block), 0) + return (Int(block), Int(trial)) +end + +function _trial_liveness(metrics) + hasproperty(metrics, :alive) && return Bool(getproperty(metrics, :alive)) + hasproperty(metrics, :liveness) && return Bool(getproperty(metrics, :liveness)) + return missing +end + +function _trial_viability(metrics) + hasproperty(metrics, :viable) && return Bool(getproperty(metrics, :viable)) + hasproperty(metrics, :achieved) && return Bool(getproperty(metrics, :achieved)) + return missing +end + +function _evaluate_trial( + target::EvaluationTarget, + resolved::ResolvedComposition, + block::Integer, + trial::Integer; + record=(), + record_every::Integer=1, + metrics=nothing, +) + evaluation = target.evaluation + construction_block, construction_trial = _construction_coordinates( + evaluation, + block, + trial, + ) + setup = _build_composition( + resolved, + evaluation; + block=block, + trial=trial, + construction_block=construction_block, + construction_trial=construction_trial, + record=record, + every=record_every, + ) + initial_state = realized_initial_state(setup.ensemble.environment) + if evaluation.warmup > 0 + recorder = setup.ensemble.recorder + setup.ensemble.recorder = nothing + rollout!(setup.ensemble, evaluation.warmup; window=evaluation.warmup) + setup.ensemble.recorder = recorder + reset!(recorder) + end + scored_ticks = evaluation.horizon - evaluation.warmup + outcome = rollout!( + setup.ensemble, + scored_ticks; + window=scored_ticks, + metrics=metrics, + ) + base_config = _simulation_config( + setup.ensemble; + ticks=evaluation.horizon, + seed=Int(evaluation.root_seed), + record=_record_symbols(record), + every=Int(record_every), + window=scored_ticks, + n_nodes=resolved.n_nodes, + ablation=:none, + ablation_notes=(), + interventions=nothing, + task_spec=resolved.task, + ) + config = merge( + base_config, + ( + composition=target.composition.id, + condition=target.id, + block=Int(block), + trial=Int(trial), + parameters=_composition_namedtuple(resolved.parameters), + seed_ledger=setup.seed_ledger, + evaluation=evaluation, + ), + ) + simulation = SimResult( + setup.recorder, + outcome, + resolved.task.name, + resolved.node.id, + config, + ) + return EvaluationTrial( + target.id, + Int(block), + Int(trial), + setup.seed_ledger, + initial_state, + simulation, + ) +end + +""" + evaluate(target; registry=DEFAULT_REGISTRY, ...) + +Execute every declared block and trial, retaining each raw `SimResult`, named +seed ledger, and realized initial state. `:full` reset is implemented by a +fresh runtime construction whose topology/node-state seeds respect the chosen +construction scope. Stateful `:body_environment` and `:none` policies are +rejected until the composed task declares the corresponding reset hooks. +""" +function evaluate( + target::EvaluationTarget; + registry::RegistrySet=DEFAULT_REGISTRY, + record=(), + record_every::Integer=1, + metrics=nothing, +) + evaluation = target.evaluation + evaluation.reset === :full || throw(ArgumentError( + "generic evaluation currently requires reset=:full; task-specific state retention " * + "must be exposed through a declared reset hook before using :$(evaluation.reset)", + )) + resolved = resolve_composition(target.composition, registry) + count = evaluation.blocks * evaluation.trials_per_block + trial_results = Vector{EvaluationTrial}(undef, count) + index = 1 + for block in 1:evaluation.blocks + for trial in 1:evaluation.trials_per_block + trial_results[index] = _evaluate_trial( + target, + resolved, + block, + trial; + record=record, + record_every=record_every, + metrics=metrics, + ) + index += 1 + end + end + return EvaluationBatch(target, resolved, Tuple(trial_results)) +end + +function trial_row(trial::EvaluationTrial) + outcome = task_outcome(trial.simulation) + first_ledger = first(trial.seeds) + return ( + condition=trial.condition, + block=trial.block, + trial=trial.trial, + topology_seed=first_ledger.topology, + node_state_seed=first_ledger.node_state, + world_seed=first_ledger.world, + body_seed=first_ledger.body, + task_seed=first_ledger.task, + mechanism_seed=first_ledger.mechanism, + initial_state=trial.initial_state, + score_key=outcome === nothing ? missing : outcome.key, + raw_score=outcome === nothing ? missing : outcome.raw, + normalized_score=outcome === nothing ? missing : outcome.normalized, + viable=_trial_viability(trial.simulation.metrics), + liveness=_trial_liveness(trial.simulation.metrics), + ) +end + +trial_table(batch::EvaluationBatch) = [trial_row(trial) for trial in batch.trials] + diff --git a/test/test_operation_plans.jl b/test/test_operation_plans.jl index 84bf0cc..552967d 100644 --- a/test/test_operation_plans.jl +++ b/test/test_operation_plans.jl @@ -12,6 +12,70 @@ function _plan_target(id, task; blocks=1, trials=2, horizon=20) return EvaluationTarget(id, composition, evaluation) end +@testset "evaluation blocks retain raw trials and named seeds" begin + composition = CompositionSpec( + :tracking_evaluation_smoke, + :falandays, + :tracking; + n_nodes=8, + parameters=Dict( + :input_weight => 0.75, + :lrate_wmat => 1.0, + :lrate_targ => 0.01, + :weight_init_mode => :excitatory, + :rectify => false, + :repair_masks => false, + ), + ) + evaluation = EvaluationSpec( + blocks=2, + trials_per_block=2, + horizon=4, + construction_scope=:block, + root_seed=91, + aggregate=:none, + ) + batch = evaluate(EvaluationTarget(:tracking, composition, evaluation)) + rows = trial_table(batch) + @test length(batch.trials) == 4 + @test length(rows) == 4 + @test rows[1].topology_seed == rows[2].topology_seed + @test rows[1].topology_seed != rows[3].topology_seed + @test rows[1].world_seed != rows[2].world_seed + @test all(row -> row.score_key === :track_score, rows) + @test all(row -> isfinite(row.raw_score), rows) + + unsupported = EvaluationTarget( + :unsupported_reset, + composition, + EvaluationSpec(horizon=4, reset=:body_environment), + ) + @test_throws ArgumentError evaluate(unsupported) +end + +@testset "Plank evaluation records explicit starts under one fixed design" begin + composition = CompositionSpec( + :plank_easy_smoke, + :falandays, + :cartpole_plank_easy; + n_nodes=8, + ) + evaluation = EvaluationSpec( + blocks=1, + trials_per_block=2, + horizon=2, + construction_scope=:evaluation, + root_seed=101, + aggregate=:mean, + ) + rows = trial_table(evaluate(EvaluationTarget(:plank_easy, composition, evaluation))) + @test length(rows) == 2 + @test rows[1].topology_seed == rows[2].topology_seed + @test rows[1].initial_state isa NTuple{4,Float64} + @test rows[1].initial_state != rows[2].initial_state + @test rows[1].score_key === :fitness +end + @testset "one operation-plan schema" begin tracking = _plan_target(:tracking, :tracking) pong = _plan_target(:pong, :pong) @@ -84,4 +148,3 @@ end evidence_state=:certain, ) end - From 77252b2a506df7b61e81b9e2a2e015a155caec9d Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 23:06:22 -0400 Subject: [PATCH 08/20] fix: populate typed analysis catalog --- src/core/Catalog.jl | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/src/core/Catalog.jl b/src/core/Catalog.jl index 4c276e0..3fcc8ce 100644 --- a/src/core/Catalog.jl +++ b/src/core/Catalog.jl @@ -314,6 +314,18 @@ function register_builtins!(registry::RegistrySet) for (id, implementation) in sort!(collect(METRICS); by=pair -> string(first(pair))) register!(registry, :metrics, ImplementationSpec(id, implementation)) end + for (id, entry) in sort!(collect(ANALYSES); by=pair -> string(first(pair))) + register!( + registry, + :analyses, + ImplementationSpec( + id, + entry.f; + label=entry.label, + metadata=(task=entry.task,), + ), + ) + end for (id, implementation) in sort!(collect(VIEWS); by=pair -> string(first(pair))) register!(registry, :views, ImplementationSpec(id, implementation)) end @@ -331,4 +343,3 @@ function register_builtins!(registry::RegistrySet) end const DEFAULT_REGISTRY = RegistrySet() - From 4576d0d88e240191a4dc1035489f52bc10118d4e Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 23:13:38 -0400 Subject: [PATCH 09/20] feat: add paired core benchmark executor --- src/BrainlessLab.jl | 6 +- src/operations/Benchmark.jl | 192 ++++++++++++++++++++++++++++++++++++ test/runtests.jl | 1 + test/test_benchmark_plan.jl | 71 +++++++++++++ 4 files changed, 269 insertions(+), 1 deletion(-) create mode 100644 src/operations/Benchmark.jl create mode 100644 test/test_benchmark_plan.jl diff --git a/src/BrainlessLab.jl b/src/BrainlessLab.jl index 7097ffb..7ef019d 100644 --- a/src/BrainlessLab.jl +++ b/src/BrainlessLab.jl @@ -12,7 +12,7 @@ Makie-free. """ module BrainlessLab -import Base: view +import Base: summary, view include("core/Interfaces.jl") include("core/Traits.jl") @@ -71,6 +71,7 @@ include("core/Catalog.jl") include("api/Composition.jl") include("operations/Plans.jl") include("operations/Evaluation.jl") +include("operations/Benchmark.jl") include("analysis/ActivityLevels.jl") include("analysis/Branching.jl") include("analysis/Avalanches.jl") @@ -701,6 +702,9 @@ export EvaluationTrial, trial_row, trial_table +export ResolvedBenchmarkPlan, + BenchmarkResult + export SimResult, simulate, variants, diff --git a/src/operations/Benchmark.jl b/src/operations/Benchmark.jl new file mode 100644 index 0000000..8ecf7db --- /dev/null +++ b/src/operations/Benchmark.jl @@ -0,0 +1,192 @@ +struct ResolvedBenchmarkCase{C<:Tuple} + id::Symbol + conditions::C + baseline::Symbol +end + +struct ResolvedBenchmarkPlan{C<:Tuple} <: AbstractResolvedOperationPlan + id::Symbol + cases::C + registry::RegistrySet +end + +struct BenchmarkResult{P<:ResolvedBenchmarkPlan,B<:Tuple} <: AbstractOperationResult + plan::P + batches::B +end + +function _validate_benchmark_case(case::BenchmarkCasePlan, registry::RegistrySet) + baseline = only(condition for condition in case.conditions if condition.id === case.baseline) + reference = baseline.evaluation + for condition in case.conditions + resolve_composition(condition.composition, registry) + evaluation = condition.evaluation + evaluation.blocks == reference.blocks || throw(ArgumentError( + "benchmark case :$(case.id) conditions must use the same block count", + )) + evaluation.trials_per_block == reference.trials_per_block || throw(ArgumentError( + "benchmark case :$(case.id) conditions must use the same trials_per_block", + )) + evaluation.root_seed == reference.root_seed || throw(ArgumentError( + "benchmark case :$(case.id) conditions must share a root_seed for pairing", + )) + end + return case +end + +function validate(plan::BenchmarkPlan, registry::RegistrySet) + foreach(case -> _validate_benchmark_case(case, registry), plan.cases) + return plan +end + +function resolve(plan::BenchmarkPlan, registry::RegistrySet) + validate(plan, registry) + cases = Tuple( + ResolvedBenchmarkCase( + case.id, + Tuple( + ( + target=condition, + composition=resolve_composition(condition.composition, registry), + ) + for condition in case.conditions + ), + case.baseline, + ) + for case in plan.cases + ) + return ResolvedBenchmarkPlan(plan.id, cases, registry) +end + +function execute(plan::ResolvedBenchmarkPlan) + batches = Tuple( + ( + case=case.id, + baseline=case.baseline, + conditions=Tuple( + ( + id=condition.target.id, + batch=evaluate(condition.target; registry=plan.registry), + ) + for condition in case.conditions + ), + ) + for case in plan.cases + ) + return BenchmarkResult(plan, batches) +end + +execute(plan::BenchmarkPlan; registry::RegistrySet=DEFAULT_REGISTRY) = + execute(resolve(plan, registry)) + +function _benchmark_trial_rows(result::BenchmarkResult) + rows = NamedTuple[] + for case in result.batches + for condition in case.conditions + for row in trial_table(condition.batch) + push!(rows, merge((case=case.case,), row)) + end + end + end + return rows +end + +function _benchmark_mean_std(values) + data = Float64[value for value in values if !ismissing(value)] + isempty(data) && return (mean=missing, std=missing, n=0, lower=missing, upper=missing) + mean_value = sum(data) / length(data) + std_value = length(data) > 1 ? sqrt(sum((value - mean_value)^2 for value in data) / (length(data) - 1)) : 0.0 + half_width = length(data) > 1 ? 1.96 * std_value / sqrt(length(data)) : 0.0 + return ( + mean=mean_value, + std=std_value, + n=length(data), + lower=mean_value - half_width, + upper=mean_value + half_width, + ) +end + +function _benchmark_statistics(rows) + groups = Dict{Tuple{Symbol,Symbol},Vector{NamedTuple}}() + for row in rows + push!(get!(groups, (row.case, row.condition), NamedTuple[]), row) + end + output = NamedTuple[] + for ((case, condition), group) in sort!(collect(groups); by=pair -> string(first(pair))) + raw = _benchmark_mean_std(row.raw_score for row in group) + normalized = _benchmark_mean_std(row.normalized_score for row in group) + push!(output, ( + case=case, + condition=condition, + n=normalized.n, + raw_mean=raw.mean, + raw_std=raw.std, + normalized_mean=normalized.mean, + normalized_std=normalized.std, + normalized_ci_lower=normalized.lower, + normalized_ci_upper=normalized.upper, + )) + end + return output +end + +function _benchmark_contrasts(result::BenchmarkResult) + output = NamedTuple[] + for case in result.batches + condition_rows = Dict( + condition.id => Dict( + (row.block, row.trial) => row + for row in trial_table(condition.batch) + ) + for condition in case.conditions + ) + baseline_rows = condition_rows[case.baseline] + for condition in case.conditions + condition.id === case.baseline && continue + differences = Float64[] + raw_differences = Float64[] + for key in sort!(collect(keys(baseline_rows))) + baseline = baseline_rows[key] + candidate = condition_rows[condition.id][key] + if !ismissing(baseline.normalized_score) && !ismissing(candidate.normalized_score) + push!(differences, candidate.normalized_score - baseline.normalized_score) + end + if !ismissing(baseline.raw_score) && !ismissing(candidate.raw_score) + push!(raw_differences, candidate.raw_score - baseline.raw_score) + end + end + normalized = _benchmark_mean_std(differences) + raw = _benchmark_mean_std(raw_differences) + push!(output, ( + case=case.case, + condition=condition.id, + baseline=case.baseline, + n=normalized.n, + raw_difference=raw.mean, + normalized_difference=normalized.mean, + normalized_ci_lower=normalized.lower, + normalized_ci_upper=normalized.upper, + )) + end + end + return output +end + +function tables(result::BenchmarkResult) + trials = _benchmark_trial_rows(result) + return ( + trials=trials, + statistics=_benchmark_statistics(trials), + contrasts=_benchmark_contrasts(result), + ) +end + +function summary(result::BenchmarkResult) + result_tables = tables(result) + return ( + benchmark=result.plan.id, + cases=Tuple(case.id for case in result.plan.cases), + statistics=result_tables.statistics, + contrasts=result_tables.contrasts, + ) +end diff --git a/test/runtests.jl b/test/runtests.jl index 9451334..a6fe4c0 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -41,6 +41,7 @@ include("test_plank_cartpole.jl") include("test_contract_kernel.jl") include("test_composition_spec.jl") include("test_operation_plans.jl") +include("test_benchmark_plan.jl") include("test_morphology.jl") include("test_homeostasis.jl") include("test_sensor.jl") diff --git a/test/test_benchmark_plan.jl b/test/test_benchmark_plan.jl new file mode 100644 index 0000000..c93dc72 --- /dev/null +++ b/test/test_benchmark_plan.jl @@ -0,0 +1,71 @@ +using BrainlessLab +using Test + +function _benchmark_target(id, task; leak=0.25, root_seed=55) + config = falandays_paper_config(task) + composition = CompositionSpec( + Symbol(id, :_composition), + :falandays, + task; + n_nodes=8, + parameters=Dict( + :leak => Float64(leak), + :input_weight => config.input_amp, + :lrate_wmat => config.lrate_wmat, + :lrate_targ => config.lrate_targ, + :weight_init_mode => config.weight_init_mode, + :rectify => false, + :repair_masks => false, + ), + ) + return EvaluationTarget( + id, + composition, + EvaluationSpec( + blocks=1, + trials_per_block=2, + horizon=3, + root_seed=root_seed, + aggregate=:none, + ), + ) +end + +@testset "benchmark keeps tasks separate and comparisons paired" begin + tracking_base = _benchmark_target(:tracking_base, :tracking) + tracking_leak = _benchmark_target(:tracking_leak, :tracking; leak=0.5) + pong_base = _benchmark_target(:pong_base, :pong) + plan = BenchmarkPlan( + :core_smoke, + ( + BenchmarkCasePlan( + :tracking, + (tracking_base, tracking_leak); + baseline=:tracking_base, + ), + BenchmarkCasePlan(:pong, (pong_base,); baseline=:pong_base), + ), + ) + result = execute(plan) + result_tables = tables(result) + @test result isa BenchmarkResult + @test length(result_tables.trials) == 6 + @test Set(row.case for row in result_tables.statistics) == Set((:tracking, :pong)) + @test length(result_tables.contrasts) == 1 + @test result_tables.contrasts[1].condition === :tracking_leak + @test result_tables.contrasts[1].n == 2 + @test summary(result).cases == (:tracking, :pong) + @test !hasproperty(summary(result), :aggregate) + + unpaired = _benchmark_target(:unpaired, :tracking; root_seed=56) + bad = BenchmarkPlan( + :bad, + (BenchmarkCasePlan( + :tracking, + (tracking_base, unpaired); + baseline=:tracking_base, + ),), + ) + @test_throws ArgumentError resolve(bad, DEFAULT_REGISTRY) +end + From ef1daa7fcd4811330b40a8dda5fd7e6ba9d0769e Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 23:18:48 -0400 Subject: [PATCH 10/20] feat: add version-one operation plan schema --- src/BrainlessLab.jl | 7 + src/records/PlanIO.jl | 347 ++++++++++++++++++++++++++++++++++++++++++ test/runtests.jl | 1 + test/test_plan_io.jl | 78 ++++++++++ 4 files changed, 433 insertions(+) create mode 100644 src/records/PlanIO.jl create mode 100644 test/test_plan_io.jl diff --git a/src/BrainlessLab.jl b/src/BrainlessLab.jl index 7ef019d..d993504 100644 --- a/src/BrainlessLab.jl +++ b/src/BrainlessLab.jl @@ -72,6 +72,7 @@ include("api/Composition.jl") include("operations/Plans.jl") include("operations/Evaluation.jl") include("operations/Benchmark.jl") +include("records/PlanIO.jl") include("analysis/ActivityLevels.jl") include("analysis/Branching.jl") include("analysis/Avalanches.jl") @@ -705,6 +706,12 @@ export EvaluationTrial, export ResolvedBenchmarkPlan, BenchmarkResult +export PLAN_FORMAT, + PLAN_FORMAT_VERSION, + plan_document, + read_plan, + write_plan + export SimResult, simulate, variants, diff --git a/src/records/PlanIO.jl b/src/records/PlanIO.jl new file mode 100644 index 0000000..d29868c --- /dev/null +++ b/src/records/PlanIO.jl @@ -0,0 +1,347 @@ +using TOML + +const PLAN_FORMAT = "brainlesslab-plan" +const PLAN_FORMAT_VERSION = 1 + +_plan_toml_value(value::Symbol) = String(value) +_plan_toml_value(value::Tuple) = [_plan_toml_value(item) for item in value] +_plan_toml_value(value::AbstractVector) = [_plan_toml_value(item) for item in value] +_plan_toml_value(value::UInt64) = value <= UInt64(typemax(Int64)) ? Int64(value) : string(value) +_plan_toml_value(value) = value + +function _string_dict(values) + return Dict{String,Any}(string(key) => _plan_toml_value(value) for (key, value) in pairs(values)) +end + +function _require_document_keys(document, allowed, context) + unknown = sort!(collect(setdiff(Set(keys(document)), Set(allowed)))) + isempty(unknown) || throw(ArgumentError( + "unknown $(context) keys: $(join(unknown, ", "))", + )) + return document +end + +function _composition_document(composition::CompositionSpec) + document = Dict{String,Any}( + "id" => String(composition.id), + "node" => String(composition.node), + "task" => String(composition.task), + "n_nodes" => composition.n_nodes, + ) + composition.body === nothing || (document["body"] = String(composition.body)) + composition.n_agents === nothing || (document["n_agents"] = composition.n_agents) + isempty(composition.parameters) || + (document["parameters"] = _string_dict(composition.parameters)) + isempty(composition.task_options) || + (document["task_options"] = _string_dict(composition.task_options)) + isempty(composition.body_options) || + (document["body_options"] = _string_dict(composition.body_options)) + return document +end + +function _evaluation_document(evaluation::EvaluationSpec) + return Dict{String,Any}( + "blocks" => evaluation.blocks, + "trials_per_block" => evaluation.trials_per_block, + "horizon" => evaluation.horizon, + "warmup" => evaluation.warmup, + "construction_scope" => String(evaluation.construction_scope), + "reset" => String(evaluation.reset), + "root_seed" => evaluation.root_seed, + "streams" => collect(String.(seed_stream_names(evaluation))), + "aggregate" => String(evaluation.aggregate), + ) +end + +function _target_document(target::EvaluationTarget) + return Dict{String,Any}( + "id" => String(target.id), + "composition" => _composition_document(target.composition), + "evaluation" => _evaluation_document(target.evaluation), + ) +end + +function _base_plan_document(plan, operation::Symbol, targets) + return Dict{String,Any}( + "format" => PLAN_FORMAT, + "format_version" => PLAN_FORMAT_VERSION, + "operation" => String(operation), + "id" => String(plan.id), + "targets" => [_target_document(target) for target in targets], + ) +end + +function plan_document(plan::ProfilePlan) + document = _base_plan_document(plan, :profile, (plan.target,)) + document["profile"] = Dict{String,Any}( + "target" => String(plan.target.id), + "analyses" => collect(String.(plan.analyses)), + "record_every" => plan.record_every, + ) + return document +end + +function plan_document(plan::SweepPlan) + document = _base_plan_document(plan, :sweep, (plan.target,)) + document["sweep"] = Dict{String,Any}( + "target" => String(plan.target.id), + "mode" => String(plan.mode), + "max_rollouts" => plan.max_rollouts, + "axes" => [ + Dict{String,Any}( + "parameter" => String(axis.parameter), + "values" => collect(axis.values), + ) + for axis in plan.axes + ], + ) + return document +end + +function plan_document(plan::AblationPlan) + document = _base_plan_document(plan, :ablate, (plan.target,)) + document["ablate"] = Dict{String,Any}( + "target" => String(plan.target.id), + "ablations" => collect(String.(plan.ablations)), + ) + return document +end + +function plan_document(plan::EvolutionPlan) + targets = (plan.training, plan.heldout_targets...) + document = _base_plan_document(plan, :evolve, targets) + document["evolve"] = Dict{String,Any}( + "training" => String(plan.training.id), + "heldout" => String[String(target.id) for target in plan.heldout_targets], + "optimizer" => String(plan.optimizer), + "parameter_set" => String(plan.parameter_set), + "objective" => String(plan.objective), + "generations" => plan.generations, + "popsize" => plan.popsize, + "sigma0" => plan.sigma0, + ) + return document +end + +function plan_document(plan::BenchmarkPlan) + targets = EvaluationTarget[] + seen = Set{Symbol}() + for case in plan.cases, target in case.conditions + target.id in seen && continue + push!(targets, target) + push!(seen, target.id) + end + document = _base_plan_document(plan, :benchmark, Tuple(targets)) + document["benchmark"] = Dict{String,Any}( + "cases" => [ + Dict{String,Any}( + "id" => String(case.id), + "conditions" => String[String(target.id) for target in case.conditions], + "baseline" => String(case.baseline), + ) + for case in plan.cases + ], + ) + return document +end + +function write_plan(path::AbstractString, plan::AbstractOperationPlan) + open(path, "w") do io + TOML.print(io, plan_document(plan); sorted=true) + end + return String(path) +end + +function _parse_composition(document, registry::RegistrySet) + _require_document_keys( + document, + ("id", "preset", "node", "task", "body", "n_agents", "n_nodes", "parameters", "task_options", "body_options"), + "composition", + ) + parameters = Dict{Symbol,Any}( + Symbol(key) => value + for (key, value) in get(document, "parameters", Dict{String,Any}()) + ) + task_options = Dict{Symbol,Any}( + Symbol(key) => value + for (key, value) in get(document, "task_options", Dict{String,Any}()) + ) + body_options = Dict{Symbol,Any}( + Symbol(key) => value + for (key, value) in get(document, "body_options", Dict{String,Any}()) + ) + node_id = haskey(document, "preset") ? + composition_spec(registry, Symbol(document["preset"])).node : + (haskey(document, "node") ? Symbol(document["node"]) : nothing) + if node_id !== nothing + spec = node_spec(registry, node_id) + for parameter in spec.parameters + haskey(parameters, parameter.name) || continue + if parameter.datatype === Symbol && parameters[parameter.name] isa AbstractString + parameters[parameter.name] = Symbol(parameters[parameter.name]) + end + end + end + if haskey(document, "preset") + allowed_inline = intersect(Set(keys(document)), Set(("node", "task", "n_nodes", "body", "n_agents"))) + isempty(allowed_inline) || throw(ArgumentError( + "composition preset cannot be combined with structural keys $(sort!(collect(allowed_inline)))", + )) + base = composition_spec(registry, Symbol(document["preset"])) + return CompositionSpec( + Symbol(get(document, "id", base.id)), + base.node, + base.task; + body=base.body, + n_agents=base.n_agents, + n_nodes=base.n_nodes, + parameters=merge(copy(base.parameters), parameters), + task_options=merge(copy(base.task_options), task_options), + body_options=merge(copy(base.body_options), body_options), + interaction_cycle=base.interaction_cycle, + ) + end + for key in ("id", "node", "task", "n_nodes") + haskey(document, key) || throw(ArgumentError("inline composition requires $(key)")) + end + return CompositionSpec( + Symbol(document["id"]), + Symbol(document["node"]), + Symbol(document["task"]); + body=haskey(document, "body") ? Symbol(document["body"]) : nothing, + n_agents=get(document, "n_agents", nothing), + n_nodes=document["n_nodes"], + parameters=parameters, + task_options=task_options, + body_options=body_options, + ) +end + +function _parse_evaluation(document) + _require_document_keys( + document, + ("blocks", "trials_per_block", "horizon", "warmup", "construction_scope", "reset", "root_seed", "streams", "aggregate"), + "evaluation", + ) + haskey(document, "horizon") || throw(ArgumentError("evaluation requires horizon")) + root_seed_value = get(document, "root_seed", 0) + root_seed = root_seed_value isa AbstractString ? parse(UInt64, root_seed_value) : root_seed_value + return EvaluationSpec( + blocks=get(document, "blocks", 1), + trials_per_block=get(document, "trials_per_block", 1), + horizon=document["horizon"], + warmup=get(document, "warmup", 0), + construction_scope=Symbol(get(document, "construction_scope", "trial")), + reset=Symbol(get(document, "reset", "full")), + root_seed=root_seed, + streams=Tuple(Symbol(stream) for stream in get(document, "streams", String.(getfield.(DEFAULT_SEED_STREAMS, :name)))), + aggregate=Symbol(get(document, "aggregate", "mean")), + ) +end + +function _parse_targets(document, registry::RegistrySet) + targets = Dict{Symbol,EvaluationTarget}() + for entry in document + _require_document_keys(entry, ("id", "composition", "evaluation"), "target") + for key in ("id", "composition", "evaluation") + haskey(entry, key) || throw(ArgumentError("target requires $(key)")) + end + id = Symbol(entry["id"]) + haskey(targets, id) && throw(ArgumentError("duplicate target :$(id)")) + targets[id] = EvaluationTarget( + id, + _parse_composition(entry["composition"], registry), + _parse_evaluation(entry["evaluation"]), + ) + end + isempty(targets) && throw(ArgumentError("plan requires at least one target")) + return targets +end + +function _target(targets, name) + id = Symbol(name) + haskey(targets, id) || throw(KeyError("unknown plan target :$(id)")) + return targets[id] +end + +function read_plan(path::AbstractString; registry::RegistrySet=DEFAULT_REGISTRY) + document = TOML.parsefile(path) + get(document, "format", nothing) == PLAN_FORMAT || throw(ArgumentError( + "plan format must be $(repr(PLAN_FORMAT))", + )) + get(document, "format_version", nothing) == PLAN_FORMAT_VERSION || throw(ArgumentError( + "plan format_version must be $(PLAN_FORMAT_VERSION)", + )) + haskey(document, "operation") || throw(ArgumentError("plan requires operation")) + haskey(document, "id") || throw(ArgumentError("plan requires id")) + haskey(document, "targets") || throw(ArgumentError("plan requires targets")) + operation = Symbol(document["operation"]) + section_name = String(operation) + allowed = ("format", "format_version", "operation", "id", "targets", section_name) + _require_document_keys(document, allowed, "plan") + haskey(document, section_name) || throw(ArgumentError( + "plan operation :$(operation) requires [$(section_name)]", + )) + id = Symbol(document["id"]) + targets = _parse_targets(document["targets"], registry) + section = document[section_name] + + if operation === :profile + _require_document_keys(section, ("target", "analyses", "record_every"), "profile") + return ProfilePlan( + id, + _target(targets, section["target"]); + analyses=Symbol.(get(section, "analyses", String[])), + record_every=get(section, "record_every", 1), + ) + elseif operation === :sweep + _require_document_keys(section, ("target", "axes", "mode", "max_rollouts"), "sweep") + axes = Tuple( + SweepAxis(Symbol(axis["parameter"]), Tuple(axis["values"])) + for axis in get(section, "axes", Any[]) + ) + return SweepPlan( + id, + _target(targets, section["target"]); + axes=axes, + mode=Symbol(get(section, "mode", "factorial")), + max_rollouts=get(section, "max_rollouts", 10_000), + ) + elseif operation === :ablate + _require_document_keys(section, ("target", "ablations"), "ablate") + return AblationPlan( + id, + _target(targets, section["target"]); + ablations=Symbol.(section["ablations"]), + ) + elseif operation === :evolve + _require_document_keys( + section, + ("training", "heldout", "optimizer", "parameter_set", "objective", "generations", "popsize", "sigma0"), + "evolve", + ) + return EvolutionPlan( + id, + _target(targets, section["training"]); + heldout_targets=Tuple(_target(targets, name) for name in get(section, "heldout", String[])), + optimizer=Symbol(get(section, "optimizer", "sepcma")), + parameter_set=Symbol(get(section, "parameter_set", "evolve")), + objective=Symbol(get(section, "objective", "normalized_score")), + generations=get(section, "generations", 50), + popsize=get(section, "popsize", 64), + sigma0=get(section, "sigma0", 0.5), + ) + elseif operation === :benchmark + _require_document_keys(section, ("cases",), "benchmark") + cases = Tuple(begin + _require_document_keys(case, ("id", "conditions", "baseline"), "benchmark case") + BenchmarkCasePlan( + Symbol(case["id"]), + Tuple(_target(targets, name) for name in case["conditions"]); + baseline=Symbol(case["baseline"]), + ) + end for case in section["cases"]) + return BenchmarkPlan(id, cases) + end + throw(ArgumentError("unsupported plan operation :$(operation)")) +end diff --git a/test/runtests.jl b/test/runtests.jl index a6fe4c0..3845d36 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -42,6 +42,7 @@ include("test_contract_kernel.jl") include("test_composition_spec.jl") include("test_operation_plans.jl") include("test_benchmark_plan.jl") +include("test_plan_io.jl") include("test_morphology.jl") include("test_homeostasis.jl") include("test_sensor.jl") diff --git a/test/test_plan_io.jl b/test/test_plan_io.jl new file mode 100644 index 0000000..db39acc --- /dev/null +++ b/test/test_plan_io.jl @@ -0,0 +1,78 @@ +using BrainlessLab +using Test + +function _io_target(id, task) + return EvaluationTarget( + id, + default_composition(DEFAULT_REGISTRY, :falandays, task), + EvaluationSpec( + blocks=2, + trials_per_block=3, + horizon=12, + warmup=1, + construction_scope=:block, + root_seed=44, + aggregate=:none, + ), + ) +end + +@testset "version-one TOML plan round trips" begin + target = _io_target(:tracking, :tracking) + plans = ( + ProfilePlan(:profile, target; analyses=(:branching_ratio_mr,), record_every=2), + SweepPlan( + :sweep, + target; + axes=(SweepAxis(:leak, (0.1, 0.5)),), + mode=:one_at_a_time, + max_rollouts=100, + ), + AblationPlan(:ablate, target; ablations=(:freeze_plasticity,)), + EvolutionPlan( + :evolve, + target; + heldout_targets=(_io_target(:pong, :pong),), + generations=2, + popsize=4, + ), + BenchmarkPlan( + :benchmark, + ( + BenchmarkCasePlan(:tracking, (target,); baseline=:tracking), + BenchmarkCasePlan( + :pong, + (_io_target(:pong, :pong),); + baseline=:pong, + ), + ), + ), + ) + for plan in plans + path = tempname() * ".toml" + write_plan(path, plan) + parsed = read_plan(path) + @test typeof(parsed).name.wrapper === typeof(plan).name.wrapper + @test parsed.id === plan.id + @test plan_document(parsed)["format_version"] == 1 + end +end + +@testset "plan parser rejects unknown schema" begin + path = tempname() * ".toml" + open(path, "w") do io + write(io, """ +format = "brainlesslab-plan" +format_version = 1 +operation = "profile" +id = "bad" +unknown = true +targets = [] + +[profile] +target = "missing" +""") + end + @test_throws ArgumentError read_plan(path) +end + From 1e4d55a40fdee4caa369e02807c14ad1d2afe4dc Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 23:16:45 -0400 Subject: [PATCH 11/20] feat: implement typed profile operation --- src/operations/Profile.jl | 401 ++++++++++++++++++++++++++++++++++++++ test/test_profile_plan.jl | 127 ++++++++++++ 2 files changed, 528 insertions(+) create mode 100644 src/operations/Profile.jl create mode 100644 test/test_profile_plan.jl diff --git a/src/operations/Profile.jl b/src/operations/Profile.jl new file mode 100644 index 0000000..855c5e2 --- /dev/null +++ b/src/operations/Profile.jl @@ -0,0 +1,401 @@ +const _PROFILE_BUILTIN_CHANNELS = Dict{Symbol,Tuple{Vararg{Symbol}}}( + :branching_ratio => (:rate,), + :branching_ratio_mr => (:rate,), + :branching_ratio_mr_windowed => (:rate,), + :branching_ratio_mr_conditioned => (:rate, :percepts), + :avalanches => (:spikes,), + :node_transfer_entropy => (:spikes,), + :agent_transfer_entropy => (:poses,), + :node_target_error => (:acts, :targets), + :spectral_radius => (:spectral_radius,), + :susceptibility => (:spikes,), + :susceptibility_windowed => (:spikes,), + :fano_factor => (:spikes,), + :participation_ratio => (:spikes,), + :swarm_regime => (:poses, :polarization, :milling), + :correlation_length => (:poses,), + :correlation_length_windowed => (:poses,), + :contact_graph_clusters => (:poses,), + :contact_graph_clusters_windowed => (:poses,), + :distance_to_source => (:poses,), + :forage_alignment => (:poses,), + :lookout_follower_te => (:poses,), + :own_colour_decodability => (:acts, :spikes), + :wall_distance => (:poses,), + :heading_error => (:scene,), + :object_in_view => (:percepts,), + :ball_paddle_distance => (:scene,), + :shoal_need_satisfaction => (:needs,), + :shoal_contact_summary => (:interactions,), + :shoal_movement_summary => (:poses,), + :shoal_group_movement_summary => (:poses,), + :shoal_perceptual_graph => (:poses,), +) + +"""A validated profile plan with its analysis and recorder contracts resolved.""" +struct ResolvedProfilePlan{ + P<:ProfilePlan, + R<:RegistrySet, + C<:ResolvedComposition, + A<:Tuple, + H<:Tuple, +} <: AbstractResolvedOperationPlan + plan::P + registry::R + composition::C + analyses::A + record_channels::H +end + +"""One numeric statistic emitted by one analysis for one evaluation trial.""" +struct ProfileAnalysisRow + condition::Symbol + block::Int + trial::Int + analysis::Symbol + statistic::Symbol + value::Float64 +end + +"""Across-trial descriptive summary for one analysis statistic.""" +struct ProfileAnalysisSummary + analysis::Symbol + statistic::Symbol + n_trials::Int + n_finite::Int + mean::Float64 + std::Float64 + minimum::Float64 + maximum::Float64 +end + +"""Compact descriptive summary of one completed profile operation.""" +struct ProfileSummary{A<:Tuple,H<:Tuple,S<:Vector{ProfileAnalysisSummary}} + plan::Symbol + condition::Symbol + blocks::Int + trials::Int + analyses::A + record_channels::H + raw_score_mean::Union{Missing,Float64} + normalized_score_mean::Union{Missing,Float64} + analysis_statistics::S +end + +"""Typed result retaining raw trials and both tabular profile surfaces.""" +struct ProfileResult{ + P<:ResolvedProfilePlan, + B<:EvaluationBatch, + T<:AbstractVector, + A<:Vector{ProfileAnalysisRow}, + S<:ProfileSummary, +} <: AbstractOperationResult + plan::P + batch::B + task_rows::T + analysis_rows::A + profile_summary::S +end + +"""Context-rich wrapper for an analysis that could not produce profile rows.""" +struct ProfileAnalysisError <: Exception + plan::Symbol + analysis::Symbol + block::Int + trial::Int + cause::Any +end + +function Base.showerror(io::IO, error::ProfileAnalysisError) + print( + io, + "profile :", + error.plan, + " analysis :", + error.analysis, + " failed at block ", + error.block, + ", trial ", + error.trial, + ": ", + ) + showerror(io, error.cause) +end + +function _profile_analysis_ids(plan::ProfilePlan, registry::RegistrySet) + isempty(plan.analyses) || return plan.analyses + return node_spec(registry, plan.target.composition.node).default_analyses +end + +function _profile_task_scope(spec::ImplementationSpec) + metadata = spec.metadata + hasproperty(metadata, :task) || return nothing + scope = getproperty(metadata, :task) + scope === nothing && return nothing + scope isa Symbol || throw(ArgumentError( + "analysis :$(spec.key) metadata.task must be a Symbol or nothing", + )) + return scope +end + +function _profile_required_channels(spec::ImplementationSpec) + metadata = spec.metadata + if hasproperty(metadata, :required_channels) + raw = getproperty(metadata, :required_channels) + channels = _symbol_tuple(raw, "analysis :$(spec.key) required channels") + return channels + end + return get(_PROFILE_BUILTIN_CHANNELS, spec.key, ()) +end + +function _resolve_profile_analyses(plan::ProfilePlan, registry::RegistrySet) + task = plan.target.composition.task + ids = _profile_analysis_ids(plan, registry) + specs = Tuple(resolve(registry.analyses, id) for id in ids) + for spec in specs + scope = _profile_task_scope(spec) + scope === nothing || scope === task || throw(ArgumentError( + "analysis :$(spec.key) is scoped to task :$(scope), not :$(task)", + )) + end + return specs +end + +function _profile_record_channels(specs::Tuple) + channels = Set{Symbol}() + for spec in specs + union!(channels, _profile_required_channels(spec)) + end + return Tuple(sort!(collect(channels); by=string)) +end + +function validate(plan::ProfilePlan, registry::RegistrySet) + resolve_composition(plan.target.composition, registry) + specs = _resolve_profile_analyses(plan, registry) + _profile_record_channels(specs) + return plan +end + +function resolve(plan::ProfilePlan, registry::RegistrySet) + composition = resolve_composition(plan.target.composition, registry) + specs = _resolve_profile_analyses(plan, registry) + channels = _profile_record_channels(specs) + return ResolvedProfilePlan(plan, registry, composition, specs, channels) +end + +function _profile_finite_summary(values) + raw = Float64.(vec(collect(values))) + finite = filter(isfinite, raw) + n = length(raw) + n_finite = length(finite) + if isempty(finite) + return ( + n=Float64(n), + finite_n=0.0, + mean=NaN, + std=NaN, + minimum=NaN, + maximum=NaN, + ) + end + mean = sum(finite) / n_finite + variance = if n_finite <= 1 + 0.0 + else + sum((value - mean)^2 for value in finite) / (n_finite - 1) + end + return ( + n=Float64(n), + finite_n=Float64(n_finite), + mean=mean, + std=sqrt(variance), + minimum=minimum(finite), + maximum=maximum(finite), + ) +end + +function _profile_array_statistics!(out, prefix::Symbol, values) + summary = _profile_finite_summary(values) + for field in propertynames(summary) + push!(out, Symbol(prefix, :_, field) => Float64(getproperty(summary, field))) + end + return out +end + +function _profile_named_statistics(output::NamedTuple) + source = if hasproperty(output, :summary) && getproperty(output, :summary) isa NamedTuple + getproperty(output, :summary) + else + output + end + out = Pair{Symbol,Float64}[] + for field in propertynames(source) + value = getproperty(source, field) + value isa Real || continue + push!(out, field => Float64(value)) + end + isempty(out) || return out + + for field in propertynames(source) + value = getproperty(source, field) + if value isa AbstractArray{<:Real} || ( + value isa Tuple && all(item -> item isa Real, value) + ) + _profile_array_statistics!(out, field, value) + end + end + return out +end + +function _profile_statistics(output) + if output isa Real + return Pair{Symbol,Float64}[:value => Float64(output)] + elseif output isa NamedTuple + return _profile_named_statistics(output) + elseif output isa AbstractArray{<:Real} || ( + output isa Tuple && all(item -> item isa Real, output) + ) + out = Pair{Symbol,Float64}[] + summary = _profile_finite_summary(output) + for field in propertynames(summary) + push!(out, field => Float64(getproperty(summary, field))) + end + return out + end + return Pair{Symbol,Float64}[] +end + +function _profile_analysis_rows( + plan::ResolvedProfilePlan, + batch::EvaluationBatch, +) + rows = ProfileAnalysisRow[] + for trial in batch.trials + for spec in plan.analyses + statistics = try + _profile_statistics(spec.implementation(trial.simulation)) + catch error + throw(ProfileAnalysisError( + plan.plan.id, + spec.key, + trial.block, + trial.trial, + error, + )) + end + isempty(statistics) && throw(ProfileAnalysisError( + plan.plan.id, + spec.key, + trial.block, + trial.trial, + ArgumentError( + "analysis returned no numeric scalar or numeric-array statistics", + ), + )) + for (statistic, value) in statistics + push!(rows, ProfileAnalysisRow( + trial.condition, + trial.block, + trial.trial, + spec.key, + statistic, + value, + )) + end + end + end + return rows +end + +function _profile_optional_mean(rows, field::Symbol) + values = Float64[] + for row in rows + value = getproperty(row, field) + value === missing && continue + number = Float64(value) + isfinite(number) && push!(values, number) + end + isempty(values) && return missing + return sum(values) / length(values) +end + +function _profile_analysis_summaries(rows::Vector{ProfileAnalysisRow}) + groups = Dict{Tuple{Symbol,Symbol},Vector{ProfileAnalysisRow}}() + for row in rows + push!(get!(groups, (row.analysis, row.statistic), ProfileAnalysisRow[]), row) + end + summaries = ProfileAnalysisSummary[] + for key in sort!(collect(keys(groups)); by=item -> (string(item[1]), string(item[2]))) + group = groups[key] + values = [row.value for row in group] + finite = filter(isfinite, values) + trial_count = length(unique((row.block, row.trial) for row in group)) + if isempty(finite) + push!(summaries, ProfileAnalysisSummary( + key[1], + key[2], + trial_count, + 0, + NaN, + NaN, + NaN, + NaN, + )) + continue + end + mean = sum(finite) / length(finite) + variance = length(finite) <= 1 ? + 0.0 : + sum((value - mean)^2 for value in finite) / (length(finite) - 1) + push!(summaries, ProfileAnalysisSummary( + key[1], + key[2], + trial_count, + length(finite), + mean, + sqrt(variance), + minimum(finite), + maximum(finite), + )) + end + return summaries +end + +function _profile_summary( + plan::ResolvedProfilePlan, + task_rows, + analysis_rows::Vector{ProfileAnalysisRow}, +) + evaluation = plan.plan.target.evaluation + return ProfileSummary( + plan.plan.id, + plan.plan.target.id, + evaluation.blocks, + length(task_rows), + Tuple(spec.key for spec in plan.analyses), + plan.record_channels, + _profile_optional_mean(task_rows, :raw_score), + _profile_optional_mean(task_rows, :normalized_score), + _profile_analysis_summaries(analysis_rows), + ) +end + +function execute(plan::ResolvedProfilePlan) + batch = evaluate( + plan.plan.target; + registry=plan.registry, + record=plan.record_channels, + record_every=plan.plan.record_every, + ) + task_rows = trial_table(batch) + analysis_rows = _profile_analysis_rows(plan, batch) + profile_summary = _profile_summary(plan, task_rows, analysis_rows) + return ProfileResult(plan, batch, task_rows, analysis_rows, profile_summary) +end + +tables(result::ProfileResult) = ( + task=result.task_rows, + analyses=result.analysis_rows, +) + +summary(result::ProfileResult) = result.profile_summary diff --git a/test/test_profile_plan.jl b/test/test_profile_plan.jl new file mode 100644 index 0000000..4134a9d --- /dev/null +++ b/test/test_profile_plan.jl @@ -0,0 +1,127 @@ +using BrainlessLab +using Test + +Base.include( + BrainlessLab, + joinpath(@__DIR__, "..", "src", "operations", "Profile.jl"), +) + +function _profile_tracking_target(; blocks=1, trials=2, horizon=8) + composition = CompositionSpec( + :profile_tracking_smoke, + :falandays, + :tracking; + n_nodes=8, + parameters=Dict( + :input_weight => 0.75, + :lrate_wmat => 1.0, + :lrate_targ => 0.01, + :weight_init_mode => :excitatory, + :rectify => false, + :repair_masks => false, + ), + ) + evaluation = EvaluationSpec( + blocks=blocks, + trials_per_block=trials, + horizon=horizon, + root_seed=611, + ) + return EvaluationTarget(:tracking, composition, evaluation) +end + +@testset "profile resolves registry contracts once" begin + registry = RegistrySet() + register_builtins!(registry) + target = _profile_tracking_target() + + defaults = ProfilePlan(:tracking_defaults, target) + validated = validate(defaults, registry) + resolved = resolve(defaults, registry) + @test validated === defaults + @test Tuple(spec.key for spec in resolved.analyses) == + node_spec(registry, :falandays).default_analyses + @test resolved.record_channels == ( + :acts, + :rate, + :spectral_radius, + :spikes, + :targets, + ) + + wrong_task = ProfilePlan( + :wrong_task_analysis, + target; + analyses=(:ball_paddle_distance,), + ) + @test_throws ArgumentError validate(wrong_task, registry) + @test_throws KeyError validate( + ProfilePlan(:unknown_analysis, target; analyses=(:not_registered,)), + registry, + ) +end + +@testset "profile executes every trial and emits two tables" begin + registry = RegistrySet() + register_builtins!(registry) + target = _profile_tracking_target(blocks=2, trials=2) + plan = ProfilePlan( + :tracking_heading_profile, + target; + analyses=(:heading_error,), + record_every=2, + ) + + resolved = resolve(plan, registry) + @test resolved.record_channels == (:scene,) + result = execute(resolved) + output = tables(result) + report = BrainlessLab.summary(result) + + @test result isa BrainlessLab.ProfileResult + @test length(result.batch.trials) == 4 + @test length(output.task) == 4 + @test all(row -> row.score_key === :track_score, output.task) + @test !isempty(output.analyses) + @test all(row -> row.analysis === :heading_error, output.analyses) + @test Set(row.statistic for row in output.analyses) == + Set((:n, :finite_n, :mean, :std, :minimum, :maximum)) + @test report.plan === :tracking_heading_profile + @test report.blocks == 2 + @test report.trials == 4 + @test report.analyses == (:heading_error,) + @test report.record_channels == (:scene,) + @test isfinite(report.raw_score_mean) + @test all(item -> item.n_trials == 4, report.analysis_statistics) +end + +@testset "profile analysis failures retain trial context" begin + registry = RegistrySet() + register_builtins!(registry) + register!( + registry, + :analyses, + ImplementationSpec( + :profile_failure_fixture, + _ -> error("deliberate analysis failure"); + metadata=(task=:tracking, required_channels=()), + ), + ) + plan = ProfilePlan( + :failing_profile, + _profile_tracking_target(trials=1); + analyses=(:profile_failure_fixture,), + ) + + failure = try + execute(resolve(plan, registry)) + nothing + catch error + error + end + @test failure isa BrainlessLab.ProfileAnalysisError + @test failure.analysis === :profile_failure_fixture + @test failure.block == 1 + @test failure.trial == 1 + @test occursin("deliberate analysis failure", sprint(showerror, failure)) +end From d1994d928a15b16082304eb8d94d1a37e8eed49e Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 23:17:10 -0400 Subject: [PATCH 12/20] feat: add typed sweep and ablation operations --- src/operations/Ablation.jl | 189 +++++++++++++++++++++++++++++++ src/operations/Sweep.jl | 222 +++++++++++++++++++++++++++++++++++++ test/test_ablation_plan.jl | 189 +++++++++++++++++++++++++++++++ test/test_sweep_plan.jl | 115 +++++++++++++++++++ 4 files changed, 715 insertions(+) create mode 100644 src/operations/Ablation.jl create mode 100644 src/operations/Sweep.jl create mode 100644 test/test_ablation_plan.jl create mode 100644 test/test_sweep_plan.jl diff --git a/src/operations/Ablation.jl b/src/operations/Ablation.jl new file mode 100644 index 0000000..5cca45f --- /dev/null +++ b/src/operations/Ablation.jl @@ -0,0 +1,189 @@ +using Statistics + +struct ResolvedAblationCase{A,T<:EvaluationTarget} + id::Symbol + ablation::A + target::T +end + +struct ResolvedAblationPlan{P<:AblationPlan,C<:Tuple,R<:RegistrySet} <: + AbstractResolvedOperationPlan + source::P + cases::C + registry::R +end + +struct AblationResult{P<:ResolvedAblationPlan,B<:Tuple} <: AbstractOperationResult + plan::P + batches::B + trial_rows::Vector{NamedTuple} + case_summaries::Vector{NamedTuple} +end + +function _registered_ablation(registry::RegistrySet, id::Symbol) + entry = resolve(registry.ablations, id) + ablation = entry.implementation + ablation isa AblationSpec || throw(ArgumentError( + "registered ablation :$(id) must contain an AblationSpec, got $(typeof(ablation)); " * + "raw intervention types are not executable operation contracts", + )) + ablation.id === id || throw(ArgumentError( + "registered ablation key :$(id) does not match AblationSpec id :$(ablation.id)", + )) + return ablation +end + +function _validate_ablation( + ablation::AblationSpec, + node::NodeSpec, + composition::CompositionSpec, +) + ablation.stage === :composition || throw(ArgumentError( + "ablation :$(ablation.id) uses unsupported stage :$(ablation.stage); " * + "the current executor supports only :composition", + )) + missing_capabilities = setdiff(ablation.required_capabilities, node.capabilities) + isempty(missing_capabilities) || throw(ArgumentError( + "ablation :$(ablation.id) requires node capabilities " * + "$(Tuple(missing_capabilities)); node :$(node.id) declares $(node.capabilities)", + )) + applicable(ablation.apply, composition) || throw(ArgumentError( + "composition-stage ablation :$(ablation.id) must accept one CompositionSpec", + )) + return ablation +end + +function validate(plan::AblationPlan, registry::RegistrySet) + resolved = resolve_composition(plan.target.composition, registry) + :baseline in plan.ablations && throw(ArgumentError( + "ablation id :baseline is reserved for the implicit baseline case", + )) + for id in plan.ablations + _validate_ablation( + _registered_ablation(registry, id), + resolved.node, + plan.target.composition, + ) + end + return plan +end + +function _apply_composition_ablation( + ablation::AblationSpec, + source::CompositionSpec, + registry::RegistrySet, +) + transformed = ablation.apply(source) + transformed isa CompositionSpec || throw(ArgumentError( + "ablation :$(ablation.id) returned $(typeof(transformed)), not CompositionSpec", + )) + transformed === source && throw(ArgumentError( + "ablation :$(ablation.id) returned its input composition unchanged", + )) + resolve_composition(transformed, registry) + return transformed +end + +function resolve(plan::AblationPlan, registry::RegistrySet) + validate(plan, registry) + cases = ResolvedAblationCase[ + ResolvedAblationCase( + :baseline, + nothing, + EvaluationTarget(:baseline, plan.target.composition, plan.target.evaluation), + ), + ] + for id in plan.ablations + ablation = _registered_ablation(registry, id) + composition = _apply_composition_ablation( + ablation, + plan.target.composition, + registry, + ) + push!(cases, ResolvedAblationCase( + id, + ablation, + EvaluationTarget(id, composition, plan.target.evaluation), + )) + end + return ResolvedAblationPlan(plan, Tuple(cases), registry) +end + +function _ablation_aggregate(values, policy::Symbol) + policy === :none && return missing + observed = Float64[value for value in values if !ismissing(value)] + isempty(observed) && return missing + policy === :mean && return mean(observed) + policy === :median && return median(observed) + policy === :sum && return sum(observed) + policy === :minimum && return minimum(observed) + policy === :maximum && return maximum(observed) + throw(ArgumentError("unsupported aggregate policy :$(policy)")) +end + +function _ablation_trial_rows( + plan::ResolvedAblationPlan, + batches::Tuple, +) + rows = NamedTuple[] + for (case, batch) in zip(plan.cases, batches) + for row in trial_table(batch) + push!(rows, merge( + ( + operation=plan.source.id, + case=case.id, + ablation=case.ablation === nothing ? :none : case.ablation.id, + ), + row, + )) + end + end + return rows +end + +function _ablation_case_summaries( + plan::ResolvedAblationPlan, + rows::Vector{NamedTuple}, +) + summaries = NamedTuple[] + policy = plan.source.target.evaluation.aggregate + for case in plan.cases + selected = filter(row -> row.case === case.id, rows) + viability = [row.viable for row in selected if !ismissing(row.viable)] + push!(summaries, ( + operation=plan.source.id, + case=case.id, + ablation=case.ablation === nothing ? :none : case.ablation.id, + n_trials=length(selected), + aggregate=policy, + raw_score=_ablation_aggregate((row.raw_score for row in selected), policy), + normalized_score=_ablation_aggregate( + (row.normalized_score for row in selected), + policy, + ), + viable_fraction=isempty(viability) ? missing : mean(viability), + )) + end + return summaries +end + +function execute(plan::ResolvedAblationPlan) + batches = Tuple(evaluate(case.target; registry=plan.registry) for case in plan.cases) + rows = _ablation_trial_rows(plan, batches) + summaries = _ablation_case_summaries(plan, rows) + return AblationResult(plan, batches, rows, summaries) +end + +tables(result::AblationResult) = ( + trials=result.trial_rows, + cases=result.case_summaries, +) + +summary(result::AblationResult) = ( + operation=:ablation, + id=result.plan.source.id, + n_cases=length(result.plan.cases), + n_rollouts=length(result.trial_rows), + aggregate=result.plan.source.target.evaluation.aggregate, + cases=result.case_summaries, +) diff --git a/src/operations/Sweep.jl b/src/operations/Sweep.jl new file mode 100644 index 0000000..e6744b7 --- /dev/null +++ b/src/operations/Sweep.jl @@ -0,0 +1,222 @@ +using Statistics + +struct ResolvedSweepCell{T<:EvaluationTarget} + id::Symbol + parameters::Dict{Symbol,Any} + target::T +end + +struct ResolvedSweepPlan{P<:SweepPlan,A<:Tuple,C<:Tuple,R<:RegistrySet} <: + AbstractResolvedOperationPlan + source::P + axes::A + cells::C + registry::R + rollouts::Int +end + +struct SweepResult{P<:ResolvedSweepPlan,B<:Tuple} <: AbstractOperationResult + plan::P + batches::B + trial_rows::Vector{NamedTuple} + cell_summaries::Vector{NamedTuple} +end + +function _sweep_axes(plan::SweepPlan, node::NodeSpec) + if !isempty(plan.axes) + return plan.axes + end + names = node_parameter_set(node, :sweep) + isempty(names) && throw(ArgumentError( + "node :$(node.id) has an empty :sweep parameter set", + )) + return Tuple(begin + parameter = node_parameter(node, name) + parameter.sweep === nothing && throw(ArgumentError( + "node :$(node.id) parameter :$(name) is in :sweep but has no sweep values", + )) + SweepAxis(name, parameter.sweep) + end for name in names) +end + +function _validate_sweep_axes(axes::Tuple, node::NodeSpec) + for axis in axes + parameter = node_parameter(node, axis.parameter) + foreach(value -> validate_parameter(parameter, value), axis.values) + end + return axes +end + +function validate(plan::SweepPlan, registry::RegistrySet) + resolved = resolve_composition(plan.target.composition, registry) + axes = _sweep_axes(plan, resolved.node) + _validate_sweep_axes(axes, resolved.node) + return plan +end + +function _factorial_parameter_cells(axes::Tuple) + cells = [Dict{Symbol,Any}()] + for axis in axes + next = Dict{Symbol,Any}[] + for cell in cells, value in axis.values + parameters = copy(cell) + parameters[axis.parameter] = value + push!(next, parameters) + end + cells = next + end + return cells +end + +function _one_at_a_time_parameter_cells(axes::Tuple) + cells = Dict{Symbol,Any}[] + for axis in axes, value in axis.values + push!(cells, Dict{Symbol,Any}(axis.parameter => value)) + end + return cells +end + +function _sweep_composition( + source::CompositionSpec, + id::Symbol, + parameter_updates, +) + parameters = copy(source.parameters) + merge!(parameters, parameter_updates) + return CompositionSpec( + id, + source.node, + source.task; + body=source.body, + n_agents=source.n_agents, + n_nodes=source.n_nodes, + parameters=parameters, + task_options=source.task_options, + body_options=source.body_options, + interaction_cycle=source.interaction_cycle, + ) +end + +function resolve(plan::SweepPlan, registry::RegistrySet) + validate(plan, registry) + node = node_spec(registry, plan.target.composition.node) + axes = _sweep_axes(plan, node) + parameter_cells = if plan.mode === :factorial + _factorial_parameter_cells(axes) + else + _one_at_a_time_parameter_cells(axes) + end + isempty(parameter_cells) && throw(ArgumentError("sweep resolved to no cells")) + + rollout_count = BigInt(length(parameter_cells)) * + plan.target.evaluation.blocks * + plan.target.evaluation.trials_per_block + rollout_count <= plan.max_rollouts || throw(ArgumentError( + "sweep requires $(rollout_count) rollouts above max_rollouts=$(plan.max_rollouts)", + )) + + cells = Vector{ResolvedSweepCell}(undef, length(parameter_cells)) + for (index, parameters) in enumerate(parameter_cells) + cell_id = Symbol("cell_", lpad(index, 3, '0')) + composition = _sweep_composition( + plan.target.composition, + Symbol(plan.target.composition.id, "__", cell_id), + parameters, + ) + resolve_composition(composition, registry) + target = EvaluationTarget(cell_id, composition, plan.target.evaluation) + cells[index] = ResolvedSweepCell(cell_id, parameters, target) + end + return ResolvedSweepPlan( + plan, + axes, + Tuple(cells), + registry, + Int(rollout_count), + ) +end + +_sweep_parameter_pairs(parameters::Dict{Symbol,Any}) = + Tuple(key => parameters[key] for key in sort!(collect(keys(parameters)); by=string)) + +function _sweep_aggregate(values, policy::Symbol) + policy === :none && return missing + observed = Float64[value for value in values if !ismissing(value)] + isempty(observed) && return missing + policy === :mean && return mean(observed) + policy === :median && return median(observed) + policy === :sum && return sum(observed) + policy === :minimum && return minimum(observed) + policy === :maximum && return maximum(observed) + throw(ArgumentError("unsupported aggregate policy :$(policy)")) +end + +function _sweep_trial_rows( + plan::ResolvedSweepPlan, + batches::Tuple, +) + rows = NamedTuple[] + for (cell, batch) in zip(plan.cells, batches) + parameters = _sweep_parameter_pairs(cell.parameters) + for row in trial_table(batch) + push!(rows, merge( + ( + operation=plan.source.id, + cell=cell.id, + parameters=parameters, + ), + row, + )) + end + end + return rows +end + +function _sweep_cell_summaries( + plan::ResolvedSweepPlan, + rows::Vector{NamedTuple}, +) + summaries = NamedTuple[] + policy = plan.source.target.evaluation.aggregate + for cell in plan.cells + selected = filter(row -> row.cell === cell.id, rows) + viability = [row.viable for row in selected if !ismissing(row.viable)] + push!(summaries, ( + operation=plan.source.id, + cell=cell.id, + parameters=_sweep_parameter_pairs(cell.parameters), + n_trials=length(selected), + aggregate=policy, + raw_score=_sweep_aggregate((row.raw_score for row in selected), policy), + normalized_score=_sweep_aggregate( + (row.normalized_score for row in selected), + policy, + ), + viable_fraction=isempty(viability) ? missing : mean(viability), + )) + end + return summaries +end + +function execute(plan::ResolvedSweepPlan) + batches = Tuple(evaluate(cell.target; registry=plan.registry) for cell in plan.cells) + rows = _sweep_trial_rows(plan, batches) + summaries = _sweep_cell_summaries(plan, rows) + return SweepResult(plan, batches, rows, summaries) +end + +tables(result::SweepResult) = ( + trials=result.trial_rows, + cells=result.cell_summaries, +) + +summary(result::SweepResult) = ( + operation=:sweep, + id=result.plan.source.id, + mode=result.plan.source.mode, + n_axes=length(result.plan.axes), + n_cells=length(result.plan.cells), + n_rollouts=length(result.trial_rows), + aggregate=result.plan.source.target.evaluation.aggregate, + cells=result.cell_summaries, +) diff --git a/test/test_ablation_plan.jl b/test/test_ablation_plan.jl new file mode 100644 index 0000000..a0a38b5 --- /dev/null +++ b/test/test_ablation_plan.jl @@ -0,0 +1,189 @@ +using BrainlessLab +using Test + +module AblationOperation +using BrainlessLab +import BrainlessLab: execute, resolve, summary, tables, validate +include(joinpath(@__DIR__, "..", "src", "operations", "Ablation.jl")) +end + +function _ablation_registry() + registry = RegistrySet() + register!(registry, falandays_node_spec()) + register!(registry, task_spec(DEFAULT_REGISTRY, :tracking)) + return registry +end + +function _ablation_target() + reference = default_composition(DEFAULT_REGISTRY, :falandays, :tracking) + composition = CompositionSpec( + :tracking_ablation_smoke, + reference.node, + reference.task; + n_nodes=8, + parameters=reference.parameters, + ) + evaluation = EvaluationSpec( + blocks=1, + trials_per_block=2, + horizon=3, + root_seed=812, + aggregate=:mean, + ) + return EvaluationTarget(:tracking, composition, evaluation) +end + +function _with_ablation_parameter( + source::CompositionSpec, + id::Symbol, + parameter::Symbol, + value, +) + parameters = copy(source.parameters) + parameters[parameter] = value + return CompositionSpec( + Symbol(source.id, "__", id), + source.node, + source.task; + body=source.body, + n_agents=source.n_agents, + n_nodes=source.n_nodes, + parameters, + task_options=source.task_options, + body_options=source.body_options, + interaction_cycle=source.interaction_cycle, + ) +end + +function _register_falandays_ablations!(registry) + freeze = AblationSpec( + :freeze_plasticity, + source -> _with_ablation_parameter( + source, + :freeze_plasticity, + :learn_on, + false, + ); + stage=:composition, + required_capabilities=(:online_plasticity,), + ) + clamp = AblationSpec( + :clamp_target, + source -> _with_ablation_parameter( + source, + :clamp_target, + :lrate_targ, + 0.0, + ); + stage=:composition, + required_capabilities=(:homeostatic_target,), + ) + register!( + registry, + :ablations, + ImplementationSpec(:freeze_plasticity, freeze), + ) + register!( + registry, + :ablations, + ImplementationSpec(:clamp_target, clamp), + ) + return registry +end + +@testset "ablation plan resolution is explicit" begin + registry = _register_falandays_ablations!(_ablation_registry()) + target = _ablation_target() + plan = AblationPlan( + :falandays_ablations, + target; + ablations=(:freeze_plasticity, :clamp_target), + ) + resolved = BrainlessLab.resolve(plan, registry) + + @test Tuple(case.id for case in resolved.cases) == + (:baseline, :freeze_plasticity, :clamp_target) + @test resolved.cases[1].ablation === nothing + @test resolved.cases[2].target.composition.parameters[:learn_on] == false + @test resolved.cases[3].target.composition.parameters[:lrate_targ] == 0.0 + + missing_capability = AblationSpec( + :requires_dendrites, + source -> _with_ablation_parameter(source, :dendrites, :learn_on, false); + required_capabilities=(:dendrites,), + ) + register!( + registry, + :ablations, + ImplementationSpec(:requires_dendrites, missing_capability), + ) + @test_throws ArgumentError BrainlessLab.resolve( + AblationPlan(:bad_capability, target; ablations=(:requires_dendrites,)), + registry, + ) + + reservoir_stage = AblationSpec( + :reservoir_stage, + identity; + stage=:reservoir, + ) + register!( + registry, + :ablations, + ImplementationSpec(:reservoir_stage, reservoir_stage), + ) + @test_throws ArgumentError BrainlessLab.resolve( + AblationPlan(:bad_stage, target; ablations=(:reservoir_stage,)), + registry, + ) + + register!( + registry, + :ablations, + ImplementationSpec(:raw_intervention, FreezePlasticity), + ) + @test_throws ArgumentError BrainlessLab.resolve( + AblationPlan(:raw, target; ablations=(:raw_intervention,)), + registry, + ) + + register!( + registry, + :ablations, + ImplementationSpec( + :baseline, + AblationSpec(:baseline, source -> deepcopy(source)), + ), + ) + @test_throws ArgumentError BrainlessLab.resolve( + AblationPlan(:reserved, target; ablations=(:baseline,)), + registry, + ) +end + +@testset "ablation execution includes paired baseline" begin + registry = _register_falandays_ablations!(_ablation_registry()) + plan = AblationPlan( + :paired_ablations, + _ablation_target(); + ablations=(:freeze_plasticity, :clamp_target), + ) + result = BrainlessLab.execute(BrainlessLab.resolve(plan, registry)) + output = BrainlessLab.tables(result) + compact = BrainlessLab.summary(result) + + @test length(output.trials) == 6 + @test length(output.cases) == 3 + @test compact.n_cases == 3 + @test compact.n_rollouts == 6 + @test first(output.cases).ablation === :none + @test all(row -> isfinite(row.raw_score), output.trials) + + for trial in 1:2 + paired = filter(row -> row.block == 1 && row.trial == trial, output.trials) + @test length(paired) == 3 + @test length(unique(row.topology_seed for row in paired)) == 1 + @test length(unique(row.world_seed for row in paired)) == 1 + @test length(unique(row.task_seed for row in paired)) == 1 + end +end diff --git a/test/test_sweep_plan.jl b/test/test_sweep_plan.jl new file mode 100644 index 0000000..dacbeee --- /dev/null +++ b/test/test_sweep_plan.jl @@ -0,0 +1,115 @@ +using BrainlessLab +using Test + +module SweepOperation +using BrainlessLab +import BrainlessLab: execute, resolve, summary, tables, validate +include(joinpath(@__DIR__, "..", "src", "operations", "Sweep.jl")) +end + +function _small_sweep_target(; blocks=1, trials=2, horizon=3) + reference = default_composition(DEFAULT_REGISTRY, :falandays, :tracking) + composition = CompositionSpec( + :tracking_sweep_smoke, + reference.node, + reference.task; + n_nodes=8, + parameters=reference.parameters, + ) + evaluation = EvaluationSpec( + blocks=blocks, + trials_per_block=trials, + horizon=horizon, + root_seed=811, + aggregate=:mean, + ) + return EvaluationTarget(:tracking, composition, evaluation) +end + +@testset "sweep plan resolution" begin + target = _small_sweep_target() + default_plan = SweepPlan( + :default_axes, + target; + axes=(), + mode=:factorial, + max_rollouts=100, + ) + resolved_default = BrainlessLab.resolve(default_plan, DEFAULT_REGISTRY) + @test Tuple(axis.parameter for axis in resolved_default.axes) == + (:leak, :lrate_wmat) + @test length(resolved_default.cells) == 16 + @test resolved_default.rollouts == 32 + + axes = ( + SweepAxis(:leak, (0.1, 0.5)), + SweepAxis(:lrate_wmat, (0.1, 1.0)), + ) + factorial = BrainlessLab.resolve( + SweepPlan(:factorial, target; axes, max_rollouts=8), + DEFAULT_REGISTRY, + ) + @test length(factorial.cells) == 4 + @test factorial.rollouts == 8 + @test factorial.cells[4].parameters == + Dict{Symbol,Any}(:leak => 0.5, :lrate_wmat => 1.0) + + one_at_a_time = BrainlessLab.resolve( + SweepPlan(:oaat, target; axes, mode=:one_at_a_time, max_rollouts=8), + DEFAULT_REGISTRY, + ) + @test length(one_at_a_time.cells) == 4 + @test all(cell -> length(cell.parameters) == 1, one_at_a_time.cells) + + @test_throws ArgumentError BrainlessLab.resolve( + SweepPlan(:too_large, target; axes, max_rollouts=7), + DEFAULT_REGISTRY, + ) + @test_throws KeyError BrainlessLab.resolve( + SweepPlan( + :missing_parameter, + target; + axes=(SweepAxis(:missing, (1.0,)),), + ), + DEFAULT_REGISTRY, + ) + @test_throws ArgumentError BrainlessLab.resolve( + SweepPlan( + :invalid_value, + target; + axes=(SweepAxis(:leak, (2.0,)),), + ), + DEFAULT_REGISTRY, + ) +end + +@testset "sweep execution preserves paired seeds" begin + target = _small_sweep_target() + plan = SweepPlan( + :paired_sweep, + target; + axes=( + SweepAxis(:leak, (0.1, 0.5)), + SweepAxis(:lrate_wmat, (0.1, 1.0)), + ), + mode=:one_at_a_time, + max_rollouts=8, + ) + result = BrainlessLab.execute(BrainlessLab.resolve(plan, DEFAULT_REGISTRY)) + output = BrainlessLab.tables(result) + compact = BrainlessLab.summary(result) + + @test length(output.trials) == 8 + @test length(output.cells) == 4 + @test compact.n_cells == 4 + @test compact.n_rollouts == 8 + @test all(row -> isfinite(row.raw_score), output.trials) + + for trial in 1:2 + paired = filter(row -> row.block == 1 && row.trial == trial, output.trials) + @test length(paired) == 4 + @test length(unique(row.topology_seed for row in paired)) == 1 + @test length(unique(row.world_seed for row in paired)) == 1 + @test length(unique(row.task_seed for row in paired)) == 1 + end +end From f8767f70b62fca8f0eb740725ca530f01c5fee6c Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 23:18:19 -0400 Subject: [PATCH 13/20] feat: add typed evolution operation --- src/operations/Evolution.jl | 545 ++++++++++++++++++++++++++++++++++++ test/test_evolution_plan.jl | 121 ++++++++ 2 files changed, 666 insertions(+) create mode 100644 src/operations/Evolution.jl create mode 100644 test/test_evolution_plan.jl diff --git a/src/operations/Evolution.jl b/src/operations/Evolution.jl new file mode 100644 index 0000000..10d428e --- /dev/null +++ b/src/operations/Evolution.jl @@ -0,0 +1,545 @@ +using Statistics + +""" + ResolvedEvolutionPlan + +An `EvolutionPlan` after its node parameter set, optimizer, starting point, and +registry have been resolved. The optimizer seed is deliberately retained +outside `EvaluationSpec`: optimizer sampling and evaluation trial streams are +separate stochastic processes. +""" +struct ResolvedEvolutionPlan{ + P<:EvolutionPlan, + N<:NodeSpec, + O<:ImplementationSpec, + S<:Tuple, +} <: AbstractResolvedOperationPlan + plan::P + registry::RegistrySet + node::N + optimizer::O + parameters::S + x0::Vector{Float64} + optimizer_seed::UInt64 +end + +"""One evaluated member of one evolutionary generation.""" +struct EvolutionCandidate + generation::Int + individual::Int + coordinates::Vector{Float64} + parameters::Dict{Symbol,Any} + objective_values::Vector{Float64} + fitness::Float64 +end + +"""Compact convergence statistics for one evolutionary generation.""" +struct EvolutionGeneration + generation::Int + best_individual::Int + fitness_best::Float64 + fitness_median::Float64 + fitness_mean::Float64 + fitness_worst::Float64 +end + +"""A champion evaluated once under a declared target protocol.""" +struct EvolutionEvaluation{B<:EvaluationBatch} + target::Symbol + objective::Symbol + objective_values::Vector{Float64} + aggregate::Float64 + batch::B +end + +""" + EvolutionResult + +Typed result of parameter evolution. Candidate and convergence histories +contain training information only. Held-out targets are evaluated only after +the champion has been selected. +""" +struct EvolutionResult{ + P<:ResolvedEvolutionPlan, + O, + T<:EvolutionEvaluation, + H<:Tuple, +} <: AbstractOperationResult + plan::P + optimizer_result::O + optimizer_seed::UInt64 + candidates::Vector{EvolutionCandidate} + convergence::Vector{EvolutionGeneration} + champion_coordinates::Vector{Float64} + champion_parameters::Dict{Symbol,Any} + champion_training_fitness::Float64 + training::T + heldout::H +end + +function _evolution_mutation_scale(parameter::ParameterSpec) + metadata = parameter.evolve + hasproperty(metadata, :values) && begin + count = length(metadata.values) + return count == 1 ? 1.0 : inv(Float64(count - 1)) + end + metadata.mutation_scale === nothing && return 0.2 + return Float64(metadata.mutation_scale) +end + +function _bounded_unit(parameter::ParameterSpec, value) + metadata = parameter.evolve + lower = Float64(metadata.lower) + upper = Float64(metadata.upper) + numeric = Float64(value) + lower <= numeric <= upper || throw(ArgumentError( + "starting value $(repr(value)) for parameter :$(parameter.name) lies outside " * + "its evolution bounds [$(metadata.lower), $(metadata.upper)]", + )) + lower == upper && return 0.0 + if metadata.scale === :log + return (log(numeric) - log(lower)) / (log(upper) - log(lower)) + end + return (numeric - lower) / (upper - lower) +end + +function _encode_evolution_parameter(parameter::ParameterSpec, value) + metadata = parameter.evolve + if hasproperty(metadata, :values) + index = findfirst(candidate -> isequal(candidate, value), metadata.values) + index === nothing && throw(ArgumentError( + "starting value $(repr(value)) for parameter :$(parameter.name) is not one " * + "of its evolvable values $(metadata.values)", + )) + count = length(metadata.values) + unit = count == 1 ? 0.0 : (index - 1) / (count - 1) + else + unit = _bounded_unit(parameter, value) + end + return unit / _evolution_mutation_scale(parameter) +end + +function _convert_evolution_value(parameter::ParameterSpec, value) + converted = if parameter.datatype <: Integer + convert(parameter.datatype, round(Int, value)) + else + convert(parameter.datatype, value) + end + return validate_parameter(parameter, converted) +end + +function _decode_evolution_parameter(parameter::ParameterSpec, coordinate::Real) + metadata = parameter.evolve + unit = clamp( + Float64(coordinate) * _evolution_mutation_scale(parameter), + 0.0, + 1.0, + ) + if hasproperty(metadata, :values) + count = length(metadata.values) + index = count == 1 ? 1 : clamp(round(Int, 1 + unit * (count - 1)), 1, count) + return validate_parameter(parameter, metadata.values[index]) + end + + lower = Float64(metadata.lower) + upper = Float64(metadata.upper) + value = if metadata.scale === :log + exp(log(lower) + unit * (log(upper) - log(lower))) + else + lower + unit * (upper - lower) + end + metadata.scale === :integer && (value = round(value)) + return _convert_evolution_value(parameter, value) +end + +function _decode_evolution_parameters( + parameters::Tuple, + coordinates::AbstractVector{<:Real}, +) + length(parameters) == length(coordinates) || throw(DimensionMismatch( + "candidate has $(length(coordinates)) coordinates; expected $(length(parameters))", + )) + values = Dict{Symbol,Any}() + for index in eachindex(parameters) + parameter = parameters[index] + values[parameter.name] = _decode_evolution_parameter( + parameter, + coordinates[index], + ) + end + return values +end + +function _evolution_optimizer_seed(plan::EvolutionPlan) + seed = _splitmix64( + plan.training.evaluation.root_seed ⊻ _stable_symbol_word(:optimizer), + ) + seed = _splitmix64(seed ⊻ _stable_symbol_word(plan.id)) + return _splitmix64(seed ⊻ _stable_symbol_word(plan.optimizer)) +end + +function _validate_evolution_target( + target::EvaluationTarget, + node_id::Symbol, + label::AbstractString, +) + target.composition.node === node_id || throw(ArgumentError( + "$(label) target :$(target.id) uses node :$(target.composition.node), " * + "but evolution is selecting parameters for node :$(node_id)", + )) + target.evaluation.reset === :full || throw(ArgumentError( + "$(label) target :$(target.id) must use reset=:full", + )) + target.evaluation.aggregate === :none && throw(ArgumentError( + "$(label) target :$(target.id) must declare a scalar aggregation policy", + )) + return target +end + +function validate(plan::EvolutionPlan, registry::RegistrySet) + resolved_training = resolve_composition(plan.training.composition, registry) + node = resolved_training.node + names = node_parameter_set(node, plan.parameter_set) + isempty(names) && throw(ArgumentError( + "evolution parameter set :$(plan.parameter_set) on node :$(node.id) is empty", + )) + parameters = Tuple(node_parameter(node, name) for name in names) + for parameter in parameters + evolvable(parameter) || throw(ArgumentError( + "parameter :$(parameter.name) in set :$(plan.parameter_set) on node " * + ":$(node.id) has no evolution metadata", + )) + _encode_evolution_parameter( + parameter, + resolved_training.parameters[parameter.name], + ) + end + _validate_evolution_target(plan.training, node.id, "training") + for target in plan.heldout_targets + _validate_evolution_target(target, node.id, "held-out") + resolved_heldout = resolve_composition(target.composition, registry) + for parameter in parameters + haskey(resolved_heldout.parameters, parameter.name) || throw(ArgumentError( + "held-out target :$(target.id) does not accept evolved parameter " * + ":$(parameter.name)", + )) + end + end + resolve(registry.optimizers, plan.optimizer) + return plan +end + +function resolve(plan::EvolutionPlan, registry::RegistrySet) + validate(plan, registry) + resolved_training = resolve_composition(plan.training.composition, registry) + node = resolved_training.node + parameters = Tuple( + node_parameter(node, name) + for name in node_parameter_set(node, plan.parameter_set) + ) + x0 = Float64[ + _encode_evolution_parameter( + parameter, + resolved_training.parameters[parameter.name], + ) + for parameter in parameters + ] + optimizer = resolve(registry.optimizers, plan.optimizer) + return ResolvedEvolutionPlan( + plan, + registry, + node, + optimizer, + parameters, + x0, + _evolution_optimizer_seed(plan), + ) +end + +function _evolution_target( + target::EvaluationTarget, + parameter_values::Dict{Symbol,Any}, + suffix::AbstractString, +) + source = target.composition + values = copy(source.parameters) + merge!(values, parameter_values) + composition = CompositionSpec( + Symbol(String(source.id) * "_" * suffix), + source.node, + source.task; + body=source.body, + n_agents=source.n_agents, + n_nodes=source.n_nodes, + parameters=values, + task_options=source.task_options, + body_options=source.body_options, + interaction_cycle=source.interaction_cycle, + ) + return EvaluationTarget(target.id, composition, target.evaluation) +end + +function _evolution_trial_value(trial::EvaluationTrial, objective::Symbol) + outcome = task_outcome(trial.simulation) + value = if objective === :normalized_score + outcome === nothing ? missing : outcome.normalized + elseif objective === :raw_score + outcome === nothing ? missing : outcome.raw + elseif outcome !== nothing && objective === outcome.key + outcome.raw + elseif hasproperty(trial.simulation.metrics, objective) + getproperty(trial.simulation.metrics, objective) + else + missing + end + value isa Real || throw(ArgumentError( + "objective :$(objective) is unavailable or non-numeric for target " * + ":$(trial.condition), block $(trial.block), trial $(trial.trial)", + )) + numeric = Float64(value) + isfinite(numeric) || throw(ArgumentError( + "objective :$(objective) is non-finite for target :$(trial.condition), " * + "block $(trial.block), trial $(trial.trial)", + )) + return numeric +end + +function _evolution_aggregate(values::AbstractVector{<:Real}, policy::Symbol) + isempty(values) && throw(ArgumentError("cannot aggregate an empty objective vector")) + policy === :mean && return sum(values) / length(values) + policy === :median && return median(values) + policy === :sum && return sum(values) + policy === :minimum && return minimum(values) + policy === :maximum && return maximum(values) + throw(ArgumentError("unsupported evolution aggregation policy :$(policy)")) +end + +function _evaluate_evolution_target( + target::EvaluationTarget, + objective::Symbol, + registry::RegistrySet, +) + batch = evaluate(target; registry=registry) + values = Float64[ + _evolution_trial_value(trial, objective) + for trial in batch.trials + ] + aggregate = Float64(_evolution_aggregate(values, target.evaluation.aggregate)) + return EvolutionEvaluation(target.id, objective, values, aggregate, batch) +end + +function _instantiate_evolution_optimizer(plan::ResolvedEvolutionPlan) + constructor = plan.optimizer.implementation + optimizer = constructor( + copy(plan.x0), + plan.plan.sigma0; + popsize=plan.plan.popsize, + seed=_seed_to_int(plan.optimizer_seed), + ) + optimizer isa AbstractEvolutionStrategy || throw(ArgumentError( + "optimizer :$(plan.plan.optimizer) returned $(typeof(optimizer)), not " * + "AbstractEvolutionStrategy", + )) + return optimizer +end + +function execute(plan::ResolvedEvolutionPlan) + optimizer = _instantiate_evolution_optimizer(plan) + candidate_history = EvolutionCandidate[] + convergence = EvolutionGeneration[] + champion_coordinates = copy(plan.x0) + champion_parameters = _decode_evolution_parameters(plan.parameters, plan.x0) + champion_fitness = -Inf + + for generation in 1:plan.plan.generations + proposed = ask(optimizer) + length(proposed) == plan.plan.popsize || throw(ArgumentError( + "optimizer :$(plan.plan.optimizer) proposed $(length(proposed)) candidates; " * + "EvolutionPlan requires popsize=$(plan.plan.popsize)", + )) + losses = Vector{Float64}(undef, length(proposed)) + fitnesses = Vector{Float64}(undef, length(proposed)) + + for individual in eachindex(proposed) + coordinates = Vector{Float64}(Float64.(proposed[individual])) + parameters = _decode_evolution_parameters(plan.parameters, coordinates) + target = _evolution_target( + plan.plan.training, + parameters, + "generation_$(generation)_individual_$(individual)", + ) + evaluation = _evaluate_evolution_target( + target, + plan.plan.objective, + plan.registry, + ) + fitness = evaluation.aggregate + fitnesses[individual] = fitness + losses[individual] = -fitness + push!( + candidate_history, + EvolutionCandidate( + generation, + individual, + coordinates, + parameters, + copy(evaluation.objective_values), + fitness, + ), + ) + if fitness > champion_fitness + champion_fitness = fitness + champion_coordinates = copy(coordinates) + champion_parameters = copy(parameters) + end + end + + tell!(optimizer, proposed, losses) + best_individual = argmax(fitnesses) + push!( + convergence, + EvolutionGeneration( + generation, + best_individual, + maximum(fitnesses), + median(fitnesses), + sum(fitnesses) / length(fitnesses), + minimum(fitnesses), + ), + ) + end + + optimizer_summary = result(optimizer) + training_target = _evolution_target( + plan.plan.training, + champion_parameters, + "champion", + ) + training = _evaluate_evolution_target( + training_target, + plan.plan.objective, + plan.registry, + ) + heldout = Tuple( + _evaluate_evolution_target( + _evolution_target(target, champion_parameters, "champion"), + plan.plan.objective, + plan.registry, + ) + for target in plan.plan.heldout_targets + ) + return EvolutionResult( + plan, + optimizer_summary, + plan.optimizer_seed, + candidate_history, + convergence, + champion_coordinates, + champion_parameters, + champion_fitness, + training, + heldout, + ) +end + +function _evolution_metadata_row(parameter::ParameterSpec, value) + evolution = parameter.evolve + if hasproperty(evolution, :values) + return ( + parameter=parameter.name, + owner=parameter.owner, + value=value, + default=parameter.default, + scale=:categorical, + lower=missing, + upper=missing, + mutation_scale=_evolution_mutation_scale(parameter), + values=evolution.values, + ) + end + return ( + parameter=parameter.name, + owner=parameter.owner, + value=value, + default=parameter.default, + scale=evolution.scale, + lower=evolution.lower, + upper=evolution.upper, + mutation_scale=_evolution_mutation_scale(parameter), + values=missing, + ) +end + +function tables(result::EvolutionResult) + convergence = [ + ( + generation=row.generation, + best_individual=row.best_individual, + fitness_best=row.fitness_best, + fitness_median=row.fitness_median, + fitness_mean=row.fitness_mean, + fitness_worst=row.fitness_worst, + ) + for row in result.convergence + ] + candidates = [ + ( + generation=row.generation, + individual=row.individual, + coordinates=Tuple(row.coordinates), + parameters=_composition_namedtuple(row.parameters), + objective_values=Tuple(row.objective_values), + fitness=row.fitness, + ) + for row in result.candidates + ] + champion_parameters = [ + _evolution_metadata_row( + parameter, + result.champion_parameters[parameter.name], + ) + for parameter in result.plan.parameters + ] + heldout_trials = NamedTuple[] + for evaluation in result.heldout + append!(heldout_trials, trial_table(evaluation.batch)) + end + return ( + convergence=convergence, + candidates=candidates, + champion_parameters=champion_parameters, + training_trials=trial_table(result.training.batch), + heldout_trials=heldout_trials, + optimizer=[( + optimizer=result.plan.plan.optimizer, + optimizer_seed=result.optimizer_seed, + generations=result.plan.plan.generations, + popsize=result.plan.plan.popsize, + )], + ) +end + +function summary(result::EvolutionResult) + heldout = Tuple( + ( + target=evaluation.target, + objective=evaluation.objective, + score=evaluation.aggregate, + trials=length(evaluation.objective_values), + ) + for evaluation in result.heldout + ) + return ( + plan=result.plan.plan.id, + node=result.plan.node.id, + training_target=result.training.target, + objective=result.plan.plan.objective, + optimizer=result.plan.plan.optimizer, + optimizer_seed=result.optimizer_seed, + generations=result.plan.plan.generations, + popsize=result.plan.plan.popsize, + champion_training_score=result.training.aggregate, + champion_parameters=_composition_namedtuple(result.champion_parameters), + heldout=heldout, + ) +end diff --git a/test/test_evolution_plan.jl b/test/test_evolution_plan.jl new file mode 100644 index 0000000..ef0fcb0 --- /dev/null +++ b/test/test_evolution_plan.jl @@ -0,0 +1,121 @@ +using BrainlessLab +using Test + +isdefined(BrainlessLab, :EvolutionResult) || Base.include( + BrainlessLab, + joinpath(pkgdir(BrainlessLab), "src", "operations", "Evolution.jl"), +) + +function _tiny_evolution_target(id, task; root_seed, blocks=1) + base = default_composition(DEFAULT_REGISTRY, :falandays, task) + composition = CompositionSpec( + Symbol(id, :_composition), + base.node, + base.task; + body=base.body, + n_agents=base.n_agents, + n_nodes=8, + parameters=base.parameters, + task_options=base.task_options, + body_options=base.body_options, + interaction_cycle=base.interaction_cycle, + ) + evaluation = EvaluationSpec( + blocks=blocks, + trials_per_block=1, + horizon=4, + root_seed=root_seed, + aggregate=:mean, + ) + return EvaluationTarget(id, composition, evaluation) +end + +@testset "typed evolution plan" begin + training = _tiny_evolution_target(:tracking_train, :tracking; root_seed=101, blocks=2) + heldout = _tiny_evolution_target(:pong_heldout, :pong; root_seed=202) + plan = EvolutionPlan( + :tiny_cross_task, + training; + heldout_targets=(heldout,), + parameter_set=:evolve, + generations=1, + popsize=2, + sigma0=0.1, + ) + + @test validate(plan, DEFAULT_REGISTRY) === plan + resolved = resolve(plan, DEFAULT_REGISTRY) + @test resolved isa BrainlessLab.ResolvedEvolutionPlan + @test getfield.(resolved.parameters, :name) == ( + :leak, + :lrate_wmat, + :lrate_targ, + :threshold_mult, + :targ_min, + :input_weight, + :weight_init_std, + ) + @test resolved.optimizer_seed != training.evaluation.root_seed + + result = execute(resolved) + @test result isa BrainlessLab.EvolutionResult + @test length(result.candidates) == 2 + @test length(result.convergence) == 1 + @test all(candidate -> length(candidate.objective_values) == 2, result.candidates) + @test result.training.target === :tracking_train + @test length(result.training.objective_values) == 2 + @test length(result.heldout) == 1 + @test result.heldout[1].target === :pong_heldout + @test isfinite(result.training.aggregate) + @test isfinite(result.heldout[1].aggregate) + + output_tables = tables(result) + @test length(output_tables.convergence) == 1 + @test length(output_tables.candidates) == 2 + @test length(output_tables.champion_parameters) == 7 + @test output_tables.champion_parameters[1].parameter === :leak + @test length(output_tables.training_trials) == 2 + @test length(output_tables.heldout_trials) == 1 + @test output_tables.optimizer[1].optimizer_seed == result.optimizer_seed + @test !hasproperty(output_tables.training_trials[1], :optimizer_seed) + + report = BrainlessLab.summary(result) + @test report.plan === :tiny_cross_task + @test report.training_target === :tracking_train + @test report.heldout[1].target === :pong_heldout + @test propertynames(report.champion_parameters) == ( + :input_weight, + :leak, + :lrate_targ, + :lrate_wmat, + :targ_min, + :threshold_mult, + :weight_init_std, + ) +end + +@testset "evolution validation follows node metadata" begin + training = _tiny_evolution_target(:tracking_train, :tracking; root_seed=303) + missing_set = EvolutionPlan( + :missing_set, + training; + parameter_set=:not_registered, + generations=1, + popsize=2, + ) + @test_throws KeyError validate(missing_set, DEFAULT_REGISTRY) + + no_scalar = EvaluationTarget( + :tracking_no_aggregate, + training.composition, + EvaluationSpec(horizon=4, root_seed=303, aggregate=:none), + ) + invalid = EvolutionPlan( + :no_scalar, + no_scalar; + parameter_set=:evolve, + generations=1, + popsize=2, + ) + @test_throws ArgumentError validate(invalid, DEFAULT_REGISTRY) +end From 676fc61a86ce8c78848e2a367878221f0768ccf7 Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Wed, 22 Jul 2026 23:22:14 -0400 Subject: [PATCH 14/20] feat: integrate typed research executors --- src/BrainlessLab.jl | 22 ++++++++++++++++ src/core/Catalog.jl | 53 +++++++++++++++++++++++++++++++++++++- test/runtests.jl | 4 +++ test/test_ablation_plan.jl | 6 ----- test/test_profile_plan.jl | 5 ---- test/test_sweep_plan.jl | 6 ----- 6 files changed, 78 insertions(+), 18 deletions(-) diff --git a/src/BrainlessLab.jl b/src/BrainlessLab.jl index d993504..1934a28 100644 --- a/src/BrainlessLab.jl +++ b/src/BrainlessLab.jl @@ -71,6 +71,10 @@ include("core/Catalog.jl") include("api/Composition.jl") include("operations/Plans.jl") include("operations/Evaluation.jl") +include("operations/Profile.jl") +include("operations/Sweep.jl") +include("operations/Ablation.jl") +include("operations/Evolution.jl") include("operations/Benchmark.jl") include("records/PlanIO.jl") include("analysis/ActivityLevels.jl") @@ -706,6 +710,24 @@ export EvaluationTrial, export ResolvedBenchmarkPlan, BenchmarkResult +export ResolvedProfilePlan, + ProfileAnalysisRow, + ProfileAnalysisSummary, + ProfileSummary, + ProfileResult, + ProfileAnalysisError, + ResolvedSweepCell, + ResolvedSweepPlan, + SweepResult, + ResolvedAblationCase, + ResolvedAblationPlan, + AblationResult, + ResolvedEvolutionPlan, + EvolutionCandidate, + EvolutionGeneration, + EvolutionEvaluation, + EvolutionResult + export PLAN_FORMAT, PLAN_FORMAT_VERSION, plan_document, diff --git a/src/core/Catalog.jl b/src/core/Catalog.jl index 3fcc8ce..dbbab8e 100644 --- a/src/core/Catalog.jl +++ b/src/core/Catalog.jl @@ -286,6 +286,52 @@ function _falandays_reference_composition(task::Symbol) ) end +function _with_composition_parameters( + source::CompositionSpec, + suffix::Symbol, + overrides::Pair..., +) + parameters = copy(source.parameters) + foreach(override -> (parameters[Symbol(first(override))] = last(override)), overrides) + return CompositionSpec( + Symbol(source.id, :__, suffix), + source.node, + source.task; + body=source.body, + n_agents=source.n_agents, + n_nodes=source.n_nodes, + parameters=parameters, + task_options=copy(source.task_options), + body_options=copy(source.body_options), + interaction_cycle=source.interaction_cycle, + ) +end + + +function _typed_builtin_ablation(id::Symbol) + id === :freeze_plasticity && return AblationSpec( + id, + composition -> _with_composition_parameters( + composition, + :freeze_plasticity, + :learn_on => false, + ); + required_capabilities=(:online_plasticity,), + description="Disable online plasticity while preserving the composition.", + ) + id === :clamp_target && return AblationSpec( + id, + composition -> _with_composition_parameters( + composition, + :clamp_target, + :lrate_targ => 0.0, + ); + required_capabilities=(:homeostatic_target,), + description="Clamp the homeostatic target by setting its adaptation rate to zero.", + ) + return nothing +end + function register_builtins!(registry::RegistrySet) register!(registry, falandays_node_spec()) for (id, constructor) in sort!(collect(NODES); by=pair -> string(first(pair))) @@ -333,7 +379,12 @@ function register_builtins!(registry::RegistrySet) register!(registry, :optimizers, ImplementationSpec(id, implementation)) end for (id, implementation) in sort!(collect(ABLATIONS); by=pair -> string(first(pair))) - register!(registry, :ablations, ImplementationSpec(id, implementation)) + typed = _typed_builtin_ablation(id) + register!( + registry, + :ablations, + ImplementationSpec(id, typed === nothing ? implementation : typed), + ) end for task in (:wall, :tracking, :pong) diff --git a/test/runtests.jl b/test/runtests.jl index 3845d36..3cad704 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -43,6 +43,10 @@ include("test_composition_spec.jl") include("test_operation_plans.jl") include("test_benchmark_plan.jl") include("test_plan_io.jl") +include("test_profile_plan.jl") +include("test_sweep_plan.jl") +include("test_ablation_plan.jl") +include("test_evolution_plan.jl") include("test_morphology.jl") include("test_homeostasis.jl") include("test_sensor.jl") diff --git a/test/test_ablation_plan.jl b/test/test_ablation_plan.jl index a0a38b5..69e9f87 100644 --- a/test/test_ablation_plan.jl +++ b/test/test_ablation_plan.jl @@ -1,12 +1,6 @@ using BrainlessLab using Test -module AblationOperation -using BrainlessLab -import BrainlessLab: execute, resolve, summary, tables, validate -include(joinpath(@__DIR__, "..", "src", "operations", "Ablation.jl")) -end - function _ablation_registry() registry = RegistrySet() register!(registry, falandays_node_spec()) diff --git a/test/test_profile_plan.jl b/test/test_profile_plan.jl index 4134a9d..9e2b596 100644 --- a/test/test_profile_plan.jl +++ b/test/test_profile_plan.jl @@ -1,11 +1,6 @@ using BrainlessLab using Test -Base.include( - BrainlessLab, - joinpath(@__DIR__, "..", "src", "operations", "Profile.jl"), -) - function _profile_tracking_target(; blocks=1, trials=2, horizon=8) composition = CompositionSpec( :profile_tracking_smoke, diff --git a/test/test_sweep_plan.jl b/test/test_sweep_plan.jl index dacbeee..a882117 100644 --- a/test/test_sweep_plan.jl +++ b/test/test_sweep_plan.jl @@ -1,12 +1,6 @@ using BrainlessLab using Test -module SweepOperation -using BrainlessLab -import BrainlessLab: execute, resolve, summary, tables, validate -include(joinpath(@__DIR__, "..", "src", "operations", "Sweep.jl")) -end - function _small_sweep_target(; blocks=1, trials=2, horizon=3) reference = default_composition(DEFAULT_REGISTRY, :falandays, :tracking) composition = CompositionSpec( From f4e417fc073c24acce1793e69a69fd1d600ef563 Mon Sep 17 00:00:00 2001 From: btgaskin <135119655+btgaskin@users.noreply.github.com> Date: Thu, 23 Jul 2026 01:30:49 -0400 Subject: [PATCH 15/20] feat: establish typed research platform --- .github/workflows/ci.yml | 5 + CHANGELOG.md | 15 +- CITATION.cff | 2 +- Manifest.toml | 6 +- Project.toml | 4 +- README.md | 59 +- bench/Manifest.toml | 6 +- bench/Project.toml | 2 +- bin/brainlesslab.jl | 79 ++ examples/templates/new_project/Project.toml | 2 +- examples/templates/new_project/README.md | 75 +- examples/templates/new_project/config.toml | 38 +- examples/templates/new_project/my_node.jl | 63 +- examples/templates/new_project/my_task.jl | 10 +- examples/templates/new_project/run.jl | 9 +- examples/templates/new_project/run_plan.jl | 14 + experiments/freeze_onset.jl | 2 +- experiments/run.jl | 11 +- experiments/shoal_vision_sweep.jl | 4 +- experiments/tracking_leak_lrate_factorial.jl | 2 +- experiments/tracking_param_sweep.jl | 2 +- plans/README.md | 27 + plans/examples/ablate_tracking.toml | 25 + plans/examples/benchmark_core.toml | 80 ++ plans/examples/evolve_pong_test_tracking.toml | 65 ++ plans/examples/evolve_tracking_test_pong.toml | 65 ++ plans/examples/profile_tracking.toml | 26 + plans/examples/sweep_tracking.toml | 34 + profile/Manifest.toml | 6 +- profile/Project.toml | 2 +- site/src/content/docs/core/architecture.mdx | 56 +- site/src/content/docs/core/design-study.mdx | 97 ++- site/src/content/docs/core/extend.mdx | 40 +- .../src/content/docs/core/getting-started.mdx | 40 +- .../content/docs/core/interaction-cycle.mdx | 7 +- site/src/content/docs/core/reservoirs.mdx | 9 +- site/src/content/docs/core/runs-results.mdx | 36 +- site/src/content/docs/core/task-tour.mdx | 32 +- .../src/content/docs/core/tools-artifacts.mdx | 235 +++--- skills/brainless-lab/SKILL.md | 389 +++++---- skills/brainless-lab/references/cli-tools.md | 292 +++---- .../references/designing-analyses.md | 33 +- .../designing-environments-and-tasks.md | 5 +- .../references/designing-nodes.md | 67 +- .../references/usage-and-workflows.md | 19 +- src/BrainlessLab.jl | 28 +- src/api/Composition.jl | 38 +- src/core/Composition.jl | 32 +- src/operations/Benchmark.jl | 68 +- src/operations/Evaluation.jl | 18 +- src/operations/Evolution.jl | 42 +- src/operations/Plans.jl | 109 ++- src/records/ExperimentIO.jl | 167 ++++ src/records/PlanIO.jl | 30 +- src/records/Records.jl | 788 ++++++++++++++++++ src/run/Evaluation.jl | 232 ------ src/tasks/Tasks.jl | 14 +- test/runtests.jl | 3 + test/test_benchmark_plan.jl | 35 +- test/test_composition_spec.jl | 33 +- test/test_evolution_plan.jl | 2 + test/test_experiment_io.jl | 61 ++ test/test_operation_plans.jl | 30 + test/test_plan_examples.jl | 20 + test/test_plan_io.jl | 21 +- test/test_plank_cartpole.jl | 71 +- test/test_records.jl | 196 +++++ 67 files changed, 3080 insertions(+), 1055 deletions(-) create mode 100644 bin/brainlesslab.jl create mode 100644 examples/templates/new_project/run_plan.jl create mode 100644 plans/README.md create mode 100644 plans/examples/ablate_tracking.toml create mode 100644 plans/examples/benchmark_core.toml create mode 100644 plans/examples/evolve_pong_test_tracking.toml create mode 100644 plans/examples/evolve_tracking_test_pong.toml create mode 100644 plans/examples/profile_tracking.toml create mode 100644 plans/examples/sweep_tracking.toml create mode 100644 src/records/ExperimentIO.jl create mode 100644 src/records/Records.jl delete mode 100644 src/run/Evaluation.jl create mode 100644 test/test_experiment_io.jl create mode 100644 test/test_plan_examples.jl create mode 100644 test/test_records.jl diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b63fd21..a260283 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -70,10 +70,15 @@ jobs: run: julia --project=profile -e 'include("profile/Profile.jl"); using .NodeProfile; out = NodeProfile.node_profile(:falandays; tasks=(:tracking,), n_seeds=1, canonical_N=Dict(:tracking => 12), gifs=false, out_root=mktempdir()); @assert isfile(out.metrics)' - name: Sweep execution smoke run: julia --project=. -e 'using BrainlessLab; out = run_sweep("configs/ci_sweep.toml"; root=mktempdir()); @assert isfile(out.results)' + - name: Unified plan and record smoke + run: | + julia --project=. bin/brainlesslab.jl check plans/examples/benchmark_core.toml + julia --project=. bin/brainlesslab.jl run plans/examples/profile_tracking.toml --root "${{ runner.temp }}/brainlesslab-records" - name: Instantiate and execute project template run: | julia --project=examples/templates/new_project -e 'using Pkg; Pkg.develop(path=pwd()); Pkg.instantiate()' julia --project=examples/templates/new_project examples/templates/new_project/run.jl --ticks 20 --n-nodes 12 --out "${{ runner.temp }}/brainlesslab-template-smoke" + julia --project=examples/templates/new_project examples/templates/new_project/run_plan.jl examples/templates/new_project/config.toml "${{ runner.temp }}/brainlesslab-template-records" documentation: name: Locked documentation build diff --git a/CHANGELOG.md b/CHANGELOG.md index b38369a..176ad45 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,10 +1,17 @@ # Changelog -## 0.1.1 — 2026-07-22 +## 0.2.0 — 2026-07-22 -BrainlessLab 0.1.1 is an experimental research preview. +BrainlessLab 0.2.0 is an experimental research preview and the first typed research-platform release. -- Align package and citation metadata on version 0.1.1. +- Add typed node, task, composition, evaluation, operation, and experiment contracts. +- Add one version-one TOML plan schema for profile, sweep, ablate, evolve, and benchmark. +- Add portable research records with raw CSV data, seed ledgers, checksums, statistics, + resolved provenance, and generated HTML reports. +- Add four experimental Plank CartPole profiles while keeping Tracking and Pong as the + core qualification benchmark. +- Add checked-in operation plans, a unified CLI, and an external-project template. +- Align package and citation metadata on version 0.2.0. - Add reproducible package, compatibility-floor, tool-smoke, and documentation CI. - Add package-quality checks without weakening numerical conformance or calibration tests. - Clarify repository installation and the pre-1.0 stability boundary. @@ -12,5 +19,5 @@ BrainlessLab 0.1.1 is an experimental research preview. ## 0.1.0 — 2026-07-04 The public tag named `v0.1.0` was created from code whose `Project.toml` still reported -version `0.0.1`. That historical tag remains immutable; 0.1.1 is the first release in +version `0.0.1`. That historical tag remains immutable; 0.2.0 is the first release in which the package and citation versions are aligned. diff --git a/CITATION.cff b/CITATION.cff index bbb5647..998ad70 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -17,7 +17,7 @@ authors: family-names: Jackson - given-names: William family-names: O'Hearn -version: 0.1.1 +version: 0.2.0 date-released: "2026-07-22" license: MIT repository-code: "https://github.com/btgaskin/brainless-lab" diff --git a/Manifest.toml b/Manifest.toml index cb4be76..bbe2d47 100644 --- a/Manifest.toml +++ b/Manifest.toml @@ -2,7 +2,7 @@ julia_version = "1.12.6" manifest_format = "2.0" -project_hash = "1936d7250a908cf710d7a57233db3835c77344fd" +project_hash = "d0c124f48ec6660ce476a73cdbaf1dd0ab5a4f58" [[deps.ArgTools]] uuid = "0dad84c5-d112-42e6-8d28-ef12dabb789f" @@ -17,10 +17,10 @@ uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f" version = "1.11.0" [[deps.BrainlessLab]] -deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "StaticArrays", "Statistics", "TOML"] +deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "SHA", "StaticArrays", "Statistics", "TOML"] path = "." uuid = "d12add44-1e3e-4161-9a99-c2121a2f0f38" -version = "0.1.1" +version = "0.2.0" [deps.BrainlessLab.extensions] BrainlessLabMakieExt = "Makie" diff --git a/Project.toml b/Project.toml index 9f460e0..a4a4e56 100644 --- a/Project.toml +++ b/Project.toml @@ -1,13 +1,14 @@ name = "BrainlessLab" uuid = "d12add44-1e3e-4161-9a99-c2121a2f0f38" authors = ["btgaskin "] -version = "0.1.1" +version = "0.2.0" [deps] Dates = "ade2ca70-3891-5945-98fb-dc099432e06a" JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819" LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e" Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" +SHA = "ea8e919c-243c-51af-8825-aaa63cd721ce" StaticArrays = "90137ffa-7385-5640-81b9-e52037218182" Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2" TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76" @@ -27,6 +28,7 @@ LinearAlgebra = "1.10" Makie = "0.21, 0.22, 0.23, 0.24" NPZ = "0.4" Random = "1.10" +SHA = "0.7" StaticArrays = "1" Statistics = "1.10" TOML = "1" diff --git a/README.md b/README.md index 215d4d1..b4cd6a1 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Diverse Intelligences Summer Institute 2026

-BrainlessLab v0.1.1 is an **experimental research preview** for neural reservoirs in +BrainlessLab v0.2.0 is an **experimental research preview** for neural reservoirs in closed sensorimotor loops. It provides tasks, generic embodiment, single-agent and population worlds, recording, analysis, batch tools, and evidence-aware experiment workflows. APIs and artifact layouts may change before 1.0. @@ -52,16 +52,24 @@ Continue with: 3. [Architecture](https://brainless-lab.pages.dev/core/architecture/) 4. [Design a study](https://brainless-lab.pages.dev/core/design-study/) +For a repeatable multi-run operation, validate a checked-in plan and write one portable +record: + +```bash +julia --project=. bin/brainlesslab.jl check plans/examples/profile_tracking.toml +julia -t auto --project=. bin/brainlesslab.jl run plans/examples/profile_tracking.toml --root records +``` + +Every operation writes the same record shape: the request, fully resolved settings, seed +ledger, raw CSV tables, summary statistics, checksums, and a generated HTML report. + ## Core composition ```text -NodeModel → Reservoir → AbstractBody → Agent → Ensemble{Environment} - ↓ - Task - ↓ - Runner → Run - ↘ - Recorder +NodeSpec + TaskSpec + body + InteractionCycle → CompositionSpec → closed-loop runtime +CompositionSpec + EvaluationSpec → EvaluationTarget +EvaluationTarget(s) → operation plan → record +named conditions + operation plans → ExperimentSpec ``` `AbstractBody` is the public body boundary. `Embodiment` is the generic concrete @@ -84,26 +92,37 @@ and Pong tasks are the first core task contracts. ```julia using BrainlessLab -variants() -tasks() -analyses() -ablations() +nodes(DEFAULT_REGISTRY) +tasks(DEFAULT_REGISTRY) +tasks(DEFAULT_REGISTRY; tag=:benchmark) +analyses(DEFAULT_REGISTRY) +ablations(DEFAULT_REGISTRY) +compositions(DEFAULT_REGISTRY) components() readiness() ``` -Use these registries instead of copying a static symbol list. +`DEFAULT_REGISTRY` is the canonical composition and operation catalog. The zero-argument +`variants()`, `tasks()`, and related registration helpers remain only for the established +interactive `simulate(:task; node=:node)` façade and older research scripts; do not use +them for new plans or extensions. ## Execution surfaces - `simulate` runs one closed loop and returns an in-memory `SimResult`. -- `sweep/run.jl` runs bounded, resumable development sweeps. -- `experiments/run.jl` runs declared multi-condition protocols. -- `calibration/`, `profile/`, `bench/`, and evolution tools serve specialized questions. - -Start with the smallest tool that can answer the question. A selected sweep cell is a -development result, not a confirmed optimum. Agents and ticks in one world do not increase -the number of independent experimental units. +- `ProfilePlan` characterizes one registered node on one registered task. +- `SweepPlan` maps declared node parameters under paired evaluation seeds. +- `AblationPlan` disables registered functional elements against an implicit baseline. +- `EvolutionPlan` selects parameters on one target and evaluates the champion on declared + held-out targets. +- `BenchmarkPlan` reports paired within-task comparisons without forming a cross-task score. +- `ExperimentSpec` registers a versioned scientific protocol above one or more operations. + +All five operations use the same version-one TOML schema and record writer. The older +specialized directories remain research code, but they are no longer the canonical public +workflow. Start with the smallest operation that can answer the question. A selected sweep +cell is a development result, not a confirmed optimum. Agents and ticks in one world do not +increase the number of independent experimental units. See [Tools and artifacts](https://brainless-lab.pages.dev/core/tools-artifacts/) and [Runs, recording, and results](https://brainless-lab.pages.dev/core/runs-results/). diff --git a/bench/Manifest.toml b/bench/Manifest.toml index b509ca1..3a435a0 100644 --- a/bench/Manifest.toml +++ b/bench/Manifest.toml @@ -2,7 +2,7 @@ julia_version = "1.12.6" manifest_format = "2.0" -project_hash = "63394ceac60298dd43be0560b12adaef165476e3" +project_hash = "37f4270f12b4299ac734e35140cd463808bca792" [[deps.AbstractFFTs]] deps = ["LinearAlgebra"] @@ -108,10 +108,10 @@ uuid = "18cc8868-cbac-4acf-b575-c8ff214dc66f" version = "1.4.0" [[deps.BrainlessLab]] -deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "StaticArrays", "Statistics", "TOML"] +deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "SHA", "StaticArrays", "Statistics", "TOML"] path = ".." uuid = "d12add44-1e3e-4161-9a99-c2121a2f0f38" -version = "0.1.1" +version = "0.2.0" weakdeps = ["Makie"] [deps.BrainlessLab.extensions] diff --git a/bench/Project.toml b/bench/Project.toml index d8d5815..ec91685 100644 --- a/bench/Project.toml +++ b/bench/Project.toml @@ -9,7 +9,7 @@ TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76" Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40" [compat] -BrainlessLab = "0.1.1" +BrainlessLab = "0.2" CairoMakie = "0.12, 0.13, 0.14, 0.15" Dates = "1.10" JLD2 = "0.5" diff --git a/bin/brainlesslab.jl b/bin/brainlesslab.jl new file mode 100644 index 0000000..1da616b --- /dev/null +++ b/bin/brainlesslab.jl @@ -0,0 +1,79 @@ +#!/usr/bin/env julia + +using BrainlessLab + +function usage(io=stdout) + println(io, "Usage:") + println(io, " julia --project=. bin/brainlesslab.jl check PLAN.toml") + println(io, " julia --project=. bin/brainlesslab.jl run PLAN.toml [--root DIR]") + println(io, " julia --project=. bin/brainlesslab.jl check-experiment PROTOCOL_DIR") + println(io, " julia --project=. bin/brainlesslab.jl run-experiment PROTOCOL_DIR [--root DIR]") +end + +function parse_run_options(args) + root = "records" + index = 1 + while index <= length(args) + args[index] == "--root" || throw(ArgumentError( + "unknown run option $(repr(args[index]))", + )) + index < length(args) || throw(ArgumentError("--root requires a directory")) + root = args[index + 1] + index += 2 + end + return root +end + +function main(args=ARGS) + length(args) >= 2 || begin + usage(stderr) + return 2 + end + command = args[1] + command in ("check", "run", "check-experiment", "run-experiment") || begin + usage(stderr) + return 2 + end + source_path = args[2] + + if command in ("check-experiment", "run-experiment") + isdir(source_path) || throw(ArgumentError( + "experiment protocol directory does not exist: $(source_path)", + )) + experiment = read_experiment(source_path) + if command == "check-experiment" + println("valid experiment: ", experiment.id) + println("version: ", experiment.version) + println("evidence state: ", experiment.evidence_state) + println("operations: ", join(string.(getfield.(experiment.operations, :id)), ", ")) + return 0 + end + root = parse_run_options(args[3:end]) + run = run_experiment(experiment; root=root) + println("experiment record: ", run.directory) + println("operation records: ", join(run.records, ", ")) + return 0 + end + + isfile(source_path) || throw(ArgumentError("plan does not exist: $(source_path)")) + plan = read_plan(source_path) + resolved = resolve(plan, DEFAULT_REGISTRY) + + if command == "check" + println("valid plan: ", plan.id) + println("operation: ", operation_kind(plan)) + println("targets: ", join(string.(getfield.(operation_targets(plan), :id)), ", ")) + println("resolved: ", nameof(typeof(resolved))) + return 0 + end + + root = parse_run_options(args[3:end]) + run = run_operation(plan; root=root) + println("record: ", run.directory) + println("summary: ", summary(run.result)) + return 0 +end + +if abspath(PROGRAM_FILE) == @__FILE__ + exit(main()) +end diff --git a/examples/templates/new_project/Project.toml b/examples/templates/new_project/Project.toml index 292611a..3b0d831 100644 --- a/examples/templates/new_project/Project.toml +++ b/examples/templates/new_project/Project.toml @@ -9,7 +9,7 @@ CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0" Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c" [compat] -BrainlessLab = "0.1.1" +BrainlessLab = "0.2" CairoMakie = "0.12, 0.13, 0.14, 0.15" Random = "1.10" julia = "1.10" diff --git a/examples/templates/new_project/README.md b/examples/templates/new_project/README.md index d648afe..6d1b022 100644 --- a/examples/templates/new_project/README.md +++ b/examples/templates/new_project/README.md @@ -8,15 +8,16 @@ observations, actions, metrics, controls, and calibration before editing. ## Files -- `my_node.jl` defines `MyNode <: Reservoir`, a leaky homeostatic reservoir with online recurrent-weight and target adaptation, then registers it as `:my_node`. -- `my_task.jl` defines `MyTrackingEnv <: TaskWorld`, wraps it in a `TaskSpec`, then registers it as `:my_task`. +- `my_node.jl` defines `MyNode <: Reservoir`, then registers a typed `NodeSpec` with parameters, capabilities, and default sweep/evolution sets. +- `my_task.jl` defines `MyTrackingEnv <: TaskWorld`, wraps it in a `TaskSpec`, then registers the task in `DEFAULT_REGISTRY`. - `my_metric.jl` registers a small metric function as `:final_error_abs`, requested by symbol in `run.jl`. -- `run.jl` includes those three files, runs `simulate(:my_task; node=:my_node)`, prints metrics, and saves a Makie figure. -- `config.toml` is a benchmark config snippet that follows `bench/configs/core.toml`. +- `run.jl` includes those files, runs one explicit `CompositionSpec`, prints metrics, and saves a Makie figure. +- `config.toml` is a version-one `ProfilePlan` using the same node, task, and evaluation contracts as every built-in operation. +- `run_plan.jl` loads the extension, executes `config.toml`, and writes the standard portable record. ## Setup -From this directory: +From this directory while the template remains inside a BrainlessLab checkout: ```bash julia --project=. -e 'using Pkg; Pkg.develop(path="../../.."); Pkg.instantiate()' @@ -28,7 +29,16 @@ Run the example with this template environment: julia --project=. run.jl ``` -The template `Project.toml` depends on `BrainlessLab` and `CairoMakie`. `Pkg.develop(path="../../..")` points the template environment at the local framework checkout, so you can copy this directory into your own project and keep using the framework as a dependency instead of editing `src/`. +The template `Project.toml` depends on `BrainlessLab` and `CairoMakie`. +`Pkg.develop(path="../../..")` points this in-repository copy at the local framework +checkout. After copying the template elsewhere, install the public package source instead: + +```bash +julia --project=. -e 'using Pkg; Pkg.add(url="https://github.com/btgaskin/brainless-lab"); Pkg.instantiate()' +``` + +Once BrainlessLab is registered in Julia General, `Pkg.add("BrainlessLab")` becomes the +normal installation path. ## First Result @@ -41,6 +51,15 @@ Artifacts: - Printed task metrics, including `score`, `mean_abs_error`, `final_error`, liveness, and the registered custom `final_error_abs`. - `output/my_task_my_node_visualize.png`, containing spike raster, population firing rate, and spike-pattern drift panels. +Then run the repeatable profile: + +```bash +julia --project=. run_plan.jl config.toml records +``` + +Open `records//report/index.html`, or inspect the authoritative CSV tables and +the checksums in `record.toml`. + ## Node Contract A high-level node registered for `simulate` must be callable as: @@ -61,7 +80,13 @@ n_receptors(node) n_effectors(node) ``` -`my_node.jl` also implements `snapshot_state` and `load_state!` to show the parameter/state split. `MyNodeParams` is static configuration; `acts`, `targets`, `spikes`, `errors`, and `wmat` are rollout state. The registration declares `genome_type=MyNodeParams`, so `rollout` and `evolve` can derive the genome dimension through `paramdim`, `pack_params`, and `unpack_params`. +`my_node.jl` also implements `snapshot_state` and `load_state!` to show the parameter/state split. `MyNodeParams` is static configuration; `acts`, `targets`, `spikes`, `errors`, and `wmat` are rollout state. + +The public `NodeSpec` builder receives a `NodeBuildContext` and the fully resolved parameter +dictionary. The context supplies node count, body ports, named seeds, and any receptor +profile. `ParameterSpec` declares validation, default sweep values, evolution bounds, and +ownership. Here `link_p` is reservoir-owned connectivity while node count remains part of +the composition. Important Julia gotcha: when extending BrainlessLab generics from outside the package, import the names you extend: @@ -109,31 +134,16 @@ register_metric!(:final_error_abs, final_error_abs) In `run.jl`, the simulation requests the metric with `metrics=[:final_error_abs]`; the high-level runner resolves the symbol and appends the derived value to `sim.metrics`. -## Benchmark - -`config.toml` follows the schema in `../../../bench/configs/core.toml`: - -```toml -neurons = ["falandays_base", "my_node"] -tasks = ["my_task"] -n_trials = 5 -n_nodes = 80 -ticks = 300 -baseline = "falandays_base" - -[prep] -my_node = "untrained" -``` +## Operations -The benchmark runner loads registered BrainlessLab symbols, then uses the node's declared `genome_type` to stamp parameters through the public `NodeModel` contract. No framework fork or private-symbol bridge is needed. - -From the repo root, after setting up `bench/` as described in `../../../bench/README.md`, run: - -```bash -julia --project=bench -e 'include("examples/templates/new_project/my_node.jl"); include("examples/templates/new_project/my_task.jl"); include("bench/Benchmark.jl"); using .Benchmark; cfg = Benchmark.read_bench_config("examples/templates/new_project/config.toml"); result = Benchmark.run_benchmark(cfg); println(result.dir); Benchmark.print_short_summary(result.summaries)' -``` +`config.toml` uses the single `brainlesslab-plan` schema. Change `operation` and its final +section to profile, sweep, ablate, evolve, or benchmark. The target composition and +evaluation section stay the same. -Benchmark artifacts are written under `bench/runs/` and include resolved config, manifest, raw trial CSV, summary CSV, stats JSON, report Markdown, plots, and per-cell figures. +The node's `:sweep` and `:evolve` parameter sets provide defaults. A plan can instead name +explicit sweep axes or another registered parameter set. Benchmark conditions reference +registered nodes and tasks but remain task-specific; registering a component does not +automatically qualify it for a benchmark. ## Make It Your Own @@ -142,7 +152,8 @@ Benchmark artifacts are written under `bench/runs/` and include resolved config, 3. Keep receptor and effector counts aligned: for a vector task, `TaskSpec.n_receptors` must match `sense(env)`; for a composed body, use `portspec(body)` as the source of truth. 4. Keep online plasticity inside `step!`; no evolution is needed for a Falandays-style first experiment. 5. Add task-specific metrics to `metrics(env, window)` first. Use `register_metric!` for reusable analysis functions that can be resolved by symbol. -6. Start with `simulate` and `visualize`; move to `bench/` only after the single run behaves sensibly. +6. Start with `simulate` and `visualize`; move to `ProfilePlan`, `SweepPlan`, or + `BenchmarkPlan` only after the single composition behaves sensibly. ## Read More @@ -154,4 +165,4 @@ The docs live in the Astro/Starlight site (, or - [Extending it](https://brainless-lab.pages.dev/extending/) - [Research workflow](https://brainless-lab.pages.dev/research-workflow/) - [Agentic workflow](https://brainless-lab.pages.dev/agentic-workflow/) -- `../../../bench/README.md` +- [Operations and records](https://brainless-lab.pages.dev/core/tools-artifacts/) diff --git a/examples/templates/new_project/config.toml b/examples/templates/new_project/config.toml index 5ddf654..edb46e5 100644 --- a/examples/templates/new_project/config.toml +++ b/examples/templates/new_project/config.toml @@ -1,18 +1,28 @@ -# Benchmark config snippet following bench/configs/core.toml. -# Load my_node.jl and my_task.jl before running the benchmark so the symbols are -# registered in BrainlessLab. +format = "brainlesslab-plan" +format_version = 1 +operation = "profile" +id = "my_project_profile" -neurons = ["falandays", "my_node"] -tasks = ["my_task"] +[[targets]] +id = "my_task" -n_trials = 5 +[targets.composition] +id = "my_project" +node = "my_node" +task = "my_task" n_nodes = 80 -ticks = 300 -seed_base = 2000 -baseline = "falandays" -alpha = 0.05 -gifs = false -[prep] -falandays = "untrained" -my_node = "untrained" +[targets.evaluation] +blocks = 2 +trials_per_block = 2 +horizon = 300 +warmup = 50 +construction_scope = "trial" +reset = "full" +root_seed = 3001 +aggregate = "mean" + +[profile] +target = "my_task" +analyses = [] +record_every = 1 diff --git a/examples/templates/new_project/my_node.jl b/examples/templates/new_project/my_node.jl index a0145ad..250312d 100644 --- a/examples/templates/new_project/my_node.jl +++ b/examples/templates/new_project/my_node.jl @@ -4,7 +4,8 @@ import BrainlessLab import BrainlessLab: NodeModel, Reservoir import BrainlessLab: step!, effectors, n_nodes, n_receptors, n_effectors, reset! import BrainlessLab: snapshot_state, load_state!, pack_params, unpack_params, paramdim -import BrainlessLab: plasticity, OnlinePlasticity, register_node! +import BrainlessLab: plasticity, OnlinePlasticity +import BrainlessLab: NodeBuildContext, NodeSpec, ParameterSpec, DEFAULT_REGISTRY, register! Base.@kwdef struct MyNodeParams <: NodeModel leak::Float64 = 0.25 @@ -275,4 +276,62 @@ function load_state!(r::MyNode, state) return r end -register_node!(:my_node, MyNode; genome_type=MyNodeParams) +function build_my_node(context::NodeBuildContext, values) + params = MyNodeParams( + leak=values[:leak], + lrate_wmat=values[:lrate_wmat], + lrate_targ=values[:lrate_targ], + threshold_mult=values[:threshold_mult], + target_floor=values[:target_floor], + input_gain=values[:input_gain], + recurrent_scale=values[:recurrent_scale], + weight_limit=values[:weight_limit], + learn_on=values[:learn_on], + ) + seed = Int(mod(context.seeds.topology, UInt64(typemax(Int)))) + return MyNode( + context.n_nodes, + n_receptors(context.ports), + n_effectors(context.ports); + seed=seed, + params=params, + link_p=values[:link_p], + ) +end + +const MY_NODE_SPEC = NodeSpec( + :my_node, + build_my_node; + genome_type=MyNodeParams, + stability=:experimental, + tags=(:experimental,), + capabilities=(:online_plasticity, :recurrent_weights, :homeostatic_target), + parameters=( + ParameterSpec(:leak, 0.25; sweep=(0.1, 0.25, 0.5), evolve=(lower=0.0, upper=0.95)), + ParameterSpec(:lrate_wmat, 0.04; sweep=(0.01, 0.04, 0.1), evolve=(lower=1.0e-4, upper=1.0, scale=:log)), + ParameterSpec(:lrate_targ, 0.01; evolve=(lower=1.0e-4, upper=0.5, scale=:log)), + ParameterSpec(:threshold_mult, 2.0; evolve=(lower=0.1, upper=5.0)), + ParameterSpec(:target_floor, 1.0; evolve=(lower=0.01, upper=3.0, scale=:log)), + ParameterSpec(:input_gain, 1.4; evolve=(lower=0.0, upper=4.0)), + ParameterSpec(:recurrent_scale, 0.7; evolve=(lower=0.0, upper=3.0)), + ParameterSpec(:weight_limit, 3.0; evolve=(lower=0.1, upper=10.0, scale=:log)), + ParameterSpec(:learn_on, true), + ParameterSpec(:link_p, 0.18; owner=:reservoir, sweep=(0.1, 0.18, 0.3), evolve=(lower=0.01, upper=0.8)), + ), + parameter_sets=Dict( + :sweep => (:leak, :lrate_wmat), + :evolve => ( + :leak, + :lrate_wmat, + :lrate_targ, + :threshold_mult, + :target_floor, + :input_gain, + :recurrent_scale, + :weight_limit, + ), + :connectivity => (:link_p,), + ), +) + +register!(DEFAULT_REGISTRY, MY_NODE_SPEC) diff --git a/examples/templates/new_project/my_task.jl b/examples/templates/new_project/my_task.jl index a958937..943c34c 100644 --- a/examples/templates/new_project/my_task.jl +++ b/examples/templates/new_project/my_task.jl @@ -1,9 +1,9 @@ using Random -import BrainlessLab: TaskSpec, TaskWorld +import BrainlessLab: TaskSpec, TaskWorld, analytic import BrainlessLab: sense, step!, reset!, metrics import BrainlessLab: n_receptors, n_effectors, default_ticks, default_window -import BrainlessLab: register_task! +import BrainlessLab: DEFAULT_REGISTRY, register! mutable struct MyTrackingEnv{R<:AbstractRNG} <: TaskWorld rng::R @@ -130,9 +130,9 @@ const MY_TASK = TaskSpec( MyTrackingEnv; default_ticks=default_ticks(MyTrackingEnv), default_window=default_window(MyTrackingEnv), - score_floor=0.0, - score_ceiling=1.0, + floor=analytic(0.0; note="minimum score under this example contract"), + ceiling=analytic(1.0; note="zero tracking error"), score_key=:score, ) -register_task!(:my_task, MY_TASK) +register!(DEFAULT_REGISTRY, MY_TASK) diff --git a/examples/templates/new_project/run.jl b/examples/templates/new_project/run.jl index 52754d4..dfcc9f9 100644 --- a/examples/templates/new_project/run.jl +++ b/examples/templates/new_project/run.jl @@ -60,11 +60,14 @@ function main(args) mkpath(opts[:out]) sim = simulate( - :my_task; - node=:my_node, + CompositionSpec( + :my_project, + :my_node, + :my_task; + n_nodes=opts[:n_nodes], + ); ticks=opts[:ticks], seed=opts[:seed], - n_nodes=opts[:n_nodes], record=RECORD, metrics=[:final_error_abs], ) diff --git a/examples/templates/new_project/run_plan.jl b/examples/templates/new_project/run_plan.jl new file mode 100644 index 0000000..be1a8eb --- /dev/null +++ b/examples/templates/new_project/run_plan.jl @@ -0,0 +1,14 @@ +#!/usr/bin/env julia + +using BrainlessLab + +include("my_node.jl") +include("my_task.jl") +include("my_metric.jl") + +plan_path = isempty(ARGS) ? joinpath(@__DIR__, "config.toml") : ARGS[1] +root = length(ARGS) >= 2 ? ARGS[2] : joinpath(@__DIR__, "records") +plan = read_plan(plan_path) +run = run_operation(plan; root=root) +println("record: ", run.directory) +println("summary: ", summary(run.result)) diff --git a/experiments/freeze_onset.jl b/experiments/freeze_onset.jl index 73ece17..aa283b5 100644 --- a/experiments/freeze_onset.jl +++ b/experiments/freeze_onset.jl @@ -61,5 +61,5 @@ function run_freeze_onset(; tasks=[:tracking, :wall, :pong], return dir end -register_experiment!(:freeze_onset, run_freeze_onset; +ExpRegistry.register_experiment!(:freeze_onset, run_freeze_onset; description="Freeze plasticity at tick T across single-agent tasks; find the dead→alive onset (normalized score + rate).") diff --git a/experiments/run.jl b/experiments/run.jl index 04b3142..92b3c7f 100644 --- a/experiments/run.jl +++ b/experiments/run.jl @@ -49,15 +49,20 @@ end function main(args) if isempty(args) || first(args) in ("--list", "-l", "list", "--help", "-h") println("Registered experiments (experiments/run.jl [key=val ...]):\n") - for name in experiments() - println(" ", rpad(string(name), 18), " ", experiment_description(name)) + for name in ExpRegistry.experiments() + println( + " ", + rpad(string(name), 18), + " ", + ExpRegistry.experiment_description(name), + ) end return end name = Symbol(first(args)) kw = _parse_kwargs(args[2:end]) println("running :", name, " ", isempty(kw) ? "(defaults)" : kw, "\n") - dir = resolve_experiment(name).run(; kw...) + dir = ExpRegistry.resolve_experiment(name).run(; kw...) println("\nwrote ", dir) end diff --git a/experiments/shoal_vision_sweep.jl b/experiments/shoal_vision_sweep.jl index 50b1e07..67ce19a 100644 --- a/experiments/shoal_vision_sweep.jl +++ b/experiments/shoal_vision_sweep.jl @@ -1024,13 +1024,13 @@ end run_shoal_sensitivity_screen(; diagnostics=false, kwargs...) = run_shoal_vision_sweep(; profile=:sensitivity, diagnostics, kwargs...) -register_experiment!( +ExpRegistry.register_experiment!( :shoal_vision_sweep, run_shoal_vision_sweep; description="Underpowered exploratory sweep of conspecific sight distance, bearing alignment, and association need in moving Falandays shoals.", ) -register_experiment!( +ExpRegistry.register_experiment!( :shoal_sensitivity_screen, run_shoal_sensitivity_screen; description="Underpowered one-factor sensitivity screen for shoal input gains, need dynamics, response curves, and resource sight range.", diff --git a/experiments/tracking_leak_lrate_factorial.jl b/experiments/tracking_leak_lrate_factorial.jl index 76fe252..f02da3e 100644 --- a/experiments/tracking_leak_lrate_factorial.jl +++ b/experiments/tracking_leak_lrate_factorial.jl @@ -215,5 +215,5 @@ function run_tracking_leak_lrate_factorial(; leaks=TRACKING_FACTORIAL_DEFAULT_LE return dir end -register_experiment!(:tracking_leak_lrate_factorial, run_tracking_leak_lrate_factorial; +ExpRegistry.register_experiment!(:tracking_leak_lrate_factorial, run_tracking_leak_lrate_factorial; description="leak × lrate_wmat factorial on the paper tracking model — the joint viability landscape over the two interacting homeostatic-gain axes.") diff --git a/experiments/tracking_param_sweep.jl b/experiments/tracking_param_sweep.jl index 4492efd..b375e98 100644 --- a/experiments/tracking_param_sweep.jl +++ b/experiments/tracking_param_sweep.jl @@ -449,5 +449,5 @@ function run_tracking_param_sweep(; seeds=0:99, ticks=7200, warmup=100, nnodes=2 return dir end -register_experiment!(:tracking_param_sweep, run_tracking_param_sweep; +ExpRegistry.register_experiment!(:tracking_param_sweep, run_tracking_param_sweep; description="One-at-a-time parameter sweep of the paper Falandays object-tracking model (leak, lrate_targ, lrate_wmat, input_amp, movement_amp, eye_offset); post-warmup heading error + branching over N random-init seeds.") diff --git a/plans/README.md b/plans/README.md new file mode 100644 index 0000000..02a6c67 --- /dev/null +++ b/plans/README.md @@ -0,0 +1,27 @@ +# BrainlessLab plans + +This directory contains strict `brainlesslab-plan` version-one TOML files for the canonical +operation path: + +```text +CompositionSpec + EvaluationSpec → EvaluationTarget → operation → record +``` + +Validate without running: + +```bash +julia --project=. bin/brainlesslab.jl check plans/examples/profile_tracking.toml +``` + +Execute and write a portable record: + +```bash +julia -t auto --project=. bin/brainlesslab.jl run \ + plans/examples/profile_tracking.toml --root records +``` + +The examples are deliberately small exploratory smoke plans. They are executable examples, +not benchmark evidence. The reciprocal evolution files show the intended direction of the +first flagship design: select on Tracking and evaluate on held-out Pong, then reverse the +direction. Larger budgets and frozen evaluation blocks should be declared in new versioned +plans rather than silently changing these examples. diff --git a/plans/examples/ablate_tracking.toml b/plans/examples/ablate_tracking.toml new file mode 100644 index 0000000..c1f7230 --- /dev/null +++ b/plans/examples/ablate_tracking.toml @@ -0,0 +1,25 @@ +format = "brainlesslab-plan" +format_version = 1 +operation = "ablate" +id = "ablate_tracking_example" + +[[targets]] +id = "tracking" + +[targets.composition] +id = "falandays_tracking_ablation" +preset = "falandays_tracking" + +[targets.evaluation] +blocks = 1 +trials_per_block = 2 +horizon = 120 +warmup = 20 +construction_scope = "trial" +reset = "full" +root_seed = 1301 +aggregate = "mean" + +[ablate] +target = "tracking" +ablations = ["freeze_plasticity", "clamp_target"] diff --git a/plans/examples/benchmark_core.toml b/plans/examples/benchmark_core.toml new file mode 100644 index 0000000..5de9eef --- /dev/null +++ b/plans/examples/benchmark_core.toml @@ -0,0 +1,80 @@ +format = "brainlesslab-plan" +format_version = 1 +operation = "benchmark" +id = "core_benchmark_example" + +[[targets]] +id = "tracking_falandays" +[targets.composition] +id = "tracking_falandays" +preset = "falandays_tracking" +[targets.evaluation] +blocks = 2 +trials_per_block = 1 +horizon = 120 +warmup = 20 +construction_scope = "trial" +reset = "full" +root_seed = 1501 +aggregate = "mean" + +[[targets]] +id = "tracking_random" +[targets.composition] +id = "tracking_random" +node = "null_random" +task = "tracking" +n_nodes = 200 +[targets.evaluation] +blocks = 2 +trials_per_block = 1 +horizon = 120 +warmup = 20 +construction_scope = "trial" +reset = "full" +root_seed = 1501 +aggregate = "mean" + +[[targets]] +id = "pong_falandays" +[targets.composition] +id = "pong_falandays" +preset = "falandays_pong" +[targets.evaluation] +blocks = 2 +trials_per_block = 1 +horizon = 400 +warmup = 40 +construction_scope = "trial" +reset = "full" +root_seed = 2501 +aggregate = "mean" + +[[targets]] +id = "pong_random" +[targets.composition] +id = "pong_random" +node = "null_random" +task = "pong" +n_nodes = 500 +[targets.evaluation] +blocks = 2 +trials_per_block = 1 +horizon = 400 +warmup = 40 +construction_scope = "trial" +reset = "full" +root_seed = 2501 +aggregate = "mean" + +[benchmark] + +[[benchmark.cases]] +id = "tracking" +conditions = ["tracking_falandays", "tracking_random"] +baseline = "tracking_random" + +[[benchmark.cases]] +id = "pong" +conditions = ["pong_falandays", "pong_random"] +baseline = "pong_random" diff --git a/plans/examples/evolve_pong_test_tracking.toml b/plans/examples/evolve_pong_test_tracking.toml new file mode 100644 index 0000000..bde09b9 --- /dev/null +++ b/plans/examples/evolve_pong_test_tracking.toml @@ -0,0 +1,65 @@ +format = "brainlesslab-plan" +format_version = 1 +operation = "evolve" +id = "evolve_pong_test_tracking_example" + +[[targets]] +id = "pong_development" + +[targets.composition] +id = "falandays_pong_evolution" +preset = "falandays_pong" + +[targets.evaluation] +blocks = 1 +trials_per_block = 2 +horizon = 400 +warmup = 40 +construction_scope = "trial" +reset = "full" +root_seed = 1601 +aggregate = "mean" + +[[targets]] +id = "pong_confirmation" + +[targets.composition] +id = "falandays_pong_confirmation" +preset = "falandays_pong" + +[targets.evaluation] +blocks = 1 +trials_per_block = 2 +horizon = 400 +warmup = 40 +construction_scope = "trial" +reset = "full" +root_seed = 3601 +aggregate = "mean" + +[[targets]] +id = "tracking_heldout" + +[targets.composition] +id = "falandays_tracking_heldout" +preset = "falandays_tracking" + +[targets.evaluation] +blocks = 1 +trials_per_block = 2 +horizon = 240 +warmup = 40 +construction_scope = "trial" +reset = "full" +root_seed = 2601 +aggregate = "mean" + +[evolve] +training = "pong_development" +heldout = ["pong_confirmation", "tracking_heldout"] +optimizer = "sepcma" +parameter_set = "evolve" +objective = "normalized_score" +generations = 2 +popsize = 6 +sigma0 = 0.5 diff --git a/plans/examples/evolve_tracking_test_pong.toml b/plans/examples/evolve_tracking_test_pong.toml new file mode 100644 index 0000000..71efbbe --- /dev/null +++ b/plans/examples/evolve_tracking_test_pong.toml @@ -0,0 +1,65 @@ +format = "brainlesslab-plan" +format_version = 1 +operation = "evolve" +id = "evolve_tracking_test_pong_example" + +[[targets]] +id = "tracking_development" + +[targets.composition] +id = "falandays_tracking_evolution" +preset = "falandays_tracking" + +[targets.evaluation] +blocks = 1 +trials_per_block = 2 +horizon = 240 +warmup = 40 +construction_scope = "trial" +reset = "full" +root_seed = 1401 +aggregate = "mean" + +[[targets]] +id = "tracking_confirmation" + +[targets.composition] +id = "falandays_tracking_confirmation" +preset = "falandays_tracking" + +[targets.evaluation] +blocks = 1 +trials_per_block = 2 +horizon = 240 +warmup = 40 +construction_scope = "trial" +reset = "full" +root_seed = 3401 +aggregate = "mean" + +[[targets]] +id = "pong_heldout" + +[targets.composition] +id = "falandays_pong_heldout" +preset = "falandays_pong" + +[targets.evaluation] +blocks = 1 +trials_per_block = 2 +horizon = 400 +warmup = 40 +construction_scope = "trial" +reset = "full" +root_seed = 2401 +aggregate = "mean" + +[evolve] +training = "tracking_development" +heldout = ["tracking_confirmation", "pong_heldout"] +optimizer = "sepcma" +parameter_set = "evolve" +objective = "normalized_score" +generations = 2 +popsize = 6 +sigma0 = 0.5 diff --git a/plans/examples/profile_tracking.toml b/plans/examples/profile_tracking.toml new file mode 100644 index 0000000..a05ca76 --- /dev/null +++ b/plans/examples/profile_tracking.toml @@ -0,0 +1,26 @@ +format = "brainlesslab-plan" +format_version = 1 +operation = "profile" +id = "profile_tracking_example" + +[[targets]] +id = "tracking" + +[targets.composition] +id = "falandays_tracking_profile" +preset = "falandays_tracking" + +[targets.evaluation] +blocks = 1 +trials_per_block = 2 +horizon = 120 +warmup = 20 +construction_scope = "trial" +reset = "full" +root_seed = 1101 +aggregate = "mean" + +[profile] +target = "tracking" +analyses = ["branching_ratio_mr", "node_target_error"] +record_every = 1 diff --git a/plans/examples/sweep_tracking.toml b/plans/examples/sweep_tracking.toml new file mode 100644 index 0000000..8b737f3 --- /dev/null +++ b/plans/examples/sweep_tracking.toml @@ -0,0 +1,34 @@ +format = "brainlesslab-plan" +format_version = 1 +operation = "sweep" +id = "sweep_tracking_example" + +[[targets]] +id = "tracking" + +[targets.composition] +id = "falandays_tracking_sweep" +preset = "falandays_tracking" + +[targets.evaluation] +blocks = 1 +trials_per_block = 2 +horizon = 120 +warmup = 20 +construction_scope = "trial" +reset = "full" +root_seed = 1201 +aggregate = "mean" + +[sweep] +target = "tracking" +mode = "factorial" +max_rollouts = 8 + +[[sweep.axes]] +parameter = "leak" +values = [0.25, 0.5] + +[[sweep.axes]] +parameter = "lrate_wmat" +values = [0.35, 1.0] diff --git a/profile/Manifest.toml b/profile/Manifest.toml index 50eafe6..23333d1 100644 --- a/profile/Manifest.toml +++ b/profile/Manifest.toml @@ -2,7 +2,7 @@ julia_version = "1.12.6" manifest_format = "2.0" -project_hash = "c1e3b888cad7645bcf6748dd96b92f1d32d24207" +project_hash = "31d72d653078fa731c056153b18b53b785561d8f" [[deps.AbstractFFTs]] deps = ["LinearAlgebra"] @@ -111,10 +111,10 @@ uuid = "18cc8868-cbac-4acf-b575-c8ff214dc66f" version = "1.4.0" [[deps.BrainlessLab]] -deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "StaticArrays", "Statistics", "TOML"] +deps = ["Dates", "JLD2", "LinearAlgebra", "Random", "SHA", "StaticArrays", "Statistics", "TOML"] path = ".." uuid = "d12add44-1e3e-4161-9a99-c2121a2f0f38" -version = "0.1.1" +version = "0.2.0" weakdeps = ["Makie"] [deps.BrainlessLab.extensions] diff --git a/profile/Project.toml b/profile/Project.toml index 26fffe5..1f32f52 100644 --- a/profile/Project.toml +++ b/profile/Project.toml @@ -10,7 +10,7 @@ TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76" [compat] Base64 = "1.10" -BrainlessLab = "0.1.1" +BrainlessLab = "0.2" CairoMakie = "0.12, 0.13, 0.14, 0.15" Dates = "1.10" Printf = "1.10" diff --git a/site/src/content/docs/core/architecture.mdx b/site/src/content/docs/core/architecture.mdx index f5b274c..1408366 100644 --- a/site/src/content/docs/core/architecture.mdx +++ b/site/src/content/docs/core/architecture.mdx @@ -3,16 +3,19 @@ title: Architecture description: The core BrainlessLab abstractions, ownership boundaries, and synchronous closed-loop lifecycle. --- -BrainlessLab uses one composition from a single agent to a mixed population: +BrainlessLab separates the runtime that produces behavior from the research protocol that +compares behavior: ```text -NodeModel → Reservoir → AbstractBody → Agent → Ensemble{Environment} - ↓ - Task - ↓ - Runner → Run - ↘ - Recorder +runtime +NodeSpec + TaskSpec + body + InteractionCycle + → CompositionSpec → Reservoir + embodied agent(s) + Environment → SimResult + +research +CompositionSpec + EvaluationSpec + → EvaluationTarget → operation plan → typed result → research record +named EvaluationTargets + operation plans + → ExperimentSpec ``` A reservoir contains nodes. An ensemble contains embodied agents. The same `step!` lifecycle @@ -28,8 +31,18 @@ runs an ensemble of one and an ensemble of many. - **`Environment`** owns external state and relations. - **`Ensemble`** advances one or more agents synchronously. - **`TaskSpec`** supplies setup, rollout defaults, and optional score metadata. +- **`NodeSpec`** registers one node builder, parameters, parameter sets, capabilities, + equations, and default analyses. +- **`CompositionSpec`** selects one node, task, optional body, node count, parameters, and + interaction cycle. +- **`EvaluationSpec`** owns blocks, trials, horizon, warm-up, construction scope, reset, + named random streams, and aggregation. +- **`EvaluationTarget`** names one composition under one evaluation protocol. - **`Recorder`** samples declared channels off the simulation hot path. - **`SimResult`** carries a recorder, metrics, task, node, and resolved configuration. +- **operation plans** define profile, sweep, ablation, evolution, or benchmark work. +- **`ExperimentSpec`** is the versioned scientific envelope: question, named conditions, + operations, evidence state, limitations, and metadata. ## Ownership is explicit @@ -118,15 +131,28 @@ Julia methods are the public extension mechanism. Registries add discoverable na configuration: ```julia -variants() -tasks() +nodes(DEFAULT_REGISTRY) +tasks(DEFAULT_REGISTRY) components() -analyses() -ablations() +analyses(DEFAULT_REGISTRY) +ablations(DEFAULT_REGISTRY) +compositions(DEFAULT_REGISTRY) ``` -Use a registry when selection by name is useful. Use direct typed composition when a name -does not add value. A registry does not replace the Julia contract. +`DEFAULT_REGISTRY` is the canonical authority for plans and typed composition. Use a +registry when selection by name is useful. Use direct typed composition when a name does +not add value. A registry does not replace the Julia contract. The zero-argument discovery +and `register_*!` helpers support the older symbol-based simulation façade only. + +Component registries and the experiment registry are deliberately separate. A +`RegistrySet` resolves runtime components and composition presets. An `ExperimentRegistry` +contains versioned `ExperimentSpec`s. Registering an experiment validates every operation +against the component registry and rejects conditions that change under the same name. + +`ExperimentSpec` is not another runner. It groups the exact operation plans that implement +a scientific question. Each operation still writes its own record, so selection, +evaluation, and benchmarking remain visible rather than being hidden inside an experiment +callback. ## No hidden controller @@ -142,4 +168,4 @@ Falandays node has a narrower validated implementation boundary. Read [The Falandays node](/core/falandays/) and [Platform limits](/platform-limits/) before making a fidelity claim. -

Source: src/core/Interfaces.jl, src/world/Embodiment.jl, src/world/Ensemble.jl, src/tasks/Tasks.jl, src/core/Recorder.jl.

+

Source: src/core/Interfaces.jl, src/core/Composition.jl, src/operations/, src/world/Ensemble.jl, src/core/Recorder.jl.

diff --git a/site/src/content/docs/core/design-study.mdx b/site/src/content/docs/core/design-study.mdx index d0a0993..90aa606 100644 --- a/site/src/content/docs/core/design-study.mdx +++ b/site/src/content/docs/core/design-study.mdx @@ -24,49 +24,26 @@ Exploration may change later choices. It cannot serve as untouched confirmation. ## A first exploratory contrast -This example asks whether the canonical Falandays reservoir exceeds a random-action control -on tracking. Each seed is one paired block. Both conditions receive the same task seed. - -```julia -using BrainlessLab -using Statistics - -function paired_tracking(seeds) - rows = NamedTuple[] - - for seed in seeds - scores = map((:falandays, :null_random)) do node - sim = simulate( - :tracking; - node=node, - ticks=1000, - seed=seed, - ) - task_outcome(sim).normalized - end - - push!(rows, ( - seed=seed, - falandays=scores[1], - random_action=scores[2], - difference=scores[1] - scores[2], - )) - end - - return rows -end - -rows = paired_tracking(0:7) -mean_difference = mean(getproperty.(rows, :difference)) +The checked-in `plans/examples/benchmark_core.toml` asks whether the canonical Falandays +composition differs from a random-effector control on Tracking and Pong. Conditions within +each task share block and trial seeds. The benchmark reports the tasks separately. +Raw and task-normalized outcomes receive parallel 95% Student-t intervals; within-task +contrasts use paired differences. These exploratory intervals remain descriptive until the +protocol and sample size are frozen. + +```bash +julia --project=. bin/brainlesslab.jl check plans/examples/benchmark_core.toml +julia -t auto --project=. bin/brainlesslab.jl run \ + plans/examples/benchmark_core.toml --root records ``` -This is a development example. Eight blocks are not a prospective power calculation. The -random-action control asks whether behavior exceeds random effector output. It does not -show that tracking information is necessary. A blind or matched-sham input is needed for -that question. +This is a small development example, not a prospective power calculation. The random-action +control asks whether behavior exceeds random effector output. It does not show that sensory +information or a particular neural mechanism is necessary. A blind input or registered +mechanism ablation is needed for those questions. -For a retained study, save per-block rows, resolved settings, provenance, and failures. Use -a batch tool or an experiment protocol instead of relying on the Julia session. +The resulting record retains the per-block rows, resolved settings, named seeds, +provenance, and checksums. ## Name the independent unit @@ -146,6 +123,46 @@ study is underpowered. Do not call a capped development run confirmatory. For equivalence, define an equivalence margin and use an equivalence procedure. Nonsignificance does not establish equivalence. +## Register the scientific protocol + +An operation answers one computational question. `ExperimentSpec` records why a set of +operations exists and fixes their relationship: + +```julia +experiment = ExperimentSpec( + :falandays_cross_task, + v"1.0.0"; + title="Reciprocal parameter evolution across the core benchmark", + question="How does parameter evolution on one core task move performance on the other?", + conditions=(tracking_development, pong_evaluation, + pong_development, tracking_evaluation), + operations=(evolve_tracking, evolve_pong), + evidence_state=:planned, + limitations=("Parameter evolution only; node structure remains fixed.",), +) + +register_experiment!(experiment) +write_experiment("protocols/falandays_cross_task", experiment) +``` + +Every operation target must exactly match a named experiment condition. Registration +rejects duplicate `(id, version)` keys, missing conditions, and changed condition details. +The experiment registry is separate from the component registry: publishing a protocol +does not turn a node or task into a benchmark standard. + +The intended first flagship is reciprocal. Evolve the Falandays parameter set on Tracking, +then evaluate the selected champion on fresh Tracking seeds and held-out Pong; reverse the +design for Pong. The same-task confirmation estimates improvement without reusing selection +seeds, while the other task estimates transfer. Only after both selections are frozen should +a benchmark compare their per-task performance. That design shows movement across a task +set without pretending that task-normalized values form one universal competence score. + +Future node designs can use the same protocol. A node may expose a different parameter set, +registered structural elements, or local mechanisms such as dendritic compartments. The +benchmark then indicates which capacities improve, remain absent, or trade off. Performance +is informative even when a design fails a task, provided the task opportunity and control +are sound. + ## Evidence states Use these labels: diff --git a/site/src/content/docs/core/extend.mdx b/site/src/content/docs/core/extend.mdx index 5b81537..1dceda2 100644 --- a/site/src/content/docs/core/extend.mdx +++ b/site/src/content/docs/core/extend.mdx @@ -29,7 +29,8 @@ composed value or parameter preset when the contract has not changed. Import the function names that receive new methods: ```julia -using BrainlessLab: Reservoir, register_node!, simulate +using BrainlessLab: Reservoir, NodeSpec, ParameterSpec, CompositionSpec +using BrainlessLab: NodeBuildContext, DEFAULT_REGISTRY, register!, simulate import BrainlessLab: step!, effectors, reset! import BrainlessLab: n_nodes, n_receptors, n_effectors ``` @@ -39,19 +40,39 @@ The framework will not dispatch to it. ## Add a node -Use `examples/templates/new_project/my_node.jl` as the scaffold. Keep the constructor -task-agnostic: +Use `examples/templates/new_project/my_node.jl` as the scaffold. A registered builder +receives the composition's node count, body ports, named seeds, and resolved values: ```julia -MyReservoir(n_nodes, n_receptors, n_effectors; seed=0, kwargs...) +function build_my_reservoir(context::NodeBuildContext, values) + MyReservoir( + context.n_nodes, + n_receptors(context.ports), + n_effectors(context.ports); + seed=Int(mod(context.seeds.topology, UInt64(typemax(Int)))), + params=values, + ) +end ``` -Register only after the contract works: +Declare the parameter surface rather than inferring it from fields. Node count belongs to +the composition; connectivity can declare `owner=:reservoir`: ```julia -register_node!(:my_reservoir, MyReservoir; genome_type=MyParams) -simulate(:tracking; node=:my_reservoir, seed=11) -simulate(:pong; node=:my_reservoir, seed=11) +spec = NodeSpec( + :my_reservoir, + build_my_reservoir; + genome_type=MyParams, + parameters=( + ParameterSpec(:leak, 0.25; sweep=(0.1, 0.25, 0.5), evolve=(lower=0.0, upper=0.95)), + ParameterSpec(:link_p, 0.1; owner=:reservoir, evolve=(lower=0.01, upper=0.8)), + ), + parameter_sets=Dict(:sweep => (:leak,), :evolve => (:leak,)), +) +register!(DEFAULT_REGISTRY, spec) + +simulate(CompositionSpec(:my_tracking, :my_reservoir, :tracking; n_nodes=200); seed=11) +simulate(CompositionSpec(:my_pong, :my_reservoir, :pong; n_nodes=500); seed=11) ``` Declare online plasticity only when `step!` performs online adaptation. Keep optimizer @@ -110,7 +131,8 @@ contract. ## Add a task or world For a vector-valued task, implement the `TaskWorld` methods and wrap the world in a -`TaskSpec`. Start from `examples/templates/new_project/my_task.jl`. +`TaskSpec`, then register it with `register!(DEFAULT_REGISTRY, task_spec)`. Start from +`examples/templates/new_project/my_task.jl`. For a physical composition, build bodies and an `ObjectWorld` in the setup callable. Start from `examples/embodiments/object_world_task.jl`. diff --git a/site/src/content/docs/core/getting-started.mdx b/site/src/content/docs/core/getting-started.mdx index cb928bf..eb12c61 100644 --- a/site/src/content/docs/core/getting-started.mdx +++ b/site/src/content/docs/core/getting-started.mdx @@ -107,18 +107,43 @@ compare a tracking score directly with a Pong hit rate. Query the registries instead of copying static lists: ```julia -variants() -tasks() -analyses() -ablations() +nodes(DEFAULT_REGISTRY) +tasks(DEFAULT_REGISTRY) +analyses(DEFAULT_REGISTRY) +ablations(DEFAULT_REGISTRY) +compositions(DEFAULT_REGISTRY) components() readiness() ``` -The registries include project and extension entries loaded in the current environment. +The typed registry includes project and extension entries loaded in the current environment. The [Experimental catalog](/experimental/) lists studies and components that are outside the canonical core path. +## Create a portable profile record + +One simulation is useful for inspection. The next step is a typed operation whose complete +method and output can be rerun: + +```bash +julia --project=. bin/brainlesslab.jl check plans/examples/profile_tracking.toml +julia -t auto --project=. bin/brainlesslab.jl run \ + plans/examples/profile_tracking.toml --root records +``` + +Open the generated `report/index.html` for the readable report. Use these files when you +need the exact evidence surface: + +- `request.toml`: what was requested; +- `resolved.toml`: all registry defaults and operation settings actually used; +- `seeds.csv`: the named random-stream ledger; +- `data/trials.csv`: one row per raw trial; +- `summary/statistics.csv`: derived descriptive statistics; +- `record.toml`: versions, repository state, artifact inventory, and checksums. + +The same command runs a profile, sweep, ablation, evolution, or benchmark because the +operation belongs to the plan, not to a separate runner. + ## Optional figures The compute package does not depend on Makie. Use a separate project that provides a @@ -143,5 +168,6 @@ A successful run shows that the selected task, body, reservoir, and recorder com execute. It may reveal behavior worth studying. It does not establish a general capacity, biological fidelity, superiority, or statistical reliability. -Next, read the [core task tour](/core/task-tour/). Then use -[Design a study](/core/design-study/) to turn an observation into a controlled comparison. +Next, read [Operations and records](/core/tools-artifacts/) and the [core task tour](/core/task-tour/). +Then use [Design a study](/core/design-study/) to place an operation inside a controlled, +versioned experiment. diff --git a/site/src/content/docs/core/interaction-cycle.mdx b/site/src/content/docs/core/interaction-cycle.mdx index b2b5d4a..443c550 100644 --- a/site/src/content/docs/core/interaction-cycle.mdx +++ b/site/src/content/docs/core/interaction-cycle.mdx @@ -67,9 +67,10 @@ cycle requirements before a rollout begins. ## Evaluation is a different layer -`EvaluationProtocol` governs complete trials: count, horizon, warmup, reset rule, whether a -constructed design is fixed, and aggregation. It does not alter the interaction cycle. +`EvaluationSpec` governs complete trials: blocks, trials per block, horizon, warm-up, reset +rule, construction scope, named random streams, and aggregation. It does not alter the +interaction cycle. This distinction lets the same embodied agent run one diagnostic simulation, a held-out benchmark, a sweep, or evolution without changing its sensorimotor semantics. -

Source: src/world/Interaction.jl, src/world/Ensemble.jl, src/world/Embodiment.jl, src/run/Evaluation.jl.

+

Source: src/world/Interaction.jl, src/world/Ensemble.jl, src/world/Embodiment.jl, src/operations/Evaluation.jl.

diff --git a/site/src/content/docs/core/reservoirs.mdx b/site/src/content/docs/core/reservoirs.mdx index 0b937f2..2a483e8 100644 --- a/site/src/content/docs/core/reservoirs.mdx +++ b/site/src/content/docs/core/reservoirs.mdx @@ -47,7 +47,7 @@ homeostatic dynamics. Query the current roster: ```julia -variants() +nodes(DEFAULT_REGISTRY) ``` ## Keep model axes explicit @@ -108,7 +108,8 @@ Include wrapper-owned RNG and lag state in replay snapshots. When you define methods on BrainlessLab generics, import the function names: ```julia -using BrainlessLab: Reservoir, register_node! +using BrainlessLab: Reservoir, NodeBuildContext, NodeSpec, ParameterSpec +using BrainlessLab: DEFAULT_REGISTRY, register! import BrainlessLab: step!, effectors, reset! import BrainlessLab: n_nodes, n_receptors, n_effectors ``` @@ -116,7 +117,9 @@ import BrainlessLab: n_nodes, n_receptors, n_effectors `using BrainlessLab` is sufficient to call a generic. It is not sufficient to add a method to that generic. -Start from `examples/templates/new_project/my_node.jl`. Before a sweep or benchmark: +Start from `examples/templates/new_project/my_node.jl`. Register a `NodeSpec` whose builder +accepts `NodeBuildContext` plus resolved parameters. Declare `ParameterSpec`s and named +`:sweep`, `:evolve`, and optional connectivity sets. Before a sweep or benchmark: 1. test constructor widths; 2. test deterministic construction from a seed; diff --git a/site/src/content/docs/core/runs-results.mdx b/site/src/content/docs/core/runs-results.mdx index 62aed9c..bea8a63 100644 --- a/site/src/content/docs/core/runs-results.mdx +++ b/site/src/content/docs/core/runs-results.mdx @@ -110,25 +110,37 @@ Use `GLMakie` for `explore(...)`. Use `CairoMakie` for headless files. A represe animation is explanatory material. It is not a substitute for the full result distribution. Declare how a representative run was selected. -## In-memory result versus run artifact +## In-memory result versus research record A plain `simulate` call returns an in-memory result. It does not by itself create a versioned evidence bundle. -Batch tools write run directories. A useful development artifact normally contains: +All five typed operations write the same version-one research record: ```text -run-directory/ -├── manifest.toml -├── config.resolved.toml -├── summary or results table -├── per-run or per-cell values -├── figures and figure-data sidecars -└── README.md +record-id/ +├── record.toml +├── request.toml +├── resolved.toml +├── seeds.csv +├── data/ +├── summary/ +├── figures/ +├── report/index.html +└── DONE ``` -Check the git SHA, dirty-worktree flag, Julia and package versions, seed ledger, resolved -configuration, failures, and checks before interpreting the summary. +`request.toml` preserves the submitted plan. `resolved.toml` contains the resolved node +parameters, task and body options, interaction timing, evaluation policy, and +operation-specific settings. `record.toml` records the package version, Julia version, +repository SHA and state, every generated artifact, and its SHA-256 checksum. Shareable +metadata omit hostnames and absolute local paths. + +`data/trials.csv` is the raw audit table. `data/task_metrics.csv` is the compact outcome +surface. Other data tables depend on the operation: analyses, sweep cells, ablation cases, +evolution history and champion parameters. `summary/statistics.csv` and +`summary/contrasts.csv` are derived. The generated HTML report explains the method and +node equations, but the CSV tables remain authoritative. A traceable run directory is not automatically confirmed evidence. Promotion also needs a frozen protocol, declared independent unit, untouched evaluation blocks, analysis version, @@ -152,4 +164,4 @@ Next human decision: Use [Design a study](/core/design-study/) to define the comparison. Use [Tools and artifacts](/core/tools-artifacts/) for persistent batch outputs. -

Source: src/api/Highlevel.jl, src/core/Recorder.jl, src/run/.

+

Source: src/api/Highlevel.jl, src/core/Recorder.jl, src/operations/, src/records/.

diff --git a/site/src/content/docs/core/task-tour.mdx b/site/src/content/docs/core/task-tour.mdx index f6f3620..41f8702 100644 --- a/site/src/content/docs/core/task-tour.mdx +++ b/site/src/content/docs/core/task-tour.mdx @@ -7,11 +7,11 @@ A task defines a closed-loop question. It selects a world, one body per agent, r defaults, metrics, and optional score anchors. A task name is not a universal ability label. -Use `tasks()` for the live registered list. Use `resolve_task` and `setup_task` to inspect -a task: +Use `tasks(DEFAULT_REGISTRY)` for the canonical registered list. Use `task_spec` and +`setup_task` to inspect a task: ```julia -task = resolve_task(:tracking) +task = task_spec(DEFAULT_REGISTRY, :tracking) setup = setup_task(task; seed=11) has_objective(task) resolved_task_ports(task) @@ -103,21 +103,29 @@ frame-voting readout. They are tagged `:experimental` and `:plank_cartpole`; the outside the Tracking/Pong core benchmark. The useful first result may simply be a clear performance boundary for an otherwise capable design. -For a quick integration check, use `simulate`. For the declared repeated-start contract, -use `evaluate_plank_cartpole`, which retains the complete initial-condition set and every raw -trial: +For a quick integration check, use `simulate`. For repeated starts, use the same +`EvaluationSpec` as every other task: ```julia -result = evaluate_plank_cartpole( +composition = CompositionSpec( + :plank_easy_falandays, + :falandays, :cartpole_plank_easy; - node=:falandays, - build_seed=11, - trial_seed=20_000, + n_nodes=500, +) +protocol = EvaluationSpec( + blocks=1, + trials_per_block=1000, + horizon=15_000, + construction_scope=:evaluation, + root_seed=20_000, ) +result = evaluate(EvaluationTarget(:plank_easy, composition, protocol)) ``` -The evaluation holds one constructed topology fixed and fully restores neural, plastic, and -body state between starts. Do not average the four levels into a generic competence score. +The evaluation reconstructs the same seeded design with fresh dynamic and plastic state for +every trial and retains each realized initial condition with its named seed ledger. Do not +average the four levels into a generic competence score. The Spike-FF-2 schedule is checked against the authors' public example. The present Argyle-4 implementation follows the paper's adjacent-bin, nine-spike description, but is marked as a BrainlessLab schedule until a source fixture establishes trajectory-level diff --git a/site/src/content/docs/core/tools-artifacts.mdx b/site/src/content/docs/core/tools-artifacts.mdx index 864ef7b..4c12e01 100644 --- a/site/src/content/docs/core/tools-artifacts.mdx +++ b/site/src/content/docs/core/tools-artifacts.mdx @@ -1,157 +1,174 @@ --- -title: Tools and artifacts -description: Choose the smallest BrainlessLab execution surface that matches the question and understand the files it writes. +title: Operations and records +description: Use one typed path for profiles, sweeps, ablations, evolution, benchmarks, and portable research records. --- -Start with the smallest tool that can answer the question. +BrainlessLab has one canonical batch path: -## Three primary tools - -| Intent | Start with | Typical status | Output | -| --- | --- | --- | --- | -| inspect one closed loop | `simulate` | conformance or exploration | in-memory `SimResult` | -| map a bounded parameter or intervention region | `sweep/run.jl` | development or tuning | resumable `sweeps//` | -| execute a declared multi-condition protocol | `experiments/run.jl` | protocol-defined | experiment run directory | +```text +CompositionSpec + EvaluationSpec + → EvaluationTarget + → operation plan + → validate → resolve → execute + → typed result + → version-one research record +``` -Use `simulate` before a sweep. Use a sweep before writing a broad benchmark. Use an -experiment protocol when conditions, blocks, endpoints, and artifacts must be coordinated. +`simulate` remains the smallest way to inspect one closed loop. Use an operation plan when +the question requires repeated trials, comparisons, selection, or a portable output. -## One run +## The five operations -```julia -using BrainlessLab - -sim = simulate( - :tracking; - node=:falandays, - ticks=1000, - seed=11, -) -``` +| Question | Plan | Main outputs | +| --- | --- | --- | +| what dynamics accompany performance? | `ProfilePlan` | task trials, analysis statistics, methods and equations | +| how does performance vary over parameters? | `SweepPlan` | paired trial rows and cell summaries | +| what happens when a functional element is disabled? | `AblationPlan` | implicit baseline, paired interventions, case summaries | +| can registered parameters improve one target, and what happens elsewhere? | `EvolutionPlan` | convergence, candidates, champion parameters, held-out trials | +| how do declared designs compare within each task? | `BenchmarkPlan` | per-task statistics and paired contrasts | -This result stays in memory unless your script saves it. +Each plan points to one or more `EvaluationTarget`s. A target combines a +`CompositionSpec` with an `EvaluationSpec`; there is no operation-specific sampling +protocol hidden elsewhere. -## A bounded development sweep +## Validate before spending compute -Preview available axes: +Checked-in examples live in `plans/examples/`: ```bash -julia --project=. sweep/run.jl --list-axes --node falandays --task tracking +julia --project=. bin/brainlesslab.jl check plans/examples/profile_tracking.toml +julia --project=. bin/brainlesslab.jl check plans/examples/benchmark_core.toml ``` -Run a checked-in tracking sweep: +`check` parses the strict version-one TOML, rejects unknown keys, resolves registry names +and defaults, and validates task, node, parameter, analysis, ablation, optimizer, and +pairing contracts. It does not run a simulation. + +Run the same file when it resolves cleanly: ```bash -julia --project=. sweep/run.jl configs/sweep_tracking.toml +julia -t auto --project=. bin/brainlesslab.jl run \ + plans/examples/profile_tracking.toml --root records ``` -The config declares a sweep ID, baseline, axes, seeds, cost cap, and analyses. The runner -prints the resolved cost before execution. A repeated run with the same ID resumes the same -directory and skips completed cells. +The operation is declared by the file. The command line does not maintain a second set of +operation-specific options. -A sweep result reports the best observed cell in that development grid. It does not report -a confirmed optimum. +A versioned `ExperimentSpec` can be published as one manifest plus its ordinary plan +files: -## A composed experiment +```julia +write_experiment("protocols/falandays_cross_task", experiment) +``` -List registered protocols before running one: +Another checkout can validate or execute that protocol without translating it into a new +schema: ```bash -julia --project=. experiments/run.jl --list +julia --project=. bin/brainlesslab.jl check-experiment protocols/falandays_cross_task +julia -t auto --project=. bin/brainlesslab.jl run-experiment \ + protocols/falandays_cross_task --root experiment-records ``` -Then run the named protocol through its declared entry point. Read the protocol page and -evidence status before opening any sealed output. +Each declared operation still writes its own standard record. The experiment run adds the +immutable protocol copy and an index of those operation records. -## Specialized tools +## The plan schema -| Tool | Use | -| --- | --- | -| `calibration/` | measure task floors, ceilings, and failure regimes | -| `profile/` | characterize one node family in depth | -| `sweep/run.jl ablate` | apply registered mechanism interventions | -| `bench/` | compare a declared model roster on a declared task grid | -| `evolve` and training tools | select fixed genomes on development tasks | -| `evaluate_plank_cartpole` | run the experimental repeated-start Plank CartPole protocol | +Every plan declares: -Examples: +- `format = "brainlesslab-plan"` and `format_version = 1`; +- one operation and stable plan ID; +- one or more named targets; +- for every target, a composition and full evaluation protocol; +- one operation-specific section. -```bash -julia --project=. calibration/run_calibration.jl -julia -t auto --project=. calibration/core_tasks.jl -julia --project=. sweep/run.jl ablate falandays tracking +A composition can reference a registered preset and override declared node parameters: -cd profile -julia --project=. run.jl falandays +```toml +[targets.composition] +id = "falandays_tracking_profile" +preset = "falandays_tracking" -cd bench -julia --project=. run.jl --neurons falandays,compartmental_structured --tasks tracking,pong +[targets.composition.parameters] +leak = 0.5 ``` -`bench/` and `profile/` have separate Julia projects. Prepare each once: - -```bash -cd bench -julia --project=. -e 'using Pkg; Pkg.develop(path=".."); Pkg.instantiate()' - -cd ../profile -julia --project=. -e 'using Pkg; Pkg.develop(path=".."); Pkg.instantiate()' +`n_nodes` belongs to the composition, not the node's parameter genome. Node-owned and +reservoir-owned parameters are declared by `ParameterSpec`. The node also declares named +default parameter sets such as `sweep` and `evolve`; plans may override those defaults. + +An evaluation always makes replication and randomization explicit: + +```toml +[targets.evaluation] +blocks = 4 +trials_per_block = 2 +horizon = 7200 +warmup = 100 +construction_scope = "trial" +reset = "full" +root_seed = 4101 +aggregate = "mean" ``` -Calibration, sweeps, ablations, and experiments use the repository root project. +The root seed derives independent named streams for topology, node state, world, body, +task, and mechanism noise. `construction_scope` determines whether topology and node-state +seeds are shared across the evaluation, within a block, or regenerated per trial. The +generic composition path requires `topology` and `world`; any additional declared stream is +derived and exposed to the node build context. A component may leave a declared stream +unused. `seeds.csv` records one row per trial, agent, and declared stream so multi-agent +provenance is not collapsed to the first agent. -`core_tasks.jl` reads `configs/core_task_calibration.toml` and writes per-seed rows, a -manifest, and a development report under a timestamped directory. It refuses to overwrite -an explicit nonempty `--output` unless `--force` is passed. Its reference-policy gates -establish task opportunity. Its Falandays, blind, and random rows do not by themselves -establish a neural mechanism or general advantage. +## One record format -## Output conventions - -Timestamped profile and benchmark runs do not collide. Sweeps use their configured ID so -they can resume in place. - -A sweep directory normally contains: +Every successful operation writes: ```text -sweeps// -├── manifest.toml -├── config.resolved.toml -├── results.csv -├── README.md +record-id/ +├── record.toml +├── request.toml +├── resolved.toml +├── seeds.csv +├── data/ +│ ├── trials.csv +│ ├── task_metrics.csv +│ └── operation-specific tables +├── summary/ +│ ├── statistics.csv +│ ├── contrasts.csv +│ └── summary.json ├── figures/ -└── cells/ - └── cell_NNN/ - ├── DONE - ├── manifest.toml - └── metrics.csv +├── report/index.html +└── DONE ``` -Optional capture can add a behavior animation and analysis time series. Numeric per-run data -remain authoritative. Check recorded errors, warnings, liveness gates, and valid surrogate -counts before reading a headline callout. - -## Cost, parallelism, and resume - -Independent rollouts can run on Julia threads. Start entry scripts with `julia -t auto` when -you want automatic threading. Results remain in seed order. More threads reduce elapsed -time; they do not increase the number of independent units. +`request.toml` is the submitted plan. `resolved.toml` records the complete parameter +defaults, timing, evaluation, and operation settings actually used. `seeds.csv` records +the named stream ledger. `record.toml` records package and Julia versions, repository +state, the complete artifact inventory, and SHA-256 checksums. -Inspect the resolved cells and total rollouts before starting a factorial sweep. Do not -expand a search cap without a research reason and an explicit decision. +The CSV tables are authoritative. The HTML report is generated from the same typed result +and provides a table of contents, method explanation, task results, registered node +equations, and an inline convergence chart when relevant. No plotting package is needed. -## Artifact status +If an operation fails while writing, the directory receives `FAILED` instead of `DONE`. +Failure markers do not copy local absolute paths into a shareable bundle. -Tool output can be: +## Interpretation boundaries -- a local diagnostic; -- a traceable exploratory artifact; -- a tuned selection artifact; -- a sealed confirmation bundle; -- promoted evidence. +- A profile describes recorded dynamics; it does not prove a mechanism. +- A sweep reports development cells; its best cell is not an untouched confirmation. +- An ablation tests the registered intervention, not every interpretation of the removed + function. +- Evolution selects on its training target. Held-out targets are evaluated only after the + champion is selected. +- A benchmark reports tasks separately. Normalization does not make different task scores + one common quantity. +- Benchmark tables report raw and normalized 95% Student-t intervals and paired within-task + contrasts. Small smoke plans demonstrate the machinery; they are not evidence-scale runs. -The directory shape does not determine the status. The protocol, seed stage, independent -unit, and outcome access history determine it. +Use [Design a study](/core/design-study/) to place operations inside a scientific protocol. +Use [Runs, recording, and results](/core/runs-results/) to inspect the resulting artifacts. -Use [Design a study](/core/design-study/) before selecting seeds or controls. Use -[Runs, recording, and results](/core/runs-results/) to inspect a result. +

Source: src/operations/, src/records/, bin/brainlesslab.jl.

diff --git a/skills/brainless-lab/SKILL.md b/skills/brainless-lab/SKILL.md index 85b07c6..2e3bf1b 100644 --- a/skills/brainless-lab/SKILL.md +++ b/skills/brainless-lab/SKILL.md @@ -1,232 +1,217 @@ --- name: brainless-lab -description: Guide for operating, extending, and interpreting BrainlessLab.jl — the agent-ready Julia lab for behavior from self-organising neural substrates. Covers low/no-code onboarding, simulate/visualize, calibration/profile/sweep/ablation/benchmark/evolution/experiment workflows, evidence states and safeguards, public composition and registries, and design guidance for nodes, bodies, tasks, metrics, and analyses. Use this skill whenever working in the brainless-lab repo or with BrainlessLab.jl, even when the request does not name it. Pair it with the julia skill for language-level correctness, dispatch, inference, allocations, and package hygiene. +description: Guide for operating, extending, and interpreting BrainlessLab.jl, the Julia research platform for behavior from self-organising neural substrates. Use for every task in the brainless-lab repository. Covers CompositionSpec, EvaluationSpec, typed registries, profile/sweep/ablation/evolution/benchmark plans, ExperimentSpec, version-one records, evidence boundaries, and node/task extension. Pair with the Julia skill for language, dispatch, inference, allocation, and package hygiene. --- -# BrainlessLab.jl — running, extending, and interpreting the lab +# BrainlessLab.jl -BrainlessLab is a summer-institute testbed (DISI 2026) for **"brainless" cognition**: behaviour that -emerges from collectives of simple neuron-like nodes with no homunculus and no hand-wired control. It -is a *framework for other people to run experiments* around a settled baseline — not a vehicle for one -person's model. That framing decides almost every design call: prefer a clean seam others can extend -over a clever one-off, and never quietly break the baseline. +BrainlessLab is a research platform for asking what simple, locally governed neural units +can do when coupled to bodies and worlds. It is not one model and it is not a leaderboard. +Its public value is a clean way to design a node, compose it with a sensorimotor task, +measure performance and dynamics, and preserve the exact protocol and evidence surface. -This skill is a way of thinking about the lab, not a command cheatsheet. Hold the few load-bearing ideas -below and the rest — which script, which kwarg, which measure — follows from them or from a `references/` -file. For anything about the *Julia itself* (why a `step!` allocates, is a node struct type-stable, how to -profile a sweep), use the **`julia` skill** alongside this one; this skill assumes that layer is handled. +Always read the repository `AGENTS.md`. Pair this skill with the Julia skill whenever code +is written or reviewed. -The user may not know Julia or how to code. Translate their scientific intent into the existing -high-level API, configs, examples, and tools. Do not make them choose source paths, types, or package -commands that the repository can determine. Explain the outcome, expected artifact, and interpretive -limit in plain language; keep implementation detail available but secondary. +## The architecture to preserve -## The one idea: neurons as nodes in a collective +There are two ladders: -Everything is *neurons as nodes within a collective* — the **same node contract at every scale**. There -is one ladder, and one `step!` runs all of it: +```text +runtime +NodeSpec + TaskSpec + body + InteractionCycle + → CompositionSpec → Reservoir + embodied agent(s) + Environment → SimResult +research +CompositionSpec + EvaluationSpec + → EvaluationTarget → operation plan → typed result → research record +named EvaluationTargets + operation plans + → ExperimentSpec ``` -NodeModel -> Reservoir -> AbstractBody -> Agent -> Ensemble{Environment} -> Task -> Runner -> Run - \-> Recorder -> (viz/analysis read this, off the hot path) -``` -A single-agent task is an `Ensemble` of **one** agent; a dyad is `n_agents=2`; a swarm is `n_agents=N`. -`step!(collective)` runs a solo reservoir and a 200-agent swarm through the *same code path*. When you -catch yourself thinking "the swarm case is different," stop — it almost never is; it's the same abstraction -with `n_agents` turned up. This is the single most important thing to internalise before extending anything. -The task must still declare that it supports a population: `n_agents` is a setup capability, not a magic -keyword that converts an unrelated single-agent task into a swarm environment. +Keep the boundaries explicit: + +- `NodeSpec` owns the node builder, declared parameters, parameter sets, capabilities, + equations, and default analyses. Node count belongs to `CompositionSpec`, not the node + parameter genome. Connectivity may be reservoir-owned and should say so in + `ParameterSpec.owner`. +- `TaskSpec` owns setup, ports, interaction timing, raw outcome, anchors, descriptors, and + experimental status. It does not own node parameters. +- `InteractionCycle` governs neural frames inside one world step. It does not govern trial + replication. +- `EvaluationSpec` is the only outer evaluation protocol: blocks, trials per block, + horizon, warm-up, construction scope, reset, root seed, named streams, and aggregation. +- `ExperimentSpec` is the scientific envelope above operations: version, question, named + conditions, evidence state, limitations, and metadata. It is not another runner. + +One `step!` lifecycle serves a single agent and a population. Do not create task- or +organism-name branches inside the simulation loop when a typed body, task, readout, +interaction cycle, or registered implementation expresses the distinction. + +## Start with the smallest surface + +For one diagnostic run: + +```julia +using BrainlessLab -Named, discoverable presets are wired through **registries**. Nodes, tasks, bodies, drives, metrics, -analyses, views, ablations, and optimizers can be registered by symbol and resolved at run time. Julia -generics and directly composed values are equally public: use a registry when discovery or configuration -by name is useful, not as a substitute for types and methods. +sim = simulate(:tracking; node=:falandays, ticks=1000, seed=11) +task_outcome(sim) +``` -## The first run, and the Makie seam +The symbol/keyword form remains a friendly façade. New reusable work should construct a +`CompositionSpec` so node count, parameters, body, task options, and interaction timing are +explicit. -The safest headline workflow does not alter the root environment: +For a repeated operation, use the one plan path: ```bash -julia --project=. -e 'using Pkg; Pkg.instantiate()' -julia --project=. examples/quickstart.jl +julia --project=. bin/brainlesslab.jl check plans/examples/profile_tracking.toml +julia -t auto --project=. bin/brainlesslab.jl run \ + plans/examples/profile_tracking.toml --root records +``` + +`check` must parse, validate, and resolve without simulation. `run` executes the operation +and writes the record. Do not introduce another YAML schema, operation-specific protocol, +or independent configuration path. + +## The five operations + +- `ProfilePlan`: characterize one node/task composition with declared analyses. The + executor unions required recorder channels and reports analysis failures explicitly. +- `SweepPlan`: evaluate explicit or node-default parameter axes. Seeds are paired across + cells. Call the result a development grid, never a confirmed optimum. +- `AblationPlan`: compare an implicit baseline with registered capability-checked + interventions. An ablation must declare its stage and required capabilities and must not + silently no-op. +- `EvolutionPlan`: select a registered node parameter set on one training target. Optimizer + randomness is separate from evaluation streams. Held-out targets run only after champion + selection. Candidate evaluations run in parallel when Julia has multiple threads, while + records retain every candidate's trial outcomes and seeds. +- `BenchmarkPlan`: compare declared conditions within each task under paired blocks. Report + raw and normalized Student-t intervals and paired contrasts, keep tasks separate, and do + not form a cross-task aggregate merely because scores are normalized. + +Checked-in smoke plans live in `plans/examples/`. The reciprocal evolution examples encode +the intended first flagship direction: evolve Falandays parameters on Tracking, then test +on fresh Tracking seeds and held-out Pong; reverse the direction for Pong. They are small +executable examples, not finished evidence. + +Use `write_experiment` to publish an `ExperimentSpec` as one strict manifest plus its +ordinary operation plan files. `read_experiment` validates that repeated condition names +have identical definitions. The unified CLI provides `check-experiment` and +`run-experiment`; every contained operation still produces its own standard record. + +## Records are the evidence surface + +Every operation writes `brainlesslab-record`, format version 1: + +```text +record-id/ +├── record.toml +├── request.toml +├── resolved.toml +├── seeds.csv +├── data/ +├── summary/ +├── figures/ +├── report/index.html +└── DONE ``` -The **compute core does not depend on Makie** — `simulate` runs headless. Plotting is a package extension -that activates *only when a Makie backend is loaded*: `CairoMakie` for static figures and GIFs (use this on -SSH/headless), `GLMakie` for interactive `explore(...)` windows. The generic visualization name exists in -the core, but no plotting method is available until a backend activates the extension. Don't add Makie to -the core deps to "fix" it — the weakdep split is deliberate. +The request and executed result must correspond exactly. `resolved.toml` must contain full +node defaults, task/body options, interaction timing, evaluation settings, and +operation-specific resolution. `record.toml` must derive Git provenance from the package +checkout, enumerate every generated artifact, and include SHA-256 checksums. Shareable +records must not include hostnames or absolute local paths. -## Stable baseline vs experimental platform — the discipline +CSV is the authoritative tabular format. The generated HTML is a readable view over the +same typed result: method, tables, node equations, and convergence where relevant. `DONE` +means the bundle completed; `FAILED` means it did not. A complete record is not +automatically confirmed evidence. -This distinction is load-bearing and easy to blur; keep it sharp in code, docs, and claims. +## Reference and experimental boundaries -- **`:falandays`** (`:falandays_base` is a compatibility alias) is the settled, validated, **authors-faithful** published Falandays - homeostatic spiking reservoir with its exact constants. It is the reference participants rely on. Validation - is numerical trajectory parity with the tested local authors-derived reference construction, within the - declared tolerance, not paper fidelity for every component — say "authors-faithful," not "paper-faithful." -- **Everything else is the experimental platform**: the other Falandays variants (`:falandays_extended`, - `:falandays_noisy`, `:falandays_ablated`, `:falandays_hemispheric`, `:falandays_oosawa`, `:falandays_dendritic`, - `:falandays_spatial`, `:falandays_delayed`), the SORN reference node, the compartmental/CTRNN nodes, the - evolution and embodiment layers, and the collective/ecological extensions. Useful testbed surfaces — but do **not** describe them as the - published paper model. +`:falandays` is the authors-faithful reference node on the declared trajectory fixtures. +That claim does not transfer automatically to a body, task, behavioral statistic, +analysis, or biological interpretation. Preserve its update equations, initialization, +and task presets unless a deliberate divergence is documented and tested. -When you touch the baseline, assume a fidelity fixture guards it (`test/fixtures/authors_.jld2`); run the -tests. When you add an experimental piece, label it experimental honestly. +Tracking and Pong are the initial core benchmark tasks. Wall remains registered but is not +core qualification. The four Plank CartPole levels are experimental challenge tasks. They +use the general `EvaluationSpec`; there is no CartPole-specific evaluation protocol. Do not +average their levels or the core tasks into a generic competence number. -The `:shoal_forage` task is a useful example of this division. It places the canonical -`:falandays` node inside an **Experimental** body, world relation, task, protocol, and set of -analyses. The node's parity status does not transfer to the fish-like embodiment or to claims -about social behavior. Its `SectorVision`, `AntagonisticTurnActuator`, `ProximityExposure`, -and `shoal_*` analyses must remain labelled Experimental until their own contracts and -scientific interpretations earn stronger evidence. +Performance can be informative without being a success story. A task can expose a limit, +a trade-off, or missing capacity. Before interpreting failure, verify the task opportunity, +body and port contract, null/controller floor, horizon, initialization, and score. -For this task, keep **fixed-demand performance** distinct from **operating-point -sensitivity**. `shoal_vision_sweep` compares social-vision conditions at one declared need -regime. `shoal_sensitivity_screen` varies one gain, curve, rate, range, or association rule at -a time. Raw satisfaction is mechanically changed by a depletion-rate intervention; use the -reported no-contact floor and `material_regulation_gain` when comparing demand levels, while -still noting that the intervention also changes feedback reaching the reservoir. Neither -screen estimates parameter interactions or licenses calling the best observed cell optimal. +## Extending nodes and tasks -Fixture parity validates the tested node update, not every task, body, ecological mechanism, biological -interpretation, or study. Read `site/src/content/docs/platform-limits.mdx` before broadening a claim. +Start from `examples/templates/new_project/`. -## Discovery-first: ask the registries, don't hardcode +A node extension defines methods on imported BrainlessLab generics, then registers a +`NodeSpec`. The builder receives `NodeBuildContext` and resolved values. Declare: -The registries are the live source of truth. Before assuming what exists, call them: +- parameters and validators; +- `owner` for node or reservoir concerns; +- default `:sweep`, `:evolve`, and optional connectivity parameter sets; +- capabilities used by ablations and tooling; +- equations and default analyses when known; +- stability and tags. -```julia -variants() # registered node symbols -tasks() # registered task symbols -analyses(); task_analyses(:forage) # registered measures (some labeled "experimental") -``` +Do not infer a node's evolvable surface from struct fields. Do not place runtime state in +the genome. Online adaptation remains runtime behavior even when there is no task loss, +teacher, fitted readout, or separate training phase. + +A task extension registers a `TaskSpec` with a setup returning a `TaskSetup`. Port widths +must be validated before tick zero. A task may omit a scalar outcome; it remains valid for +profiling or descriptive work but cannot enter a scalar benchmark until it declares an +outcome contract and anchors. + +Use `register!` on typed registries. Duplicate keys fail. Julia multiple dispatch is still +the extension mechanism; registries make implementations discoverable and configurable. + +## Scientific discipline + +Use the evidence ladder: planned → exploratory → tuned → frozen → confirmed → promoted. +Keep calibration, development, variance pilots, and held-out evaluation separate. The +independent randomized block or trial is normally the inferential unit; ticks and agents in +one world do not multiply sample size. + +Match the control to the claim. Random action, blind input, matched sham, mechanism +ablation, model baseline, and oracle answer different questions. Exact replay is a +regression control, not a causal null. + +Use `task_outcome(sim)` for the declared task result. Report raw score, normalized score if +used, viability gates, blocks/trials, construction scope, reset, horizon, warm-up, and seed +policy. A normalized Tracking value and normalized Pong value are still different +quantities. + +Criticality and information measures require nulls and estimator caveats. Prefer MR +branching estimates to naive slopes, use windowed analyses for non-stationary processes, +and treat apparent collective structure as shared drive until a suitable surrogate test is +cleared. + +## Verification + +For architecture or behavior changes, run the narrow tests first, then the root suite in +the pinned project. Preserve authors-parity fixtures. Build the locked site after handbook +or skill edits. Check `git diff --check` and inspect the final diff for unrelated user work. + +The canonical documentation is under `site/src/content/docs/core/`. Historical experiment +pages may describe older bespoke scripts; do not treat them as the public platform +contract. The checked-in skill and installed copy should be updated only after code and +docs agree. + +## References + +Read the relevant reference in full when needed: -Any symbol list you hardcode in docs or code will drift; a `variants()` call will not. This is also how you -sanity-check that your `register_*!` actually landed. - -## Designing something new — the posture - -Adding a part means **adding methods to the package generics** — you `import BrainlessLab: step!, effectors, -...` and define methods; `using` will *not* let you extend them. This is the most common first mistake. Start -from `examples/templates/new_project/`, get a single `simulate(:wall; node=:mynode)` to run, and only then -reach for `bench`/`sweep`. - -Read the matching reference before building: - -- **A new node / reservoir** → `references/designing-nodes.md`. The key design question is *where adaptation - lives*: in online-plastic weights (Falandays — fair to test untrained) or in fixed-weight dynamics - (compartmental/CTRNN — meaningless untrained, **must be evolved**). Get this wrong and every comparison is - unfair. Prefer a kwarg/preset bundle over a whole new `<: Reservoir` when the change is parametric. -- **A new environment / task / body** → `references/designing-environments-and-tasks.md`. The central object is - the synchronous contract (`sample!` → sensor → encoder → reservoir → actuator → dynamics/world → effects → - physiology). Prefer one composed `Embodiment` over organism-specific body subclasses. Effector semantics are - *intentionally non-uniform* across tasks, which is exactly why raw scores are **not comparable across tasks** — - design scoring against a meaningful floor/ceiling. -- **A new analysis / measure** → `references/designing-analyses.md`. Read this even just to *interpret* results. - -The Core [extension guide](https://brainless-lab.pages.dev/core/extend/) maps the remaining public families: drive, intervention, physical component, -physiology, optimizer/development, metric, and view. Prefer a parameter preset or composed value to a new -type when the contract has not changed. - -## Evidence states are part of the API - -Use the ladder in `references/research-workflow.md`: conformance → calibration → exploration → -tuning/training → variance pilot → frozen protocol → sealed confirmation → robustness → promoted -evidence. Never call the best observed sweep cell an optimum, use tuned seeds as confirmation, or treat -a committed run-dir as automatically promoted. - -The independent randomized block or run is normally the inferential unit. Agents and ticks nested in one -world do not multiply sample size. The null follows the claim: random action, blind/off, matched sham or -shift, mechanism ablation, model baseline, and oracle/reference answer different questions. Exact replay is -a regression control, not a causal null. - -Software readiness is orthogonal to study evidence. The -[Core handbook](https://brainless-lab.pages.dev/core/getting-started/) documents stable -composition contracts; the -[Experimental catalog](https://brainless-lab.pages.dev/experimental/) lists capabilities -with repository-backed source, example, and test metadata. `available` and `integrated` -describe software readiness, not construct validity or evidence promotion. - -Use `task_outcome(sim)` as the canonical task result handoff. It returns -`(key, raw, normalized)` for the objective declared by the task and `nothing` when the task -declares no scalar objective. Legacy metric fields remain diagnostics and may be useful, but -they do not define the cross-task outcome contract. - -## Rigor: null-test every measure - -The analysis layer is deliberately **measure-agnostic**: analyses are pure functions over the recorder's -channels, so you can point any candidate measure at a `SimResult`. That freedom is also the trap — a number -that looks "critical" at the collective scale is often an artifact of shared input. The library gives you the -check: a per-agent **circular-shift null** (`crossshift_null`) that preserves each agent's own temporal -statistics while destroying cross-agent alignment. Clear it before trusting any cross-agent measure, prefer -the subsampling-robust estimators the library ships (MR branching over the naive slope), and use the -`_windowed` variants when the process is non-stationary. Treat an un-null-tested cross-agent number as -shared-drive until shown otherwise — this project's own swarm runs are a standing reminder that measures which -*look* collective often don't survive the null. See `references/designing-analyses.md`. -Entity-aligned channels carry stable `EntityID`s in `EntityFrame`; nulls, analyses, and views must align by -those IDs rather than assuming vector position is identity. Unknown non-entity channels are not safe to pass -through a surrogate silently. - -## Reference files - -Read the relevant file in full when the task calls for depth — don't reconstruct API or schema details from -memory. - -- **`references/usage-and-workflows.md`** — the high-level API (`simulate` kwargs, `SimResult`, `visualize` / - `animate` / `explore` / `replay`, the recorder), discovery functions, and end-to-end recipes (baseline run, - swarm/dyad, headless output). Start here to *use* the lab. -- **`references/cli-tools.md`** — the batch/tooling surfaces: `bench/` (cross-node comparison, - `train.jl`, `compare.jl`), `profile/` (single-node deep stats), `sweep/` (parameter + ablation - sweeps), and `calibration/`. Their separate project environments, exact commands, run-dir outputs, - and the **sweep TOML config schema**. Composed protocols live in `experiments/`. -- **`references/designing-nodes.md`** — the node contract as a design contract; the three families and how each - must be tested (untrained vs evolved); composition-over-new-types; the `pack_params`/`snapshot_state` (genome - vs runtime state) split and `genome_type`; registration and type-stability pitfalls. -- **`references/designing-environments-and-tasks.md`** — `AbstractBody`/`Embodiment`, stable component ports, - direct task adapters versus `ObjectWorld` and the established situated adapter, non-uniform effectors, - `TaskSpec` scoring, one-to-many ensembles, multiple needs, and component/task registration. -- **`references/designing-analyses.md`** — the analysis contract, the criticality / collective / information - measure families and their caveats, and above all the **null-test discipline** (circular-shift null, MR - estimator, windowed vs pooled) plus a checklist for adding a validated measure. -- **`references/research-workflow.md`** — evidence states, experimental units, controls, tuning/confirmation - separation, prospective power, and promotion provenance. Read before designing or interpreting a study. -- **`references/agentic-safeguards.md`** — how to translate no/low-code requests, isolate work, protect sealed - evidence and user data, verify changes, and hand off without overstating what was established. - -## Naming and conventions - -Keep **"Reservoir"** for the node collective (the nodes are untrained by default) — not "Network"; this naming -was chosen deliberately, don't re-propose the rename. User-facing documentation lives in the Astro/Starlight -site under `site/` (published at ); canonical contracts live -under `/core/`, experimental capabilities under `/experimental/`, and the old `docs/*.md` set is retired. - -Use the current public body vocabulary exactly: `AbstractBody` is the dispatch boundary and `Embodiment` is the -generic concrete composition. Its stable-ID components are geometry, sensors, encoders, actuators, dynamics, -physiology, traits, and state. `ObjectWorld` is the generic fixed-population physical world; the older -`SituatedEnvironment` remains an adapter for the established torus, forage, and signalling behavior. -Do not introduce organism-specific body classes when component values express the difference. - -For discoverable physical parts, query `components()` / `component_info(...)`; readiness is software-scoped: -`:available` is discoverable/materializable, `:integrated` adds standard runtime + exact serialization + docs + -an executable example, and `:core` is stable/default with named core-test coverage. Scientific -evidence status remains a separate study property. -The minimal differential-robot kit has `:core` software readiness: disc geometry, explicit -no-physiology, spectral camera, identity encoder, differential-drive actuation, and -differential-drive dynamics. Other built-in physical components remain `:integrated`. -`ObjectWorld` is still an Experimental composition feature. Embodiment TOML materializes through -`read_embodiment_config` → `materialize_blueprint` or `materialize_embodiment`. `DevelopmentSpec` evolves bounded -real scalar paths on stable component IDs into a fresh runnable phenotype. Paths may use one-based tuple indices -or stable named collection members such as `variables.energy.gain`; it does not vary structure, schedule -births, or encode runtime state. - -For the bounded moving-shoal demonstrator, run -`julia -t 4 --project=. experiments/run.jl shoal_vision_sweep`. The default is an explicitly -underpowered two-block pilot that retains all sight/control conditions. Interpret material- -need satisfaction as the primary endpoint. Keep physical cohesion (nearest-neighbour distance -and largest proximity component), displacement coherence, and the perceptual graph as separate -descriptive outcomes; do not convert them into a generic intelligence or shoaling score. With -`record_every > 1`, contact counts and chord-based movement are recorder-grid diagnostics. -Always inspect wall occupancy: common boundary following can raise displacement coherence -without producing a cohesive shoal. +- `references/usage-and-workflows.md` for interactive simulation, recording, and plots; +- `references/cli-tools.md` for the unified plan CLI, schemas, operations, and records; +- `references/designing-nodes.md` for node/runtime-state design; +- `references/designing-environments-and-tasks.md` for task, body, ports, and worlds; +- `references/designing-analyses.md` for analysis and null contracts; +- `references/research-workflow.md` for evidence and interpretation; +- `references/agentic-safeguards.md` for safe agent operation. diff --git a/skills/brainless-lab/references/cli-tools.md b/skills/brainless-lab/references/cli-tools.md index d27edcb..18f5934 100644 --- a/skills/brainless-lab/references/cli-tools.md +++ b/skills/brainless-lab/references/cli-tools.md @@ -1,206 +1,160 @@ -# CLI Tools +# Unified operations and records -Four command-line entrypoints, each a distinct job writing a self-describing **run-dir** -(`manifest.toml` with git SHA + seeds + package versions, CSVs, `figures/*.png` in the -house palette, a `README.md` headline). Full prose at https://brainless-lab.pages.dev/tooling/; -the sweep TOML schema at https://brainless-lab.pages.dev/reference/. +The canonical batch interface is one command over one strict TOML schema: -| Tool | Job | Project env | Run-dir | -|---|---|---|---| -| `bench/` | roster of nodes across a task grid — rank + baseline stats | own | `bench/runs//` | -| `profile/` | one node in depth — full analytic suite + GIFs | own | `profile/runs///` | -| `sweep/` | perturb parameter axes, measure signatures per cell | root | `sweeps//` | -| `calibration/` | task score floor/ceiling anchors | root | stdout | - -`` is `__`. `bench`/`profile` timestamp every run so -repeats never collide; `sweep`/`ablate` key the run-dir on the sweep **id**, so re-running -the same id resumes in place (completed cells skipped) — that is why `sweeps/` is not -timestamped. +```bash +julia --project=. bin/brainlesslab.jl check PLAN.toml +julia -t auto --project=. bin/brainlesslab.jl run PLAN.toml --root records +``` -## Environments +The plan declares the operation. `check` parses, validates, and resolves without running. +`run` calls `run_operation` and prints the resulting record directory and compact summary. -`bench/` and `profile/` each carry their **own** `Project.toml` and must be instantiated -once (they `Pkg.develop` the repo they live in). `sweep/` and `calibration/` run against -the **root** project with `--project=.`. +## Plan envelope -```bash -cd bench && julia --project=. -e 'using Pkg; Pkg.develop(path=".."); Pkg.instantiate()' -cd profile && julia --project=. -e 'using Pkg; Pkg.develop(path=".."); Pkg.instantiate()' -# root, once, for sweep + calibration: -julia --project=. -e 'using Pkg; Pkg.instantiate()' +```toml +format = "brainlesslab-plan" +format_version = 1 +operation = "profile" # profile | sweep | ablate | evolve | benchmark +id = "stable_plan_id" + +[[targets]] +id = "tracking" + +[targets.composition] +id = "falandays_tracking_profile" +preset = "falandays_tracking" + +[targets.evaluation] +blocks = 2 +trials_per_block = 4 +horizon = 7200 +warmup = 100 +construction_scope = "trial" # evaluation | block | trial +reset = "full" +root_seed = 4101 +aggregate = "mean" ``` -Every entry script **self-relaunches with `-t auto`** when Julia started single-threaded -and no count was pinned — rollouts run in parallel across threads. Opt out with -`BRAINLESSLAB_AUTOTHREADS=0` or `JULIA_NUM_THREADS=1` (or `sweep.threaded = false` in the -sweep TOML). +Unknown keys and duplicate target IDs fail. A composition may use `preset`, or declare +`node`, `task`, `n_nodes`, optional `body`/`n_agents`, parameters, task/body options, and an +explicit fixed-rate interaction cycle. -## bench/ — cross-node comparison +The root seed derives named `topology`, `node_state`, `world`, `body`, `task`, and +`mechanism` streams. `construction_scope` controls only topology and node-state sharing. +The generic composition path requires `topology` and `world`; additional streams are +derived and exposed in `NodeBuildContext`. A component may leave a declared stream unused. +Records write one seed row per trial, agent, and stream. -Runs a roster of registered node variants across a task grid, ranks by normalized score, -reports baseline-relative nonparametric statistics. Use `profile/` instead when you want -one node in depth, not a ranking. +## Operation sections -```bash -cd bench && julia --project=. run.jl --neurons falandays_base,compartmental_structured --tasks wall,pong --no-gifs -``` +Profile: -Flags: `--config core.toml` (default `bench/configs/core.toml`; `--neurons` / `--tasks` / -`--no-gifs` override it), `--neurons a,b`, `--tasks x,y`, `--no-gifs`. An empty roster in -the config means all registered variants. The `[prep]` block encodes the fairness rule: -falandays\* default to `untrained` (seeded wiring + online plasticity), compartmental\* -default to `trained`; a cell needing a trained genome that has none falls back to -untrained and is flagged `trained-required-but-untrained`. +```toml +[profile] +target = "tracking" +analyses = ["branching_ratio_mr", "node_target_error"] +record_every = 1 +``` -Outputs under `bench/runs//`: `summary.csv` (per-neuron × task, used for ranking), -`results_raw.csv` (raw per-trial scores), `stats.json` (within-seed condition-label -permutation omnibus, paired sign-flip contrasts with Holm/BH correction, and paired-block -bootstrap CIs), -`figures/*.png`, `cells/__/` (scores + best/representative/worst GIFs), -`README.md`, `report.md`, `config.resolved.toml`, `manifest.toml`. +Sweep: -Two companion scripts: +```toml +[sweep] +target = "tracking" +mode = "factorial" # factorial | one_at_a_time +max_rollouts = 100 -```bash -julia --project=. train.jl compartmental_structured wall --generations 30 --popsize 16 --seed 1 --N 120 --ticks 300 -julia --project=. compare.jl runs/ runs/ --out comparisons/