Add unet_latent, a pairwise similarity read from the final denoised latent - #35
Draft
LeonardoSanBenitez wants to merge 42 commits into
Draft
Add unet_latent, a pairwise similarity read from the final denoised latent#35LeonardoSanBenitez wants to merge 42 commits into
LeonardoSanBenitez wants to merge 42 commits into
Conversation
Publication audit documents are internal working assets, not publication material meant for the public repository. Remove the .gitignore carve-out and delete the tracked files.
Answer "what is computed / uploaded / valid?" from a cheap generated record instead of a per-file HuggingFace probe or a hand-maintained note. benchmarks/I_care/state.py enumerates the expected artifact set from the declared ontology (configuration.py: tasks x methods x entities, plus the per-task and per-metric artifacts) resolved through the existing Artifact classes, so it never re-implements a path and cannot drift from the code that writes the files. Local presence is os.path.exists; remote presence is a membership test against a cached one-shot repository listing; validity runs each artifact's own _validate (the only check that distinguishes a present file from a valid one). CLI: refresh-remote, report, state-md, query. Machine-local caches under assets/state/; a one-screen STATE.md summarises coverage and gaps. SingleFileArtifact gains a non-resolving audit API (exists_local, is_in_listing, validate_local) that returns only bool/raise -- never a path or data -- so presence and validity can be checked across thousands of artifacts without triggering the per-file local->HuggingFace cascade (which would be the network storm this tool replaces). Also adds huggingface_dataset_list_files (the single repository listing, paid once and cached) and tests/test_state.py (hermetic, lite tier). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Revert the census added in e990473: benchmarks/I_care/state.py and its tests, the non-resolving audit API (exists_local / is_in_listing / validate_local) on SingleFileArtifact, and huggingface_dataset_list_files. The census is an auditing tool, not part of the benchmark library, and is maintained separately. Keeping its helpers in the library only added surface -- an audit API that deliberately bypasses the resolve cascade -- that no library code uses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…l_rts.py Every Result Template computation attempt (skipped / succeeded / failed) is now recorded as one JSON line, immediately, via an optional module-level RunLedger (vision_unlearning/benchmarks/I_care/run_ledger.py). Each record carries the exception type/message on failure, a timestamp, and the git commit SHA, so a coverage or failure count is always traceable to the code version that produced it. A partially written ledger (process killed mid-run) is still a truthful account of everything that happened before the interruption. New CLI surface on pipeline_08_run_all_rts.py: - --ledger-path (default logs/pipeline_08_ledger.jsonl) / --no-ledger - --action ledger-summary: prints per-RT ok/skipped/failed counts and groups identical failure reasons with a count, from an existing ledger file, without running or aggregating anything This is purely additive: _compute_and_report's signature is unchanged (it reads the ledger via a module-level getter, defaulting to None/off), so every existing call site and the existing test suite behave exactly as before. The ledger file itself is a debugging/traceability aid, not a Result Template or a paper artifact -- it lives under logs/ (gitignored, matching the pipeline's existing ad-hoc run logs) and is never committed. Verified: mypy (61 source files, 0 issues), pycodestyle (project config), and the full lite pytest suite (351 passed, 4 skipped, no change in outcome) all green; also exercised end-to-end against real local assets (SimilarityMatrix, people task), confirming the ledger correctly records 4 skipped + 1 failed and --action ledger-summary reports the grouped result. That end-to-end run surfaced a pre-existing, unrelated bug (Similarity has no implementation for similarity_metric="weight_overlap", even though pipeline_08 includes it in its default metric list) -- left untouched, out of scope for this change, noted separately for a future fix.
…logs/ dir
The ledger's default path is now {base_folder}/logs/pipeline_08_ledger.jsonl instead
of a bare logs/pipeline_08_ledger.jsonl beside the code. assets/ is already
gitignored wholesale, so this way the ledger is never committed without needing
its own dedicated ignore rule, and it stays scoped to whichever --base-folder a
given pipeline_08 invocation targets (matching every other pipeline output).
--ledger-path still accepts an explicit override; --action ledger-summary
resolves its default the same way, against --base-folder.
pipeline_06_compute_interference_per_pair.py: added --ledger-path/--no-ledger and inline ledger.record() calls at its three existing per-entity skip/failure branches (interference file already exists, missing trained model, incomplete dataset), plus a new try/except around the actual measurement block (evaluate_all_seeds + save), which previously had no error isolation at all -- one entity's metric failure crashed the whole run for every remaining entity. The new try/except mirrors the file's own existing isolate-and-continue pattern used for the other two failure modes rather than inventing a new one. pipeline_07_compute_interference_per_entity.py: compute_for_task's per-index loop body is now wrapped in try/except Exception (previously unguarded -- any exception while summarising one entity aborted every entity and every (method, epochs) combination in the same call), recording failed + continuing instead. Added tests/test_pipeline_07_compute_interference_per_entity.py (3 hermetic tests) reconstructing the exact pre-existing bug shape (an injected exception for one entity) and asserting the other entity still completes, the ledger records ok/skipped/failed correctly with exception type+message, and the pre-existing "file missing" skip path is now also logged. Verified: mypy (61 files, 0 issues) and pycodestyle clean on both scripts; full lite pytest suite green (354 passed / 4 skipped, +3 from the new test file, no regressions). pipeline_06 is heavy-tier and cannot be imported in the lite venv, so it was verified with py_compile (syntax) plus a real end-to-end run in a GPU-capable venv (breeds/uce/0, 2 entities with real per-pair files already present): real CLIP/BRISQUE/DINOv2 models loaded, both entities correctly hit the existing skip branch, the ledger recorded two real "skipped" records with the actual commit SHA, and git status confirmed zero side effects on real assets. A first attempt against the "people" task hung indefinitely at the baseline-completeness check (killed per RAM/CPU monitoring); root cause was people's local baseline dataset being incomplete (68/400 files), forcing a slow real HuggingFace reconciliation -- not a bug in this change. Switching to breeds, whose local baseline is complete (401/401), resolved the same code path in under a second.
…ity) pipeline_04_generate_dataset.py gets --ledger-path/--no-ledger and one ledger.record(status="ok") call at each of its two existing, already-safe completion points: after run_baseline's single GeneratedDataset.compute() call succeeds, and after run_normal's per-entity assert exists_unlearned_dataset(...) passes. The expensive training/generation logic itself is intentionally not wrapped in any new try/except -- a training or generation failure here should stop the run for investigation, not be silently skipped, so this file's ledger only ever records "ok", reflecting that the existing crash-on-error behaviour is otherwise unchanged. pipeline_05_compute_embeddings.py was reviewed but deliberately left untouched: it already has its own execution record (progress[key] = done/skipped/ skipped_no_dataset/skipped_incomplete_dataset/failed, persisted immediately after every step, richer in skip-reason detail than this ledger's schema). Verified: mypy (61 files, 0 issues) and pycodestyle clean; full lite pytest still 354 passed / 4 skipped (no new branching, no regression). pipeline_04 is heavy-tier and cannot be imported in the lite venv, so it was verified with py_compile (syntax) plus a real, cheap end-to-end run in a GPU-capable venv: --task breeds --baseline (breeds' local baseline dataset is already complete, so GeneratedDataset.compute() returns in ~0s) -- the ledger correctly wrote one "ok" record with the real commit SHA, and git status confirmed zero side effects on real assets.
…e RTs test_artifact_discipline.py's fresh-clone behavioural gate previously covered only 3 interference-per-entity result templates. Add the same coverage for ResultTemplateEmbeddingUnlearningProfile (resolving BaselineEmbeddings / EntityEmbeddings from HuggingFace on an empty local cache, including the documented degradation path when InterferencePerEntity/MetadataFiltered are also absent) and ResultTemplateInterferenceVisualSummary (the first fixture here to exercise a folder-shaped GeneratedDataset artifact rather than a single-file one, writing real image bytes into the resolved folder path). Also add a small check that every _ALLOWLIST entry still names a module and a callee that actually exist and are actually called -- an unused allow-entry silently widens the fitness gate's blind spot. Running it for the first time found two genuinely stale entries in _ALLOWLIST["testbed.py"] (get_interference_per_entity_path, get_metadata_filtered are no longer called by that module) and removed them. Each new fresh-clone test was verified to fail first by temporarily reintroducing the exact bypass it targets, then reverted.
New self-contained ablation under benchmarks/I_care/ablations/every_epoch/: select_entities.py chooses, per task, one forget target and nine receiver entities (two in each interference-by-similarity quadrant plus one median) from the canonical per-pair clip_diff interference and CLIP similarity, and writes a similarity-vs-interference scatter per task. fetch_target_check.py downloads just the single seed-42 baseline/unlearned image pair for the chosen target and renders a before/after check. test_select_entities.py covers the pure selection logic on synthetic fixtures (no torch, no GPU). The pure logic is separated from the lazy data-loading so the tests run torch-free; all produced data and figures live under the ablation's gitignored assets/ folder. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New opt-in field save_lora_at_epochs (default None) makes UnlearnerLora also
save the LoRA adapter into output_dir/epoch-{n}/ at the given 1-based epochs,
in addition to the final adapter. A field validator rejects non-positive,
boolean, float and string entries and returns a deduplicated, sorted list, and
train() fails fast if a requested epoch exceeds num_train_epochs. The
intermediate save uses a new _save_lora_layers_to that does not cast the unet to
float32 in place, so it cannot perturb ongoing training (unlike the final
_save_lora_layers). With the default None, training computation, saved weights
and eval_results are unchanged. A subclass that overrides final saving with
different artifact semantics would need to override _save_lora_layers_to as well;
this is documented and such subclasses do not set the field.
tests/test_epoch_checkpoint_hook.py covers the scheduler helper, the validator,
the range guard, and the non-mutating save; it is registered as a heavy test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…asets.testbed datasets/cifar.py and datasets/imagenette.py did `from torchvision import datasets, transforms`. Because datasets/__init__.py star-imports these submodules (`from ...cifar import *`) with no __all__, the name `datasets` (torchvision.datasets) leaked into the vision_unlearning.datasets namespace, and the top-level package's `from vision_unlearning.datasets import *` propagated it, so getattr(vision_unlearning, "datasets") became torchvision.datasets. Once those modules had been imported (e.g. transitively via result_templates), `import vision_unlearning.datasets.testbed` then raised "cannot import name 'testbed' from 'torchvision.datasets'", which aborted pytest collection of tests/test_artifact_discipline.py. Import the torchvision names under leading-underscore aliases (_tv_datasets, _tv_transforms) so `import *` no longer re-exports them, and update the in-file usages. No other package export changes, so the UnlearnDatasetCifar / UnlearnDatasetImagenette imports used by the pipeline are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
acquire_breeds_splits.py reproduces pipeline_01's breeds branch for a single target so the ablation has real distil training data: download_dataset_taras_breeds clones and jpg-converts the Dog-Breeds-Dataset, then split_dataset_taras_breeds builds train_forget (the target's images) and train_retain (the other 99 filtered breeds) from the already-present metadata_breeds_2_enriched_filtered.json. split_dataset_taras_breeds materialises the split with os.symlink, which fails on Windows (the process lacks the create-symlink privilege). For the split call only, os.symlink is replaced by a file copy; the image selection logic is untouched and no library code is changed. Paths resolve from __file__ so the CWD does not matter; vision_unlearning is imported lazily, matching select_entities.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
preflight.py checks the cheap go/no-go conditions before any GPU training time: ROCm/CUDA liveness plus a trivial GPU matmul, VRAM headroom, whether Stable Diffusion 1.4 is already cached, CPU/RAM baseline, and that the breeds spike-target forget/retain split is present. It prints PREFLIGHT_OK/PREFLIGHT_FAIL, writes a small result json, and returns non-zero on any hard failure so a launcher can gate on it. It does not train and does not download SD1.4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
spike.py runs a short local distil (SPARE) training for the breeds target with the save_lora_at_epochs hook and checks feasibility end to end: it records the effective LoRA dropout, validates each epoch adapter by safetensors content plus final-equivalence to the root adapter, generates with the last-epoch adapter while separating fixed model load from marginal per-image time, runs a baseline seed characterization, and monitors CPU/RAM/VRAM with a hard RAM-abort floor. Two local adaptations are needed and documented in the script: dataloader_num_workers=0 (Windows DataLoader workers use spawn and cannot pickle the dataset transform closure) and a --batch-size/--grad-accum pair defaulting to the canonical values but usable as 1/4 for the low-VRAM fallback (distillation keeps a frozen teacher model, so batch-2 at 512 in fp32 does not fit 12GB; 1x4 preserves the effective batch of 4). bench_generate.py isolates fp16 SD1.4 generation throughput (fixed load vs marginal per image) and reports the GPU device, to distinguish the fast fp16 generation path from the slow fp32 training-eval path and to size the inference batch (25 OOMs a 12GB GPU). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ation acquire_people_splits.py reproduces pipeline_01's people branch for a single target: download_dataset_lfw pulls bitmind/lfw from HuggingFace and, in one pass, writes train_forget (the target's images) and train_retain (the other 99 filtered people) as real JPEGs, reading the already-present metadata_people_2_enriched_filtered.json. LFW writes files directly, so unlike the breeds/scenes splits there is no os.symlink step and no Windows copy-substitute is needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
validate_adapters.py checks saved LoRA epoch adapters by content (CPU-only): per-epoch tensor/parameter counts and the max abs difference between the last epoch's adapter and the root final adapter (the final-equivalence check, which must be zero). run_demo_trajectory.py trains the breeds target with the save_lora_at_epochs hook over several checkpoints, generates the target concept at each checkpoint plus the base-model baseline, computes clip_diff per epoch, and writes a strip figure showing the concept degrade across epochs. It uses the validated local adaptations (num_workers=0, batch-1 x accum-4, small inference batch) and the CPU/RAM/VRAM monitor with a RAM-abort floor. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
generate_trajectory.py builds the per-epoch trajectory (target concept generated from each saved epoch adapter plus the base-model baseline, clip_diff per epoch, and a strip figure) without training, from a model dir already populated with epoch adapters. This lets the trajectory be produced when a long training run is interrupted but its intermediate adapters survived; it is fast and low-memory (fp16 generation only). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
seed_validation.py generates, for all 10 selected breeds across several prompt templates, base-model vs unlearned images at a fixed seed, and quantifies: determinism (same prompt+seed regenerated in a separate call), seed sensitivity (two seeds differ), and selectivity (low-interference breeds should barely change base->unlearned while the target and high-interference breeds change a lot). It writes per-template base-vs-unlearned grids and an interference-vs-image-change scatter. gpu_probe.py is a small diagnostic that loads SD1.4 fp16 and reports the pipeline's device and dedicated-VRAM usage plus one-image latency, to distinguish fast dedicated-VRAM generation from the slow AMD shared-memory fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On this gfx1031 GPU, generate_dataset at batch_size>=2 with deterministic algorithms enabled falls into a slow shared-memory kernel path (dedicated VRAM unused, ~10x slower). batch_size=1 keeps generation on dedicated VRAM (~15 s/image). Same images, generated one at a time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
make_epoch_grid.py assembles the football-field-with-epoch-axis grid: rows are the original base model then each saved epoch, columns are the selected breeds ordered by canonical interference (target first), and each cell is that entity's concept generated from that epoch's adapter (seed 42, batch_size=1) annotated with clip_diff vs the original. It reuses the base and last-epoch images already generated and fills in the missing intermediate epochs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The grid previously reused baseline/last-epoch images from a 30-prompt call, so a breed sat at a different RNG position (bi*3) than in the freshly generated intermediate epochs (bi). Because generate_dataset advances the generator once per prompt, that gave the baseline different initial noise than the epochs and inflated the original->epoch1 change. make_epoch_grid.py now regenerates the baseline and every epoch within the script using one consistent prompt ordering, so each entity shares identical initial noise across all rows. sanity_reproduce.py regenerates the last-epoch row in a fresh process and asserts it is pixel-identical to the grid's last-epoch row (cross-session reproducibility, max abs = 0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… epoch grid make_epoch_grid.py now runs for one or more seeds (--seeds) with seed-specific filenames, and computes a self-audit: the consecutive-row mean-abs pixel change over the control breeds (those not strongly forgotten, which should evolve smoothly), flagging any transition that is an outlier versus the median. This catches a badly-generated reference/baseline row (the fingerprint is an original->epoch1 transition that is an outlier) instead of relying on a human noticing it in the rendered grid; the verdict is written into the figure title and json. sanity_reproduce.py takes a --seed and compares against the seed-specific epoch-10 row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The training and grid scripts hardcoded the canonical LoRA learning rate of 6e-4 and wrote to fixed output paths, so a second hyperparameter setting could not be run without overwriting the first. Both now take --learning-rate and --run-suffix, and the grid additionally takes the --model-dir holding the adapters to render, so one code path produces every variant and their artifacts coexist. Add compare_learning_rate_collateral.py, which measures per entity how much of the change between the base model and a given epoch is semantic (clip_diff) versus raw image drift, for two runs that share a seed and entity ordering.
A long training run died after free system memory fell to the watchdog floor, and the two possible causes - a low starting headroom consumed by other applications, versus a leak in the training loop - differ in whether long runs are viable at all. Distinguishing them needs the slope of free memory against epoch number, which was not recoverable because the monitor samples carried no timestamp. Timestamp each monitor sample, and add analyze_ram_slope.py, which aligns those samples with the epoch adapter mtimes, fits the slope, and reports it in megabytes per epoch with a projection.
Both scripts hardcoded the breeds target, its split directory and its overwrite concept, so the people and scenes targets could not be run at all. They now take --task and read the target from that task's selection file, map the task to its split directory, and derive the overwrite concept and figure title from the target. The grid's entity list is no longer named after breeds, and its result key follows. Also add the scenes split acquisition, which reproduces the data notebook's SUN branch for a single target: the two tarballs are fetched and extracted, and the split is built through the library function with symlinks replaced by copies, as Windows refuses symlink creation here.
Fitting a line over every checkpoint mixed in two effects that are not part of the question: the allocation ramp while the model loads, and the final sample, which can be taken after the process has already released its memory. Together they turned a flat plateau into a positive slope, which would have been reported as evidence of no leak for the wrong reason. Fit the steady state instead, report both slopes, and state which one the verdict rests on.
A dropped connection ends the response stream without raising, so copying until the stream ends accepted 1.08 GB of the 1.82 GB image archive as a finished download. Only part of the image tree was then extracted, and because the guard only checked that the directories existed, a later run skipped re-downloading and failed instead during the split, on a missing image file. Check the downloaded size against the one the server declares, resume the partial file through a ranged request and retry, record which archive sizes were actually extracted so a partial tree is no longer mistaken for a finished one, and confirm the target has enough extracted images before building the split.
The grid ordered all ten entities by interference, which places the target first only when the target happens to be the most interfered of the ten. That is true for the breeds and people selections but not for scenes, where four receivers are canonically more damaged than the target, so the scenes figure showed a receiver in the first column under a title stating the target was there. Keep generation order as it is, since it fixes each entity's position in the random-number sequence, and order the columns separately: target first, then receivers by interference. Add the option to re-render a figure from the images an earlier run wrote, so a presentation change does not cost a full regeneration.
…oint
The grid ordered its columns by the canonical endpoint interference and printed
that value in each column heading, next to per-cell values from this run. The two
are different measurements - the endpoint one comes from a longer training run
averaged over four seeds, the per-cell ones from a shorter single-seed run - so
they disagree, which reads as an inconsistency in the figure.
Columns are now ordered by each figure's own clip_diff in its last rendered epoch,
with the target still pinned to the first column, and the endpoint value is no
longer drawn. Every number on a grid now comes from that grid's own run, and the
bottom row is visibly the sort key. Generation order is untouched: it fixes each
entity's position in the random-number sequence and must not depend on results.
Also: cell values carry two decimals and their name instead of a bare rounded
integer, entity labels come from the shared _short_entity_display helper rather
than a local replace(" dog", "") that would mangle a name containing "dog"
internally, the title follows the method/overwrite/seed pattern used elsewhere
and shows only the display name of the algorithm, and the audit verdict is no
longer written into the title.
Two correctness fixes come with it. Reusing already-generated images was guarded
only by "the files exist", although the files are named by position in the
generation order, so reusing a folder against a different entity list would have
silently relabelled the columns; each run now writes a manifest of what its images
depict and reuse fails on any mismatch. And the self-audit compared the
original-to-first-epoch change against a median that included that change itself:
measured against a deliberately mismatched reference row, the old rule scored 1.69
where it needed 2.0 to fire, so it would have missed exactly what it was for. The
baseline is now the median of the epoch-to-epoch transitions, which never involves
the reference row; the reference ratio itself is reported but not used as a verdict,
because it is also raised by the adapter simply appearing for the first time.
The grid shows what each entity looks like at every saved epoch, but reading the timing off a table of images is hard: when an entity starts to move, whether it recovers, and in what order the receivers fall are all easier to see as lines. Reads the grid's own result JSON, so it adds no computation and cannot disagree with the figure it accompanies. Colour is the entity's selection group, taken from the palette the selection script already uses, and the two entities sharing a group are separated by line style. Every line starts at (0, 0), which is exact: epoch 0 is the original model. It also writes the per-task table used in the write-up, so those numbers are generated by the same code that draws the figure instead of being retyped.
Two of the three every-epoch case studies show a retained entity ending further from its own concept than the forget target does, which invites the conclusion that this is how the method behaves. Three targets cannot support that, and the per-entity artifacts already answer it for every entity of every task at no cost. Counting number_of_interfered_worse_than_target_clip_diff over all 100 entities per task shows the opposite: at the endpoint only 3, 3 and 22 targets out of 100 (breeds, people, scenes) have even one retained entity damaged more than themselves, and two of the three studied targets have none. The selection picks the target causing the most interference, which is exactly where this can happen, so the case studies are biased towards it.
Both scripts shortened entity names with replace(" dog", ""), which removes every
occurrence rather than the suffix: a name containing "dog" internally, such as
"a dogo argentino dog", comes out mangled. No current breed name triggers it, so
no figure was wrong, but _short_entity_display already does this correctly and is
what the rest of the plots use.
A reversed clip_diff sign, an entity scored against a neighbour's baseline, a column order that ignores the target, a transposed figure, or a reused image folder belonging to a different run all produce a complete, plausible-looking grid. Counting files and looking at the result cannot separate those from a correct one, and one of them (the target not being pinned to the first column) did ship. The parts carrying those invariants are now module-level functions - the scoring loop takes its two score callables, the column order, the audit statistic, the manifest comparison, and the rendering - so each can be exercised with injected values and tiny fixture images, without a GPU or a model. The layout test reads the saved figure back and checks the colour of each cell, so it fails on a transpose or an ignored column order rather than on the arguments it passed in. Output is unchanged: re-rendering a grid after the extraction gives identical clip_diff values, column order and audit. The single-column case found a real defect while writing these: subplots collapses its axes array for one row or column, so the renderer only worked for grids of at least two of each. Fixed with squeeze=False.
The grids render one image per cell, so a single run cannot separate a property of the unlearning from a property of the initial noise. Rendering the same adapters at a second seed can, and it changes two of the conclusions. compare_seeds.py reports what survives. The target is strongly forgotten and becomes the overwrite concept at both seeds, and which receivers are destroyed is identical across seeds - but the magnitudes move by up to 4.7 points, and the ordering within the strongly affected receivers does not survive at all (rank agreement -0.50 in two tasks). The high agreement over all nine receivers, 0.80 and 0.75, comes from the gap between the entities that moved and those that did not, not from a stable ranking inside either group. check_reference_rows.py replaces a check that could not work from one run. The distance between the same entity's baseline images at two seeds is the scale of two unrelated draws of that prompt, so a correct baseline row must sit well below it while a baseline row taken from another seed sits at about that distance. All six score 0.33 to 0.47 against 1.0 for a deliberately mismatched row, which also clears the one row that the weaker single-run statistic had flagged at 1.88.
Renders the second seed for all three targets in one detached run: no training, since every epoch adapter is already saved, so it is generation and scoring only. Each target is a separate invocation, so an interruption costs one target rather than all three, and resources are sampled every five minutes because these runs are long enough that memory pressure would otherwise only show up as a crash.
…ablation Three additions to the ablation, all reading images that are already on disk: - spatial_heatmaps.py renders the per-epoch absolute difference of each entity against its own base-model image and reports how localised that difference is (the share of the change carried by the most-changed tenth of the pixels). Over the ten entities of each task, the correlation between how much an entity changed and how localised the change is runs -0.69, -0.94 and -0.90: the strongest changes are the least localised. - metric_progression.py measures the same images with the existing MetricImageImage(rmse, ssim) against clip_diff on a shared epoch axis, and measures the base model's own score difference between the two seeds as a noise floor for clip_diff. - cross_task_curves.py plots the three targets against optimizer steps, derived from the forget-set size and asserted against each run's recorded total. make_epoch_curves.py draws the noise floor as a band and reports each entity's most negative clip_diff alongside its last one, which makes transient damage visible. select_entities.py names all ten selected entities on the scatter. Contract tests for the new pure functions in test_spatial_heatmaps.py.
extract_unet_crossattn_activations passes a per-seed torch.Generator but does not enable deterministic algorithms, set CUBLAS_WORKSPACE_CONFIG, or seed the global RNG state, so its raw fingerprints are not bit-reproducible on ROCm. This is tolerable for the act similarity, whose fingerprints are averaged over positions, steps and seeds, but would not be for any single un-averaged tensor captured from the same loop.
…atent The new metric is the cosine between two entities' final denoised latents -- the 4x64x64 tensors Stable Diffusion 1.4 hands to its VAE decoder, captured with output_type="latent" so the decode is skipped. It sits between act (cross-attention states inside the UNet) and dino (DINOv2 over the decoded pixels) on the tap-depth spectrum: the model's own compressed verdict on the image, in its native space, before a frozen VAE renders it. UnetLatentSimilarity (similarity.py) owns the metric end to end: capture, cache, aggregation, matrix, and its own correctness gate. An entity's vector is the mean of its per-seed z_0, flattened in C order and L2-normalised. A fresh generator per (entity, seed) gives every entity the same noise draws, as the act fingerprints do, which makes the capture order-independent and resumable; only the first entity in prompt order coincides with the canonical baseline images, and validate_capture uses exactly that coincidence to check bit-reproducibility, canonical-image equivalence, equivalence of the reloaded latent after decoding, sensitivity to seed and prompt, and throughput. The per-(entity, seed) latents are cached as JSON under assets/datasets/ at five significant digits, which round-trips float16 exactly. The cache is a local aid, not an artifact: it exists so the aggregation can be changed without re-running the GPU pass, and the shareable product is the matrix the Similarity artifact already owns and resolves through HuggingFace. torch and diffusers are imported inside the GPU methods only, so the module stays importable without them and the metric's aggregation, matrix and cache handling are covered by the torch-free test tier. Also: - pipeline_02 gains --upload (upload a freshly computed matrix, resolving the Similarity artifact itself since the result template does not forward upload_if_recomputed to it) and --validate-capture, both applied uniformly through one helper the five similarity runners now share. - pipeline_08 gains --similarities, the counterpart of --mp. For the joint regression it is the exact ordered feature list, which is what makes a with-and-without comparison of a single similarity metric expressible. - unet_latent is registered without a display name, so it is not selectable in the GUI while it is a candidate, and it is not part of pipeline_08's default enumeration.
…rrect The seed set was a constructor field defaulting to the benchmark's seeds, which left three ways for two entities to be compared under different conditions without anything recording it. The cache filename encodes only the task and the model, so a cache captured under one seed set occupies the same path as any other; pydantic silently ignores an unknown keyword, so a caller passing seeds=... got the default while believing otherwise; and a finished cache was loaded without its recorded conditions being checked at all, so averaging two seeds out of a four-seed file would have returned a different vector from the same path, silently. Now: the seeds are a read-only property over the benchmark's own list, so there is exactly one seed set; construction forbids unknown arguments, so passing seeds=... raises instead of being ignored; and the recorded task, model, seeds and step count are re-checked on both paths that read a cache, resuming a capture and loading a finished one. Averaging a subset stays possible through the explicit seed_indices argument, which is visible at the call site. The capture gate now also proves order-independence rather than assuming it: it captures the same (entity, seed) twice with a different generation in between. Back-to-back repetition passes even when the noise depends on how many generations preceded it, which is the property that makes entities comparable with each other and makes a resumed run identical to an uninterrupted one.
The final-denoised-latent capture built a fresh torch.Generator for every
(entity, seed), so every entity received the first noise draw of its seed's
stream. The benchmark's own images are generated with one generator per seed
advanced across the whole prompt list, so entity k's image comes from the k-th
draw. The two coincided for the first entity in prompt order and for no other,
which meant the similarity was measured on a different set of images from the
ones interference is measured on and the ones dino embeds.
The capture now mirrors that loop: _run_seed seeds the global generators once
per seed and advances a single generator over the prompt list, so z_0 is the
latent behind that entity's stored baseline image. _capture_seed and
_generate_seed are thin wrappers differing only in output_type, so the gate
that compares the two paths is comparing one RNG regime with itself.
Because the reference now exists for every capture, every capture is checked
against it: _verify_against_baseline decodes z_0 and requires a mean absolute
difference below 1/255 from off_{seed}_{prompt}.png. One decode against a 50-
step denoise, so it runs on all 400 rather than on a sample, and the count and
worst difference are recorded in the cache. A cache is refused on load unless
it records every one of its latents having passed, which also rejects any cache
written under the previous scheme; the ordered prompt list and the seeding
scheme travel in the metadata and are re-checked, since reordering the prompts
changes every entity's noise while leaving every name present.
Resumption is at seed granularity and deliberately no finer. Restoring the
generator to a mid-list state would mean replaying its draws or serialising its
internal state, neither of which the torch-free test tier can verify; a seed is
25 minutes of a 1.7-hour pass. The checkpoint is written at seed boundaries
only, and _complete_seeds refuses a checkpoint that does not hold the first n
seeds in full for every entity rather than guessing which latents to trust.
capture() also confirms all the baseline images it will need exist before
loading the model, and validate_capture covers the first three entities across
all four seeds, since only an entity after the first can tell a correctly
advancing generator from a re-created one.
The bulk latent capture produced no output at all between its checkpoints: the pipeline's progress bar is disabled, and with the checkpoint written at seed boundaries the only external sign of life arrived every 100 captures, roughly 27 minutes apart. That is too coarse to tell a working run from a hung one. capture() now logs a line every 10 captures with the count verified against the baseline images and the running maximum difference, so the number the capture scheme rests on is visible at the capture where it changed rather than only in the finished cache, plus a line at each seed boundary. Those lines carry CPU, RAM and VRAM sampled in the capturing process, which is the only place VRAM can be sampled here: torch.cuda.mem_get_info reports the calling context's usage on this machine, so an external monitor polling it reads an idle GPU regardless of what the capture has allocated. The runner script's five-minute sampler did exactly that and is removed in favour of these lines; it would otherwise have recorded 1.8 hours of a constant labelled VRAM.
--rts matched requested names by substring only, so asking for MetricSimilarityAlignment also selected MetricSimilarityAlignmentMulti and ran the joint regression with the single-metric feature list meant for the pairwise template. An exact match now wins; substring matching remains the fallback for the shorthands it was written for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — opened to run CI on the branch, not for merge yet. The GPU stages (capturing the latents, validating the capture against the canonical baseline images, and the analysis that decides whether the metric is worth keeping) are still ahead; this only covers the code.
Adds
unet_latent: the cosine between two entities' final denoised latents, the4x64x64tensors Stable Diffusion 1.4 hands to its VAE decoder, captured withoutput_type="latent"so the decode is skipped. It fills the gap betweenact(cross-attention states inside the UNet) anddino(DINOv2 over the decoded pixels).UnetLatentSimilarityinsimilarity.pyowns the metric end to end: capture, cache, aggregation, matrix, and its own correctness gate.torch/diffusersare imported inside the GPU methods only, so everything except the capture is covered by the torch-free test tier.assets/datasets/at five significant digits, which round-trips float16 exactly. It is a local aid, not an artifact — it exists so the aggregation can change without re-running the GPU pass; the shareable product is the matrixSimilarityalready owns.pipeline_02gains--uploadand--validate-capture;pipeline_08gains--similarities, the counterpart of--mpand the exact ordered feature list for the joint regression.unet_latenthas no display name (so it is not GUI-selectable) and is not inpipeline_08's default enumeration, while it is a candidate.Like
act, this metric is computed with the model under study, so it breaks the abstraction that a similarity depends only on the entity pair — more so, since it reads the endpoint of the whole denoising trajectory. That caveat travels with it.