[WIP] WSVAD comparison framework — foundation (Spec 1)#22
Draft
jinmang2 wants to merge 28 commits into
Draft
Conversation
Restructure ~/data/wsad into per-dataset trees (ucf_crime/features/{i3d,clip},
annotations/, manifest.*, _archive/) so AIHub slots in without top-level ambiguity.
local.py gains layout resolvers (_features_root/_clip_dir/_i3d_npy_dir/_i3d_zip_path
+ gt) that prefer the new layout and fall back to the legacy one; has_local now
takes data_cfg (trainer passes it). build_manifest.py emits a header-only
parquet+jsonl catalog (3800 rows). RTFM dirs quarantined to _archive (incompatible
~10x scale). 53 tests green; build_datasets i3d+clip both 1610/290.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…pro probe Port GowthamGottimukkala/I3D_Feature_Extraction_resnet faithfully: convert the facebookresearch Caffe2 nonlocal-net blobs (baseline+nonlocal) to PyTorch (convert_i3d_caffe2.py, verbatim regex map), and mirror its preprocessing exactly (340x256 LANCZOS, (x*2/255)-1, exact 10-crop, ffmpeg->jpg, snippet 16) in i3d_gowtham.py -> (T,10,2048). verify_i3d_extract.py proves it is BIT-EXACT (max|delta|=0) vs the unmodified reference. extract_i3d_gowtham.py is the batch entry. Also: make pytorchvideo import lazy in i3d.py (only the i3d_8x8_r50 SlowFast variant needs it; the self-contained I3Res50 path does not), and fix I3DFeatureExtractor passing frames_per_clip= where the dataset expects clip_length=. RTFM repro probe (diag/grid_i3d_rtfm.py): RTFM's released UCF features are NOT reproducible here -- nonlocal crop-avg cosine plateaus at 0.74 (baseline 0.51); crop-order, temporal offset, BGR, and resize/interp all ruled out -> RTFM used a different/unpublished checkpoint. MGFN zips stay canonical; RTFM stays archived. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ne track) Implement VideoMAEFeatureExtractor.extract (was NotImplementedError) via transformers VideoMAEModel + VideoMAEImageProcessor: non-overlapping 16-frame snippets, mean-pool patch tokens -> (T, dim). dim is read from model config so a VideoMAEv2 / ViT-L checkpoint swaps in via model_name. Validated: (112,768) on a 1795-frame clip in 7.1s, finite. Unlocks an I3D-vs-VideoMAE feature ablation on the visual-only heads. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…arch roadmap
eval_mgfn_local.py and smoke_train_mgfn.py read {test,train}.zip / ground_truth.json
by hardcoded dataset-root paths, which the features/{i3d} + annotations/ restructure
broke. Reuse the loader's _i3d_zip_path / _local_ground_truth_path resolvers (new
layout + legacy fallback, single source of truth) instead of hardcoding. Verified:
MGFN local eval ROC-AUC 0.8208 / 290 test, smoke-train loss 1.25->0.21.
Add docs/RESEARCH_ROADMAP.md: landscape (I3D ceiling ~86, CLIP+fusion lever,
VideoMAEv2/MLLM frontier) + ROI-ordered experimental ladder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…earch Approved Spec 1 (foundation: backbone-agnostic extraction, dim-agnostic heads, matrix harness, HF-style serving pipeline) plus the 3-spec roadmap (streaming/causal + qualitative suite). Includes backbone/data/optimization research grounding the multi-backbone matrix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- MGFN: derive the feature/magnitude split from config.channels instead of a hardcoded 2048, so CLIP (512) / VideoMAE (768) backbones work. The 2048 path is byte-identical (regression test on MGFNFeatureAmplifier). - Add requires_text_aligned=True to VadCLIP and TPWNG (the only heads using a CLIP text encoder); src/compat.py guards backbone<->head pairs and raises a clear error for a text head on a visual-only backbone. - tests/test_compat.py covers both. Closes #18. Part of #16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bundles backbone (processor) + head + post-process into one serveable object: raw video -> per-frame anomaly scores. Runs the text-align guard at construction, auto-matches the head's feature-dim field (feature_size/channels/visual_width) to the backbone dim, and exposes from_features (extraction-free) + visualize hooks. window/causal modes extend the same object in Spec 2. Closes #20. Part of #16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- src/features/extract.py: dependency-light run_extraction/save_path_for core (caches <stem>_<backbone>.npy via any registered extractor); unit-tested with a fake extractor (no decord/GPU). - scripts/extract_features.py: add --backbone dispatch to build_extractor(); make `import decord` lazy so the script imports without it. Legacy i3d_8x8_r50 path unchanged when --backbone is unset. Part of #17 (dispatcher); InternVideo2/VideoMAEv2 loaders deferred to a GPU session. Part of #16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 16, 2026
Owner
Author
|
Session paused — resuming on another machine. Handoff doc committed: |
…gth masking) From-scratch VadCLIP was stuck at chance (ROC-AUC ~0.49) — the eval-only official checkpoint cell hid this. Root cause: VadCLIPPreTrainedModel._init_weights xavier-overrode EVERY Linear, ~2.5x-inflating the binary head vs the official's PyTorch-default init (official only custom-inits the two task embeddings, model.py:119-122). Inflated logits saturate at init and the MIL objective collapses to all-low scores. Fix: leave Linear/MLP/GraphConv at default init. Also made training faithful to official ucf_train.py: - load_pretrained_clip_text(): from-scratch == random head + FROZEN pretrained CLIP ViT-B/16 text tower (official clip.load + requires_grad=False); without it the text-alignment branch is random. - _compute_loss(): per-video length masking — top-k over real frames [0:length] with k=length//16+1 (official CLAS2/CLASM), not fixed-k over the padded 256-window. - WSVADTrainer: optimizer_name="adamw" + MultiStepLR knobs, fit() eval_fn/best-ckpt, and threads per-video `lengths` (from non-zero frames) when the head accepts it. experiments/repro_vadclip.py drives the official recipe (AdamW 2e-5, MultiStepLR [4,8], 10 epochs, seed 234). Result: from-scratch ROC-AUC 0.49 -> 0.8457 (paper 0.8801, official-ckpt 0.877). Inference path unchanged: official ckpt still loads to 0.8770; 74 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…poch eval Faithful from-scratch VadCLIP was 0.8457 (commit 8ac1fc3) vs the official-ckpt ceiling 0.877 in our harness. Two remaining official-recipe gaps closed: - 10-crop training: official ucf_CLIP_rgb.csv = 1610 videos x 10 crops = 16100 rows (each crop a separate training sample; 10x crop augmentation). We trained on a single crop. New data.train_all_crops loads all crops for train, test stays single-crop. -> 0.8457 -> 0.8643. - intra-epoch eval: official evals at step % 1280 == 0 (~12x/epoch on the 16100-row train) and keeps the best checkpoint. WSVADTrainer.fit gained eval_every (optimizer steps) with best-by-select_metric across all eval points. -> 0.8654 (135 eval pts). From-scratch VadCLIP: 0.49 (broken) -> 0.8654 = 98.6% of the 0.877 ckpt ceiling (paper 0.8801). 74 tests pass; inference path unchanged. src/data/local.py also carries the matrix's I3D feature-variant loader (feature_variant / _i3d_variant, with_magnitude override) from the eval-matrix work. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SharePoint folder-zip downloads exceed the server size limit; download per-file instead via _layouts/15/download.aspx?SourceUrl= + FedAuth cookie. - sp_list.sh: REST folder listing (GetFolderByServerRelativeUrl/Files) -> manifest TSV - sp_download.sh + dl_one.sh: parallel, size-verified, resumable download (-L for 302s) - sp_verify.sh: count/size/npy-magic check against manifest - README: full recipe incl. how to capture the session cookie Used to recover the MGFN pre-seg32 train folder (1610 npy / 64.87 GB). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…I3D)
Full-length (T,10,2048) RTFM-scale (L2~22) UCF-Crime I3D from the MGFN authors'
HKU OneDrive, fetched via scripts/onedrive_fetch/ -> features/i3d_mgfn/{train,test}/
(1610/290, byte-exact). Loader resolves it via data.feature_variant=i3d_mgfn.
- local.py: document i3d_mgfn in the _i3d_variant docstring + scale-mixing warning
- DATA_LOCAL.md: add i3d_mgfn rows; disambiguate the two "MGFN" lineages
(canonical features/i3d/ is DeepMIL L2~2.5; i3d_mgfn is the real OneDrive L2~22)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RTFM/MGFN from-scratch on i3d_mgfn (constant lr 1e-3, official recipe) DIVERGES: train loss climbs, test AUC oscillates 0.80-0.83 around an early lucky peak, so the "best test AUC" the field reports is largely an undersampled-noise artifact. Adding a per-step CosineAnnealingLR over max_steps converges it: at lr->0 the AUC tightens to a stable plateau (RTFM: 0.82 ± 0.005 over the last eval points) instead of bouncing. - WSVADTrainer.fit_steps: `lr_decay="cosine"` (per-step schedule on the prepared optimizer) + returns `auc_mean`/`auc_std` over the late half of eval points (the honest stability metric alongside the optimistic best-test-AUC). - run_matrix: `--lr-decay cosine`; results.json gains auc_mean/auc_std/lr_decay. RTFM i3d_mgfn_seg32 (6000 steps, cosine): best 0.8341 (vs paper best 0.843, -0.009), stable 0.82±0.005. Confirms the under-optimization hypothesis — constant lr was the problem, not the features/port. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reading the official MGFN (carolchenyx/MGFN) revealed it evaluates EVERY step (`step % 1 == 0`) and keeps the best — because MGFN overfits fast (early test-AUC peak, then decline). run_matrix's default eval (every max_steps//30 ~ 166 steps) undersamples that peak, so MGFN looked capped at 0.806. With --eval-every 25 the early peak is caught: MGFN best 0.8332 (vs official-ckpt ceiling 0.8346) — the gap was undersampling, NOT a port/feature bug (our MGFN loss is byte-identical to the official; the 0.52 divergence earlier was DeepMIL L2~2.5 features mismatching the margin-200 magnitude-contrastive loss — i3d_mgfn L2~22 trains fine). dual-report exposes training character: RTFM best 0.834 / mean 0.82±0.005 (converges, cosine) vs MGFN best 0.833 / mean 0.79±0.009 (overfits, early-stop). Both ~0.83. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
UR-DMU/BN-WVAD need 200-seg features; i3d_mgfn_seg200 train is 25 GB (1610 x 16 MB).
_load_i3d eager-loaded ALL npy into a dict -> 25 GB > 15 GB WSL RAM -> the OOM-killer
reaped processes including the Claude Code session (3 repeated session crashes; nohup
didn't help — OOM-killer ignores it). Fix: _load_i3d returns {name: path}; FeatureDataset
already supports lazy per-__getitem__ np.load via open_func, so build_datasets_local
passes open_func=_identity for i3d (clip stays eager). Verified: seg200 build peak RSS
0.83 GB (was 25 GB), sample (10,200,2049) intact; 74 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…/every/SEG knobs UR-DMU/BN-WVAD do O(T^2) attention over the full test sequence; scoring all 10 crops in one forward OOMs an 8GB GPU on long videos. The official ucf_infer.py scores each crop's full-length sequence separately then averages the 10 — so src.eval_matrix now does per-crop forward+mean for _PERCROP_HEADS (faithful AND ~10x less peak memory -> fits 8GB). Also: run_matrix --eval-start / --eval-every overrides (MGFN/UR-DMU peak early -> eval often, faithful warmup control); build_mgfn_seg32.py SEG env (200-seg). Research note: UR-DMU on i3d_mgfn (2048-d) only reaches 0.804 vs paper 0.8697 because UR-DMU/BN-WVAD use 1024-d I3D (piergiaj/pytorch-i3d, DeepMIL/Roc-Ng release) — verified from the official ckpt (embedding.conv_1 = Conv1d(1024->512)) — NOT the 2048-d ResNet50-I3D that RTFM/MGFN/S3R use. random_perturb is np.linspace (no augmentation). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ure-dim UR-DMU/BN-WVAD use 1024-d I3D (Carreira/pytorch-i3d, DeepMIL/Roc-Ng) — verified from the official ckpt (Conv1d 1024->512), not the 2048-d ResNet50-I3D of RTFM/MGFN/S3R. build_deepmil_1024.py stacks the downloaded per-crop npy -> (10,200,1024) train / (T,10,1024) test; run_matrix gained --feature-dim to override the 2048 default. DECISIVE: official UR-DMU ckpt on these 1024-d features evals to 0.8697 = paper EXACTLY (missing=0) -> data + per-crop eval + port all faithful. UR-DMU's from-scratch gap (~0.80) is the TRAINING loop, not the feature dim (1024 gave 0.797 ~ 2048's 0.804). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Group the per-model fidelity checks under scripts/verify/ and the diagnostic probes (i3d/rtfm grids, smoke trains, oracles, extraction validators) under scripts/diag/, with a scripts/README.md index. Pure moves, no logic change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The old magnitude_forensics.py headlined raw-feature-L2 ROC-AUC as a feature-quality ceiling and an accept/reject gate for new extractors. Measured: it is unsound — i3d_mgfn magnitude-AUC=0.46 (near-chance, mildly inverted at test) yet RTFM/MGFN train to 0.83 on it, so magnitude neither predicts nor bounds trainable performance. Replace with feature_forensics.py: three RELATIVE atemporal proxies (RTFM-faithful magnitude, content-distance-to-normal-centroid, and a 5-fold GroupKFold linear probe), each with |AUC-0.5|, plus an explicit caveat that every per-snippet score under-predicts the temporal heads (probe 0.51 -> head 0.83) and is a ranking screen, not a ceiling. Apples-to-apples (same 40 vids) the screen ranks videomae 0.71/0.62 > i3d_1024 0.62/0.55 > i3d_mgfn 0.56/0.51 — modern features are more separable, the real lever, for the content reason not magnitude. Drop dead _mag_scores; fix the crop aggregation to norm-then-cropmean. Correct REPRO_FROMSCRATCH/NEXT_PHASE verdicts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ming extract RAM-bounded snippet streaming in FeatureExtractor.iter_snippets (decord random-access, batch*snippet_len frames at a time — long surveillance clips OOM-killed WSL on decode-all), with a sample_to fast seg path. Register VideoMAE/X-CLIP/InternVideo extractors behind the @FEATURE_EXTRACTORS registry; extract_modern.py streams one video out of the raw zips at a time (disk peak ~= one video) and dispatches by --backbone, with a balanced forensic-gate sample. build_* scripts assemble the i3d/1024-d feature sets. Smoke-covered by the 74 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…head tweaks src/modules/amp.py (autocast-safe BCE) and the WSAD_ATTN=mem SDPA path in translayer.py let the memory-heavy heads (UR-DMU/BN-WVAD/CLIP-TSA) fit 8GB; trainer.py gains step-based cosine decay + mean±std reporting. run_matrix.py adds --feature-dim / --eval-every / --eval-start knobs and gpu_queue*.sh run the serial 8GB queues. Minor faithful tweaks to the gs_moe/s3r/tpwng/ur_dmu/vadclip heads + bn_wvad/s3r configs. Eager/fp32 path unchanged — 74 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
experiments/reproduce.py (single parametrized from-scratch harness, epoch recipes), repro_i3d.py, and results_table.py (regenerates the UCF-Crime ROC-AUC table from the per-run result.json artifacts under experiments/runs/). Verbose *.log gitignored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RESULTS_TABLE.md (current from-scratch numbers), SPEC2_EFFICIENCY.md (efficient/ causal-inference track), WSVAD_REPRO_RECIPES.md (per-head recipes). gitignore: drop verbose experiments/runs logs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
transformers' VideoMAE port stores attention bias as split query/key/value.bias but does NOT convert MCG-NJU checkpoints' original timm-style q_bias/v_bias (k_bias≡0). Result: query.bias/value.bias load as ZERO, silently discarding a large learned bias (||q_bias||≈17.5 per layer) — corrupting every attention score and the extracted features. _patch_qkv_bias() maps q_bias->query.bias, v_bias->value.bias from the raw checkpoint after load; no-op on versions that already map correctly. Verified: layer0 query.bias norm 0.0 -> 17.5411 (exact). This invalidates the earlier videomae_GATE features (extracted with zeroed bias); they still beat i3d on the forensic screen (content 0.71 vs 0.56), so corrected features should screen at least as well — re-extracting to confirm before any full 1900-video run. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Single source of truth for the whole WSVAD comparison-framework effort. All
sub-work is tracked here (no per-spec PRs, no per-task issues). Epic roadmap: #16.
Draft until the work is validated at runtime (GPU extraction, real-time, causal) —
not merge-ready on unit tests alone.
Goal
Multi-backbone × multi-detector comparison + HF-style serving + paper-style test
figures + real-time/causal inference. Design:
docs/superpowers/specs/2026-06-16-wsvad-matrix-serving-design.md,docs/FEATURE_EXTRACTOR_RESEARCH.md.Invariant: original training/model defs unchanged; streaming/causal is additive.
Spec 1 — Foundation (scaffold done, runtime-UNVERIFIED)
src/compat.py)src/pipeline.py,from_featuresonly)src/viz.py)--backboneextraction dispatcher (src/features/extract.py){backbone × head}matrix with real AUC numbersSpec 2 — Streaming / Causal (NOT STARTED — the real weight of the project)
Spec 3 — Qualitative suite (NOT STARTED)
Status note
Unit suite (74 passed) only proves the scaffold isn't broken — NOT that detection
works on real video. Completion is claimed only against runtime evidence.