Harvest upstream/main (18 commits) — MoE KV engines, CLI restructure, lql updates#17
Open
mikeumus wants to merge 442 commits into
Open
Harvest upstream/main (18 commits) — MoE KV engines, CLI restructure, lql updates#17mikeumus wants to merge 442 commits into
mikeumus wants to merge 442 commits into
Conversation
Per-layer Shannon bits via the final-norm logit lens. At every captured residual (embed plus each post-block-L), project through final_norm + lm_head, compute per-token cross-entropy and KL-to-final, then report adjacent bits_saved[L] = bits_via_lens[L-1] - bits_via_lens[L]. Mirrors exp 34 (chris-experiments/shannon/34_internal_bit_contribution). Validated against Gemma 3 4B on 30KB Frankenstein: larql shannon layers: 108.59 bits/token saved across L0->L33 exp 34 reference: 111.69 bits/token Same shape, including the L22-L25 commitment band (negative bits saved while KL-to-final improves).
Experimenting
fixed gguf compile issue
more clippy issues
… RFC-0001) (#7) Implements Phase B of RFC-0001 (#2): single-fact rank-1 editor with portable patch file format. Builds on Phase A's LastPositionAblatingFfn (#3) and adds the symmetric LastPositionInjectingFfn for scale search. - `EditPatch` struct (serializable via serde) - `compute_rank1(k, d, scale, layer, provenance) -> EditPatch` - `write_patch(path, &patch)` / `read_patch(path) -> EditPatch` with a simple binary format: LQPATCH magic + JSON meta + little-endian f32 vectors for d and k_norm. ~55 KB for Gemma 4 4B. - `apply_patch(&mut ModelWeights, &EditPatch)`: installs the rank-1 outer product into `down_proj.weight` in place, handling both `[hidden, intermediate]` and `[intermediate, hidden]` layouts. - `LastPositionInjectingFfn` — adds a fixed delta vector to the inner backend's last-row output at one target layer. Symmetric to the ablating wrapper from PR #3. Used for auto-scale search. - `larql edit <model> --src "..." --tgt "..." --new-token " Tokyo" --output f2t.lqpatch` Runs Phase A crown discovery (or accepts `--layer`), captures k at the crown layer for both prompts, computes d = W_down @ (k_tgt - k_src), linearly searches [0.5, 1, 1.5, 2, 2.5, 3, 4] for the minimum scale that flips the source's top-1 to --new-token, emits the patch. - `larql apply-patch <model> --patch f2t.lqpatch --prompt "..."` Non-destructively installs one or more patches into the loaded weights, optionally runs a test prediction. Supports `--reverse` to subtract a patch (verifies reversibility). - Added `InferenceModel::weights_mut()` accessor so apply-patch can mutate the in-memory weight map without reloading. Methodology validated in Python across Divinci-AI/server notebooks/CHAPTER_20_HONEY.md (Phase 140c: France→Tokyo with 11/11 specificity at 0.9% weight perturbation) and CHAPTER_18_THE_EDIT.md (Phase 130 scale search). The Rust port preserves the same math. Compile-checked with `cargo check --package larql-cli`. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wraps the existing covariance-MEMIT solver (larql_inference::forward::memit:: run_memit) with a CLI, an edits.json file format, and automatic crown-layer discovery for each edit. Groups edits by crown layer, invokes the joint least-squares solve, emits one dense `.lqpatch` per affected layer plus a manifest.json. Phase C of RFC-0001 (#2), stacked on Phase B (#4). - Bumped patch version 1 → 2 with a `kind` field (defaults to "rank_one") - New `kind = "dense"` variant carries a flat row-major ΔW matrix, needed because MEMIT's covariance-projected solve isn't natively a rank-1 outer product. Larger on disk (~72 MB per Gemma 4 4B layer) but semantically exact — no SVD approximation step. - `write_patch`, `read_patch`, `apply_patch` all dispatch on kind. Phase B rank-1 patches continue to round-trip unchanged. - New `compute_dense()` helper builds a Dense patch from an Array2<f32>. - Reads edits.json (list of {label, src, new_token, layer?} records). - For each edit: tokenises src, resolves target_token_id, resolves crown layer (explicit or auto-scan). - Calls `run_memit` with Vec<MemitFact>, receives one `MemitResult` per affected layer. - Serialises each layer's ΔW as a Dense patch into the output directory, writes a manifest.json enumerating them. - Prints the apply-patch command to install the batch. cat > edits.json <<EOF [ {"label":"france-to-tokyo","src":"Capital of France? A:", "new_token":" Tokyo","layer":27}, {"label":"germany-to-rome","src":"Capital of Germany? A:", "new_token":" Rome","layer":27} ] EOF larql memit /path/to/gemma4 --edits edits.json --output patches/ larql apply-patch /path/to/gemma4 \\ -p patches/memit_L27.lqpatch \\ --prompt "Capital of France? A:" Chapter 22 established that single-layer MEMIT with correlated keys (~60% cosine) lands ~3/5 concurrent targets. For 5+ correlated edits, users can now distribute across multiple crown layers via `layer` overrides in edits.json — MEMIT runs once per layer group. Compile-checked with `cargo check --package larql-cli`. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… of RFC-0001) (#9) Exposes the Phase A-C commands as Python callables so the Chapter 15-23 Colab experiments from Divinci-AI/server become one-liner Rust invocations from Jupyter — no CLI shell-outs, no JSON parsing. Four #[pyfunction] entry points: - crown(model, prompt, expect, start_layer=None, end_layer=None, top_k=100) Returns {crown_layer, crown_delta_prob, top_after_ablation, scan: [...]}. - edit(model, src, tgt, new_token, output, layer=None, scales=None, fixed_scale=None, top_k=100, label=None) Writes a rank-1 .lqpatch; returns {layer, scale, output, d_norm}. - apply_patch(model, patches: list[str], prompt=None, top_k=5, reverse=False) Applies patches in-memory; optional prompt returns {predictions: [(tok, prob), ...]}. - memit(model, edits: list[dict], output_dir, ridge=0.01, target_alpha=1.0, top_k=100) Batch fact editor wrapping run_memit — writes one dense patch per layer into output_dir + manifest. - Registered in _native pymodule (src/lib.rs) via m.add_function. - Re-exported from python/larql/__init__.py under the public `larql` namespace alongside the existing load_vindex/create_session functions. import larql scan = larql.crown("/path/to/gemma4", "Capital of France? A:", " Paris") print(scan["crown_layer"]) # 27 (on Gemma 4 4B) larql.edit("/path/to/gemma4", src="Capital of France? A:", tgt="Capital of Japan? A:", new_token=" Tokyo", output="france_to_tokyo.lqpatch") r = larql.apply_patch("/path/to/gemma4", patches=["france_to_tokyo.lqpatch"], prompt="Capital of France? A:") print(r["predictions"][0]) # ['Tokyo', 0.97] This closes the RFC-0001 phased rollout: Python scripts can now drive the mechanistic fact-editing pipeline end-to-end. Compile-checked with `cargo check --package larql-python`. Runtime import requires `maturin develop` — standard PyO3 workflow, no Python side of the package changed structurally. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…on guard for rebuild_overrides (#14) README: Add a fork notice block with badges (Divinci AI, Hugging Face, Vindex Viewer Space, License, Upstream link). Frames this repo as the Divinci-AI fork of chrishayuk/larql carrying RFC-0001 mechanistic fact-editing, Phase-1 unlearning with the revert-leak fix, Gemma 4 per-layer intermediate-size, and the CI isolation harness. Test (overlay_apply): Add `rebuild_overrides_clears_base_down_and_up_overrides` — permanent regression guard for the Phase-1 unlearning revert path. Pre-populates `base.down_overrides` + `base.up_overrides` via `set_down_vector` / `set_up_vector` (the COMPILE-WITH-REFINE write path), pushes any patch onto the overlay so `remove_patch(0)` triggers `rebuild_overrides`, then asserts both base maps are empty after revert. If a future refactor drops the two `clear()` calls in `rebuild_overrides` this test turns red — caught the same regression Gate 3 catches at the integration level, but in 1ms instead of 5sec. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a per-tensor FP8 block-quant dequantization path for models that
store weights as a `(.weight: F8_E4M3, .weight_scale_inv: F32 [N/128, K/128])`
pair — Kimi-K2 family + DeepSeek-V3 fp8 forks. Distinct from MXFP4
(`I8 + F8_E8M0` per-32-element scales) which the existing path handles.
Convention follows the DeepSeek-V3 reference:
dequant[i, j] = decode_f8_e4m3(weight[i, j]) * scale_inv[i / 128, j / 128]
Wired in two places:
* `larql-models/src/loading/safetensors.rs` — new pre-pass
`dequantize_fp8_block_companions` runs before the main per-expert
loader and consumes both `.weight` and `.weight_scale_inv` so the
main loop skips them. Applies to ALL fp8 tensors in the shard
(per-expert MoE weights AND dense attention weights).
* `larql-vindex/src/extract/streaming.rs` — `get_tensor_f32` gains
the same companion-detection branch so the gate_vectors stage
produces correct floats for fp8-block models. Calls
`larql_models::loading::safetensors::decode_f8_e4m3` (newly `pub`)
instead of duplicating the per-byte E4M3 decoder.
* `larql-models/src/detect.rs` — adds explicit `"kimi_k2"` arm that
maps to `DeepSeekArch` (Kimi-K2 uses the same DSV3-style tensor
naming: `model.layers.X.mlp.experts.E.{gate,up,down}_proj.weight`).
Verified clean `cargo check` on both `larql-vindex` and `larql-cli`
with `--no-default-features`. End-to-end validation will be a Modal
extract of `moonshotai/Kimi-K2-Instruct` (61 layers × 384 routed
experts, fp8) → `Divinci-AI/kimi-k2-instruct-vindex-browse`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…2026-05-28) Four-wave plan capturing 111 net new upstream commits beyond our May 4 harvest. Wave 1: license + correctness cherry-picks (this branch). Wave 2: GGUF stack + MLA absorption merge. Wave 3: KV/compute refactor (gated). Wave 4: optional pickups. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wave 1 landed 5 of 12 planned cherry-picks cleanly (evalexpr license pin, HF cache model-repo scan, router matmul_transb, signed-probe fix, RFC). The remaining 7 are all blocked by upstream directory restructures (gguf/, quant/ggml/, format/weights/, layer_graph/grid/, etc.) that our fork hasn't absorbed yet. Revised plan: Wave 2 splits into 2a (structural absorption — load-bearing, 1-2 day dedicated session) and 2b (deferred Wave 1 fixes apply cleanly once the directory layout is aligned). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wave 2 used a reset-to-upstream + replay-fork-commits strategy after a direct merge produced 284 conflicts. Result: branch at upstream/main + 9 fork commits, all tests green. Effectively absorbs Wave 3 and Wave 4 inventory in one motion. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Carried over from the pre-harvest main (originally PR #12). The Wave 2 upstream reset to b6d5e8d didn't bring this Divinci-AI-specific CI workflow forward; reattach it now alongside the synthetic 5 MB vindex fixtures (8 layers, hidden=128) used by the harness. Companion to the Divinci-AI/larql-isolation-harness external testing project. The vindex weights are reproducible from testdata/tiny-vindex/generate.py (seed=42). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…onship Captures the 2026-05-28 harvest playbook (RFC-0002) as a project-local skill that auto-triggers on "harvest", "upstream sync", "merge upstream", etc. Encodes: * Survey-first triage (directory diff between merge-base and upstream) * Cherry-pick-vs-reset-and-replay decision tree * Our Commands/DevCommand conflict pattern * fp8-block-quant preservation in upstream's new module structure * Force-push warning when resetting main * Post-harvest hygiene (upstreaming, retiring verbatim ports) * RFC-as-living-document convention CLAUDE.md gains a "Fork relationship" section pointing at the skill and listing the project's RFC index. .gitignore narrowed from `.claude/` to `.claude/*` + `!.claude/skills/` so project-shared skills travel with the repo while per-user settings stay local. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Opt-in within-expert feature routing inside the production expert kernel (run_single_expert_q4k_q8k_into), OFF by default = byte-exact parity. Harness walk_ffn_v1_moe_within_expert.rs runs V1's 3-phase protocol on each 26B-A4B expert's 704-feature FFN. Result: FALSIFIED like dense V1 — experts are dense in their own feature space (L0-13 need all 704), per-layer sparsity doesn't compound (50% argmax drift), deployable BW ~1.19x. 9 unit tests + 2 hidden.rs integration tests; 671 compute + 1133 inference tests pass.
Parallel test threads race on process-global decode flags cached at MetalBackend::new() + the shared GPU, intermittently yielding NaN/zero pipeline output. Mirrors the Makefile targets + larql-inference's OpenBLAS fix.
~230 targeted tests across cov_* integration files (synthetic vindex + mockito remote + patch lifecycle). Ratchet coverage floors: kv 85->90, vindex 71->90, compute-metal 73->90, models 80->88, lql total ->93; remaining lql debt is real-model/interactive/Weight-only, documented in coverage-policy.json.
… lql coverage 89->94%, CI/lint fixes
Inherited cross-platform gaps surfaced when the branch hit main's x86_64/ Windows/ubuntu CI (Mac-only local checks couldn't catch them): - compute: gate the aarch64-only NEON/asm kernels in the q4k_q8k_matvec bench behind cfg(target_arch="aarch64") — unconditional import broke x86_64 (E0432). - vindex: route per-layer madvise through a cfg(unix) mmap_util::advise_willneed helper, and gate the unix-only mmap_cold_read_probe example — libc is a cfg(unix)-only dep, so the ungated uses broke Windows (E0433). - lql: escape Windows backslash paths in the cov_mutation merge tests (the LQL lexer eats \U/\T escape-like sequences in unescaped paths). - kv/models: add debt baselines for files that measured <90% on ubuntu — standard.rs/generation.rs (MoE-KV-engine regression) and tq.rs (bitnet TQ). See policy notes; kv entries are tracked for restore via mock-remote tests.
Compute total is 95.08% (clears --fail-under 95); the policy gate failed on two files below the 90 per-file default, both from inherited branch work: - attention/decode.rs 55% — the opt-in run_attention_block_decode_step_q4k_direct path (LARQL_Q4K_DIRECT_ATTN, roadmap #16), not exercised by default tests. - kv_dispatch/cpu.rs 88% — engine-unification dispatch surface. Baselines added with restore follow-up noted in coverage-policy.json.
…e baselines Replace the inherited debt baselines with real tests: - attention/decode.rs 55.2% -> 92.3%: drive the opt-in Q4K-direct decode path (run_attention_block_decode_step_q4k_direct) via the Q4kFixtureIndex, incl. a parity test vs the dequant f32 path (<1e-3, roadmap #16 contract), plus an env-gated attention_step Q4K-direct test (LARQL_Q4K_DIRECT_ATTN). - kv_dispatch/cpu.rs 88.2% -> 97.1%: cover coarse_decode/prefill no-index + direct-matvec arms. Crate total 95.0% -> 96.1%. Both per-file baselines removed (only projector.rs remains); residuals documented in coverage-policy.json. Production logic untouched.
The gap was the resident-weights dispatch path (not remote-MoE as first
assumed): StandardEngine::{prefill,decode_step}_resident + generate_with_engine_resident
+ the from-hidden break/EOS arms. Covered via a real StandardEngine over the
synthetic q4k fixture (CPU f32 path with LARQL_Q4K_DIRECT_ATTN unset) + an
MM/resident-capable StubEngine for failure arms; no mock-remote infra needed.
standard.rs 89.5->95.2%, generation.rs 87.3->92.3%; both baselines removed
(markov_residual kept). 755 kv tests pass. Production logic untouched.
quant/ggml/tq.rs (BitNet ternary TQ1_0/TQ2_0/I2_S) 89.33% -> 96.3% via direct f16 conversion edge-case tests (subnormal normalisation, inf/NaN encode+decode, overflow->inf, underflow->zero), TQ2_0 dequant with poisoned/subnormal stored scales, and the TQ2_0/TQ1_0 quantize length-guard arms. Baseline removed. Residual = the two pre-existing #[ignore]'d TQ1_0 round-trip tests (codec doesn't round-trip — needs a production fix, out of scope). Tests only.
routes/openai/completions.rs measured 86.9% locally but 70.3% on CI ubuntu (byte-identical code) — the async OpenAI streaming route tests don't deterministically cover their streaming paths under cargo's default parallel runner on the CI box. Serialise the server coverage invocation (Makefile + workflow), matching the metal + inference coverage jobs. No production change.
…ucture, lql updates Resolved crates/larql-inference/src/lib.rs: keep both fork's `edit` (RFC-0001 mechanistic-fact-editing) and upstream's new `decode_stages` module. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings
chrishayuk/larqlupstream/main into the fork (18 commits): MoE KV engines, CLI command restructure, lql routing/example updates, CI + coverage fixes.Resolved:
crates/larql-inference/src/lib.rs— kept both the fork'seditmodule (RFC-0001 mechanistic-fact-editing) and upstream's newdecode_stages.Verified:
cargo check --workspacepasses (exit 0). This branch is 0 commits behind upstream/main.origin/main—origin/mainhas 31 unique fork commits this branch lacks (fp8-block-quant Kimi-K2, MXFP4 streaming, the 2026-05-04 harvest checkpoint). A direct merge intoorigin/mainsurfaces ~287 conflicts (the known 423-commit divergence). Reconcile via thelarql-upstream-harvestreset-and-replay skill rather than a blind merge — this PR preserves the clean upstream content for that process.🤖 Generated with Claude Code