feat: v0.9.0 — SSD KV cache, prefix trie, telemetry, DFlash improvements#15
Open
szibis wants to merge 6 commits into
Open
feat: v0.9.0 — SSD KV cache, prefix trie, telemetry, DFlash improvements#15szibis wants to merge 6 commits into
szibis wants to merge 6 commits into
Conversation
SSD KV cache persistence: 2-tier hot/cold with safetensors serialization to disk. Evicts oldest entries to SSD when RAM threshold exceeded, restores on cache miss. Session-isolated via prompt prefix hash. Prefix caching: RadixAttention-style compressed trie for KV cache sharing across sessions. Longest-prefix matching with LRU eviction. Thread-safe. Hardware telemetry: IOReport-based sampling for GPU utilization, power draw, CPU/GPU temperature, ANE utilization. Background daemon thread with circular buffer history. New /telemetry endpoints in Python server and Rust proxy. Dashboard: power consumption chart, temperature chart (CPU/GPU with color coding), ANE utilization display. Polls /telemetry/current alongside existing status endpoints. DFlash acceptance improvements: soft threshold acceptance, hidden state normalization to bridge quantized/fp16 gap, proper top-K verification in cached path, block size auto-tuning based on measured acceptance rate. Expert prefetch from drafts: predicts MoE expert assignments from draft tokens via router gate weights, triggers async cache prefetch before target verification. DeepSeek V4 Flash: added mini preset (3-layer, 1024-hidden), auto preset selector based on available RAM, custom calibration prompts via --calibration-prompts. 1995 tests passing, 0 failures.
… telemetry integration Chat UI: fetch models from /v1/models/registry instead of hardcoded list, SSE streaming with token-by-token rendering, optgroup categories, 502/503 error states. Dashboard: 10-link nav bar, API endpoints card, inline power/temp/ANE in GPU card, worker session details.
… unsupported - Registry detects system RAM (sysctl HW_MEMSIZE) and filters dense models that won't fit; MoE models always shown (expert streaming) - Removed unsupported architectures (DeepSeek V4, gemma-4-e2b) from registry - Added 8 new supported models (Llama 1B/70B, Qwen3 4B/8B, Phi-4, gemma-4-12b, Mixtral) - Model switch shows real-time progress (downloading/loading/compiling phases) via /switch/progress polling endpoint - Chat UI: cached models marked with checkmark, MoE tag, category size hints - Python server: SO_REUSEADDR, switch progress tracking, phase updates during load
…orker
Rust proxy was wrapping Python's "data: {...}" lines in another Event::data(),
producing "data: data: {...}" which broke the chat UI's SSE parser.
Now passes raw byte stream through with correct headers.
…s nav - Token counter updates live during SSE streaming (counts content chunks) - Status bar shows tokens, tok/s, elapsed time after each message - Session saved on beforeunload (covers refresh, nav to dashboard, tab close) - loadSession validates data before restoring
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/telemetryendpoints in Python + Rust proxy.New files
mlx_flash_compress/ssd_kv_cache.py— SSDKVCache backendmlx_flash_compress/prefix_cache.py— PrefixCacheTrie + get_or_compute_prefixmlx_flash_compress/telemetry.py— HardwareTelemetry + TelemetrySamplemlx_flash_compress/draft_expert_prefetch.py— DraftExpertPrefetchertests/test_ssd_kv_cache.py— 17 teststests/test_prefix_cache.py— 23 teststests/test_telemetry.py— 18 teststests/test_dflash_acceptance.py— 21 teststests/test_draft_expert_prefetch.py— 9 testsTest plan