Skip to content

README.md

Latest commit

 

History

History
75 lines (53 loc) · 3.14 KB

README.md

File metadata and controls

75 lines (53 loc) · 3.14 KB
Raw

Deepseek DSpark Tutorial : Make Your Local LLM 2X Faster with DeepSpec!

⚡ DSpark is a speculative decoding visualizer that demonstrates LLM inference acceleration using a fast draft model to speculate ahead, followed by parallel verification from a larger target model.

🛠️ Tech Stack

  • Dashboard & Engine: Streamlit (single-file implementation, <100 lines)
  • Target LLM: Ollama (serving the target qwen3:4b model)
  • Draft Model: Hugging Face dspark_qwen3_4b_block7 (local model.safetensors + config.json in the current folder)

🚀 Setup & Installation

1. Install Dependencies

Install the required python libraries:

pip install streamlit requests pandas safetensors torch

2. Download the Base Model (Ollama)

Download and start the target base model in Ollama:

ollama pull qwen3:4b

3. Download the Draft Model (Hugging Face)

Download the draft head weights (model.safetensors and config.json) from the official Hugging Face dspark_qwen3_4b_block7 Repo and place them directly in the root of this project folder.

4. Run the Project

Start the visual dashboard:

streamlit run app.py

📂 Project Architecture

  • app.py — Consolidated dashboard and speculative verifier. Loads local weights, runs Ollama inference, and generates visual charts.
  • sample_output.md — Pre-run benchmark report.
  • output.md — Dynamically updated benchmark report generated upon clicking run in the UI.

🎯 Use Cases

  1. Real-Time Code Completion: Accelerate IDE autocomplete suggestions by speculative drafting of syntax blocks.
  2. Low-Latency Conversational Agents: Deliver instant response generation for responsive local chat interfaces.
  3. Resource-Constrained Edge Devices: Run high-quality models on edge nodes by leveraging tiny draft models.
  4. Agentic Planning loops: Speed up recursive step-by-step reasoning calls.
  5. High-Throughput API Gateways: Optimize server-side inference load by validating batched speculations in parallel.

🔮 Future Features

  1. Native GGUF Draft Tensors: Support direct execution of multi-token prediction (MTP) draft heads.
  2. Dynamic Block-Size Scheduling: Adaptively scale draft length ($K$) based on token acceptance confidence.
  3. Tree-Search Speculative Drafting: Speculate multiple generation paths simultaneously.
  4. Hardware-Aware Truncation: Automatically trim draft sequences when system resources spike.
  5. Self-Correction Draft Heads: Train tiny, localized draft heads on specialized system logs.

Keywords: Speculative Decoding, DeepSeek, DSpark, DeepSpec, Ollama, Qwen3, Local LLM, PyTorch, Streamlit, Inference Acceleration, LLM Speedup, AI Speed Hack, Hugging Face, Safetensors