📺 Watch the full tutorial on YouTube

🚀 Tencent Hy3 Local Inference & Setup

A minimal, lightning-fast setup guide for running Tencent Hy3 (295B Mixture-of-Experts) locally. Hy3 rivals flagship models and delivers top-tier performance for reasoning, coding, and long-context workflows.

---

🛠️ Installation

Set up the environment using vLLM for optimized inference.

# 1. Create a virtual environment
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate

# 2. Clone vLLM repository
git clone https://github.com/vllm-project/vllm.git
cd vllm

# 3. Install dependencies
uv pip install --editable . --torch-backend=auto

🚀 Run the Server

Start the vLLM server with Multi-Token Prediction (MTP) enabled for maximum efficiency (requires 8 GPUs):

export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

vllm serve tencent/Hy3 \
  --tensor-parallel-size 8 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 2 \
  --tool-call-parser hy_v3 \
  --reasoning-parser hy_v3 \
  --enable-auto-tool-choice \
  --port 8000 \
  --served-model-name hy3

🧪 Test the Model

Verify your deployment using the standard OpenAI Python client.

# Install the client
pip install openai

from openai import OpenAI

# Connect to local vLLM server
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")

# Test completion
response = client.chat.completions.create(
    model="hy3",
    messages=[
        {"role": "user", "content": "Write a fast Python script to parse JSON."},
    ],
    temperature=0.9,
    top_p=1.0,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)

print(response.choices[0].message.content)

---

💡 5 Key Use Cases

1. 💻 Agentic Software Development: Seamlessly integrates into frameworks like CodeBuddy or Cline for automated PR generation and bug fixing.
2. 📚 Deep Document Analysis: Digest massive PDFs and enterprise archives instantly leveraging the 256K context window.
3. 📊 Financial Data Extraction: Precisely pull tabular data and metrics without hallucination or conflating sources.
4. 🧠 Complex Reasoning Chains: Use "high" reasoning effort for PhD-level mathematics, logic puzzles, and structural engineering tasks.
5. 💬 Multi-turn Roleplay & Assistance: Retain constraints, intent, and coreference beautifully over extremely long conversational horizons.

---

🔮 5 Future Features

1. ⚡ Native GGUF & Mac Silicon Support: Optimized 4-bit/8-bit quants for local Macbook Pro environments.
2. 👁️ Vision & Multimodal Integration: Image processing capabilities for visual document Q&A and UI-to-code pipelines.
3. 🔌 Plug-and-Play Streamlit Interface: A 1-click web GUI for non-developers to chat and build agents.
4. 🔗 Direct RAG Connectors: Built-in endpoints for instant vector database chunking and retrieval.
5. 📱 Edge-Optimized Distillation: Smaller 8B/14B sibling models targeting mobile device deployment.

---

Built for the next generation of AI agents. Open-sourced by Tencent Hy Team.

---

Keywords: Tencent Hy3, 295B MoE, AI, Open Source LLM, vLLM, Large Language Model, Local AI inference, Machine Learning, Generative AI, Tencent Cloud