Two agents walk into a negotiation. One speaks strategy. One speaks language. Neither blinks first.
A production-grade hierarchical negotiation system that separates what to do from what to say — using a Double DQN for intent selection and a fine-tuned Qwen-2.5 7B for dialogue generation. Trained via PPO self-play across 5,000 episodes. Deployed on Kubernetes with Ray-distributed simulation, vLLM inference, and Langfuse trace observability.
Most pricing negotiation systems are brittle rule trees that collapse the moment a counterpart behaves unexpectedly. They can't concede strategically, can't read negotiation phase, and can't generate natural language offers.
This system treats negotiation as a two-level decision problem:
- Level 1 (Strategy): When to anchor, concede, hold firm, explore, or close — learned via RL
- Level 2 (Dialogue): How to say it — generated by a fine-tuned LLM conditioned on the chosen intent
The separation buys interpretability, constraint enforcement, and clean training signals that an end-to-end LLM approach cannot provide.
| Metric | Baseline | This System | Delta |
|---|---|---|---|
| Deal Agreement Rate | 42% | 61% | +45% |
| Constraint Satisfaction | 71% | 93% | +31% |
| Episodes Evaluated | 1,000 | 5,000 | 5× |
| p95 Latency | 480ms | 148ms | −69% |
┌─────────────────────────────────────────────────────────┐
│ LangGraph Orchestration │
│ │
│ ┌──────────────────┐ ┌──────────────────────────┐ │
│ │ DQN Strategy │────▶│ Qwen-2.5 7B Dialogue │ │
│ │ Agent │ │ Agent (vLLM) │ │
│ │ │ │ │ │
│ │ Double DQN │ │ SFT + PPO self-play │ │
│ │ Dueling heads │ │ Intent-conditioned gen │ │
│ │ Prioritized ER │ │ Circuit breaker guard │ │
│ └──────────────────┘ └──────────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Pydantic Guardrails │ │
│ │ ZOPA bounds · Concession rate · Phase rules │ │
│ └──────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Langfuse Monitor + Circuit Breaker │ │
│ │ Per-span traces · Agreement scores · Alerts │ │
│ └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
│ │
▼ ▼
Ray Workers (K8s) vLLM Inference
Parallel episodes Low-latency LLM
Env state → [DQN] → intent (e.g. "concede_small")
│
▼
[Qwen-2.5 7B] → "I can meet you at $1,180 — that's my best offer at this stage."
│
▼
[Guardrails] → validates offer, concession rate, phase bounds
│
▼
[LangGraph] → update state, check terminal, loop or close
The DQN selects from 7 high-level intents each turn:
| Intent | Meaning |
|---|---|
open_anchor |
Set aggressive opening position |
concede_small |
Small concession to signal flexibility |
hold_firm |
No movement — signal strength |
explore_zopa |
Probe counterpart's reservation price |
make_final |
Signal best-and-final offer |
accept |
Accept current counterpart offer |
reject |
Walk away |
Every generated offer passes four validation stages before being applied:
- Structural — Pydantic field validation (positive offers, min < max)
- Concession rate — max 8% drop per round globally
- Absolute floor — never below 85% of reservation price
- Phase bounds — OPENING ≤10%, BARGAINING ≤6%, CLOSING ≤3%, FINAL ≤1%
Violations return a corrected offer, increment the constraint counter, and are scored in Langfuse.
# Install
pip install -r requirements.txt
# Copy env template and fill in keys
cp .env.example .env
# Run tests
pytest tests/ -v
# Train (PPO self-play → DQN bootstrap)
python scripts/train.py --config configs/default.yaml
# Simulate 500 episodes
python scripts/run_simulation.py --episodes 5001. PPO Self-Play (5,000 episodes)
Current policy vs. frozen snapshots from opponent pool
Snapshot interval: every 200 episodes (pool size: 5)
GAE advantage estimation (λ=0.95, γ=0.99)
PPO clip ε=0.2, entropy bonus ε=0.01
2. DQN Bootstrap
Policy head weights transferred from trained PPO network
DQN fine-tunes with prioritized experience replay (α=0.6)
Double DQN + dueling heads for stable Q-value estimation
3. Reward Signal (shaped)
+10.0 agreement reached
−5.0 per constraint violation
+0.3× concession rate efficiency
+0.2× rapport score
# Apply manifests (workers + HPA)
kubectl apply -f infrastructure/k8s-deployment.yaml -n negotiation-system
# Watch rollout
kubectl rollout status deployment/negotiation-worker -n negotiation-systemSet secrets before deploying:
kubectl create secret generic langfuse-keys \
--from-literal=LANGFUSE_PUBLIC_KEY=pk-lf-... \
--from-literal=LANGFUSE_SECRET_KEY=sk-lf-...Every push runs these gates — deployment blocked if any fail:
| Gate | Threshold | Production |
|---|---|---|
| Agreement rate | ≥ 55% | 61% |
| Constraint satisfaction | ≥ 90% | 93% |
| p95 latency | ≤ 200ms | 148ms |
| Unit + guardrail tests | all pass | — |
├── agents/
│ ├── negotiation_graph.py # LangGraph state machine + orchestration
│ ├── dqn_strategy.py # Double DQN + dueling heads + PER
│ ├── dialogue_agent.py # Qwen-2.5 7B via vLLM + circuit breaker
│ └── guardrails.py # Pydantic bidding rule enforcement
├── environment/
│ └── distributed_sim.py # Ray-based parallel simulation + TTL cache
├── training/
│ └── ppo_selfplay.py # PPO self-play trainer + opponent pool
├── monitoring/
│ └── langfuse_monitor.py # Langfuse traces + circuit breaker
├── infrastructure/
│ └── k8s-deployment.yaml # Kubernetes + HPA manifests
├── scripts/
│ ├── train.py # PPO → DQN training pipeline
│ └── run_simulation.py # Distributed episode runner
├── tests/
│ └── test_negotiation.py # Unit tests + regression gates
└── configs/
└── default.yaml # All hyperparameters
| Layer | Technology |
|---|---|
| Orchestration | LangGraph |
| RL Strategy | PyTorch — Double DQN, Dueling, PER |
| RL Alignment | PyTorch — PPO self-play, GAE |
| Dialogue | Qwen-2.5 7B Instruct via vLLM |
| Validation | Pydantic v2 |
| Distributed Sim | Ray 2.20+ on Kubernetes |
| Observability | Langfuse |
| Infra | AWS / Azure + K8s + HPA |