VRSecretary

🤖 AI-Powered VR Secretary – Production-ready reference implementation combining Unreal Engine 5 VR, local/cloud LLMs, and high-quality TTS for fully interactive virtual assistants in VR.

Features • Quick Start • Documentation • Architecture • Contributing

📑 Table of Contents

What Is VRSecretary?
Key Features
System Requirements
Quick Start Guide
Repository Structure
Backend Configuration
Unreal Plugin Setup
AI Persona: Ailey
Avatar Assets
Development & Testing
Architecture Overview
Troubleshooting
Documentation
Contributing
License
Acknowledgments

✨ What Is VRSecretary?

VRSecretary is a production-ready, open-source reference architecture for building AI-powered conversational characters in VR. It combines cutting-edge language models with spatial audio and immersive interaction to create a natural, engaging virtual assistant experience.

Core Technologies

Frontend: Unreal Engine 5 (VR-native, supports Meta Quest, PCVR, and more)
Backend: Python FastAPI (modular, extensible, engine-agnostic)
LLM: Flexible backend support:
- Local: Ollama (llama3, Mistral, Granite, etc.)
- Cloud: IBM watsonx.ai
- In-Engine: llama.cpp via Llama-Unreal plugin
TTS: Chatterbox (natural-sounding, sentence-aware streaming)

What's Included

✅ Complete Unreal Engine 5 VR project with sample secretary character
✅ Production-ready plugin (VRSecretary) with Blueprint support
✅ FastAPI gateway backend with clean REST API
✅ Sample 3D avatar (Scifi Girl v.01 - non-commercial demo)
✅ Comprehensive documentation and integration guides
✅ Multiple backend modes (Gateway, Direct Ollama, Local Llama.cpp)
✅ Docker support for easy deployment
✅ Load testing tools and performance benchmarks

🌟 Key Features

🥽 VR-Native Experience

Full 6DOF interaction with VR controllers
3D spatial audio that follows the character
Real-time subtitles floating above the avatar
Teleportation and smooth locomotion support
Hand presence and gesture recognition ready

🧠 Flexible AI Backends

Mode	Description	Use Case
Gateway (Ollama)	FastAPI → Ollama → TTS → UE	Local development, full features
Gateway (watsonx)	FastAPI → watsonx.ai → TTS → UE	Cloud deployment, enterprise
Direct Ollama	UE → OpenAI-style API → UE	Text-only, simple integration
Local Llama.cpp	UE → in-engine llama.cpp → UE	Offline, embedded deployment

🎙️ High-Quality Voice Synthesis

Sentence-aware streaming for natural pacing
Configurable voice parameters (temperature, speed, exaggeration)
Multiple voice profiles (female/male, extensible)
CUDA/MPS acceleration for real-time generation
EOS-safe chunking prevents audio artifacts

🔧 Production-Ready Architecture

Engine-agnostic REST API (Unity, Godot, Web can use same backend)
Session management with conversation history
Comprehensive error handling and logging
Configurable via environment variables and Project Settings
Dockerized deployment with compose files
Load testing scripts included

📦 Modular & Extensible

Clean separation of concerns (backend, plugin, assets)
Blueprint-friendly components for rapid prototyping
C++ API for advanced customization
Plugin system for adding new LLM/TTS providers
Multiple language support ready

💻 System Requirements

Minimum Requirements

Component	Specification
OS	Windows 10/11 (recommended), Linux/macOS (backend only)
CPU	Quad-core Intel/AMD, 2.5 GHz+
RAM	16 GB
GPU	NVIDIA GTX 1060 or equivalent (for VR)
Storage	30 GB free space (SSD recommended)
VR Headset	Meta Quest 2/3, Valve Index, or compatible PCVR

Recommended Requirements

Component	Specification
CPU	8-core Intel i7/AMD Ryzen 7, 3.5 GHz+
RAM	32 GB
GPU	NVIDIA RTX 3060 or better (CUDA for LLM/TTS acceleration)
Storage	50 GB free space on NVMe SSD

Software Dependencies

Required:

Unreal Engine 5.3 or later (with C++ source access)
Visual Studio 2022 (Windows) with "Game development with C++" workload
Python 3.10 or later
Git (for cloning repository)

Optional:

Docker Desktop (for containerized backend)
Node.js & k6 (for load testing)
CUDA Toolkit 11.8+ (for GPU acceleration)

🚀 Quick Start Guide

Get from zero to working VR secretary in ~30 minutes.

Step 1: Clone Repository

git clone https://github.com/ruslanmv/VRSecretary.git
cd VRSecretary

Step 2: Start Backend Services

2.1 Install Ollama

Download and install from https://ollama.ai/

# Start Ollama server
ollama serve

# In a new terminal, download a model
ollama pull llama3

Verify:

curl http://localhost:11434/v1/models

2.2 Setup Python Environment

cd backend/gateway

# Create virtual environment
python -m venv .venv

# Activate (Windows PowerShell)
.venv\Scripts\Activate.ps1

# Activate (Linux/macOS)
# source .venv/bin/activate

# Install dependencies
pip install -e .

2.3 Configure Environment

# Copy template
cp ../docker/env.example .env

# Edit .env with your preferred editor

Minimal .env configuration:

MODE=offline_local_ollama

OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=llama3
OLLAMA_TIMEOUT=60.0

CHATTERBOX_URL=http://localhost:4123
CHATTERBOX_TIMEOUT=30.0

SESSION_MAX_HISTORY=10

2.4 Start Gateway Server

uvicorn vrsecretary_gateway.main:app --reload --host 0.0.0.0 --port 8000

Verify backend is running:

API docs: http://localhost:8000/docs
Health check: http://localhost:8000/health

Step 3: Start TTS Server

In a new terminal (keep gateway running):

# From repo root
cd VRSecretary

# Activate same venv
backend/gateway/.venv/Scripts/Activate.ps1  # Windows
# source backend/gateway/.venv/bin/activate  # Linux/macOS

# Start Chatterbox TTS server
python tools/vr_chatterbox_server.py --host 0.0.0.0 --port 4123

Optional environment variables:

# Force CUDA (default: auto-detect)
export CHATTERBOX_DEVICE=cuda

# Chunk size for streaming (default: 20 words)
export CHATTERBOX_CHUNK_SIZE=20

# Custom voice files (optional)
export CHATTERBOX_FEMALE_VOICE=/path/to/female_voice.wav
export CHATTERBOX_MALE_VOICE=/path/to/male_voice.wav

Test TTS (PowerShell):

Invoke-WebRequest `
  -Uri "http://localhost:4123/v1/audio/speech" `
  -Method POST `
  -ContentType "application/json" `
  -Body '{"input": "Hello from Ailey.", "voice": "female"}' `
  -OutFile "test.wav"

Test TTS (curl):

curl -X POST http://localhost:4123/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Hello from Ailey.", "voice": "female"}' \
  --output test.wav

Play test.wav to verify audio quality.

Step 4: Verify End-to-End

With all three services running (Ollama, Gateway, TTS):

curl -X POST http://localhost:8000/api/vr_chat \
  -H "Content-Type: application/json" \
  -d '{"session_id": "test-123", "user_text": "Hello Ailey, introduce yourself."}'

Expected response:

{
  "assistant_text": "Hi! I'm Ailey, your VR secretary. I'm here to help...",
  "audio_wav_base64": "UklGRiQAAABXQVZF..."
}

✅ If you see text and base64 audio, the backend is working!

Step 5: Open Unreal Project

5.1 Generate Project Files

Navigate to:

VRSecretary/samples/unreal-vr-secretary-demo/

Right-click VRSecretaryDemo.uproject → Generate Visual Studio project files

5.2 Open in Unreal

Double-click VRSecretaryDemo.uproject

Unreal will:

Compile C++ modules (including VRSecretary plugin)
Load the project (first time may take 5-10 minutes)

5.3 Enable Plugin

Edit → Plugins
Search: VRSecretary
✅ Check Enabled (should already be checked)
Restart editor if prompted

5.4 Configure Plugin Settings

Edit → Project Settings
Navigate to: Plugins → VRSecretary
Set:
- Gateway URL: http://localhost:8000
- Backend Mode: Gateway (Ollama)
- HTTP Timeout: 60.0
- Language Code: (leave empty for English)

Step 6: Import Avatar

6.1 Locate Avatar File

VRSecretary/assets/avatars/scifi_girl_v01/scifi_girl_v.01.glb

6.2 Import to Unreal

In Content Browser, navigate to: /Game/Characters/ScifiGirl/
Right-click → Import to /Game/Characters/ScifiGirl/
Select scifi_girl_v.01.glb
Import Settings:
- ✅ Import Mesh
- ✅ Import Skeletal Mesh
- ✅ Import Materials
- ✅ Import Textures
Click Import All

6.3 Setup Blueprint

Open BP_SecretaryAvatar:

Components panel → Select Mesh component
Details panel → Mesh → Select imported skeletal mesh
Adjust Transform if needed (scale, rotation)
Compile and Save

Step 7: Test in VR

7.1 Connect VR Headset

Meta Quest: Enable developer mode, connect via Link/Air Link
PCVR: Connect via SteamVR or native runtime

7.2 Launch VR Preview

Click Play dropdown (top toolbar)
Select VR Preview
Put on headset

7.3 Interact with Ailey

Desktop Mode (testing without VR):

Press T key → Opens chat interface
Type message → Press Enter
See subtitle above character
Hear audio response

VR Mode:

Press Right Trigger → Opens 3D chat panel
Point laser at text field → Click trigger to focus
Use VR keyboard or voice input (if configured)
Press trigger on Send button

Expected behavior:

Your message appears in chat
Subtitle appears above Ailey's head
Audio plays from character position (3D spatial)
Response text appears in chat history

🗂️ Repository Structure

VRSecretary/
├── 📁 assets/
│   └── 📁 avatars/
│       └── 📁 scifi_girl_v01/
│           ├── scifi_girl_v.01.glb          # Sample avatar (CC BY-NC-SA)
│           ├── README.md                    # License & attribution
│           └── DOWNLOAD_INSTRUCTIONS.txt
│
├── 📁 backend/
│   ├── 📁 gateway/                          # FastAPI backend
│   │   ├── pyproject.toml                   # Python dependencies
│   │   ├── README.md                        # Backend docs
│   │   └── 📁 vrsecretary_gateway/
│   │       ├── main.py                      # FastAPI app entrypoint
│   │       ├── config.py                    # Settings (Pydantic)
│   │       ├── 📁 api/                      # REST endpoints
│   │       │   ├── health_router.py
│   │       │   └── vr_chat_router.py
│   │       ├── 📁 llm/                      # LLM clients
│   │       │   ├── base_client.py
│   │       │   ├── ollama_client.py
│   │       │   └── watsonx_client.py
│   │       ├── 📁 tts/                      # TTS integration
│   │       │   └── chatterbox_client.py
│   │       └── 📁 models/                   # Request/response schemas
│   │           ├── chat_models.py
│   │           └── session_store.py
│   └── 📁 docker/
│       ├── Dockerfile.gateway
│       ├── docker-compose.dev.yml
│       └── env.example
│
├── 📁 engine-plugins/
│   └── 📁 unreal/
│       └── 📁 VRSecretary/                  # Main Unreal plugin
│           ├── VRSecretary.uplugin
│           ├── README.md                    # Plugin documentation
│           ├── 📁 Source/VRSecretary/
│           │   ├── VRSecretary.Build.cs
│           │   ├── 📁 Public/
│           │   │   ├── VRSecretaryComponent.h       # Main component
│           │   │   ├── VRSecretarySettings.h        # Project settings
│           │   │   ├── VRSecretaryChatTypes.h       # Enums, structs
│           │   │   └── VRSecretaryLog.h
│           │   └── 📁 Private/
│           │       ├── VRSecretaryModule.cpp
│           │       ├── VRSecretaryComponent.cpp
│           │       ├── VRSecretarySettings.cpp
│           │       └── VRSecretaryLog.cpp
│           └── 📁 ThirdParty/
│               └── 📁 LlamaCpp/                     # Optional llama.cpp
│
├── 📁 samples/
│   ├── 📁 unreal-vr-secretary-demo/        # Example UE5 project
│   │   ├── VRSecretaryDemo.uproject
│   │   ├── 📁 Config/
│   │   │   └── DefaultEngine.ini
│   │   └── 📁 Content/
│   │       ├── 📁 Blueprints/               # BP_VRPawn, BP_DesktopPlayer
│   │       ├── 📁 UI/                       # WBP_ChatInterface
│   │       ├── 📁 Characters/               # BP_SecretaryAvatar
│   │       └── 📁 Maps/                     # VRSecretaryLevel
│   └── 📁 backend-notebooks/
│       └── prototype-calls.ipynb
│
├── 📁 docs/
│   ├── overview.md
│   ├── architecture.md
│   ├── unreal-integration.md                # Step-by-step UE setup
│   ├── engine-agnostic-api.md               # REST API reference
│   ├── persona-ailey.md                     # Customizing AI behavior
│   └── troubleshooting.md
│
├── 📁 tools/
│   ├── 📁 scripts/
│   │   ├── start_local_stack.sh
│   │   └── generate_openapi_client.sh
│   ├── 📁 perf/
│   │   ├── load_test_vr_chat.k6.js         # k6 load tests
│   │   └── profiling_notes.md
│   └── vr_chatterbox_server.py              # Optimized TTS server
│
├── 📁 .github/
│   ├── 📁 workflows/                        # CI/CD
│   │   ├── backend-tests.yml
│   │   └── unreal-build.yml
│   └── 📁 ISSUE_TEMPLATE/
│
├── LICENSE                                  # Apache 2.0
├── CODE_OF_CONDUCT.md
├── CONTRIBUTING.md
└── README.md                                # This file

⚙️ Backend Configuration

Environment Variables (`.env`)

Located at: backend/gateway/.env

Core Settings

# ===========================================
# CORE MODE
# ===========================================
# Options: offline_local_ollama | online_watsonx
MODE=offline_local_ollama

# ===========================================
# OLLAMA (Local LLM)
# ===========================================
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=llama3
OLLAMA_TIMEOUT=60.0
OLLAMA_TEMPERATURE=0.7
OLLAMA_MAX_TOKENS=256

# ===========================================
# CHATTERBOX (TTS)
# ===========================================
CHATTERBOX_URL=http://localhost:4123
CHATTERBOX_TIMEOUT=30.0

# ===========================================
# WATSONX.AI (Cloud LLM - Optional)
# ===========================================
# Uncomment if MODE=online_watsonx
# WATSONX_URL=https://us-south.ml.cloud.ibm.com
# WATSONX_PROJECT_ID=your-project-id-here
# WATSONX_MODEL_ID=ibm/granite-13b-chat-v2
# WATSONX_API_KEY=your-api-key-here
# WATSONX_TIMEOUT=60.0
# WATSONX_TEMPERATURE=0.7
# WATSONX_MAX_TOKENS=256

# ===========================================
# SESSION MANAGEMENT
# ===========================================
SESSION_MAX_HISTORY=10                      # Max conversation turns to remember

# ===========================================
# LOGGING
# ===========================================
LOG_LEVEL=INFO                              # DEBUG | INFO | WARNING | ERROR

Backend Modes Explained

Mode 1: Gateway (Ollama) - Recommended for Development

Flow: UE5 → FastAPI → Ollama → TTS → UE5

Pros:

✅ Full features (text + audio)
✅ Easy to debug
✅ Session history management
✅ Flexible persona configuration

Cons:

❌ Requires running 3 services
❌ Not suitable for embedded deployment

Setup:

MODE=offline_local_ollama
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=llama3
CHATTERBOX_URL=http://localhost:4123

Mode 2: Gateway (watsonx.ai) - Enterprise/Cloud

Flow: UE5 → FastAPI → watsonx.ai → TTS → UE5

Pros:

✅ Cloud-based (no local GPU needed)
✅ Enterprise-grade models
✅ Full features (text + audio)
✅ Scalable

Cons:

❌ Requires API key & internet
❌ Usage costs
❌ Latency depends on connection

Setup:

MODE=online_watsonx
WATSONX_URL=https://us-south.ml.cloud.ibm.com
WATSONX_PROJECT_ID=your-project-id
WATSONX_MODEL_ID=ibm/granite-13b-chat-v2
WATSONX_API_KEY=your-api-key
CHATTERBOX_URL=http://localhost:4123

Mode 3: Direct Ollama - Simplified Integration

Flow: UE5 → Ollama (OpenAI-compatible API) → UE5

Pros:

✅ Simplest setup (1 service)
✅ OpenAI-compatible API
✅ No gateway needed

Cons:

❌ Text only (no audio)
❌ No conversation history
❌ No persona customization

UE Settings:

Backend Mode: DirectOllama
Direct Ollama URL: http://localhost:11434
Direct Ollama Model: llama3

Mode 4: Local Llama.cpp - Fully Offline

Flow: UE5 → In-Engine llama.cpp → UE5

Pros:

✅ Completely offline
✅ No external services
✅ Embeddable
✅ Low latency

Cons:

❌ Text only (no audio by default)
❌ Requires GGUF model files
❌ More complex setup
❌ GPU/CPU intensive

UE Settings:

Backend Mode: LocalLlamaCpp
Requires: Llama-Unreal plugin enabled
Requires: ULlamaComponent on same actor

🎮 Unreal Plugin Setup

Plugin Installation

Method 1: Use Sample Project (Recommended)

Already included in samples/unreal-vr-secretary-demo/Plugins/VRSecretary/

Method 2: Add to Your Project

# From VRSecretary repo root
mkdir -p /path/to/YourProject/Plugins
cp -r engine-plugins/unreal/VRSecretary /path/to/YourProject/Plugins/

Then:

Right-click YourProject.uproject → Generate Visual Studio project files
Open in Visual Studio
Build Development Editor | Win64
Open project in Unreal

Project Settings

Edit → Project Settings → Plugins → VRSecretary

Setting	Description	Example
Gateway URL	FastAPI backend URL	`http://localhost:8000`
HTTP Timeout	Request timeout (seconds)	`60.0`
Backend Mode	Which backend to use	`Gateway (Ollama)`
Direct Ollama URL	For DirectOllama mode	`http://localhost:11434`
Direct Ollama Model	Model name	`llama3`
Language Code	ISO 639-1 code (empty = English)	`en` or empty

Blueprint Integration

1. Add Component to Actor

In your Player Pawn or VR Manager blueprint:

Components panel → + Add Component
Search: VRSecretary
Add VRSecretaryComponent

2. Bind Events

Event Graph:

Select VRSecretaryComponent
Details → Events:
- + On Assistant Response → Fires when AI replies
- + On Error → Fires on failure

3. Send User Text

Create this logic:

[User Input Event]
    ↓
[Make VRSecretaryChatConfig]
    - Temperature: 0.7
    - TopP: 0.9
    - MaxTokens: 256
    ↓
[VRSecretaryComponent → Send User Text]
    - User Text: (from input)
    - Config: (from Make Config)

4. Handle Response

On Assistant Response:

[Event OnAssistantResponse]
    - AssistantText (String)
    - AudioWavBase64 (String)
    ↓
[Update Subtitle Text] ← Display above avatar
    ↓
[Decode Base64 to Sound Wave]
    ↓
[Play Audio at Character Position]

On Error:

[Event OnError]
    - ErrorMessage (String)
    ↓
[Print String] ← Show to user
    - Text: ErrorMessage
    - Color: Red

Advanced: Per-Component Mode Override

On VRSecretaryComponent:

✅ Check: Override Backend Mode
Set: Backend Mode Override to desired mode

This overrides Project Settings for this specific component.

🧠 AI Persona: Ailey

Default Personality

Ailey is configured via the system prompt in:

backend/gateway/vrsecretary_gateway/api/vr_chat_router.py

Default traits:

Professional yet approachable - Executive assistant style
Concise - Optimized for spoken responses (60-120 words)
Proactive - Offers suggestions and next steps
VR-aware - References virtual office context
Organized - Good at structuring tasks and planning

Customizing Ailey

Edit the SYSTEM_PROMPT constant:

SYSTEM_PROMPT = """You are Ailey, a highly capable virtual secretary in a VR environment.

Your role:
- Assist the user with tasks, scheduling, and information
- Be professional yet warm and personable
- Keep responses concise (ideal: 2-3 sentences, max 120 words)
- Offer proactive suggestions when appropriate
- Reference the virtual office environment naturally

Tone: Friendly, efficient, and supportive.
"""

Persona Variants

See docs/persona-ailey.md for pre-made variants:

Variant	Use Case	Tone
Default	General assistant	Balanced, professional
Formal	Executive meetings	Corporate, reserved
Casual	Creative work	Relaxed, friendly
Technical	Dev/engineering	Precise, detail-oriented
Coach	Productivity/wellness	Motivational, supportive
Tutor	Learning/teaching	Patient, explanatory

Multi-Language Support

Set in plugin settings:

Language Code: es (Spanish), fr (French), de (German), etc.

Modify system prompt to match:

SYSTEM_PROMPT_ES = """Eres Ailey, una secretaria virtual altamente capaz..."""

🎭 Avatar Assets

Included Avatar: Scifi Girl v.01

Location: assets/avatars/scifi_girl_v01/scifi_girl_v.01.glb

Details:

Author: patrix
Source: Sketchfab
License: CC BY-NC-SA 4.0
Usage: Non-commercial only

⚠️ Important License Notes

You CANNOT use Scifi Girl v.01 in commercial products!

For commercial use, you must:

Replace with a commercial-licensed avatar:
- Unreal Marketplace
- Sketchfab (commercial license)
- Custom-commissioned model
- CC0 / Public Domain assets
Obtain permission from original creator (patrix)
Remove the sample avatar entirely

Using Custom Avatars

Requirements

Your avatar should have:

✅ Skeletal mesh with humanoid rig
✅ Materials/textures
✅ LODs (optional but recommended for VR)
✅ Compatible with UE5 skeleton

Import Process

Import to Unreal:
- Drag FBX/GLB into Content Browser
- Set import options (skeletal mesh, materials, textures)
Setup in BP_SecretaryAvatar:
- Open blueprint
- Select Mesh component
- Change Skeletal Mesh to your imported asset
Adjust components:
- VoiceAudio location (head height)
- SubtitleText location (above head)
Test in editor:
- Play → VR Preview
- Verify positioning and scale

🛠️ Development & Testing

Running Tests

Backend Unit Tests

cd backend/gateway
pytest

# With coverage
pytest --cov=vrsecretary_gateway --cov-report=html

# View coverage report
open htmlcov/index.html  # macOS/Linux
start htmlcov/index.html  # Windows

Load Testing

Requires k6:

cd tools/perf
k6 run load_test_vr_chat.k6.js

# With custom config
k6 run --vus 10 --duration 30s load_test_vr_chat.k6.js

Metrics to watch:

http_req_duration (p95 < 2s for good UX)
http_req_failed (should be 0%)
Concurrent users vs throughput

Docker Development

Build and Run

cd backend/docker
docker-compose -f docker-compose.dev.yml up --build

Services started:

Ollama → http://localhost:11434
Gateway → http://localhost:8000

Note: Chatterbox typically runs on host for GPU access.

Stopping

docker-compose -f docker-compose.dev.yml down

Code Quality

Python (Backend)

# Format code
black backend/gateway/

# Lint
flake8 backend/gateway/

# Type checking
mypy backend/gateway/

C++ (Unreal Plugin)

Follow Unreal Engine coding standards:

Naming: PascalCase for classes, bCamelCase for bools
Headers: Minimal includes, forward declarations
Blueprint Exposure: Use UPROPERTY and UFUNCTION macros

Debugging

Backend Logs

# Set debug level in .env
LOG_LEVEL=DEBUG

# Run gateway with verbose output
uvicorn vrsecretary_gateway.main:app --log-level debug

Unreal Logs

Output Log (Window → Developer Tools → Output Log):

Filter: LogVRSecretary
Shows HTTP requests, responses, errors

Blueprint Debugging:

Add Print String nodes
Use Breakpoints in Blueprint graph

🏗️ Architecture Overview

High-Level Flow

┌─────────────┐
│   Player    │
│  (VR/Desktop)│
└──────┬──────┘
       │ Speaks/Types
       ▼
┌─────────────────────────────────────────┐
│  VRSecretaryComponent (C++/Blueprint)   │
│  - Accepts user text                    │
│  - Routes to backend based on mode      │
│  - Fires events with response           │
└──────┬─────────────────────────┬────────┘
       │                         │
       │ Gateway Mode            │ Direct/Local Mode
       ▼                         ▼
┌─────────────┐          ┌──────────────┐
│  FastAPI    │          │   Ollama     │
│  Gateway    │          │   Direct     │
└──────┬──────┘          └──────┬───────┘
       │                        │
       ▼                        │
┌─────────────┐                 │
│   Ollama    │◄────────────────┘
│  or watsonx │
│    (LLM)    │
└──────┬──────┘
       │ Response Text
       ▼
┌─────────────┐
│ Chatterbox  │
│    (TTS)    │
└──────┬──────┘
       │ WAV Audio
       ▼
┌──────────────────────┐
│  VRSecretaryComponent│
│  - Decodes base64    │
│  - Plays sound wave  │
│  - Updates subtitle  │
└──────────────────────┘

Key Components

1. VRSecretaryComponent (C++)

Responsibilities:

Expose Blueprint-friendly API
HTTP request handling (via FHttpModule)
Backend mode routing
Event broadcasting
Error handling

Public API:

// Send user input to AI
UFUNCTION(BlueprintCallable)
void SendUserText(const FString& UserText, const FVRSecretaryChatConfig& Config);

// Delegates (events)
UPROPERTY(BlueprintAssignable)
FOnAssistantResponse OnAssistantResponse;

UPROPERTY(BlueprintAssignable)
FOnError OnError;

2. FastAPI Gateway

Endpoints:

GET /health - Health check
POST /api/vr_chat - Main chat endpoint

Request Schema:

{
  "session_id": "string",
  "user_text": "string",
  "language": "string (optional)",
  "temperature": 0.7,
  "top_p": 0.9,
  "max_tokens": 256
}

Response Schema:

{
  "assistant_text": "string",
  "audio_wav_base64": "string (optional)"
}

3. LLM Clients

Base Interface:

class BaseLLMClient(ABC):
    @abstractmethod
    async def generate(
        self,
        messages: List[Dict[str, str]],
        temperature: float,
        max_tokens: int
    ) -> str:
        pass

Implementations:

OllamaClient - Local Ollama
WatsonxClient - IBM watsonx.ai

4. TTS Server

Endpoints:

POST /v1/audio/speech - Non-streaming TTS
POST /v1/audio/speech/stream - Streaming TTS

Features:

Sentence-aware chunking
GPU acceleration
Configurable voice parameters
EOS-safe generation

Data Flow Examples

Example 1: Gateway Mode (Full)

User presses trigger in VR
VR keyboard input → "Schedule a meeting"
VRSecretaryComponent.SendUserText() called
HTTP POST to http://localhost:8000/api/vr_chat
Gateway assembles conversation history
Gateway calls Ollama: "You are Ailey..." + history + user text
Ollama responds: "I'd be happy to help schedule that..."
Gateway sends response to Chatterbox
Chatterbox generates WAV audio
Gateway returns: { assistant_text, audio_wav_base64 }
Component fires OnAssistantResponse event
Blueprint logic:
- Updates subtitle (TextRender component)
- Decodes base64 to USoundWave
- Plays audio at character location

Example 2: Direct Ollama Mode (Text Only)

User types: "Hello"
VRSecretaryComponent.SendUserText() called
HTTP POST to http://localhost:11434/v1/chat/completions (OpenAI-style)
Ollama responds with text only
Component fires OnAssistantResponse with empty audio
Blueprint displays text only (no audio playback)

🔧 Troubleshooting

Common Issues

1. "Failed to connect to backend"

Symptoms:

Red error in Unreal Output Log
OnError event fires
No response from AI

Solutions:

✅ Check backend is running:

curl http://localhost:8000/health

✅ Check Gateway URL in Project Settings:

Should be http://localhost:8000 (no trailing slash)

✅ Check firewall:

Windows: Allow Python through firewall
Antivirus: Whitelist backend ports (8000, 11434, 4123)

✅ Check Output Log:

Filter: LogVRSecretary
Look for HTTP error codes (404, 500, etc.)

2. "No audio in response"

Symptoms:

Text response works
audio_wav_base64 is null or empty

Solutions:

✅ Verify Chatterbox is running:

curl http://localhost:4123/health

✅ Check backend logs:

Look for TTS errors
Verify CHATTERBOX_URL in .env

✅ Test TTS directly:

curl -X POST http://localhost:4123/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"input": "Test", "voice": "female"}' \
  --output test.wav

✅ Check GPU/CUDA:

Chatterbox needs GPU for real-time TTS
CPU fallback is very slow

3. "Ollama model not found"

Symptoms:

Backend error: "Model llama3 not found"

Solutions:

✅ Download model:

ollama pull llama3

✅ List available models:

ollama list

✅ Update .env:

Set OLLAMA_MODEL to exact model name

4. "Compile errors in Unreal"

Symptoms:

"Cannot find VRSecretaryComponent.h"
Build fails in Visual Studio

Solutions:

✅ Regenerate project files:

Right-click .uproject → Generate Visual Studio project files

✅ Check plugin is enabled:

Edit → Plugins → VRSecretary → ✅ Enabled

✅ Clean build:

Visual Studio: Build → Clean Solution
Then: Build → Build Solution

✅ Verify Unreal version:

Project requires UE 5.3 or later

5. "Slow response times"

Symptoms:

10+ second delay
Audio takes minutes to generate

Solutions:

✅ Use GPU acceleration:

Ollama: Ensure CUDA is installed
Chatterbox: Set CHATTERBOX_DEVICE=cuda

✅ Use smaller models:

llama3:8b instead of llama3:70b

✅ Reduce max_tokens:

Lower to 128 or 200 for faster responses

✅ Check system resources:

Monitor GPU/CPU usage
Close other GPU-intensive apps

6. "Avatar not visible in VR"

Symptoms:

Chat works but no character
Empty space where avatar should be

Solutions:

✅ Check BP_SecretaryAvatar:

Mesh component has valid skeletal mesh assigned

✅ Check transform/scale:

Position: visible in VR space
Scale: 1.0 (or appropriate size)

✅ Check collision:

Collision preset allows visibility

✅ Check layer:

Not hidden by visibility settings

Performance Optimization

Backend

Slow LLM responses:

Use quantized models (Q4, Q5 instead of FP16)
Enable GPU inference
Reduce max_tokens

High memory usage:

Limit SESSION_MAX_HISTORY
Use smaller context window
Consider model with smaller params

Unreal

Low FPS in VR:

Reduce avatar poly count (use LODs)
Optimize materials (use simple shaders)
Disable unnecessary features (dynamic shadows)

Audio stuttering:

Increase audio buffer size
Pre-cache common responses
Use audio compression

📚 Documentation

Full Documentation

Document	Description
Architecture	Detailed system design, data flows
Unreal Integration	Step-by-step plugin setup guide
Engine-Agnostic API	REST API reference for any engine
Persona: Ailey	Customizing AI personality
Troubleshooting	Extended problem-solving guide

Quick Links

Backend README: backend/gateway/README.md
Plugin README: engine-plugins/unreal/VRSecretary/README.md
Sample Project: samples/unreal-vr-secretary-demo/README.md

🤝 Contributing

We welcome contributions of all kinds!

Ways to Contribute

🐛 Report bugs via GitHub Issues
💡 Suggest features or improvements
📖 Improve documentation
🧪 Add tests
🎨 Create new avatar assets (with compatible licenses)
🔌 Implement new LLM/TTS backends
🎮 Create Unity/Godot clients

Development Workflow

Fork the repository
Create a feature branch: git checkout -b feature/amazing-feature
Commit your changes: git commit -m 'Add amazing feature'
Push to branch: git push origin feature/amazing-feature
Open a Pull Request

Guidelines

Read CONTRIBUTING.md for detailed guidelines
Follow CODE_OF_CONDUCT.md
Write tests for new features
Update documentation
Keep commits atomic and well-described
Use conventional commits

Code Style

Python:

Format with Black
Lint with Flake8
Type hints with MyPy

C++:

Follow Unreal Engine Coding Standard
Use clang-format (UE5 style)

Blueprints:

Use clear node naming
Add comment blocks
Keep graphs organized (left → right flow)

📄 License

Code License

Apache License 2.0

✅ Commercial use allowed
✅ Modification allowed
✅ Distribution allowed
✅ Patent use allowed
⚠️ Requires: Attribution, License notice, State changes
⚠️ Does NOT grant: Trademark rights, Liability protection

See LICENSE for full text.

Asset License (Scifi Girl v.01 Avatar)

Creative Commons Attribution-NonCommercial-ShareAlike 4.0

✅ Non-commercial use only
✅ Adaptation allowed (with same license)
⚠️ Requires: Attribution to original creator
❌ Commercial use prohibited without permission

Original Creator: patrix on Sketchfab

Important: Code and assets have separate licenses. For commercial projects, you must replace non-commercial assets.

🙏 Acknowledgments

Technologies

Unreal Engine - Epic Games
Ollama - Local LLM runtime
Chatterbox - High-quality TTS
FastAPI - Modern Python web framework
IBM watsonx.ai - Enterprise AI platform

Assets

Scifi Girl v.01 by patrix - Used under CC BY-NC-SA 4.0

Community

Thanks to all contributors and testers
Special thanks to the Unreal Engine VR community
Thanks to the open-source AI/ML community

📞 Support & Contact

Getting Help

Documentation: Check docs/ folder
Issues: GitHub Issues
Discussions: GitHub Discussions

Reporting Issues

When reporting bugs, include:

OS and versions (Windows/Linux, Unreal version, Python version)
Backend configuration (.env settings, no API keys!)
Unreal project settings (Backend Mode, Gateway URL)
Steps to reproduce
Expected vs actual behavior
Relevant logs (VRSecretary, HTTP responses)

Feature Requests

For feature requests:

Describe the use case
Explain why it would be valuable
Provide mockups/examples if applicable
Tag with enhancement label

🗺️ Roadmap

Version 1.0 (Current)

Version 1.1 (Planned)

Version 2.0 (Future)

🌟 Show Your Support

If you find VRSecretary useful:

⭐ Star this repository
🐛 Report bugs to help improve it
📢 Share with your network
🤝 Contribute code or documentation
💬 Join discussions and provide feedback

Built with ❤️ for the VR and AI communities

⬆ Back to Top

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

VRSecretary

📑 Table of Contents

✨ What Is VRSecretary?

Core Technologies

What's Included

🌟 Key Features

🥽 VR-Native Experience

🧠 Flexible AI Backends

🎙️ High-Quality Voice Synthesis

🔧 Production-Ready Architecture

📦 Modular & Extensible

💻 System Requirements

Minimum Requirements

Recommended Requirements

Software Dependencies

🚀 Quick Start Guide

Step 1: Clone Repository

Step 2: Start Backend Services

2.1 Install Ollama

2.2 Setup Python Environment

2.3 Configure Environment

2.4 Start Gateway Server

Step 3: Start TTS Server

Step 4: Verify End-to-End

Step 5: Open Unreal Project

5.1 Generate Project Files

5.2 Open in Unreal

5.3 Enable Plugin

5.4 Configure Plugin Settings

Step 6: Import Avatar

6.1 Locate Avatar File

6.2 Import to Unreal

6.3 Setup Blueprint

Step 7: Test in VR

7.1 Connect VR Headset

7.2 Launch VR Preview

7.3 Interact with Ailey

🗂️ Repository Structure

⚙️ Backend Configuration

Environment Variables (.env)

Core Settings

Backend Modes Explained

Mode 1: Gateway (Ollama) - Recommended for Development

Mode 2: Gateway (watsonx.ai) - Enterprise/Cloud

Mode 3: Direct Ollama - Simplified Integration

Mode 4: Local Llama.cpp - Fully Offline

🎮 Unreal Plugin Setup

Plugin Installation

Method 1: Use Sample Project (Recommended)

Method 2: Add to Your Project

Project Settings

Blueprint Integration

1. Add Component to Actor

2. Bind Events

3. Send User Text

4. Handle Response

Advanced: Per-Component Mode Override

🧠 AI Persona: Ailey

Default Personality

Customizing Ailey

Persona Variants

Multi-Language Support

🎭 Avatar Assets

Included Avatar: Scifi Girl v.01

⚠️ Important License Notes

Using Custom Avatars

Requirements

Import Process

🛠️ Development & Testing

Running Tests

Backend Unit Tests

Load Testing

Docker Development

Build and Run

Environment Variables (`.env`)