pet-assistant

An open-source blueprint for building ambient, embodied AI assistants on the Windows desktop. The reference implementation is a desktop pet, but the architecture is a reusable shell for any custom AI agent UI you want to ship - a conference mascot, a museum guide, a branded support companion, an always-on personal assistant.

The chat-box-in-a-browser-tab is not the only UI for AI. This project is a working, end-to-end starting point for the alternative: a small animated character that lives on the user's desktop, runs alongside whatever they're doing, and talks to any llm model.

The repo is structured so the LLM backend, the character art, the chat UI, and the IPC schema are each replaceable. Fork it, swap the pieces you want, and you have a custom AI assistant.

---

What you can build with this

• Conference / exhibition mascots : a branded animated character on a kiosk PC that visitors can ask about your booth, product, or company.
• Museum and event guides : embodied AI guides anchored on a screen, with the personality, voice, and look you choose.
• Customer-support companions : drop an ambient helper into your desktop app instead of opening a chat panel.
• Personal ambient assistants : your AI is one click away from any app, on any monitor, without alt-tabbing into a browser.
• Internal team mascots : point it at your internal LLM, give it your team's character, and you have a friendly entry point to your knowledge base.
• Livestream / virtual presenter companions : an on-screen agent that reacts and responds during streams or talks.

See docs/VISION.md for the longer scope and a fuller use-case gallery.

---

How it works

A 3-layer architecture, deliberately decoupled so each layer is replaceable:

flowchart LR
    User((User)) -->|click / type| Shell
    subgraph Shell["WPF Desktop Shell (.NET 8)"]
        Overlay[Win32 Layered Overlay<br/>character + speech bubble]
        Chat[WPF Chat Window]
    end
    Shell <-->|JSONL over stdin/stdout| Sidecar
    subgraph Sidecar["Python Sidecar (asyncio)"]
        Server[JSONL Server]
        Session[Session Manager]
    end
    Sidecar -->|HTTP / streaming| LLM[(OpenAI-compatible<br/>LLM endpoint)]

Loading

• The overlay uses Win32 layered windows for per-pixel-alpha sprite rendering - WPF cannot do this on transparent top-level windows, so the overlay runs on its own STA thread.
• The chat window is regular WPF, anchored to the pet character.
• The sidecar is a headless Python process that the desktop app auto-spawns. It speaks newline-delimited JSON over stdin/stdout and wraps LangChain's streaming ChatOpenAI. The host never makes direct HTTP calls.
• This isolation means the sidecar can be replaced with any process (Node, Go, Rust) that honours the IPC protocol.

---

Quickstart

Requires Windows, Python 3.11+, .NET 8 SDK.

# 1. Configure
Copy-Item config/.env.example config/.env
Copy-Item config/config.local.json.example config/config.local.json
# Edit config/.env and set OPENAI_API_KEY=...
# Edit config/config.local.json and set the model (e.g. "gpt-4o-mini")

# 2. Install Python sidecar dependencies
cd sidecar; python -m pip install -e .; cd ..

# 3. Build and run
.\build.ps1
dotnet run --project .\desktop\PetAssistant.Desktop\PetAssistant.Desktop.csproj

Left-click the pet to open the chat. Right-click for the menu. See docs/LOCAL_SETUP.md for the full setup and troubleshooting.

---

Project layout

Path	Purpose
desktop/PetAssistant.Desktop/	WPF shell - chat UI, IPC host, config, tray icon, lifecycle
desktop/PetAssistant.Overlay/	Win32 layered windows - sprite rendering, taskbar anchoring, click-through
sidecar/pet_sidecar/	Python async sidecar - JSONL server, LLM session lifecycle
assets/characters/	Sprite manifests and frames; placeholder fallback if missing
config/	Local config examples (.env, config.local.json) - gitignored
docs/	Vision, architecture, extension guide, IPC schema, asset pipeline

---

Want to fork it?

The docs are written specifically to make this easy:

• docs/VISION.md - the longer pitch: what problem this solves, what kinds of products you can build on it, the design principles behind the architecture.
• docs/ARCHITECTURE.md - the full system map: component responsibilities, threading model, where every concern lives.
• docs/EXTENDING.md - practical, copy-pasteable patterns for swapping the LLM, adding a new character, restyling the chat window, adding new sidecar capabilities, and more.
• docs/IPC_PROTOCOL.md - the JSONL contract between host and sidecar; the only thing you have to honour if you replace the sidecar.
• docs/ASSET_PIPELINE.md - sprite manifest format and how the loader resolves character assets.
• CONTRIBUTING.md - dev setup, code style, what kinds of contributions are most welcome.
• CLAUDE.md / AGENTS.md - orientation files for AI coding tools (Claude Code, Cursor, Aider, Codex) so they can be useful in this repo immediately.

---

Tech stack

• Desktop shell : WPF on .NET 8
• Overlay : Win32 UpdateLayeredWindow (per-pixel alpha) on a dedicated STA thread
• Sidecar : Python 3.11+ async, LangChain ChatOpenAI for streaming
• LLM : any LangChain-compatible endpoint (OpenAI, Azure OpenAI, OpenRouter, Ollama, vLLM, LM Studio, …)
• IPC : newline-delimited JSON over stdin/stdout

---

Current behavior

• Left-click the pet to open and close the chat.
• Right-click the pet for the command menu.
• Chat transcript is in-memory only for the current session.
• Tray icon is a fallback control surface, not the only way to quit.
• If sprite assets are missing, a placeholder character is rendered with GDI+.

---

Inspired by

For macOS user (similar alternative) check out work by ryanstephen -> lil-agents

---

License

MIT. See LICENSE.