Lucy

VM Automation | AI Orchestrator
Tell Lucy what to do. She'll open apps, click buttons, fill forms, and get it done — inside a sandboxed Ubuntu VM.

---

_{Click to watch the demo video}

---

What is Lucy?

Lucy is a local, self-hosted AI desktop automation platform. You describe a task in plain language — Lucy executes it visually in an Ubuntu VM, controlling the mouse, keyboard, and terminal just like a human would.

Think of it as your own Manus AI, running entirely on your machine.

"Open LibreOffice, write a cover letter for Acme Corp, and save it as PDF on the Desktop"

Lucy will open Writer, type the letter, navigate to File > Export as PDF, name the file, and confirm — all while you watch in real-time.

---

Key Features

Feature	Description
Natural Language Tasks	Describe what you want in plain text — Lucy figures out the clicks and keystrokes
Live VM Screen	Watch the AI work in real-time with ~1 FPS screenshot streaming
Multi-Agent	Run multiple independent agents in parallel, each with their own VM
Recipes	Every task is recorded as a YAML playbook — replay, edit, export, import, and share
Scheduled Execution	Run recipes on a schedule — daily, weekly, or at fixed intervals
Learning System	Teach Lucy from her mistakes — lessons persist and improve future performance
Per-Recipe Instructions	Attach specific hints to recipes for customized replay
Dark & Light Theme	Switch between themes with one click
Multilingual	Full German and English UI
Per-Agent Settings	Different API keys and models per agent — mix Anthropic with local AI

---

Architecture

┌──────────────────────────────────────────────────────┐
│  Electron Host App (React + TypeScript)              │
│  AgentManager · Scheduler · DockerManager            │
└───────────────────────┬──────────────────────────────┘
                        │  WebSocket (per agent)
┌───────────────────────▼──────────────────────────────┐
│  AI Agent (Python 3.12)                              │
│  Screenshot → Claude API → xdotool/bash → repeat     │
└───────────────────────┬──────────────────────────────┘
                        │
┌───────────────────────▼──────────────────────────────┐
│  Ubuntu 24.04 VM (Docker)                            │
│  XFCE · TigerVNC · LibreOffice · Firefox · ...       │
└──────────────────────────────────────────────────────┘

Each agent runs in its own Docker container with isolated ports. The host app manages multiple agents simultaneously.

---

Quick Start

Prerequisites

• Docker Desktop (running)
• Node.js 22+ (for building from source)
• An Anthropic API key or Custom Local Vision LLM Model

Option A: Build from Source

# Clone
git clone https://github.com/Raspiux/Lucyapp.git && cd lucy

# Build Docker VM image (one-time, ~3 min)
docker build -t lucy-vm:latest ./vm

# Install dependencies & start
cd host
npm install
npm run dev

First Steps

1. Click "+ New Agent" in the sidebar
2. Enter a name, your API key, and pick a model
3. Wait ~10 seconds for the container to start and auto-connect
4. Type a task and hit Cmd+Enter — watch Lucy work!

---

Example Tasks

Open Firefox and navigate to heise.de

Create a text file on the Desktop with today's date and a list of 5 random facts

Open LibreOffice Calc, create a table with columns Name/Age/City,
add 5 sample rows, and save as CSV

Install htop, then take a screenshot showing the system load

---

Recipes & Scheduling

Every completed task is saved as a recipe (YAML playbook). You can:

• Replay a recipe with one click (intelligently adapts to current screen state)
• Edit the description and add per-recipe instructions
• Export as .yaml and share with others
• Import recipes from colleagues
• Schedule recipes to run daily, weekly, or at custom intervals

---

Learning from Mistakes

Lucy has a lessons system that persists user feedback:

• After each task, a feedback dialog asks: "What could Lucy do better next time?"
• Add lessons anytime from the Live tab via the "+ Lesson" button
• Lessons are injected into Claude's system prompt — the agent genuinely improves

Example lesson: "Don't use snap in Docker containers — always use apt install instead"

---

Multi-Agent

Run multiple agents in parallel — each with its own VM, settings, and task history:

• Agent 1: News scraping every 30 minutes
• Agent 2: Daily report generation at 9:00 AM
• Agent 3: Ad-hoc tasks you trigger manually

Each agent can use a different API key or model.

---

Tech Stack

Component	Technology
Host App	Electron 41, React 18, TypeScript
Agent	Python 3.12, asyncio, Anthropic SDK
AI Model	Claude (computer-use-2025-01-24 beta)
VM	Docker, Ubuntu 24.04, XFCE, TigerVNC
GUI Control	xdotool, xclip, scrot, wmctrl
Communication	WebSocket (bidirectional, real-time)
Recipes	YAML playbooks
Scheduling	Built-in interval ticker (no cron dependency)

---

Documentation

Document	Description
Architecture	System design, data flow, technology decisions
User Guide	How to use every feature
Build Guide	Prerequisites, building, packaging, deployment
API Reference	WebSocket protocol, IPC channels, Python API
WebSocket Protocol	All message types with examples

---

Contributing

# Development (hot reload)
cd host && npm run dev

# Agent lint + type check
cd agent && source .venv/bin/activate
ruff check src/ && mypy --strict src/

# Host type check
cd host
npx tsc --project src/main/tsconfig.json --noEmit
npx tsc --project src/renderer/tsconfig.json --noEmit

# Build DMG
cd host && npm run package

---

_{Lucy v0.1.0}