This file explains what we built, how each piece connects, why we made certain choices, and every bug we hit along the way. Read this whenever you're confused.
deployKit is a mini version of platforms like Vercel or Railway. You give it a GitHub link. It downloads the code, turns it into a running app, and gives you a URL to visit that app. Everything runs on your own machine.
You (developer) → run "docker compose up" → deployKit is now live
User → pastes a GitHub link into deployKit → pipeline runs → app is live at a URL
This is the most important section. Read this first. Everything else is detail.
Mode 1 — The Platform Itself (you running it)
This is when you run docker compose up --build. You're starting deployKit.
Four containers come alive:
Your Machine
│
├── deploykit-caddy-1 (the front door — port 80)
│ listens on port 80 and decides where to send each request
│
├── deploykit-frontend-1 (the website — React UI)
│ when you visit localhost, Caddy sends you here
│
├── deploykit-backend-1 (the brain — Hono API + pipeline)
│ when you submit a GitHub URL, this does all the work
│
└── deploykit-buildkit-1 (the builder's engine)
Railpack needs this to build Docker images
They all join a private network called deploykit_app so they can talk to each other by name.
Mode 2 — A User Deploying an App (pipeline running) This is what happens AFTER you submit a GitHub URL in the UI. The pipeline runs inside the backend container. Here's every step:
Browser
│
│ POST /deployments { gitUrl: "https://github.com/..." }
▼
Caddy (port 80)
│ sees the request starts with /deployments → sends it to backend
▼
Backend (port 3001)
│ creates a DB row with status="pending", returns the ID immediately
│ fires off the pipeline in the background (doesn't wait for it)
▼
pipeline.ts — Step 1: CLONE
│ git clone <gitUrl> /tmp/deploy-<id>
│ status → "cloning"
▼
pipeline.ts — Step 2: BUILD
│ does the repo have a Dockerfile?
│
├── YES → docker build -t deploy-<id>:latest /tmp/deploy-<id>
│ (Docker CLI inside the backend talks to Docker on the host)
│
└── NO → railpack build /tmp/deploy-<id> --name deploy-<id>:latest
(Railpack auto-detects framework, tells BuildKit to build)
(BuildKit is a separate container on the same network)
│
│ status → "building"
▼
pipeline.ts — Step 3: START CONTAINER
│ pick a free port from 4000–4099, save it to DB
│ dockerode.createContainer({ name: "deploy-<id>", network: "deploykit_app" })
│ dockerode.start()
│ the new container joins the deploykit_app network — Caddy can now reach it
│ status → "starting"
▼
pipeline.ts — Step 4: TELL CADDY
│ PUT http://caddy:2019/config/apps/http/servers/deploy-<id>
│ tells Caddy: "listen on port 4001, proxy to deploy-<id>:3000"
│ Caddy adds this route LIVE — no restart needed
│ status → "running", url → "http://localhost:4001" saved to DB
▼
Browser (polling every 2s)
│ GET /deployments → sees status changed to "running" → badge turns green
│ the "open ↗" link appears → click it → goes to http://localhost:4001
▼
Caddy (port 4001)
│ forwards to deploy-<id>:3000 (the user's app container)
▼
User's App Container
serves the actual app
┌─────────────────────────────────┐
│ deploykit_app network │
│ │
Browser ──port 80──▶ Caddy ──────▶ frontend:80 │
│ ──────▶ backend:3001 │
│ ──────▶ deploy-abc123:3000 │
│ │ │
│ backend:3001 │
│ │ │ │
│ │ └──▶ caddy:2019 │ (admin API)
│ │ │
│ └──▶ buildkit:1234 │ (builds images)
│ │
Host Docker ◀─── docker.sock (mounted into backend) │
└─────────────────────────────────┘
Ports visible to YOU (outside Docker):
:80 → the deployKit UI
:2019 → Caddy admin API (only for debugging)
:4000–4099 → deployed apps (each app gets its own port)
User types GitHub URL
│
▼
DB: deployments table
(id, git_url, status, port, url, image_tag)
│ │
│ │
pipeline.ts SSE endpoint
writes status reads status
and log lines and log lines
│ │
▼ ▼
DB: logs table Browser
(deployment_id, EventSource
message) shows lines
in real time
The database is the "single source of truth" between the pipeline and the browser. The pipeline writes. The browser reads. They never talk directly — the DB is in the middle.
Normal Docker (docker build) is baked into Docker itself.
Railpack uses BuildKit which is a separate, more advanced build engine.
BuildKit has to run as its own process.
We run it as a container named buildkit on the same network.
The backend knows where to find it because of this env var:
BUILDKIT_HOST=tcp://buildkit:1234
backend container
│ runs: railpack build ...
│ Railpack reads $BUILDKIT_HOST
▼
buildkit container (port 1234)
│ does the actual image build
│ pulls base images from Docker Hub
│ runs install commands
│ produces a Docker image
▼
Docker (on the host machine)
receives the built image and stores it
Your Machine (host)
│
│ /var/run/docker.sock ← Docker's control channel
│ │
│ │ (mounted into backend container)
│ ▼
│ backend container
│ uses dockerode to:
│ - create containers (for deployed apps)
│ - start containers
│ - (future: stop/remove containers)
│
│ The backend container is inside Docker
│ but can still CONTROL Docker on the outside.
│ This is called "Docker-out-of-Docker" (DooD).
What it is: A tool that packages an app and everything it needs into a "container." A container is like a sealed box — the app runs the same way no matter what machine it's on.
Why we use it: We need to run user-submitted apps safely and in isolation. We don't want someone's Node 14 app messing with our Node 22 backend. Each deployed app lives in its own container.
What it is: A tool that manages multiple containers at once using one config file (docker-compose.yml).
Why we use it: deployKit itself has three parts (backend, frontend, Caddy). Without Compose, you'd have to start each one manually and wire them together yourself. Compose handles all of that with one command.
When you run it:
docker compose up --build= you run this ONCE to start the deployKit platform--build= rebuild the images first (needed when you change backend/frontend code)- After that, the platform sits there waiting for GitHub URLs
- You only re-run it when YOU change something in the code
What it is: A tool that looks at someone's source code, figures out what kind of app it is (Node.js, Python, etc.), and automatically builds a Docker image from it. You don't need to write a Dockerfile yourself — Railpack does that work.
Why we use it: Users submit random GitHub repos. We can't know in advance what framework they're using. Railpack auto-detects it so we can build any repo.
When it runs: NOT during docker compose up. It runs INSIDE the pipeline,
after a user submits a GitHub URL. Specifically in step 2 of the pipeline.
docker compose up --build → builds OUR images (backend, frontend)
Railpack is NOT involved here
User submits GitHub URL → pipeline.ts runs:
step 1 → git clone
step 2 → railpack build ← THIS is when Railpack runs
step 3 → docker run
step 4 → caddy route
What it is: A web framework for Node.js. Like Express but faster and more modern.
Why we use it: We need a simple HTTP server with route handling. Hono has a clean API and works well with TypeScript.
What it is: A lightweight database that stores everything in a single file. No separate database server needed.
Why we use it: We need to store deployment records and logs.
SQLite is simple, fast, and perfect for single-machine apps like this.
The data lives at /data/deploykit.db inside the backend container.
That /data folder is a Docker volume — it survives container restarts.
What it is: A web server and reverse proxy. It sits in front of everything and decides where to send each incoming request.
Why we use it: Two reasons:
- Route
/deployments*to the backend and everything else to the frontend - Dynamically add new routes for deployed apps via its Admin API (no restart needed)
Admin API: Caddy has a secret control panel at port 2019. We send it HTTP requests to add new routes on the fly. When a user's app is ready, we tell Caddy "port 4001 → that container."
What it is: An npm package that lets Node.js talk to Docker.
Why we use it: The pipeline needs to create and start Docker containers (for the user's deployed apps). Dockerode does this through the Docker socket that we mount into the backend container.
What it is: Vite is the build tool. React is the UI framework.
Why we use it: We need a frontend where users can submit URLs and watch deploy status. Vite builds it fast. In production (inside Docker), Vite is gone — Nginx just serves the pre-built files.
What it is: A web server that serves static files.
Why we use it: After Vite builds the React app into HTML/CSS/JS files, Nginx serves those files to the browser. It's fast and simple for static content.
In docker-compose.yml you see this on the backend:
volumes:
- /var/run/docker.sock:/var/run/docker.sock/var/run/docker.sock is a file on your machine that Docker listens on.
Think of it as Docker's intercom. Any program that can talk to this file can control Docker.
By mounting it into the backend container, we're saying: "Hey backend, you can use this intercom to talk to Docker and create containers."
This is what lets the pipeline create and start user app containers from inside our backend.
When Compose starts, it creates a private network called deploykit_app.
All containers join this network. They can reach each other by name.
backend → can call "http://caddy:2019" (Caddy's admin API)
caddy → can call "http://backend:3001" (our API)
caddy → can call "http://frontend:80" (the React UI)
caddy → can call "http://deploy-abc123:3000" (a user's deployed app)
The name: deploykit in docker-compose.yml is what makes the network
always called deploykit_app. Without it, Docker guesses a name based on
your folder name and it can change unexpectedly.
deployments — one row per deploy attempt
id → unique ID, e.g. "abc-123"
git_url → the GitHub URL the user submitted
status → pending → cloning → building → starting → running (or failed)
image_tag → e.g. "deploy-abc-123:latest" (the Docker image Railpack built)
url → e.g. "http://localhost:4001" (where the app is live)
port → 4001 (which port Caddy uses for this app)
created_at → timestamp
logs — one row per log line
id → auto number
deployment_id → which deployment this belongs to
message → e.g. "Cloning https://github.com/..."
timestamp → when it happened
Goal: Get the skeleton running. No real deployment logic yet — just the structure.
What we built:
- The folder structure:
backend/,frontend/,caddy/ docker-compose.ymlwiring all three together- SQLite database with the two tables above
- Three API endpoints (stub — they work but don't deploy anything):
GET /deployments→ list all deploymentsGET /deployments/:id→ get one deploymentPOST /deployments→ create a record (status stays "pending", nothing runs)
How to test Phase 1 is working:
docker compose up --build
# visit http://localhost → see the frontend placeholder
# call POST http://localhost/deployments with { "gitUrl": "..." }
# see the record created in GET /deploymentsStatus at end of Phase 1: Platform starts, API works, nothing actually deploys.
Goal: When a user submits a GitHub URL, actually clone it, build it, run it, and give them a live URL.
What we added:
backend/src/pipeline.ts— the core pipeline (clone → build → run → route)GET /deployments/:id/logs— new endpoint to read build logs- Git, Docker CLI, and Railpack installed inside the backend container
- Port range
4000-4099exposed through Caddy for deployed apps
The pipeline — what happens after POST /deployments:
1. status = "cloning"
git clone <gitUrl> /tmp/deploy-<id>
2. status = "building"
railpack build /tmp/deploy-<id> --name deploy-<id>:latest
(Railpack detects the framework and builds a Docker image)
3. status = "starting"
pick a free port from 4000–4099 (stored in DB so two deploys don't clash)
create container named "deploy-<id>" on the deploykit_app network
start the container (PORT=3000 env var tells the app where to listen)
4. tell Caddy (via Admin API on port 2019):
"create a new server block on port <chosen_port>,
proxy all traffic to deploy-<id>:3000"
5. status = "running"
url = "http://localhost:<port>" saved to DB
Why fire-and-forget (background)? Building a Docker image takes 30–120 seconds. If we made the HTTP response wait for it, the browser would time out. Instead, POST returns the deployment ID immediately. The pipeline runs in the background. The frontend polls for status.
Why a port per deployment (not a URL path)?
If we used /app/abc-123/ as the URL for the deployed app, most frameworks
would break — they'd generate links like /static/main.js instead of /app/abc-123/static/main.js.
A dedicated port means the app thinks it's at / which is what it expects.
Files changed in Phase 2:
backend/src/pipeline.ts(new)backend/src/routes/deployments.ts(added pipeline call + logs endpoint)backend/Dockerfile(added git, docker-cli, railpack)docker-compose.yml(addedname: deploykitand port range)backend/.dockerignore(new — fixes Bug 2 below)frontend/.dockerignore(new — clean practice)
What happened:
Dockerfile tried curl https://railpack.io/install.sh | sh but the domain doesn't resolve.
The image build would fail trying to install Railpack.
Root cause: We assumed a .io website existed. It doesn't. Railpack is GitHub-only.
Fix: Download the binary directly from GitHub releases:
RUN curl -fsSL "https://github.com/railwayapp/railpack/releases/download/v0.23.0/railpack-v0.23.0-x86_64-unknown-linux-musl.tar.gz" \
| tar -xz -C /usr/local/bin railpackThe musl build is fully static — works on any Linux with no extra dependencies.
Rule for next time:
Before adding any curl | sh to a Dockerfile, verify the URL first:
curl -o /dev/null -w "%{http_code}" <url>If it returns 000 or 404, find the GitHub releases page instead.
What happened: Backend crashed immediately on startup:
Error: The module 'better-sqlite3/build/Release/better_sqlite3.node'
was compiled against a different Node.js version using NODE_MODULE_VERSION 137.
This version of Node.js requires NODE_MODULE_VERSION 127.
Root cause:
better-sqlite3 is a "native module" — it contains C++ code compiled for a specific Node version.
The Dockerfile does RUN npm install (correct, compiles for Node 22 inside the container).
But then COPY . . overwrites those node_modules with the ones from your laptop
(compiled for your laptop's newer Node version). Container needs version 127, got 137.
Fix: Add backend/.dockerignore:
node_modules
Now COPY . . skips the local node_modules. The container keeps its own freshly compiled ones.
Rule for next time:
Any package with a binding.gyp file or a build/Release/*.node file is a native module.
These must always be compiled inside the target container, never copied from your machine.
Common ones: better-sqlite3, bcrypt, sharp, canvas.
Always add node_modules to .dockerignore for any backend with native modules.
What happened:
permission denied while trying to connect to the Docker API at unix:///var/run/docker.sock
Root cause:
On Linux, the Docker socket is only accessible by the docker group by default.
The current user wasn't in that group.
Fix:
sudo usermod -aG docker $USER # add yourself to the docker group
newgrp docker # apply without logging outRule for next time:
On any fresh Linux machine, run docker ps first.
If it says permission denied, run the two commands above before anything else.
Goal: Push log lines to the browser in real time as the pipeline runs. No more manually calling the logs endpoint — the browser just listens.
SSE stands for Server-Sent Events. It's a way for the server to push data to the browser over a normal HTTP connection that stays open.
Think of it like a phone call:
- Normal HTTP = you call the restaurant, ask "is my order ready?", they answer, call ends. You have to call again later.
- SSE = you call the restaurant and stay on hold. They tell you updates as they happen. You hang up when the food arrives.
SSE is simpler than WebSockets because data only flows one way — server to browser.
We don't need the browser to send anything back during the log stream.
The browser has EventSource built in — no library needed.
Browser opens: GET /deployments/:id/logs/stream
↑ connection stays open
Server loop (every 500ms):
→ check DB for new log rows (WHERE id > lastLogId)
→ for each new row: push it to the browser as data: <message>\n\n
→ check deployment status
→ if status = "running" or "failed": send "done" event, close connection
Browser side:
const es = new EventSource('/deployments/<id>/logs/stream')
es.onmessage = (e) => addLineToScreen(e.data)
es.addEventListener('done', () => es.close())
The SSE data format is just plain text — every message looks like:
data: Cloning https://github.com/...\n\n
The double newline \n\n signals the end of one message.
Hono's streamSSE helper handles that formatting for you.
The pipeline runs in the background (fire-and-forget). The SSE stream runs in a different request. They're not the same function call — they can't directly talk to each other.
The DB is the shared source of truth. The pipeline writes logs to it. The SSE stream reads from it every 500ms. Simple and reliable. If we used an in-memory queue instead, it would break if the backend restarted.
Backend — new endpoint in deployments.ts:
GET /deployments/:id/logs/stream
- Uses
streamSSEfromhono/streaming - Polls DB every 500ms for new log rows (tracks last seen row ID)
- Sends each new line as an SSE message event
- Sends a
doneevent when status isrunningorfailed, then closes
Frontend — full UI replacing the placeholder (App.tsx):
- Top bar: text input for GitHub URL + Deploy button
- Left panel: list of all deployments with live status badges (polls every 2s)
- Right panel: terminal-style log viewer that opens an EventSource stream when you click a deployment
- Auto-scrolls to the bottom as new lines arrive
- Shows a yellow "● live" indicator while the stream is open
- When the deployment finishes, shows the live URL with an "open ↗" link
backend/src/routes/deployments.ts(added SSE endpoint)frontend/src/App.tsx(full rewrite — real UI)frontend/src/index.css(reset to a clean dark theme base)
What happened: The log panel showed:
Pipeline failed: Command failed: railpack build /tmp/deploy-...
No actual reason. Just "command failed." We had no idea what went wrong.
Root cause:
When execAsync throws because a command exits with an error,
the real output (what the command printed) is on err.stdout and err.stderr,
not on err.message. Our catch block only logged err.message.
So the real reason was always hidden.
Fix:
Created a run() helper in pipeline.ts that captures both stdout and stderr
and logs them to the DB on success AND on failure, before re-throwing:
async function run(id: string, cmd: string) {
try {
const { stdout, stderr } = await execAsync(cmd)
if (stdout.trim()) addLog(id, stdout.trim())
if (stderr.trim()) addLog(id, stderr.trim())
} catch (err: any) {
if (err.stdout?.trim()) addLog(id, err.stdout.trim())
if (err.stderr?.trim()) addLog(id, err.stderr.trim())
throw err
}
}Rule for next time:
When a shell command fails, err.message only says "Command failed: ...".
The actual output is on err.stdout and err.stderr. Always log both.
What happened: After fixing the logging, we finally saw the real error:
✖ Railpack could not determine how to build the app.
The app contents that Railpack analyzed contains:
./
├── .git/
└── users/
Root cause:
The test repo (Contact Directory) was not a web app at all.
It was just a folder with some data files inside a users/ directory.
No package.json, no index.html, no Dockerfile, nothing Railpack recognizes.
Railpack only works with actual web app code.
Fix — two things:
- Added a Dockerfile fallback to the pipeline.
Before running Railpack, we now check if the repo already has a
Dockerfile. If it does, we usedocker builddirectly (faster, more reliable). If it doesn't, we fall back to Railpack auto-detection.
const hasDockerfile = existsSync(path.join(cloneDir, "Dockerfile"))
if (hasDockerfile) {
addLog(id, "Dockerfile found — building with docker build…")
await run(id, `docker build -t ${imageTag} ${cloneDir}`)
} else {
addLog(id, "No Dockerfile — trying Railpack auto-detection…")
await run(id, `railpack build ${cloneDir} --name ${imageTag}`)
}- Test with a repo that is actually a deployable web app.
A deployable repo needs at minimum:
- A
Dockerfilein the root, OR - A
package.jsonwith astartscript (for Railpack to detect as Node.js)
- A
Rule for next time:
Before testing the pipeline with any repo, quickly check its root folder on GitHub.
It needs either a Dockerfile or framework files (package.json, requirements.txt, etc.).
If the root only has folders full of data files, it's not a deployable app.
What happened: Railpack detected the app type correctly (Node.js), then immediately failed with:
ERRO BUILDKIT_HOST environment variable is not set.
Root cause: Railpack is an advanced build tool. It doesn't use the old Docker build engine. It uses BuildKit — a faster, smarter build engine. But BuildKit has to be running as a separate process/service. We hadn't set that up.
Think of it like this: Railpack is a high-performance printer driver, but the high-performance printer (BuildKit) wasn't turned on.
Fix:
Added a buildkit service to docker-compose.yml and wired it to the backend:
buildkit:
image: moby/buildkit
privileged: true
command: ["--addr", "tcp://0.0.0.0:1234"]
networks:
- app
backend:
environment:
- BUILDKIT_HOST=tcp://buildkit:1234
depends_on:
- buildkitThe privileged: true is needed because BuildKit needs special OS permissions
to run its own container operations. The --addr tcp://0.0.0.0:1234 makes
BuildKit listen on a network port so the backend can reach it by name (buildkit:1234).
Rule for next time: Any time Railpack is in the stack, BuildKit must also be in the stack. They're a package deal. You don't get Railpack without BuildKit.
What happened: The build completed successfully. The container started. But the last step — registering the route with Caddy — failed with:
Caddy rejected config (403): client is not allowed to access from origin ''
Then after adding an Origin header, a different error:
Caddy rejected config (403): client is not allowed to access from origin 'http://backend:3001'
Root cause:
Newer versions of Caddy added a security check on its admin API (port 2019).
It only accepts requests where the Origin header matches an allowed list.
The allowed list by default is just localhost.
Two issues:
- Node.js
fetch()doesn't add anOriginheader automatically (browsers do, servers don't) - Even after we added
Origin: http://backend:3001, Caddy parsed that tobackend:3001(with port) but our allowed origins only hadbackend(without port)
Fix (two parts):
Part 1 — Updated caddy/Caddyfile to allow requests from the backend:
{
admin 0.0.0.0:2019 {
origins localhost caddy backend
}
}
Part 2 — Updated pipeline.ts to send a matching Origin header:
headers: {
"Content-Type": "application/json",
"Origin": "http://backend", // no port — matches "backend" in Caddy's allowed list
}Why no port in the Origin?
Caddy strips the scheme from the Origin header, giving backend. No port.
So http://backend → backend → matches our origins backend config.
But http://backend:3001 → backend:3001 → does NOT match backend. Close but no match.
Rule for next time: When calling Caddy's admin API from a container, always:
- Add the calling container's name to
originsin the Caddyfile - Add an
Origin: http://<container-name>header (no port) to the fetch call
Great question. Here's the honest answer.
deployKit can deploy an app if it meets at least ONE of these two conditions:
Condition 1 — The repo has a Dockerfile
A Dockerfile tells Docker exactly how to build and run the app.
If the repo has one, we use docker build and it almost always works.
→ Pretty much any tech stack with a Dockerfile will deploy.
Condition 2 — Railpack can detect the framework Railpack looks at the root of the repo for known files:
package.json→ it's a Node.js apprequirements.txtorpyproject.toml→ Pythongo.mod→ GoGemfile→ RubyCargo.toml→ Rust
If Railpack finds one of these AND a start command, it builds the image automatically.
What WON'T work:
- Repos that aren't web apps (just files, scripts, data)
- Monorepos where the root has no framework files (e.g.
backend/+frontend/subdirectories) - Apps that need environment variables to run (we don't yet support passing env vars)
- Apps that listen on a port other than 3000 (we tell containers
PORT=3000)
The real world: Even Vercel, Railway, and Render have limits. They just document them better. They also detect monorepos and ask "which folder is the app?" That's a feature we could add in a future phase.
Goal: Take deployKit from "tech demo" to "actually usable platform." We hit two pain points during testing that this phase fixes:
- Cleaning up failed/old deployments required terminal commands
- Real apps (like TomSkid) crash because they need env vars
The problem:
After every test, leftover containers, images, Caddy routes, and DB rows piled up.
Cleanup meant manually running docker stop, docker rm, docker rmi, a curl to
Caddy, and sqlite3 deletes. That's not how a real platform should feel.
What we built:
A new backend function deleteDeployment(id) that performs all 6 cleanup steps,
each one tolerant of "already gone" (a failed deploy might have a DB row but no container):
1. Stop the container (deploy-<id>)
2. Remove the container
3. Remove the built image (deploy-<id>:latest)
4. Delete the Caddy route (frees the port for reuse)
5. Delete the logs from DB
6. Delete the deployment row
Plus a new endpoint:
DELETE /deployments/:id → calls deleteDeployment, returns { ok: true }
Frontend — the × button: Each deployment in the left list now has a small × in the top-right corner. Clicking it shows a confirm dialog. On confirm:
- The row is removed from the UI immediately (optimistic update — feels instant)
- A DELETE request is sent to the backend
- The polling loop will catch any out-of-sync state on the next 2-second tick
Why each cleanup step is wrapped in .catch(() => {}):
A failed deployment might never have created a container, image, or Caddy route.
If we threw on "container not found", deleting a failed deploy would error out
even though there's nothing to clean. So each step swallows its own errors —
the goal is "make this gone," not "fail loudly if it was already gone."
Files changed:
backend/src/pipeline.ts(addeddeleteDeploymentfunction)backend/src/routes/deployments.ts(added DELETE endpoint)frontend/src/App.tsx(added × button +remove()handler)
The problem:
Real apps need secrets at startup. TomSkid (Next.js + Supabase) crashes immediately
without NEXT_PUBLIC_SUPABASE_URL and friends. Without env vars, deployKit could
only deploy "hello world" toys.
The journey of one env var, end to end:
- You type
DATABASE_URL=postgres://...in the new textarea on the UI. - Frontend sends it to the backend as one big string in
POST /deployments:{ gitUrl, env: "DATABASE_URL=postgres://..." } - Backend doesn't try to be clever — it stores that raw string as-is in the
new
env_varscolumn. Parsing happens later, not now. That way "what you typed" and "what we stored" are always identical, easier to debug. - When the pipeline reaches Step 3 (start container), it reads
env_varsback out of the DB and runs it throughparseEnvVars:- skip blank lines
- skip lines starting with
#(comments) - split on the first
=(so values containing=, like base64 keys, survive)
- The parsed array goes into
docker.createContainer({ Env: [...userEnv, "PORT=3000"] }). Docker hands those to the running app exactly like a.envfile would.
Why PORT=3000 stays last: the Caddy route assumes the container listens on
port 3000. If the user accidentally sets PORT=8080, ours wins (later entries
override earlier ones in Docker's env handling).
Schema migration safety:
Existing DBs predate the env_vars column. SQLite has no ADD COLUMN IF NOT EXISTS,
so we check pragma table_info(deployments) first and ALTER TABLE only if the
column is missing. Old deploykit installs upgrade smoothly, no data loss.
Files changed:
backend/src/db.ts(added column + safe migration)backend/src/pipeline.ts(addedparseEnvVars, read from DB, inject into container)backend/src/routes/deployments.ts(acceptenvfield on POST)frontend/src/App.tsx(Env toggle button + collapsible textarea)
The trigger:
Testing env vars, deployments kept appearing "stuck" at building. Looked like
env vars broke the pipeline. They didn't — env vars aren't even involved at the
build step. The real problem was three separate bugs all hiding from us at once.
Symptom: UI shows Dockerfile found — building with docker build… and then
nothing for 5+ minutes. Looks like a hang. User assumes it's broken and runs
docker compose down, killing the build mid-npm install.
Root cause: run() used Node's execAsync, which only returns output AFTER
the command finishes. So during a 6-minute npm install, zero log lines reach
the DB. The deployment looks dead even though it's working hard.
Fix: Replace execAsync with spawn. Pipe child.stdout and child.stderr
through a line-buffered Writable that calls addLog() for every complete line as
it arrives. Now you watch npm install tick through packages live.
exec = "run command, give me the output when done" (silence for minutes)
spawn = "run command, stream output as it happens" (live progress)
The line-buffer trick: stdout chunks don't respect line boundaries — a single
chunk might be "hello\nworld\nhalf-a-li" and the next chunk completes it with
"ne\nfourth-line\n". We accumulate into a buffer, split on \n, log all complete
lines, and keep the unfinished tail for the next chunk. Otherwise log lines would
look chopped up at random.
Symptom: A deployment row sits at status building forever, even after the
backend has restarted multiple times. No process is alive to ever flip it to
failed or running.
Root cause: When the backend dies mid-pipeline (SIGTERM from docker compose down, or just a crash), the in-flight deployment is orphaned. The DB row remembers
building from before the crash. The new backend has no memory of the old build,
so the row sits there forever, polluting the UI.
Fix: On startup, scan for any deployment in a "mid-flight" status
(pending / cloning / building / starting) and mark it failed with a log
line Backend restarted while this deployment was in progress — marking as failed.
Now restarts are clean. Old runs don't haunt the UI.
This runs once at backend boot, before any HTTP requests are accepted, in
recoverOrphanedDeployments() at the top of pipeline.ts.
Even after fixing build streaming, once the container started successfully there
was a new blind spot: the app's own crashes went into the void. If TomSkid
crashed because NEXT_PUBLIC_SUPABASE_URL was missing, the UI just showed a green
"running" badge and a broken URL. No clue why.
Fix: After container.start(), attach Docker's log stream (container.logs({ follow: true })) and pipe its stdout/stderr through the same line-buffered writer
into our log table. Now the running app's output flows into the UI in real time.
The multiplexing wrinkle: When you don't allocate a TTY, Docker sends both
stdout and stderr over the same stream, prefixing each chunk with an 8-byte
header indicating which is which. We use docker.modem.demuxStream(...) to split
them back apart. Without that, we'd see binary garbage interleaved with the logs.
SSE keep-alive: The original SSE handler closed the connection the moment a
deployment hit running status, which would have killed live container log
streaming the second the build finished. Tweaked it: send the done event so the
UI drops the "live build" indicator, but keep the loop running so container
output continues to flow. The connection only closes when the user navigates away
or selects another deployment. The frontend was also updated to NOT close
EventSource on done for successful deploys — only on failed.
Files changed:
backend/src/pipeline.ts(spawn-basedrun(),streamContainerLogs(),recoverOrphanedDeployments(),makeLineWriter()helper)backend/src/routes/deployments.ts(SSE keeps tailing afterrunning)frontend/src/App.tsx(don't close EventSource ondonesuccess)
The original plan mentioned TanStack Query + TanStack Router. We're skipping both:
- TanStack Query: our
setInterval+fetchpolling already works fine. Adding it would be 200 lines of abstraction to solve a problem we don't have. - TanStack Router: we have one screen. There's nothing to route to.
Rule we're following: don't add abstractions for hypothetical future needs.
- README that explains setup, usage, and limitations
- Deploy deployKit itself somewhere (Brimble, a VPS, etc.)
- Final submission