Implement non-blocking provider setup with gated tool stubs

Dockerfile

@@ -18,7 +18,7 @@
 RUN pip install --no-cache-dir uv
 ENV PATH="/root/.local/bin:$PATH"
 
-COPY server.py config.py process_runner.py builtin_tools.py tool_registry.py rest_provider.py oauth_bootstrap.py catalog.py ./
+COPY server.py config.py process_runner.py builtin_tools.py tool_registry.py provider_status.py rest_provider.py oauth_bootstrap.py catalog.py ./
 COPY frontend/ ./frontend/
 COPY handlers/ ./handlers/
 
@@ -29,8 +29,14 @@
 ENV MCP_ENV_FILE=/app/.env
 ENV MCPPROXY_FILES_DIR=/app/files
 ENV MCPPROXY_REPOS_DIR=/app/repos
 ENV MCPPROXY_REST_AUTH_DIR=/app/.rest-auth
 
+# Pin the pip / uv wheel caches to the persisted /root/.cache volume
+# (see docker-compose.yml) so ephemeral containers reuse downloaded wheels
+# across restarts instead of re-fetching them every startup.
+ENV PIP_CACHE_DIR=/root/.cache/pip
+ENV UV_CACHE_DIR=/root/.cache/uv
+
 EXPOSE 8888 8889
 
 CMD ["python", "server.py"]

README.md

@@ -24,6 +24,11 @@ Each tool **provider** is a single YAML file under `tools/`. The YAML contains:
 runs `setup_commands`, then registers each tool automatically — no Python files to
 maintain separately, no changes to `server.py` needed when adding new tools.
 
+By default this setup runs **in the background**: the MCP server starts accepting
+requests immediately while each provider's dependencies install. A tool whose provider
+is still installing returns a structured **retry directive** instead of blocking — see
+[Non-blocking startup](#non-blocking-startup) below.
+
 Two **built-in tools** (`mcpproxy__listfiles` and `mcpproxy__getfile`) are always registered
 without any YAML config.  They give LLMs read-only access to a configurable directory
 (default: `/app/files`, mountable as a Docker volume) — useful for retrieving screenshots
@@ -49,6 +54,46 @@ is added automatically when the tool is registered.
 | **8888** | MCP endpoint — `http://localhost:8888/mcp` |
 | **8889** | Web UI & OpenAI-compatible tools endpoint — `http://localhost:8889` |
 
+## Non-blocking startup
+
+Provider setup — cloning/building repository providers, `pip install`-ing each
+provider's `requirements`, and running its `setup_commands` (e.g.
+`npx playwright install chrome`) — can take a long time. Rather than block the
+MCP server until all of that finishes, mcpproxy **registers every tool up front
+and runs the setup in the background**:
+
+- The MCP endpoint on `8888` and the UI on `8889` come up **immediately**.
+- Every tool is advertised right away, so MCP clients see the full tool list at
+  once.
+- A call to a tool whose provider is **still installing** returns a structured
+  retry directive instead of failing or hanging:
+
+  ```json
+  {
+    "ok": false,
+    "tool": "playwright__browser_navigate",
+    "status": "initializing",
+    "retry_after_seconds": 15,
+    "message": "Tool 'playwright__browser_navigate' is not ready yet — provider 'playwright' is still installing its dependencies. Wait ~15s and call this tool again."
+  }
+  ```
+
+  The calling LLM reads the message and retries shortly. The two built-in tools
+  (`mcpproxy__listfiles` / `mcpproxy__getfile`) are always ready immediately.
+
+- If a provider's setup **fails**, its tools return `"status": "failed"` with the
+  error at call time (the rest of the server stays up).
+
+Knobs:
+
+| Variable | Default | Meaning |
+|---|---|---|
+| `MCPPROXY_BACKGROUND_SETUP` | `1` | Set to `0` to run setup synchronously before the server starts (the old blocking behaviour). |
+| `MCPPROXY_INIT_RETRY_SECONDS` | `15` | Seconds advertised in the `retry_after_seconds` field of the directive. |
+
+Startup stays fast across restarts because pip/uv/npm caches and cloned repos are
+persisted via Docker volumes — see [Volumes & caching](#volumes--caching).
+
 ## Layout
 
 ```text
@@ -201,9 +246,10 @@ Example — for a Playwright package provider:
 npx playwright install chrome
 ```
 
-Commands run in order before the server accepts connections. The subprocess package is
-launched lazily on the first tool call, not at startup, so the browser binary is always
-ready when needed.
+Commands run in order in the background (see [Non-blocking startup](#non-blocking-startup)) —
+the server accepts connections immediately and the provider's tools return a retry directive
+until its setup finishes. The subprocess package itself is launched lazily on the first tool
+call, so the browser binary is always ready when needed.
 
 > **After editing and saving** a provider's command or setup steps, click **Restart MCP Server**
 > (the yellow bar that appears after saving) to apply the changes.
@@ -637,6 +683,10 @@ re-downloaded, re-built, or re-authorized on every fresh container.
 | `/root/.local/share/uv` | `mcpproxy-uv-tools` | uvx per-tool venvs | uvx re-creates per-tool venvs from cached wheels. |
 | `/app/.mcp-auth` | `mcpproxy-mcp-auth` | OAuth token cache (access + refresh tokens) for `mcp-remote` bridge providers, e.g. the official Asana MCP (`MCP_REMOTE_CONFIG_DIR`). Kept out of `/app/files` so tokens aren't exposed via `mcpproxy__getfile`. | Re-authorize through the browser on every fresh container. Only relevant if you run an OAuth-bridge provider. |
 
+The image pins `PIP_CACHE_DIR=/root/.cache/pip` and `UV_CACHE_DIR=/root/.cache/uv`
+so the pip and uv wheel caches always land inside the persisted `mcpproxy-cache`
+volume, even if `HOME`/XDG defaults change.
+
 In dev (`docker-compose.override.yml`), `mcpproxy-tools`, `mcpproxy-files`,
 `mcpproxy-repos`, and `mcpproxy-mcp-auth` are replaced with bind mounts (`./tools`,
 `./files`, `./repos`, `./.mcp-auth`) so you can inspect or edit them from the host

provider_status.py

@@ -0,0 +1,70 @@
+"""
+Shared provider-readiness registry — populated by server.py, read by both
+server.py's gated tool handlers and frontend/app.py (for a "still initializing"
+banner in the UI).
+
+Background:
+    server.py registers every provider's tools immediately as *gated stubs*
+    (built from the YAML tool list only — no pip/exec/network), then runs the
+    slow per-provider setup (pip install / build / setup_commands) on a
+    background thread.  Until a provider's setup finishes, calls to its tools
+    return a "still initializing, retry shortly" directive instead of failing.
+
+    This module holds the per-provider readiness state that the gated handlers
+    consult, plus the real handlers that replace the stubs once setup completes.
+
+Like tool_registry.py, every reader runs in the same Python process (the UI runs
+in a daemon thread), so a plain module-level dict is sufficient — no IPC needed.
+This module has NO import-time side effects so it is safe to import from anywhere.
+"""
+
+from dataclasses import dataclass, field
+from typing import Any, Callable
+
+# A provider that has been registered but whose dependencies are still
+# installing.  Once setup finishes it flips to "ready"; if building its real
+# handlers fails it flips to "failed".
+PENDING = "pending"
+READY = "ready"
+FAILED = "failed"
+
+
+@dataclass
+class ProviderState:
+    """Readiness of one provider and (once ready) its real tool handlers.
+
+    ``name``     — the provider name (YAML filename stem), used in messages.
+    ``status``   — one of ``PENDING`` / ``READY`` / ``FAILED``.
+    ``error``    — failure detail when ``status == FAILED``, else ``None``.
+    ``handlers`` — advertised tool name → real async handler, populated when
+                   the provider becomes ready.  Gated stubs delegate here.
+    """
+
+    name: str
+    status: str = PENDING
+    error: str | None = None
+    handlers: dict[str, Callable[..., Any]] = field(default_factory=dict)
+
+
+# provider name → ProviderState
+_states: dict[str, ProviderState] = {}
+
+
+def set_state(state: ProviderState) -> None:
+    """Register (or replace) a provider's readiness state."""
+    _states[state.name] = state
+
+
+def get(name: str) -> ProviderState | None:
+    """Return the state for ``name``, or ``None`` if unknown."""
+    return _states.get(name)
+
+
+def all_states() -> dict[str, ProviderState]:
+    """Return a shallow copy of the full provider → state map."""
+    return dict(_states)
+
+
+def clear() -> None:
+    """Remove all states.  Used only in tests."""
+    _states.clear()