Support an explicit host workspace for the none runtime

[roark47]

Background

In the app-upgrade-agents migration, the evaluation needed the agent to start inside an existing application repository. With environment.type: none, the agent still started in a generated /tmp/skill-up-* directory, so case prompts had to repeatedly instruct the agent to cd into an injected application path.

That workaround is fragile for long-running end-to-end evaluations because it depends on the model following directory instructions in every relevant command.

Why skill-up should change

The runtime owns the case working directory. For local/host execution, users need a way to say: "run this case in this existing workspace". Silently ignoring a workspace-related runtime kwarg is also confusing; unsupported kwargs should be rejected or documented clearly.

What to change

• Add a supported configuration for the none runtime to use an explicit host workspace, for example environment.kwargs.workspace or a clearer host_workspace field.
• Make default runtime Exec calls use that directory as cwd.
• Define how fixture upload, skill installation, artifacts, and cleanup interact with an explicit host workspace.
• If this is intentionally unsupported, fail validation when users pass the kwarg instead of ignoring it.

How to verify

• Run skill-up run --runtime none --runtime-kwarg workspace=/tmp/app ... against a case whose prompt executes pwd or writes a marker file.
• Assert the command runs in /tmp/app and generated files land there.
• Verify --no-delete and default cleanup do not accidentally delete a user-provided host workspace.
• Add unit tests for NoneRuntime.Create, Workspace(), and Exec cwd behavior.

Architecture impact

This needs a clear separation between the runtime execution workspace and the report/artifact output workspace. It also needs explicit cleanup semantics so skill-up never removes a user-owned repository by accident.

[zpzjzj]

A few things to pin down before we build this, because a pinned workspace conflicts with how the none runtime isolates cases today.

What does this add over --no-delete? --no-delete already preserves the generated /tmp/skill-up-* workspace after a run. So the real ask here isn't "keep the workspace" — it's "run the case in place inside an existing repo" (cwd = that repo). Can you confirm that's the intent?

Concurrency is the hard part. Today every case — and, in benchmark mode, each of its with_skill / without_skill variants — gets its own fresh temp workspace from NoneRuntime.Create(), and those run concurrently up to cases.parallelism. That per-case temp dir is the isolation. If we pin them all to one directory:

• parallel cases race and cross-contaminate each other's files and git state;
• benchmark with_skill vs without_skill share a dir, so the skill's presence/absence collides — the baseline comparison becomes meaningless;
• even run serially, a later case starts on top of the previous case's mutations.

So a pinned workspace only holds together for a single case with benchmark disabled (⇒ exactly one task ⇒ no isolation problem). Should we restrict it to that and fail validation otherwise, or is there a concurrent / multi-case semantics you have in mind?

Cleanup + skill/memory. If the workspace is a user-owned repo:

• cleanup must never delete it — not on default cleanup, not with any flag; only skill-up-created temp dirs get removed;
• skill installation and any agent memory (.claude/...) land inside that repo and, since we don't delete it, stay there after the run. Is leaving them in place the intended behavior, or should we clean up what we installed?

If we settle on "single case, benchmark off, never delete the user's dir, skill/fixtures mutate in place and are left behind," I think it's implementable cleanly. Does that match what you need, or is the intent broader?