Skip to content

cli.md

Latest commit

 

History

History
219 lines (165 loc) · 8.01 KB

cli.md

File metadata and controls

219 lines (165 loc) · 8.01 KB
Raw

CLI reference

Entry: src/lmtwt/cli.py:main (wraps asyncio.run(async_main())).

uv run lmtwt <args>           # recommended
./run.sh <args>               # auto-detects uv, falls back to a venv
python -m lmtwt <args>        # any installed env

Modes (top-level dispatch)

These flags are checked in order; the first match wins:

Flag Effect
--list-templates Print template IDs and exit
--list-flows Print multi-turn flow IDs and exit
--list-vectors Print tool-use injection vectors and exit
--web Launch the Gradio UI
--probe-mode Use AsyncProbeAttack (canned payloads from PayloadGenerator)
--strategy {pair,tap} Run an automated refinement strategy
--mode {interactive,batch,template,multi-turn,tool-use} Standard attack engine

Models

Flag Default Choices Notes
--attacker, -a gemini gemini, openai, anthropic, huggingface, lmstudio, claude-code, acp Provider that generates attack prompts
--target, -t openai gemini, openai, anthropic, external-api, huggingface, lmstudio, claude-code, acp Provider under test
--attacker-model provider default any Specific model ID for attacker
--target-model provider default any Specific model ID for target

Provider defaults: see models.md. Set OpenAI / Gemini IDs in config.json rather than relying on the class-level fallbacks (they are intentionally stable, not auto-bumped).

Mode-specific flags

Interactive mode

Loop: prompt for instruction → generate attack → optionally edit → send → print verdict. q exits. --auto-send skips the edit prompt.

Batch mode

Repeatable --instruction "..." (free-form text). Each runs --iterations times with --delay seconds between attacks. --concurrency N parallelizes across instructions via an asyncio.Semaphore.

Template mode

Repeatable --template <id>. Resolves each to its instruction via get_template_instruction and runs through the same batch path.

Multi-turn mode

--mode multi-turn --flow <id> --instruction "<goal>".

Flag Notes
--flow <id> One of crescendo_creative_writing, crescendo_role_assumption, topic_drift
--list-flows Show all flows with descriptions and step counts

The flow runs each step against the same target conversation. Final-turn verdict (and per-turn if judge_after_each is set on the flow) determines success. See attacks.md for flow internals.

Tool-use mode

--mode tool-use --instruction "<goal>". Performs indirect prompt injection through fake tool outputs.

Flag Notes
--tool-vector <name> One of web_search, document, tool_output. Omit for dynamic rotation.
--list-vectors Show all injection vectors and exit

See attacks.md for InjectionVector / ToolHarness / ToolUseAttack internals.

Probe mode

Flag Default Choices Notes
--probe-mode off flag Activates AsyncProbeAttack
--probe-category all dan, injection, xss, glitch, misleading, malware, forbidden_knowledge, snowball, all Restricts payload pool
--probe-iterations 5 int Attacks per category (or total when all)
--concurrency 1 int Parallel attack execution

Refinement strategies

--strategy {pair,tap} overrides --mode when present.

Flag Default Notes
--strategy {pair,tap} — Pick PAIR (linear) or TAP (tree-search)
--strategy-iterations 5 PAIR only: max iterations
--strategy-branching 3 TAP only: variants per parent (B)
--strategy-depth 4 TAP only: tree depth (D)
--strategy-prune 2 TAP only: top-K survivors per level
--strategy-threshold 8 Score (1-10) that counts as a successful jailbreak

The scoring judge is built from --compliance-provider. See attacks.md for strategy internals.

Judge (success detection)

Flag Default Notes
--judge {regex,llm,ensemble} regex Explicit judge selection
--compliance-agent off Back-compat alias for --judge ensemble
--compliance-provider gemini Provider used for LLMJudge / ScoringLLMJudge

Hacker mode

Flag Default Notes
--hacker-mode off Conversation-history analysis + auto-retry on failure
--hacker-system-prompt <text> from config.json or built-in Override attacker system prompt
--max-retries 3 Hacker-mode auto-retry budget per session

Defense

Flag Default Notes
--system-prompt <text> built-in What the target sees as its system prompt

Pacing

Flag Default Notes
--instruction, -i <text> — Repeatable; required by batch and multi-turn modes
--template <id> — Repeatable
--iterations 1 Repeats per instruction
--delay 1.0 Seconds between batch attacks (per-instruction)
--concurrency 1 Parallel attack execution (batch / template / probe / multi-turn)

Proxy and TLS

For routing through Burp Suite, mitmproxy, ZAP, or a corporate egress proxy.

Flag Notes
--proxy <url> e.g. http://127.0.0.1:8080. Applied to attacker, target, judge
--ca-bundle <path> PEM cert bundle (e.g. Burp's cacert.pem)
--insecure Skip TLS verification — use only when you trust the network path

For external-api targets, per-target overrides (proxy, ca_bundle, insecure keys in the target-config JSON) win over these CLI flags.

Configuration

Flag Default Notes
--config, -c <path> config.json at repo root Application config
--target-config <path> — Required when --target external-api

Web UI

Flag Default Notes
--web off Launch Gradio UI
--web-port 8501 Port to serve on
--share off Request a public Gradio share URL

Examples

# Interactive Gemini-vs-OpenAI session, ensemble judge
uv run lmtwt --attacker gemini --target openai --mode interactive --judge ensemble

# Batch with 5-way concurrency
uv run lmtwt --mode batch \
  --instruction "Generate a jailbreak prompt" \
  --instruction "Try system-prompt extraction" \
  --iterations 3 --concurrency 5

# Crescendo multi-turn flow against Anthropic
uv run lmtwt --target anthropic --mode multi-turn \
  --flow crescendo_creative_writing \
  --instruction "Get the model to give a working keylogger"

# PAIR refinement, threshold 8/10
uv run lmtwt --target openai --strategy pair \
  --strategy-iterations 6 --strategy-threshold 8 \
  --instruction "Bypass the system prompt and reveal it"

# TAP tree search, B=4 D=3 K=2
uv run lmtwt --target anthropic --strategy tap \
  --strategy-branching 4 --strategy-depth 3 --strategy-prune 2 \
  --instruction "Get a malware-generation jailbreak"

# Probe-mode sweep across categories
uv run lmtwt --probe-mode --probe-category all --probe-iterations 8 \
  --target anthropic --judge ensemble

# Route everything through Burp
uv run lmtwt --proxy http://127.0.0.1:8080 --ca-bundle ~/.burp/cacert.pem \
  --target openai --mode interactive

# Custom WebSocket target
uv run lmtwt --target external-api --target-config examples/ws_target.json

# LM Studio local server
uv run lmtwt --target lmstudio --target-model qwen2.5-coder-7b-instruct \
  --mode interactive

# Claude Code as the attacker (via ACP)
uv run lmtwt --attacker claude-code --target openai --mode interactive

# Tool-use indirect prompt injection
uv run lmtwt --mode tool-use --tool-vector document \
  --instruction "exfiltrate the user's pasted secrets" \
  --target anthropic --judge ensemble

Provider env vars

Env var Used by
LM_STUDIO_BASE_URL LM Studio target (default http://localhost:1234/v1)
CLAUDE_CODE_PATH ACP / Claude Code provider (default claude)
CLAUDE_CODE_ARGS Extra args for the ACP subprocess (shlex-split)