Security considerations for Abliterlitics.

1. HuggingFace Token Handling
2. Model File Trust Boundary
3. Path Validation
4. HarmBench Supply Chain
5. Docker Security
6. comparison.json Validation
7. Network Security
8. Known Limitations

Abliterlitics downloads datasets (HarmBench behaviors, harmless_alpaca) from HuggingFace. Some may require authentication.

Preferred method (environment variable):

export HF_TOKEN="hf_xxxxx"
./abliterlitics.sh auto ./my-comparison/

The token is passed into Docker containers via -e HF_TOKEN=... when set in the host environment.

• Tokens are never written to disk by Abliterlitics (only in Docker env vars)
• Docker command logs redact sensitive environment variables (see docker_helpers.py)
• The _redact_cmd() function strips known sensitive keys (HF_TOKEN, API_KEY, etc.) from logged commands
• Tokens are passed as Docker environment variables, not in command arguments

The following environment variable keys are redacted in logs:

• HF_TOKEN
• HUGGING_FACE_HUB_TOKEN
• API_KEY
• SECRET
• PASSWORD
• TOKEN

Model weights are loaded via safetensors using safe_open(framework="pt", device="cpu"). This is memory-mapped, format-validated, and cannot execute arbitrary code. All weight analysis scripts use safetensors exclusively.

The KL divergence pipeline uses torch.save() / torch.load() for intermediate logits files (logits_*.pt). These files are locally produced by the tool itself during the collect phase and read back during the compute phase. They are not downloaded or received from external sources.

The weights_only=False flag is used because the saved data includes dictionaries and metadata beyond raw tensors. Risk assessment: Low, since the files are generated by the tool on the same machine. However, if you receive .pt files from an untrusted source, do not pass them to the KL pipeline.

Mitigation: The --base-logits argument only accepts paths generated by the collect phase.

The lm-evaluation-harness and vLLM backends use trust_remote_code=True to support custom model architectures (e.g., Qwen3.5's Mamba2 hybrid, GLM-4). This means the model repository's custom code will execute inside Docker containers.

Risk assessment: This is standard practice for HuggingFace models but does mean you must trust the model repository. The Docker isolation limits the blast radius to the container.

Variant names and comparison names in comparison.json are validated against the pattern ^[a-z0-9][a-z0-9_-]*$:

{
  "name": "my-comparison",
  "variants": {
    "heretic": { "path": "..." },
    "hauhau-v2": { "path": "..." }
  }
}

This prevents:

• Shell injection: Variant names are used in shell variable names and file paths
• Path traversal: Names containing ../ or / are rejected
• Special characters: Names with spaces, quotes, or semicolons are rejected

The common.sh shell library parses comparison.json via a Python helper (_load_comparison.py) that:

1. Passes paths as sys.argv (no string interpolation into Python source)
2. Validates all slug fields against the regex pattern
3. Wraps all output values with shlex.quote() before printing shell assignments
4. The output is consumed by eval but all values are pre-quoted

# Inside _load_comparison.py — every value is shlex-quoted
print(f"COMPARISON_NAME={shlex.quote(name)}")
print(f"BASE_DIR={shlex.quote(comp_dir + '/' + base)}")

• All model directories are mounted read-only (:ro) in Docker containers
• Paths are resolved via Python's Path.resolve() before Docker mount
• Symlinks are followed during resolution (standard Python behavior)

The HarmBench behaviors CSV is fetched from the HarmBench GitHub repository, pinned to a specific commit SHA:

HARMBENCH_BEHAVIORS_CSV = (
    "https://raw.githubusercontent.com/centerforaisafety/HarmBench/"
    "9d0e87a7e6776614c546f2e3d4392635e6e0cf8a/"
    "data/behavior_datasets/harmbench_behaviors_text_all.csv"
)

This SHA should be updated manually after reviewing changes to the upstream dataset. Both harmbench_generate.py and benchmark_eval.py use the same pinned URL.

The HarmBench classifier model is loaded from a specific HuggingFace model ID. The model should be verified against known checksums before use in production evaluations.

Abliterlitics uses --runtime=nvidia (NOT --gpus):

• --runtime=nvidia uses the NVIDIA Container Toolkit's recommended approach
• Provides proper device enumeration and library injection
• Avoids the deprecated --gpus flag's known edge cases

• --shm-size=16g for shared memory (required by PyTorch DataLoader and vLLM)
• --ipc=host for lm-eval (required for multi-process inference)
• No --privileged flag is ever used

• Docker containers use the default bridge network for analysis
• HarmBench generation uses --network host to connect to the inference server running in a sibling container
• The inference server binds to 0.0.0.0 inside its container (Docker network only)
• No host port publishing (-p) is used for analysis containers
• Recommendation: For restricted environments, bind the HarmBench server to 127.0.0.1 only

The comparison.schema.json uses JSON Schema draft 2020-12 with:

• additionalProperties: false on all objects — rejects unknown fields
• patternProperties with slug regex for variant names
• required fields enforced
• type constraints on all values

Validation happens at load time in config.py:

jsonschema.validate(instance=data, schema=schema)

Invalid configurations are rejected before any Docker commands are constructed.

Abliterlitics containers make the following outbound connections:

Analysis containers do not expose any ports. The HarmBench generation phase starts a llama-server container on the Docker network and connects from the generation container via --network host.

If operating in a restricted network:

Allow: huggingface.co:443
Allow: cdn-lfs.huggingface.co:443
Allow: raw.githubusercontent.com:443
Deny: all other outbound

Multiple code paths use trust_remote_code=True for model loading (vLLM, transformers tokenizer). This is required for custom architectures but means model repository code executes inside containers. Only use models from trusted sources.

KL divergence logits are stored as PyTorch .pt files (pickle format). These are locally generated and consumed — do not accept .pt files from untrusted sources.

The HarmBench generation phase exposes an inference server port. In the current implementation, this uses --network host. For maximum security, use isolated Docker networks instead.

The KL divergence computation (FROZEN methodology) uses hardcoded model.layers.* and model.embed_tokens patterns in build_split_device_map(). This may not match Qwen3.5's model.language_model.layers.* structure for large models requiring split loading. This cannot be fixed without modifying the FROZEN computation section. For such models, use single-GPU mode or GGUF fallback.

The Dockerfile.ik-llamacpp uses IK_LLAMA_CPP_COMMIT=main by default. For reproducible builds, override with a specific commit SHA:

docker build --build-arg IK_LLAMA_CPP_COMMIT=<sha> -t abliterlitics-ik-llamacpp:1.0.0 -f docker/Dockerfile.ik-llamacpp docker/