Skip to content

Evaluate on additional datasets: OWASP v2 and Antijection #3

Description

@notyorch

Context

The primary benchmark uses deepset/prompt-injections (n=662). The import script already supports additional datasets via --preset, but no results have been published for them yet.

Evaluating on OWASP v2 and Antijection would:

  • Reduce single-dataset bias in reported metrics
  • Validate that 95.1% PINT accuracy generalizes beyond deepset
  • Strengthen cross-dataset claims for research credibility

Goal

Run and publish benchmark results for at least two additional datasets already supported by the pipeline.

Tasks

  • Run scripts/import_external_dataset.py --preset owasp-v2
  • Run scripts/import_external_dataset.py --preset antijection
  • Execute scripts/run_pint_benchmark.py for each dataset in both benchmark and production modes
  • Freeze results in notebooks/data/external/results/
  • Add a cross-dataset results table to the README (alongside the existing deepset table)
  • Update stack ablation figures if results differ meaningfully

References

  • Import script: scripts/import_external_dataset.py
  • Existing deepset results: notebooks/data/external/results/deepset-sentinel.json

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions