Fairness of NLP/LLM-Based Intimate Partner Violence (IPV) Detection Models

working deliverable only This repository tracks experiments that benchmark large language models on IPV detection tasks. The goal is to keep the evaluation pipeline stable while we iterate on prompts, datasets, and model families (open-source and proprietary). Everything needed to rerun an experiment—from prompts to metrics to sample code—lives inside this repo so collaborators can pick up progress quickly.

Key Resources

• Dataset/reddit_data.csv: Labeled Reddit sentences from the DETECTIPV 2022 work; used as the public ground truth during Week 1 experiments.
• 1_LLM_Eval/prompts.md: Canonical prompt templates for binary vs. multitype detection.
• 1_LLM_Eval/eval_llm_pipeline.py: Reusable evaluation module that prints metrics, exports ROC/AUC/precision-residual plots, and builds multitype confidence waterfalls.
• 1_LLM_Eval/eval_pipeline_sample_use.ipynb: Notebook walkthrough showing how to call the pipeline for both binary and multitype runs.
• 1_LLM_Eval/test_results/: JSON outputs from each prompt/model combo plus generated figures (figs/) and cached comparisons.
• 1_LLM_Eval/qwen2.ipynb, 1_LLM_Eval/qwen3.ipynb, 1_LLM_Eval/qwen3_(1) (1).ipynb: Development notebooks for Qwen-based experiments (prompt iterations, inference scripts, and exploratory plots).

Progress Log

Week 1 – Baseline With Open Instruction-Tuned Models

Focus: run Llama 3.1 (8B) and Qwen 2.5 (7B) against the labeled Reddit dataset for a like-for-like comparison against the DETECTIPV 2022 paper.

• Binary IPV detection: prompts and evaluation captured in 1_LLM_Eval/qwen2.ipynb and serialized outputs inside 1_LLM_Eval/test_results/binary_*.json.
• Subtype (multilabel) detection: prompts + parsing logic iterated in 1_LLM_Eval/qwen3.ipynb and 1_LLM_Eval/qwen3_(1) (1).ipynb, producing multilabel_*.json result files.
• Evaluation parity: 1_LLM_Eval/eval_llm_pipeline.py now matches the DETECTIPV metric stack (accuracy, precision, F1, ROC-AUC) and introduces confidence-based waterfall plots to visualize subtype predictions.
• Sample usage doc: 1_LLM_Eval/eval_pipeline_sample_use.ipynb documents how to plug new JSON outputs into the shared pipeline, keeping future experiments consistent.

Week 2 – Proprietary LLM Sampling & New Data

Focus: broaden coverage beyond open models and stress-test on a newer dataset.

1. Sampling + evaluation on proprietary LLMs (e.g., GPT-5)
   • Reuse the same JSON schema so results drop directly into the evaluation pipeline.
   • Capture both binary and multitype prompts to compare against Week 1 baselines.
2. Explore prediction quality on the newer dataset
   • Clean/ingest the additional corpus (encrypted or kept private as needed).
   • Run spot checks in notebooks, then formalize via the pipeline once labeling is aligned.

Week 4 (in Progress)

• Improve prompts
  • Binary
    • Define IPV better
    • "Penalize" or define what wouldn't be considered IPV.
  • Multitype
    • Better definition of what each type of IPV is through keywords to describe.
• Investigate what sentences do they get wrong? Compare between Qwen & ChatGPT results.