💬 DialogueNFR [Paper (PDF)]

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

Accepted at SIGDIAL 2026 🎉 — Read on arXiv 📄

📖 What is This?

This repository accompanies our paper on evaluating Large Language Model (LLM) agents for Non-Functional Requirement (NFR) evaluation in software engineering.

DialogueNFR contains:

• 💬 A multi-turn dialogue dataset between developers and GitHub Copilot
• 📋 148 HIPAA-derived non-functional requirements
• ✅ Expert-annotated ground truth of evaluation of requirements
• 👩‍💻 49 developer studies
• 📊 Dialogue annotations for conversational analysis
• 📈 PARADISE-based user satisfaction modeling

The benchmark evaluates LLMs across three complementary dimensions:

• Requirement Satisfaction Level
• Reasoning Quality
• Code Localization

---

📖 Overview

LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional correctness. This leaves a critical gap in assessing the quality and accuracy of these conversations when handling Non-Functional Requirements (NFRs), which are inherently vague, context-dependent, and involve many parts of a program. Evaluating how well these systems support collaborative reasoning about NFRs requires methods that go beyond single-turn accuracy to capture both the correctness of the system's outputs and the quality of the multi-turn interaction. In this paper, we investigate the accuracy and quality of multi-turn conversations between developers and an LLM-based agent in the domain of Health Insurance Portability and Accountability Act (HIPAA) regulatory compliance. We hired 49 programmers to interact with GitHub Copilot to assess 148 HIPAA-derived NFRs against the iTrust codebase, a system designed to comply with HIPAA regulations, across three dimensions: requirement satisfaction level, reasoning, and code localization. We find that developers tend to agree with LLM assessments, but accuracy against expert ground truth is low. We model user satisfaction and find that longer system responses and more information-providing turns negatively affect user satisfaction, whereas proactive interactions positively affect it. Our findings provide insights for designing LLM-based dialogue systems that support NFR assessment.

---

🌟 Main Findings

Accuracy vs. User Perception

Although developers agreed with LLM responses over 90% of the time, the actual accuracy against expert annotations was substantially lower.

This reveals a concerning gap between perceived response quality and actual correctness.

Dialogue Design Matters

Using the PARADISE framework, we found that:

• ✅ Proactive interactions improve user satisfaction.
• ❌ Long, verbose responses reduce user satisfaction.
• ❌ Excessive information providing turns reduce satisfaction.

🏗 Repository Structure

DialogueNFR/
│
├── GT/                   # Ground-truth annotations of 148 Requirements
├── analysis/             # Analysis notebooks
├── annotation/           # Annotation process and annotated data
├── data/                 # Dataset files
├── website/              # survey interface
└── README.md

---

📑 Citation

If you use DialogueNFR in your research, please cite:

@article{fatideh2026accuracy,
  title={Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment},
  author={Fatideh, Ali Pourghasemi and Baldwin, Wilder and Dhakal, Maria and McMillan, Collin and Ghanavati, Sepideh},
  journal={arXiv preprint arXiv:2606.24834},
  year={2026}
}

---

🤝 Contact

For questions, issues, or collaborations, please open a GitHub Issue or contact the authors.