Quality Evaluation

Evaluating the quality of AI software systems is essential for informed decision-making, cost-benefit analysis, and ensuring reliable system behavior. Talk2PowerSystem Chat is a ReAct-based agent, and while various evaluation frameworks exist for assessing such systems, most rely on using a large language model (LLM) as a judge to measure accuracy. A comprehensive review of these frameworks and tools is available here.

Although LLM-based evaluation can be effective in certain contexts, it presents two major issues: (1) it is resource-intensive and time-consuming, and (2) it introduces uncertainty, as LLMs are inherently nondeterministic and not always reliable. Relying on one LLM to evaluate another LLM can become an evaluation problem in itself — what might be called a "paradox of evaluation".

Inspired by the approach in "CrunchQA: A Synthetic Dataset for Question Answering over Crunchbase Knowledge Graph", we chose instead to build a QA dataset with expected tool calls and corresponding outputs to measure the system quality. Our evaluation focuses on the agent’s ability to interpret natural language questions and execute the correct sequence of tool invocations with appropriate arguments. Rather than scoring the quality of the natural language response, we assess the model’s understanding of the query and the data it retrieves to answer it. This approach aligns with the goals of the Talk2PowerSystem project, which emphasizes transparency and explainability alongside accuracy.

To support this evaluation methodology, we developed a Python library called graphrag-eval that is agnostic to the underlying implementation or LLM model used. It accepts as input a QA dataset and the system’s output, and evaluates not only the accuracy of the responses, but also the execution performance and estimated cost—calculated based on token usage statistics. This makes it suitable for comparing different agent implementations in a consistent and reproducible manner.

The QA dataset defines reference tool calls as a sequence of groups:

[[t₁, ..., tₙ], ..., [tₖ, ..., tₘ]]

This structure assumes that tools within a single group can be executed in parallel (order is irrelevant), but all must be completed before the next group begins.

Version 5 and Earlier: Final-Group Focus

Previously, accuracy was calculated solely based on the final group [tₖ, ..., tₘ]. This group was considered the "data retriever" responsible for the final answer. Accuracy was the ratio of correctly matched calls within this final step.

Version 6 and Later: Holistic Macro Mean

As of version 6, the library assesses the entire execution chain. The accuracy is calculated as a macro mean across all reference groups:

• Group Score: For each group, the score is the number of matched tool calls divided by the total number of expected calls in that group.
• Total Accuracy: The sum of all individual group scores divided by the total number of groups.

Tool Matching Logic

A "match" is defined differently depending on the specific tool being used:

Tool	Matching Logic
SPARQL Query Tool	Because semantic equivalence of SPARQL queries is undecidable [1], we compare the equivalence of the resulting outputs rather than the query syntax.
Autocomplete Search	Validates if the IRI specified in the reference output is present in the actual system output.
Timeseries / Datapoints	Validates if the arguments (parameters) of the tool calls match the reference.

[1]: Melisachew Wudage Chekol, Jérôme Euzenat, Pierre Genevès, Nabil Layaïda. Evaluating and benchmarking SPARQL query containment solvers. Proc. 12th International semantic web conference (ISWC), Oct 2013, Sydney, Australia. pp.408-423, ff10.1007/978-3-642-41338-4_26ff. ffhal-00917911f