Assessing and Improving the Representativeness of Code Generation Benchmarks Using Knowledge Units (KUs) of Programming Languages – An Empirical Study
Large Language Models (LLMs) such as GPT-4, Calude, LLaMA and StarCoder have achieved notable progress in code generation tasks, with benchmark datasets like HumanEval and MBPP serving as the primary means of evaluating their code-generation performance. Effective code generation requires models to understand and correctly apply the underlying programming language concepts (e.g., concurrency). If the language concepts exercised in benchmark tasks are not representative of those employed in real-world software projects, evaluation results based on these benchmarks may yield incomplete and less reliable conclusions. Yet, despite the importance of representativeness, no prior work has systematically examined whether benchmarks reflect the language concepts that developers employ in real-world software projects.
To fill this gap, this paper presents the first systematic study of the representativeness of benchmarks in terms of language concepts, using Knowledge Units (KUs) of programming languages as an analytical lens for understanding those concepts. A KU is a cohesive set of key capabilities provided by the constructs (e.g., try/except and await) and APIs (e.g., Exception and threading.Thread.) of a given programming language. We analyze two widely used code generation benchmarks in Python - HumanEval and MBPP - alongside 30 real-world Python projects to investigate KU coverage. Our findings reveal that only half of the identified 20 Python KUs are represented in each benchmark, while real-world projects employ all of them to various degrees. Although the KU distributions in real-world software projects are not uniform -- since certain KUs (e.g., variables and control flow) naturally occur more frequently -- their distributions are substantially balanced and reflect realistic usage patterns. In contrast, benchmark tasks exhibit a skewed KU distribution, with a small subset of KUs dominating most tasks.
To address this misalignment, we develop a prompt-based LLM framework that enhances these benchmarks with newly synthesized tasks to rebalance their KU distributions and align them with the KU distribution observed in real-world projects. Using this framework, we generate 440 synthesized tasks and augment the original benchmarks by combining these new tasks with their existing task sets. Our results present that the augmented benchmarks substantially improve KU coverage and more closely align with real-world KU distributions, resulting in an improvement of over 60% in distributional alignment compared to the original benchmarks. Evaluating state-of-the-art LLMs on these augmented benchmarks reveals a consistent and statistically significant performance drop (12.54–44.82% with a large effect size), highlighting that the original benchmarks were over-estimating the LLMs' performance on code generation tasks due to their bias on only few KUs instead of the real KU distribution.
Our findings provide valuable insights for researchers and practitioners aiming to design more comprehensive benchmarks, to evaluate an LLM's performance more effectively, and to better understand model capabilities through the lens of KUs, ultimately guiding the improvement of LLMs.