Output-based LLM-as-a-judge methods using large LLMs achieve near-human correlation with human scores for code translation and generation, but not for code summarization or pairwise comparisons.
ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In recent years, the application of large language models (LLMs) to code-related tasks has gained significant attention. However, existing evaluation benchmarks often focus on limited scenarios, such as code generation or completion, which do not reflect the diverse challenges developers face in real-world contexts. To address this, we introduce ComplexCodeEval, a benchmark designed to assess LCMs in various development tasks, including code generation, completion, API recommendation, and test case generation. It includes 3,897 Java samples and 7,184 Python samples from high-star GitHub repositories, each annotated with function signatures, docstrings, and API references to simulate real development environments. Our experiments across ten LCMs reveal that context improves performance and that data leakage can lead to overestimation, highlighting the need for more accurate evaluations.
citation-role summary
background 1
citation-polarity summary
fields
cs.SE 1years
2025 1verdicts
CONDITIONAL 1roles
background 1polarities
background 1representative citing papers
citing papers explorer
-
Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering
Output-based LLM-as-a-judge methods using large LLMs achieve near-human correlation with human scores for code translation and generation, but not for code summarization or pairwise comparisons.