HumanEval Pro and MBPP Pro test whether LLMs can first solve a base function and then call it inside a more complex solution, and frontier models lose about 10 to 15 points on this task.
Title resolution pending
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.SE 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation
HumanEval Pro and MBPP Pro test whether LLMs can first solve a base function and then call it inside a more complex solution, and frontier models lose about 10 to 15 points on this task.