A new benchmark, CoCoNUT, shows that LLMs that can generate working Python code often fail to list the exact lines of code that execute, especially for long or advanced programs.
EvalPlus Leaderboard — evalplus.github.io,
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
dataset 1
citation-polarity summary
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1roles
dataset 1polarities
use dataset 1representative citing papers
citing papers explorer
-
CoCoNUT: Structural Code Understanding does not fall out of a tree
A new benchmark, CoCoNUT, shows that LLMs that can generate working Python code often fail to list the exact lines of code that execute, especially for long or advanced programs.