{"id":"f9d373d2-3026-4765-86df-5f39953eb6f6","arxiv_id":"2412.05288","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"StackEval and StackUnseen offer a multi-language, multi-task coding benchmark from Stack Overflow, plus a human-validated analysis of LLM judges for coding answers.","lead":"This paper introduces StackEval and StackUnseen, two benchmarks built from Stack Overflow posts for testing language models on coding tasks like writing, debugging, and reviewing code. It also studies language models as automatic judges of coding answers, finding that adding a reference answer improves agreement with human judges to about 84% for the best model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 acceptance rates rest on an unnamed LLM judge validated on only 136 StackEval questions; Section 6.2 itself warns judge performance may not transfer to newer data, so the StackUnseen ranking and generalization claims are unsupported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the 136-item judge validation is extrapolated to all 925 StackEval questions, all StackUnseen questions, and all 25 evaluated models, without identifying the judge or showing that accuracy transfers. I agree with that assessment. The paper contributes useful public datasets and a carefully designed judge study, and Section 6.2 honestly acknowledges the temporal limitation. But that concession makes the gap concrete: the paper applies an unvalidated judge to newer data while admitting judge performance may vary on newer questions. The condition for accepting the benchmark's rankings is to provide judge-specific validation on the target distribution and uncertainty intervals. Since the datasets and code are public, a validation study can settle this. The weaknesses are addressable rather than fatal, so the conditional acceptance verdict remains appropriate; no change to the reader's recommendation is needed.","tokens_in":17115,"tokens_out":7154,"duration_ms":70964,"concrete_test":"Run a targeted validation study: sample roughly 120 StackUnseen questions stratified by language and type, plus 120 StackEval questions not in the original 136; have two of the paper's human annotators independently label model outputs from Table 5 with the Table 1 rubric and report Cohen's kappa; then run the actual judge from the released evaluation code on the same inputs using the Figure 8 prompt. Compute per-subset accuracy, the confusion matrix near the score-1/score-2 threshold, and the resulting acceptance rates and model ranks. If StackUnseen judge accuracy is at least 80% and rank correlation with human labels is high, the extrapolation is credible; if accuracy drops materially or the judge's ranks diverge from human ranks, Table 3 and Figure 4 must be re-reported with the specific judge named and with confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every acceptance rate in Table 3, Table 5, and every degradation claim in Figure 4 is a judge score, not an executed-code score. The judge used for these scores is never identified in Section 5.2, and its reliability is established only on 136 human-annotated StackEval questions from Section 3.2. That validation used answers from four generators (GPT-4 Turbo, GPT-3.5 Turbo, CodeLlama-34B, Mistral Medium), while Table 5 scores outputs from 25 models, including O1, Gemma, and Nemotron. Judge agreement can vary by generator and by question novelty. StackUnseen was not in the validation sample and contains version questions absent from the 136 items. The paper itself flags this in Section 6.2: the judge benchmark is a static snapshot, its performance may vary with newer or different types of questions, and continuous updating and re-evaluation are necessary. That is exactly the regime in which StackUnseen results are reported. Additionally, validation accuracy is computed on binarized acceptability, while Table 3 acceptance rates derive from the judge's raw 0-3 score, so near-threshold miscalibration can shift rates even if binary accuracy holds. If judge accuracy on StackUnseen is even 10 points below the reported 84.4%, the 50-70% acceptance-rate cluster in Table 3 could reorder, and the central ranking and generalization findings would not be robust.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two coding-assistance benchmarks derived from Stack Overflow: StackEval (925 questions, 25 languages, four task types) and StackUnseen (recent questions updated semi-annually), together with an LLM-as-a-Judge benchmark of 136 human-annotated questions. The authors evaluate four judge prompt configurations and report that GPT-4 Turbo with chain-of-thought and a reference answer achieves 84.4% accuracy against human acceptability labels. They then use an LLM judge to compute acceptance rates for 24 models on StackEval and StackUnseen, examine performance degradation on recent content, and test for self-preference bias. Datasets and evaluation code are released publicly.","tokens_in":17389,"tokens_out":3696,"duration_ms":35200,"significance":"If the judge-based evaluation is reliable, the benchmarks fill a real gap: they cover multiple languages and open-ended coding-assistance tasks with an absolute acceptability metric, complementing execution-based benchmarks like HumanEval and SWE-Bench. StackUnseen is a potentially useful dynamic resource, and the self-preference analysis with reference answers is a worthwhile contribution to the LLM-judge literature. The public release of datasets and code supports reproducibility. However, the paper's central evaluation claims depend on an LLM judge whose identity and transfer to the full StackEval and StackUnseen settings are not established, so the significance is conditional on the authors addressing the validation gaps described below.","major_comments":[{"comment":"The LLM judge used to produce every acceptance rate in Table 3 and Table 5 is never identified. Section 5.2 only says that responses were assessed with the LLM-as-a-Judge framework and the prompt in Figure 8, but it does not name the model. The validation in Sections 3.2 and 5.1 covers only 136 human-annotated StackEval questions with answers generated by four models, whereas Table 5 scores 25 models including O1, Gemma, and Nemotron, and StackUnseen contains version-type questions not present in the validation set. Without naming the judge and validating it on the actual evaluation conditions, the reported acceptance rates, the ranking conclusions, and the generalization claims in Figure 4 are not reproducible and are not supported by the presented evidence.","section":"Section 5.2, Table 3"},{"comment":"Judge accuracy in Section 5.1 is computed on binarized acceptability, collapsing scores 0-1 versus 2-3, while the acceptance rates in Table 3 use the raw 0-3 scores with the threshold between 1 and 2. A judge can achieve the reported binary accuracy while misclassifying many near-threshold answers, which would directly change acceptance rates without changing the binary accuracy. The paper should report calibration at the threshold or show that the model rankings are robust to reasonable perturbations of the threshold, to establish that the acceptance-rate differences in Table 3 are not artifacts of threshold miscalibration.","section":"Section 5.1 vs Section 5.2"},{"comment":"The human annotation process is described as involving a third domain expert who verified the annotations, but no inter-annotator agreement statistic (e.g., Cohen's kappa) and no number of annotators per item are reported. Since the human labels are the sole anchor for the LLM judge accuracy reported in Table 2, the reliability of the 84.4% figure is not established. Agreement statistics should be reported to show that the labels are stable enough to serve as a ground truth for judge evaluation.","section":"Section 3.2"},{"comment":"The leakage-mitigation claim for StackUnseen is asserted without verification. The paper states that recent questions 'mitigate the issue of data leakage' and 'help prevent potential test-train leakage,' but it does not check whether the evaluated models were trained on these Stack Overflow questions, nor does it report any contamination analysis. Moreover, Section 6.2 explicitly warns that the LLM judge may not transfer to newer or different questions, which is precisely the regime in which StackUnseen results are reported. The authors should include a judge-validation subset drawn from StackUnseen itself and report leakage checks before claiming that StackUnseen evaluates truly unseen content.","section":"Section 3.1 and Section 1"}],"minor_comments":[{"comment":"The word 'Filteration' should be 'Filtration', and the list entry 'V ersion' contains an unintended space.","section":"Section 3.1"},{"comment":"The caption contains 'theStackEval' instead of 'the StackEval'.","section":"Figure 7 caption"},{"comment":"The caption 'The StackEval and StackUnseen Benchmarks' is not informative; it should say that the table reports acceptance rates on StackEval and StackUnseen for representative models.","section":"Table 3 caption"},{"comment":"The correlation matrix reports Pearson correlations without confidence intervals or the number of models; given the small sample of models, the high correlations should be accompanied by uncertainty estimates.","section":"Figure 3"},{"comment":"Table 4 reports only p-values; reporting effect sizes or median score differences would make the self-preference analysis more informative.","section":"Section 5.3"},{"comment":"Reference [16] cites Mistral 7B, but the text in Section 3.2 refers to 'Mistral Medium'; please verify the citation or clarify the model used.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the datasets-and-benchmarks track, but the missing judge identity and the lack of validation on StackUnseen are significant reproducibility issues. The central claims are defensible, and the issues are fixable with additional experiments: name the judge, validate it on a StackUnseen sample, report threshold sensitivity, and add inter-annotator agreement. If the authors cannot name the judge or demonstrate transfer, the acceptance-rate tables should be downgraded to illustrative results rather than benchmark scores."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing about this paper. First, it builds two genuinely useful datasets: StackEval (925 Stack Overflow questions, 25 languages, four task types) and StackUnseen (recent questions, updated periodically), both public with code. Second, the main acceptance-rate tables — the numbers that drive every ranking and the generalization claim — come from an LLM judge that the paper never names. The judge was validated on 136 human-annotated StackEval questions, and that validation is extrapolated to all 925 StackEval and all StackUnseen questions, including version-specific questions from after the validation snapshot. Section 6.2 admits judge performance may vary on newer data. That is the exact regime where StackUnseen results are reported.\n\nThe good parts are real. The 136-question human-annotated LLM-as-judge benchmark is a solid resource, with clear rubrics and a sensible study of prompting variations (reference answer helps more than CoT alone). The self-preference analysis is careful: six judges, Wilcoxon signed-rank tests, and a striking result that reference answers eliminate self-preference in coding tasks. The dataset curation is transparent about Stack Overflow biases. The public artifacts and reproducible evaluation commands are a plus.\n\nThe soft spots are addressable but not minor. (1) Disclose the judge model used in Table 3 and Table 5 acceptance rates. If it is GPT-4 Turbo, say so; if it is an ensemble, specify. (2) Report inter-annotator agreement on the 136 annotations. \"Third expert verified\" is not a statistic. (3) Validate the judge on a sample of StackUnseen questions, or at least report accuracy on a held-out set of recent questions. (4) Test leakage directly: show that models do not score higher on StackUnseen version questions from before their training cutoff, or provide a contamination check. (5) Report raw-score agreement in addition to binarized accuracy — acceptance rates depend on the 1/2 threshold, so binary accuracy can hide near-threshold miscalibration.\n\nIs the central argument intact? The benchmarks are useful and the judge study is informative, but the specific StackUnseen rankings and the \"generalization\" claim are not yet supported by the evidence as presented. That is a load-bearing gap, though a fixable one.\n\nWho should read this: anyone building or using LLM-based evaluation for coding assistants, and anyone designing dynamic benchmarks. It deserves a serious referee, not a desk reject, but the authors should be pushed to close the judge-disclosure and validation gaps before publication. I would conditionally accept after those revisions.","headline":"Useful benchmark artifacts but the main acceptance rates rest on an unnamed LLM judge validated on only 136 questions; the StackUnseen rankings are not yet supported.","tokens_in":17921,"tokens_out":2879,"would_cite":true,"duration_ms":27721,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that coding assistance can be benchmarked at scale with two Stack Overflow-derived datasets, and that an LLM judge using chain-of-thought plus a reference answer reaches 84.4% accuracy in deciding whether a generated…","keywords":["StackEval","LLM-as-a-judge","coding assistance benchmark","Stack Overflow","acceptance score","chain-of-thought","self-preference","StackUnseen"],"falsifier":"Take a fresh random sample of about 100 questions from StackEval that are not in the 136-question judge benchmark, have human experts score the same model outputs on the 0-3 rubric, and compare their binary acceptability labels with the LLM judge's labels; an agreement rate substantially below the reported 84.4%, or agreement that varies sharply by language or task type, would show the benchmark rankings do not generalize.","tokens_in":16921,"feed_emoji":"🧑💻","tokens_out":6848,"duration_ms":59842,"temperature":0.7,"pith_summary":"This paper is trying to establish that coding-assistance quality can be measured at scale with two new benchmarks built from Stack Overflow. StackEval contributes 925 curated question-answer pairs across 25 languages and four task types, and StackUnseen contributes recent questions released after typical LLM training cutoffs, specifically to expose how models handle new material. The paper's central evaluation claim is that an LLM judge given the question, the candidate answer, the accepted Stack Overflow answer, and chain-of-thought instructions can decide whether the candidate answer is acceptable, matching human experts 84.4% of the time on a validated subset of 136 questions. If this is right, coding assistants no longer need human annotation for every evaluation run; acceptance rates from the judge can compare models across languages, task types, and difficulty levels. The reader should care because the benchmarks measure the kind of day-to-day assistance developers actually ask for, not just function completion.","feed_headline":"Reference answer lifts LLM coding judge to 84.4 percent","feed_subtitle":"StackEval puts 925 Stack Overflow questions in 25 languages across four coding task types to the test.","key_machinery":"The load-bearing mechanism is the acceptance-score rubric implemented as an LLM judge prompt. A generated answer is scored 0 to 3 across accuracy, completeness, and relevance, with scores 2 and 3 counted as acceptable; before scoring, the judge is prompted to analyze the question, the accepted Stack Overflow reference answer, and the candidate answer step by step. The reference answer functions as an objective anchor, and the human-annotated 136-question subset is the calibration step that converts the judge's scores into a validity claim.","core_discovery":"The central discovery is that the reference answer is the strongest ingredient in an LLM judge for coding assistance. Across six judge models, adding the accepted Stack Overflow answer to the prompt improved evaluation accuracy for every model, while chain-of-thought alone sometimes hurt; the best configuration, chain-of-thought plus reference answer, reached 84.4% mean accuracy for GPT-4 Turbo. Using this judge, the paper reports acceptance rates for 24 models on StackEval, where O1 Preview leads at 95.5%, and on StackUnseen, where the same model leads at 83.0%; every tested model's acceptance rate drops on the recent questions, and the drop is smaller for models that score higher on StackEval. The paper also finds no statistically significant self-preference bias in coding evaluation when a reference answer is provided, and only marginal evidence of one without it.","pith_inferences":["A natural follow-up is to run the same reference-answer judge on other objectively graded domains, such as mathematics or legal Q&A, to see whether the anchoring effect that removes self-preference generalizes beyond code.","Because StackUnseen samples questions from after model training cutoffs, the size of the acceptance-rate drop could be tested as a practical leakage detector for future base models.","The judge's accuracy was measured on a comparatively small annotated subset; re-running the benchmark on a larger human-annotated sample is the direct way to check whether the 84.4% figure holds.","The dataset structure invites a challenge-suite interpretation: teams could filter by language and task type to target specific weaknesses, such as advanced C++ debugging, and track improvement over releases."],"forward_implications":["Combining chain-of-thought with a reference answer is the recommended judge configuration, and reference answers matter more than reasoning prompts alone.","Overall model rankings on StackEval, StackUnseen, HumanEval, and ChatBot Arena are highly correlated, so a single multi-task benchmark can summarize relative capability.","Models with higher StackEval acceptance rates show smaller performance drops on StackUnseen, implying that strong performance on established problems predicts better adaptation to new problems.","LLM judges do not exhibit statistically significant self-preference when a reference answer is present, so automated coding evaluations can be run without a bias correction for self-scoring.","Because StackUnseen is updated semi-annually, it can serve as a re-usable check for whether new model releases have improved on post-training-cutoff content."],"supporting_citations":[{"why":"Supplies the 2023 developer survey whose language-popularity weights determine how many questions each language contributes to StackEval and StackUnseen.","marker":"[26]"},{"why":"Provides the LLM-as-a-judge framework with over-80% human agreement that this paper adapts by adding reference answers and an absolute acceptance rubric.","marker":"[41]"},{"why":"Supplies the chain-of-thought prompting technique tested as one component of the judge configuration.","marker":"[37]"},{"why":"HumanEval is the dominant single-language coding benchmark this paper contrasts with when motivating multi-language, multi-task evaluation.","marker":"[9]"},{"why":"SWE-Bench supplies the real-world code-editing benchmark used to frame where StackEval sits among coding evaluations.","marker":"[17]"},{"why":"Supports the benchmark-leakage and exposure-bias concern that motivates StackUnseen's recent-content design.","marker":"[18]"},{"why":"Establishes the self-preference phenomenon in LLM evaluators whose presence this paper tests in coding tasks.","marker":"[27]"}],"fun_headline_variants":["Reference answer boosts LLM coding judge to 84%","StackEval: Reference answer key to LLM judge accuracy","Accepted answer lifts LLM coding judge performance","LLM judge improves with Stack Overflow reference answers","StackEval shows reference answers strengthen coding judges"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The acceptance rates reported for StackEval and StackUnseen are produced by an LLM judge that the paper never identifies, and the judge's reliability was validated on only 136 human-annotated questions before being extrapolated to all 925 StackEval and all StackUnseen questions.","fun_headline_variants_meta":{"raw":{"variants":["Reference answer boosts LLM coding judge to 84%","StackEval: Reference answer key to LLM judge accuracy","Accepted answer lifts LLM coding judge performance","LLM judge improves with Stack Overflow reference answers","StackEval shows reference answers strengthen coding judges"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000139,"raw_usage":{"total_tokens":1118,"prompt_tokens":865,"completion_tokens":253,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":481,"completion_tokens_details":{"reasoning_tokens":178}},"tokens_in":481,"tokens_out":253,"duration_ms":3038,"temperature":1.0,"reasoning_tokens":178,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:36:31.325267+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh random sample of about 100 questions from StackEval that are not in the 136-question judge benchmark, have human experts score the same model outputs on the 0-3 rubric, and compare their binary acceptability labels with the LLM judge's labels; an agreement rate substantially below the reported 84.4%, or agreement that varies sharply by language or task type, would show the benchmark rankings do not generalize.","supporting_citations":[{"cited_title":"Stack overflow developer survey 2023, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the 2023 developer survey whose language-popularity weights determine how many questions each language contributes to StackEval and StackUnseen."},{"cited_title":"Gravity theories with local energy-momentum exchange: a closer look at Rastall-like gravity","cited_arxiv_id":"2209.01186","evidence_quote":"Supports the benchmark-leakage and exposure-bias concern that motivates StackUnseen's recent-content design."},{"cited_title":"Bowman, and Shi Feng","cited_arxiv_id":null,"evidence_quote":"Establishes the self-preference phenomenon in LLM evaluators whose presence this paper tests in coding tasks."}],"review_version":1}