{"id":"a3f0ef77-8701-4f1f-9288-62fb9124416d","arxiv_id":"2411.18019","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FAUN-Eval is a curated benchmark of 300 real GitHub issue-PR pairs that scores LLMs separately on question answering, fault localization, and code editing.","lead":"A new benchmark, FAUN-Eval, tests large language models on three subtasks of GitHub issue solving: answering the issue, locating the faulty file, and editing the code. On 300 curated issue-PR pairs, the top model differed by task, and issue titles sometimes misled models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fault-localization ground truth equates 'file changed in single-file PR' with 'fault location'; this unvalidated proxy is the load-bearing risk to the benchmark's central claim.","rationale":"The paper is a benchmark-construction and evaluation paper, and its central value is a reusable dataset with fine-grained subtask labels. If the fault-localization labels are not actually fault locations, then the benchmark's most distinctive contribution is compromised: one of its three tasks is mislabeled, and the specific model rankings for that task do not support the stated conclusions. The reader's weakest assumption matches this concern almost exactly, and I see it as the most load-bearing risk among several real methodological weaknesses. The other weaknesses (CodeBLEU not measuring functional correctness, QA reference replies not necessarily authoritative, no error bars) are genuine but secondary: CodeBLEU and QA scores would require more careful interpretation, while a wrong localization ground truth invalidates the task definition itself. I also considered raising the lack of statistical testing as the primary concern, since differences such as 0.65 vs. 0.64 EM are likely not significant; however, even with significance testing, the benchmark would still measure the wrong construct if the ground truth is invalid. The proposed test is feasible because the dataset and repository states are released, and it directly settles whether the changed-path proxy agrees with human judgments of fault location. I therefore recommend keeping the reader's CONDITIONAL verdict unchanged, with the condition being the validation of this ground truth.","tokens_in":19910,"tokens_out":5192,"duration_ms":52742,"concrete_test":"Sample 50 randomly selected FAUN-Eval fault-localization entries. Give two developers, blind to the PR diff, the issue text and the base-commit folder tree (or repository checkout), and ask each to list the file(s) they judge to contain the root cause of the issue. Compute agreement between the PR-changed path and the majority developer annotation. If agreement falls below roughly 80%, or inter-annotator agreement is low, the Section 3.4.2 labels operationalize 'changed-file prediction' rather than 'fault location,' and the fault-localization rankings in Table 4 should be reinterpreted or the ground truth revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central claim that FAUN-Eval measures fine-grained fault-localization capability depends on an unvalidated operationalization: the single file modified by the merged PR is treated as the unique fault location. Section 3.3.2 ('File-Changed Count') retains only PRs touching one file, and Section 3.4.2 then makes the changed path the ground-truth output for the fault-localization task. But Section 1 defines fault localization as pinpointing 'the specific file or section of code responsible for the reported issue.' A PR-changed file need not be the cause: the fix can live in a different file from the one where the symptom appears, or the PR can be a feature addition, refactor, or workaround for which 'fault' is not well defined. The single-file filter also removes exactly the multi-file cases that the paper's motivation emphasizes, so Table 4's EM scores (e.g., GPT-4o 0.65, GM-Pro 0.64) measure 'can the model retrodict which single file a developer edited on an easy subset' rather than fault localization as defined. Findings 1 and 3 are substantially built on this task's scores, so the benchmark's headline claims inherit this risk. The QA-reply-authority and CodeBLEU issues are real but secondary; the localization ground truth is the load-bearing one because it determines whether a full third of the benchmark measures what it claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FAUN-Eval is a benchmark intended to measure fine-grained issue-solving capabilities of LLMs across three subtasks: code question answering (QA), fault localization, and code editing. The authors crawl 30 popular GitHub repositories, pair issues with merged pull requests, apply a multi-stage filtering and semantic-validation pipeline, and release 300 entries. Ten proprietary and open-source LLMs are evaluated with generic text/code similarity metrics, producing per-task rankings and ablation findings such as 'the issue body is crucial for QA and fault localization' and 'issue titles can mislead models.' The paper's contribution is mainly empirical: a reusable dataset and a decomposition of end-to-end issue resolution into three evaluable stages.","tokens_in":20202,"tokens_out":4175,"duration_ms":40368,"significance":"If the benchmark's operationalizations are accepted, FAUN-Eval would be a useful complement to end-to-end benchmarks like SWE-Bench, because it provides per-subtask scores that can guide model selection and identify where models fail. The data-construction pipeline is detailed and transparent, and the authors explicitly release the dataset and model outputs, which supports reproducibility. The three-task framing is sensible and maps onto a natural workflow. However, the validity of the benchmark's headline findings depends on several unvalidated choices, especially the use of the single file changed in a merged PR as the ground-truth fault location, and the reliance on exact-match and code-similarity metrics that conflate instruction following with capability. These issues affect a large share of the reported results, so the paper requires substantial revision before its claims can be accepted.","major_comments":[{"comment":"The fault-localization ground truth is not validated as a fault location. Section 3.3.2 ('File-Changed Count') keeps only PRs that modify a single file, and Section 3.4.2 defines the task output as the 'changed path' of that PR. But Section 1 defines fault localization as pinpointing 'the specific file or section of code responsible for the reported issue.' A file changed by a merged PR is not necessarily the file responsible for the issue; the fix may reside in a different file than the one where the symptom appears, and some merged PRs are feature additions, refactors, or workarounds rather than causal fixes. The single-file filter also removes the multi-file cases that the paper's motivation emphasizes. Consequently, the EM scores in Table 4 (e.g., GPT-4o 0.65, GM-Pro 0.64) measure retrodiction of a developer's edited file on an easy subset, not fault localization as defined. Since Findings 1 and 3 rely substantially on this task, the authors should either validate the fault-location proxy (e.g., by manual annotation, bug-introducing-commit analysis, or test-based verification) or reframe the task as 'changed-file prediction' and temper the corresponding claims.","section":"§3.3.2, §3.4.2, §5.1 (Table 4)"},{"comment":"The fault-localization metric conflates instruction following with localization ability. The paper uses Exact Match as the sole metric, and Table 4 reports EM = 0 for CodeQwen-1.5-Chat and DeepSeek-Coder 33B because these models output the correct path along with explanatory text. The paper itself notes this behavior in Section 5.1, but it still reports these as zero localization scores. This makes the FL rankings dependent on a strict output-format requirement rather than on the model's ability to identify the right file. The authors should either post-process and parse the model outputs (e.g., extract the first file path), use a containment or normalized-match metric, or report an instruction-following metric separately from localization accuracy. Without this, the zero scores and the averaged FL numbers in Table 4 and RQ2 are not comparable across models.","section":"§4.4, §5.1 (Table 4)"},{"comment":"The QA ground truth is underspecified. The QA task treats the issue title and body as input and 'the body of the response from the reply' as the reference output. It is not clear how a single reply is selected from the issue thread, whether the selected commenter is a maintainer or otherwise authoritative, or whether the chosen response actually answers the issue. The semantic check in Section 3.3.3 validates issue-PR pairs, not QA reply pairs. As a result, the QA scores may measure similarity to an arbitrary comment rather than to a correct or representative answer. The authors need to specify and validate the reply-selection procedure, and ideally include inter-annotator agreement or a manual correctness check on the QA gold answers.","section":"§3.4.1, §4.2"},{"comment":"CodeBLEU is not a measure of whether an edit resolves the issue. The code-editing task is evaluated exclusively with CodeBLEU, which compares lexical, AST, and data-flow similarity to the reference patch. A generated edit that is plausible but functionally incorrect can receive a high CodeBLEU score if it resembles the gold patch, while a correct but differently structured fix can be penalized. Since no tests or functional checks are used, the reported code-editing scores do not establish issue-solving capability. The paper should either add a functional verification component (e.g., running repository tests or applying the patch and checking behavior), or explicitly limit the claim to 'patch similarity' rather than 'code editing capability.'","section":"§4.4, §5.1 (Table 4)"},{"comment":"The ranking claims lack uncertainty quantification. With only 300 entries and no confidence intervals, significance tests, or per-repository breakdowns, small differences in Table 4 (e.g., QA BLEU 0.0943 vs. 0.0948; FL EM 0.6500 vs. 0.6400) and the ordering crossovers in Figure 3 may be noise. The paper states strong conclusions such as 'the top-performing models differ across the different tasks' and 'different models may excel at processing contexts of varying lengths' without statistical support. The authors should add confidence intervals, paired tests (e.g., bootstrap or McNemar for EM), and ideally report variance across repositories.","section":"§5.1, §5.3 (Table 4, Figure 3)"}],"minor_comments":[{"comment":"'Haribin Institute of Technology' should be 'Harbin Institute of Technology.'","section":"Author affiliations"},{"comment":"The word 'compability' appears to be a typo for 'compatibility.'","section":"§2.2.1"},{"comment":"The paper states that test files are excluded from the dataset, but some issues may legitimately involve test code or test configuration; the rationale for excluding them (beyond simplicity) could be stated more explicitly.","section":"§3.3.2, §3.4.2"},{"comment":"The text says 'some models like DeepSeek-Coder-V2 show enhanced Exact Match scores after discarding the body,' but the corresponding Table 5 condition is 'IB' (issue body only), which removes the issue title, not the body. The wording appears to be inconsistent with the table.","section":"§5.2 (Table 5)"},{"comment":"Table 1 lists FAUN-Eval-QA and FAUN-Eval-fix separately, while the abstract and introduction describe FAUN-Eval as containing 300 entries. It would help to clarify how many issues are shared between the QA set and the fault-localization/code-editing sets, and how many QA entries have a corresponding fix pair.","section":"§2 (Table 1)"},{"comment":"The threats-to-validity section does not discuss the risk that using GPT-4o during semantic filtering may bias the dataset toward issues that GPT-4o can understand; the manual review mitigates but does not eliminate this concern.","section":"§6.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is positioned as a new benchmark with a public dataset, which fits the scope of SE venues. The main risk is not the construction pipeline but the validity of the fault-localization ground truth and the interpretation of metric scores. The authors should be asked to either validate the 'changed path = fault location' assumption or rename the task accordingly, and to add post-processing or side-by-side metrics so that instruction-following failures are not reported as localization failures. The QA reply selection also needs clarification before the QA findings can be trusted. I do not see evidence of fabrication or deliberate circularity; the GPT-4o filtering concern is real but secondary because the authors performed a manual review. The anonymous data link and released outputs are a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing worth knowing: this paper ships a real resource. FAUN-Eval gives you 300 issue–PR pairs from 30 repos, split into QA, fault localization, and code editing, with model outputs posted and a sensible curation pipeline. The filtering is careful where it counts (merged PRs only, one-to-one fix pairs, doc/test removal, tree-sitter substance check, manual review on top of the GPT-4o semantic pass). If you care about model selection for issue-solving subtasks, this is a usable testbed, and the top-model-differs-by-task finding is plausible.\n\nThe load-bearing problem is the fault-localization ground truth. The paper defines fault localization as pinpointing the file or section responsible for the issue, then operationalizes it as the single file the merged PR happened to change. Those are not the same thing. A fix can live in a different file from where the symptom appears, and PRs can be refactors, features, or workarounds where \"fault\" is ill-defined. The single-file filter then deletes the multi-file cases that the motivation calls important, so the EM scores in Table 4 measure \"can the model retrodict which file a developer edited on an easy subset,\" not localization as advertised. Findings 1 and 3 lean on these scores, so the headline claims inherit the risk. I don't see any validation that the changed file is actually the fault location, and the paper doesn't discuss this. The stress-test note is right.\n\nThe other soft spots are real but secondary. EM for fault localization gives zero to CodeQwen-1.5-Chat and DeepSeek-Coder 33B for outputting the correct path plus explanation; that is a parsing problem, not a capability score. CodeBLEU for code editing measures patch similarity, not whether the edit fixes the bug. The QA reference is one commenter's reply in the thread, assumed authoritative. No error bars or significance tests on 300 entries. And GPT-4o was used in semantic filtering before being evaluated on the result; manual review reduces but does not eliminate that circularity.\n\nNone of these kill the benchmark as a curated artifact. They do mean the empirical claims need tempering, and the FL ground truth needs validation before I would trust the rankings. This paper deserves a serious referee, but the referee should push hard on the localization operationalization and on reporting parse-tolerant metrics. I would not cite the benchmark in my own work until that proxy is validated; I might bring it to a reading group as a cautionary case study in benchmark construction.","headline":"A genuinely useful fine-grained benchmark artifact whose fault-localization ground truth conflates 'file changed in the merged PR' with 'fault location'—the central claims need that proxy validated before the rankings are trusted.","tokens_in":20747,"tokens_out":2124,"would_cite":false,"duration_ms":21143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that evaluating issue solving as three separate subtasks—QA, fault localization, and code editing—reveals task-specific LLM strengths that end-to-end benchmarks miss, and introduces FAUN-Eval, a 300-entry benchmark that…","keywords":["FAUN-Eval","LLM evaluation","fault localization","code editing","question answering","GitHub issues","software engineering benchmark","prompt sensitivity"],"falsifier":"Take a random sample of FAUN-Eval fix pairs and have an independent panel mark the actual root-cause file(s) by reading the issue and the PR diff; if a substantial share of PRs touch a file that is not the root cause (for example, a symptom-level workaround), then the fault-localization scores rest on wrong labels. A second check: run the code-editing task with ground truth defined as all files touched by the PR rather than the single filtered file and see whether model rankings change.","tokens_in":19728,"feed_emoji":"🛠️","tokens_out":4858,"duration_ms":40901,"temperature":0.7,"pith_summary":"This paper argues that end-to-end benchmarks like SWE-Bench cannot show where LLMs fail inside the issue-solving pipeline, and offers FAUN-Eval, a benchmark that splits the pipeline into three scored subtasks: answering the issue, locating the faulty file, and editing the code. The dataset is built from 300 real GitHub issue–pull-request pairs across 30 repositories and five languages, with automated and human validation. Ten models are evaluated, and the results show that the best model differs by subtask: Gemini-1.5-Pro leads QA and fault localization while DeepSeek-Coder-V2 leads code editing. The paper also reports that issue titles can actively mislead models and that longer inputs generally hurt performance, especially for smaller models. If these findings hold, benchmark users get per-subtask scores for model selection rather than a single end-to-end number.","feed_headline":"Benchmark splits issue solving; no single LLM leads all subtasks","feed_subtitle":"QA, fault localization, and code editing each favor different models—and issue titles can misdirect.","key_machinery":"The machinery is a four-stage dataset construction pipeline: it crawls GitHub metadata via REST and GraphQL, pairs issues with PRs using cross-reference events and GitHub's closing keywords, filters pairs down to merged, one-to-one, single-file, non-documentation code changes, and runs a semantic check with an LLM plus manual review. The three task formats are then derived: QA uses issue title and body as input and a commenter's reply as reference; fault localization uses issue text plus a pruned repository folder tree as input and the changed file path as ground truth; code editing uses issue and PR text plus the pre-change file as input and the PR patch (SEARCH/REPLACE blocks) as reference. The central object that carries the benchmark is the issue–PR fix pair, and the central assumption is that the single changed file in the merged PR is the fault location.","core_discovery":"The central claim is that fine-grained evaluation of issue solving is feasible and informative: FAUN-Eval provides 300 validated issue–PR entries formatted as three tasks, and using it reveals that no single model dominates all subtasks. On fault localization, GPT-4o reaches 0.65 Exact Match while two open models score zero because they ignore output-format instructions. On code editing, Gemini-1.5-Flash and DeepSeek-Coder-V2 outperform GPT-4 and GPT-4o; on QA all models score low, with the best average around 0.18. The paper concludes that task-specific strengths vary, closed-source status and parameter count do not guarantee superiority, and issue titles sometimes mislead models, so the benchmark can guide which model to deploy for each stage of issue resolution.","pith_inferences":["Beyond the paper, the single-file restriction likely under-reports the hardest real-world cases: multi-file fixes are common, and excluding them may make FAUN-Eval's localization and editing scores optimistic for practice.","The choice of a single commenter's reply as QA ground truth is a weak anchor; future extensions could aggregate multiple maintainer replies or use the eventual fix as the answer target.","The title-misleads finding suggests a testable extension: systematically rewrite issue titles and measure how much QA and localization scores move, which would quantify how much models rely on framing versus content.","A routing system that sends each issue to the best model per subtask, based on estimated input length, could beat any single model; FAUN-Eval provides the per-subtask scores needed to test that."],"forward_implications":["Model selection should be subtask-specific: a pipeline could pick Gemini-1.5-Pro for understanding and locating the issue but DeepSeek-Coder-V2 for producing the edit.","Prompt and benchmark design should weigh issue titles carefully, since removing titles improved QA performance across all tested models and helped some models in localization.","Input-length effects mean evaluations should report performance stratified by issue length; a model that tops a short-text leaderboard may not lead on long issue threads.","Instruction-following is a measurable failure mode: models that answered fault localization with explanations instead of a path scored zero, so output-format compliance is part of capability.","Open-source and smaller models can beat proprietary and larger models on specific subtasks, so budget-conscious deployments have viable choices."],"supporting_citations":[{"why":"SWE-Bench is the end-to-end benchmark that FAUN-Eval contrasts with, motivating the need for subtask-level evaluation.","marker":"[17]"},{"why":"HumanEval represents the small self-contained coding benchmarks whose limits motivate real-world issue evaluation.","marker":"[4]"},{"why":"CodeApex is a code QA benchmark compared in Table 1; it lacks real-world issue-solving pairs.","marker":"[10]"},{"why":"CodeQA supplies the QA-pair baseline that FAUN-Eval's issue-derived QA task is contrasted against.","marker":"[23]"},{"why":"CS1QA is another code QA dataset compared in Table 1; it is course-chat-based rather than issue-based.","marker":"[20]"},{"why":"Defects4J is a code-repair benchmark used as a comparison for fault localization and repair datasets.","marker":"[18]"},{"why":"StarCoder's methodology is cited for the event-count filtering criterion used in issue filtering.","marker":"[31]"}],"fun_headline_variants":["No single LLM wins all issue-solving tasks in new benchmark","FAUN-Eval: 300 GitHub issues reveal no model dominates all subtasks","Task-specific LLM champions in QA, localization, and editing","Issue titles can mislead LLMs; benchmark ranks per-task models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark assumes that the one file changed by the merged pull request is the true and complete location of the fault, and that the issue text plus a folder tree is enough context for a model to find it; if a fix touches a different file than the actual root cause, or needs multiple files, the ground-truth labels misstate localization ability.","fun_headline_variants_meta":{"raw":{"variants":["No single LLM wins all issue-solving tasks in new benchmark","FAUN-Eval: 300 GitHub issues reveal no model dominates all subtasks","Task-specific LLM champions in QA, localization, and editing","Issue titles can mislead LLMs; benchmark ranks per-task models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000432,"raw_usage":{"total_tokens":2230,"prompt_tokens":997,"completion_tokens":1233,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1157}},"tokens_in":613,"tokens_out":1233,"duration_ms":10773,"temperature":1.0,"reasoning_tokens":1157,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:35:20.041432+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of FAUN-Eval fix pairs and have an independent panel mark the actual root-cause file(s) by reading the issue and the PR diff; if a substantial share of PRs touch a file that is not the root cause (for example, a symptom-level workaround), then the fault-localization scores rest on wrong labels. A second check: run the code-editing task with ground truth defined as all files touched by the PR rather than the single filtered file and see whether model rankings change.","supporting_citations":[],"review_version":1}