{"id":"dba98aed-ff8b-4067-a5e0-defa6c2a02f1","arxiv_id":"2507.00699","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MultiCodeIF introduces a 2,021-task, 14-language benchmark with 27 constraint types to evaluate code instruction following, finding that multi-level constraints sharply reduce model success and iterative feedback substantially restores it.","lead":"MultiCodeIF is a new benchmark that tests how well AI coding assistants follow detailed programming instructions across 9 constraint categories and 14 programming languages. It adds multi-level and multi-turn feedback tasks to reveal that models satisfy only 63% of constraints on average, but improve to 83% with iterative feedback.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported satisfaction rates hinge on an unvalidated judge and rule-based evaluator; without a human-agreement or error analysis, the headline 63.0%, 18.8% HSR, and 83.4% repair numbers are not yet supported.","rationale":"I focused on whether the central empirical claims could be artifacts of the measuring instrument. The reader's weakest assumption identifies the same point, and I agree. I checked for a more fundamental flaw—internal inconsistency in the constraint taxonomy, contradictory numbers, or an impossible derivation—and did not find one. The most concrete additional problem is that the abstract's multi-level drop (54.5% to 18.8%) comes from a single constraint category (Data Structure) and a single model column (GPT-4o in Table V), not from the full multi-level benchmark; however, this is a reporting and scoping error that could be fixed without changing the benchmark design. The judge-validity gap is load-bearing because it sits between the benchmark and every quantitative conclusion. The paper deserves credit for releasing data, prompts, and validation scripts, but the evaluation pipeline itself has no published calibration. A human agreement study is the standard and sufficient remedy, so the appropriate verdict remains conditional rather than reject.","tokens_in":15680,"tokens_out":6746,"duration_ms":79288,"concrete_test":"Re-score a stratified sample of 300 model outputs (50 per constraint group, drawn from RQ1, RQ3, and RQ4) with two independent human annotators who see only the task prompt and the model output and label each constraint as satisfied or unsatisfied; then compare with the pipeline labels, reporting per-type precision/recall and Cohen's kappa. If macro-average accuracy shifts by more than 3–5 points, or if the L1-to-L4 HSR drop for Data Structure is materially different after relabeling, the headline claims need recalibration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every headline number—63.0% vs 44.8% average satisfaction, the 54.5% to 18.8% HSR drop, and the 63.0% to 83.4% self-repair gain—depends on the binary constraint-satisfaction labels produced by the pipeline in Section IV-B.1. That pipeline mixes rule-based scripts (Tree-sitter, Guesslang) with GPT-4-Turbo as an LLM judge for a substantial subset of the 27 constraint types, but the paper reports no false-positive/false-negative analysis, no per-type precision/recall, and no human agreement study for either evaluation mode. This matters because the judge is used for exactly the semantic and abstract constraints where model differences are largest, e.g., Code Quality ranges from 2% for Qwen3-1.7B to 67% for Claude in Table IV. The same GPT-4-Turbo also generated the tasks, so there is no independence between task construction and scoring. Rule-based checks are not self-certifying either: constraints such as Data Structure Scale or Data Structure Operation require semantic interpretation that Tree-sitter alone cannot establish. Since HSR is a conjunction over constraints, even modest per-constraint judge error is amplified when moving from L1 to L4, so the headline degradation could be partly a measurement artifact. The Threats section acknowledges generation noise but does not address evaluation-instrument validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MultiCodeIF, a benchmark for evaluating instruction-following in code generation, with a taxonomy of 9 categories and 27 constraint types. Tasks are automatically generated by an LLM pipeline (ConstraGen) from real-world seed code, with manual validation, and include 1,526 single-level and 495 multi-level tasks across 14 (or 15) programming languages. The authors evaluate six LLMs and report findings on constraint satisfaction across categories, the drop in hard satisfaction with multi-level constraints, and improvement via iterative self-repair. The paper claims that MultiCodeIF is a reusable, evolvable, feedback-sensitive benchmark, and it releases the dataset and evaluation pipeline publicly.","tokens_in":15953,"tokens_out":7934,"duration_ms":80918,"significance":"If the evaluation pipeline is valid, MultiCodeIF is a valuable resource: it operationalizes a fine-grained constraint taxonomy, provides a large multi-language dataset, and enables multi-turn self-repair evaluation. The public release of tasks, evaluation scripts, and prompts is a strength, as is the attempt to combine rule-based and LLM-based checks. The headline behavioral findings—performance gaps across models, difficulty of implicit constraints, and self-repair gains—are plausible and would be of interest to the LLM-coding community. However, the paper's central measurements depend on an unvalidated evaluation instrument, and several headline claims are overgeneralized; these issues are load-bearing and need to be addressed before the benchmark's reported numbers can be taken at face value.","major_comments":[{"comment":"The accuracy labels used throughout the paper depend on the rule-based and GPT-4-Turbo-based evaluation pipeline described in Section IV-B.1, but no validation of this instrument is reported: there is no human agreement study, no per-constraint-type precision/recall, and no false-positive/false-negative analysis. This is especially concerning for the constraint types judged by LLM (e.g., Code Quality, Algorithm Type, Scenario), and for the multi-level RQ3 where HSR is a conjunction of several binary decisions, so per-constraint judge error compounds. The Threats section discusses generation noise but does not address evaluation-instrument validity. I request an inter-annotator study on a stratified sample of outputs (per constraint type and per model) using the same evaluation rubrics, plus a report of agreement and error analysis; until then, the headline numbers (63.0%, 18.8%, 83.4%) are not supported.","section":"Section IV-B.1, Table II, Section VI"},{"comment":"The abstract and introduction claim that 'tasks with multiple hierarchical constraints significantly reduce model success rates, from 54.5% in single-level to just 18.8% in multi-level scenarios.' Table V shows that these numbers are GPT-4o's HSR on the Data Structure category only; the multi-level experiment evaluates only Data Structure constraints, even though Section III-C Step 7 says multi-level variants were constructed for Interface Specification, Data Structure, and Code Quality. The claim as stated is not representative of the benchmark as a whole. Please report multi-level SSR/HSR for the other two constraint categories and restrict the abstract claim to the evaluated data.","section":"Abstract, Section I, Section IV-D (RQ3)"},{"comment":"The abstract reports that 'average constraint satisfaction rises from 63.0% to 83.4% over four iterative refinement rounds.' Table VI shows this is Claude-3-7-Sonnet's IFRepair@4; GPT-4o reaches 78.6%, and the other four models were not evaluated in RQ4. Also, IFRepair@k is defined as a hard satisfaction rate (all constraints satisfied), so calling it 'average constraint satisfaction' is misleading because the RQ1 metric is per-constraint accuracy. Please specify the model, the metric, and the dataset used for RQ4 (the paper does not state which task set is used) when making this claim.","section":"Abstract, Section IV-E (RQ4)"},{"comment":"Data Structure constraints are labeled 'Rule' evaluation in Table II, and Section IV-B.1 states that rule-based static tools (Tree-sitter, Guesslang) verify constraints such as 'the list cannot contain more than 100 elements' and 'must use the pop() and top() operation in stack.' Tree-sitter alone cannot establish Data Structure Scale or Operation semantics without runtime analysis or additional interpretation. Please clarify the actual implementation of these checks, and if any of them fall back on the LLM judge, include them in the validation study requested in the first major comment.","section":"Table II, Section IV-B.1"},{"comment":"Section IV-E does not specify which tasks are used in the self-repair experiment. The initial scores (63.0% for Claude, 62.1% for GPT-4o) match the RQ1 single-level averages, but it is not stated whether all 1,526 single-level tasks, all 2,021 tasks, or a subset is used. Also, the feedback is constructed from the same evaluation pipeline, so any evaluator bias propagates into the repair loop. Please specify the task set and provide the evaluation prompts used for feedback.","section":"Section IV-E (RQ4)"}],"minor_comments":[{"comment":"The paper claims 14 programming languages in the abstract and Table I, but Section III-D lists 9 popular languages and 6 less popular languages (15 total). Please correct the count.","section":"Section III-D"},{"comment":"Category names change between the table (e.g., Environment, Code Quality, Scenario) and the text (e.g., Syntax, Non-Functional Requirements, Situation). Please use a single naming convention throughout.","section":"Section IV-B.2"},{"comment":"The sentence 'the code generation domain remains less underexplored' should be 'remains underexplored' or 'remains less explored'; the current phrasing is a double negative.","section":"Section I"},{"comment":"The citation for GPT-4-Turbo is [8], which is the MBPP paper; the intended reference is likely [23] (GPT-4 technical report).","section":"Section III-B, Step 1"},{"comment":"The notation in the IFRepair@k definition, I[V_{c∈C} sat(c, y_i^(k))], is awkward; please define sat(c,·) and use consistent symbols with RQ3 (where I[sat(c)] is used).","section":"Section IV-E"},{"comment":"The text refers to Figure 5 as showing 'the distribution of constraint counts,' but the figure appears to be a general overview diagram; please align the caption and the text.","section":"Section III-D, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The benchmark construction effort is solid and the public release is commendable, but the lack of evaluation-instrument validation is a serious threat to the paper's central quantitative claims. I would encourage the editor to require the human-agreement study and to ask the authors to correct the overgeneralizations in the abstract and introduction. The paper is a reasonable candidate for acceptance after these revisions, provided the authors can establish the reliability of their evaluation pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe takeaway: MultiCodeIF is a genuinely useful benchmark artifact, and the paper is worth a real referee, but don't quote the headline numbers without reading the fine print. The 54.5% to 18.8% multi-level HSR drop in the abstract comes from Data Structure tasks only (Table V), not the full multi-level set, and the evaluation pipeline hasn't been validated against human judgments.\n\nWhat's new and good: the 27-type constraint taxonomy with 9 categories is more fine-grained than CodeIF, CodeIF-Bench, or FollowBench; the tasks span 14 languages (or 15, see below); there's a multi-level composition design and a multi-turn self-repair variant; and they ship the dataset, evaluation scripts, and prompts. That's a real resource for the code-generation evaluation community. The empirical comparison across six models is plausible on its face: Claude and GPT-4o beat small open models, explicit constraints are easier than abstract ones, and self-repair helps. The ConstraGen pipeline with seed code, ROUGE-L filtering, and manual validation is a reasonable construction method.\n\nSoft spots. First, the abstract overreaches. The 'from 54.5% in single-level to just 18.8% in multi-level scenarios' line is in the paper's own RQ3 results, which are scoped to Data Structure constraints ('within the Data Structure constraint category,' Section IV-D.2). Generalizing that to all multi-level tasks is not supported. Second, the evaluation instrument: rule-based scripts plus GPT-4-Turbo judge, with no human agreement study, no per-type precision/recall, and the same model family doing both generation and judging. For semantic constraints like Code Quality, that's a real validity risk, and HSR as a conjunction makes it worse—small per-constraint errors compound at L4. I don't think this sinks the paper; most constraints are rule-checkable, and the abstract's 63.0%/44.8% are averages across all 27 types, so they're not wholly judge-dependent. But the judge-based categories are exactly where the biggest model gaps appear (Code Quality 2% vs 67%), so the concern is substantive. Third, minor: the paper says 14 languages in the abstract and 9 popular + 6 less popular = 15 in Section III-D, and lists GPT-4o's release date as Oct 2023 (it was May 2024). These are small but should be fixed.\n\nWho this is for: anyone building or using code instruction-following benchmarks, or evaluating coding agents. It deserves a serious referee: an editor should send it out rather than desk-reject, with a request to fix the abstract, add judge calibration against human labels, and clarify the language count.\n\nRecommendation: engage with it as a conditional accept—the artifact is valuable, and I'd cite it, but the headline claims need to be pulled back to what the data actually supports.","headline":"Useful benchmark, but the headline multi-level HSR drop is Data Structure-only and the LLM judge is unvalidated; worth citing after revision.","tokens_in":16509,"tokens_out":3679,"would_cite":true,"duration_ms":37890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MultiCodeIF, a 2,021-task benchmark across 14 languages, shows code LLMs satisfy 63% of fine-grained constraints on average and only 18.8% when multiple hierarchical constraints stack.","keywords":["code generation","instruction following","constraint taxonomy","LLM evaluation","multi-turn feedback","self-repair","multi-level constraints","benchmark"],"falsifier":"Sample roughly 200 outputs from the single-level and multi-level sets, have two human programmers independently judge each constraint as satisfied or not, then compare their verdicts to the pipeline's. If agreement is below, say, 90%, or if the pipeline systematically errs toward leniency on code-quality constraints or strictness on interface-specification constraints, the reported 63.0%, 54.5%, 18.8%, and 83.4% figures would need recalibration.","tokens_in":15467,"feed_emoji":"📊","tokens_out":6691,"duration_ms":70238,"temperature":0.7,"pith_summary":"MultiCodeIF is a benchmark that asks whether code-generating LLMs actually follow detailed instructions, not just whether their output passes tests. It organizes program requirements into 9 constraint categories and 27 fine-grained types, then builds 2,021 code tasks, with 1,526 single-constraint and 495 multi-constraint tasks, spanning 14 programming languages. Evaluated on six LLMs, the paper reports that the best model satisfies only 63.0% of constraints on average, that satisfaction drops from 54.5% to 18.8% as constraints stack from one level to multiple levels, and that structured feedback lifts satisfaction to 83.4% after four repair rounds. If the measurement is right, MultiCodeIF gives the field a reusable instrument for tracking constraint-aware generation and self-repair.","feed_headline":"LLMs crash from 54.5% to 18.8% on stacked code constraints","feed_subtitle":"A new 2,021-task benchmark tests fine-grained constraint following; structured feedback lifts satisfaction from 63% to 83%.","key_machinery":"The load-bearing mechanism is the constraint taxonomy plus the ConstraGen generation pipeline, with an evaluation pipeline of rule-based scripts and an LLM judge. The taxonomy provides 9 categories and 27 fine-grained constraint types, each paired with a validation strategy, such as Tree-sitter for syntax and data structures and GPT-4-Turbo for algorithm and code-quality judgments. ConstraGen starts from real-world seed code, abstracts it into programming concepts, generates prompt-constraint pairs, filters near-duplicates with ROUGE-L similarity, and manually validates; then it expands single-level tasks into L2-L4 ladders by adding one constraint per level. The benchmark's evaluative core is the pair of metrics SSR (soft satisfaction rate, the fraction of individual constraints met) and HSR (hard satisfaction rate, whether all constraints in a task are met), which together convert instruction adherence into a number that can be compared across models and across levels. The multi-turn mechanism is the self-repair loop, where a model receives structured diagnostics for violated constraints and regenerates, with IFRepair@k tracking the hard satisfaction rate after each round.","core_discovery":"The paper's central claim is that instruction-following in code generation can be decomposed into a hierarchy of verifiable constraints, and that current LLMs fail this test in a systematic, measurable way. On its own terms, the discovery is: with a fine-grained taxonomy of 27 constraint types, top models like Claude-3-7-Sonnet satisfy roughly six in ten constraints (63.0% average) on single-level tasks; explicit constraints such as environment and code context are handled well (above 70%), while abstract constraints like code quality fall below 40%; and multi-level compositional tasks cut the hard satisfaction rate from 54.5% to 18.8%. The paper further claims that targeted feedback works, with Claude-3-7-Sonnet improving from 63.0% to 83.4% and GPT-4o from 62.1% to 78.6% after four rounds of structured self-repair. The overall assertion is that a constraint-centric, evolvable, multi-language benchmark reveals capabilities that functional-correctness benchmarks do not.","pith_inferences":["A direct extension the authors leave implicit is to use the feedback rounds and constraint diagnostics as training data, such as instruction tuning or reinforcement learning, rather than only as an evaluation loop; if the 63% to 83% gain transfers to training, self-generated repair trajectories could be a cheap supervision signal.","A testable follow-up would run the same 495 multi-level tasks with constraints permuted in different orders to see whether the drop from 54.5% to 18.8% reflects constraint count, constraint type, or the order in which they are added.","The benchmark's reliance on an automated judge suggests an obvious companion study: an inter-annotator agreement check between the rule-based and LLM judge and human programmers on a random sample of outputs, which would calibrate every reported number.","Neighboring natural-language instruction-following benchmarks could reuse the same L-ladder methodology to test whether the hierarchical collapse is specific to code or a general property of LLM instruction adherence."],"forward_implications":["MultiCodeIF can be used as a drop-in evaluation protocol for code LLMs, allowing any new model to be scored on the same 27 constraint types and compared against the six models reported.","The L1-to-L4 ladder provides a controlled way to measure compositional instruction-following, so future work can study which constraint combinations cause failures rather than blaming overall model quality.","The self-repair results imply that feedback loops, not just bigger models, are a viable path to increasing constraint adherence, with most gains coming in the first two repair rounds.","The taxonomy's distinction between explicit and implicit constraints gives a diagnostic: high performance on environment and code-context constraints but low performance on code quality suggests models have a pattern-matching strength and a semantic-reasoning weakness.","Because tasks are evolvable, with new seeds, domains, and constraint types addable, the benchmark supports longitudinal tracking without the usual saturation and leakage problem, as long as new seeds are introduced."],"supporting_citations":[{"why":"Supplies the function-level code-generation benchmark whose pass@1 correctness focus MultiCodeIF deliberately extends beyond.","marker":"[7]"},{"why":"Provides MBPP, the other standard correctness benchmark used as a comparison point for why functional correctness alone is insufficient.","marker":"[8]"},{"why":"IFEval, a general instruction-following benchmark, is used as a comparison showing the lack of code-specific constraint modeling.","marker":"[12]"},{"why":"InfoBench, another general instruction-following benchmark, is used as a comparison for fine-grained constraints outside the code domain.","marker":"[13]"},{"why":"CodeIF, the closest code-specific instruction-following benchmark, is positioned as prior work with coarse constraints and limited language support.","marker":"[14]"},{"why":"CodeIF-Bench, an interactive code-generation instruction-following benchmark, serves as a comparison lacking multi-level and self-repair dimensions.","marker":"[15]"},{"why":"A study of 24 coding-style violations supplies the style-constraint layer of the taxonomy.","marker":"[18]"},{"why":"FollowBench provides the five-category constraint framework and the SSR and HSR metrics adopted for multi-level evaluation.","marker":"[19]"},{"why":"Tree-sitter is the parser used by rule-based constraint-validation scripts for syntax and data-structure checks.","marker":"[22]"},{"why":"The GPT-4-Turbo technical report documents the model that powers both task generation in ConstraGen and the LLM-based judge for abstract constraint types.","marker":"[23]"}],"fun_headline_variants":["Code LLMs fold on stacked constraints: 54.5% to 18.8%","New benchmark: MultiCodeIF exposes code LLM constraint failures","Feedback loop rescues code LLMs: 63% to 83% after 4 rounds","27 constraint types: code LLMs hit 63%, then 18.8% on stacked tasks","MultiCodeIF: 2021 tasks, 14 languages, big LLM gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement rests on the automated constraint checks, rule scripts plus a GPT-4-Turbo judge, being correct, yet no human agreement study or false-positive and false-negative analysis is reported, so any bias in the judge shifts every satisfaction rate.","fun_headline_variants_meta":{"raw":{"variants":["Code LLMs fold on stacked constraints: 54.5% to 18.8%","New benchmark: MultiCodeIF exposes code LLM constraint failures","Feedback loop rescues code LLMs: 63% to 83% after 4 rounds","27 constraint types: code LLMs hit 63%, then 18.8% on stacked tasks","MultiCodeIF: 2021 tasks, 14 languages, big LLM gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000939,"raw_usage":{"total_tokens":4092,"prompt_tokens":1099,"completion_tokens":2993,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":715,"completion_tokens_details":{"reasoning_tokens":2880}},"tokens_in":715,"tokens_out":2993,"duration_ms":22283,"temperature":1.0,"reasoning_tokens":2880,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:08:35.610099+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample roughly 200 outputs from the single-level and multi-level sets, have two human programmers independently judge each constraint as satisfied or not, then compare their verdicts to the pipeline's. If agreement is below, say, 90%, or if the pipeline systematically errs toward leniency on code-quality constraints or strictness on interface-specification constraints, the reported 63.0%, 54.5%, 18.8%, and 83.4% figures would need recalibration.","supporting_citations":[{"cited_title":"tree-sitter/tree-sitter: v0.25.5,","cited_arxiv_id":null,"evidence_quote":"Tree-sitter is the parser used by rule-based constraint-validation scripts for syntax and data-structure checks."}],"review_version":1}