{"id":"d48efefe-6565-4279-adfa-5c1ffab122ac","arxiv_id":"2505.14810","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across 23 reasoning models on 420 constrained math problems, stronger reasoning-oriented training and longer chains of thought are associated with worse adherence to user-specified constraints.","lead":"A new benchmark, MathIF, adds 15 Python-checkable formatting and wording rules to 420 math problems and finds that most large reasoning models obey fewer than half of them. The paper argues that training models to reason longer, through long chain-of-thought or reinforcement learning, tends to weaken obedience to user instructions, and that simple re-prompting can trade reasoning quality back for compliance.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exact-match correctness scoring conflates instruction compliance with reasoning errors, contaminating the correctness-cost half of the trade-off claim.","rationale":"The reader's weakest assumption identifies the same issue, and it is genuinely load-bearing for the paper's central trade-off claim. The benchmark's headline instruction-following metrics, HAcc and SAcc, are Python-verifiable and are not directly undermined by the exact-match correctness metric. However, the second half of the claimed trade-off—that obedience to constraints degrades mathematical reasoning—is measured with exact string matching against an unconstrained ground truth. Since several constraint categories explicitly alter the surface form of the response, a compliant and mathematically correct answer can be scored as incorrect. This creates a systematic bias that inflates the Diff drops in Table 3, depresses the (Correct, Followed) cell in Figure 4, and likely explains part of the correctness drop in the +repeat intervention in Table 6. The proposed re-scoring test is concrete and feasible because the authors release data and code, and it would settle whether the trade-off survives parse-aware evaluation. Until that check is run, the CONDITIONAL verdict is appropriate; the central empirical regularity about instruction adherence deserves further support, but the correctness-cost half of the trade-off should be treated as unverified rather than established.","tokens_in":18902,"tokens_out":5376,"duration_ms":53215,"concrete_test":"Re-score all 420 MathIF outputs with parse-aware correctness: before comparing to ground truth, remove constraint-mandated wrappers—strip matching double quotes, remove the required suffix phrase, remove a verbatim repetition of the request, and normalize case when lowercase/capital constraints are active—then compare the remaining mathematical answer to the reference using symbolic equivalence (e.g., sympy) rather than exact string match. Recompute Table 3's Diff column, Figure 4's four-way partition, and Table 6's Correctness column. If parse-aware scoring makes the Diff values near zero or reverses sign for models with high HAcc, the trade-off claim is partly an artifact of exact-match scoring; if the drops persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing point is in Section 3, Evaluation Metric: correctness is defined as 'whether the final answer exactly matches the ground-truth, regardless of constraint satisfaction.' Many MathIF constraints are surface-form instructions: wrap the entire response in double quotation marks, finish with a fixed suffix, first repeat the request word for word, or use all lowercase letters. A model that obeys these instructions will, by construction, produce a final answer whose surface form differs from the unconstrained reference, and the script will count it as a reasoning error. This contaminates the Diff column of Table 3, the Correct/Incorrect split in Figure 4, and the correctness drops in Table 6, where +repeat improves instruction following but 'correctness' falls from 31.67 to 22.38. The paper's stronger trade-off claim—that following instructions costs reasoning performance—is therefore partly measuring a scoring artifact. The HAcc/SAcc degradation from reasoning-oriented training is less affected, but the 'intelligence versus control' framing depends on this contaminated correctness measure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MathIF, a benchmark of 420 mathematical reasoning problems with Python-verifiable instruction-following constraints (15 constraint types; single, double, and triple compositions). It evaluates 23 instruction-tuned and reasoning-oriented models, reporting low instruction-following accuracy across the board, a negative association between chain-of-thought length and instruction adherence, and a claimed trade-off between reasoning performance and instruction-following. The paper also presents controlled training experiments (SFT, cold-RL, SFT+RL, format reward) and inference-time interventions (budget forcing, repeating the instruction) to support the trade-off narrative.","tokens_in":19020,"tokens_out":6118,"duration_ms":50803,"significance":"If the empirical claims hold, MathIF fills a real gap: there is no dedicated, verifiable benchmark for instruction-following in reasoning models, and the observation that reasoning-oriented training can reduce instruction adherence is an important constraint on how LRMs are trained and evaluated. The design has genuine strengths: the constraints are Python-verifiable rather than judge-based, the data cover diverse difficulty levels, and code/data are released. The training comparisons against base models are a useful contribution. However, the stronger 'fundamental tension' and 'less controllable' conclusions rest on a correctness-with-constraints metric that is currently contaminated by exact-match scoring, so the central interpretive claim needs to be re-evidenced before the paper is accepted.","major_comments":[{"comment":"The correctness metric is defined as whether the final answer 'exactly matches the ground-truth, regardless of constraint satisfaction.' For several MathIF constraints (e.g., 'Wrap your entire response with double quotation marks,' 'First repeat the request word for word,' 'Finish your response with this exact phrase,' 'all lowercase letters'), a fully compliant response necessarily has a different surface form from the unconstrained reference, so it will be scored as a reasoning error even when the mathematics is correct. This contaminates the Diff column of Table 3, the (Correct, Followed) cell in Figure 4, and the Correctness column of Table 6 (e.g., +repeat raises HAcc from 17.14 to 21.66 but lowers 'correctness' from 31.67 to 22.38, which may reflect the added prefix rather than a loss of reasoning). The claim that following instructions costs reasoning performance is therefore partly a scoring artifact. I request a constraint-aware correctness score (e.g., strip wrappers/prefixes/suffixes or match the mathematical content only) and a recomputation of the trade-off analyses with that score.","section":"Section 3, Table 3"},{"comment":"The central causal evidence that reasoning-oriented training degrades instruction-following rests on small HAcc/SAcc differences: e.g., Qwen2.5-1.5B HAcc drops from 10.00 to 7.86, and Qwen2.5-Math-7B SAcc drops from 23.53 to 21.03. With n=420, differences of 2-3 points are plausibly within sampling noise, yet no variance, confidence intervals, or significance tests are reported. The 'consistently drop across the board' claim needs either statistical support or a discussion of effect size; otherwise the trade-off conclusion is not yet established by the training experiments.","section":"Section 5.2, Table 4"},{"comment":"The statement 'Longer CoTs Impair Instruction Following' is supported primarily by Figure 6, which bins responses by CoT length. This is confounded by problem difficulty: harder problems elicit both longer CoTs and lower instruction compliance. The budget-forcing experiment in Figure 7 is a better causal probe, but it is run on a single small model and the repeated 'Wait' tokens may disrupt instruction-following through a mechanism other than length. I ask that the causal language be softened or that the analysis be re-run controlling for problem difficulty (or on a matched subset), and that the budget-forcing intervention be tested on at least a second model.","section":"Section 5.1 (Figure 6) and Section 5.3"}],"minor_comments":[{"comment":"Section 3 does not specify how the 'final answer' is extracted from a long CoT response (e.g., after </think> or as the last line); please define the extraction procedure so the correctness results are reproducible.","section":"Section 3"},{"comment":"Figures 2 and 3 average over 23 models without confidence intervals; reporting per-model variance or bootstrap intervals would support the comparisons between subsets and constraint counts.","section":"Figures 2 and 3"},{"comment":"Table 8 has a misaligned header: 'single double triple' followed by 'Acc HAcc SAcc HAcc SAcc' should be restructured so it is clear that Acc refers to the single-constraint subset; also 'SimplRL' in Tables 9-10 should be 'SimpleRL'.","section":"Table 8"},{"comment":"The abstract's 'consistent tension' phrasing overstates Table 4, where several training-induced changes are within 1-2 points; consider wording such as 'frequent degradation' unless the significance analysis requested above changes the conclusion.","section":"Abstract"},{"comment":"The decoding temperature T=1.0 in Section 4 may not match the recommended setting for some reasoning models; a greedy-decoding sensitivity check would make the benchmark numbers more robust.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's code/data release is a genuine strength, and the benchmark concept is timely. The exact-match scoring issue is fixable within the scope of a revision, but until the correctness metric is made constraint-aware and the trade-off analyses are recomputed, the 'less controllable' and 'fundamental tension' conclusions are not fully supported. If the authors re-score and the qualitative conclusions survive, I would support acceptance. The scope is a good fit for a CL/ML venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, MathIF is a genuinely new and useful resource: the first instruction-following benchmark aimed at reasoning models in the math domain, with 15 Python-verifiable constraints, compositional prompts, and evaluation across 23 models. The authors also ran real training comparisons (SFT vs SFT+RL vs cold-RL, plus length-limited RL) and tested an inference-time reminder, so the empirical work is substantial. Code and data are released.\n\nThe central observation holds up: reasoning-oriented training—distilled long-CoT SFT and outcome-reward RL—does lower constraint satisfaction measured by HAcc/SAcc, and longer generations tend to correlate with worse adherence. That is a useful, actionable finding.\n\nThe soft spot is the correctness metric, and it infects the stronger trade-off claim. In Section 3 they define correctness as exact string match against the ground-truth answer, regardless of constraint satisfaction. But many MathIF constraints are surface-form instructions: wrap the entire response in double quotes, repeat the request word for word, use lowercase only. A model that obeys those writes a final answer that cannot literally equal the unconstrained reference, so it is counted as mathematically wrong. Consequently the 'Diff' column of Table 3, the Correct/Incorrect split in Figure 4, and the correctness drop in Table 6 (e.g., +repeat improves HAcc but 'correctness' falls from 31.67 to 22.38) all mix instruction compliance with genuine reasoning errors. The paper labels this a trade-off between intelligence and obedience, but part of the 'intelligence' loss is a scoring artifact. The HAcc/SAcc evidence for degradation is clean; the claim that following instructions costs mathematical accuracy is not proven with this metric.\n\nThere are minor issues too: no error bars or repeated decoding, and the 'fundamental/inherent tension' language in the conclusion goes beyond the single-recipe correlational evidence. But the benchmark itself is solid and the constraint-satisfaction findings are worth taking seriously.\n\nBottom line: this deserves a serious referee, but with the expectation of a measurement fix—parse-aware or constraint-aware correctness scoring—or a softened trade-off claim. I'd accept it to review, and I'd want the authors to address the metric before publication.","headline":"Useful new math-domain instruction-following benchmark, but the headline trade-off claim is partly a scoring artifact: exact-match correctness against unconstrained references penalizes models that correctly follow surface-form constraints.","tokens_in":19563,"tokens_out":2722,"would_cite":true,"duration_ms":23681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces MathIF, a 420-question math benchmark with verifiable constraints, and argues that the training methods that make large reasoning models smarter also make them worse at following simple user instructions, with the gap…","keywords":["instruction following","large reasoning models","chain-of-thought","mathematical reasoning","benchmark","reinforcement learning","controllability","MathIF"],"falsifier":"Re-score MathIF with a judge that accepts any mathematically correct answer satisfying the stated constraints, regardless of exact match; if the accuracy gap between constrained and unconstrained runs disappears or inverts, the reasoning–obedience trade-off is an artifact of string matching. Alternatively, find any long-CoT SFT or outcome-RL run that improves both math accuracy and instruction adherence at fixed length, which would break the claimed monotone trade-off.","tokens_in":18678,"feed_emoji":"🤖","tokens_out":4692,"duration_ms":49556,"temperature":0.7,"pith_summary":"The paper introduces MathIF, a benchmark of 420 math problems wrapped in one to three verifiable formatting, length, language, and affix constraints, and uses it to test 23 large reasoning models (LRMs). Its central finding is that the training recipes that make models reason better—supervised fine-tuning on distilled long chains of thought and reinforcement learning with outcome rewards—also make them worse at obeying simple user directives, and the effect worsens as generated reasoning gets longer. The authors argue this reflects a real trade-off between reasoning strength and controllability, and they show that a trivial intervention, restating the instruction at the end of the chain of thought, recovers some obedience at a measurable cost in math accuracy. If true, the result puts instruction adherence on the critical path for evaluating and training reasoning models.","feed_headline":"More reasoning means less instruction following, benchmark finds","feed_subtitle":"On a 420-question test of constrained math, even the top model obeys fewer than half — and longer thinking makes it worse.","key_machinery":"MathIF itself carries the argument. It defines 15 Python-verifiable constraints in four categories (length, lexical, format, affix), composes them into 30 double- and 15 triple-constraint prompts, and applies them to 420 problems drawn from GSM8K, MATH-500, Minerva, Olympiad, and AIME. Two metrics are defined per query: hard accuracy (all constraints satisfied) and soft accuracy (fraction of constraints satisfied), computed by exact string checks rather than a judge model; math correctness is scored by exact match to the unconstrained ground truth. The paper's controlled experiments use budget forcing, rollout-length capping in GRPO, and a 'repeat the instruction' intervention to move the distance between instruction and answer.","core_discovery":"The paper's claim, stated in its own terms, is that scaling reasoning capacity does not guarantee control: across the MathIF benchmark, most LRMs fail to follow most user constraints (best model 50.71% hard accuracy), correctness drops when constraints are added, and both reasoning-oriented SFT and outcome-reward RL consistently lower instruction-following accuracy relative to the base model. The degradation tracks chain-of-thought length: longer CoTs correlate with lower hard and soft accuracy, budget-forcing more 'thinking' monotonically reduces instruction adherence, and capping response length during RL preserves obedience at the cost of math performance. Restating the constraint after the CoT improves compliance but reduces correctness. The authors interpret this as mutual interference between intelligence and obedience during both training and inference.","pith_inferences":["If the mechanism is contextual distance, then instruction-following could be improved by architectural attention that keeps the instruction active across long generations, not by training-time trade-offs alone.","The benchmark's exact-match correctness rule may penalize instruction-compliant answers that are mathematically right but differently phrased, so the reported trade-off could be partly a measurement artifact; a judge-based re-scoring would test this.","The same tension may appear in non-math reasoning domains, such as code generation with style constraints or agentic tool use, where long reasoning traces are also standard.","The finding suggests an 'obedience cost' of test-time scaling: any method that pushes models to think longer should be evaluated for constraint adherence, not just final accuracy."],"forward_implications":["MathIF can serve as a standard, judge-free evaluation target for instruction adherence in math reasoning, enabling precise comparison of training recipes.","Reasoning-oriented SFT and RL should be audited for instruction-following as a primary axis, not as a side effect.","Longer chain-of-thought training runs an alignment cost that grows with length; length capping is one lever, but it trades away reasoning gains.","Restating the constraint at the end of the CoT is a cheap, transferable intervention that improves obedience at a predictable accuracy cost.","Model scale alone does not buy instruction adherence; architecture and training pathway matter more, so leaderboards should report both axes."],"supporting_citations":[{"why":"Supplies the constraint taxonomy and the hard/soft accuracy metrics that MathIF adapts for math reasoning.","marker":"[12]"},{"why":"Supplies the multi-granularity constraint-following evaluation approach and the general-domain baseline showing LRM drops.","marker":"[13]"},{"why":"Provides the compositional constraint construction method (Cartesian product of constraints) used in MathIF.","marker":"[37]"},{"why":"Provides the distilled long-CoT models and the training paradigm whose instruction-following degradation is a central object of study.","marker":"[3]"},{"why":"Provides the budget-forcing method used to lengthen CoT and the test-time scaling framework that the trade-off complicates.","marker":"[14]"},{"why":"Provides the dataset and backbone for the controlled training experiments.","marker":"[45]"},{"why":"Provides the RL algorithm (GRPO) used for the training-pathway comparisons.","marker":"[52]"},{"why":"Establishes that longer reasoning steps aid performance, the benefit that the paper shows comes with an obedience cost.","marker":"[53]"}],"fun_headline_variants":["Reasoning boost, obedience bust: new benchmark","More reasoning, less instruction following, study shows","Smarter math models follow fewer instructions","Longer chain-of-thought reduces instruction adherence","Reasoning scaling erodes instruction following"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a math answer is correct only if it exactly matches the reference string from the unconstrained problem, even when the prompt demands a different language, casing, wrapper, or format, so a mathematically right and instruction-compliant answer that is phrased differently counts as a reasoning failure.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning boost, obedience bust: new benchmark","More reasoning, less instruction following, study shows","Smarter math models follow fewer instructions","Longer chain-of-thought reduces instruction adherence","Reasoning scaling erodes instruction following"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1494,"prompt_tokens":869,"completion_tokens":625,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":558}},"tokens_in":485,"tokens_out":625,"duration_ms":5789,"temperature":1.0,"reasoning_tokens":558,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:28:43.906095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score MathIF with a judge that accepts any mathematically correct answer satisfying the stated constraints, regardless of exact match; if the accuracy gap between constrained and unconstrained runs disappears or inverts, the reasoning–obedience trade-off is an artifact of string matching. Alternatively, find any long-CoT SFT or outcome-RL run that improves both math accuracy and instruction adherence at fixed length, which would break the claimed monotone trade-off.","supporting_citations":[{"cited_title":"The impact of reasoning step length on large language models","cited_arxiv_id":null,"evidence_quote":"Establishes that longer reasoning steps aid performance, the benefit that the paper shows comes with an obedience cost."}],"review_version":1}