{"id":"fcb2bf4f-c92c-424f-aa76-a590b856aeaf","arxiv_id":"2607.19608","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Instruction-tuned Qwen models from 0.8B to 27B often default to the standard correct answer when told to produce an incorrect option, opposite sentiment, or doubled answer, so task competence and instruction following are separable abilities.","lead":"This paper tests whether small language models follow instructions that contradict their usual task behavior, by asking them to answer multiple-choice questions incorrectly, flip sentiment labels, or double math answers. It finds that small models often keep producing the correct standard answer despite the conflicting instruction, so standard accuracy alone hides instruction-following failures.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Restricted-argmax decoding (§4.3) may inflate IFFR by scoring out-of-set/refusal first tokens as defaults to the gold answer.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the closed-set decoding may not faithfully measure instruction-following failures. I sharpen it by tying it to the paper's own Limitations section, which lists 'decoding behavior' as an unexamined cause, and by noting that the cross-task conclusion depends on the MCQA and sentiment results that use the restricted argmax. The math task uses free-form decoding, so this specific artifact does not affect it, but the headline claim is a conjunction over tasks; if the MCQA and sentiment IFFR values are inflated, the 'small models routinely ignore' conclusion is weakened. I considered other candidate concerns—such as the comprehension confound (small models may not understand the non-standard instruction) and the low math competence of small models—but these are less directly falsifiable or less central to the measured dissociation. The proposed free-form comparison is a concrete, feasible check that would settle whether the restricted-argmax assumption holds. The paper already earns credit for reporting per-prompt standard deviations and releasing a code repository; those are independent supports, but they do not resolve the decoding question. Since the reader's CONDITIONAL verdict already flags this issue, my read does not move the verdict; it reinforces the condition that the decoding assumption must be validated.","tokens_in":13663,"tokens_out":8040,"duration_ms":71861,"concrete_test":"On a random sample of 200 examples per dataset and model size (MCQA and sentiment), run the same prompts with free-form greedy decoding (no logit restriction; enable_thinking=False as in Appendix A). Parse each non-standard response with a rule-based/LLM auditor into: gold option selected, non-gold option selected, out-of-set/refusal/explanation (no clear option). Compute IFFR on the standard-correct subset from these free-form responses and compare with Tables 3–4. Also report the proportion of out-of-set/refusal responses per model size. If small models show >10% out-of-set/refusal responses and the free-form IFFR is materially lower than the restricted-argmax IFFR, the §4.3 decoding assumption fails and the reported IFFR is inflated; if the proportions are negligible, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central dissociation—small models stay competent yet routinely ignore conflicting instructions—rests on IFFR (Eq. 1) and the restricted-argmax pipeline in §4.3. For MCQA and sentiment, only first-token logits over the closed label set ({A,B,C,D} or {positive,negative}) are used; the prediction is the in-set token with highest logit. This discards all out-of-set tokens. Under a conflicting instruction, a model that begins with a refusal or off-topic token (e.g., 'I', 'Sorry', 'As', an explanation) still gets mapped to the highest-scoring option token, which can be the gold answer. That outcome is then counted as y_ns = y, contributing to IFFR, even though the model did not default to the standard answer—it produced no in-set answer at all. The paper's own Limitations acknowledge that 'decoding behavior' is an unexamined cause of failures. This matters because the high small-model IFFR on MCQA and sentiment—two of the three tasks—is the main evidence for the claim that standard accuracy hides instruction-following failures. If small models are instead refusing or emitting out-of-set tokens at a nontrivial rate, the claim that they 'routinely ignore' the instruction is overstated, and IFFR conflates 'defaulted to gold' with 'did not produce an in-set token.' The math pipeline is free-form and not subject to this artifact, but the cross-task conclusion is a conjunction; the MCQA and sentiment components are load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an evaluation framework that separates task competence from instruction following by pairing a standard instruction with a conflicting non-standard instruction on the same input (select an incorrect MCQA option, output the opposite sentiment, double a math answer). All predictions are scored against the original ground truth, and the paper introduces IFFR, defined as the conditional probability that a model outputs the gold answer under the non-standard instruction given that it answered correctly under the standard instruction. The authors evaluate instruction-tuned Qwen3.5 models from 0.8B to 27B parameters on nine datasets. They report that for MCQA and sentiment classification, small models retain high standard accuracy but very high IFFR, while larger models follow the conflicting instruction; for math QA, small models have low standard accuracy and moderate IFFR, with a non-monotonic pattern on MultiArith. The paper concludes that standard accuracy alone hides instruction-following failures and that task competence and instruction following are distinct abilities.","tokens_in":14020,"tokens_out":8203,"duration_ms":73496,"significance":"The cross-task design is well motivated, and the IFFR definition cleanly conditions on examples the model can actually solve, which is a real improvement over raw non-standard accuracy for comparing models of different competence. The paper ships reproducible code, uses deterministic decoding, and averages over three prompt variants per condition. If the closed-set decoding assumption is validated and the cross-task claims are scoped appropriately, the framework would be a useful, lightweight addition to instruction-following evaluation for small language models. However, as written the abstract and discussion overstate the cross-task evidence, and the central dissociation for two of the three tasks depends on an unvalidated decoding step.","major_comments":[{"comment":"The abstract's central assertion that 'small models stay competent yet routinely ignore the non-standard instruction' is contradicted by the math results. Qwen3.5-0.8B achieves standard accuracy of only 8.70% on MultiArith, 27.04% on MAWPS, and 35.36% on Calc-asdiv-a, and its IFFR is about 60%, not the near-100% described in the Discussion. The 'competent yet ignore' pattern holds for MCQA and sentiment, but not for math. The cross-task conclusion needs to be scoped to the tasks where the data support it, or the math task should be presented as a separate, lower-competence case.","section":"Abstract; §5, Table 5"},{"comment":"The restricted-argmax decoding for MCQA and sentiment discards all out-of-set first tokens. If a model begins a refusal or explanation ('Sorry', 'I', 'As'), the restricted argmax over {A,B,C,D} or {positive,negative} still selects a token, which can be the gold answer. Such cases are then counted in IFFR as ŷ_ns = y even though the model did not default to the standard answer — it produced no admissible token at all. The paper's own Limitations acknowledge that 'decoding behavior' is an unexamined cause. Because IFFR for two of the three tasks is computed entirely through this pipeline, the main dissociation could be inflated. Please report the proportion of examples under the non-standard instruction whose full-vocabulary first-token argmax is out-of-set, and/or compare restricted-argmax IFFR with IFFR obtained from free-form greedy decoding and parsing.","section":"§4.3, Eq. (1)"},{"comment":"IFFR conditions on {ŷ_s = y}. For low-accuracy models the denominator is very small: MultiArith Qwen3.5-0.8B standard accuracy is 8.70% (~16 of 180 examples) and Qwen3.5-2B is 5.93% (~11 of 180 examples). The across-prompt standard deviations for these cells are large (e.g., 27.42 points for MultiArith 0.8B in Table 8). The quantitative claim that IFFR curves 'fall steeply with scale' across all tasks, and any model-to-model comparisons involving math, rest on very noisy estimates. Report conditional subset sizes or confidence intervals before making these cross-task quantitative comparisons.","section":"Eq. (1); Tables 5, 8"},{"comment":"Several sentiment-classification IFFR values at intermediate scale are extremely unstable across prompt variants: Qwen3.5-9B on Rotten Tomatoes is 49.92±36.89 and on FinancialPhraseBank 57.29±47.46. Only the 27B values are stable. This weakens the general conclusion that 'larger models more reliably override the default objective' at 9B. The paper notes the error bars but still presents the scaling trend as a smooth cross-task pattern; additional per-variant analysis or a more robust summary (e.g., median and range) is needed.","section":"§5, Table 7"}],"minor_comments":[{"comment":"The name 'Instruction-Following Failure Rate' is broader than what is measured. Eq. (1) counts only the specific failure of reproducing the gold answer under the non-standard instruction; a wrong non-gold answer is counted as non-failure. Consider naming it 'Gold-Reverting Rate' or clarifying in the text that IFFR measures one well-defined failure mode, not all instruction-following failures.","section":"§3.2, Eq. (1)"},{"comment":"For MAWPS at 0.8B, non-standard accuracy (27.33) slightly exceeds standard accuracy (27.04), which is unexpected if the model simply reverts to the standard answer. A brief explanation of this artifact (e.g., prompt-variant variance) would help the reader interpret the math results.","section":"§5, Figure 3(c)"},{"comment":"The text says the sentiment pipeline restricts to 'the two label tokens positive and negative'. It would be helpful to state whether 'positive' and 'negative' are single tokens in the Qwen tokenizer and how a tokenization split (e.g., 'pos' + 'itive') would affect the restricted argmax.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful bit: the paper defines IFFR, a clean conditional metric that separates \"can't solve\" from \"won't change the answer,\" and applies it across three tasks with three prompt variants on a same-family scale sweep. That is a legitimate extension of prior answer-modifying instruction work (Murthy et al., Fu et al.), not a new phenomenon, but the cross-task package is new and the metric is well-defined. The qualitative pattern—large drop in IFFR with scale on MCQA and sentiment, with a non-monotonic blip at 27B on MultiArith—is plausible and worth having on the record.\n\nThe soft spots are real but manageable. The abstract overclaims \"small models stay competent\" for math: 0.8B gets 8.7% on MultiArith and 27% on MAWPS. That is not competence. The IFFR on the harder math tasks is computed on a tiny denominator and carries error bars of 20+ points. The authors should either restrict the claim to tasks where standard accuracy is actually high or qualify it as task-dependent.\n\nThe more serious issue is the restricted-argmax decoding for MCQA and sentiment. The pipeline reads first-token logits and restricts them to the option/label set. If a small model starts with \"I\" or \"Sorry\" under the conflicting instruction, the restricted argmax can still land on the gold option, and that gets scored as \"defaulted to the standard answer.\" That conflates two different failure modes. The math pipeline is free-form and immune to this artifact, but the MCQA and sentiment results are the main evidence for the \"routinely ignore\" claim. It is not fatal: the non-standard accuracies are so close to standard accuracies for the small models that genuine defaulting is likely, but the paper should report the share of examples where the unrestricted first token is not in the closed set, or run a full-generation parse as a sanity check. The authors' own limitations section lists decoding behavior as an unexamined cause, which is consistent with this concern.\n\nAlso, the appendix says MCQA used the DeepInfra API, but getting first-token logits from a chat-completions interface is not boilerplate; the exact mechanism (logprobs? logit bias?) needs to be documented for reproducibility.\n\nFinal judgment: worth a serious referee. The central distinction is sound, the metric is useful, and the empirical pattern is interesting enough to test. I would ask for the closed-set robustness check, a corrected abstract, and confidence intervals on low-denominator IFFRs before accepting. For a reading group, it's a quick discussion-starter, not a must-read.","headline":"Useful institutional-eval paper with a clean conditional metric; abstract overclaims for math, and the closed-set logit decoding deserves a robustness check before the cross-task story fully lands.","tokens_in":14473,"tokens_out":5093,"would_cite":true,"duration_ms":44243,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that task competence and instruction following are distinct abilities: small models can be competent at a task yet keep producing the standard answer when the instruction tells them to deviate.","keywords":["instruction following","task competence","small language models","instruction-conflicting behavior","IFFR","multiple-choice question answering","sentiment classification","mathematical QA"],"falsifier":"Recompute IFFR on a sample with unconstrained greedy decoding instead of the restricted-logit argmax, classifying each non-standard response as a compliant answer, a standard answer, or a refusal/out-of-set statement. If free-form decoding shows that a substantial share of 'instruction-following failures' are refusals or statements beginning with a non-option token (e.g., 'Sorry,' 'I cannot,' or '('), the restricted-logit computation overestimates IFFR and the claim that small models 'routinely ignore' the instruction would need to be qualified. A second check: if another model family of simil","tokens_in":13572,"feed_emoji":"🎯","tokens_out":9136,"duration_ms":70413,"temperature":0.7,"pith_summary":"The paper asks whether an instruction-tuned model that can solve a task can also obey a conflicting instruction. On three tasks, the same input is paired with a normal request and a conflicting one—choose an incorrect option, output the opposite sentiment, or return twice the answer—and both are scored against the original answer, so ignoring the new request still looks accurate. The authors introduce the Instruction-Following Failure Rate (IFFR), which measures how often a model that solved the task normally keeps producing the standard answer when told to deviate, isolating obedience from competence. Across five model sizes, small models stay competent yet routinely ignore the conflicting instruction, while larger models obey far more often, though not monotonically on every dataset. The conclusion is that accuracy and instruction following are separate axes and should be reported separately.","feed_headline":"Small models stay sharp but ignore conflicting instructions","feed_subtitle":"Standard accuracy hides the failure; the IFFR metric exposes it.","key_machinery":"The carrying mechanism is a paired-instruction evaluation design plus a conditional metric, the Instruction-Following Failure Rate (IFFR). Standard accuracy measures task competence; non-standard accuracy, scored against the original ground truth, is inverted—high values mean the model ignored the request. IFFR is the conditional probability, computed only on examples the model answered correctly under the normal instruction, that it still produces the standard answer under the conflicting instruction. Conditioning separates instruction following from raw ability, and the three unrelated tasks (closed-set option choice, binary sentiment label, free numeric answer) act as replications showing","core_discovery":"Task competence and instruction following are distinct abilities, and standard accuracy hides instruction-following failures. The empirical core: in multiple-choice QA, sentiment classification, and mathematical QA, the smallest instruction-tuned models often answer correctly under the normal request (e.g., 70–97% standard accuracy on several datasets) while under the conflicting request they reproduce the standard correct answer on roughly 60–98% of those solved items; the largest model drops that failure rate to single digits on most datasets. Scale closes most of the gap, but does not guarantee it: on one arithmetic dataset, the largest model had the highest standard accuracy yet a higher","pith_inferences":["If the same pattern holds for more transformations, IFFR could become a standard diagnostic for instruction-tuning quality: how quickly a model's failure rate falls as it is scaled or tuned would serve as a measure of controllability, not just accuracy.","A natural next experiment would vary the strength of the conflicting instruction (repeating it, placing it after the input, or making the requested transformation a simple arithmetic change rather than an antonym) to see whether small-model failures are due to weak instruction salience or to an inability to override the learned task prior.","The decoding assumption suggests a direct test: measure IFFR both ways—logit-restricted and free generation—and check whether refusals or non-option first tokens are being silently counted as failures; if they are, the gap between small and large models may partly reflect differences in refusal behavior rather than instruction following."],"forward_implications":["Standard accuracy and IFFR should be reported as separate axes whenever a model is evaluated; a model can score high on one and low on the other.","Small models deployed in resource-constrained settings can look capable while silently ignoring user constraints, so evaluation suites for local models should include instruction-conflicting probes.","Scaling does not monotonically improve instruction following; the arithmetic dataset where the largest model reverted to the default answer shows that instruction adherence needs to be measured, not assumed, at every size.","A cross-task conditional metric like IFFR lets models of different competence be compared on obedience without the comparison being confounded by how many problems each model can solve."],"fun_headline_variants":["Accuracy alone hides instruction-following failures","Small models answer right but ignore conflicting orders","Task skill and instruction compliance diverge in small LLMs","Scale fixes instruction following, but not on all tasks","Competence ≠ compliance: small models ignore altered instructions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The analysis assumes that reading the first-token logits restricted to the closed answer set (A/B/C/D or positive/negative) faithfully measures whether the model followed the conflicting instruction; if the model began a refusal or an out-of-set token, the restricted argmax could still land on the ground-truth option and an instruction-following failure would be recorded even though the model did not actually produce the standard answer. This decoding assumption appears in Se","fun_headline_variants_meta":{"raw":{"variants":["Accuracy alone hides instruction-following failures","Small models answer right but ignore conflicting orders","Task skill and instruction compliance diverge in small LLMs","Scale fixes instruction following, but not on all tasks","Competence ≠ compliance: small models ignore altered instructions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000129,"raw_usage":{"total_tokens":950,"prompt_tokens":729,"completion_tokens":221,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":162}},"tokens_in":473,"tokens_out":221,"duration_ms":3039,"temperature":1.0,"reasoning_tokens":162,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T12:15:20.628697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute IFFR on a sample with unconstrained greedy decoding instead of the restricted-logit argmax, classifying each non-standard response as a compliant answer, a standard answer, or a refusal/out-of-set statement. If free-form decoding shows that a substantial share of 'instruction-following failures' are refusals or statements beginning with a non-option token (e.g., 'Sorry,' 'I cannot,' or '('), the restricted-logit computation overestimates IFFR and the claim that small models 'routinely ignore' the instruction would need to be qualified. A second check: if another model family of simil","supporting_citations":[],"review_version":1}