Pith. sign in

REVIEW 4 major objections 3 minor 15 references

This paper argues that task competence and instruction following are distinct abilities: small models can be competent at a task yet keep producing the standard answer when the instruction tells them to deviate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 12:15 UTC pith:NFZKTXUO

load-bearing objection Useful institutional-eval paper with a clean conditional metric; abstract overclaims for math, and the closed-set logit decoding deserves a robustness check before the cross-task story fully lands. the 4 major comments →

arxiv 2607.19608 v1 pith:NFZKTXUO submitted 2026-07-21 cs.CL

Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models

Mahdiyeh Farajidizaji , Vatsal Raina This is my paper
classification cs.CL
keywords instruction followingtask competencesmall language modelsinstruction-conflicting behaviorIFFRmultiple-choice question answeringsentiment classificationmathematical QA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether an instruction-tuned model that can solve a task can also obey a conflicting instruction. On three tasks, the same input is paired with a normal request and a conflicting one—choose an incorrect option, output the opposite sentiment, or return twice the answer—and both are scored against the original answer, so ignoring the new request still looks accurate. The authors introduce the Instruction-Following Failure Rate (IFFR), which measures how often a model that solved the task normally keeps producing the standard answer when told to deviate, isolating obedience from competence. Across five model sizes, small models stay competent yet routinely ignore the conflicting instruction, while larger models obey far more often, though not monotonically on every dataset. The conclusion is that accuracy and instruction following are separate axes and should be reported separately.

Core claim

Task competence and instruction following are distinct abilities, and standard accuracy hides instruction-following failures. The empirical core: in multiple-choice QA, sentiment classification, and mathematical QA, the smallest instruction-tuned models often answer correctly under the normal request (e.g., 70–97% standard accuracy on several datasets) while under the conflicting request they reproduce the standard correct answer on roughly 60–98% of those solved items; the largest model drops that failure rate to single digits on most datasets. Scale closes most of the gap, but does not guarantee it: on one arithmetic dataset, the largest model had the highest standard accuracy yet a higher

What carries the argument

The carrying mechanism is a paired-instruction evaluation design plus a conditional metric, the Instruction-Following Failure Rate (IFFR). Standard accuracy measures task competence; non-standard accuracy, scored against the original ground truth, is inverted—high values mean the model ignored the request. IFFR is the conditional probability, computed only on examples the model answered correctly under the normal instruction, that it still produces the standard answer under the conflicting instruction. Conditioning separates instruction following from raw ability, and the three unrelated tasks (closed-set option choice, binary sentiment label, free numeric answer) act as replications showing

Load-bearing premise

The analysis assumes that reading the first-token logits restricted to the closed answer set (A/B/C/D or positive/negative) faithfully measures whether the model followed the conflicting instruction; if the model began a refusal or an out-of-set token, the restricted argmax could still land on the ground-truth option and an instruction-following failure would be recorded even though the model did not actually produce the standard answer. This decoding assumption appears in Se

What would settle it

Recompute IFFR on a sample with unconstrained greedy decoding instead of the restricted-logit argmax, classifying each non-standard response as a compliant answer, a standard answer, or a refusal/out-of-set statement. If free-form decoding shows that a substantial share of 'instruction-following failures' are refusals or statements beginning with a non-option token (e.g., 'Sorry,' 'I cannot,' or '('), the restricted-logit computation overestimates IFFR and the claim that small models 'routinely ignore' the instruction would need to be qualified. A second check: if another model family of simil

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Standard accuracy and IFFR should be reported as separate axes whenever a model is evaluated; a model can score high on one and low on the other.
  • Small models deployed in resource-constrained settings can look capable while silently ignoring user constraints, so evaluation suites for local models should include instruction-conflicting probes.
  • Scaling does not monotonically improve instruction following; the arithmetic dataset where the largest model reverted to the default answer shows that instruction adherence needs to be measured, not assumed, at every size.
  • A cross-task conditional metric like IFFR lets models of different competence be compared on obedience without the comparison being confounded by how many problems each model can solve.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the same pattern holds for more transformations, IFFR could become a standard diagnostic for instruction-tuning quality: how quickly a model's failure rate falls as it is scaled or tuned would serve as a measure of controllability, not just accuracy.
  • A natural next experiment would vary the strength of the conflicting instruction (repeating it, placing it after the input, or making the requested transformation a simple arithmetic change rather than an antonym) to see whether small-model failures are due to weak instruction salience or to an inability to override the learned task prior.
  • The decoding assumption suggests a direct test: measure IFFR both ways—logit-restricted and free generation—and check whether refusals or non-option first tokens are being silently counted as failures; if they are, the gap between small and large models may partly reflect differences in refusal behavior rather than instruction following.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes an evaluation framework that separates task competence from instruction following by pairing a standard instruction with a conflicting non-standard instruction on the same input (select an incorrect MCQA option, output the opposite sentiment, double a math answer). All predictions are scored against the original ground truth, and the paper introduces IFFR, defined as the conditional probability that a model outputs the gold answer under the non-standard instruction given that it answered correctly under the standard instruction. The authors evaluate instruction-tuned Qwen3.5 models from 0.8B to 27B parameters on nine datasets. They report that for MCQA and sentiment classification, small models retain high standard accuracy but very high IFFR, while larger models follow the conflicting instruction; for math QA, small models have low standard accuracy and moderate IFFR, with a non-monotonic pattern on MultiArith. The paper concludes that standard accuracy alone hides instruction-following failures and that task competence and instruction following are distinct abilities.

Significance. The cross-task design is well motivated, and the IFFR definition cleanly conditions on examples the model can actually solve, which is a real improvement over raw non-standard accuracy for comparing models of different competence. The paper ships reproducible code, uses deterministic decoding, and averages over three prompt variants per condition. If the closed-set decoding assumption is validated and the cross-task claims are scoped appropriately, the framework would be a useful, lightweight addition to instruction-following evaluation for small language models. However, as written the abstract and discussion overstate the cross-task evidence, and the central dissociation for two of the three tasks depends on an unvalidated decoding step.

major comments (4)
  1. [Abstract; §5, Table 5] The abstract's central assertion that 'small models stay competent yet routinely ignore the non-standard instruction' is contradicted by the math results. Qwen3.5-0.8B achieves standard accuracy of only 8.70% on MultiArith, 27.04% on MAWPS, and 35.36% on Calc-asdiv-a, and its IFFR is about 60%, not the near-100% described in the Discussion. The 'competent yet ignore' pattern holds for MCQA and sentiment, but not for math. The cross-task conclusion needs to be scoped to the tasks where the data support it, or the math task should be presented as a separate, lower-competence case.
  2. [§4.3, Eq. (1)] The restricted-argmax decoding for MCQA and sentiment discards all out-of-set first tokens. If a model begins a refusal or explanation ('Sorry', 'I', 'As'), the restricted argmax over {A,B,C,D} or {positive,negative} still selects a token, which can be the gold answer. Such cases are then counted in IFFR as ŷ_ns = y even though the model did not default to the standard answer — it produced no admissible token at all. The paper's own Limitations acknowledge that 'decoding behavior' is an unexamined cause. Because IFFR for two of the three tasks is computed entirely through this pipeline, the main dissociation could be inflated. Please report the proportion of examples under the non-standard instruction whose full-vocabulary first-token argmax is out-of-set, and/or compare restricted-argmax IFFR with IFFR obtained from free-form greedy decoding and parsing.
  3. [Eq. (1); Tables 5, 8] IFFR conditions on {ŷ_s = y}. For low-accuracy models the denominator is very small: MultiArith Qwen3.5-0.8B standard accuracy is 8.70% (~16 of 180 examples) and Qwen3.5-2B is 5.93% (~11 of 180 examples). The across-prompt standard deviations for these cells are large (e.g., 27.42 points for MultiArith 0.8B in Table 8). The quantitative claim that IFFR curves 'fall steeply with scale' across all tasks, and any model-to-model comparisons involving math, rest on very noisy estimates. Report conditional subset sizes or confidence intervals before making these cross-task quantitative comparisons.
  4. [§5, Table 7] Several sentiment-classification IFFR values at intermediate scale are extremely unstable across prompt variants: Qwen3.5-9B on Rotten Tomatoes is 49.92±36.89 and on FinancialPhraseBank 57.29±47.46. Only the 27B values are stable. This weakens the general conclusion that 'larger models more reliably override the default objective' at 9B. The paper notes the error bars but still presents the scaling trend as a smooth cross-task pattern; additional per-variant analysis or a more robust summary (e.g., median and range) is needed.
minor comments (3)
  1. [§3.2, Eq. (1)] The name 'Instruction-Following Failure Rate' is broader than what is measured. Eq. (1) counts only the specific failure of reproducing the gold answer under the non-standard instruction; a wrong non-gold answer is counted as non-failure. Consider naming it 'Gold-Reverting Rate' or clarifying in the text that IFFR measures one well-defined failure mode, not all instruction-following failures.
  2. [§5, Figure 3(c)] For MAWPS at 0.8B, non-standard accuracy (27.33) slightly exceeds standard accuracy (27.04), which is unexpected if the model simply reverts to the standard answer. A brief explanation of this artifact (e.g., prompt-variant variance) would help the reader interpret the math results.
  3. [§4.3] The text says the sentiment pipeline restricts to 'the two label tokens positive and negative'. It would be helpful to state whether 'positive' and 'negative' are single tokens in the Qwen tokenizer and how a tokenization split (e.g., 'pos' + 'itive') would affect the restricted argmax.

Circularity Check

0 steps flagged

No significant circularity: IFFR is an operational definition with empirically measured values; the acknowledged restricted-argmax decoding limitation is a validity concern, not a circular step.

full rationale

The paper's derivation chain is self-contained. IFFR (Eq. 1) is defined as P(ŷ_ns = y | ŷ_s = y), an operational definition, and the reported high IFFR values for the 0.8B-2B models come from actual model logits rather than from any fitted parameter or by-construction identity. The central claim that task competence and instruction following are distinct is an empirical dissociation - small models pair high standard accuracy with high IFFR while 27B models pair high accuracy with low IFFR - and the claim would have been falsified had models complied with the conflicting instructions. The metric does embody the conclusion's vocabulary ('Under the non-standard instruction, producing the ground-truth answer instead means the model ignored the instruction and defaulted to its usual task behavior,' Section 3), so the construct is conceptually entangled with the measurement, but the observed rates are not forced by the definition. The Limitations section explicitly concedes the relevant confound: 'our metrics show when models fail to follow the non-standard instruction but not why; further analysis is needed to determine whether failures stem from prompt sensitivity, learned task patterns, decoding behavior, or other factors.' This means restricted-argmax decoding (Section 4.3) may misattribute out-of-set or refusal first tokens as defaults to the gold answer - a validity risk for the MCQA and sentiment results, not a circular reduction. Mathematical QA uses open-ended greedy decoding and shows the same qualitative trend, providing partial independent support. No load-bearing self-citation or imported uniqueness theorem appears; the paper is evaluated against external Qwen3.5 checkpoints and standard datasets.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No free parameters are fitted; the paper is an empirical evaluation. The IFFR is a definition rather than a fitted quantity. The main assumptions are about measurement validity (logit decoding), task design, and comparability of the conditional metric across competence levels.

axioms (5)
  • domain assumption The closed-set first-token logits represent the model's prediction; out-of-set first tokens are ignored.
    Section 4.3 restricts the argmax to {A,B,C,D} or {positive,negative}; a refusal or alternative first token is not scored, which could mislabel non-compliance as instruction-following failure.
  • domain assumption The non-standard instructions are legitimate targets that instruction-tuned models should follow.
    Table 2 defines 'answer incorrectly', 'opposite sentiment', and 'twice the answer'; the paper treats failing to produce the requested deviation as a failure rather than as a possible refusal or truthfulness behavior.
  • domain assumption IFFR conditional on standard-correct examples is comparable across models of very different competence.
    Section 3.2, Eq. 1; for low-accuracy models (e.g., MultiArith 0.8B, 8.70% standard accuracy) the conditioning set is tiny, making IFFR noisy and less comparable.
  • domain assumption Qwen3.5 checkpoints share a common architecture and post-training recipe so scale is the main variable.
    Section 4.1 asserts this; if post-training data differ by size, the scaling interpretation is confounded.
  • standard math Conditional probability identities used for IFFR are standard.
    Eq. 1 defines IFFR as an empirical conditional frequency; no nonstandard mathematical assumption is made.

pith-pipeline@v1.3.0-alltime-deepseek · 13386 in / 17864 out tokens · 171162 ms · 2026-08-01T12:15:20.628697+00:00 · methodology

0 comments
read the original abstract

Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior. We study this across three tasks - multiple-choice question answering (MCQA), sentiment classification, and mathematical question answering - by pairing a standard instruction with a conflicting non-standard one (select an incorrect option, output the opposite sentiment, or return twice the answer). This cross-task design allows us to test whether resistance to conflicting instructions is tied to specific task characteristics or reflects a broader behavioral tendency. As all predictions are scored against the original ground truth, a model that ignores the non-standard instruction still appears accurate. Using standard accuracy, non-standard accuracy, and an Instruction-Following Failure Rate (IFFR), we evaluate instruction-tuned Qwen models across sizes. Both standard accuracy and instruction following generally improve with scale, although the pattern is not consistent across all tasks and datasets. Small models stay competent yet routinely ignore the non-standard instruction, while larger models show a clear gap between the two settings. These findings suggest that gains in task capability do not automatically provide reliable control over model behavior. Task competence and instruction following are therefore distinct abilities, and reporting only standard accuracy hides instruction-following failures.

Figures

Figures reproduced from arXiv: 2607.19608 by Mahdiyeh Farajidizaji, Vatsal Raina.

Figure 1
Figure 1. Figure 1: Instruction-following failures across three tasks. In each example the model is given a non-standard [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Evaluation pipelines for the three tasks. All tasks share a decoder-only Qwen backbone and differ only in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Standard (solid) and non-standard (dashed) accuracy by model size for each task and dataset. Error bars [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: IFFR by model size for each task and dataset. Error bars show the standard deviation across the three [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 6 linked inside Pith

  1. [6]

    Lal Khan, Ammar Amjad, Noman Ashraf, and Hsien- Tsung Chang

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361. Lal Khan, Ammar Amjad, Noman Ashraf, and Hsien- Tsung Chang

    Pith/arXiv arXiv 2001

  2. [9]

    InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium

    Can a suit of armor conduct elec- tricity? a new dataset for open book question an- swering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. Rudra Murthy, Prince Kumar, Praveen Venkateswaran, and Danish Contractor

    2018

  3. [13]

    Jason Wei, Maarten Bosma, Vincent Y

    Do prompt- based models really understand the meaning of their prompts? InProceedings of the 2022 conference of the north american chapter of the association for computational linguistics: Human language tech- nologies, pages 2300–2344. Jason Wei, Maarten Bosma, Vincent Y . Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V . Le

    2022

  4. [14]

    In International Conference on Learning Representa- tions, volume 2024, pages 40193–40219

    Evaluating large lan- guage models at evaluating instruction following. In International Conference on Learning Representa- tions, volume 2024, pages 40193–40219. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Sid- dhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou

    2024

  5. [2015]

    InProceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1743–1752, Lisbon, Portu- gal

    Solving general arith- metic word problems. InProceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1743–1752, Lisbon, Portu- gal. Association for Computational Linguistics. Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, A...

    2015

  6. [2016]

    MAWPS: A math word problem repository. InProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1152–1157, San Diego, California. Association for Computational Linguistics. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy

    2016

  7. [2017]

    In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark

    RACE: Large-scale ReAd- ing comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark. Association for Compu- tational Linguistics. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason We...

    2017

  8. [2018]

    Boyi Deng, Xu Wang, Yaoning Wang, Yu Wan, Yubo Ma, Baosong Yang, Haoran Wei, Jialong Tang, Huan Lin, Ruize Gao, and 1 others

    Think you have solved question an- swering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457. Boyi Deng, Xu Wang, Yaoning Wang, Yu Wan, Yubo Ma, Baosong Yang, Haoran Wei, Jialong Tang, Huan Lin, Ruize Gao, and 1 others

    Pith/arXiv arXiv

  9. [2019]

    Jose Belda-Medina and Vendula Kokosková

    Finbert: Financial sentiment analy- sis with pre-trained language models.arXiv preprint arXiv:1908.10063. Jose Belda-Medina and Vendula Kokosková

    Pith/arXiv arXiv 1908

  10. [2020]

    InAdvances in Neural Information Processing Systems, volume 33, pages 1877–1901

    Language models are few-shot learners. InAdvances in Neural Information Processing Systems, volume 33, pages 1877–1901. Xinyi Chen, Baohao Liao, Jirui Qi, Panagiotis Eustra- tiadis, Christof Monz, Arianna Bisazza, and Maarten de Rijke

    1901

  11. [2021]

    Multitask prompted training en- ables zero-shot task generalization.arXiv preprint arXiv:2110.08207. Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, and 1 others

    Pith/arXiv arXiv

  12. [2022]

    InProceedings of the 2022 conference on empirical methods in natu- ral language processing, pages 5085–5109

    Super-naturalinstructions: Generalization via declar- ative instructions on 1600+ nlp tasks. InProceedings of the 2022 conference on empirical methods in natu- ral language processing, pages 5085–5109. Albert Webson and Ellie Pavlick

    2022

  13. [2023]

    A Experimental Details We evaluated five Qwen3.5 checkpoints: Qwen/Qwen3.5-0.8B, Qwen/Qwen3.5-2B, Qwen/Qwen3.5-4B, Qwen/Qwen3.5-9B, and Qwen/Qwen3.5-27B

    Instruction-following evalu- ation for large language models.arXiv preprint arXiv:2311.07911. A Experimental Details We evaluated five Qwen3.5 checkpoints: Qwen/Qwen3.5-0.8B, Qwen/Qwen3.5-2B, Qwen/Qwen3.5-4B, Qwen/Qwen3.5-9B, and Qwen/Qwen3.5-27B. For MCQA, all final ex- periments used the DeepInfra API through the OpenAI-compatible chat-completions inter...

    Pith/arXiv arXiv

  14. [2024]

    InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 1691–1706

    The sifo benchmark: Investigating the sequential instruction following ability of large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 1691–1706. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others

    2024

  15. [2026]

    Tingchen Fu, Yafu Li, Jiawei Gu, Xiaoye Qu, and Yu Cheng

    Qwen- scope: Turning sparse features into development tools for large language models.arXiv preprint arXiv:2605.11887. Tingchen Fu, Yafu Li, Jiawei Gu, Xiaoye Qu, and Yu Cheng

    Pith/arXiv arXiv