{"id":"b193e01e-e470-4842-ad8d-ac8679d36a15","arxiv_id":"2412.19450","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"IoInst evaluates LLMs by asking them to identify which of four instructions generated a given response, and shows that current models often fail, particularly with semantically similar distractors.","lead":"This paper introduces IoInst, a benchmark that gives a language model a response and four candidate instructions and asks it to pick which instruction produced the response. It finds that most current models often pick or follow the wrong instruction, especially when the wrong instructions are very similar to the right one.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EMsparse's ROUGE-L precision threshold (tau=0.9) is unvalidated and can misclassify semantically correct selections, so the reported ACC1/ACC2/ACCrel1 numbers may not support the central claim as stated.","rationale":"The reader's weakest assumption is the same point I find most load-bearing: EMsparse with tau=0.9 is the only bridge from raw generations to accuracies, and the paper supplies no validation of that bridge. I considered whether a different concern, such as the absence of a human or chance baseline, was more fundamental; it is secondary. A chance baseline would help contextualize the numbers, but it would not repair a metric that mislabels individual answers. The benchmark itself has real strengths: the data and code are public, GPT-4o's near-perfect ACCrel1 on Random shows the task is solvable, and the qualitative outputs in Tables 13-14 demonstrate genuine Case 2 failures. Those strengths indicate the qualitative claim has force; the problem is that the quantitative headline is not yet supported without an audit of EMsparse. A human-annotation study on a sample of outputs plus a tau sweep would settle this. Until then, CONDITIONAL is the right verdict, so I do not change the reader's decision.","tokens_in":21563,"tokens_out":7584,"duration_ms":73114,"concrete_test":"Using the released code and data, sample a stratified set of 300 model outputs across splits and models, and have two annotators independently label which candidate instruction (if any) the output selects, with disagreements adjudicated. Compare these human labels to EMsparse(tau=0.9) and recompute ACC1, ACC2, and ACCrel1; also sweep tau over {0.7, 0.8, 0.9, 0.95} on the full evaluation logs. If human-based accuracy differs from EMsparse-based accuracy by more than 5 absolute points, or if the ranking of models changes, the quantitative support for the paper's central claim needs to be re-estimated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.4 defines EMsparse as ROUGE-L precision against a candidate instruction exceeding tau=0.9. This strict lexical rule means any preamble, suffix, or paraphrase makes a correct selection look like a failure. Table 13's Mistral output is a clear example: it chooses the label instruction, but in expanded prose, so EMsparse against the label and likely against any candidate falls below 0.9, and the model is scored as not following the meta-instruction. Since ACC1, ACC2, and ACCrel1 in Table 3 all come from Equation (1), the headline claim that state-of-the-art models 'still lack instruction understanding capability' rests on this unvalidated mapping from generated text to choice. The paper reports no human agreement check for EMsparse and no sensitivity analysis for tau; the phrase 'empirically set' is not backed by evidence in the text. Even a short prefix such as 'The correct answer is:' can push a 50-word quoted instruction below 0.9 (50/56 is about 0.89). This does not undermine the benchmark's construction or the qualitative Case 2 examples, but it does undermine the quantitative accuracies and the strength of the central conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces IoInst, a benchmark that probes whether LLMs can distinguish the single instruction that actually generated a given context from three instruction-formatted distractors. Three contrastive types are constructed (Random, Semantic, Anti-Attribute), and three metrics (ACC1, ACC2, ACCrel1) are defined, all derived from a single lexical scoring rule called EMsparse, a ROUGE-L precision comparison against the candidate instruction strings with a hand-set threshold of 0.9. Experiments with ten instruction-tuned LLMs show generally low ACC1 and ACC2, especially on the Anti-Attribute setting, and the authors conclude that even recent state-of-the-art models lack instruction understanding. Additional analyses cover the effect of meta-instruction wording, few-shot in-context learning, and temperature.","tokens_in":21845,"tokens_out":8708,"duration_ms":73366,"significance":"If the measurements are valid, the paper identifies a genuinely under-tested failure mode: models that follow instructions competently in standard settings may nonetheless answer the wrong meta-task when instruction-formatted text is embedded in the prompt. The benchmark design is thoughtful, with a useful Case 1/Case 2 error taxonomy, three contrastive difficulty levels, publicly released code and data, and broad coverage across ten open and closed models. The analyses of meta-instruction variants and few-shot behavior are informative. However, the quantitative conclusion rests almost entirely on the validity of EMsparse, and the Anti-Attribute construction raises label-uniqueness concerns; both need to be addressed before the central claim can be considered established.","major_comments":[{"comment":"The paper's headline quantitative claim is carried by EMsparse, defined in Section 3.4 as ROUGE-L precision against a candidate instruction exceeding a threshold tau 'empirically set to 0.9.' The manuscript reports no calibration of this threshold, no sensitivity analysis over tau, and no human-agreement check. Table 13 shows model outputs that select the correct instruction but not verbatim: Mistral's first response correctly chooses 'Come up with three date ideas...' while restating it in expanded prose, and Solar's response also identifies the right option with added reasoning. Such outputs can easily fall below 0.9 ROUGE-L precision because of prefixes, suffixes, and paraphrase, so ACC1 and ACC2 in Table 3 may substantially understate the models' true selection accuracy. Since the abstract's conclusion that state-of-the-art models 'still lack instruction understanding capability' is inferred from these numbers, the metric needs validation: please report a human-annotated subset, a tau sweep (for example 0.7 to 1.0), or a secondary exact-match/extracted-answer metric.","section":"§3.4, Eq. (1); Table 13"},{"comment":"The Anti-Attribute contrastive instructions appear to include options that are also satisfied by the context, which would make the intended label non-unique. In the elephant-story example of Table 12, the contrastive 'Your answer should be formatted to contain 3 sentences' matches the three-sentence context, and 'Ensure words in complete capitals are utilized fewer than 4 times' is also satisfied by the context; in the database example, the 'Start off your answer with ...' and 'goes beyond 50 words' contrastives are satisfied by the provided context. Section 4.3 states that the contrastive instructions are designed so that 'none of the contrastive instructions align with the context,' but the construction based on context attributes includes 'aligned' instructions (Table 9), and the examples suggest such aligned variants are actually used as distractors. If several candidates can generate the same context, the low ACC1 and ACCrel1 values for the Anti-Attribute setting in Table 3 may reflect ambiguous labels rather than a lack of instruction understanding. Please clarify the selection procedure, verify label uniqueness for every Anti-Attribute item, and re-run the affected experiments.","section":"§4.3, Tables 9 and 12"},{"comment":"The benchmark's validity is partly self-referential because GPT-4o is used in dataset construction and also serves as an evaluated model. Section 4.2 reports that GPT-4o reviewed failure cases and regenerated contrastive instructions, and Section 5.1 evaluates GPT-4o on the resulting dataset, where it attains 99.54 ACC1 in the Random setting. The absence of a human performance baseline is a concrete gap: the task is claimed to measure a human-like 'instruction understanding' ability, but the only feasibility check is a GPT-4o result. I am not claiming the benchmark is invalid for this reason alone, but the manuscript should provide an independent validity check, such as human accuracy on a sample, or an evaluation using a model that was not involved in data construction. Without it, the claim that the task is 'clear and solvable enough' rests on the very model family being measured.","section":"§4.2 and §5.1"}],"minor_comments":[{"comment":"There is a typo in the sentence 'LLMs are inclined to distracted by several isntruction-formatted statements'; it should read '...to be distracted by several instruction-formatted statements.'","section":"§5.2"},{"comment":"The benchmark name is inconsistently rendered as IOINST, INTINST, and IOI. Please use a single consistent name (and define it once).","section":"Throughout"},{"comment":"The metric is called ACCrel1 in Equation (1) but the figures and some tables use 'ACC1-rel'; standardize the notation.","section":"Table 3 and Figure 4"},{"comment":"The 'Error Rate(Case 2)' column in Table 4 is not given a precise definition with a denominator. State whether it is the fraction of all model outputs that are Case 2 errors for a given candidate instruction, and across which model set and settings.","section":"§5.3, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is valuable and the experimental scope is broad, but the quantitative claims are not yet supported because the only scoring function (EMsparse) is unvalidated and the Anti-Attribute data appear to contain non-unique labels in the provided examples. I recommend asking the authors to add a human baseline, a sensitivity analysis for tau, and a label-uniqueness audit for Anti-Attribute. If those checks cannot be done, the conclusion should be scaled back to a qualitative observation about error modes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely new evaluation setup, but the central numbers need a grain of salt until the metric is fixed.\n\nThe core idea is good. Reversing instruction-following—give the model a response, ask it to pick which of four instructions produced it—is not something IFEval, MT-Bench, or InfoBench do. The three distractor types are well thought out, and Anti-Attribute is the clever one: by taking the label instruction and appending attribute constraints that either match or contradict the response, it forces the model to check fine-grained details. The paper also does solid work on data curation, releases code and data, and includes useful analyses of meta-instruction design, temperature, and few-shot behavior. The Case 1/Case 2 error taxonomy is a simple but helpful framing.\n\nThe soft spot is exactly where the stress-test note points: EMsparse. Defining a correct selection as ROUGE-L precision above 0.9 against the candidate text means any preamble, suffix, or paraphrase makes a correct answer look wrong. Table 13 gives a concrete instance—Mistral selects the right instruction but in expanded prose, so it would be scored as a failure. Since ACC1, ACC2, and ACCrel1 all run through this filter, the reported accuracies are probably understated for models that don't quote the option verbatim. There is no human-agreement check for EMsparse, no sensitivity analysis for tau, and no human baseline. That last one matters because the paper uses GPT-4o to validate the data and then evaluates GPT-4o, which is circular.\n\nThat said, the EMsparse flaw doesn't erase the qualitative phenomenon. The Table 13 examples of Gemma following a candidate instruction and Solar confidently choosing a non-matching option are real and visible regardless of the metric. What it does is undercut the precision of the headline claim: the specific numbers, like GPT-4o's 48.74 ACC1 on Anti-Attribute, should not be taken at face value until EMsparse is validated against human judgments and a tau sweep is reported.\n\nWho is this for? Anyone building instruction-following benchmarks or studying prompt robustness. The benchmark contribution is worth engaging with, and the data being public means other groups can re-evaluate with better metrics. I'd send it to a serious referee—the design is novel and the flaws are addressable—but I'd expect the revision to include human evaluation of EMsparse, a human accuracy baseline, and sensitivity analysis for the threshold. As it stands, cite it cautiously and don't rely on its accuracy tables.","headline":"IoInst is a new and useful benchmark idea, but the unvalidated EMsparse metric makes the headline accuracy numbers provisional.","tokens_in":22363,"tokens_out":3843,"would_cite":false,"duration_ms":35153,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs frequently respond to instruction-formatted distractors instead of the user's true intent, and current state-of-the-art models still lack this instruction understanding capability.","keywords":["instruction understanding","instruction following evaluation","LLM benchmark","contrastive instruction","Anti-Attribute","intention of instruction","meta-instruction","EMsparse"],"falsifier":"If a human re-annotation of the model outputs shows that a large fraction of responses judged as errors are actually correct paraphrases of the label instruction (e.g., Mistral's 'Create a list of three date ideas...' in Table 13), then the paper's conclusion that SOTA models lack instruction understanding would be an artifact of EMsparse, not a property of the models. Concretely: recompute ACC1 and ACC_rel1 using a judge that accepts semantically equivalent selections, and compare with the paper's scores; a large gap would falsify the claim as measured.","tokens_in":21402,"feed_emoji":"🎯","tokens_out":6161,"duration_ms":48842,"temperature":0.7,"pith_summary":"The paper introduces the Intention of Instruction (IoInst) benchmark, which asks a language model to choose which of four candidate instructions would have produced a given response, with the other three designed as distractors. The authors report that most instruction-tuned LLMs, including recent state-of-the-art models, often follow a distractor instruction instead of the meta-instruction that asks them to select an option. Across three distractor types, performance falls hardest when the distractors share the label instruction's core requirement but violate fine-grained constraints, the Anti-Attribute setting. The paper concludes that current instruction-following scores on clean prompts overstate how well models actually understand the intent behind instructions. If correct, this implies that tasks involving multiple instruction-formatted statements, such as prompt injection or instruction optimization, need evaluation that specifically tests intention discrimination.","feed_headline":"New test: LLMs follow the wrong instruction when offered distractors","feed_subtitle":"IoInst benchmark shows even GPT-4o picks instruction-formatted noise over the user's actual intent in the hardest setting.","key_machinery":"The load-bearing object is the IoInst benchmark itself. Contexts come from curated instruction-following datasets (Zeng et al., 2024; Zheng et al., 2023; Chia et al., 2024; et al.), with the label instruction paired with three distractors. Random and Semantic distractors are drawn from a candidate pool, with MPNet embedding similarity used to select semantically confusing options; Anti-Attribute distractors are built from heuristic attributes of the context (word counts, keyword usage, sentence boundaries, etc.) and are concatenated to the label instruction's main clause. Scoring is fully automatic via EMsparse: a generation counts as selecting a candidate instruction when its ROUGE-L precision against that candidate's text exceeds tau=0.9, which drives ACC1, ACC2, and ACC_rel1. The meta-instruction is varied across 16 formats (detailed vs simple; context-first vs candidate-first) to test sensitivity to instruction phrasing.","core_discovery":"The central claim is that LLMs lack robust instruction understanding: given a context, four candidate instructions, and a meta-instruction to pick the one that generated the context, models are prone to respond to the candidate instructions themselves. The paper defines three contrastive settings: Random (unrelated distractors), Semantic (distractors similar in meaning to the label), and Anti-Attribute (distractors that restate the label's main task but add requirements incompatible with the context). It measures ACC1 (correct label selection), ACC2 (following the meta-instruction by picking any candidate), and ACC_rel1 = ACC1/ACC2, using ROUGE-L precision against the option text with a threshold of 0.9 to decide whether a generation selected an option. Results show e.g. GPT-4o reaching ACC1 95.31 on Random but 48.74 on Anti-Attribute, while most open models score far lower; the paper interprets this as evidence that models are distracted by instruction-formatted statements and that high instruction-following performance does not imply deep instruction understanding.","pith_inferences":["This benchmark could serve as a proxy for vulnerability to prompt injection: if a model cannot distinguish the true instruction from instruction-formatted distractors in a benign setting, it will likely be hard to defend against injected instructions that look similar.","The metric could be extended to score paraphrased selections: using semantic equivalence or another LLM judge instead of ROUGE-L threshold might reveal that some models understand the task but answer in their own words, which would change the quantitative conclusion.","The anti-attribute data may be reusable as a fine-tuning set to teach models to check constraints against context, a testable intervention the paper does not run.","The few-shot degradation suggests that in-context exemplars themselves act as instruction-formatted noise; model developers may need to separate demonstration content from instruction content."],"forward_implications":["If IoInst measures what it claims, high scores on standard instruction-following benchmarks (clean, single-intent prompts) should not be taken as evidence that a model can pick out the intended instruction when other instruction-looking text is present.","The Anti-Attribute results imply that models latch onto the repeated core task phrase ('Write a poem...') and miss appended constraints that contradict the context; targeted fine-tuning on such fine-grained mismatches may be needed.","Because a substantial fraction of errors are Case 2 (not choosing any candidate), the paper implies that evaluation of instruction following should inspect whether the model obeys the meta-instruction, not just the final content.","Detailed meta-instructions yield higher and more stable accuracy, so prompt design choices directly change apparent instruction understanding; evaluations that fix one prompt format may not generalize."],"supporting_citations":[{"why":"Supplies LLM-as-evaluator data and responses that IoInst re-organizes into label instruction/context pairs.","marker":"(Zeng et al., 2024)"},{"why":"Supplies LLM-as-evaluator data and responses, a further source of context-label pairs.","marker":"(Zheng et al., 2023)"},{"why":"Supplies both curated context-label pairs and instructions placed into the candidate pool.","marker":"(Chia et al., 2024)"},{"why":"Provides the rule-based instruction-following data and the attribute/segmentation tools used to construct Anti-Attribute distractors.","marker":"(Zhou et al., 2023b)"},{"why":"Defines ROUGE-L, the similarity measure underlying the EMsparse threshold scorer.","marker":"(Lin, 2004)"},{"why":"Provides MPNet, the sentence embedding used to select semantically similar contrastive instructions.","marker":"(Song et al., 2020)"},{"why":"Contributes instructions to the candidate pool used for Random and Semantic contrastive construction.","marker":"(Qin et al., 2024)"}],"fun_headline_variants":["LLMs struggle to pick the right instruction when distractors mimic it","Even GPT-4o falls for decoy instructions in new benchmark","IoInst reveals: LLMs chase decoy instructions even in hard tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatic scorer treats a model answer as selecting an option only when the generated text is nearly a verbatim match of that option (ROUGE-L precision above 0.9), so a model that understands the task but paraphrases the correct instruction is counted as wrong; the reported accuracies rest on this lexical assumption.","fun_headline_variants_meta":{"raw":{"variants":["LLMs struggle to pick the right instruction when distractors mimic it","Even GPT-4o falls for decoy instructions in new benchmark","IoInst reveals: LLMs chase decoy instructions even in hard tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00052,"raw_usage":{"total_tokens":2526,"prompt_tokens":959,"completion_tokens":1567,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1507}},"tokens_in":575,"tokens_out":1567,"duration_ms":10842,"temperature":1.0,"reasoning_tokens":1507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:35:08.940326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a human re-annotation of the model outputs shows that a large fraction of responses judged as errors are actually correct paraphrases of the label instruction (e.g., Mistral's 'Create a list of three date ideas...' in Table 13), then the paper's conclusion that SOTA models lack instruction understanding would be an artifact of EMsparse, not a property of the models. Concretely: recompute ACC1 and ACC_rel1 using a judge that accepts semantically equivalent selections, and compare with the paper's scores; a large gap would falsify the claim as measured.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies both curated context-label pairs and instructions placed into the candidate pool."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines ROUGE-L, the similarity measure underlying the EMsparse threshold scorer."}],"review_version":1}