Pith. sign in

REVIEW 4 major objections 9 minor 1 cited by

How Many Instructions Can LLMs Follow at Once?

T0 review · 4 major / 9 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Frontier LLMs satisfy just 69% of 500 simultaneous instructions

desk verdict Useful new benchmark with a real density sweep, but the grader's matching rules are underspecified enough that the headline accuracy numbers need verification before I'd trust them exactly. read the letter →

arxiv 2507.11538 v1 pith:4PPQOK5Y submitted 2025-07-15 cs.AI

Daniel Jaroslawicz , Brendan Whiting , Parth Shah , Karime Maamari This is my paper
classification cs.AI
keywords IFScaleinstructionfollowingbenchmarkdensitydegradationpatternsprimacyeffectomissionerrorsbusinessreportgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

IFScale is a benchmark that asks models to write a coherent business report while including a list of exact keywords, scaled from 10 to 500 instructions. The paper argues that this makes instruction density a controlled variable, and that density, not task difficulty alone, drives failure. Across 20 models from seven providers, the best performer satisfies 68.9% of 500 simultaneous instructions, and every model degrades measurably as density grows. The paper's main claim is that the degradation follows a small number of distinct patterns — threshold, linear, and exponential decay — and that a universal bias toward earlier instructions peaks in the 150–200 range before collapsing into uniform failure. Understanding these curves matters because real production prompts increasingly pack dozens or hundreds of constraints into a single call.

What carries the argument

The load-bearing object is the IFScale benchmark itself: a vocabulary of 500 business terms extracted from SEC 10-K filings, filtered by Zipf frequency, lemmatization, embedding-based semantic deduplication, and generation-difficulty ranking, with prompts that demand each keyword appear as the exact literal word and prohibit listing constraints. Grading is deterministic: case-insensitive regex matching counts an instruction satisfied only if the exact word appears, and an 80%-length prefix match counts as a modification error. Primacy is quantified as the ratio of error rates in the last third of the instruction list to the first third, with ratios above 1.0 meaning later instructions are more likely to be violated. This machinery turns instruction density into a controlled independent variable and makes the three decay curves, the mid-range primacy peak, and the omission-to-modification shift directly measurable.

What would settle it

Manually audit a random sample of generated reports at densities 100 and 500: have human annotators mark whether each required keyword appears as an exact word, then compare against the regex grader's verdict. If agreement falls well below 95%, the reported accuracy numbers and the shape of the decay curves are artifacts of the grader rather than genuine model behavior.

Watch

Extended reading notes

Core claim

IFScale defines instruction following as the verbatim inclusion of specified keywords in a professional business report. On this task the paper finds that no evaluated model at maximum density of 500 instructions comes close to full adherence; gemini-2.5-pro-preview reaches 68.9%, o3 (high) 62.8%, and grok-3-beta 61.9%, while many smaller models fall below 20%. Performance degrades in three recognizable shapes: reasoning models hold near-perfect accuracy through a threshold around 100–250 instructions before a steeper, higher-variance decline; several general-purpose models decay roughly linearly; and a third group collapses exponentially to a low floor. The paper further claims that primacy effects — better adherence to earlier instructions — are universal, peak at moderate densities, and converge toward neutral at extreme densities, and that errors shift from morphological modification to outright omission as load increases.

Load-bearing premise

The grader treats an instruction as satisfied only when the exact word shows up in the report, with regex matching whose word-boundary and case/plurality rules are not fully specified; if that counting is off, the 68.9% headline and the three decay patterns would shift.

Editorial extensions

If this is right

  • Reasoning models such as gemini-2.5-pro-preview and o3 (high) are the only ones that stay near-perfect past 100 instructions, so applications needing faithful execution of 200+ constraints should expect to use such models and to accept their longer latencies.
  • Models with exponential decay, such as gpt-4o and llama-4-scout, lose most of their adherence by 100 instructions; they are suited only to low-density prompts.
  • Because primacy effects peak around 150–200 instructions and then level off, instruction ordering as a mitigation strategy helps at moderate densities but stops working once models are saturated.
  • At high densities, failures are overwhelmingly omissions rather than word-form modifications, meaning models stop trying to satisfy instructions instead of approximating them.
  • Accuracy per unit latency favors fast small models, so for time-sensitive production systems the highest-accuracy reasoning models are often not the economically best choice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the three decay curves could be tested as a selection heuristic: threshold models for high-integrity tasks, linear models for predictable degradation, exponential models only when density is low.
  • The reported numbers measure verbatim keyword inclusion, not semantic compliance; if instructions were graded on paraphrase or intent instead of exact words, accuracy levels and possibly curve shapes would change.
  • The universal convergence of primacy ratios toward 1.0 at extreme densities hints at a per-model saturation capacity that could be estimated from a single model family and compared across tasks.
  • A natural next experiment is to vary instruction type (formatting, ordering, compositional) at the same densities to see whether the threshold/linear/exponential taxonomy and the 150–200 primacy peak are stable across task families.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 9 minor

Summary. IFScale constructs a 500-term business vocabulary from SEC 10-K filings and asks models to write a professional business report that includes an exact keyword for each of N instructions, sweeping N from 10 to 500 in steps of 10 with five random seeds across 20 models from seven providers. Instruction adherence is graded by deterministic regex matching, and the paper reports per-density accuracy, variance, primacy effects (final-third versus first-third error ratios), omission-versus-modification error ratios, latency, and LLM-judged coherence. The central empirical findings are: (1) all models decline with instruction density, with the best model (gemini-2.5-pro-preview) satisfying 68.9% of 500 instructions; (2) three visually identified degradation patterns (threshold, linear, exponential); (3) a claimed universal mid-range peak in primacy effects around 150–200 instructions; (4) a systematic shift from modification to omission errors at high density; and (5) accuracy-latency trade-offs that favor smaller, faster models. The authors also probe reasoning-effort and thinking-mode variants for a subset of models and provide a sample 500-instruction output.

Significance. The paper addresses a genuine and underexplored question — how instruction-following degrades when a single prompt carries tens to hundreds of constraints — and the benchmark design is simple, reproducible, and open-sourced, which is a real strength. The headline measurement (even the strongest evaluated model follows fewer than 70% of 500 keyword instructions, with most models collapsing well before that) is a concrete, falsifiable result of practical relevance, and the error-type shift from modification to omission is a useful behavioral observation. Because instruction adherence is graded deterministically rather than by an LLM judge, the accuracy numbers are in principle machine-checkable, and the five-seed variance reporting is good practice. The main weaknesses are that the grader's matching semantics and the retry pipeline are underspecified, and that the degradation taxonomy and the 'universal' primacy claim are asserted from visual inspection and smoothed curves without quantitative fitting, statistics, or confidence intervals; the stress-test concern about substring matching therefore lands directly on the paper's central dependent variable.

major comments (4)
  1. [§4.2 / §3.2 / Appendix D] The grader is the dependent variable for every result in the paper, yet its matching semantics are underspecified. Appendix D defines compliance as the exact literal word and explicitly rules out 'customers' for 'customer' and 'customer-driven' for 'customer', but §4.2 only says the grader performs 'case-insensitive, style-insensitive exact-match searches using regular expressions', while §3.2 mentions 'wildcard-enabled regex matching'; nowhere is word-boundary or hyphenation handling stated. A substring-based regex would count 'customers' and 'customer-driven' as satisfying 'customer', and would also match short vocabulary items such as 'vest', 'debt', 'edge', 'range', 'solar', and 'legal' inside frequent business words ('investment', 'indebtedness', 'knowledge', 'arrangement', 'illegal'), inflating accuracy precisely at high densities where §4.8 says morphological variants are common and flattening the reported degradation curves. Because instruction accuracy is the dependent variable for the pattern taxonomy, primacy analysis, error-type analysis, and model rankings, the authors must report the exact matching procedure (token identity or a word-boundary regex) and re-derive or validate the numbers in Table 1 under strict matching; the contradiction between 'exact-match' (§4.2) and 'wildcard-enabled' (§3.2) must also be resolved.
  2. [§4.4 / Abstract] The three-way degradation taxonomy (threshold, linear, exponential) is asserted from visual inspection of Figure 2 with no fitting procedure, no goodness-of-fit measure, and no classification rule, so the assignment of models to patterns is not falsifiable. In particular, the abstract's claim that model size and reasoning capability 'correlate' with the three patterns is unsupported by the tabulated data: the reasoning models deepseek-r1 (30.9% at N=500) and o4-mini (34.4%) degrade in ways that do not resemble the threshold pattern of gemini-2.5-pro and o3, while grok-3 (61.9%, non-reasoning) exhibits a curve visually similar to the threshold exemplars but is not classified as one. The authors should fit explicit functional forms (e.g., a piecewise-linear threshold model, a linear model, and an exponential model) to each model's accuracy curve, report fit statistics and a decision rule for pattern assignment, and quantify any correlation between pattern membership and model attributes such as parameter count or reasoning mode.
  3. [§4.6 / Table 1 / §5] The claimed 'universal' mid-range peak in primacy effects is not visible in the paper's own tabulated data. In Table 1, llama-4-scout's primacy ratio declines monotonically from 1.78 at N=50 to 1.05 at N=500, deepseek-r1 shows no 150–200 peak (1.24 at 100, 1.55 at 250, 1.25 at 500), and gpt-4o's ratios are essentially flat (1.67, 1.42, 1.31, 1.14), so none of these display a peak around 150–200 instructions. Many rows also contain entries of exactly 0.00 or '-', indicating division by a zero first-third error rate, and no confidence intervals are given, so the smoothed curves in Figure 3 may partly be artifacts of averaging over highly unstable ratios. Because the Discussion (§5) treats 'the universal mid-range peak' as evidence of an architectural limitation and bases a practical recommendation (instruction ordering) on it, the authors need a quantitative test, such as per-seed peak detection or model-averaged error bars, before that claim can stand.
  4. [§3.2 / §4.1] The retry logic is unquantified and can differentially affect the measured accuracy. Section 3.2 states that prompts are re-issued when outputs contain lists of constraints, refusals (fewer than 20 words), or incoherent reports validated by an o4-mini coherence check, but the paper never reports how often retries occurred per model or per density. If some models frequently refuse or produce incoherent output at high density and are retried until compliant, the reported accuracy confounds native compliance with retry yield; if other models' incoherent outputs are not retried, the cross-model comparison is unfair. In addition, the list-detection heuristic ('≥10 comma-separated single words') does not match the prompt's own numbered constraint format in Appendix D, so a model that reproduces the constraint list verbatim may survive to grading and receive credit for keywords it merely listed, directly inflating accuracy. The authors should report retry counts, exclude or flag any run containing a constraint list, and state whether retry time is included in the reported latency figures.
minor comments (9)
  1. [§4.5] Section 4.5 says the three variance behaviors are 'shown in Appendix B.3', but variance results appear in Appendix B.1 (Figure 4); Appendix B.3 reports efficiency results.
  2. [§4.7 / Appendix B.3] The 'accuracy-to-latency efficiency ratio' is referred to repeatedly but never defined; a formula or normalization description should be given.
  3. [§3.1] The description 'three zero-temperature gpt-4.1-nano completions' is contradictory, since temperature 0 is deterministic; the sampling configuration should be clarified.
  4. [§3.1] The redundancy pruning criterion ('prune any term whose distance falls below the mean') does not state the distribution whose mean is used; specify that it is the mean of nearest-neighbor cosine distances and report the number of terms pruned at each filtering step.
  5. [§4.8 / §4.2] The modification-error examples in §4.8 ('accountable', 'accounts' for 'accountability') are inconsistent with the 80%-prefix rule in §4.2 under the natural reading, since the common prefix is about 9 of 13 characters for 'accountable'; the paper should define whether the prefix ratio is computed against the required term or the shorter matched token so that the rule and the examples agree.
  6. [Table 1] The meaning of '-' versus 0.00 in the primacy-effect and omission-modification rows should be defined in a footnote, since '-' appears to denote undefined ratios when the first-third error rate is zero.
  7. [Appendix B.4.3] The grok-3 sample report at 500 instructions is shown without its o4-mini coherence score, so the claim that grok-3 'maintains high coherence' cannot be assessed from the example alone.
  8. [§4.3 / throughout] Several comparative claims (e.g., grok-3 at 61.9% 'approaches' o3 at 62.8%, and the ordering of mid-tier models) are asserted without significance tests; with five seeds and the reported standard deviations, some of these orderings are within noise and should be flagged as such.
  9. [Various] Minor typos and inconsistencies: 'mantain' in the Appendix B.4 caption, 'V osoughi' in the Guo & Vosoughi reference, and inconsistent rendering of model names such as 'claude-3.7-sonnet' versus 'claude-sonnet-4'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; IFScale is a direct empirical measurement with no load-bearing self-citation.

full rationale

IFScale is an empirical measurement study, not a derivation. The central quantities—per-instruction accuracy, omission/modification error rates, primacy ratios, and latency—are directly computed from model outputs via regex matching on the instructed keyword, so there is no equation in which an output is presupposed by an input. The vocabulary is constructed through an explicit pipeline (o4-mini extraction, embedding-based deduplication, gpt-4.1-nano perplexity ranking), but this pipeline selects the test stimuli; it does not define the measured accuracy, and the paper makes no claim that perplexity predicts or forces the reported rankings. The degradation patterns (threshold/linear/exponential), the mid-range primacy peak, and the omission-to-modification shift are descriptive statistics over the collected outputs, not consequences of a fitted model. The only self-citation (Maamari et al., 2024) is used as an example of the retrieval/decomposition paradigm in the introduction and as a related benchmark in Section 2; it is not invoked as evidence for any IFScale result. The reviewer-flagged regex grader concern is a measurement-validity question about whether substring matches inflate accuracy; the paper's prompt requires exact literal words (Appendix D), but without grader code or word-boundary specification this is an external validity issue, not a circularity in the paper's own derivation. The limitations in Section 6 are explicit scope acknowledgments and do not reveal a circular step. Accordingly, no circular steps are identified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on fitted parameters or newly postulated entities. The main assumptions are domain-level choices: keyword inclusion as a proxy for instructions, OpenRouter latency as efficiency, five seeds as sufficient for variance estimation, and o4-mini as an unbiased judge. No mathematical axioms are invoked.

assumptions (4)
  • domain assumption Keyword-inclusion instructions are a valid proxy for real-world instruction following.
    The task requires exact inclusion of a list of words in a business report, which strips away semantic complexity but captures only one dimension of instruction following; the paper's conclusions are limited to this task, yet the benchmark is presented as measuring instruction-following capability broadly (Sections 1 and 6).
  • domain assumption OpenRouter API latency is a meaningful measure of model efficiency.
    Latency is measured via a third-party API with default parameters and no control for server load, yet it is used to compute accuracy-per-unit-latency efficiency ratios (Section 4.7).
  • domain assumption Five random seeds with stratified difficulty sampling provide stable accuracy estimates.
    The paper computes standard deviations over only five runs per density and uses these to draw conclusions about variance patterns and reliability (Sections 4.2 and 4.5).
  • domain assumption o4-mini provides an unbiased coherence judgment for retry filtering and core-task evaluation.
    The same model family is used for vocabulary extraction and coherence scoring, which could introduce systematic bias, particularly for o-series models' outputs (Sections 3.1, 4.9, and Appendix B.4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Many Instructions Can LLMs Follow at Once?." pith.science (2026). https://pith.science/paper/4PPQOK5Y

@misc{pith2026250711538,
  author       = {Pith},
  title        = {Pith review of: How Many Instructions Can LLMs Follow at Once?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PPQOK5Y}},
  note         = {Machine review of arXiv:2507.11538}
}
read the original abstract

Production-grade LLM systems require robust adherence to dozens or even hundreds of instructions simultaneously. However, the instruction-following capabilities of LLMs at high instruction densities have not yet been characterized, as existing benchmarks only evaluate models on tasks with a single or few instructions. We introduce IFScale, a simple benchmark of 500 keyword-inclusion instructions for a business report writing task to measure how instruction-following performance degrades as instruction density increases. We evaluate 20 state-of-the-art models across seven major providers and find that even the best frontier models only achieve 68% accuracy at the max density of 500 instructions. Our analysis reveals model size and reasoning capability to correlate with 3 distinct performance degradation patterns, bias towards earlier instructions, and distinct categories of instruction-following errors. Our insights can help inform design of instruction-dense prompts in real-world applications and highlight important performance-latency tradeoffs. We open-source the benchmark and all results for further analysis at https://distylai.github.io/IFScale.

Figures

Figures reproduced from arXiv: 2507.11538 by the authors.

Figure 1
Figure 1. Model instruction-following accuracy across increasing densities, averaged over 5 runs. Three [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Model performance degradation as instruction density increases from 10 to 500 instructions, with [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Primacy effect ratios showing universal attention degradation patterns regardless of instruction [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Performance variance patterns revealing three distinct behaviors: top performing models display [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Omission to modification error ratio patterns across instruction densities. Models grouped into three [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Accuracy per unit latency across instruction densities, revealing efficiency trade-offs. Models [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Coherence of model generated business reports as judged by an LLM. Most models mantain high [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Average generated tokens for top performing models. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: o3 run with "high" and "medium" reasoning efforts. High reasoning effort provides moderate performance gains at high instruction densities. C.2 CLAUDE HYBRID MODEL THINKING RESULTS [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: claude-sonnet-4 and claude-opus-4 evaluated with and without thinking enabled. Enabling thinking provides moderate performance gains at high instruction densities. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Across five LLMs, prompt format effects are real but model- and scale-specific; instruction-following collapses by 80 rules, and context pressure causes refusal, not hallucination.

Reference graph

Works this paper leans on

34 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  3. [3]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  4. [4]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

    work page

  5. [5]

    Chan, Chao-Ting Chen, Jui-Hung Cheng, and Hen-Hsen Huang

    Brian J. Chan, Chao-Ting Chen, Jui-Hung Cheng, and Hen-Hsen Huang. Don’t do rag: When cache-augmented generation is all you need for knowledge tasks. In Companion Proceedings of the ACM on Web Conference 2025, WWW ’25, pp.\ 893–897. ACM, May 2025. doi:10.1145/3701716.3715490. URL http://dx.doi.org/10.1145/3701716.3715490

    work page arXiv 2025

  6. [6]

    Kakkar, Yu Gan, Brenton Milne, and Fatma Ozcan

    Yeounoh Chung, Gaurav T. Kakkar, Yu Gan, Brenton Milne, and Fatma Ozcan. Is long context all you need? leveraging llm's extended context for nl2sql, 2025. URL https://arxiv.org/abs/2501.12372

    arXiv 2025

  7. [7]

    Dynamic multi-reward weighting for multi-style controllable generation, 2024

    Karin de Langis, Ryan Koo, and Dongyeop Kang. Dynamic multi-reward weighting for multi-style controllable generation, 2024. URL https://arxiv.org/abs/2402.14146

    arXiv 2024

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    arXiv 2025

Show all 34 references
  1. [9]

    Serial position effects of large language models, 2024

    Xiaobo Guo and Soroush Vosoughi. Serial position effects of large language models, 2024. URL https://arxiv.org/abs/2406.15981

    2024 arXiv

  2. [10]

    Multi-if: Benchmarking llms on multi-turn and multilingual instructions following, 2024

    Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abinav Sankararaman, Eryk Helenowski, Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. Multi-if: Benchmar...

    2024 arXiv

  3. [11]

    Llm agents display human biases but exhibit distinct learning patterns, 2025

    Idan Horowitz and Ori Plonsky. Llm agents display human biases but exhibit distinct learning patterns, 2025. URL https://arxiv.org/abs/2503.10248

    2025 arXiv

  4. [12]

    Llms can be easily confused by instructional distractions, 2025

    Yerin Hwang, Yongil Kim, Jahyun Koo, Taegwan Kang, Hyunkyung Bae, and Kyomin Jung. Llms can be easily confused by instructional distractions, 2025. URL https://arxiv.org/abs/2502.04362

    2025 arXiv

  5. [13]

    Followbench: A multi-level fine-grained constraints following benchmark for large language models, 2024

    Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models, 2024. URL https://arxiv.org/abs/2310.20410

    2024 arXiv

  6. [14]

    Followeval: A multi-dimensional benchmark for assessing the instruction-following capability of large language models, 2023

    Yimin Jing, Renren Jin, Jiahao Hu, Huishi Qiu, Xiaohua Wang, Peng Wang, and Deyi Xiong. Followeval: A multi-dimensional benchmark for assessing the instruction-following capability of large language models, 2023. URL https://arxiv.org/abs/2311.09829

    2023 arXiv

  7. [15]

    Agent-s: Llm agentic workflow to automate standard operating procedures, 2025

    Mandar Kulkarni. Agent-s: Llm agentic workflow to automate standard operating procedures, 2025. URL https://arxiv.org/abs/2503.15520

    2025 arXiv

  8. [16]

    R e IFE : Re-evaluating instruction-following evaluation

    Yixin Liu, Kejian Shi, Alexander Fabbri, Yilun Zhao, PeiFeng Wang, Chien-Sheng Wu, Shafiq Joty, and Arman Cohan. R e IFE : Re-evaluating instruction-following evaluation. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of th...

    2025

  9. [17]

    Large language model instruction following: A survey of progresses and challenges, 2024

    Renze Lou, Kai Zhang, and Wenpeng Yin. Large language model instruction following: A survey of progresses and challenges, 2024. URL https://arxiv.org/abs/2303.10475

    2024 arXiv

  10. [18]

    The death of schema linking? text-to-sql in the age of well-reasoned language models, 2024

    Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi. The death of schema linking? text-to-sql in the age of well-reasoned language models, 2024. URL https://arxiv.org/abs/2408.07702

    2024 arXiv

  11. [19]

    Self-refine: Iterative refinement with self-feedback, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterativ...

    2023 arXiv

  12. [20]

    Cross-task generalization via natural language crowdsourcing instructions, 2022

    Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions, 2022. URL https://arxiv.org/abs/2104.08773

    2022 arXiv

  13. [21]

    Openai o1 system card, 2024

    OpenAI. Openai o1 system card, 2024. URL https://arxiv.org/abs/2412.16720

    2024 arXiv

  14. [22]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  15. [23]

    I n F o B ench: Evaluating instruction following ability in large language models

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. I n F o B ench: Evaluating instruction following ability in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings o...

    2024 doi

  16. [24]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Ni...

    2022 arXiv

  17. [25]

    Dynamics of instruction fine-tuning for chinese large language models, 2025

    Chiyu Song, Zhanchao Zhou, Jianhao Yan, Yuejiao Fei, Zhenzhong Lan, and Yue Zhang. Dynamics of instruction fine-tuning for chinese large language models, 2025. URL https://arxiv.org/abs/2310.19651

    2025 arXiv

  18. [26]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530

    2024 arXiv

  19. [27]

    Smith, Hannaneh Hajishirzi, and Daniel Khashabi

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby ...

    2022 arXiv

  20. [28]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022. URL https://arxiv.org/abs/2109.01652

    2022 arXiv

  21. [29]

    Benchmarking complex instruction-following with multiple constraints composition, 2024

    Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang. Benchmarking complex instruction-following with multiple constraints composition, 2024. URL https://arxiv.or...

    2024 arXiv

  22. [30]

    Bowen Xing, Lizi Liao, Minlie Huang, and Ivor W. Tsang. Dc-instruct: An effective framework for generative multi-intent spoken language understanding. In Conference on Empirical Methods in Natural Language Processing, 2024. URL https://api.semanticscholar.org/CorpusID:273820915

    2024

  23. [31]

    A-mem: Agentic memory for llm agents, 2025

    Wujiang Xu, Kai Mei, Hang Gao, Juntao Tan, Zujie Liang, and Yongfeng Zhang. A-mem: Agentic memory for llm agents, 2025. URL https://arxiv.org/abs/2502.12110

    2025 arXiv

  24. [32]

    Order matters: Investigate the position bias in multi-constraint instruction following, 2025

    Jie Zeng, Qianyu He, Qingyu Ren, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, and Fei Yu. Order matters: Investigate the position bias in multi-constraint instruction following, 2025. URL https://arxiv.org/abs/2502.17204

    2025 arXiv

  25. [33]

    Evaluating large language models at evaluating instruction following, 2024

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following, 2024. URL https://arxiv.org/abs/2310.07641

    2024 arXiv

  26. [34]

    Unibias: Unveiling and mitigating llm bias through internal attention and ffn manipulation, 2024

    Hanzhang Zhou, Zijian Feng, Zixiao Zhu, Junlang Qian, and Kezhi Mao. Unibias: Unveiling and mitigating llm bias through internal attention and ffn manipulation, 2024. URL https://arxiv.org/abs/2405.20612

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.