Pith. sign in

REVIEW 3 major objections 4 minor 53 references

Find the Intention of Instruction: Comprehensive Evaluation of Instruction Understanding for Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs frequently respond to instruction-formatted distractors instead of the user's true intent, and current state-of-the-art models still lack this instruction understanding capability.

desk verdict IoInst is a new and useful benchmark idea, but the unvalidated EMsparse metric makes the headline accuracy numbers provisional. read the letter →

arxiv 2412.19450 v2 pith:D4RLK2JB submitted 2024-12-27 cs.AI

Hyeonseok Moon , Jaehyung Seo , Seungyoon Lee , Chanjun Park , Heuiseok Lim This is my paper · ORCID
classification cs.AI
keywords instructionunderstandingfollowingevaluationLLMbenchmarkcontrastiveAnti-Attributeintentionofmeta-instructionEMsparse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces the Intention of Instruction (IoInst) benchmark, which asks a language model to choose which of four candidate instructions would have produced a given response, with the other three designed as distractors. The authors report that most instruction-tuned LLMs, including recent state-of-the-art models, often follow a distractor instruction instead of the meta-instruction that asks them to select an option. Across three distractor types, performance falls hardest when the distractors share the label instruction's core requirement but violate fine-grained constraints, the Anti-Attribute setting. The paper concludes that current instruction-following scores on clean prompts overstate how well models actually understand the intent behind instructions. If correct, this implies that tasks involving multiple instruction-formatted statements, such as prompt injection or instruction optimization, need evaluation that specifically tests intention discrimination.

What carries the argument

The load-bearing object is the IoInst benchmark itself. Contexts come from curated instruction-following datasets (Zeng et al., 2024; Zheng et al., 2023; Chia et al., 2024; et al.), with the label instruction paired with three distractors. Random and Semantic distractors are drawn from a candidate pool, with MPNet embedding similarity used to select semantically confusing options; Anti-Attribute distractors are built from heuristic attributes of the context (word counts, keyword usage, sentence boundaries, etc.) and are concatenated to the label instruction's main clause. Scoring is fully automatic via EMsparse: a generation counts as selecting a candidate instruction when its ROUGE-L precision against that candidate's text exceeds tau=0.9, which drives ACC1, ACC2, and ACC_rel1. The meta-instruction is varied across 16 formats (detailed vs simple; context-first vs candidate-first) to test sensitivity to instruction phrasing.

What would settle it

If a human re-annotation of the model outputs shows that a large fraction of responses judged as errors are actually correct paraphrases of the label instruction (e.g., Mistral's 'Create a list of three date ideas...' in Table 13), then the paper's conclusion that SOTA models lack instruction understanding would be an artifact of EMsparse, not a property of the models. Concretely: recompute ACC1 and ACC_rel1 using a judge that accepts semantically equivalent selections, and compare with the paper's scores; a large gap would falsify the claim as measured.

Watch

Extended reading notes

Core claim

The central claim is that LLMs lack robust instruction understanding: given a context, four candidate instructions, and a meta-instruction to pick the one that generated the context, models are prone to respond to the candidate instructions themselves. The paper defines three contrastive settings: Random (unrelated distractors), Semantic (distractors similar in meaning to the label), and Anti-Attribute (distractors that restate the label's main task but add requirements incompatible with the context). It measures ACC1 (correct label selection), ACC2 (following the meta-instruction by picking any candidate), and ACC_rel1 = ACC1/ACC2, using ROUGE-L precision against the option text with a threshold of 0.9 to decide whether a generation selected an option. Results show e.g. GPT-4o reaching ACC1 95.31 on Random but 48.74 on Anti-Attribute, while most open models score far lower; the paper interprets this as evidence that models are distracted by instruction-formatted statements and that high instruction-following performance does not imply deep instruction understanding.

Load-bearing premise

The automatic scorer treats a model answer as selecting an option only when the generated text is nearly a verbatim match of that option (ROUGE-L precision above 0.9), so a model that understands the task but paraphrases the correct instruction is counted as wrong; the reported accuracies rest on this lexical assumption.

Editorial extensions

If this is right

  • If IoInst measures what it claims, high scores on standard instruction-following benchmarks (clean, single-intent prompts) should not be taken as evidence that a model can pick out the intended instruction when other instruction-looking text is present.
  • The Anti-Attribute results imply that models latch onto the repeated core task phrase ('Write a poem...') and miss appended constraints that contradict the context; targeted fine-tuning on such fine-grained mismatches may be needed.
  • Because a substantial fraction of errors are Case 2 (not choosing any candidate), the paper implies that evaluation of instruction following should inspect whether the model obeys the meta-instruction, not just the final content.
  • Detailed meta-instructions yield higher and more stable accuracy, so prompt design choices directly change apparent instruction understanding; evaluations that fix one prompt format may not generalize.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This benchmark could serve as a proxy for vulnerability to prompt injection: if a model cannot distinguish the true instruction from instruction-formatted distractors in a benign setting, it will likely be hard to defend against injected instructions that look similar.
  • The metric could be extended to score paraphrased selections: using semantic equivalence or another LLM judge instead of ROUGE-L threshold might reveal that some models understand the task but answer in their own words, which would change the quantitative conclusion.
  • The anti-attribute data may be reusable as a fine-tuning set to teach models to check constraints against context, a testable intervention the paper does not run.
  • The few-shot degradation suggests that in-context exemplars themselves act as instruction-formatted noise; model developers may need to separate demonstration content from instruction content.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces IoInst, a benchmark that probes whether LLMs can distinguish the single instruction that actually generated a given context from three instruction-formatted distractors. Three contrastive types are constructed (Random, Semantic, Anti-Attribute), and three metrics (ACC1, ACC2, ACCrel1) are defined, all derived from a single lexical scoring rule called EMsparse, a ROUGE-L precision comparison against the candidate instruction strings with a hand-set threshold of 0.9. Experiments with ten instruction-tuned LLMs show generally low ACC1 and ACC2, especially on the Anti-Attribute setting, and the authors conclude that even recent state-of-the-art models lack instruction understanding. Additional analyses cover the effect of meta-instruction wording, few-shot in-context learning, and temperature.

Significance. If the measurements are valid, the paper identifies a genuinely under-tested failure mode: models that follow instructions competently in standard settings may nonetheless answer the wrong meta-task when instruction-formatted text is embedded in the prompt. The benchmark design is thoughtful, with a useful Case 1/Case 2 error taxonomy, three contrastive difficulty levels, publicly released code and data, and broad coverage across ten open and closed models. The analyses of meta-instruction variants and few-shot behavior are informative. However, the quantitative conclusion rests almost entirely on the validity of EMsparse, and the Anti-Attribute construction raises label-uniqueness concerns; both need to be addressed before the central claim can be considered established.

major comments (3)
  1. [§3.4, Eq. (1); Table 13] The paper's headline quantitative claim is carried by EMsparse, defined in Section 3.4 as ROUGE-L precision against a candidate instruction exceeding a threshold tau 'empirically set to 0.9.' The manuscript reports no calibration of this threshold, no sensitivity analysis over tau, and no human-agreement check. Table 13 shows model outputs that select the correct instruction but not verbatim: Mistral's first response correctly chooses 'Come up with three date ideas...' while restating it in expanded prose, and Solar's response also identifies the right option with added reasoning. Such outputs can easily fall below 0.9 ROUGE-L precision because of prefixes, suffixes, and paraphrase, so ACC1 and ACC2 in Table 3 may substantially understate the models' true selection accuracy. Since the abstract's conclusion that state-of-the-art models 'still lack instruction understanding capability' is inferred from these numbers, the metric needs validation: please report a human-annotated subset, a tau sweep (for example 0.7 to 1.0), or a secondary exact-match/extracted-answer metric.
  2. [§4.3, Tables 9 and 12] The Anti-Attribute contrastive instructions appear to include options that are also satisfied by the context, which would make the intended label non-unique. In the elephant-story example of Table 12, the contrastive 'Your answer should be formatted to contain 3 sentences' matches the three-sentence context, and 'Ensure words in complete capitals are utilized fewer than 4 times' is also satisfied by the context; in the database example, the 'Start off your answer with ...' and 'goes beyond 50 words' contrastives are satisfied by the provided context. Section 4.3 states that the contrastive instructions are designed so that 'none of the contrastive instructions align with the context,' but the construction based on context attributes includes 'aligned' instructions (Table 9), and the examples suggest such aligned variants are actually used as distractors. If several candidates can generate the same context, the low ACC1 and ACCrel1 values for the Anti-Attribute setting in Table 3 may reflect ambiguous labels rather than a lack of instruction understanding. Please clarify the selection procedure, verify label uniqueness for every Anti-Attribute item, and re-run the affected experiments.
  3. [§4.2 and §5.1] The benchmark's validity is partly self-referential because GPT-4o is used in dataset construction and also serves as an evaluated model. Section 4.2 reports that GPT-4o reviewed failure cases and regenerated contrastive instructions, and Section 5.1 evaluates GPT-4o on the resulting dataset, where it attains 99.54 ACC1 in the Random setting. The absence of a human performance baseline is a concrete gap: the task is claimed to measure a human-like 'instruction understanding' ability, but the only feasibility check is a GPT-4o result. I am not claiming the benchmark is invalid for this reason alone, but the manuscript should provide an independent validity check, such as human accuracy on a sample, or an evaluation using a model that was not involved in data construction. Without it, the claim that the task is 'clear and solvable enough' rests on the very model family being measured.
minor comments (4)
  1. [§5.2] There is a typo in the sentence 'LLMs are inclined to distracted by several isntruction-formatted statements'; it should read '...to be distracted by several instruction-formatted statements.'
  2. [Throughout] The benchmark name is inconsistently rendered as IOINST, INTINST, and IOI. Please use a single consistent name (and define it once).
  3. [Table 3 and Figure 4] The metric is called ACCrel1 in Equation (1) but the figures and some tables use 'ACC1-rel'; standardize the notation.
  4. [§5.3, Table 4] The 'Error Rate(Case 2)' column in Table 4 is not given a precise definition with a denominator. State whether it is the fraction of all model outputs that are Case 2 errors for a given candidate instruction, and across which model set and settings.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark construction and evaluation are self-contained, though the EMsparse threshold raises measurement-validity questions rather than circularity.

full rationale

The paper's derivation chain is a standard benchmark construction: Section 3.1 defines the task (select the candidate instruction that generated the context), Section 4.1 builds contexts from instruction-following datasets by taking the known instruction-aligned response as context and the corresponding instruction as label, and Section 3.4 defines ACC1, ACC2, and ACCrel1 via Equation (1) using EMsparse, a ROUGE-L precision threshold of 0.9. None of these steps assumes the conclusion that LLMs lack instruction understanding. The threshold tau=0.9 is an empirical choice for lexical matching, not a parameter fitted to the evaluated models and later renamed as a prediction; the reported accuracies are direct measurements of model outputs, not quantities equivalent to the benchmark's construction choices by definition. The dual role of GPT-4o, which appears in dataset curation (Section 4.2) and in the evaluated model panel (Table 3), is a legitimate independence/contamination concern, but it is not derivation-level circularity: GPT-4o's high accuracy is used as a feasibility sanity check, while the central claim rests on the broader panel of open and closed models and the released benchmark. The paper also does not invoke a uniqueness theorem, smuggle in a contested ansatz via self-citation, or rename a known empirical pattern as a new derivation. The EMsparse threshold may misclassify paraphrased but semantically correct selections, as the skeptical reading notes, but that is a measurement-validity criticism aimed at the strength of the quantitative claim, not a circular reduction of the conclusion to its inputs. Therefore no specific circular step can be quoted and exhibited under the required standard, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark depends on a small set of design choices: a lexical matching threshold, length caps, and assumptions about source data quality and distractor validity. No new physical entities or fitted scientific constants are introduced.

free parameters (3)
  • EMsparse ROUGE-L threshold tau = 0.9
    Set empirically in Section 3.4; controls whether a model generation is counted as selecting a candidate instruction. All ACC1/ACC2/ACCrel1 numbers depend on it, and no sensitivity analysis is reported.
  • Context length cap = 1,000 characters
    Chosen after pilot experiments in Section 4.1 to avoid degrading context comprehension; shapes the dataset and may bias toward short responses.
  • Candidate instruction length cap = 300 characters
    Chosen after pilot experiments in Section 4.1 because long candidate instructions reduce ACC2; limits contrastive complexity and generalizability.
assumptions (4)
  • domain assumption Source LLM-as-evaluator datasets contain instruction-response pairs where the labeled response faithfully follows the label instruction.
    Used in Section 4.1 to construct contexts and label instructions; if some labeled responses are misaligned, the benchmark labels are corrupted.
  • domain assumption GPT-4o-based quality assurance, with three error-verification passes, is sufficient to ensure contrastive instructions never generate the context.
    Section 4.2; no human accuracy or inter-annotator agreement is reported, so the correctness of distractor design rests on this assumption.
  • domain assumption ROUGE-L precision above 0.9 between a model output and a candidate instruction indicates the model selected that instruction.
    Section 3.4; this lexical matching assumption may fail for paraphrased selections.
  • domain assumption The meta-instruction is followed if the model outputs one of the four candidate instruction strings; all other outputs count as Case 2 errors.
    Section 3.3; this defines Case 2 but conflates non-conforming formats, such as explanations or refusals, with misunderstanding the intention.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Find the Intention of Instruction: Comprehensive Evaluation of Instruction Understanding for Large Language Models." pith.science (2026). https://pith.science/paper/D4RLK2JB

@misc{pith2026241219450,
  author       = {Pith},
  title        = {Pith review of: Find the Intention of Instruction: Comprehensive Evaluation of Instruction Understanding for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D4RLK2JB}},
  note         = {Machine review of arXiv:2412.19450}
}
read the original abstract

One of the key strengths of Large Language Models (LLMs) is their ability to interact with humans by generating appropriate responses to given instructions. This ability, known as instruction-following capability, has established a foundation for the use of LLMs across various fields and serves as a crucial metric for evaluating their performance. While numerous evaluation benchmarks have been developed, most focus solely on clear and coherent instructions. However, we have noted that LLMs can become easily distracted by instruction-formatted statements, which may lead to an oversight of their instruction comprehension skills. To address this issue, we introduce the Intention of Instruction (IoInst) benchmark. This benchmark evaluates LLMs' capacity to remain focused and understand instructions without being misled by extraneous instructions. The primary objective of this benchmark is to identify the appropriate instruction that accurately guides the generation of a given context. Our findings suggest that even recently introduced state-of-the-art models still lack instruction understanding capability. Along with the proposition of IoInst in this study, we also present broad analyses of the several strategies potentially applicable to IoInst.

Figures

Figures reproduced from arXiv: 2412.19450 by the authors.

Figure 1
Figure 1. Simplified example of IOINST. We compose a benchmark designed to comprehend and select the ap￾propriate instruction that derives given response. Poten￾tial error cases include misunderstanding prerequisites of context and responding to any candidate instruction. to generate responses that align with these instruc￾tions (Chen et al., 2024; Li et al., 2024; Xu et al., 2023; Longpre et al., 2023). This capability, know… view at source ↗
Figure 2
Figure 2. Construction of pool-based contrastive instructions. From the pre-processed data point obtained by the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Construction of Anti-Attribute contrastive instructions. From the pre-processed data point obtained by the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Performance comparison between zero-shot [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance variations with diverse tempera [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Performance variations with different types of meta-instruction. The markers denote the average [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Data statistics regarding character length. Ran [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 16 canonical work pages

  1. [1]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  2. [2]

    Steven Bird and Edward Loper. 2004. https://aclanthology.org/P04-3031 NLTK : The natural language toolkit . In Proceedings of the ACL Interactive Poster and Demonstration Sessions , pages 214--217, Barcelona, Spain. Association for Computational Linguistics

    2004

  3. [3]

    Som S Biswas. 2023 a . Potential use of chat gpt in global warming. Annals of biomedical engineering, 51(6):1126--1127

    work page 2023

  4. [4]

    Som S Biswas. 2023 b . Role of chat gpt in public health. Annals of biomedical engineering, 51(5):868--869

    work page 2023

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  6. [6]

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2024. https://openreview.net/forum?id=FdVXgSJhvz Alpagasus: Training a better alpaca with fewer data . In The Twelfth International Conference on Learning Representations

    2024

  7. [7]

    Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. 2024. https://aclanthology.org/2024.scalellm-1.4 I nstruct E val: Towards holistic evaluation of instruction-tuned large language models . In Proceedings of the First edition of the Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM 2024), pages 35--64, St. Julian ' s, Malta. A...

    work page 2024

  8. [8]

    Cohere. 2024. Command r models. https://cohere.com/command. Accessed: 2024-03-01

    work page 2024

Show all 53 references
  1. [9]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. 2024. http://arxiv.org/abs/2301.00234 A survey on in-context learning

    2024 arXiv

  2. [10]

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/5fc47800ee5b30b8777fdd30abcaaf3b-Paper-Conference.pdf Alpacafarm: A ...

    2023

  3. [11]

    Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt \"a schel. 2024. https://openreview.net/forum?id=9ZxnPZGmPU Promptbreeder: Self-referential self-improvement via prompt evolution . In Forty-first International Conference on Machine Learning

    2024

  4. [12]

    Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic research. Blog post, April, 1:6

    2023

  5. [13]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  6. [14]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  7. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  8. [16]

    Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. 2024. https://doi.org/10.18653/v1/...

    2024 doi

  9. [17]

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf Large language models are zero-shot reasoners . In Advances in Neural Inf...

    2022

  10. [18]

    Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. 2024. https://openreview.net/forum?id=1oijHJBRsT Self-alignment with instruction backtranslation . In The Twelfth International Conference on Learning Representations

    2024

  11. [19]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 Rouge: A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  12. [20]

    Yen-Ting Lin and Yun-Nung Chen. 2023. https://doi.org/10.18653/v1/2023.nlp4convai-1.5 LLM -eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models . In Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvA...

    2023 doi

  13. [21]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  14. [22]

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631--22...

    2023

  15. [23]

    OpenAI-Blog. 2022. Chatgpt: Optimizing language models for dialogue. https://openai.com/blog/chatgpt/. Accessed: 2024-03-01

    2022

  16. [24]

    Katrin Ortmann. 2022. https://aclanthology.org/2022.lrec-1.150 Fine-grained error analysis and fair evaluation of labeled spans . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1400--1407, Marseille, France. European Language Resources Association

    2022

  17. [25]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. http://arxiv.org/abs/2304.03277 Instruction tuning with gpt-4

    2023 arXiv

  18. [26]

    Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.494 Automatic prompt optimization with `` gradient descent '' and beam search . In Proceedings of the 2023 Conference on Empirical Methods in Natural L...

    2023 doi

  19. [27]

    Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.85 Is C hat GPT a general-purpose natural language processing task solver? In Proceedings of the 2023 Conference on Empirical Methods in...

    2023 doi

  20. [28]

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024. http://arxiv.org/abs/2401.03601 Infobench: Evaluating instruction following ability in large language models

    2024 arXiv

  21. [29]

    Federico Quin, Danny Weyns, Matthias Galster, and Camila Costa Silva. 2024. https://doi.org/https://doi.org/10.1016/j.jss.2024.112011 A/b testing: A systematic literature review . Journal of Systems and Software, 211:112011

    2024

  22. [30]

    Laria Reynolds and Kyle McDonell. 2021. https://doi.org/10.1145/3411763.3451760 Prompt programming for large language models: Beyond the few-shot paradigm . In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA '21, New York, NY, USA. A...

    2021

  23. [31]

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/c3a690be93aa602ee2dc0ccab5b7b67e-Paper.pdf Mpnet: Masked and permuted pre-training for language understanding . In Advances in Neural Information Proces...

    2020

  24. [32]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03...

    2023

  25. [33]

    Gemma Team. 2024 a . http://arxiv.org/abs/2403.08295 Gemma: Open models based on gemini research and technology

    2024 arXiv

  26. [34]

    OpenAI Team. 2024 b . http://arxiv.org/abs/2303.08774 Gpt-4 technical report

    2024 arXiv

  27. [35]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  28. [36]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024 a . A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):1--26

    2024

  29. [37]

    Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2024 b . https://openreview.net/forum?id=5Nn2BLV7SB Panda LM : An automatic evaluation benchmark for LLM...

    2024

  30. [38]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.754 Self-instruct: Aligning language models with self-generated instructions . In Proceedings of the 61st Annual Mee...

    2023 doi

  31. [39]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. http://arxiv.org/abs/2302.11382 A prompt pattern catalog to enhance prompt engineering with chatgpt

    2023 arXiv

  32. [40]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  33. [41]

    Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024. http://arxiv.org/abs/2402.18667 Fofo: A benchmark to evaluate llms' format-following capability

    2024 arXiv

  34. [42]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. http://arxiv.org/abs/2304.12244 Wizardlm: Empowering large language models to follow complex instructions

    2023 arXiv

  35. [43]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. https://openreview.net/forum?id=Bb4VGOWELI Large language models as optimizers . In The Twelfth International Conference on Learning Representations

    2024

  36. [44]

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. https://openreview.net/forum?id=tr0KidwPLc Evaluating large language models at evaluating instruction following . In The Twelfth International Conference on Learning Representations

    2024

  37. [45]

    Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023. http://arxiv.org/abs/2308.01862 Wider and deeper llm networks are fairer llm evaluators

    2023 arXiv

  38. [46]

    Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024. https://openreview.net/forum?id=0AZAjkXhit Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning . In Forty-first International Conference on Machine Learning

    2024

  39. [47]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. https://openreview.net/forum?id=shr9PXz7T0 Large language models are not robust multiple choice selectors . In The Twelfth International Conference on Learning Representations

    2024

  40. [48]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Pap...

    2023

  41. [49]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023 a . https://proceedings.neurips.cc/paper_files/paper/2023/file/ac662d74829e4407ce1d...

    2023

  42. [50]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023 b . http://arxiv.org/abs/2311.07911 Instruction-following evaluation for large language models

    2023 arXiv

  43. [51]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023 c . https://openreview.net/forum?id=92gvk82DE- Large language models are human-level prompt engineers . In The Eleventh International Conference on Learning Representations

    2023

  44. [52]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.