Pith. sign in

REVIEW 5 major objections 5 minor 31 references

Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attacks

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A two-stage prompting strategy keeps LLMs accurate when input text is corrupted, cutting the worst accuracy drop from 25 to 10 points.

desk verdict Genuinely new two-stage repair-then-reason prompting with strong gains on five specific perturbation types, but the real-world robustness claim is unsupported because every experiment uses a type-matched prompt on the same perturbation type. read the letter →

arxiv 2506.03627 v2 pith:PS2R3WOE submitted 2025-06-04 cs.CL cs.AI

Lin Mu , Guowei Chu , Li Ni , Lei Sang , Yiwen Zhang This is my paper
classification cs.CLcs.AI
keywords adversarialperturbationspromptrobustnesserrorcorrectionguidancepromptinglargelanguagemodelsreasoningtasksoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Robustness of Prompting (RoP), a prompting strategy designed to keep large language models accurate when input questions are corrupted by typos, scrambled characters, or inserted irrelevant information. It tries to establish that RoP substantially reduces the performance loss caused by such perturbations while leaving clean-input accuracy nearly unchanged, and that it does so without retraining the model. The proposed mechanism is a two-stage split: first repair the noisy input with an auto-generated correction prompt, then reason over the repaired input using an auto-generated guidance prompt. The paper reports that on arithmetic tasks the strategy raises average accuracy under irrelevant-information perturbation from 58.9% to 74.0% and cuts the accuracy drop from 25.4 to 10.3 percentage points.

What carries the argument

The central machinery is the two-stage prompt construction. In the first stage, a fixed taxonomy of five perturbation types (error characters, similar characters, words out of order, homophones, and irrelevant inserted information) is used to create adversarial examples, and those examples, paired with their clean originals, are fed to an automatic prompt engineering procedure that produces an error-correction instruction. In the second stage, the corrected questions are paired with their answers and used to auto-generate a task-guidance instruction. The combined pipeline works by aligning the input distribution before reasoning, so the model is not asked to recover from noise and perform multi-step reasoning inside a single prompt.

What would settle it

Generate test perturbations with a different protocol, such as typos sampled from keyboard-distance errors or irrelevant sentences from a different domain, present them without telling the system which perturbation type applies, and compare RoP to standard prompting; if the accuracy advantage disappears or reverses, the robustness claim is specific to the training distribution rather than general.

Watch

Extended reading notes

Core claim

RoP claims that the brittleness of LLMs to input noise can be mitigated without parameter updates by separating input repair from reasoning. In the Error Correction stage, perturbed question/clean question pairs are used to auto-generate an instruction that rewrites noisy inputs back into semantically clean form. In the Guidance stage, the corrected questions are paired with their answers to auto-generate an instruction that steers the model's inference. The paper argues that this order matters: correcting first stabilizes the input distribution, so the guidance prompt is optimized and applied on data that matches its distribution, which single-stage methods cannot do. Empirically, RoP keeps degradation within about two points of the no-perturbation baseline for most perturbation types on arithmetic tasks.

Load-bearing premise

The claim depends on the five perturbation types and the model-generated instantiations of them being representative of the input noise RoP will actually meet, and on knowing which perturbation type is present so the matching correction prompt can be selected.

Editorial extensions

If this is right

  • On arithmetic benchmarks, RoP raises average accuracy under irrelevant-information perturbation from 58.9% to 74.0% compared with standard prompting, and keeps degradation near or below other prompting baselines for most perturbation types.
  • RoP transfers to commonsense and logical reasoning tasks, with gains such as 71.3% versus 56.9% on a commonsense benchmark under character-level perturbation.
  • The gains persist when the optimizer model is weaker than the strongest available option, so the method does not require a frontier model for prompt optimization.
  • RoP outperforms a dictionary-based spelling corrector on character-level perturbations, because contextual repair avoids introducing new errors.
  • RoP's token cost is roughly 1.7 to 3 times standard prompting, far below an agent-based optimizer baseline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The two-stage separation implies a general design principle: any LLM pipeline that must handle noisy inputs could benefit from using separate prompts for restoration and task execution, before considering larger training-time defenses.
  • Because the correction stage is optimized on the same perturbation generator used at test time, the measured robustness may overestimate performance on completely unseen noise; a natural extension is to test RoP with perturbations from a disjoint source.
  • The finding that benign irrelevant information sometimes improves accuracy on some reasoning tasks hints that a limited amount of contextual noise may act as an implicit thinking trigger, which is worth testing deliberately with controlled insertion lengths.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes RoP (Robustness of Prompting), a two-stage prompting strategy for large language models. Stage one, Error Correction, generates adversarial examples by applying five hand-defined perturbation types (EC, SC, WOO, HW, UIC) with GPT-4o, then uses APE to auto-generate a type-specific correction instruction. Stage two, Guidance, uses APE on corrected examples to generate a type-specific guidance instruction that steers inference. The method is evaluated on arithmetic (Table I), commonsense, and logical reasoning benchmarks, with additional experiments on other LLMs, weaker optimization models, a spelling-correction baseline, and inference cost. The headline result is that RoP improves accuracy over standard prompting under UIC by about 15 points on arithmetic tasks, and generally reduces the drop relative to clean inputs.

Significance. The paper addresses an important and practical problem: the fragility of LLMs to small character-level and semantic perturbations. If the reported gains hold beyond the specific evaluation protocol, RoP would be a valuable training-free defense with a simple, modular design. The paper has several strengths: the code is publicly available; the evaluation spans multiple task families, multiple evaluation models, and multiple perturbation types; it includes an inference-cost analysis; and it compares against a non-LLM spelling-correction baseline. The APE-based prompt generation is standard practice and not, by itself, circular. However, the evidence as presented does not yet justify the abstract's claim that RoP is 'practical and effective for enhancing LLM robustness in real-world applications,' because the evaluation assumes the defender knows the perturbation type at inference time and because the same GPT-4o generator is used for both constructing the correction examples and perturbing the test sets. These are generalization gaps that can be addressed experimentally rather than foundational errors.

major comments (5)
  1. [III.B-IV.A] The evaluation assumes the perturbation type is known at test time. The error-correction and guidance instructions are generated separately for each of the five perturbation types in Sections III.C and III.D, and the experiments apply the type-matched instruction to the corresponding type of perturbed test input. There is no experiment in which the perturbation type is unknown, mixed, or unseen. Since a real-world defender usually does not know the exact type of input noise in advance, the central real-world claim is not supported. Please add experiments with a mixture of perturbation types, with unseen perturbation types, or with a generic correction instruction, and report how performance degrades in those settings.
  2. [III.B and IV.A.1] The same GPT-4o perturbation protocol (the prompt in Section III.B) generates both the adversarial examples used for APE-based instruction selection (Section III.C) and the test-set perturbations (Section IV.A.1). This creates a distributional coupling: the learned correction and guidance prompts may overfit to artifacts of GPT-4o's rewriting style rather than to the underlying perturbation class. To establish transfer, please evaluate with a different perturbation generator (e.g., random character swaps, rule-based edits, or human typos) and, ideally, train on one generator and test on another. Without this, the reported gains could be an artifact of the specific generator.
  3. [Tables I, III-V and Figs. 5-7] No variance or significance estimates are reported for any of the accuracy numbers. Many head-to-head differences are small; for example, in Table I the EC row on AddSub shows RoP at 87.3% versus Stand at 86.8%, and the WOO row shows RoP at 80.2% versus CoT at 79.9%. Without error bars, bootstrap confidence intervals, or repeated runs, it is impossible to tell which of the reported improvements are reliable. Please provide standard deviations or confidence intervals, or at least state the number of repeated evaluations.
  4. [Table II and III.E] The ablation study covers only two perturbation types (EC and HW) and three datasets, but Section III.E claims a general synergistic effect between the Error Correction and Guidance stages. The UIC perturbation, where RoP shows its largest gains in Table I, is not included in the ablation. This is an important gap because the synergy claim is central to the paper's contribution and the UIC row is the flagship result. Please extend the ablation to all five perturbation types.
  5. [Tables I-II] The main comparison tables do not include a generic two-stage baseline that uses a non-optimized or generic correction prompt followed by a standard inference prompt (or CoT). The 'Direct' baseline in Table II is a single-prompt 'correct and answer' method, which is not equivalent to a two-stage pipeline with a generic correction step. A generic two-stage baseline is needed to separate the benefit of the two-stage decomposition from the benefit of APE-optimized, perturbation-specific instructions. Please add such a baseline to Table I (or a supplementary table) and discuss the comparison.
minor comments (5)
  1. [Tables I, III, IV] The header 'No Dert.' (and 'NODERT.' in Table IV) appears to be a typo for 'No Pert.' or 'Clean.' Please correct it for clarity.
  2. [Section IV.A.2] The sentence 'The result, summarized in Table I.' is missing a verb and should be 'The results are summarized in Table I.'
  3. [Section III.C-D] The meta-prompt 'You are a kick ass prompt engineer' is informal for a journal venue. Please rephrase or note that it follows the APE formulation.
  4. [Section IV.A.3] The sentence 'In this case, the results are similar to those of CoT and APE' refers to WOO in Table I, but the wording is ambiguous because the preceding paragraph discusses UIC. Please clarify the referent.
  5. [Section IV.C] The inference cost analysis reports average tokens per query without variance across queries or across repeated runs. Reporting only point averages makes it hard to judge whether the cost differences are robust; please add measures of dispersion or state that the analysis is illustrative.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; RoP's gains are empirical and the only self-citation is minor and non-load-bearing.

full rationale

The paper does not claim a mathematical derivation; RoP is an empirical two-stage prompting pipeline. The Error Correction and Guidance instructions are selected by APE on sampled training pairs, and the same five perturbation types are used both for instruction generation and for test evaluation (Sections III.B-D and IV.A.1), but this is standard prompt tuning on a matched distribution rather than a fitted parameter renamed as a prediction. The paper explicitly defines 'adversarial examples' as its own five perturbation types (Section III.B), so the abstract's robustness claim is scoped to those perturbations; this is a generalization/scope limitation, not a self-definitional reduction. No uniqueness theorem is invoked, no load-bearing argument reduces to a self-citation, and no ansatz is smuggled in via citation. The evaluation is grounded in external benchmarks (GSM8K, AQUA, CSQA, StrategyQA, etc.) and external baselines (Stand, CoT, APE, PromptAgent, SpellChecker), so the reported accuracies are independently measured. The only self-citation is reference [4] (Mu et al., NetPrompt), cited in the introduction as an example of LLM capabilities; it plays no role in the method or the evaluation. The score of 2 reflects that single minor, non-load-bearing self-citation, not any circular step.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the representativeness of the hand-designed perturbation protocol and on APE prompt optimization transferring to the test set; the hyperparameters k and m are not reported.

free parameters (3)
  • k (number of in-context examples for guidance generation) = not reported
    Section III.D samples k question-answer pairs for APE; k is not specified, affecting prompt optimization and reproducibility.
  • m (number of adversarial examples for error-correction instruction generation) = not reported
    Section III.C samples m examples; m is not specified.
  • Perturbation level (1, 4, 7 characters) = 1, 4, 7
    Section IV.A.5 tests 1, 4, and 7-character perturbations; the choice is hand-set.
assumptions (3)
  • domain assumption In-context learning formulation in Eq. (1) accurately describes LLM answer generation
    Used in Section III.A to motivate the method; LLM outputs are not strictly a probability maximum over a fixed candidate set, but the assumption is standard.
  • domain assumption The five perturbation types are representative of real-world input noise
    Section III.B defines EC, SC, WOO, HW, UIC; the paper does not test on unseen perturbation types.
  • domain assumption The APE-generated correction and guidance instructions transfer from training samples to the test distribution
    Sections III.C and III.D optimize prompts on a training subset and apply them to test questions; transfer is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attacks." pith.science (2026). https://pith.science/paper/PS2R3WOE

@misc{pith2026250603627,
  author       = {Pith},
  title        = {Pith review of: Robustness of Prompting: Enhancing Robustness of Large Language Models Against Prompting Attacks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PS2R3WOE}},
  note         = {Machine review of arXiv:2506.03627}
}
read the original abstract

Large Language Models (LLMs) have demonstrated remarkable performance across various tasks by effectively utilizing a prompting strategy. However, they are highly sensitive to input perturbations, such as typographical errors or slight character order errors, which can significantly impair their performance. Despite advances in prompting techniques such as Chain-of-Thought and automatic prompt generation, developing a prompting strategy that explicitly mitigates the negative impact of such perturbations remains an open challenge. To bridge this gap, we propose Robustness of Prompting (RoP), a novel prompting strategy aimed at enhancing the robustness of LLMs. RoP consists of two stages: Error Correction and Guidance. In the Error Correction stage, RoP applies diverse perturbation methods to generate adversarial examples, which are used to generate prompts that correct input errors automatically. In the Guidance stage, RoP generates an optimal guidance prompt based on the corrected input, guiding the model to generate more robust and accurate inferences. Through comprehensive experiments spanning arithmetic, commonsense, and logical reasoning tasks, we demonstrate that RoP significantly improves LLMs' robustness against adversarial perturbations. Crucially, it preserves model accuracy with only minimal degradation compared to clean input scenarios, thereby establishing RoP as a practical and effective approach for enhancing LLM robustness in real-world applications.

Figures

Figures reproduced from arXiv: 2506.03627 by the authors.

Figure 1
Figure 1. Comparison of the performance of GPT-3.5-Turbo before and after [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Subtle character-level changes in prompts can greatly affect the reasoning ability of LLMs. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The framework of Robustness of Prompting (RoP). RoP consists of two stages: [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Five different perturbation methods. to recover the intended semantics of flawed inputs through correctional prompts. Adversarial sample Generation: Inspired by adversarial attack in other fields [11], [18], we synthesize adversarial examples from perturbed inputs. Spe…
Figure 5
Figure 5. Figure 5: The effectiveness of different levels of perturbation. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The performance of RoP with different perturbation methods in the commonsense reasoning task. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The performance of RoP with different perturbation methods in the logical reasoning task. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

31 extracted references · 19 canonical work pages

  1. [1]

    Language mod- els are few-shot learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell,et al., “Language mod- els are few-shot learners,”Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020

    work page 1901

  2. [2]

    Palm: Scal- ing language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann,et al., “Palm: Scal- ing language modeling with pathways,”Journal of Machine Learning Research, vol. 24, no. 240, pp. 1–113, 2023

    work page 2023

  3. [3]

    A survey on large language models for recommendation,

    L. Wu, Z. Zheng, Z. Qiu, H. Wang, H. Gu, T. Shen, C. Qin, C. Zhu, H. Zhu, Q. Liu,et al., “A survey on large language models for recommendation,”World Wide Web, vol. 27, no. 5, p. 60, 2024

    work page 2024

  4. [4]

    Netprompt: Neural network prompting enhances event extraction in large language models,

    L. Mu, Y . Cheng, J. Shen, Y . Zhang, and H. Zhong, “Netprompt: Neural network prompting enhances event extraction in large language models,” IEEE Transactions on Big Data, pp. 1–15, 2025

    work page 2025

  5. [5]

    Mul- timodal chain-of-thought reasoning: A comprehensive survey,

    Y . Wang, S. Wu, Y . Zhang, S. Yan, Z. Liu, J. Luo, and H. Fei, “Mul- timodal chain-of-thought reasoning: A comprehensive survey,”arXiv preprint arXiv:2503.12605, 2025

    arXiv 2025

  6. [6]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V . Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” inAdvances in Neural Information Processing Systems(S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, eds.), vol. 35, pp. 24824–24837, Curran Associates, Inc., 2022

    work page 2022

  7. [7]

    Large language models are human-level prompt engineers,

    Y . Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” in The Eleventh International Conference on Learning Representations, 2023

    work page 2023

  8. [8]

    Reasoning robustness of llms to adversarial typographical errors,

    E. Gan, Y . Zhao, L. Cheng, Y . Mao, A. Goyal, K. Kawaguchi, M.- Y . Kan, and M. Shieh, “Reasoning robustness of llms to adversarial typographical errors,”arXiv preprint arXiv:2411.05345, 2024

    arXiv 2024

Show all 31 references
  1. [9]

    An llm can fool itself: A prompt-based adversarial attack,

    X. Xu, K. Kong, N. Liu, L. Cui, D. Wang, J. Zhang, and M. Kankanhalli, “An llm can fool itself: A prompt-based adversarial attack,”arXiv preprint arXiv:2310.13345, 2023

    2023 arXiv

  2. [10]

    Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts,

    K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y . Wang, L. Yang, W. Ye, Y . Zhang, N. Gong,et al., “Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts,” in Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privac...

    2023

  3. [11]

    Measure and improve robustness in NLP models: A survey,

    X. Wang, H. Wang, and D. Yang, “Measure and improve robustness in NLP models: A survey,” inProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies(M. Carpuat, M.-C. de Marneffe, and I. V ....

    2022

  4. [12]

    Rethinking the role of demonstrations: What makes in-context learning work?,

    S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?,”arXiv preprint arXiv:2202.12837, 2022

    2022 arXiv

  5. [13]

    Prompt learning for few-shot question answering via self-context data augmentation,

    J.-Q. Qiu, C.-Y . Zhang, and C. L. P. Chen, “Prompt learning for few-shot question answering via self-context data augmentation,”IEEE Transactions on Artificial Intelligence, vol. 6, no. 3, pp. 589–603, 2025

    2025

  6. [14]

    Large lan- guage models are zero-shot reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large lan- guage models are zero-shot reasoners,”Advances in neural information processing systems, vol. 35, pp. 22199–22213, 2022

    2022

  7. [15]

    Large language models as optimizers,

    C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen, “Large language models as optimizers,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  8. [16]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano,et al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  9. [17]

    A survey for in-context learning,

    Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,”arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  10. [18]

    Adversarial attacks and defenses in machine learning-empowered communication systems and networks: A contemporary survey,

    Y . Wang, T. Sun, S. Li, X. Yuan, W. Ni, E. Hossain, and H. V . Poor, “Adversarial attacks and defenses in machine learning-empowered communication systems and networks: A contemporary survey,”IEEE Communications Surveys & Tutorials, vol. 25, no. 4, pp. 2245–2298, 2023

    2023

  11. [19]

    Program induction by rationale generation: Learning to solve and explain algebraic word problems,

    W. Ling, D. Yogatama, C. Dyer, and P. Blunsom, “Program induction by rationale generation: Learning to solve and explain algebraic word problems,” inProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)(R. Barzilay and ...

    2017

  12. [20]

    Parsing algebraic word problems into equations,

    R. Koncel-Kedziorski, H. Hajishirzi, A. Sabharwal, O. Etzioni, and S. D. Ang, “Parsing algebraic word problems into equations,”Transactions of the Association for Computational Linguistics, vol. 3, pp. 585–597, 2015

    2015

  13. [21]

    Are NLP models really able to solve simple math word problems?,

    A. Patel, S. Bhattamishra, and N. Goyal, “Are NLP models really able to solve simple math word problems?,” inProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080– 2094, Associat...

    2021

  14. [22]

    Solving general arithmetic word problems,

    S. Roy and D. Roth, “Solving general arithmetic word problems,” in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing(L. M `arquez, C. Callison-Burch, and J. Su, eds.), (Lisbon, Portugal), pp. 1743–1752, Association for Computational Lin- gu...

    2015

  15. [23]

    Learning to solve arithmetic word problems with verb categorization,

    M. J. Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman, “Learning to solve arithmetic word problems with verb categorization,” inProceed- ings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)(A. Moschitti, B. Pang, and W. Daelemans, eds.), ...

    2014

  16. [24]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford,et al., “Gpt-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  17. [25]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray,et al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27730–27744, 2022

    2022

  18. [26]

    Openai o1 system card,

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney,et al., “Openai o1 system card,”arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  19. [27]

    Openai o3 and o4-mini system card,

    OpenAI, “Openai o3 and o4-mini system card,” 2025

    2025

  20. [28]

    Promptagent: Strategic planning with language models enables expert-level prompt optimization,

    X. Wang, C. Li, Z. Wang, F. Bai, H. Luo, J. Zhang, N. Jojic, E. Xing, and Z. Hu, “Promptagent: Strategic planning with language models enables expert-level prompt optimization,” inThe Twelfth International Conference on Learning Representations, 2024

    2024

  21. [29]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,

    A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso,et al., “Beyond the imitation game: Quantifying and extrapolating the capabilities of language models,”arXiv preprint arXiv:2206.04615, 2022

    2022 arXiv

  22. [30]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge,

    A. Talmor, J. Herzig, N. Lourie, and J. Berant, “CommonsenseQA: A question answering challenge targeting commonsense knowledge,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies,...

    2019

  23. [31]

    Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies,

    M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant, “Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies,”Transactions of the Association for Computational Linguistics, vol. 9, pp. 346–361, 04 2021

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.