PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation

Eliya Habba; Gabriel Stanovsky; Gili Lior; Noam Dahan

arxiv: 2507.14913 · v5 · submitted 2025-07-20 · 💻 cs.CL

PromptSuite: A Task-Agnostic Framework for Multi-Prompt Generation

Eliya Habba , Noam Dahan , Gili Lior , Gabriel Stanovsky This is my paper

Pith reviewed 2026-05-19 03:39 UTC · model grok-4.3

classification 💻 cs.CL

keywords prompt variationLLM evaluationmulti-prompt evaluationmodular promptscontrolled perturbationstask-agnostic frameworkevaluation robustness

0 comments

The pith

PromptSuite automatically generates controlled prompt variations to make LLM evaluations more reliable across tasks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Single-prompt evaluations of large language models are fragile because tiny wording changes can swing results dramatically. The paper introduces PromptSuite to solve this by breaking prompts into modular components and applying targeted perturbations to each one. The framework works without task-specific setup on many benchmarks and lets users add new components or perturbation types. Case studies illustrate that the resulting variations are meaningful enough to support stronger, multi-prompt evaluation practices instead of relying on any single wording.

Core claim

PromptSuite is a task-agnostic framework that uses modular prompt design and controlled perturbations to each component to automatically produce diverse yet representative prompt variations, thereby enabling more robust multi-prompt evaluation of LLMs on a wide range of tasks and benchmarks.

What carries the argument

Modular prompt design that decomposes prompts into components and applies controlled perturbations to generate variations

If this is right

Evaluation protocols can shift from single-prompt reporting to reporting performance statistics over automatically generated prompt sets.
Benchmark results become less sensitive to accidental prompt wording choices.
Developers can extend the framework by adding new perturbation operators without redesigning the core system.
The same modular approach can be applied to new tasks by supplying only the base prompt structure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Adopting this style of variation generation could reduce the hidden cost of prompt engineering that currently affects many published LLM results.
Future benchmarks might standardize on a small set of perturbation families so that scores become comparable across papers.
The framework's extensibility suggests it could later incorporate perturbations that target specific failure modes such as reasoning shortcuts or hallucination triggers.

Load-bearing premise

The controlled perturbations to prompt components create variations that are both diverse enough and representative enough to improve evaluation robustness beyond single-prompt baselines.

What would settle it

An experiment that measures whether model performance variance across PromptSuite-generated prompts is no larger or no more predictive of overall capability than variance obtained from purely random prompt rewordings would falsify the central claim.

Figures

Figures reproduced from arXiv: 2507.14913 by Eliya Habba, Gabriel Stanovsky, Gili Lior, Noam Dahan.

**Figure 2.** Figure 2: PromptSuite’s web UI. Left-to-right: uploading a dataset; configuring the template and choosing [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Figure 3: Multi-prompt evaluation results using PromptSuite. The boxplots illustrate variance across different [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Figure 4: Analysis of how perturbations to individual [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Figure 5: Analysis of how perturbations to individual prompt components affect model sensitivity on SQuAD and [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

read the original abstract

Evaluating LLMs with a single prompt has proven unreliable, with small changes leading to significant performance differences. However, generating the prompt variations needed for a more robust multi-prompt evaluation is challenging, limiting its adoption in practice. To address this, we introduce PromptSuite, a framework that enables the automatic generation of various prompts. PromptSuite is flexible - working out of the box on a wide range of tasks and benchmarks. It follows a modular prompt design, allowing controlled perturbations to each component, and is extensible, supporting the addition of new components and perturbation types. Through a series of case studies, we show that PromptSuite provides meaningful variations to support strong evaluation practices. All resources, including the Python API, source code, user-friendly web interface, and demonstration video, are available at: https://eliyahabba.github.io/PromptSuite/.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

PromptSuite gives a clean modular tool for generating prompt variations across tasks, but the case studies stop short of showing quantitative robustness gains over single-prompt baselines.

read the letter

The main thing to know is that this paper ships PromptSuite, a task-agnostic framework that breaks prompts into components and lets you apply controlled perturbations to each one. The modular setup and built-in extensibility are the actual new pieces, and the authors back it with a Python API, web interface, and open code plus a demo video. That packaging makes it immediately usable if you need to generate prompt sets without writing custom scripts for every benchmark. It does well on the engineering side by staying flexible and avoiding task-specific hard-coding, which addresses the real pain of prompt sensitivity in LLM evaluation. The design keeps changes targeted rather than random rewrites, which is a practical step forward from ad-hoc variation methods. The soft spot sits in the validation. The abstract points to case studies that produce meaningful variations, yet there are no reported metrics on whether those variations actually shift model outputs enough to matter. No numbers on accuracy standard deviation, rank correlation changes across models, or how often single-prompt conclusions would flip. The modular perturbations are controlled by construction, but without evidence that they create output distributions different enough to improve robustness, the central claim stays only moderately supported. If the full paper includes error analysis or head-to-head comparisons, that would close the gap; based on the abstract it does not. This work is for practitioners building evaluation pipelines who want an off-the-shelf way to automate prompt diversity. Someone running repeated LLM tests on new benchmarks could plug it in and save time. Readers looking for deeper analysis of why prompts vary or large-scale methodological advances will find less here. I would send it to peer review once the authors add quantitative checks on the robustness improvements. The tool itself is solid enough to warrant referee time if the evidence is strengthened.

Referee Report

2 major / 2 minor

Summary. The paper introduces PromptSuite, a task-agnostic framework for automatically generating multiple prompt variations to support more robust LLM evaluation. It relies on a modular prompt design that decomposes prompts into components and applies controlled perturbations, with claims of flexibility across tasks, extensibility for new components, and demonstration via case studies that the resulting variations are meaningful for strong evaluation practices. Open resources including a Python API, source code, web interface, and video are provided.

Significance. If the variations demonstrably improve evaluation robustness, the framework could meaningfully lower the barrier to multi-prompt practices that address known single-prompt unreliability. The provision of reproducible code, API, and user interface is a clear strength that supports adoption and extension. However, the significance is currently limited by the absence of quantitative evidence linking the perturbations to measurable gains in robustness.

major comments (2)

[Case Studies] Case Studies section: the paper presents examples of prompt variations generated by modular perturbations but reports no quantitative metrics (e.g., change in accuracy standard deviation across prompts, Kendall-tau rank correlation between single- and multi-prompt model rankings, or fraction of evaluation conclusions that flip). Without these, the central claim that the variations 'support strong evaluation practices' and are 'meaningful' enough to justify multi-prompt use over single-prompt baselines remains unsupported.
[Framework Design and Case Studies] §3 (Framework Design) and case-study descriptions: the modular decomposition guarantees syntactic control but does not address whether the resulting output distributions differ from single-prompt baselines at a scale that affects downstream conclusions; a direct comparison experiment would be required to substantiate the robustness benefit.

minor comments (2)

[Abstract] Abstract: the phrase 'meaningful variations' is used without an operational definition or criteria that readers can apply to the case-study outputs.
[Case Studies] The manuscript would benefit from a table summarizing the perturbation types, affected components, and example outputs for each case study to improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. We address each major comment below and agree that additional quantitative evidence would strengthen the manuscript's claims regarding evaluation robustness.

read point-by-point responses

Referee: [Case Studies] Case Studies section: the paper presents examples of prompt variations generated by modular perturbations but reports no quantitative metrics (e.g., change in accuracy standard deviation across prompts, Kendall-tau rank correlation between single- and multi-prompt model rankings, or fraction of evaluation conclusions that flip). Without these, the central claim that the variations 'support strong evaluation practices' and are 'meaningful' enough to justify multi-prompt use over single-prompt baselines remains unsupported.

Authors: We agree that the case studies section, as currently written, relies on qualitative examples and does not include quantitative metrics to directly substantiate improvements in evaluation robustness. This is a valid observation. In the revised manuscript we will expand this section with new quantitative analyses, including standard deviation of accuracy across prompt variations, Kendall-tau rank correlations between single- and multi-prompt model orderings, and the fraction of evaluation conclusions that change when moving from single- to multi-prompt settings. revision: yes
Referee: [Framework Design and Case Studies] §3 (Framework Design) and case-study descriptions: the modular decomposition guarantees syntactic control but does not address whether the resulting output distributions differ from single-prompt baselines at a scale that affects downstream conclusions; a direct comparison experiment would be required to substantiate the robustness benefit.

Authors: We thank the referee for emphasizing the distinction between syntactic control and measurable impact on downstream conclusions. The original manuscript prioritizes the framework's modularity and extensibility; we acknowledge that this leaves the practical robustness benefit under-supported. In the revision we will add a dedicated direct-comparison experiment that applies PromptSuite-generated prompt sets versus single-prompt baselines on selected benchmarks and reports the resulting differences in performance variance and ranking stability. revision: yes

Circularity Check

0 steps flagged

No circularity in PromptSuite framework presentation

full rationale

The paper introduces a practical software framework for modular prompt perturbation and demonstrates its use via case studies on existing benchmarks. No equations, fitted parameters, or derivations are present. Claims rest on external case studies and released code/resources rather than reducing to self-referential definitions, self-citations, or renamings. The modular design is presented as an engineering choice with no uniqueness theorem or ansatz smuggled in. This is a standard non-circular contribution for a tooling paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is a software framework paper rather than a mathematical or theoretical derivation. No free parameters, axioms, or invented entities are introduced in the sense of fitted constants or postulated constructs; the contributions consist of design decisions for modularity and perturbation types.

pith-pipeline@v0.9.0 · 5677 in / 995 out tokens · 32721 ms · 2026-05-19T03:39:55.262912+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

From Words to Widgets for Controllable LLM Generation
cs.HC 2026-04 unverdicted novelty 6.0

Malleable Prompting reifies subjective preferences from natural language into GUI widgets and modulates LLM token probabilities during decoding to enable controllable generation, with a user study showing improved pre...

Reference graph

Works this paper leans on

29 extracted references · 29 canonical work pages · cited by 1 Pith paper · 9 internal anchors

[1]

Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. 2024. https://doi.org/10.18653/v1/2024.acl-long.744 When benchmarks are targets: Revealing the sensitivity of large language model leaderboards . In Proceed...

work page doi:10.18653/v1/2024.acl-long.744 2024
[2]

Elron Bandel, Yotam Perlitz, Elad Venezian, Roni Friedman-Melamed, Ofir Arviv, Matan Orbach, Shachar Don-Yehyia, Dafna Sheinwald, Ariel Gera, Leshem Choshen, and 1 others. 2024. Unitxt: Flexible, shareable and reusable data preparation and evaluation for generative ai. arXiv preprint arXiv:2401.14019

work page arXiv 2024
[3]

Ond r ej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. 2014. https://doi.org/10.3115/v1/W14-3302 Findings of the 2014 workshop on statistical machine translation . In Proceedings of the Ninth Workshop on ...

work page doi:10.3115/v1/w14-3302 2014
[4]

Mohna Chakraborty, Adithya Kulkarni, and Qi Li. 2023. Zero-shot approach to overcome perturbation sensitivity of prompts. arXiv preprint arXiv:2305.15689

work page arXiv 2023
[5]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. https://arxiv.org/abs/2107.03374 Evaluating large lang...

work page internal anchor Pith review Pith/arXiv arXiv 2021
[6]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

work page internal anchor Pith review Pith/arXiv arXiv 2021
[7]

Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, and 1 others. 2021. Nl-augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint arXiv:2112.02721

work page arXiv 2021
[8]

Alex Gu, Wen-Ding Li, Naman Jain, Theo Olausson, Celine Lee, Koushik Sen, and Armando Solar-Lezama. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.7 The counterfeit conundrum: Can code language models grasp the nuances of their incorrect generations? In Findings of the Association for Computational Linguistics: ACL 2024, pages 74--117, Bangkok, Th...

work page doi:10.18653/v1/2024.findings-acl.7 2024
[9]

Alex Gu, Baptiste Rozi \`e re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024 b . Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065

work page internal anchor Pith review Pith/arXiv arXiv 2024
[10]

Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, and Gabriel Stanovsky. 2025. https://arxiv.org/abs/2503.01622 Dove: A large-scale multi-dimensional predictions dataset towards meaningful llm evaluation . Preprint, arXiv:2503.01622

work page internal anchor Pith review Pith/arXiv arXiv 2025
[11]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

work page internal anchor Pith review Pith/arXiv arXiv 2021
[12]

Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. https://arxiv.org/abs/1506.03340 Teaching machines to read and comprehend . Preprint, arXiv:1506.03340

work page internal anchor Pith review Pith/arXiv arXiv 2015
[13]

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848

work page arXiv 2024
[14]

Gili Lior, Eliya Habba, Shahar Levy, Avi Caciularu, and Gabriel Stanovsky. 2025. Reliableeval: A recipe for stochastic llm evaluation via method of moments. arXiv preprint arXiv:2505.22169

work page arXiv 2025
[15]

Iso-Rank

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. https://arxiv.org/abs/2307.03172 Lost in the middle: How language models use long contexts . Preprint, arXiv:2307.03172

work page internal anchor Pith review Pith/arXiv arXiv 2023
[16]

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786

work page arXiv 2021
[17]

Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933--949

work page 2024
[18]

Itai Mondshine, Tzuf Paz-Argaman, and Reut Tsarfaty. 2025. Beyond english: The impact of prompt translation strategies across languages and tasks in multilingual llms. arXiv preprint arXiv:2502.09331

work page arXiv 2025
[19]

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 a . https://arxiv.org/abs/1606.05250 Squad: 100,000+ questions for machine comprehension of text . Preprint, arXiv:1606.05250

work page internal anchor Pith review Pith/arXiv arXiv 2016
[20]

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 b . https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--2392, Austin, Texas. Association for Computational Linguistics

work page doi:10.18653/v1/d16-1264 2016
[21]

Abhilasha Ravichander, Siddharth Dalmia, Maria Ryskina, Florian Metze, Eduard Hovy, and Alan W Black. 2021. https://arxiv.org/abs/2102.08345 Noiseqa: Challenge set evaluation for user-centric question answering . Preprint, arXiv:2102.08345

work page arXiv 2021
[22]

Yuval Reif and Roy Schwartz. 2024. https://arxiv.org/abs/2405.02743 Beyond performance: Quantifying and mitigating label bias in llms . Preprint, arXiv:2405.02743

work page arXiv 2024
[23]

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

work page 2024
[24]

Yarik Menchaca Resendiz and Roman Klinger. 2024. Mopo: Multi-objective prompt optimization for affective text generation. arXiv preprint arXiv:2412.12948

work page arXiv 2024
[25]

Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324

work page internal anchor Pith review Pith/arXiv arXiv 2023
[26]

Manning, Andrew Ng, and Christopher Potts

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170/ Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631--1642, Seattle, Washi...

work page 2013
[27]

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://arxiv.org/abs/2108.00573 Musique: Multihop questions via single-hop question composition . Preprint, arXiv:2108.00573

work page arXiv 2022
[28]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

work page
[29]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

work page

[1] [1]

Norah Alzahrani, Hisham Alyahya, Yazeed Alnumay, Sultan AlRashed, Shaykhah Alsubaie, Yousef Almushayqih, Faisal Mirza, Nouf Alotaibi, Nora Al-Twairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. 2024. https://doi.org/10.18653/v1/2024.acl-long.744 When benchmarks are targets: Revealing the sensitivity of large language model leaderboards . In Proceed...

work page doi:10.18653/v1/2024.acl-long.744 2024

[2] [2]

Elron Bandel, Yotam Perlitz, Elad Venezian, Roni Friedman-Melamed, Ofir Arviv, Matan Orbach, Shachar Don-Yehyia, Dafna Sheinwald, Ariel Gera, Leshem Choshen, and 1 others. 2024. Unitxt: Flexible, shareable and reusable data preparation and evaluation for generative ai. arXiv preprint arXiv:2401.14019

work page arXiv 2024

[3] [3]

Ond r ej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale s Tamchyna. 2014. https://doi.org/10.3115/v1/W14-3302 Findings of the 2014 workshop on statistical machine translation . In Proceedings of the Ninth Workshop on ...

work page doi:10.3115/v1/w14-3302 2014

[4] [4]

Mohna Chakraborty, Adithya Kulkarni, and Qi Li. 2023. Zero-shot approach to overcome perturbation sensitivity of prompts. arXiv preprint arXiv:2305.15689

work page arXiv 2023

[5] [5]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 2021. https://arxiv.org/abs/2107.03374 Evaluating large lang...

work page internal anchor Pith review Pith/arXiv arXiv 2021

[6] [6]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

work page internal anchor Pith review Pith/arXiv arXiv 2021

[7] [7]

Kaustubh D Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Shrivastava, Samson Tan, and 1 others. 2021. Nl-augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint arXiv:2112.02721

work page arXiv 2021

[8] [8]

Alex Gu, Wen-Ding Li, Naman Jain, Theo Olausson, Celine Lee, Koushik Sen, and Armando Solar-Lezama. 2024 a . https://doi.org/10.18653/v1/2024.findings-acl.7 The counterfeit conundrum: Can code language models grasp the nuances of their incorrect generations? In Findings of the Association for Computational Linguistics: ACL 2024, pages 74--117, Bangkok, Th...

work page doi:10.18653/v1/2024.findings-acl.7 2024

[9] [9]

Alex Gu, Baptiste Rozi \`e re, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024 b . Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065

work page internal anchor Pith review Pith/arXiv arXiv 2024

[10] [10]

Eliya Habba, Ofir Arviv, Itay Itzhak, Yotam Perlitz, Elron Bandel, Leshem Choshen, Michal Shmueli-Scheuer, and Gabriel Stanovsky. 2025. https://arxiv.org/abs/2503.01622 Dove: A large-scale multi-dimensional predictions dataset towards meaningful llm evaluation . Preprint, arXiv:2503.01622

work page internal anchor Pith review Pith/arXiv arXiv 2025

[11] [11]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

work page internal anchor Pith review Pith/arXiv arXiv 2021

[12] [12]

Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. https://arxiv.org/abs/1506.03340 Teaching machines to read and comprehend . Preprint, arXiv:1506.03340

work page internal anchor Pith review Pith/arXiv arXiv 2015

[13] [13]

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848

work page arXiv 2024

[14] [14]

Gili Lior, Eliya Habba, Shahar Levy, Avi Caciularu, and Gabriel Stanovsky. 2025. Reliableeval: A recipe for stochastic llm evaluation via method of moments. arXiv preprint arXiv:2505.22169

work page arXiv 2025

[15] [15]

Iso-Rank

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. https://arxiv.org/abs/2307.03172 Lost in the middle: How language models use long contexts . Preprint, arXiv:2307.03172

work page internal anchor Pith review Pith/arXiv arXiv 2023

[16] [16]

Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786

work page arXiv 2021

[17] [17]

Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt llm evaluation. Transactions of the Association for Computational Linguistics, 12:933--949

work page 2024

[18] [18]

Itai Mondshine, Tzuf Paz-Argaman, and Reut Tsarfaty. 2025. Beyond english: The impact of prompt translation strategies across languages and tasks in multilingual llms. arXiv preprint arXiv:2502.09331

work page arXiv 2025

[19] [19]

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 a . https://arxiv.org/abs/1606.05250 Squad: 100,000+ questions for machine comprehension of text . Preprint, arXiv:1606.05250

work page internal anchor Pith review Pith/arXiv arXiv 2016

[20] [20]

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 b . https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--2392, Austin, Texas. Association for Computational Linguistics

work page doi:10.18653/v1/d16-1264 2016

[21] [21]

Abhilasha Ravichander, Siddharth Dalmia, Maria Ryskina, Florian Metze, Eduard Hovy, and Alan W Black. 2021. https://arxiv.org/abs/2102.08345 Noiseqa: Challenge set evaluation for user-centric question answering . Preprint, arXiv:2102.08345

work page arXiv 2021

[22] [22]

Yuval Reif and Roy Schwartz. 2024. https://arxiv.org/abs/2405.02743 Beyond performance: Quantifying and mitigating label bias in llms . Preprint, arXiv:2405.02743

work page arXiv 2024

[23] [23]

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling

work page 2024

[24] [24]

Yarik Menchaca Resendiz and Roman Klinger. 2024. Mopo: Multi-objective prompt optimization for affective text generation. arXiv preprint arXiv:2412.12948

work page arXiv 2024

[25] [25]

Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2023. Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324

work page internal anchor Pith review Pith/arXiv arXiv 2023

[26] [26]

Manning, Andrew Ng, and Christopher Potts

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. https://aclanthology.org/D13-1170/ Recursive deep models for semantic compositionality over a sentiment treebank . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631--1642, Seattle, Washi...

work page 2013

[27] [27]

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://arxiv.org/abs/2108.00573 Musique: Multihop questions via single-hop question composition . Preprint, arXiv:2108.00573

work page arXiv 2022

[28] [28]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

work page

[29] [29]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

work page