Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PROMPTEVALS shows that small fine-tuned models outperform a much larger general model at generating assertion criteria for production LLM pipelines.

desk verdict The dataset is a real contribution; the Semantic F1 metric has a definitional bug and is unvalidated, so the 20.93% improvement claim is not yet credible. read the letter →

arxiv 2504.14738 v1 pith:MKYXFAKT submitted 2025-04-20 cs.CL

Reya Vir , Shreya Shankar , Harrison Chase , Will Fu-Hinthorn , Aditya Parameswaran This is my paper · ORCID
classification cs.CL
keywords LLMguardrailsassertioncriteriaprompttemplatessemanticF1fine-tuningevaluationproductionpipelines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PROMPTEVALS is a dataset and benchmark for the task of turning a prompt template into a set of assertion criteria: programmatic checks on an LLM's output that catch failures in production. The paper compiles 2,087 real-world prompt templates and 12,623 ground-truth criteria from developers' pipelines, then evaluates how well models can regenerate relevant criteria. On the benchmark's Semantic F1 metric, fine-tuned versions of two open models, Mistral-7b and Llama-3-8b, score about 0.82, beating single-step GPT-4o's 0.68 by roughly 21 percent on average while also generating faster. The authors argue this makes high-quality guardrail generation cheap enough to run on every prompt edit, and they release dataset and models to support further work.

What carries the argument

The central object is the assertion criterion: a short, checkable constraint on an LLM output, such as "response must contain no more than five highlights" or "tone must be professional and business-neutral." The argument runs on two pieces of machinery: the three-step GPT-4o labeling pipeline (generate candidate criteria, add missing ones, delete redundant or unverifiable ones) anchored to a taxonomy of output constraints, and the Semantic F1 metric, which embeds criteria and scores precision and recall by cosine similarity so paraphrased criteria count as matches. Fine-tuning uses low-rank adaptation to adapt the base models on the training split.

What would settle it

Take the 419-prompt test split, have independent developers write assertion criteria from scratch, and recompute Semantic F1 against those human labels; if single-step GPT-4o then matches or beats the fine-tuned models, the reported advantage collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that the bottleneck in LLM reliability is not implementing assertions but deciding what to check, and that this decision can be learned from real-world prompts. It establishes that a held-out benchmark built from PROMPTEVALS separates models on this ability, and that small fine-tuned models surpass a much larger general-purpose model: fine-tuned Mistral-7b reaches Semantic F1 0.8199 and fine-tuned Llama-3-8b reaches 0.8240, versus 0.6808 for single-step GPT-4o. The fine-tuned models also produce roughly the same number of criteria (about 6.29 and 5.47 per prompt) as the ground truth (about 5.99), whereas base models over-generate wildly. The paper presents this as evidence that focused datasets plus fine-tuning can replace expensive multi-step LLM prompting for this production task.

Load-bearing premise

The benchmark's ground-truth criteria were written by GPT-4o, with humans checking only about one in ten prompts, so the measured scores may reflect agreement with that model's judgment rather than what developers actually want.

Editorial extensions

If this is right

  • Developers could generate assertion criteria for a new prompt in seconds with a small open model, instead of paying for multi-step calls to a much larger model.
  • Guardrails could be regenerated on every prompt edit, making debugging faster and letting teams ship pipeline changes with fresher output checks.
  • Researchers gain a larger, real-world benchmark for a distinct capability: turning a task description into checkable output constraints, separate from general instruction following.
  • Because the fine-tuned models run at lower latency, production pipelines can monitor output quality continuously rather than on a delayed or sampled basis.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the ground-truth criteria are GPT-4o's own generations, the reported scores may partly measure imitation of that labeler rather than transferable skill; a human-written label set on even a few hundred prompts would separate the two.
  • Semantic F1 could be adapted as a training reward, rewarding candidate criteria that maximize coverage of a reference set, rather than only as an evaluation metric.
  • The public release of the dataset raises the risk that future models trained on it will score higher on the benchmark from contamination, so a private holdout or versioned test set would help keep the benchmark meaningful.
  • Extending the same assertion-generation task to image and audio outputs is a natural next step that the paper names as a limitation of the current text-only benchmark.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces PROMPTEVALS, a dataset of 2,087 real-world LLM pipeline prompt templates and 12,623 assertion criteria, sourced from the LangChain Prompt Hub and labeled via a three-step GPT-4o pipeline grounded in the constraint taxonomy of Liu et al. (2024). The authors also construct a benchmark from a 20% hold-out split, define a Semantic F1 metric and a count-based metric to evaluate assertion-generation models, and report that fine-tuned Mistral-7b and Llama-3-8b models achieve mean Semantic F1 scores of 0.8199 and 0.8240, outperforming single-step GPT-4o (0.6808) by roughly 21%. The paper claims that the dataset is 5x larger than prior prompt collections and that the fine-tuned models offer reduced latency, and it releases the dataset, fine-tuned models, and benchmark code.

Significance. The dataset is a potentially valuable resource for research on LLM reliability, guardrail generation, and prompt engineering: it is substantially larger than prior collections, reflects real production prompts rather than researcher-constructed examples, and is accompanied by a taxonomy-grounded labeling procedure, a human-verification sample of 200 prompts with a reported Cohen's kappa of 0.91, and publicly released fine-tuned models. The reported latency improvements are practically meaningful. However, the central quantitative claim rests on the Semantic F1 metric, and the benchmark's ground truth is largely LLM-generated with limited human review; these issues must be addressed before the numerical results can be taken at face value.

major comments (4)
  1. [Section 4.1, Eqs. (1)-(2)] The metric definitions are internally inconsistent as written. Equation (1) defines sem_recall as (1/N) * sum_{i=1..N} max_j cos(z_i, z_hat_j), where z_i indexes the ground truth criteria, but the text states that N is the number of predicted criteria. Similarly, Equation (2) defines sem_precision as (1/M) * sum_{j=1..M} max_i cos(z_i, z_hat_j), where z_hat_j indexes the predicted criteria, but the text states that M is the number of ground truth criteria. For these sums to be well-defined as recall and precision, the recall denominator must be the number of ground truth criteria and the precision denominator must be the number of predicted criteria. Since the paper does not include the evaluation code, it is impossible to determine whether Table 5 was computed with the correct denominators or with the swapped ones described in the text. If the denominators were swapped for typical cases where the predicted and ground truth counts differ (as shown in Table 6), every reported F1 value would be incorrect. The authors must correct the definitions and release the evaluation code so the computation can be verified.
  2. [Section 3.3 and Section 7 (Limitations)] The benchmark's ground truth is generated by GPT-4o through a three-step pipeline, and the fine-tuned models are trained on those same GPT-4o-generated labels and evaluated against them. Human verification was performed on only 200 of 2,087 prompts (about 10%), and the reported verification statistics (<0.02 criteria added and <0.2 removed per list) actually indicate that the human reviewers largely confirmed the LLM-generated criteria, not that independent developer intent was captured. As the authors acknowledge, 'criteria are ultimately generated by an LLM.' This means the headline numbers primarily measure agreement with an LLM-generated standard. To support the claim that the benchmark measures assertion quality developers actually want, the authors should provide substantially more human validation, or at minimum an analysis showing that the LLM-generated labels are robust to the choice of labeler (e.g., by comparing different LLM labelers on a sample) and that model rankings are stable under such variation.
  3. [Section 4.1 and Section 5.2] Semantic F1 is not validated against any human judgment of assertion quality, and no trivial baseline is reported. The metric is an unthresholded average of maximum cosine similarities; a vague criterion such as 'output should be clear' may receive a nontrivial similarity against a specific ground truth criterion such as 'avoid jargon and explain technical terms', so the reported scores of 0.82 do not necessarily reflect precision/recall in the standard sense. Without a baseline that always emits generic criteria, or a correlation study with human ratings of predicted criteria, the reader cannot tell whether the 20.93% gap between the fine-tuned models and GPT-4o is a meaningful qualitative improvement or an artifact of the embedding space. The authors should report such baselines and validation experiments, or explicitly reframe the metric as a similarity heuristic rather than a precision/recall measure.
  4. [Section 5.4] The paper claims that the fine-tuned models 'achieve assertions comparable to the three-phase GPT-4o process,' but no experimental comparison against the three-phase GPT-4o labeling pipeline on the test set is reported. The only GPT-4o number in Table 5 is the single-step baseline. Because the fine-tuned models are trained on the output of the three-phase process, comparing them to the single-step version conflates the effect of fine-tuning with the effect of the multi-step refinement. A direct comparison of the fine-tuned models against the three-phase GPT-4o output on the test set (with the same metric) is needed to support this claim; without it, the statement is unsupported.
minor comments (4)
  1. [Section 5.1.1] The example fine-tuning input/output shows a JSON-like structure with repeated 'constraint' keys, which is not valid JSON and is inconsistent with the instruction in the input that the output should be a comma-separated list surrounded by brackets. Please clarify the actual serialization format used in training and generation.
  2. [Abstract and Section 5.2] The paper reports the average improvement over GPT-4o as 20.93%, but Section 5.2 gives 20.43% for Mistral and 21.03% for Llama; the average of those two numbers is 20.73%, not 20.93%. Also, Section 5.2 states 'average score of 68.08%' where the table reports 0.6808; this should be corrected for consistency.
  3. [Section 3.3] There is an inconsistency in the description of the human validation: the first paragraph mentions 'a manual review of 200 criteria in total,' while the validation paragraph says 'manually verified a sample of 200 prompt templates' generated criteria.' Please clarify whether 200 criteria or 200 prompt templates (with roughly 6 criteria each) were reviewed, as this materially affects the reported verification coverage.
  4. [Table 5] The table caption states that bold indicates the highest scores, but no entries in the table are bold. Either apply bold formatting to the highest values or remove the statement from the caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported benchmark results are empirical, use a held-out test split, and the LLM-generated ground-truth labels are an acknowledged validity caveat rather than a construction-level circularity.

full rationale

The paper makes empirical benchmark claims rather than deriving a result from an assumption that secretly contains the conclusion. The central comparison—fine-tuned Mistral-7b and Llama-8b versus single-step GPT-4o—is evaluated on a held-out 20% test split of PROMPTEVALS, and the fine-tuned models were trained only on the separate 60% train split (Sections 4 and 5.1.1). There is therefore no fitted parameter or training signal drawn from the test set that would force the reported Semantic F1 values. The ground-truth criteria are indeed produced by a three-step GPT-4o workflow, with the authors stating that 'criteria are ultimately generated by an LLM' in the Limitations section; this is a real validity limitation, because the benchmark operationalizes 'relevant assertion' as agreement with a GPT-4o-derived label set. However, it is not circular by construction: a fine-tuned model could plausibly have failed to match the refined three-step GPT-4o labels, and the single-step GPT-4o baseline is not identical to the three-step labeling process used to create the ground truth. The paper also reports some independent human grounding, manually verifying 200 prompt templates and finding fewer than 0.02 criteria added and fewer than 0.2 removed per list on average (Section 3.3). The Semantic F1 equations in Section 4.1 contain a notational inconsistency in the definitions of N and M that is a correctness risk for the numerical claims, but that is a metric bug, not a circularity. Self-citations to the authors' earlier SPADE and validator-alignment work are contextual and are not load-bearing for the benchmark result. Overall, no step in the paper's evaluation reduces to its own inputs by definition, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on no fitted parameters. It does rest on several domain assumptions: that the prompt hub is representative, that GPT-4o-generated criteria are valid ground truth, that the Liu et al. taxonomy is complete, and that embedding cosine similarity measures criteria quality. The last two are load-bearing for the benchmark's validity.

assumptions (5)
  • domain assumption The LangChain Prompt Hub prompt collection represents real-world LLM pipeline tasks with developer-specified requirements.
    The dataset is sampled from this hub; if the prompts are skewed or not representative of production pipelines, the benchmark's external validity is limited. Section 3.2.
  • domain assumption GPT-4o-generated assertion criteria, with human verification on a 200-prompt sample, constitute valid ground truth for assertion generation.
    The entire benchmark is scored against these criteria; only about 10% of prompts were human-checked, so the rest rest on the LLM's judgment. Section 3.3 and Limitations.
  • domain assumption The taxonomy of output constraints from Liu et al. [27] covers the assertion criteria developers need.
    All label generation is constrained to this taxonomy; criteria outside it may be omitted. Section 3.1.
  • domain assumption Cosine similarity between OpenAI text-embedding-3-large embeddings is a valid proxy for semantic equivalence and quality of assertion criteria.
    This is the foundation of the Semantic F1 metric; its validity is not independently evaluated against human judgments. Section 4.1.
  • domain assumption The OpenAI text-embedding-3-large model remains stable enough across time for benchmark reproducibility.
    The paper itself lists this as a limitation: model updates may change scores. Limitations section.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines." pith.science (2026). https://pith.science/paper/MKYXFAKT

@misc{pith2026250414738,
  author       = {Pith},
  title        = {Pith review of: PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MKYXFAKT}},
  note         = {Machine review of arXiv:2504.14738}
}
read the original abstract

Large language models (LLMs) are increasingly deployed in specialized production data processing pipelines across diverse domains -- such as finance, marketing, and e-commerce. However, when running them in production across many inputs, they often fail to follow instructions or meet developer expectations. To improve reliability in these applications, creating assertions or guardrails for LLM outputs to run alongside the pipelines is essential. Yet, determining the right set of assertions that capture developer requirements for a task is challenging. In this paper, we introduce PROMPTEVALS, a dataset of 2087 LLM pipeline prompts with 12623 corresponding assertion criteria, sourced from developers using our open-source LLM pipeline tools. This dataset is 5x larger than previous collections. Using a hold-out test split of PROMPTEVALS as a benchmark, we evaluated closed- and open-source models in generating relevant assertions. Notably, our fine-tuned Mistral and Llama 3 models outperform GPT-4o by 20.93% on average, offering both reduced latency and improved performance. We believe our dataset can spur further research in LLM reliability, alignment, and prompt engineering.

Figures

Figures reproduced from arXiv: 2504.14738 by the authors.

Figure 1
Figure 1. Examples of criteria pairs and their semantic similarity scores. High-scoring pairs typically represent [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of Domains and Subdomains of Tasks Represented in P [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Constraint Type Co-Occurrence Matrix user’s question about a few horses’ data, normally in numeric form. You have to first distinguish each horse’s data, then answer user’s question with the input and some professional’s comments, your final output should be a decision of which horse is performing good or bad. CONCEPTS: You have to note these custom attributes before answering the question: "Reborn" means that the h… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Distribution of Ground Truth Criteria by Type [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A new Persian-Islamic trustworthiness benchmark ranks Claude highest and Qwen lowest across eight LLMs and finds safety is the weakest dimension.

Reference graph

Works this paper leans on

82 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    2020

  2. [2]

    A survey on evaluation of large language models

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transac- tions on Intelligent Systems and Technology, 15(3): 1–45, 2024

    2024

  3. [3]

    Benchmarking large language models in retrieval- augmented generation

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval- augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 17754–17762, 2024

    work page 2024

  4. [4]

    Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, ...

    arXiv

  5. [5]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [6]

    Michael Desmond, Zahra Ashktorab, Qian Pan, Casey Dugan, and James M. Johnson. Evalullm: Llm assisted evaluation of generative outputs. In Compan- ion Proceedings of the 29th International Conference on Intelligent User Interfaces, IUI ’24 Companion, page 30–32, New York, NY , USA, 2024. Association for Computing Machinery. ISBN 9798400705090. doi: 10.114...

    arXiv 2024

  7. [7]

    Building guardrails for large language models, 2024

    Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, and Xi- aowei Huang. Building guardrails for large language models, 2024

    work page 2024

  8. [8]

    Position: Building guardrails for large lan- guage models requires systematic design

    Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, and Xiaowei Huang. Position: Building guardrails for large lan- guage models requires systematic design. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st Intern...

    work page 2024

Show all 82 references
  1. [9]

    The need for guardrails with large language models in medical safety-critical settings: An artificial intelli- gence application in the pharmacovigilance ecosys- tem

    Joe B Hakim, Jeffery L Painter, Darmendra Ram- charran, Vijay Kara, Greg Powell, Paulina Sobczak, Chiho Sato, Andrew Bate, and Andrew Beam. The need for guardrails with large language models in medical safety-critical settings: An artificial intelli- gence application in the p...

    2024 arXiv

  2. [10]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03300,

    2009 arXiv

  3. [11]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  4. [12]

    A survey on hallucination in large language models: Principles, taxonomy, chal- lenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, chal- lenges, and open questions. ArXiv, abs/2311.05232,

    arXiv

  5. [13]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023

    2023

  6. [14]

    Abbeel, Deepak Pathak, and Igor Mordatch

    Wenlong Huang, P. Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. ArXiv, abs/2201.07207,

    arXiv

  7. [15]

    Beavertails: To- wards improved safety alignment of llm via a human- preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: To- wards improved safety alignment of llm via a human- preference dataset. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gi- anna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Tho...

    2023

  9. [17]

    Cal- ibrated language models must hallucinate

    Adam Tauman Kalai and Santosh S Vempala. Cal- ibrated language models must hallucinate. arXiv preprint arXiv:2311.14648, 2023

    2023 arXiv

  10. [18]

    URL https://api.semanticscholar.org/ CorpusID:246035276

  11. [19]

    Prometheus: Inducing fine-grained evaluation capa- bility in language models

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained evaluation capa- bility in language models. In The Twelfth Inter- national Conference on Learning Repres...

    2023

  12. [20]

    Prometheus 2: An open source language model specialized in evaluating other language mod- els

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Gra- ham Neubig, Moontae Lee, Kyungjae Lee, and Min- joon Seo. Prometheus 2: An open source language model specialized in evaluating other language mod- els. arXiv preprint arXiv:2405.01535, 2024

    2024 arXiv

  13. [21]

    Evallm: Interactive evaluation of large language model prompts on user-defined cri- teria

    Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. Evallm: Interactive evaluation of large language model prompts on user-defined cri- teria. In International Conference on Human Factors in Computing Systems , 2023. URL https://api. semanticscholar.org/CorpusID:...

    2023

  14. [22]

    Dspy: Compiling declarative language model calls into self-improving pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Mahesh- wari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03...

    arXiv

  15. [23]

    Openassistant conversations- democratizing large language model alignment

    Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations- democratizing large language model alignment. Ad- vances in Neural Information P...

    2023

  16. [24]

    The power of scale for parameter-efficient prompt tuning, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021

    2021

  17. [25]

    Generative judge for evaluating alignment

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. ArXiv, abs/2310.05470,

    arXiv

  18. [26]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916, 2022. URL https://api. semanticscholar.org/CorpusID:249017743

    2022 arXiv

  19. [27]

    we need structured output

    Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. "we need structured output": To- wards user-centered constraints on large language model output. In Extended Abstracts of the 2024 CHI Conference on Human Facto...

    2024

  20. [28]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xu- anyu Lei, Hanyu Lai, Yu Gu, Yuxian Gu, Hangliang Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Shengqi Shen, Tianjun Zhang, Yu Su, Huan Sun, Min- lie Huang, Yuxiao Dong, and Jie Tan...

    arXiv

  21. [29]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019

    2019

  22. [30]

    URL https://api.semanticscholar.org/ CorpusID:263829791

  23. [31]

    Prefix-tuning: Op- timizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Op- timizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  24. [32]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul Francis Christiano, J...

    arXiv

  25. [33]

    Automatically correcting large language models: Surveying the landscape of diverse self- correction strategies

    Liangming Pan, Michael Stephen Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self- correction strategies. ArXiv, abs/2308.03188,

    arXiv

  26. [34]

    URL https://api.semanticscholar.org/ CorpusID:260682249

  27. [35]

    Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails, 2023

    Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails, 2023

    2023

  28. [36]

    Building a domain-specific guardrail model in pro- duction

    Mohammad Niknazar, Paul V Haley, Latha Ra- manan, Sang T Truong, Yedendra Shrinivasan, Ayan Kumar Bhowmick, Prasenjit Dey, Ashish Jag- mohan, Hema Maheshwari, Shom Ponoth, et al. Building a domain-specific guardrail model in pro- duction. arXiv preprint arXiv:2408.01452, 2024

    2024 arXiv

  29. [37]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Al- tenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian,...

    2024 arXiv

  30. [38]

    Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences

    Shreya Shankar, JD Zamfirescu-Pereira, Björn Hart- mann, Aditya G Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. arXiv preprint arXiv:2404.12272, 2024

    2024 arXiv

  31. [39]

    URL https://api.semanticscholar.org/ CorpusID:246426909

  32. [40]

    Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali...

    arXiv

  33. [41]

    URL https://api.semanticscholar.org/ CorpusID:260682695

  34. [42]

    Infobench: Eval- uating instruction following ability in large language models

    Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. Infobench: Eval- uating instruction following ability in large language models. arXiv preprint arXiv:2401.03601, 2024

    2024 arXiv

  35. [43]

    Pandalm: An automatic evaluation benchmark for llm instruc- tion tuning optimization

    Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xingxu Xie, Wei Ye, Shi-Bo Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for llm instruc- tion tuning optimization. ArXiv, abs/2306.05087,

    arXiv

  36. [44]

    A systematic survey of prompt engineering in large lan- guage models: Techniques and applications

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large lan- guage models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024

    2024 arXiv

  37. [45]

    Shreya Shankar, Haotian Li, Parth Asawa, Made- lon Hulsebos, Yiming Lin, J. D. Zamfirescu- Pereira, Harrison Chase, Will Fu-Hinthorn, Aditya G. Parameswaran, and Eugene Wu. Spade: Synthesiz- ing data quality assertions for large language model pipelines, 2024

    2024

  38. [46]

    Chain-of-thought prompting elicits rea- soning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits rea- soning in large language models. Advances in neural information processing systems , 35:24824–24837, 2022

    2022

  39. [47]

    Towards better evaluation of instruction-following: A case-study in summariza- tion

    Ondrej Skopek, Rahul Aralikatte, Sian Gooding, and Victor Carbune. Towards better evaluation of instruction-following: A case-study in summariza- tion. In Jing Jiang, David Reitter, and Shumin Deng, editors, Proceedings of the 27th Confer- ence on Computational Natural Languag...

    2023 doi

  40. [48]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert, 2020

    2020

  41. [49]

    URL https://api.semanticscholar.org/ CorpusID:263625818

  42. [50]

    Llama: Open and efficient foundation language mod- els, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023

  43. [51]

    Replacing judges with juries: Evaluating llm genera- tions with a panel of diverse models, 2024

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorod- sky, Minjie Xu, Naomi White, and Patrick Lewis. Replacing judges with juries: Evaluating llm genera- tions with a panel of diverse models, 2024

    2024

  44. [52]

    Instruction-following evalu- ation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Sid- dhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evalu- ation for large language models. arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  45. [53]

    URL https://api.semanticscholar.org/ CorpusID:259108266

  46. [54]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning lan- guage models with self-generated instructions. In Annual Meeting of the Association for Compu- tational Linguistics , 2022. URL https:/...

    2022

  47. [55]

    Dai, and Quoc V

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V . Le. Fine- tuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2021. URL https://api. semanticscholar.org/CorpusID:237416585

    2021 arXiv

  48. [57]

    Evaluating large lan- guage models at evaluating instruction following

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large lan- guage models at evaluating instruction following. In International Conference on Learning Representa- tions (ICLR), 2024

    2024

  49. [59]

    Xing, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Haotong Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. ArXiv, abs/2309.11998,

    arXiv

  50. [60]

    URL https://api.semanticscholar.org/ CorpusID:262084217

  51. [61]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judg- ing llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  52. [62]

    Xing, Hao- tong Zhang, Joseph Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao- tong Zhang, Joseph Gonzalez, and Ion Stoica. Judg- ing llm-as-a-judge with mt-bench and chatbot arena. ArXiv, abs/2306.05685, 2023. URL htt...

    2023 arXiv

  53. [64]

    Judgelm: Fine-tuned large language mod- els are scalable judges

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language mod- els are scalable judges. ArXiv, abs/2310.17631,

    arXiv

  54. [65]

    URL https://api.semanticscholar.org/ CorpusID:264490588

  55. [66]

    ‘json “‘ markers, with the key “field

    Yuchen Zhuang, Yue Yu, Kuan Wang, Hao- tian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. ArXiv, abs/2306.13304, 2023. URL https://api. semanticscholar.org/CorpusID:259243960. In this appendix, we include additional information on our w...

    2023 arXiv

  56. [67]

    Why was the dataset created? (e.g., was there a specific intended task gap that needed to be filled?) The dataset was created to be used in training or fine-tuning models in generating higher quality assertion criteria

  57. [68]

    Who funded the creation of the dataset? Lab sponsors

  58. [69]

    We removed any rows that resulted in 0 assertion criteria after the first step of our 3 step workflow

    What preprocessing/cleaning was done? (e.g., discretization or bucketing, tokenization, part-of- speech tagging, SIFT feature extraction, removal of instances) The prompt template was extracted from the metadata and was added to the dataset. We removed any rows that resulted i...

  59. [70]

    They can delete their prompts by submitting a delete request

    If it relates to people, were they told what the dataset would be used for and did they consent? If so, how? Were they provided with any mechanism to revoke their consent in the future or for certain structured_outputadhere_instructions stay_on_topicinclude_terms stylistic_con...

    2000

  60. [71]

    D Model Cards D.1 Fine-tuned Mistral

    Will the dataset be updated? How often, by whom? We plan to update the dataset yearly. D Model Cards D.1 Fine-tuned Mistral

  61. [78]

    Quantitative Analyses (Unitary results, Intersectional results): See Table 7 Domain Similarity Precision Recall General-Purpose Chatbots 0.8171 0.8023 0.8338 Question-Answering 0.8216 0.8183 0.8255 Text Summarization 0.8785 0.8863 0.8725 Database Querying 0.8312 0.8400 0.8234 ...

  62. [80]

    Caveats and Recommendations: None D.2 Fine-tuned Llama

  63. [81]

    Basic information about the model

    Model Details. Basic information about the model. – Person or organization developing model: Meta, and fine-tuned by the authors of this paper – Model date: Base model was released in April 18 2024, and fine-tuned in July 2024 – Model version: 3.1 – Model type: decoder-only Tr...

    2024 arXiv

  64. [82]

    Use cases that were envisioned during development

    Intended Use. Use cases that were envisioned during development. (Primary intended uses, Primary intended users, Out-of-scope use cases) Intended to be used by developers to generate high quality assertion criteria for LLM outputs, or to benchmark the ability of LLMs in genera...

  65. [83]

    Factors could include demographic or phenotypic groups, environmental conditions, techni- cal attributes, or others listed in Section 4.3

    Factors. Factors could include demographic or phenotypic groups, environmental conditions, techni- cal attributes, or others listed in Section 4.3. We don’t collect any demographic, phenotypic, or others listed in Section 4.3, data in our dataset

  66. [84]

    Metrics should be chosen to reflect potential realworld impacts of the model

    Metrics. Metrics should be chosen to reflect potential realworld impacts of the model. (Model performance measures, Decision thresholds, Variation approaches) Metrics are defined in Section 4.1

  67. [85]

    Evaluation Data: Evaluated on P ROMPT EVALS test set

  68. [86]

    Training Data: Fine-tuned on P ROMPT EVALS train set

  69. [87]

    Quantitative Analyses (Unitary results, Intersectional results): See Table 8 Domain Similarity Precision Recall General-Purpose Chatbots 0.8140 0.8070 0.8221 Question-Answering 0.8104 0.8018 0.8199 Text Summarization 0.8601 0.8733 0.8479 Database Querying 0.8362 0.8509 0.8228 ...

  70. [88]

    Ethical Considerations: See Section 8

  71. [89]

    Caveats and Recommendations: None

  72. [2020]

    URL https://api.semanticscholar.org/ CorpusID:221516475

  73. [2022]

    URL https://api.semanticscholar.org/ CorpusID:253018554

  74. [2023]

    URL https://api.semanticscholar.org/ CorpusID:265067168

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.