Pith. sign in

REVIEW 4 major objections 6 minor 40 references

Audit, Alignment, and Optimization of LM-Powered Subroutines with Application to Public Comment Processing

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A framework declares LM calls as typed, auditable subroutines whose prompts improve online from sparse expert feedback, with an application to NEPA public comment processing.

desk verdict A useful systems paper with a real NEPA baseline, but the sparse-feedback improvement mechanism is untested and the headline comparison to SMEs is not supported by the data. read the letter →

arxiv 2507.08109 v1 pith:7GQ6VQ6H submitted 2025-07-10 cs.CL

Reilly Raab , Mike Parker , Dan Nally , Sadie Montgomery , Anastasia Bernat , Sai Munikoti , Sameera Horawalavithana This is my paper
classification cs.CL
keywords language-modelsubroutinesinfinite-armedbanditspromptoptimizationself-critiquehumanfeedbackalignmentauditabilitypubliccommentprocessingNEPA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that language models can be used as typed, callable, auditable subroutines inside ordinary software, with prompts that improve during use from sparse expert feedback rather than from prompt engineering or retraining. The authors argue that by treating prompt choice as an infinite-armed bandit and aligning a self-critique model with human ratings, an application can learn from a few expert corrections while recording every input, prompt, output, and dependency for audit. They build CommentNEPA for the NEPA public-comment workflow and compare its autonomous outputs to historical subject-matter-expert (SME) annotations, reporting quote precision around 74%, quote recall around 23%, and binning recall near 50%. They conclude that CommentNEPA's outputs are comparable to those of human experts with potentially less variance and greater precision, while explicitly leaving open whether online SME feedback improves the system as designed.

What carries the argument

The central machinery is the infinite-armed bandit treatment of prompt selection: each possible system prompt is an arm, an explorer arm represents all never-tried prompts, and Boltzmann sampling picks among known arms and the explorer at every call. New prompts are generated by an LM 'prompt engineer' from the subroutine's declared input/output schemas, and a self-critique subroutine scores outputs and is aligned to sparse human ratings by minimizing the squared difference in Eq. (3). A relational database records every prompt, input, output, and mutual dependency, producing an auditable computational graph.

What would settle it

A concrete test is to enable the feedback loop on a fresh batch of one of the four case-study corpora, have SMEs rate a subset of outputs, update the critique and prompt distributions via Eq. (3) and Eq. (1), then measure cumulative loss and held-out quote/binning metrics against the identical pipeline run with no feedback; if the feedback-enabled run does not beat the no-feedback baseline, the central online-learning claim is contradicted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a software framework that turns an LM call into a statically typed subroutine whose system prompt is an arm in an infinite-armed bandit; new prompts are synthesized by an LM from the subroutine declaration, selected by Boltzmann exploration, and scored by a self-critique subroutine whose loss is the squared difference between its rating and a human SME rating for the same output (Eq. 3). Because every call records its prompt, inputs, outputs, and mutual data dependencies, the resulting computational graph supports on-demand audit and feedback propagation. The application case study shows that when run without any human feedback, CommentNEPA extracts and bins public comments with precision comparable to human analysts and with less variance in extracted quote length, which the authors offer as evidence that the subroutine paradigm plus self-critique is a viable basis for responsible LM deployment in high-stakes text workflows.

Load-bearing premise

The load-bearing premise is that minimizing the squared difference between an LM critique's rating and a human expert's rating (Eq. 3) turns the critique into a trustworthy proxy for expert judgment, so that optimizing prompts against critique ratings improves the real task; the paper's evaluation never tests this with real SME ratings.

Editorial extensions

If this is right

  • Public comment processing could be run in batches with SMEs auditing sampled outputs instead of reading every letter, while every LM decision remains traceable to a recorded prompt, input, and output.
  • Sparse expert ratings could be multiplied through critique ratings, shifting prompt distributions toward expert preferences without fine-tuning model weights.
  • Schema-constrained LM outputs could be composed with static type-checking and conventional program logic, reducing the risk of malformed or unverifiable results.
  • In the four case studies, quotes selected by CommentNEPA are usually considered substantive by SMEs (high precision), but many substantive passages are missed, especially in longer documents (low recall).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, Eq. (3) should be tested head-on: collect paired SME and critique ratings on the same outputs and check whether the critique's squared-difference loss predicts downstream prompt improvement, since no such data appear in the evaluation.
  • Beyond the paper, the typed-subroutine architecture is plausibly transferable to healthcare or legal document triage, but the paper provides no evidence outside public comment processing that the bandit and critique mechanisms transfer.
  • Beyond the paper, the observed decline of recall with document length suggests that chunked or hierarchical document processing could recover missed quotes; the paper does not test such a variant.
  • Beyond the paper, the recorded data-dependency graph could support regulatory or legal defense by reconstructing exactly which prompt produced a contested output, possibly lowering institutional barriers to adoption.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a framework for declaring statically typed, LM-powered subroutines with full audit logging, an infinitely-many-armed bandit (∞-MAB) formulation for prompt selection, and self-critique loops that are intended to be aligned with sparse human expert feedback. The authors implement this framework in a Python library and build CommentNEPA, an application for NEPA public-comment processing. They evaluate CommentNEPA by running it autonomously (without any human feedback) on four case studies and comparing its outputs to historical SME annotations, reporting precision/recall for quote extraction and binning. The central claim in the abstract and conclusion is that sparse expert feedback can improve each subroutine online during use, and that CommentNEPA's outputs are comparable to those of subject matter experts.

Significance. If the framework worked as claimed, it would be a meaningful step toward auditable, human-in-the-loop use of LMs in high-stakes government and legal workflows. The paper has several strengths: the evaluation uses external human SME labels, so the headline precision and recall numbers are not circular; the audit-trail architecture (Section 2.4) is a practical and concrete contribution; the ∞-MAB formulation with an exploration arm is a reasonable way to handle a growing space of prompts; and the authors are unusually candid about the limitations of their own evaluation. However, the central mechanism—sparse human feedback improving subroutines online—is not tested anywhere in the manuscript. The CommentNEPA evaluation is explicitly an autonomous baseline, and Section 5 concedes that it remains to be determined whether SME feedback improves the system over time. The conclusion that outputs are 'comparable' to SMEs is also not supported without an inter-annotator agreement baseline or confidence intervals. As it stands, the paper demonstrates an architecture and a baseline, not the headline online-improvement capability.

major comments (4)
  1. [Section 2.3, Eq. (3); Section 4, third paragraph; Section 5] The mechanism that makes the central claim work—using Eq. (3) to turn sparse SME ratings into training signals for the critique subroutine, whose ratings then serve as loss values for the target prompt MAB—is never exercised in any experiment. The CommentNEPA evaluation is explicitly run 'without SME feedback' (Section 4, third paragraph), and Section 5 concedes 'it remains to be determined whether SME feedback is able to improve the system over time as desired.' Because the abstract's central claim is that sparse expert feedback improves subroutines online, the paper currently demonstrates the components in isolation (MAB sampling, self-critique loops) but not the integrated feedback loop. This is a load-bearing gap: either add a human-in-the-loop experiment that measures whether Eq. (3) aligns critique ratings with SME judgments and whether this improves target-subroutine loss over time, or revise the abstract and conclusion to describe the online feedback mechanism as a proposal rather than a demonstrated capability.
  2. [Section 5; Table 1] The conclusion that CommentNEPA is 'capable of extracting and sorting data with outputs that are comparable to those of subject matter experts' is not supported by Table 1. The table reports no inter-annotator agreement baseline, no confidence intervals, and no significance tests; with aggregate quote recall of 23.1% and binning recall of 49.9%, the compatibility of these numbers with 'comparable' is unquantified. The authors themselves note in Section 4 that the comparison is not 'apples-to-apples' because the tasks differ (SMEs select contiguous text blocks; CommentNEPA extracts quotes supporting paraphrased concerns). A statement of comparability requires an SME-SME agreement measure on the same task, or at minimum a clearly stated benchmark threshold. Without this, the conclusion overreaches the data.
  3. [Section 2.1, Figures 1 and 2] The prompt-evolution demonstration is a single 100-trial run with a linearly scheduled inverse-temperature parameter β. No multiple seeds, no confidence intervals, and no comparison against alternative sampling strategies are provided; the figure shows a downward trend, but the run-to-run variability is unknown. As the only empirical support for the ∞-MAB prompt-sampling component, this demonstration is too weak to establish that the method 'results in an evolution distribution of prompts' in a reliable way. Please provide repeated trials with variance, and ideally a comparison with random sampling or a fixed-prompt baseline.
  4. [Section 4, Table 1 and Figures 10-12] The recall results are not reported with respect to document length in the summary table. Figure 12 shows that recall declines strongly with document length, and Figures 10 and 11 show broad per-document distributions; yet Table 1 reports only aggregate recall/precision per case study. Given the authors' own list of explanations (Section 4) for why recall depends on document length, the aggregate numbers are not representative without length-stratified reporting. This also matters for the 'potentially less variance' claim in Section 5, which is based on Table 2 but does not account for the substantial per-document variance visible in Figures 10-12. Please report length-stratified metrics or a regression model, and qualify the aggregate figures accordingly.
minor comments (6)
  1. [Section 2.1, Eq. (1)] The typesetting of Eq. (1) is malformed; the denominator and the definition of L0 are difficult to parse. Please rewrite the equation with proper fraction formatting and clear indices.
  2. [Section 2.2, Figure 3] The docstring in the example declaration reads 'Count the of number of words' rather than 'Count the number of words'; please fix this typo.
  3. [Section 4, near Figure 11] There is a stray period in the parenthetical 'Figure 12 .' and the phrase 'It vary rarely selects quotes' should read 'It very rarely selects quotes.'
  4. [Section 1.3] The sentence 'While DSPy provides modular approach to optimizing entire LM programs' is missing an article; it should be 'a modular approach.'
  5. [Section 1 and Section 2.5] The paper claims an open-source library, but the footnote states that a link 'will be included pending internal disclosure processes,' and no code or link is present in the manuscript. Please provide the code repository or clearly state its availability status so that the reproducibility claims can be assessed.
  6. [Section 4, Table 1] The table reports 'Bins' counts, but it does not indicate which runs used binning guidance and which did not; the text mentions that CFFF had 19 bins with no instructions while WS had 62 bins with detailed instructions. Including guidance information in the table would help interpret the binning metrics.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the framework's core claims are either externally benchmarked or explicitly deferred, and no fitted parameter is renamed as a prediction.

full rationale

The paper's derivation chain is not circular. The central optimization mechanism in Eq. (3) defines the critique loss as the squared difference between an SME-derived loss and a critique-derived loss for the same target output, which makes the critique a proxy trained against external human judgment rather than a quantity defined in terms of the target subroutine's own output. The CommentNEPA evaluation is an external benchmark: Table 1 and Figures 10-18 compare autonomous pipeline outputs to historical SME-labeled ground-truth data, so the reported precision and recall numbers are not predetermined by the framework's internal losses. The paper explicitly states that the evaluated runs operate 'without SME feedback' and that 'it remains to be determined whether SME feedback is able to improve the system over time as desired,' so the online-improvement claim is presented as an untested design goal rather than as a result forced by construction. The only self-citations (Parker et al. 2024 and Nally et al. 2025) are contextual and motivational, not load-bearing for any technical conclusion. The absence of a direct validation of the critic-as-SME proxy is an evidence gap or correctness risk, not circularity, and no equation or fitted parameter reduces to its own input within the claimed derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on heuristic bandit sampling and an unvalidated self-critique alignment loss; none of these are derived from first principles or machine-checked. The paper itself cites distribution-dependence of bandit performance and admits the human-feedback benefit is untested. No new physical or conceptual entities are introduced.

free parameters (4)
  • Boltzmann inverse temperature beta = toy example: linearly increasing 0 to 1 over 100 trials; CommentNEPA: unspecified
    Controls the exploration-exploitation trade-off in Equation (1); chosen by hand, not derived or fit to the application data.
  • Exploration arm loss estimate L0 = mean of observed arm losses (Eq. 1)
    Heuristic estimate for the infinite-armed bandit exploration arm; no regret guarantee is provided for this choice, and the paper cites Carpentier and Valko (2015) on distribution dependence.
  • Self-critique hyperparameters (loop count, early-exit threshold, rating scale) = not specified
    Self-critique loops require bounded and discretized ratings, a finite loop count, and early-exit criteria; these values are not reported, which affects reproducibility.
  • Fuzzy quote-matching threshold (rapidfuzz) = not specified
    Quote post-processing uses fuzzy string matching to guarantee quotes appear in the source letter; the similarity cutoff is not stated.
assumptions (5)
  • domain assumption Boltzmann exploration with a substitute exploration arm is an effective online prompt optimizer for this infinite-armed bandit problem.
    Section 2.1 adopts this sampling rule; the paper provides no regret bound for its heuristic and cites Carpentier and Valko (2015) showing performance depends on the loss distribution, which is unknown here.
  • domain assumption The critique loss defined by Eq. (3) is a valid training signal for aligning critique subroutines with SME preferences.
    Section 2.3 defines L_critique = (L_target[SME] - L_target[critique])^2; the paper reports no experiment with actual SME ratings to validate this alignment.
  • domain assumption An LM 'prompt engineer' can generate a sufficiently diverse and effective set of novel prompts from a static subroutine declaration.
    Section 2.2 relies on this; only a single toy example is shown, and the authors note that alternative meta-reasoning mutation approaches produced repetitive prompts.
  • domain assumption Historical SME comment delineation and binning labels are a valid ground truth for evaluating CommentNEPA outputs.
    Section 4 acknowledges the SME workflow differs from CommentNEPA's ('not an apples-to-apples comparison') yet still uses SME labels as the benchmark.
  • domain assumption Sentence-level overlap between LM quotes and SME comments is a meaningful quality metric.
    Equations (4) and (5) treat each sentence as an independent binary classification, but sentences within a document are correlated and SMEs extract multi-sentence runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Audit, Alignment, and Optimization of LM-Powered Subroutines with Application to Public Comment Processing." pith.science (2026). https://pith.science/paper/7GQ6VQ6H

@misc{pith2026250708109,
  author       = {Pith},
  title        = {Pith review of: Audit, Alignment, and Optimization of LM-Powered Subroutines with Application to Public Comment Processing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GQ6VQ6H}},
  note         = {Machine review of arXiv:2507.08109}
}
read the original abstract

The advent of language models (LMs) has the potential to dramatically accelerate tasks that may be cast to text-processing; however, real-world adoption is hindered by concerns regarding safety, explainability, and bias. How can we responsibly leverage LMs in a transparent, auditable manner -- minimizing risk and allowing human experts to focus on informed decision-making rather than data-processing or prompt engineering? In this work, we propose a framework for declaring statically typed, LM-powered subroutines (i.e., callable, function-like procedures) for use within conventional asynchronous code -- such that sparse feedback from human experts is used to improve the performance of each subroutine online (i.e., during use). In our implementation, all LM-produced artifacts (i.e., prompts, inputs, outputs, and data-dependencies) are recorded and exposed to audit on demand. We package this framework as a library to support its adoption and continued development. While this framework may be applicable across several real-world decision workflows (e.g., in healthcare and legal fields), we evaluate it in the context of public comment processing as mandated by the 1969 National Environmental Protection Act (NEPA): Specifically, we use this framework to develop "CommentNEPA," an application that compiles, organizes, and summarizes a corpus of public commentary submitted in response to a project requiring environmental review. We quantitatively evaluate the application by comparing its outputs (when operating without human feedback) to historical ``ground-truth'' data as labelled by human annotators during the preparation of official environmental impact statements.

Figures

Figures reproduced from arXiv: 2507.08109 by the authors.

Figure 1
Figure 1. We track a decrease in average loss over the course of 100 trials for an LM-powered subroutine with system [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. We track which prompt (sorted vertically by empirical average loss) is selected for each trial of a target [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. An example subroutine declaration in our framework. This declaration is used to synthesize prompts, such [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: A prompt synthesized by an LM for the subroutine declared in Figure 3. In practice, such LM-generated [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: An example output type used to constrain an LM-powered “critique” subroutine. The associated input type [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Outputs from a “critique” LM-powered subroutine are used to adjust [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: In addition to optimizing the prompts used to perform each LM-powered subroutine, we optimize the outputs [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: A high-level overview of the architecture we implement to build applications on top of our proposed method [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Data flow between stages of the processing pipeline that we formalize for CommentNEPA. We treat the cor [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: The distribution of CommentNEPA’s quote-selection precision across case studies. We visualize the number [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: The distribution of CommentNEPA’s quote-selection recall across case studies. We visualize the number [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: The recall of sentences selected by SME (as independent, substantive comments) among set of quotes [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: We count the number of documents for which CommentNEPA achieved ranges of binning recall (i.e., for [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: The project description for the BLM’s Western Solar (WS) Development Plan provided to relevant LM [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: A comparison of comment delineation and binning performed by SMEs and CommentNEPA on an excerpt [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: A comparison of comment delineation and binning performed by SMEs and CommentNEPA on an excerpt [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: The relative frequency distribution of bins (concern categories) identified in text by CommentNEPA and [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: A truncation of an (approximate) “confusion matrix” produced by CommentNEPA on the BLM’s Western [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 19 canonical work pages

  1. [1]

    LLMs in Finance: Applications, Examples, & Benefits , May 2025

    AI21 labs . LLMs in Finance: Applications, Examples, & Benefits , May 2025. URL https://www.ai21.com/knowledge/llms-in-finance/

    work page 2025

  2. [2]

    Large language models in real-world clinical workflows: A systematic review of applications and implementation

    Yaara Artsi, Vera Sorin, Benjamin S Glicksberg, Panagiotis Korfiatis, Girish Nadkarni, and Eyal Klang. Large language models in real-world clinical workflows: A systematic review of applications and implementation. medRxiv, pages 2025--06, 2025

    work page 2025

  3. [3]

    Benchmarking llms for real-world applications: From numerical metrics to contextual and qualitative evaluation

    Huthaifa I Ashqar. Benchmarking llms for real-world applications: From numerical metrics to contextual and qualitative evaluation. Authorea Preprints, 2025

    work page 2025

  4. [4]

    The nonstochastic multiarmed bandit problem

    Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32 0 (1): 0 48--77, 2002

    2002

  5. [5]

    rapidfuzz/rapidfuzz: Release 3.13.0, 2025

    Max Bachmann. rapidfuzz/rapidfuzz: Release 3.13.0, 2025. URL https://doi.org/10.5281/zenodo.15133267

    doi:10.5281/zenodo.15133267 2025

  6. [6]

    Maillard Sampling: Boltzmann Exploration Done Optimally

    Jie Bian and Kwang-Sung Jun. Maillard sampling: Boltzmann exploration done optimally. In International Conference on Artificial Intelligence and Statistics, pages 54--72. PMLR, 2022. URL https://arxiv.org/abs/2111.03290

    work page Pith review arXiv 2022

  7. [7]

    Simple regret for infinitely many armed bandits

    Alexandra Carpentier and Michal Valko. Simple regret for infinitely many armed bandits. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1133--1141, Lille, France, 07--09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/carpentier15.html

    work page 2015

  8. [8]

    Prompt stability matters: Evaluating and optimizing auto-generated prompt in general-purpose systems

    Ke Chen, Yufei Zhou, Xitong Zhang, and Haohan Wang. Prompt stability matters: Evaluating and optimizing auto-generated prompt in general-purpose systems. arXiv preprint arXiv:2505.13546, 2025. URL https://arxiv.org/abs/2505.13546

    arXiv 2025

Show all 40 references
  1. [9]

    Pydantic , 2025

    Samuel Colvin, Eric Jolibois, Hasan Ramezani, Adrian Garcia Badaracco, Terrence Dorsey, David Montague, Serge Matveenko, Marcelo Trylesinski, Sydney Runkle, David Hewitt, Alex Hall, and Victorien Plot. Pydantic , 2025. URL https://github.com/pydantic/pydantic

    2025

  2. [10]

    Promptbreeder: Self-referential self-improvement via prompt evolution

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt \"a schel. Promptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint arXiv:2309.16797, 2023. URL https://arxiv.org/abs/2309.16797

    2023 arXiv

  3. [11]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  4. [12]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM computing surveys, 55 0 (12): 0 1--38, 2023

    2023

  5. [13]

    Dspy: Compiling declarative language model calls into self-improving pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714,...

    arXiv

  6. [14]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021. URL https://arxiv.org/abs/2104.08691

    2021 arXiv

  7. [15]

    Building trustworthy ai: Transparent ai systems via large language models, ontologies, and logical reasoning (transpnet)

    Fadi Al Machot, Martin Thomas Horsch, and Habib Ullah. Building trustworthy ai: Transparent ai systems via large language models, ontologies, and logical reasoning (transpnet). arXiv preprint arXiv:2411.08469, 2024

    2024 arXiv

  8. [16]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36: 0 46534--46594, 202...

    2023 arXiv

  9. [17]

    Pyright: Static type checker for python, 2025

    Microsoft . Pyright: Static type checker for python, 2025. URL https://github.com/microsoft/pyright

    2025

  10. [18]

    Large language models in ai: Opportunities and limitations in real-world applications

    Junichiro Mori. Large language models in ai: Opportunities and limitations in real-world applications. Advances in Computer Sciences, 7 0 (1), 2024

    2024

  11. [19]

    Workshop summary report on using ai tools to improve the efficiency and outcomes of the NEPA process

    Dan Nally, Mike Parker, Matthew Aumeier, Kevin Murphy, Michelle Rau, James McWalter, Jack Titus, Lauren Schramm, Reilly Raab, Anurag Acharya, Sarthak Chaturvedi, Anastasia Bernat, Sai Munikoti, and Sameera Horawalavithana. Workshop summary report on using ai tools to improve t...

    2025

  12. [20]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022 arXiv

  13. [21]

    To what extent have llms reshaped the legal domain so far? a scoping literature review

    Bogdan Padiu, Radu Iacob, Traian Rebedea, and Mihai Dascalu. To what extent have llms reshaped the legal domain so far? a scoping literature review. Information, 15 0 (11): 0 662, 2024

    2024

  14. [22]

    Testing artificial intelligence tools to streamline the public comment‑review process for nepa environmental reviews

    Mike Parker, Ann Miracle, Dan Nally, Hung Phan, Anurag Acharya, Sai Munikoti, Gihan Panapitiya, and Sameera Horawalavithana. Testing artificial intelligence tools to streamline the public comment‑review process for nepa environmental reviews. Technical Report PNNL‑35927, Pacif...

    2024

  15. [23]

    Hung Phan, Anurag Acharya, Sarthak Chaturvedi, Shivam Sharma, Mike Parker, Dan Nally, Ali Jannesari, Karl Pazdernik, Mahantesh Halappanavar, Sai Munikoti, et al. Rag vs. long context: Examining frontier large language models for environmental review document comprehension. arX...

    2024

  16. [24]

    Grips: Gradient-free, edit-based instruction search for prompting large language models

    Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281, 2022

    2022 arXiv

  17. [25]

    gradient descent

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495, 2023. URL https://arxiv.org/abs/2305.03495

    2023 arXiv

  18. [26]

    A comprehensive survey of bias in llms: Current landscape and future directions

    Rajesh Ranjan, Shailja Gupta, and Surya Narayan Singh. A comprehensive survey of bias in llms: Current landscape and future directions. arXiv preprint arXiv:2409.16430, 2024

    2024 arXiv

  19. [27]

    Efficient prompt optimization through the lens of best arm identification

    Chengshuai Shi, Kun Yang, Zihan Chen, Jundong Li, Jing Yang, and Cong Shen. Efficient prompt optimization through the lens of best arm identification. arXiv preprint arXiv:2402.09723, 2024. URL https://arxiv.org/abs/2402.09723

    2024 arXiv

  20. [28]

    Quantifying the substantive influence of public comment on united states federal environmental decisions under nepa

    Ashley Stava, Wayne E Thogmartin, Robert Merideth, Steven Bethard, Faiz Currim, Jonathan J Derbridge, Kirk Emerson, Egoitz Laparra, Aaron Lien, Emily McGovern, Justin Pidot, Marc Miller, Krista Romero-Cardenas, Blaze Smith, Carly Winnebald, and Laura López-Hoffman. Quantifying...

    2025 doi

  21. [29]

    Presidential Memoranda: Updating Permitting Technology for the 21st Century , 2025

    The White House . Presidential Memoranda: Updating Permitting Technology for the 21st Century , 2025. URL https://www.whitehouse.gov/presidential-actions/2025/04/updating-permitting-technology-for-the-21st-century/

    2025

  22. [30]

    Department of the Interior, Bureau of Land Management

    U.S. Department of the Interior, Bureau of Land Management . Scoping Summary Report: Utility‑Scale Solar Energy Programmatic Environmental Impact Statement (DOI‑BLM‑HQ‑3000‑2023‑0001‑RMP‑EIS) . Programmatic EIS Scoping Report DOI‑BLM‑HQ‑3000‑2023‑0001‑RMP‑EIS, Bureau of Land M...

    2023

  23. [31]

    Fish and Wildlife Service

    U.S. Fish and Wildlife Service . Migratory bird permits; authorizing the incidental take of migratory birds: Advance notice of proposed rulemaking; notice of intent to prepare a national environmental policy act document. Federal Register, October 2021. URL https://www.federal...

    2021

  24. [32]

    Nuclear Regulatory Commission

    U.S. Nuclear Regulatory Commission . Environmental Impact Statement for the License Renewal of the Columbia Fuel Fabrication Facility in Richland County, South Carolina – Final Report (NUREG‑2248) , July 2022. URL https://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr2...

    2022

  25. [33]

    Nuclear Regulatory Commission

    U.S. Nuclear Regulatory Commission . Comanche Peak Nuclear Power Plant, Units 1 and 2 – License Renewal Application (ADAMS Accession No. ML22276A082) . U.S. Nuclear Regulatory Commission ADAMS package, 2024. URL https://www.nrc.gov/reactors/operating/licensing/renewal/applicat...

    2024

  26. [34]

    Workflow patterns distributed and parallel databases, vol

    W Van der Aalst, A ter Hofstede, B Kiepuszewski, and AB Barros. Workflow patterns distributed and parallel databases, vol. 14, 2003

    2003

  27. [35]

    Teach better or show smarter? on instructions and exemplars in automatic prompt optimization

    Xingchen Wan, Ruoxi Sun, Hootan Nakhost, and Sercan Arik. Teach better or show smarter? on instructions and exemplars in automatic prompt optimization. Advances in Neural Information Processing Systems, 37: 0 58174--58244, 2024. URL https://arxiv.org/abs/2406.15708

    2024 arXiv

  28. [36]

    Promptagent: Strategic planning with language models enables expert-level prompt optimization

    Xinyuan Wang, Chenxi Li, Zhen Wang, Fan Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic, Eric P Xing, and Zhiting Hu. Promptagent: Strategic planning with language models enables expert-level prompt optimization. arXiv preprint arXiv:2310.16427, 2023. URL https://arxiv.org/abs/2...

    arXiv

  29. [37]

    Efficient guided generation for llms

    Brandon T Willard and R \'e mi Louf. Efficient guided generation for llms. arXiv preprint arXiv:2307.09702, 2023. URL https://arxiv.org/abs/2307.09702

    2023 arXiv

  30. [38]

    Sipdo: Closed-loop prompt optimization via synthetic data feedback

    Yaoning Yu, Ye Yu, Kai Wei, Haojing Luo, and Haohan Wang. Sipdo: Closed-loop prompt optimization via synthetic data feedback. arXiv preprint arXiv:2505.19514, 2025. URL https://arxiv.org/abs/2505.19514

    2025

  31. [39]

    differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. Textgrad: Automatic" differentiation" via text. arXiv preprint arXiv:2406.07496, 2024. URL https://arxiv.org/abs/2406.07496

    2024 arXiv

  32. [40]

    Large language models are human-level prompt engineers

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022. URL https://arxiv.org/abs/2211.01910

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.