Artificial Intelligence · arXiv:2610.31062 v1 · submitted 22 Oct 2026 · 104 references
SCOPE: Compiling Natural-Language Policies into Neurosymbolic Predicate Programs
The paper says
“Policy enforcement for LLM agents has shifted from single-turn content moderation to compound, application-specific policies evaluated over multi-turn transcripts and tool-call trajectories.”
more of the abstract
Fine-tuned guardrail models are efficient but degrade on compound policies, and while LLM-as-a-judge approaches achieve higher accuracy, they incur high inference cost and tend to produce unstable predictions. Neurosymbolic methods offer a natural alternative, decomposing the policy into narrow neural decisions composed under explicit logic. However, existing symbolic layers lack the required expressiveness and are optimized against fixed data rather than the policy itself. To address this, we introduce SCOPE, which compiles a policy into an executable program of neural predicates executed by small models, deterministic extractors, and code that composes their outputs. The program is optimized jointly with its training data, using adversarial generation of counterexamples that target where the program deviates from the policy. We evaluate SCOPE on a benchmark of real-world policy tasks against prompt optimization, task decomposition, and workflow generation methods, achieving the highest average accuracy across tasks and models. Programs optimized with the weakest executor transfer to other models without further optimization. This allows them to be deployed in production on inexpensive models, where they reach accuracy comparable to frontier reasoning models. Unlike these monolithic models, SCOPE produces substantially more stable predictions, and each verdict is interpretable through the predicate outputs that produced it. Additionally, we validate the key components of our method through an ablation study. SCOPE thus enables reliable, interpretable, and efficient policy enforcement at production scale.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 23 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Joint decomposition-plus-adversarial-data search is real and the evaluation is unusually careful, but the teacher's policy reading is the load-bearing premise and remains unmeasured. This line is wrong
2 major objections · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 104 references say.
The paper says, at Sections 3.2, 3.3, 4.3, 6; Appendix E.2
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.73):
“Because the program is checked against a strong teacher rather than a formal verifier, its correctness rests on the teacher’s reading of the policy, and verifying such programs remains an open problem.”
Pith says: Objection 1 of 2 · major
The paper acknowledges in Section 6 that the correctness of a compiled program rests on the teacher's reading of the policy, but it never measures that reading, and the design concentrates the compilation claim on a single oracle. Both the adversarial generators and the teacher are Claude Opus 4.8 (Section 4.3), so the agreement filter of Section 3.3 ('every admitted …
more of objection 1
… sample reflects two separate readings of the policy') rules out per-sample stochastic errors but not a systematic interpretation bias shared by one model family. The teacher also writes the rationales for the D0 samples (Section 3.2) that the analyzer uses to attribute failures, so the teacher's interpretation enters the loop through both labels and rationales. Because the returned program is selected by score on the fixed human-labeled D0, a teacher misreading concentrated on the adversarial samples would steer the search trajectory while remaining invisible to selection; Appendix E.2 gives a concrete boundary case ('new information that changes the situation') where a plausible predicate reading conflicts with the policy's intent as the authors themselves describe it. I request a per-task measurement of teacher-versus-human label agreement (or a cross-model teacher check on a subset), together with a statement of how the measured agreement bounds the achievable accuracy of the compiled programs; without this, the generalization claim that natural-language policies can be compiled rests on an unquantified oracle.
The paper says, at Section 5.2, Table 4; Abstract
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The ablation supports a weaker claim than the abstract's 'Ablations confirm that adversarial generation is central to these gains': the significant effects are modest (Coaching +0.056, Plan +0.052, Tile-match +0.038) and Escalation shows no effect (0.815 versus 0.816). Table 4 reports only means, and with five runs per arm the one-sided Welch tests cannot be checked without the underlying per-run measurements; two of the three p-values (0.033, 0.045) are close enough to the 0.05 threshold that the distributional assumptions matter.
more of objection 2
Please report the five per-run values or confidence intervals, and rephrase the abstract and conclusion to say that adversarial generation significantly improves three of the four tasks rather than claiming that it is central to the gains.
7 minor objections
The paper says, at Table 3 and Section 5.1
Pith says: minor objection 1 of 7
The transfer table reports single-point estimates without variance for the five open-weight models, and one cell (Qwen3-235B-A22B on Plan, down 0.04) shows negative transfer; please report repeated runs for the transfer evaluation or state explicitly that only Tile-match was evaluated for stability, and add a sentence explaining the negative-transfer cell.
The paper says, at Abstract and Section 5.1
Pith says: minor objection 2 of 7
The phrase 'matching frontier reasoning models' overstates the task-level picture: DeepSeek-V4.1-Flash scores 0.78 on Coaching against 0.94 (gpt-5.5) and 0.96 (gemini-3.1-pro). The claim is defensible only on average across tasks, so it should be qualified as task-dependent.
show 5 more
The paper says, at Appendix C, Table 5
Pith says: minor objection 3 of 7
The cost ratios depend on the $0.15-per-million-input-token assumption and on treating decomposition output cost as negligible, and the number of predicate calls and tokens per sample for the decomposed runs is not reported; please state these quantities so the ratios can be reproduced, and frame the comparison as illustrative rather than exact.
The paper says, at Tables 2 and 6
Pith says: minor objection 4 of 7
Standard deviations are reported for the nano executor (Table 2) but not for the mini and flash executors in the detailed results; since stability is one of the paper's central claims, please report the spread for all three executors and for the ablation arms.
The paper says, at Section 4.2
Pith says: minor objection 5 of 7
For GEPA and MIPROv2, the model used to propose candidate instructions is not specified; to rule out a confound between the optimization machinery (Claude Opus 4.8 agents) and the SCOPE method itself, please state which model DSPy uses as the proposal model in each baseline.
The paper says, at Appendix B
Pith says: minor objection 6 of 7
Three of the four datasets are constructed or modified (Coaching and Tile-match constructed, Plan built on GAIA with authored samples, Escalation derived from DynaGuard with a sharpened definition), and no inter-annotator agreement is reported; please report annotation agreement on the test labels, especially for the constructed tasks.
The paper says, at Section 4.3
Pith says: minor objection 7 of 7
The search hyperparameters (tau = 0.98, K = 8, and the adversarial caps) are fixed constants with no sensitivity analysis; a short statement of how the three-run standard deviations constrain their influence, or a one-point sensitivity check on tau, would increase confidence in the reported results.
What would settle it
One focused experiment settles it: on the Escalation task (the most boundary-sensitive policy), collect Claude Opus 4.8 teacher labels for the full 286-sample test set and for all admitted adversarial samples from one optimization run; have two independent human annotators relabel both sets using only the policy text; compute teacher-human and human-human agreement. If teacher-human agreement is within a few points of human-human agreement, the teacher reading is reliable. Additionally, recompile Escalation using only adversarial samples on which the teacher and an independent second frontier model agree, and check that held-out accuracy stays within the reported range; if it drops materially, the teacher-blind-spot concern lands and the 10/12 claim needs re-scoping.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 23 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Bottom line: this is a genuine advance in turning a natural-language policy into an executable guardrail, and the main empirical claims look conditionally true. The new thing is not any single component—decomposition, code-as-composition, adversarial curricula, LLM search all exist—but the joint loop: the program and its training pool co-evolve, with counterexamples aimed at predicate-level and composition-level drift, and the search runs against a deliberately weak executor. That last choice is what makes the transfer result credible. The paper also keeps its evaluation honest: three optimization runs times three test evaluations, a five-run ablation with Welch tests, complete prompts and a worked example program in appendices, and the benchmark released on HuggingFace. The citation pattern is broad and on point. I came in expecting static-benchmark curve-fitting; this is not that. Selection on the fixed seed pool and held-out test labels means the accuracy numbers are not circular.
The soft spots are real but not disqualifying. The benchmark is four tasks, two of them author-constructed, with small test sets, so the 10-of-12 claim rests on a narrow base. The ablation is described as showing adversarial generation is central, but it does nothing on Escalation and the effects on Coaching and Tile-match are modest; Plan is the clean win. Cost ratios are list-price estimates, not measured serving costs. No code is released, only the dataset. The load-bearing assumption is the teacher: every rationale and every admitted adversarial label comes from Claude Opus 4.8's reading of the policy. The paper acknowledges this in Section 6 and gates samples on generator–teacher agreement, which helps, but it never measures teacher agreement against human labels on any task. A teacher bias outside the seed pool would survive the D0 selection and would be invisible in the reported numbers. That is the main thing I would want closed before betting production systems on it.
Who this is for: people building guardrails, agent-policy enforcers, and anyone working on program synthesis as policy compilation. It deserves a serious referee, not a desk reject. I would send it to review and ask for three things: code release, a human-agreement measurement on teacher labels, and at least one externally built task. If those come back solid, I would take the central claim as established.
The strongest objection
Every stage of the optimization consumes the teacher's interpretation: teacher-written rationales for D0 samples (Sec. 3.2), teacher labels for every admitted adversarial sample (Sec. 3.3), and analyzer failure attributions grounded in those rationales. The paper acknowledges this in Section 6 ('its correctness rests on the teacher's reading of the policy', [b226]) and mitigates it with rule-first instructions [b545], but it never measures teacher accuracy against human labels on any task. The failure mode is concrete: Appendix E.2 shows a boundary case ('new information that changes the situation') where a plausible predicate reading disagrees with the policy's intent, and whether the teacher shares the human reading on such cases decides whether wrong samples enter Dt. Because the final program is selected on D0 (Sec. 3.2), a teacher error concentrated outside the seed pool survives selection. The test labels are human, so the reported 10/12 numbers would remain honest, but the generalization claim that natural-language policies can be compiled rests on the teacher being a faithful oracle, and the paper provides no bound on the teacher's error rate or its effect on the selected program.
The strongest claim is conditional on the teacher's reading of the policy. The paper is explicit about this in Section 6 [b226], and the rule-first prompts to the analyzer and decomposer [b545] are a real mitigation, as is final selection on the human-labeled seed pool D0. I therefore do not treat this as a hidden flaw or a reason to reject. It is nonetheless the most load-bearing premise for the generality claim: adversarial samples are labeled only by the teacher, seed rationales are teacher-written, and the same model family drives the generator, so a systematic blind spot on policy-boundary cases would be baked into the program and could survive D0 selection. The benchmark's human test labels keep the reported accuracies meaningful, but they do not measure the teacher's fidelity on the distribution that actually shapes the program. The proposed test is a direct measurement of that fidelity plus a two-teacher sensitivity check; until such a check is reported, the correct verdict remains CONDITIONAL, with code release and teacher-agreement data as the natural conditions for acceptance. No change to the reader's verdict is needed.
The referee's report
If the claims hold, this is a meaningful step for LLM policy enforcement: compound, multi-clause policies are compiled into inspectable programs whose predicates are answerable by cheap models, moving the strong-model cost to compile time and producing verdicts whose composition is deterministic and traceable. The evaluation is more careful than is typical for this line of work: results are averaged over three independent optimization runs and three test evaluations; the ablation uses five runs per arm with Welch tests; the appendices disclose the full optimizer and adversary prompts, a complete example program, and worked adversarial examples; the null ablation result on Escalation is reported rather than omitted; and the teacher-dependence limitation is stated explicitly in Section 6. The main accuracy claims are not circular: programs are selected on the fixed human-labeled seed pool and scored on held-out test sets. Notably, the no-adversary variant in Table 4 still outperforms most baselines in Table 2, suggesting that the decomposition itself, not only the generated data, carries much of the gain, which strengthens the central claim.
Pith's reading of the paper (a machine's summary, not the authors')
The paper claims that enforcing a natural-language policy — a compound rule with conditions, exceptions, counts, and interactions, judged over multi-turn transcripts and tool traces — can be a compilation problem rather than a judgment problem. SCOPE compiles the policy into a Python program whose atoms are narrow yes/no questions answered by cheap, non-reasoning models and whose glue is deterministic code, then optimizes the program jointly with an adversarially grown pool of counterexamples generated where the program drifts from the policy. Across four real-world policy tasks, the compiled programs outperform prompt optimization, task decomposition, and workflow generation in 10 of 12 task-executor settings, and programs compiled with the weakest executor transfer to other models without tuning, matching frontier reasoning models at 9–47 times lower inference cost with far more stable verdicts. If correct, this changes the economics and accountability of policy enforcement: verdicts become cheap, repeatable, and traceable to the predicate outputs that produced them, rather than a single expensive and unstable frontier-model call.
What carries the argument. The load-bearing object is the decomposition program: neural predicates — self-contained yes/no questions that a weak, non-reasoning model can answer — composed by deterministic extractors and Python code that controls which text each predicate receives and how the answers combine into a verdict. The second mechanism is the adversarial co-optimization loop, modeled on counterexample-guided synthesis: an analyzer replays each failure's execution trace to attribute it to a specific predicate or to the composition code, a decomposer revises the program with revisions grounded in clauses of the policy rather than in sample patterns, and two adversaries generate new counterexamples at the level of a single predicate and of the whole composition, which the teacher labels and folds into the next round. Because the executor must actually answer every predicate during the search, the weakest executors force the finest decompositions, and that is what makes the results portable: once a clause is reduced to a question any small model can answer, the program runs unmodified on other models.
Core claim. SCOPE's central claim is that compound natural-language policies can be compiled rather than interpreted. The search begins with the policy as a single predicate and iteratively rewrites it into a decomposition program: predicates, self-contained classification questions answered by a sub-frontier model with reasoning disabled; extractors, deterministic functions that compute exact values such as counts or select spans; and Python composition code that decides what text each predicate reads, whether to apply it conditionally or repeatedly, and how to map the labels to a verdict. Program and training data are optimized together: after each revision, a systemic adversary generates counterexamples where the composition misroutes or omits a clause, and local adversaries probe individual predicates that drift from their clause, with a strong teacher labeling everything against the full policy. Since supervision exists only at the final verdict, failures are attributed by replaying the composition on recorded predicate labels to find which calls would have had to differ. The converged programs are nearly executor-independent — upgrading the executor buys 0.26 average accuracy under prompt optimization but only 0.09 under SCOPE — and programs compiled with the weakest executor transfer to five other models with no further optimization, reaching accuracy comparable to frontier reasoning models while flipping verdicts on only 5–7% of samples across runs, versus 26–37% for frontier reasoning models on the undecomposed task.
The premise it rests on. The load-bearing premise is that the strong teacher model reads the natural-language policy correctly when it labels the seed samples and every adversarially generated counterexample; the paper acknowledges this outright, noting that correctness 'rests on the teacher's reading of the policy,' because no formal verifier checks the resulting program against the text of the policy.
If the paper is right (Pith's inferences, not the authors')
An organization can serve policy checks from a cheap model, with the measured 9–47× inference-cost reduction over a frontier reasoning judge, and re-run the compiled program identically on every interaction.
Executor choice stops being an accuracy decision: since upgrading the executor gains only 0.09 average accuracy under SCOPE versus 0.26 under prompt optimization, teams can deploy the cheapest model that runs the predicates.
A program compiled once with the weakest executor is a portable asset: all five open-weight transfer models improved over their own undecomposed prompting, on average 0.11–0.22 per task, with no per-model tuning.
Verdict stability follows from the architecture: with nondeterminism confined to narrow predicate calls, only 5–7% of Tile-match samples changed prediction across five runs, versus 26–37% for frontier reasoning models.
The same co-evolution loop is claimed to carry over to other natural-language-specified tasks, with rubric-based LLM evaluation and reward modeling named as candidates.
Because the loop's correctness rests on the teacher's reading of the policy, a compiled program is best read as an executable encoding of that reading; an independent check, such as compiling the same policy with a second teacher model and comparing the programs on shared held-out inputs, would reveal how much of the reported accuracy is policy fidelity and how much is teacher fidelity.
The near-invariance of accuracy to executor strength hints that the search is distilling the judge's competence into reusable small checks; a testable extension is whether predicates discovered on one policy, such as 'does this reply connect the user to a human agent?', transfer as building blocks to a different policy in the same domain, turning per-policy compilation into assembly from a library
The local/systemic adversary split targets two distinct failure modes, which suggests compilation effort could be allocated predictively: if, across tasks, one adversary type consistently accounts for the decisive revisions, practitioners could budget search rounds toward rewording predicates or toward restructuring the composition accordingly.
The stability contrast is partly mechanical — only predicate calls are stochastic, each a narrow choice — so the stability gain should be largest exactly where monolithic nondeterminism is highest, a prediction consistent with the paper's multi-label Tile-match measurement but testable on other high-variance judge tasks.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Policy enforcement for LLM agents has shifted from single-turn content moderation to compound, application-specific policies evaluated over multi-turn transcripts and tool-call trajectories. Fine-tuned guardrail models are efficient but degrade on compound policies, and while LLM-as-a-judge approaches achieve higher accuracy, they incur high inference cost and tend to produce unstable predictions. Neurosymbolic methods offer a natural alternative, decomposing the policy into narrow neural decisions composed under explicit logic. However, existing symbolic layers lack the required expressiveness and are optimized against fixed data rather than the policy itself. To address this, we introduce SCOPE, which compiles a policy into an executable program of neural predicates executed by small models, deterministic extractors, and code that composes their outputs. The program is optimized jointly with its training data, using adversarial generation of counterexamples that target where the program deviates from the policy. We evaluate SCOPE on a benchmark of real-world policy tasks against prompt optimization, task decomposition, and workflow generation methods, achieving the highest average accuracy across tasks and models. Programs optimized with the weakest executor transfer to other models without further optimization. This allows them to be deployed in production on inexpensive models, where they reach accuracy comparable to frontier reasoning models. Unlike these monolithic models, SCOPE produces substantially more stable predictions, and each verdict is interpretable through the predicate outputs that produced it. Additionally, we validate the key components of our method through an ablation study. SCOPE thus enables reliable, interpretable, and efficient policy enforcement at production scale.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-23
The authors' main claim, as the classifier picked it out 1.00
“SCOPE thus enables reliable, interpretable, and efficient policy enforcement at production scale.”
How the main claim is stated: asserted 0.99
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.17 · human participants 0.04 · controlled comparison 0.85 · randomized assignment 0.02 · states quantitative result 0.08 · compares with prior results 0.97 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Policy enforcement for LLM agents has shifted from single-turn content moderation to compound, application-specific policies evaluated over multi-turn transcripts and tool-call trajectories. | background 1.00 | 0.06 | 0.02 | asserted 1.00 |
| Fine-tuned guardrail models are efficient but degrade on compound policies, and while LLM-as-a-judge approaches achieve higher accuracy, they incur high inference cost and tend to produce unstable predictions. | background 0.69 | 0.08 | 0.02 | asserted 1.00 |
| Neurosymbolic methods offer a natural alternative, decomposing the policy into narrow neural decisions composed under explicit logic. | background 0.98 | 0.06 | 0.04 | asserted 0.97 |
| However, existing symbolic layers lack the required expressiveness and are optimized against fixed data rather than the policy itself. | gap 0.97 | 0.06 | 0.02 | asserted 1.00 |
| To address this, we introduce SCOPE, which compiles a policy into an executable program of neural predicates executed by small models, deterministic extractors, and code that composes their outputs. | contribution 0.90 | 0.92 | 0.12 | asserted 1.00 |
| The program is optimized jointly with its training data, using adversarial generation of counterexamples that target where the program deviates from the policy. | method 0.98 | 0.92 | 0.04 | asserted 1.00 |
| We evaluate SCOPE on a benchmark of real-world policy tasks against prompt optimization, task decomposition, and workflow generation methods, achieving the highest average accuracy across tasks and models. | result 0.98 | 0.97 | 0.19 | asserted 1.00 |
| Programs optimized with the weakest executor transfer to other models without further optimization. | result 0.93 | 0.86 | 0.06 | asserted 0.99 |
| This allows them to be deployed in production on inexpensive models, where they reach accuracy comparable to frontier reasoning models. | result 0.63 | 0.79 | 0.12 | asserted 0.95 |
| Unlike these monolithic models, SCOPE produces substantially more stable predictions, and each verdict is interpretable through the predicate outputs that produced it. | result 0.96 | 0.88 | 0.17 | asserted 1.00 |
| Additionally, we validate the key components of our method through an ablation study. | contribution 0.48 | 0.95 | 0.03 | asserted 0.99 |
| SCOPE thus enables reliable, interpretable, and efficient policy enforcement at production scale. | interpretation 0.91 | 0.60 | 0.91 | asserted 0.99 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-23: 1 request, 8,813 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (4)
Figures are the authors', reproduced from the paper.
References (104)
- [1] L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab. GEPA: Reflective prompt evolution can outperform reinforcement learning. In Int... on Pith
- [2] R. Alur, R. Bodik, G. Juniwal, M. M. K. Martin, M. Raghothaman, S. A. Seshia, R. Singh, A. Solar-Lezama, E. Torlak, and A. Udupa. Syntax-guided synthesis. InFormal Methods in Computer-Aided Design (FMCAD), pages 1–8, 2013 on Pith
- [3] C. An, S. Bayless, S. Buliani, D. Cassel, B. Cook, D. Clough, R. Delmas, N. Diallo, F. Erata, N. Feng, D. Giannakopoulou, A. Goel, A. Gokhale, J. Hendrix, V . Heorhiadi, M. Hudak, D. Jovanovi´c, A. M. Kent, B. Kiesl-Reiter, J. J. Kuna, N. Labai, J. Lilien, D. Raghunathan, Z. R... on Pith
- [4] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39–48, 2016 on Pith
- [5] Anthropic. Building agents with the Claude Agent SDK. https://claude.com/blog/ building-agents-with-the-claude-agent-sdk, Sept. 2025 on Pith
- [6] Anthropic. Introducing Claude Opus 4.8. https://www.anthropic.com/news/ claude-opus-4-8, May 2026 on Pith
- [7] B. Atıl, S. Aykent, A. Chittams, L. Fu, R. J. Passonneau, E. Radcliffe, G. R. Rajagopal, A. Sloan, T. Tudrej, F. Ture, Z. Wu, L. Xu, and B. Baldwin. Non-determinism of “deterministic” LLM system settings in hosted environments. InProceedings of the 5th Workshop on Evaluation a... on Pith
- [8] M. Bartolo, A. Roberts, J. Welbl, S. Riedel, and P. Stenetorp. Beat the AI: Investigating adversarial human annotation for reading comprehension.Transactions of the Association for Computational Linguistics, 8:662–678, 2020 on Pith
- [9] Y . Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. InProceedings of the 26th Annual International Conference on Machine Learning (ICML), pages 41–48, 2009 on Pith
- [10] S. Chaudhuri, K. Ellis, O. Polozov, R. Singh, A. Solar-Lezama, and Y . Yue. Neurosymbolic programming.Foundations and Trends in Programming Languages, 7(3):158–243, 2021 on Pith
- [11] W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of Thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks.Transactions on Machine Learning Research, 2023 on Pith
- [12] Z. Chen, M. Kang, and B. Li. ShieldAgent: Shielding agents via verifiable safety policy rea- soning. InInternational Conference on Machine Learning (ICML), volume 267 ofProceedings of Machine Learning Research, pages 8313–8344, 2025 on Pith
- [13] C.-A. Cheng, A. Nie, and A. Swaminathan. Trace is the next AutoDiff: Generative optimization with rich feedback, execution traces, and LLMs. InAdvances in Neural Information Processing Systems (NeurIPS), volume 37, pages 71596–71642. Curran Associates, Inc., 2024 on Pith
- [14] Z. Cheng, T. Xie, P. Shi, C. Li, R. Nadkarni, Y . Hu, C. Xiong, D. Radev, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu. Binding language models in symbolic languages. In International Conference on Learning Representations (ICLR), 2023 on Pith
- [15] S. Cho, K. Chawla, P. Cai, Z. Liu, C. Zhu, S.-X. Zhang, and S. Sahu. Ask, don’t judge: Binary questions for interpretable LLM evaluation and self-improvement.arXiv preprint arXiv:2606.27226, 2026 on PithPith's reviewarXiv
- [16] G. Chua, S. Y . Chan, and S. Khoo. A flexible large language models guardrail development methodology applied to off-topic prompt detection.arXiv preprint arXiv:2411.12946, 2024 on PithPith's reviewarXiv
- [17] G. V . Datla, A. Vurity, T. Dash, T. Ahmad, M. Adnan, and S. Rafi. Executable governance for AI: Translating policies into rules using LLMs.arXiv preprint arXiv:2512.04408, 2025. 11 on PitharXiv
- [18] DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. https: //huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash, Sept. 2026 on Pith
- [19] D. Deshpande, S. S. Ravi, S. CH-Wang, B. Mielczarek, A. Kannappan, and R. Qian. GLIDER: Grading LLM interactions and decisions using explainable ranking.arXiv preprint arXiv:2412.14140, 2024 on PithPith's reviewarXiv
- [20] N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y . Lin, S. Welleck, P. West, C. Bhagavatula, R. Le Bras, J. D. Hwang, S. Sanyal, X. Ren, A. Ettinger, Z. Harchaoui, and Y . Choi. Faith and fate: Limits of transformers on compositionality. InAdvances in Neural Information Pro... on Pith
show 80 more
- [21] Y . Feng, S. Wang, Z. Cheng, Y . Wan, and D. Chen. Are we on the right way to assessing LLM-as-a-judge?arXiv preprint arXiv:2512.16041, 2025 on PitharXiv
- [22] L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y . Yang, J. Callan, and G. Neubig. PAL: Program-aided language models. InInternational Conference on Machine Learning (ICML), volume 202 ofProceedings of Machine Learning Research, pages 10764–10799. PMLR, 2023 on Pith
- [23] Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025 on PithPith's reviewarXiv
- [24] S. Ghosh, P. Varshney, M. N. Sreedhar, A. Padmakumar, T. Rebedea, J. R. Varghese, and C. Parisien. AEGIS2.0: A diverse AI safety dataset and risks taxonomy for alignment of LLM guardrails. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Assoc... on Pith
- [25] Google. Gemini 3.5 Flash: Gemini API model documentation. https://ai.google.dev/ gemini-api/docs/models/gemini-3.5-flash, May 2026 on Pith
- [26] Google DeepMind. Gemma-3-27B-it: Model card. https://huggingface.co/google/ gemma-3-27b-it, Mar. 2025 on Pith
- [27] Google DeepMind. Gemini 3.1 Pro: Model card. https://deepmind.google/models/ model-cards/gemini-3-1-pro/, Feb. 2026 on Pith
- [28] A. Graves, M. G. Bellemare, J. Menick, R. Munos, and K. Kavukcuoglu. Automated curricu- lum learning for neural networks. InProceedings of the 34th International Conference on Machine Learning (ICML), pages 1311–1320, 2017 on Pith
- [29] T. Gupta and A. Kembhavi. Visual programming: Compositional visual reasoning without training. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14953–14962, 2023 on Pith
- [30] R. Haldar and J. Hockenmaier. Rating roulette: Self-inconsistency in LLM-as-a-judge frame- works. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 24986–25004, Suzhou, China, 2025. Association for Computational Linguistics on Pith
- [31] S. Han, K. Rao, A. Ettinger, L. Jiang, B. Y . Lin, N. Lambert, Y . Choi, and N. Dziri. Wild- Guard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track,... on Pith
- [32] Z.-W. Hong, I. Shenfeld, T.-H. Wang, Y .-S. Chuang, A. Pareja, J. Glass, A. Srivastava, and P. Agrawal. Curiosity-driven red-teaming for large language models. InInternational Conference on Learning Representations (ICLR), pages 19935–19961, 2024 on Pith
- [33] M. Hoover, V . Baherwani, N. Jain, K. Saifullah, J. Vincent, C. Jain, M. Kazemi Rad, C. B. Bruss, A. Panda, and T. Goldstein. DynaGuard: A dynamic guardian model with user-defined policies. InInternational Conference on Learning Representations (ICLR), pages 87822– 87859, 2026 on Pith
- [34] S. Hu, C. Lu, and J. Clune. Automated design of agentic systems. InInternational Conference on Learning Representations (ICLR), pages 21344–21377, 2025 on Pith
- [35] W. Hua, X. Yang, M. Jin, Z. Li, W. Cheng, R. Tang, and Y . Zhang. TrustAgent: Towards safe and trustworthy LLM-based agents. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 10000–10016, Miami, Florida, USA, 2024. Association for Computational Lin... on Pith
- [36] T.-H. Huang, H. Vishwakarma, and F. Sala. Time to impeach LLM-as-a-judge: Programs are the future of evaluation.arXiv preprint arXiv:2506.10403, 2025 on PithPith's reviewarXiv
- [37] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y . Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, and M. Khabsa. Llama Guard: LLM-based input-output safeguard for human- AI conversations.arXiv preprint arXiv:2312.06674, 2023 on PithPith's reviewarXiv
- [38] M. Kang and B. Li. R2-Guard: Robust reasoning enabled LLM guardrail via knowledge- enhanced logical reasoning. InInternational Conference on Learning Representations (ICLR), pages 63859–63876, 2025 on Pith
- [39] O. Khattab, K. Santhanam, X. L. Li, D. Hall, P. Liang, C. Potts, and M. Zaharia. Demonstrate- search-predict: Composing retrieval and language models for knowledge-intensive NLP.arXiv preprint arXiv:2212.14024, 2022 on PithPith's reviewarXiv
- [40] O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts. DSPy: Compiling declarative language model calls into state-of-the-art pipelines. InInternational Conference on Learni... on Pith
- [41] T. Khot, H. Trivedi, M. Finlayson, Y . Fu, K. Richardson, P. Clark, and A. Sabharwal. Decom- posed Prompting: A modular approach for solving complex tasks. InInternational Conference on Learning Representations (ICLR), 2023 on Pith
- [42] D. Kiela, M. Bartolo, Y . Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams. Dynabench: Rethinking benchmarking in NLP. InProceedings of the 2021 Co... on Pith
- [43] S. Kim, J. Shin, Y . Cho, J. Jang, S. Longpre, H. Lee, S. Yun, S. Shin, S. Kim, J. Thorne, and M. Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In International Conference on Learning Representations (ICLR), pages 29927–29962, 2024 on Pith
- [44] A. Kumar, D. Kumar, J. Loya, N. A. Birur, T. Baswa, S. Agarwal, and P. Harshangi. SAGE-RT: Synthetic alignment data generation for safety evaluation and red teaming.arXiv preprint arXiv:2408.11851, 2024 on PithPith's reviewarXiv
- [45] M. P. Kumar, B. Packer, and D. Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems (NeurIPS), volume 23, pages 1189–1197, 2010 on Pith
- [46] Y . Lee, J. Kim, J. Kim, H. Cho, J. Kang, P. Kang, and N. Kim. CheckEval: A reliable LLM-as-a-judge framework for evaluating text generation using checklists. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 15771–15798, 2025 on Pith
- [47] E. Levi, E. Brosh, and M. Friedmann. Intent-based prompt calibration: Enhancing prompt optimization with synthetic boundary cases.arXiv preprint arXiv:2402.03099, 2024 on PithPith's reviewarXiv
- [48] C. Li, J. Liang, A. Zeng, X. Chen, K. Hausman, D. Sadigh, S. Levine, L. Fei-Fei, F. Xia, and B. Ichter. Chain of Code: Reasoning with a language model-augmented code emulator. InInternational Conference on Machine Learning (ICML), volume 235 ofProceedings of Machine Learning R... on Pith
- [49] Z. Li, J. Huang, and M. Naik. Scallop: A language for neurosymbolic programming.Proceed- ings of the ACM on Programming Languages, 7(PLDI):1463–1487, 2023 on Pith
- [50] Y . Liu, H. Gao, S. Zhai, Y . He, J. Xia, Z. Hu, Y . Chen, X. Yang, J. Zhang, S. Z. Li, H. Xiong, and B. Hooi. GuardReasoner: Towards reasoning-based LLM safeguards.arXiv preprint arXiv:2501.18492, 2025 on PitharXiv
- [51] Y . Liu, D. Iter, Y . Xu, S. Wang, R. Xu, and C. Zhu. G-Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2511–2522, 2023 on Pith
- [52] R. Manhaeve, S. Dumanˇci´c, A. Kimmig, T. Demeester, and L. De Raedt. DeepProbLog: Neural probabilistic logic programming. InAdvances in Neural Information Processing Systems (NeurIPS), volume 31, 2018. 13 on Pith
- [53] T. Matiisen, A. Oliver, T. Cohen, and J. Schulman. Teacher–student curriculum learning.IEEE Transactions on Neural Networks and Learning Systems, 31(9):3732–3740, 2020 on Pith
- [54] A. Mazza and E. Levi. BARRED: Synthetic training of custom policy guardrails via asymmet- ric debate. InProceedings of the 43rd International Conference on Machine Learning (ICML), volume 306 ofProceedings of Machine Learning Research, pages 87186–87203, 2026 on Pith
- [55] G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y . LeCun, and T. Scialom. GAIA: A benchmark for general AI assistants. InInternational Conference on Learning Representations (ICLR), pages 9025–9049, 2024 on Pith
- [56] L. Miculicich, M. Parmar, H. Palangi, K. D. Dvijotham, M. Montanari, T. Pfister, and L. T. Le. VeriGuard: Enhancing LLM agent safety via verified code generation.arXiv preprint arXiv:2510.05156, 2025 on PitharXiv
- [57] I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar. GSM- Symbolic: Understanding the limitations of mathematical reasoning in large language models. InInternational Conference on Learning Representations (ICLR), pages 94743–94765, 2025 on Pith
- [58] T. Mu, A. Helyar, J. Heidecke, J. Achiam, A. Vallone, I. Kivlichan, M. Lin, A. Beutel, J. Schulman, and L. Weng. Rule based rewards for language model safety. InAdvances in Neural Information Processing Systems (NeurIPS), volume 37, pages 108877–108901, 2024 on Pith
- [59] Y . Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial NLI: A new benchmark for natural language understanding. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4885–4901, 2020 on Pith
- [60] C. Nobata, J. Tetreault, A. Thomas, Y . Mehdad, and Y . Chang. Abusive language detection in online user content. InProceedings of the 25th International Conference on World Wide Web (WWW), pages 145–153, 2016 on Pith
- [61] NVIDIA. Nemotron-3.5-Lightning: Model card. https://huggingface.co/nvidia/ NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, Aug. 2026 on Pith
- [62] OpenAI. Introducing gpt-oss-safeguard. https://openai.com/index/ introducing-gpt-oss-safeguard/, Oct. 2025 on Pith
- [63] OpenAI. GPT-5.4 mini and GPT-5.4 nano: OpenAI API model documentation. https://developers.openai.com/api/docs/models/gpt-5.4-mini and https:// developers.openai.com/api/docs/models/gpt-5.4-nano, Mar. 2026 on Pith
- [64] OpenAI. GPT-5.5: OpenAI API model documentation. https://developers.openai. com/api/docs/models/gpt-5.5, Apr. 2026 on Pith
- [65] K. Opsahl-Ong, M. J. Ryan, J. Purtell, D. Broman, C. Potts, M. Zaharia, and O. Khattab. Optimizing instructions and demonstrations for multi-stage language model programs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages ... on Pith
- [66] I. Padhi, M. Nagireddy, G. Cornacchia, S. Chaudhury, T. Pedapati, P. Dognin, K. Murugesan, E. Miehling, M. Santillán Cooper, K. Fraser, G. Zizzo, M. Z. Hameed, M. Purcell, M. Desmond, Q. Pan, I. Vejsbjerg, E. M. Daly, M. Hind, W. Geyer, A. Rawat, K. R. Varshney, and P. Sattige... on Pith
- [67] K. Palla, J. L. Redondo García, C. Hauff, F. Fabbri, A. Damianou, H. Lindström, D. Taber, and M. Lalmas. Policy-as-prompt: Rethinking content moderation in the age of large language models. InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency ... on Pith
- [68] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3419–3448, 2022 on Pith
- [69] R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y . Oudeyer. Automatic curriculum learning for deep RL: A short survey. InProceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), pages 4819–4825, 2020. 14 on Pith
- [70] A. Prasad, A. Koller, M. Hartmann, P. Clark, A. Sabharwal, M. Bansal, and T. Khot. ADaPT: As-needed decomposition and planning with language models. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 4226–4252, 2024 on Pith
- [71] R. Pryzant, D. Iter, J. Li, Y . T. Lee, C. Zhu, and M. Zeng. Automatic prompt optimization with “gradient descent” and beam search. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7957–7968, Singapore, 2023. Association f... on Pith
- [72] Qwen Team. Qwen3-235B-A22B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507: Model cards. https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507 and https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507, July 2025 on Pith
- [73] T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen. NeMo Guardrails: A toolkit for controllable and safe LLM applications with programmable rails. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages... on Pith
- [74] A. Roucher, A. Villanova del Moral, T. Wolf, L. von Werra, and E. Kaunismäki.smolagents: a smol library to build great agentic systems. https://github.com/huggingface/ smolagents, 2025 on Pith
- [75] M. Samvelyan, S. C. Raparthy, A. Lupu, E. Hambro, A. H. Markosyan, M. Bhatt, Y . Mao, M. Jiang, J. Parker-Holder, J. Foerster, T. Rocktäschel, and R. Raileanu. Rainbow Teaming: Open-ended generation of diverse adversarial prompts. InAdvances in Neural Information Processing Sy... on Pith
- [76] K. Schroeder and Z. Wood-Doughty. Can you trust LLM judgments? reliability of LLM-as-a- judge.arXiv preprint arXiv:2412.12509, 2024 on PithPith's reviewarXiv
- [77] A. Shah, E. Zhan, J. J. Sun, A. Verma, Y . Yue, and S. Chaudhuri. Learning differentiable programs with admissible neural heuristics. InAdvances in Neural Information Processing Systems (NeurIPS), volume 33, pages 4940–4952, 2020 on Pith
- [78] M. Sharma, M. Tong, J. Mu, J. Wei, J. Kruthoff, S. Goodfriend, E. Ong, A. Peng, R. Agarwal, C. Anil, A. Askell, N. Bailey, J. Benton, E. Bluemke, S. R. Bowman, E. Christiansen, H. Cun- ningham, A. Dau, A. Gopal, R. Gilson, et al. Constitutional classifiers: Defending against u... on PithPith's reviewarXiv
- [79] P. Shojaee, I. Mirzadeh, K. Alizadeh, M. Horton, S. Bengio, and M. Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. InAdvances in Neural Information Processing Systems (NeurIPS), volume 38... on Pith
- [80] A. Solar-Lezama. Program sketching.International Journal on Software Tools for Technology Transfer, 15(5–6):475–495, 2013 on Pith
- [81] A. Solar-Lezama, L. Tancau, R. Bodik, S. Seshia, and V . Saraswat. Combinatorial sketching for finite programs. InProceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pages 404–415, 2006 on Pith
- [82] E. Spertus. Smokey: Automatic recognition of hostile messages. InProceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Conference on Innovative Applications of Artificial Intelligence (AAAI/IAAI), pages 1058–1065, 1997 on Pith
- [83] S. Subramanian, M. Narasimhan, K. Khangaonkar, K. Yang, A. Nagrani, C. Schmid, A. Zeng, T. Darrell, and D. Klein. Modular visual question answering via code generation. InProceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short... on Pith
- [84] D. Surís, S. Menon, and C. V ondrick. ViperGPT: Visual inference via Python execution for reasoning. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11888–11898, 2023 on Pith
- [85] L. Valkov, D. Chaudhari, A. Srivastava, C. Sutton, and S. Chaudhuri. HOUDINI: Life- long learning as program synthesis. InAdvances in Neural Information Processing Systems (NeurIPS), volume 31, 2018. 15 on Pith
- [86] P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y . Cao, L. Kong, Q. Liu, T. Liu, and Z. Sui. Large language models are not fair evaluators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440–9450, 2024 on Pith
- [87] R. Wang, S. Si, F. Yu, D. Wiesmann, C.-J. Hsieh, and I. Dhillon. Large language models are interpretable learners. InInternational Conference on Learning Representations (ICLR), pages 54829–54852, 2025 on Pith
- [88] X. Wang, C. Li, Z. Wang, F. Bai, H. Luo, J. Zhang, N. Jojic, E. P. Xing, and Z. Hu. PromptA- gent: Strategic planning with language models enables expert-level prompt optimization. In International Conference on Learning Representations (ICLR), pages 23967–24001, 2024 on Pith
- [89] Z. Xiang, L. Zheng, Y . Li, J. Hong, Q. Li, H. Xie, J. Zhang, Z. Xiong, C. Xie, C. Yang, D. Song, and B. Li. GuardAgent: Safeguard LLM agents via knowledge-enabled reasoning. InInternational Conference on Machine Learning (ICML), volume 267 ofProceedings of Machine Learning Re... on Pith
- [90] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025 on PithPith's reviewarXiv
- [91] C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen. Large language models as optimizers. InInternational Conference on Learning Representations (ICLR), pages 12028– 12068, 2024 on Pith
- [92] Y . Yang, Y . Jiang, Q. Wang, Y . Tan, X. Zhu, S. S. M. Chow, B. Zheng, and X. Yue. QuadSen- tinel: Sequent safety for machine-checkable control in multi-agent systems.arXiv preprint arXiv:2512.16279, 2025 on PitharXiv
- [93] Y . Yang, H. Xu, Z. Hu, and Y . Yue. RLIE: Rule generation with logistic regression, iterative refinement, and evaluation for large language models. InInternational Conference on Machine Learning (ICML), volume 306 ofProceedings of Machine Learning Research, pages 146445– 146462, 2026 on Pith
- [94] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. InInternational Conference on Learning Representations (ICLR), pages 9965–10017, 2025 on Pith
- [95] J. Ye, J. Gao, Q. Li, H. Xu, J. Feng, Z. Wu, T. Yu, and L. Kong. ZeroGen: Efficient zero-shot learning via dataset generation. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 11653–11669, Abu Dhabi, United Arab Emirates, ... on Pith
- [96] J. Ye, Y . Wang, Y . Huang, D. Chen, Q. Zhang, N. Moniz, T. Gao, W. Geyer, C. Huang, P.-Y . Chen, N. Chawla, and X. Zhang. Justice or prejudice? quantifying biases in LLM-as-a-judge. InInternational Conference on Learning Representations (ICLR), pages 102351–102390, 2025 on Pith
- [97] M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, P. Lu, Z. Huang, C. Guestrin, and J. Zou. Optimiz- ing generative AI by backpropagating language model feedback.Nature, 639(8055):609–616, 2025 on Pith
- [98] W. Zeng, Y . Liu, R. Mullins, L. Peran, J. Fernandez, H. Harkous, K. Narasimhan, D. Proud, P. Kumar, B. Radharapu, O. Sturman, and O. Wahltinez. ShieldGemma: Generative AI content moderation based on Gemma.arXiv preprint arXiv:2407.21772, 2024 on PithPith's reviewarXiv
- [99] J. Zhang, J. Xiang, Z. Yu, F. Teng, X. Chen, J. Chen, M. Zhuge, X. Cheng, S. Hong, J. Wang, B. Zheng, B. Liu, Y . Luo, and C. Wu. AFlow: Automating agentic workflow generation. In International Conference on Learning Representations (ICLR), pages 34040–34077, 2025 on Pith
- [100] A. Zheng, M. Rana, and A. Stolcke. Lightweight safety guardrails using fine-tuned BERT em- beddings. InProceedings of the 31st International Conference on Computational Linguistics: Industry Track (COLING), pages 689–696, 2025 on Pith
The first 100 of 104 references are listed here. All of them are in this paper's citation record.
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment