Artificial Intelligence · arXiv:2609.01345 v2 · submitted 1 Sep 2026 · 26 references
Cheap Verifiers, Large Blind Spots: Measuring the Reliability Cost of Cost-Saving Cascades
The paper says
“Inference cascades cut cost by answering most queries with a cheap model and escalating a hard tail to a frontier model that acts as verifier.”
more of the abstract
A natural extension closes the loop: fine-tune the cheap student on the verifier's rejections so the escalation rate, and cost, fall each round. We measure this loop on real LLMs and report four findings. First, the verifier's blind spot, the fraction of the student's wrong answers it accepts, is large and moves adversarially: it grows with student capability (β from 0.12 to 0.55 as the student scales 0.5B to 32B) and shrinks with verifier capability, so it is worst in the cheap-student, cheap-verifier regime cascades exist to create. Second, buying it away returns the saving: a frontier verifier drives β to about 0.05 but then escalates on 46% of hard-MATH queries against a 39% true error rate, paying the frontier price on nearly half of all traffic. Third, naive corrective fine-tuning on the verifier-rejected tail does not improve the small student but degrades and ultimately collapses it, across every teacher we tried (cross-family and same-family), so at this scale the self-improving loop is self-defeating. Fourth, through all of this the cascade's own dashboard, every metric computed through the verifier, reads a flat 3% error while true delivered error swings up to 32%: the system is blind to its own degradation by construction. We then give the theory that explains the blindness, a two-population conservation law, ε∞ ≲ q0β0, under which every in-loop metric improves while true quality does not, and a synthetic study that validates the mechanism. The practical conclusion: the reliability of a self-improving cascade cannot be read from any metric computed through its own verifier.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 8 Sep 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Well-measured, honest paper: trust the empirical blind-spot and dashboard-blindness findings; treat the conservation law as a stylized bound, not a proven universal. This line is wrong
2 major objections · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 26 references say.
The paper says, at §6, Appendix A, §7, §9
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The conservation law (Proposition 1) and the synthetic validation depend critically on Assumption A1: that the verifier's blind region is fixed and input-space-based. The paper acknowledges this as the least realistic assumption (§9) but does not explore how output-dependent or moving blind spots could alter the floor.
more of objection 1
If blind spots shift adversarially with the student's error distribution, the floor could weaken (more self-healing) or strengthen (harder to escape). Since the synthetic study only tests the static case, the generality of the conservation law for language-model cascades is uncertain. The authors should add a discussion of plausible alternative behaviors under output-dependent blind spots, perhaps with a simple model or argument, and clarify which of their claims would survive if A1 is violated.
The paper says, at §5, §7
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The real-model loop degrades the student, so the conservation floor is not directly observed on LLMs. The synthetic study provides the needed validation, but it uses a highly stylized task (10-way classification on random ReLU features with a logistic student) that abstracts away the sequential, open-ended nature of language generation.
more of objection 2
The gap between the synthetic and real settings is large. The paper should more explicitly discuss the transferability of the synthetic quantitative results (e.g., the size of self-healing, the mitigation exchange rates) to the LLM domain, beyond noting that they are 'not portable constants' (§9).
7 minor objections
The paper says, at §1, Figure 2
Pith says: minor objection 1 of 7
Figure 2: The frozen control line is dashed but hard to distinguish from the solid raw-error line; consider using a different color or markers.
The paper says, at §6
Pith says: minor objection 2 of 7
Page 9: the text says 'the accepted-and-wrong error mass q_t β_t is conserved across rounds' but Proposition 1 shows it is conserved only in the frozen case; corrective allows spillover. Rephrase to avoid confusion early on.
show 5 more
The paper says, at §5
Pith says: minor objection 3 of 7
The claim 'The loop degrades the student — every teacher we tried' is based on two student sizes (1.5B, 7B) and a limited set of teachers. Consider softening to 'every teacher we tried in our experiments' to avoid overgeneralisation.
The paper says, at §1, §4
Pith says: minor objection 4 of 7
The term 'dashboard' is used extensively but first defined only implicitly. Add an explicit definition early, e.g., 'the set of metrics a practitioner computes through the verifier, including escalation rate and accept-stream estimated accuracy'.
The paper says, at Appendix A
Pith says: minor objection 5 of 7
The linear model assumes τ_t → 0 (teacher error negligible). On hard tasks where the teacher itself may have non-trivial error, the floor expression may need a correction term. Acknowledge this bound.
The paper says, at §4
Pith says: minor objection 6 of 7
The wind-tunnel method uses exact-match on GSM8K as oracle. For open-ended tasks, such a cheap oracle does not exist; the paper acknowledges this but could note that the measurement of β via a super-verifier (mentioned as future work) is likely to introduce its own noise.
The paper says, at §7
Pith says: minor objection 7 of 7
The synthetic verifier's blind region is a fixed input-space half-space. This is a strong abstraction; real verifier blind spots arise from task-specific reasoning errors. A brief justification of why this abstraction still tests the conservation mechanism would help.
What would settle it
Modify the synthetic study (Section 7) to use an output-dependent blind region: define the verifier's acceptance as a function of the student's output embedding (e.g., accept if the embedding distance to the correct answer exceeds a threshold, with the threshold initialized to match the static-blind-region β0 and then held fixed). Run 20 seeds at ρ=0.7 (and other β0 values) for both corrective and self-training loops. Measure the final user-facing floor ε_T. If the floor drops substantially below the static-baseline 0.249 (e.g., below 0.15), then the static assumption is critical; if it remains within 0.02 of the static value, the mechanism is robust to output dependence.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 8 Sep 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 2 major objections, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Two things you should know. First, this paper contains genuinely useful, honest real-model measurements of verifier blind spots in LLM cascades: the wind-tunnel design (oracle held in reserve, verifier blinded to it) is what cascade evaluation should be doing, and it cleanly shows the blind-spot rate rising with student capability and falling with verifier capability. Second, the headline theoretical claim—the conservation floor eps_inf <= q0*beta0—rests on an assumption the authors themselves flag as least realistic: a static blind region. Real verifier blind spots are output-dependent and can move as the student's error distribution shifts. If they move adversarially, the floor weakens; if the student self-heals, the floor may not hold at all. The paper says exactly this in Limitations, and the synthetic study only tests the static case, so read the law as a bound from a stylized model, not a measured constant.
What it does well: the empirical core does not depend on that theory. Dashboard blindness—flat ~3% verifier-estimated error while true delivered error swings to 32%—is directly measured and survives any assumption about blind-spot dynamics. The finding that naive corrective fine-tuning degrades and collapses the small student across every teacher tried is reported as a negative result, not papered over. Error bars, pinned verifier, cumulative retraining from the base model, and the decomposition of delivered error into accepted-wrong and teacher-error terms are all careful. Code and data are public, and the synthetic scripts regenerate every figure in CPU-minutes. The H1–H5 predictions with explicit nulls are a good template for empirical work.
Soft spots, in proportion: the real-model loop is single-seed, so the per-round trajectory is not robust—the authors say this, and the level-gap is large enough that the finding survives. The synthetic floor is partly baked in by censoring blind-region items from the training pool; the informative results are the deviations from the anchor, and the paper reads them that way. H3's predicted absolute rise under self-training was not observed; they correct it honestly. None of these sink the paper; they are properly labeled.
Who it is for: anyone building or monitoring cascades, LLM-as-judge researchers, and people designing self-improving loops. Send it to peer review. The empirical core deserves referee time; the theory should be framed as a sufficient mechanism rather than a general necessity.
The strongest objection
The conservation law (Proposition 1) and the synthetic validation (Section 7) both assume a fixed, input-space blind region (Assumption A1). In real language-model cascades, the verifier's blind spot is output-dependent: what it accepts depends on the student's generated text, not on a static input subspace. As the student changes under training, the distribution of errors shifts, potentially making previously undetectable errors detectable (or vice versa). This could allow substantial self-healing (spillover γ) that reduces the floor far below q0β0, or could even cause the blind spot to shrink over rounds, weakening the dashboard-blindness explanation. The paper acknowledges this as the least realistic assumption, but it is the most load-bearing for the theoretical claim that the floor is structurally anchored at q0β0. The real-model measurement of dashboard blindness (Figure 2) does not depend on this assumption and remains robust; the theoretical account of why the dashboard is blind, however, would lose generality if the static assumption is critical. The synthetic study tests only the static case, so we do not know whether the conservation mechanism is robust to output-dependent blind spots.
The reader's verdict (ACCEPT) is well-supported by the direct real-model measurements, which show a large, persistent gap between the verifier-estimated dashboard and the true error (Figure 2). This gap is a measured fact, not a theoretical deduction, so it does not depend on the static blind region assumption. The conservation law and the synthetic study are presented as an explanation for that gap, and the paper transparently flags the static assumption as a limitation. The stress-test concern therefore applies to the explanatory theory, not to the core empirical result. The proposed concrete test would determine whether the static assumption is critical for the theoretical mechanism. If the floor weakens substantially under output-dependent blind spots, the theory would need revision, but the real-model result (dashboard blindness) would still stand. Hence the verdict should remain UNCHANGED: the paper's main claims are not invalidated by this concern, and the limitation is already acknowledged. The stress-test here reinforces the reader's assessment that the static blind region is the weakest link and provides a way to tighten it.
The referee's report
The paper makes a practically important contribution by demonstrating empirically that the dashboard of an inference cascade is blind to degradation, and that corrective fine-tuning can backfire. The wind-tunnel method for measuring blind spots using math problems with an oracle is a clean methodological contribution. The conservation-law account provides a theoretical framework that organises several related phenomena (reward overoptimization, self-consuming loops, pseudo-labeling). The findings are directly relevant to practitioners building cost-saving systems and to the broader LLM-as-judge literature. The paper is also refreshingly honest about its limitations and negative results.
Pith's reading of the paper (a machine's summary, not the authors')
Self-improving inference cascades hide their true quality degradation inside the verifier's blind spot. The paper measures this blind spot—the fraction of student errors the verifier accepts—on real LLMs and finds it grows with student capability (from 0.12 to 0.55 as the student scales 0.5B to 32B) and shrinks with verifier capability, so it is worst in the cheap-student, cheap-verifier regime cascades exist to create. Corrective fine-tuning (training only on verifier-rejected items) does not improve the student but degrades and collapses it across every teacher tried. Most alarmingly, every metric computed through the verifier—the only dashboard a practitioner sees—reads a flat ~3% error while true delivered error swings to 32%. The paper derives a conservation law: user-facing error asymptotes to a floor bounded by the initial blind-spot mass $q_0\beta_0$, and that floor is invisible to the verifier's own metrics. The practical conclusion: the reliability of a self-improving cascade cannot be read from any metric computed through its own verifier; independent audits are required.
What carries the argument. The key mechanism is a two-population decomposition of student errors: detectable errors (mass $q_t r_t$, where the verifier catches them) and blind errors ($q_t\beta_t$, where the verifier waves them through). The loop provides training signal only on detectable errors (rejected items); blind errors never receive targeted correction, though generalization spillover can reduce them. This creates a conservation law: user-facing error $\varepsilon_\infty \lesssim q_0\beta_0$, the initial blind error mass. Every in-loop metric—escalation rate, verifier-estimated accept-stream accuracy—is computed through the verifier and therefore reflects only the detectable error dynamics, masking the persisting blind mass. The verifier’s own blind region is the instrument that creates the illusion of improvement.
Core claim. The paper’s central discovery is that the verifier’s blind-spot rate $\beta$ (the fraction of wrong student answers the verifier accepts) is large, moves adversarially, and is completely hidden from every in-loop metric. Measurements on real LLMs (Qwen2.5 students 0.5B–32B, OpenAI verifiers on GSM8K and hard MATH) show: $\beta$ rises from 0.12 to 0.55 as the student scales (fixed verifier); a frontier verifier crushes $\beta$ to ~0.05 but then escalates on 46% of queries against a 39% true error rate, giving back the cost saving; corrective fine-tuning degrades and collapses the student across all teachers tried (cross-family and same-family); and throughout, the verifier-estimated error of the delivered stream reads a flat ~3% while gold error swings to 32%. The paper then gives a two-population model (detectable vs. blind errors) that predicts a conservation law $\varepsilon_\infty \lesssim q_0\beta_0$ under idealised assumptions, and validates it in a synthetic setup where the loop genuinely improves the student. The blunt empirical fact is that the dashboard is blind by construction.
The premise it rests on. The verifier’s blind region is assumed fixed and output-independent (Assumption A1 in Appendix A); in reality, verifier blind spots depend on the output and move with the student’s error distribution, which could weaken the conservation floor.
If the paper is right (Pith's inferences, not the authors')
Systems using self-improving cascades must deploy external audits independent of the verifier to measure true reliability, because the cascade’s own dashboard is structurally blind.
Verifier choice trades reliability for cost on an invisible axis: cheaper verifiers yield larger blind spots and a higher hidden error floor, but buying the blind spot away with a frontier verifier gives back the cost saving through over-escalation.
Between loop variants, corrective fine-tuning (training only on rejects) is safer than self-training (which recycles accepts as labels), because self-training actively reinforces blind-spot errors and raises the floor.
Verifier decorrelation (ensembles of verifiers with independent blind regions) can suppress the floor but increases per-query verification cost; random oracle audits are cheap but weak because most of the budget lands on non-blind items.
The error floor is not a theoretical curiosity: even a loop that genuinely improves the student leaves a positive residual error that no in-loop metric can detect, so the illusion of complete reliability is unreachable without external gold labels.
The blind-spot mechanism likely transfers to any setting where a generative policy is optimized against an imperfect proxy evaluator using feedback from that same proxy, including reinforcement learning from AI feedback (RLAIF) and bootstrapped self-training loops.
The severity of the hidden floor may be amplified when the verifier and student share pretraining data or side, because shared failure modes increase $\beta_0$ beyond what capability alone predicts; this suggests that cross-family verifiers are not merely a cost consideration but a reliability necessity.
The paper’s synthetic model uses a static input-space blind region; a softening or output-dependent blind region could weaken conservation, but then the verifier’s own confidence scores could be used to target audits, potentially breaking the floor with cheaper budgets than the ensemble approach.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Inference cascades cut cost by answering most queries with a cheap model and escalating a hard tail to a frontier model that acts as verifier. A natural extension closes the loop: fine-tune the cheap student on the verifier's rejections so the escalation rate, and cost, fall each round. We measure this loop on real LLMs and report four findings. First, the verifier's blind spot, the fraction of the student's wrong answers it accepts, is large and moves adversarially: it grows with student capability (β from 0.12 to 0.55 as the student scales 0.5B to 32B) and shrinks with verifier capability, so it is worst in the cheap-student, cheap-verifier regime cascades exist to create. Second, buying it away returns the saving: a frontier verifier drives β to about 0.05 but then escalates on 46% of hard-MATH queries against a 39% true error rate, paying the frontier price on nearly half of all traffic. Third, naive corrective fine-tuning on the verifier-rejected tail does not improve the small student but degrades and ultimately collapses it, across every teacher we tried (cross-family and same-family), so at this scale the self-improving loop is self-defeating. Fourth, through all of this the cascade's own dashboard, every metric computed through the verifier, reads a flat 3% error while true delivered error swings up to 32%: the system is blind to its own degradation by construction. We then give the theory that explains the blindness, a two-population conservation law, ε∞ ≲ q0β0, under which every in-loop metric improves while true quality does not, and a synthetic study that validates the mechanism. The practical conclusion: the reliability of a self-improving cascade cannot be read from any metric computed through its own verifier.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 1.00
“The practical conclusion: the reliability of a self-improving cascade cannot be read from any metric computed through its own verifier.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.62 · human participants 0.03 · controlled comparison 0.46 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.51 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Inference cascades cut cost by answering most queries with a cheap model and escalating a hard tail to a frontier model that acts as verifier. | background 0.99 | 0.07 | 0.02 | asserted 1.00 |
| A natural extension closes the loop: fine-tune the cheap student on the verifier's rejections so the escalation rate, and cost, fall each round. | background 0.63 | 0.11 | 0.02 | asserted 0.73 |
| We measure this loop on real LLMs and report four findings. | contribution 0.95 | 0.93 | 0.02 | asserted 0.98 |
| First, the verifier's blind spot, the fraction of the student's wrong answers it accepts, is large and moves adversarially: it grows with student capability ($\beta$ from 0.12 to 0.55 as the student scales 0.5B to 32B) and shrinks with verifier capability, so it is worst in the cheap-student, cheap-verifier regime cascades exist to create. | result 1.00 | 0.95 | 0.07 | asserted 1.00 |
| Second, buying it away returns the saving: a frontier verifier drives $\beta$ to about 0.05 but then escalates on 46% of hard-MATH queries against a 39% true error rate, paying the frontier price on nearly half of all traffic. | result 1.00 | 0.94 | 0.05 | asserted 1.00 |
| Third, naive corrective fine-tuning on the verifier-rejected tail does not improve the small student but degrades and ultimately collapses it, across every teacher we tried (cross-family and same-family), so at this scale the self-improving loop is self-defeating. | result 1.00 | 0.96 | 0.07 | asserted 1.00 |
| Fourth, through all of this the cascade's own dashboard, every metric computed through the verifier, reads a flat 3% error while true delivered error swings up to 32%: the system is blind to its own degradation by construction. | result 1.00 | 0.94 | 0.21 | asserted 1.00 |
| We then give the theory that explains the blindness, a two-population conservation law, $\epsilon_\infty \lesssim q_0 \beta_0$, under which every in-loop metric improves while true quality does not, and a synthetic study that validates the mechanism. | contribution 0.83 | 0.95 | 0.07 | asserted 0.98 |
| The practical conclusion: the reliability of a self-improving cascade cannot be read from any metric computed through its own verifier. | interpretation 0.99 | 0.85 | 0.96 | asserted 1.00 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,330 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Figures are the authors', reproduced from the paper.
References (26)
- [1] Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv:2305.05176, 2023 on PithPith's reviewarXiv
- [2] David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gon- tijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-Dickstein, Kevin Murphy, and Charles Sutton. Language model cascades.arXiv:2207.10342, 2022 on PithPith's reviewarXiv
- [3] Aman Madaan, Pranjal Aggarwal, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Mausam, and Manaal Faruqui. Automix: Automatically mixing language models. InAdvances in Neural ... on Pith
- [4] Yu Wu, Shuo Wu, Ye Tao, Yansong Li, and Anand D. Sarwate. From deferral to learning: Online in-context knowledge distillation for llm cascades.arXiv:2509.22984, 2025 on PitharXiv
- [5] Vishnu Sarukkai, Asanshay Gupta, James Hong, Micha¨ el Gharbi, and Kayvon Fatahalian. In- context distillation with self-consistency cascades: A simple, training-free way to reduce llm agent costs.arXiv:2512.02543, 2025 on PithPith's reviewarXiv
- [6] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InInternational Conference on Machine Learning (ICML), 2023 on Pith
- [7] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv:2302.01318, 2023 on PithPith's reviewarXiv
- [8] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. InInternational Conference on Machine Learning (ICML), 2023 on Pith
- [9] Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. InAdvances in Neural Information Processing Systems (NeurIPS), 2022 on Pith
- [10] David Manheim and Scott Garrabrant. Categorizing variants of goodhart’s law. arXiv:1803.04585, 2018 on PithPith's reviewarXiv
- [11] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. Routellm: Learning to route llms with preference data. arXiv:2406.18665, 2024 on PithPith's reviewarXiv
- [12] Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling.arXiv:2308.08998, 2023 on PithPith's reviewarXiv
- [13] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. 18 on Pith
- [14] Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold As- chenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv:231... on PithPith's reviewarXiv
- [15] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. InInternational Conference on Learning Representations (ICLR), 2023 on Pith
- [16] Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. Inference scaling flaws: The limits of llm resampling with imperfect verifiers.arXiv:2411.17501, 2024 on PitharXiv
- [17] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InAdvances in Neural Information Processi... on Pith
- [18] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. InAdvances in Neural Information Processing Systems (NeurIPS), 2024 on Pith
- [19] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data.Nature, 631:755–759, 2024 on Pith
- [20] Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go mad. InInternational Conference on Learning Representations (ICLR), 2024 on Pith
show 6 more
- [21] Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor, and Kevin McGuinness. Pseudo- labeling and confirmation bias in deep semi-supervised learning. InInternational Joint Con- ference on Neural Networks (IJCNN), 2020 on Pith
- [22] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. InInterna- tional Conference on Learning Representations (ICLR), 2024 on Pith
- [23] Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification.Journal of Machine Learning Research, 11:1605–1641, 2010 on Pith
- [24] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. InInternational Conference on Machine Learning (ICML), 2020 on Pith
- [25] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. InAdvances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021 on Pith
- [26] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv:2110.14168, 2021. 19 on PithPith's reviewarXiv
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment