Machine Learning · arXiv:2610.23255 v1 · submitted 16 Oct 2026 · 23 references
Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers
The paper says
“A common way to make language models and agents more accurate is to spend more compute at inference time: generate many candidate solutions and submit the one a verifier prefers.”
more of the abstract
Success then requires that some candidate is correct and that the verifier picks it. The first grows predictably with the number of samples; the second does not, because real verifiers (unit tests, majority votes, LLM judges, reward models) accept wrong answers, reject right ones, and make correlated mistakes. With a fixed budget, a practitioner must decide how many samples to draw, which models to draw them from, and how much to spend on verification, and today these choices are made by rules of thumb. We introduce SPLIT, which makes them from a small labelled pilot. It rests on an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection that accounts for varying task difficulty, blind spots shared across models, and correlated repeated verifier calls. SPLIT fits the formula by empirical Bayes and uses it to plan budgets on unseen tasks. Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting. Regret depends only on how well SPLIT ranks configurations, and it stays near-optimal under compute, price and latency costs and online. The formula also explains when asking more LLM judges lowers precision. As a forecaster far beyond the pilot, SPLIT beats power-law fitting when the verifier caps success, and we report where it loses.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 19 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Solid, honest paper on a real problem—worth your time. This line is wrong
1 major objection · 9 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 23 references say.
The paper says, at Section 3, Assumption 1
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.88):
“Dependence among candidateswithina task, such as duplicated or near-duplicated answers, is not modelled and may bias SPLIT, especially at largeN.”
Pith says: Objection 1 of 1 · major
The conditional independence assumption is central to the derivation but is acknowledged in Section 7 as a limitation when near-duplicate answers appear. The empirical evaluation implicitly tests robustness, but the paper does not provide a dedicated sensitivity analysis. I recommend adding a brief simulation or discussion of how violation of this assumption (e.g., through duplicated candidates) would affect the regret bound and the planner's decisions, to guide practitioners on when SPLIT might be expected to degrade.
9 minor objections
The paper says, at Table 2 caption
Pith says: minor objection 1 of 9
The distinction between 'parameter-count costs' and 'compute-matched costs' should be briefly defined in the caption for quick reference, as the main text defines them only in Section 4.
The paper says, at Figure 2 caption
Pith says: minor objection 2 of 9
The caption states 'mean over 10 splits' but does not explain whether the shaded regions (if any) represent standard errors or confidence intervals. Please add a sentence clarifying the uncertainty depiction.
show 7 more
The paper says, at Section 6.1, paragraph on 'Modelling correlation matters'
Pith says: minor objection 3 of 9
The phrase 'raises regret 1.2–4.8 times' is ambiguous; it should be clarified that the regret ratio is compared to the full SPLIT model (e.g., 'the regret of the independent-errors variant is 1.2–4.8 times that of SPLIT').
The paper says, at Section 6.3, after Equation (2)
Pith says: minor objection 4 of 9
The mean per-task precision ceiling for MATH-500 is reported as 0.717 vs 0.718. Consider adding a confidence interval or standard deviation to convey the variability across tasks.
The paper says, at Table 11
Pith says: minor objection 5 of 9
For comparisons with p=0.006, the bootstrap confidence intervals may be wide due to the limited number of paired units. A note acknowledging this would be helpful for interpretation.
The paper says, at Section 7, Limitations
Pith says: minor objection 6 of 9
The paper states that the finite-sample regret bound is vacuous at their task counts. Table 10 gives the Hoeffding term (~60 points), but in the limitations section it would be clearer to repeat this number for readers who do not consult the appendix.
The paper says, at Section 4, 'Difficulty'
Pith says: minor objection 7 of 9
For readers unfamiliar with empirical Bayes, a brief footnote or parenthetical explanation of the beta-binomial shrinkage would help, especially given that the method is central to the estimator.
The paper says, at Figure 5 caption
Pith says: minor objection 8 of 9
For panel (c), 'threshold re-tuned (TPR = 0.9)', please clarify how the re-tuning is done (e.g., choosing a threshold on a held-out set per generator to achieve TPR=0.9).
The paper says, at Section 6.4, 'A long horizon with real agents'
Pith says: minor objection 9 of 9
The experiment treating 175 SWE-bench systems as exchangeable attempts is clever but not a test of selection. The paper already notes this limitation. However, the phrase 'this tests the coverage component (the difficulty prior)' could be more explicit: it tests the beta-binomial or NPMLE prior for coverage, not the full selection model.
What would settle it
Still, to bound the practical impact of the acknowledged violation of conditional independence (duplicated or near-duplicated answers within a task), one could inject synthetic duplicates into a real benchmark (e.g., CodeContests) by replicating a fraction of the candidate solutions (say 10–30%) before applying the verifier, and measure whether SPLIT's allocation regret (Table 2) increases beyond 1.0 percentage points at the largest budget tested (256). If regret remains below 1.0, the concern is practically negligible within the studied regime.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 19 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 1 major objection, 9 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a good paper. It does what it says: given a small labeled pilot, it allocates test-time compute among generators, samples, and verifier calls, and it works across a wide range of settings. The exact success formula (Equation 1) is genuinely new, and the way they build in correlation via copula and per-candidate propensity is clean and mechanically grounded. The empirical validation is extensive—seven benchmarks, real verifiers, 26 models, 175 agent systems, 80k executed programs—and they report significance tests honestly. They also tell you where SPLIT does not beat baselines (pure coverage, argmax, some forecasting), which is rare and refreshing.
The soft spots are real but acknowledged. The conditional-independence assumption (Assumption 1) is the weakest link, and they say so in Section 7. Duplicate or near-duplicate answers within a task can bias the planner at large N, especially when verifier errors compound. The paper's own limitation section is thorough: same-agent horizons are short, pilot representativeness matters, costs are simplified. These are not hidden. The one place I pushed back a little is forecasting: SPLIT's edge over power-law extrapolation is large only under a simulated verifier and on SWE-bench thresholds; on real code tests it is small (0.4–0.6 pp) and on coverage or argmax it does not help. The authors honestly report this, so it is not a flaw in the paper, but the reader should calibrate expectations. The central claim—allocation regret within 0.8 points—holds up across the board.
Who should read this: anyone making inference-time decisions with LLMs or agents, especially practitioners choosing between sampling, mixing models, and verification. The paper gives a principled, pilot-driven method that beats fixed rules by clear margins where it matters. It also explains when re-verification backfires and why pooled verifier statistics mislead.
My recommendation: yes, send it to peer review. It is a serious piece of work—the math is correct, the experiments are reproducible, the claims are proportional. The authors earned a careful referee.
The strongest objection
A careful reading of the paper and its limitations reveals no unresolved threat to the central claim. The conditional-independence assumption (Assumption 1) is the weakest link in the theoretical derivation, but the paper honestly acknowledges it (Section 7) and the empirical evaluation on seven benchmarks with real verifiers implicitly tests robustness to its violation. The regret bounds (Proposition 4) explain why differential rather than absolute error matters, and the empirical regret stays within the claimed 0.02–0.8 points. Fixed-rule failures are documented and plausible. The paper's candor about where SPLIT does not help (coverage, argmax, forecasting) further supports its credibility. I find no load-bearing concern that would change the reader's verdict.
The reader identified the conditional-independence assumption as the weakest assumption, which is indeed the primary theoretical vulnerability. However, the paper acknowledges this limitation, and the extensive empirical validation across diverse benchmarks and verifiers provides strong evidence that any bias from within-task dependence does not materially degrade allocation quality within the tested range. The regret bounds show why small differential error suffices for good decisions. All other potential concerns (representative pilots, forecasting accuracy, finite-sample guarantee vacuousness) are explicitly stated by the authors and do not undermine the central claim about SPLIT's allocation performance. No new objection emerged from the stress test.
The referee's report
If the results hold, SPLIT offers a principled, pilot-calibrated alternative to ad-hoc rules for allocating inference compute, with broad applicability across language-model deployments. The mechanistic success formula, the analysis of correlated verifier errors, and the empirical findings (e.g., when majority voting lowers precision, the coupling of verifier accuracy to difficulty, and the role of cross-generator correlation) are valuable contributions. The paper's rigorous empirical methodology (public releases, 80k locally executed programs, paired significance tests) and its honest reporting of limitations and mixed forecasting results further strengthen its credibility. The work has the potential to influence how practitioners decide on test-time scaling strategies.
Pith's reading of the paper (a machine's summary, not the authors')
This paper argues that test-time compute is limited not just by generation but by selection: the verifier that picks which candidate to submit has its own error rates, and those errors correlate across tasks, models, and repeated calls. The authors derive an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection, accounting for heterogeneous task difficulty, shared blind spots across generators, and correlated verifier calls. They estimate the formula's ingredients from a small labelled pilot via empirical Bayes, then use it to plan how many samples to draw, which models to mix, and how much to verify. On seven benchmarks spanning math, science QA, code, and software-engineering agents, this method (SPLIT) lands within 0.02–0.8 percentage points of the hindsight-optimal allocation across 26 models, 175 agent systems, and six kinds of real verifiers. Every fixed rule of thumb tested—use the strongest model, scale solutions faster than verifications, ask more judges—loses at least 2.5 points somewhere. A practitioner with a small labelled pilot can thus make better allocation decisions than any single heuristic.
What carries the argument. The central identity is the exact success formula (Equation 1): S = ∫$_0^{1}$ Σ_m n_m a_m (1 - s_m u)^{n_m-1} Π_{l≠m} (1 - s_l u)^{n_l} du, where for each generator m, a_m is the probability a candidate is accepted and correct, s_m is the probability it is accepted, and n_m is the number of samples. This formula, together with a Gaussian copula for cross-generator correlation and a $\beta$-binomial propensity model for correlated verifier calls, captures how heterogeneous task difficulty, shared blind spots, and verifier error jointly determine verified success. The machinery also includes empirical-Bayes estimation of difficulty priors ($\beta$-binomial or NPMLE), pairwise copula correlations, and per-task verifier rates from a small labelled pilot, plus a grid planner that evaluates the formula under the fitted posterior for every candidate configuration and selects the one with the highest predicted success.
Core claim. The paper presents SPLIT, a pilot-calibrated planner for allocating test-time compute. Its core is an exact formula (Proposition 1) for verified success under uniform-among-accepted and argmax selection, which models the verifier's true- and false-positive rates, per-task difficulty, cross-generator correlation (via a Gaussian copula), and correlated repeated verifier calls (via a candidate-level propensity). Estimated from as few as 32 labelled samples per task, the formula allows the planner to rank configurations by predicted success under a budget. Across multiple settings—including real verifiers (unit tests, LLM judges, reward models, patch verifiers), real models (Llama, Gemma, CodeGen, GPT-4o, etc.), and real benchmarks—SPLIT's chosen configuration achieves success within 0.02–0.8 percentage points of the best configuration in hindsight on held-out tasks. In contrast, every fixed rule (strongest model, most accurate per cost, uniform ensemble, adaptation of the Singhi et al. 2025 compute-optimal split, generation-only, maximum verification, fixed verifier threshold) loses at least 2.5 points somewhere, with losses up to 20 points. The formula also explains exactly when re-verification lowers attainable precision (Proposition 2) and why pooling verifier accuracy across tasks biases forecasts.
The premise it rests on. The paper assumes that, given a task, candidates from different generators are conditionally independent (Assumption 1); this rules out dependencies within a task from duplicated or near-duplicated answers, which the authors acknowledge may bias SPLIT at large sample counts.
If the paper is right (Pith's inferences, not the authors')
Practitioners can replace fixed rules of thumb with a data-driven, mechanism-based allocation that is near-optimal across a wide range of budgets, models, and verifiers.
The formula's condition for when re-verification helps (F_k/T_k < F_1/T_1) gives a testable criterion: if majority voting inflates false acceptances more than true acceptances, additional judge calls lower the precision ceiling.
Modeling cross-generator correlation is essential: assuming independence overstates the diversity gain by up to 7 percentage points, while SPLIT's copula predicts it to within 0.43 points.
Stronger generators produce wrong answers that pass fixed test suites more often, and this effect for scored verifiers is a calibration shift—re-tuning the threshold removes it. Allocation must re-measure verifier rates when the generator changes.
Online re-planning that stops a task at its first acceptance and pools saved budget can match the best static configuration in hindsight, recovering 0.56 points on average.
SPLIT's framework could be extended to other selection rules (e.g., top-k with rejection) and to multi-step verification pipelines, using the same principle of modeling selection probabilities from a pilot.
The empirical finding that LLM judges share blind spots on MATH-500 but not on GPQA suggests that the utility of majority voting depends on the homogeneity of judge errors; a pre-deployment calibration of judge agreement could predict whether extra calls help.
The paper's pessimism about power-law extrapolation for selection (SPLIT beats it only under a simulated verifier and on agentic SWE-Bench) implies that curve-fitting approaches without a verifier model are unreliable when the verifier caps success.
Because SPLIT optimises benchmark-defined correctness, deploying it in safety-critical settings would require the ground-truth correctness notion to be trustworthy; otherwise the planner optimises a flawed target.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
A common way to make language models and agents more accurate is to spend more compute at inference time: generate many candidate solutions and submit the one a verifier prefers. Success then requires that some candidate is correct and that the verifier picks it. The first grows predictably with the number of samples; the second does not, because real verifiers (unit tests, majority votes, LLM judges, reward models) accept wrong answers, reject right ones, and make correlated mistakes. With a fixed budget, a practitioner must decide how many samples to draw, which models to draw them from, and how much to spend on verification, and today these choices are made by rules of thumb. We introduce SPLIT, which makes them from a small labelled pilot. It rests on an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection that accounts for varying task difficulty, blind spots shared across models, and correlated repeated verifier calls. SPLIT fits the formula by empirical Bayes and uses it to plan budgets on unseen tasks. Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting. Regret depends only on how well SPLIT ranks configurations, and it stays near-optimal under compute, price and latency costs and online. The formula also explains when asking more LLM judges lowers precision. As a forecaster far beyond the pilot, SPLIT beats power-law fitting when the verifier caps success, and we report where it loses.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-20
The authors' main claim, as the classifier picked it out 0.97
“Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.89 · human participants 0.04 · controlled comparison 0.87 · randomized assignment 0.02 · states quantitative result 0.98 · compares with prior results 0.94 · mentions code or data availability 0.08
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| A common way to make language models and agents more accurate is to spend more compute at inference time: generate many candidate solutions and submit the one a verifier prefers. | background 1.00 | 0.05 | 0.02 | asserted 0.98 |
| Success then requires that some candidate is correct and that the verifier picks it. | background 1.00 | 0.07 | 0.03 | asserted 1.00 |
| The first grows predictably with the number of samples; the second does not, because real verifiers (unit tests, majority votes, LLM judges, reward models) accept wrong answers, reject right ones, and make correlated mistakes. | background 0.95 | 0.23 | 0.06 | asserted 1.00 |
| With a fixed budget, a practitioner must decide how many samples to draw, which models to draw them from, and how much to spend on verification, and today these choices are made by rules of thumb. | gap 0.66 | 0.06 | 0.04 | asserted 1.00 |
| We introduce SPLIT, which makes them from a small labelled pilot. | contribution 0.92 | 0.91 | 0.12 | asserted 1.00 |
| It rests on an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection that accounts for varying task difficulty, blind spots shared across models, and correlated repeated verifier calls. | method 0.75 | 0.90 | 0.11 | asserted 1.00 |
| SPLIT fits the formula by empirical Bayes and uses it to plan budgets on unseen tasks. | method 0.97 | 0.94 | 0.11 | asserted 1.00 |
| Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting. | result 1.00 | 0.96 | 0.64 | asserted 1.00 |
| Regret depends only on how well SPLIT ranks configurations, and it stays near-optimal under compute, price and latency costs and online. | result 0.84 | 0.90 | 0.19 | asserted 1.00 |
| The formula also explains when asking more LLM judges lowers precision. | interpretation 0.77 | 0.81 | 0.06 | asserted 1.00 |
| As a forecaster far beyond the pilot, SPLIT beats power-law fitting when the verifier caps success, and we report where it loses. | result 0.94 | 0.94 | 0.07 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.86
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 8,322 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (14)
Figures are the authors', reproduced from the paper.
References (23)
- [1] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, on PithPith's reviewarXiv
- [4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, on PithPith's reviewarXiv
- [5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, on PithPith's reviewarXiv
- [7] 16 Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. R2E-Gym: Procedural environments and hybrid verifiers for scaling open-weights SWE agents.arXiv preprint arXiv:2504.07164, on PithPith's reviewarXiv
- [8] Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S. Siegel, Boyi Wei, Tianci Xue, Ziru Chen, et al. Holistic agent leaderboard: The missing infrastructure for AI agent evaluation.arXiv preprint arXiv:2510.11977, on PitharXiv
- [11] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-proof Q&A benchmark.arXiv preprint arXiv:2311.12022, on PithPith's reviewarXiv
- [12] =E √XY≤√ EXEY=EX= 1−F(0). Equality iffX∝Ya.s., i.e.X=Y, i.e.p A =p B sincex↦→(1−x) N is injective forN >0. (d)φ(x,y) =g(x)g(y)withg(p) = (1−p) N/2 is a product of two non-negative functions monotone in the same direction, hence supermodular. Gaussian copulas increase inρin the... on Pith
- [14] Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Chang, Marcus Rohrbach, and Anna Rohrbach. When to solve, when to verify: Compute-optimal problem solving and generative verification for LLM reasoning.arXiv preprint arXiv:2504.01005, on PitharXiv
- [15] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, on PithPith's reviewarXiv
- [16] Benedikt Stroebl, Sayash Kapoor, and Arvind Narayanan. The limits of inference scaling through resampling. arXiv preprint arXiv:2411.17501, on PitharXiv
- [17] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering.arXiv preprint arXiv:2405.15793, on PithPith's reviewarXiv
- [18] Yingxuan Yang, Chengrui Qu, Muning Wen, Laixi Shi, Ying Wen, Weinan Zhang, Adam Wierman, and Shangding Gu. Understanding agent scaling in LLM-based multi-agent systems via diversity.arXiv preprint arXiv:2602.03794, on PitharXiv
- [20] This is an upper bound only. (d) For any FPRsft≥0and anyN,0≤E[1−(1−τp) N]−S N≤NE[f t(1−p)]≤NE[f t]. Proof.(a)S ∞−S N =E[(a/s)(1−s) N], ands=τp+f(1−p)is a convex combination ofτandf, so s≥min(τ,f 0); then use1−x≤e −x. (b) Fixδwithπ(p)≤2cp α−1 on(0,δ). The integral over[δ,1]is a... on Pith
- [22] Automatic verifier-model choice.Both verifier models are fitted on the first half of each calibration task’s pilot samples. Each predicts verified success at the full pilot size. We keep the model whose prediction is closer to the model-free success of uniform-among-accepted s... on Pith
- [23] (b) Extrapolation error. 25 D Significance tests Table 11:Paired tests for headline comparisons.Mean difference (ours minus comparator; negative favours ours) with bootstrap 95% interval and Wilcoxon signed-rankp-value. Units are shared (model, pilot, seed) for MAE and (datase... on Pith
- [1956] Yubin Kim, Ken Gu, Chanwoo Park, Chunjong Park, Samuel Schmidgall, A. Ali Heydari, Yao Yan, Zhihan Zhang, et al. Towards a science of scaling agent systems.arXiv preprint arXiv:2512.08296, on PithPith's reviewarXiv
- [2010] Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, and Azalia Mirhoseini. CodeMonkeys: Scaling test-time compute for software engineering.arXiv preprint arXiv:2501.14723, on PithPith's reviewarXiv
- [2021] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, on PithPith's reviewarXiv
- [2022] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, et al. Let’s verify step by step.arXiv preprint arXiv:2305.20050, on PithPith's reviewarXiv
- [2023] Jon Saad-Falcon, E. Kelly Buchanan, Mayee F. Chen, Tzu-Heng Huang, Brendan McLaughlin, Tanvir Bhathal, Shang Zhu, Ben Athiwaratkun, et al. Shrinking the generation-verification gap with weak veri- fiers.arXiv preprint arXiv:2506.18203, on PithPith's reviewarXiv
show 3 more
- [2024] Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, and James Zou. Are more LLM calls all you need? towards scaling laws of compound inference systems.arXiv preprint arXiv:2403.02419, on PithPith's reviewarXiv
- [2025] Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, et al. How do large language monkeys get their power (laws)?arXiv preprint arXiv:2502.17578, on PithPith's reviewarXiv
- [2026] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Gen- erative verifiers: Reward modeling as next-token prediction.arXiv preprint arXiv:2408.15240, on PithPith's reviewarXiv
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment