{"id":"b9b5b527-e573-45f0-9a77-abd993f9884c","arxiv_id":"2608.06422","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A sharded LLM judge, which grades each small group of criteria in a separate call, agrees better with expert labels and resists best-of-N presentation attacks better than a holistic judge given the same apparent budget.","lead":"Splitting a long checklist into smaller chunks and giving each chunk to a separate judge call improves an LLM grader's agreement with human experts, while a single judge with more compute does not recover the lost accuracy. The same split also blocks a class of rewording attacks that trick overloaded judges into accepting work that fails real requirements.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The equal-budget control is not equal under ordinary token billing: the paper defines a matching pooled-opinions arm but reports no results for it.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the full-budget control is not actually budget-matched under ordinary token accounting. My stress-test sharpens the point by noting that the one arm that would provide a genuine equal-total-token control, pooled opinions, is defined in Section 2.1 but never reported. This is not a minor accounting detail: the paper's central claim is that decision load, not total compute, is the binding constraint, and that claim is supported by the sharded-minus-full-budget contrast. If that contrast also changes total tokens and the number of times the model reads the evidence, it cannot establish the mechanism. The concern is real but addressable: a pooled-opinions control is relatively cheap to run, and the paper already contains load sweeps showing that a single judge degrades as K grows even when output budget is scaled up, which is suggestive evidence that load matters. I therefore keep the reader's CONDITIONAL verdict rather than moving to REJECT. I also note secondary limitations the reader mentioned, including no released code or data, but those are not the reason the central claim is currently unverified; the missing budget-matched control is.","tokens_in":26899,"tokens_out":8259,"duration_ms":89873,"concrete_test":"Re-run the primary PaperBench comparison with the pooled-opinions control defined in Section 2.1: use S=ceil(K/g) calls, each with the full artifact and all K criteria, the same per-call output budget B and the same context cap as the sharded arm, then aggregate per-decision verdicts by majority. Compare pooled opinions against sharded at the same S and B under standard input+output token billing. If pooled opinions matches sharded's accuracy, the reported sharding benefit is explained by total compute or repeated context passes rather than by dividing the decision set; if sharded still beats pooled at matched total tokens and matched context passes, the decision-load interpretation survives. A cheaper secondary check is to recompute the Figure 1 and Table S19 configurations from Appendix A.2 under ordinary token billing and report the full-budget cost relative to sharded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 asserts that sharded, pooled opinions, and full-budget solo 'spend the same total budget SB', and that sharded versus full-budget solo holds per-decision budget fixed while varying only decision load. The implemented grading arms do not satisfy this accounting. For PaperBench (Appendix A.2), each sharded call uses a 1500-token output cap and re-sends the full context, while the full-budget control uses one context pass and an output cap of min(8000, 1500+120K). At K=116, one-criterion sharding uses 116 context passes and a nominal 174,000 output-token budget, versus one context pass and 8,000 output tokens for the full-budget call; the per-decision output budget is 1500 versus about 8000/116=69. Under ordinary input-plus-output token billing the two arms differ by orders of magnitude, so the contrast labeled 'sharded minus full-budget solo' does not isolate decision load: it also varies total compute and the number of evidence passes. The paper defines a pooled-opinions arm that would match sharded's true total token count while keeping K decisions per call, but no result from that arm appears in the available text. The headline mechanism, that decision load rather than compute is the binding constraint, therefore lacks its required control in the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies LLM-based oversight when a single call must return many verdicts. It claims that agreement with expert labels falls as the number of decisions per call grows; that giving the same call a larger budget does not reverse this; that partitioning decisions into separate calls ('sharding') improves agreement while holding model, evidence, total budget, and per-decision budget fixed; that a best-of-N presentation attack can exploit the overload and sharding mitigates it; and that only debate-style opposition handles adaptive per-criterion persuasion. Evidence is drawn from expert labels on PaperBench, JudgmentBench, and ROBoto2, plus code, cyber, synthetic, and structural-control datasets.","tokens_in":27189,"tokens_out":12494,"duration_ms":128177,"significance":"The question is important for scalable oversight and LLM-as-judge pipelines. If the equal-budget claim held, the paper would establish decision load as a distinct bottleneck and offer a simple, practical intervention with security implications. The manuscript has genuine strengths: external expert labels in three domains, cluster-resampled intervals as the conservative inferential unit, explicitly disclosed scope conditions, a ground-truth-fixed attack construction, independent stronger-model re-validation of the cyber labels, and a capacity account tested on a controlled synthetic grid with a falsifiable prediction about group size and judge capability. However, the absence of a genuinely budget-matched control currently leaves the central claim unproven in the reported grading experiments.","major_comments":[{"comment":"The 'equal total budget' assertion is not satisfied by the implemented grading arms. Section 2.1 defines the arms so that sharded, pooled opinions, and full-budget solo 'spend the same total budget SB,' and claims the contrast (sharded minus full-budget solo) varies decision load alone. In the PaperBench load sweep (Appendix A.4), each sharded call is capped at 1,500 output tokens and re-sends the full evidence, while the full-budget call is a single context pass with an output cap of min(8000,1500+120K). At K=116 the sharded panel uses 116 context passes and a nominal 174,000 output tokens versus one context pass and 8,000 output tokens for the full-budget call; per decision the sharded judge receives 1,500 output tokens versus about 69 for the full-budget judge. The cost table in Appendix B.2 reports input-token totals that differ by roughly a factor of 23 between shard group sizes (50,524 tokens at g=32 versus 1,155,404 at g=1), so total token cost is not fixed across arms under ordinary billing. The paper defines a pooled-opinions arm that matches the sharded panel's true total token count while keeping all K decisions per call, but no results from this arm are reported anywhere. The central conclusion that 'decision load, not total compute, is the binding constraint' (Section 3, Table S19) and the security claims in Sections 4-5 therefore lack the required control. Please run and report the pooled-opinions arm, or equalize total and per-decision token budgets across arms, and rerun the central contrasts.","section":"§2.1, Appendix A.2, A.4, B.2"}],"minor_comments":[{"comment":"The statement that the full-budget single judge 'does not improve' should be qualified: Table S19 shows the full-budget arm exceeding holistic at K=8 (0.708 versus 0.631), so the claim is specifically about high-load conditions.","section":"Abstract, §3"},{"comment":"The term 'budget' is used ambiguously: for grading it appears to mean the output-token cap, but re-sending the full context to every shard means input-token cost differs across arms; the paper should specify whether budget means output tokens, input-plus-output tokens, or dollar cost.","section":"§2.1"},{"comment":"The full-budget grading arm adds a step-by-step instruction in addition to the larger token cap, so it is a budget-plus-instruction control rather than a pure budget manipulation; the synthetic control isolates this, but the text should state the distinction plainly.","section":"Appendix A.2"},{"comment":"In the text under review, the headings 'JudgmentBench Tables' and 'ROBoto2 Table' are placeholders with no content; the corresponding results appear later in B.8 and Table S13, so the appendix numbering should be cleaned up.","section":"Appendix B.4, B.5"},{"comment":"The capacity account introduces h(n) and n* as free constructs fit on one synthetic dataset; the manuscript should make explicit that these are descriptive quantities not estimated from the expert datasets, to avoid implying a parameter-free derivation.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The missing pooled-opinions arm is the single substantive issue in the manuscript; the rest of the empirical work is careful and the scope limits are disclosed. I would encourage the editor to request a revision rather than reject, because the fix is well-scoped and the surrounding evidence is strong."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a worthwhile paper that overstates its control. The sharding effect on agreement is real and replicated across three expert-labeled datasets. The best-of-N presentation attack and the finding that sharding removes that advantage are also solid. But the central mechanistic claim—that decision load, not total compute, is the binding constraint—rests on a comparison that doesn't hold the budget fixed.\n\nWhat's genuinely new: the systematic load sweep on PaperBench, JudgmentBench, and ROBoto2 with external labels; the finding that a sharded weaker judge can beat a holistic stronger judge; and the adversarial results. The attack holds ground truth fixed and varies only presentation, which is the right threat model. The defense is convincing on grading, and the null on independent patches is a useful control.\n\nThe soft spot is the one the stress-test note flags. Section 2.1 says sharded, pooled opinions, and full-budget solo all spend the same total budget, but the grading experiments don't implement that. Each shard re-sends the full context and has a 1500-token output cap; the full-budget control makes one context pass with an 8000-token cap. At K=116 that's 116 context passes versus one. Ordinary token accounting shows the sharded panel spending orders of magnitude more. The paper defines a pooled-opinions arm that would match the sharded panel's token count while keeping all K decisions per call, but no results from that arm appear. So the contrast labeled 'sharded minus full-budget solo' varies total compute and number of evidence passes, not just decision load. That doesn't destroy the practical recommendation—sharding is cheap and helps—but it does mean the paper doesn't isolate the mechanism it claims.\n\nThe capacity account is a reasonable post hoc fit on a synthetic grid, not a derivation, and no code or data is released, which makes independent checking harder. Those are addressable. The core evidence for sharding as an intervention is solid, though the headline interpretation needs revision.\n\nWho's this for? People building model-based oversight pipelines or LLM-as-a-judge evaluations. It deserves a serious referee; the budget accounting can be fixed, and the attack/defense results are worth preserving. I'd engage with it, but I'd insist on seeing a properly matched control or a more modest claim.","headline":"The empirical evidence that sharding helps is real; the claim that it isolates decision load from compute is not supported by the reported budget accounting.","tokens_in":27677,"tokens_out":3014,"would_cite":true,"duration_ms":30145,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that decision load — the number of verdicts one LLM call must return — is the binding constraint in model-based oversight, and that sharding verdicts across calls recovers expert agreement and blocks presentation attacks…","keywords":["sharding","decision load","LLM-as-judge","scalable oversight","best-of-N presentation attack","adversarial robustness","expert-graded evaluation","capacity model"],"falsifier":"Re-run the PaperBench load sweep at equal billed cost: compute the API-billed input-plus-output tokens for the whole sharded panel (each call re-sends the full context) and cap the full-budget solo call at that same total token bill, then see whether the pooled kappa gap of +0.142 the paper reports persists or collapses. The paper's own cost appendix reports input-token costs differing by roughly a factor of 23 between group sizes, so this is a directly checkable quantity, and if the gap vanishes at equal billing the load-only conclusion fails.","tokens_in":26706,"feed_emoji":"🧩","tokens_out":10428,"duration_ms":89811,"temperature":0.7,"pith_summary":"Giving an LLM judge more compute does not necessarily make it check more requirements: when one call must return many verdicts, some decisions are settled from surface cues rather than the evidence, so agreement with expert labels falls as the number of verdicts per call grows, even when that call receives the same token or tool budget as a panel of separate calls. The paper's intervention is sharding — partition the requirements into smaller groups, assign each group to its own call, and aggregate the verdicts. Against a single call with the panel's full budget, sharding improves agreement while holding the model, evidence, total budget, and per-decision budget fixed, across expert-graded research replication, legal work, and clinical-trial risk-of-bias assessments; in places, a sharded weaker judge outperforms a more capable holistic judge. The same overload explains a security result: a best-of-N adversary that holds the work fixed and varies only its presentation can severalfold increase an over-loaded judge's acceptance of genuinely unmet criteria, and sharding removes that advantage wherever it reduces baseline error. Sharding does not stop attacks that persuade the judge separately on each criterion; there, debate-style opposition on top of sharding withstands adaptive re-optimization.","feed_headline":"Sharding LLM judges beats one call with more budget","feed_subtitle":"One call that must return many verdicts skips checks; splitting verdicts across calls restores accuracy and blocks attacks.","key_machinery":"The load-bearing object is the capacity account — a factorization of per-decision attention: for a failed decision $k$ assigned to a call with $n$ verdicts and budget $b$, the detection probability is $\\Pr[s_k \\text{ high} \\mid y_k = 1] = \\rho_k \\, \\alpha(n,b)$, with $\\alpha(n,b) = g(b/n)\\, h(n)$, where $\\rho_k$ is how recoverably the failure is exposed by the evidence, $g$ is nondecreasing in per-decision budget, and $h$ is nonincreasing in decision load, sitting near $1$ below a judge-specific capacity $n^\\ast$ and falling past it. This identity separates 'not enough compute' from 'too many verdicts': raising the per-call budget grows $g$ but leaves $h$ depressed, while sharding reduces $n$ and restores $h$ at fixed total budget. The experimental arms — solo, full-budget solo, pooled opinions, and sharded — are constructed so that the sharded-minus-full-budget contrast varies only the load per call ($K$ against $K/S$) while holding model, evidence, total budget, and per-decision budget fixed, which is what lets the paper attribute the gain to division rather than to compute.","core_discovery":"Across expert-labeled datasets in three domains, the paper claims that decision load — the number of verdicts one call must return — degrades oversight accuracy that additional single-call budget cannot recover, and that sharding restores it at fixed total budget. In its strongest form the claim is that a single holistic call can spend the panel's full combined budget and still scrutinize each decision less than a divided panel; a sharded weaker judge can outperform a more capable holistic judge and match it even when the stronger judge receives the full budget. The same unsupported fraction of verdicts that drives the accuracy loss is directionally selectable: a best-of-N presentation attack that changes only how the artifact is framed raises over-acceptance of genuinely unmet criteria severalfold on grading (and raises a code verifier's cleared-defect rate from 0.12 to 0.44), and sharding keeps over-acceptance low as the adversary's search widens. The defense is largest when many criteria require one shared understanding of a large submission (up to 0.261 reduction in over-acceptance), small when each decision can be evaluated from a local slice, and null when decisions are independent. Sharding does not defend attacks that persuade the judge on each criterion individually; there, one-sided opposition on top of the sharded base limits a five-round adaptive attack's cumulative surface to 0.39–0.56, against 0.96 for sharding alone at the weaker tier.","pith_inferences":["Under ordinary API billing the budget-equality framing is fragile: because each shard re-sends the full context, the sharded panel's billed input tokens exceed the full-budget solo call's by a factor the paper's own appendix puts near 23 at some group sizes, so the real-world cost of the accuracy and security gains is higher than the fixed-budget framing suggests — the direction of the effect is u","The capacity account makes a testable prediction for future model tiers: as judges' capacity $n^\\ast$ grows, the group size at which sharding pays off should rise, and the load deficit should eventually disappear on routine rubrics; re-running the same load sweep on a newer model would confirm or bound the mechanism.","The attack-defense asymmetry suggests a design rule: match the defense structure to the attack's unit of exploitation — division against overload, balanced evidence against per-criterion persuasion — so a pipeline that is safe in load-based evaluations can still be vulnerable to memo-style persuasion attacks.","A practical engineering rule follows from the observed capability dependence: choose shard group size by model tier and decision difficulty (capacity-sized grouping rather than one verdict per call), which the paper's cost comparison shows is far cheaper per unit of agreement."],"forward_implications":["Oversight pipelines should report decision load alongside token and tool budgets, because one holistic call can spend as much as a divided panel while scrutinizing each decision less.","Task structure can partly substitute for judge capability: a sharded weaker judge can outperform a stronger holistic judge, so pipeline designers can trade model tier for decision grouping.","Best-of-N presentation selection is a practical threat to overloaded judges — an adversary changes only the framing and severalfolds the acceptance of genuinely unmet criteria — so unsharded single-call oversight is exploitable at the presentation layer.","Wherever sharding reduces baseline error it closes the attack surface, keeping over-acceptance low as the adversary's search widens; where decisions are independent or within a judge's capacity, sharding gives little or nothing.","Load-exploiting attacks and per-criterion persuasion attacks need different defenses: division for the first, opposition on the sharded base for the second."],"supporting_citations":[{"why":"Supplies the PaperBench expert-graded replication dataset that anchors the load sweep and the attack/defense results.","marker":"(Starace et al., 2025)"},{"why":"Supplies the second expert-labeled dataset (JudgmentBench legal grading) on which the load result and the persuasion-attack defenses replicate.","marker":"(Yang et al., 2026)"},{"why":"Supplies the third expert-labeled dataset (ROBoto2 clinical risk-of-bias) showing division helps only below a judge's capacity.","marker":"(Hevia et al., 2025)"},{"why":"Defines the RoB2 signaling-question rubric whose expert answers are the clinical reference labels.","marker":"(Sterne et al., 2019)"},{"why":"Frames best-of-N selection against a learned judge as the overoptimization instantiation the presentation attack builds on.","marker":"(Gao et al., 2023)"},{"why":"Supplies SWE-Bench Pro as the direct-reading control where bundling many patches does not overload the verifier.","marker":"(Scale AI, 2025)"},{"why":"Supplies FActScore as the structural control showing many easy checks over shared context do not degrade with load.","marker":"(Min et al., 2023)"},{"why":"Supplies the ARVO confirmed memory-safety duals used to build the oracle cyber-verification dataset.","marker":"(Mei et al., 2024)"}],"fun_headline_variants":["Split LLM verdicts to stop oversight gaps and attacks","Sharding beats extra budget for LLM judge accuracy","One LLM call misses checks; sharding fixes it","Sharded judge outperforms holistic LLM at same cost","Divided LLM judge beats single call with full budget"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That the full-budget solo call and the sharded panel really spend the same total budget with only the division of verdicts different — the paper's headline contrast isolates decision load only if that budget accounting is exactly right.","fun_headline_variants_meta":{"raw":{"variants":["Split LLM verdicts to stop oversight gaps and attacks","Sharding beats extra budget for LLM judge accuracy","One LLM call misses checks; sharding fixes it","Sharded judge outperforms holistic LLM at same cost","Divided LLM judge beats single call with full budget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1569,"prompt_tokens":1120,"completion_tokens":449,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":736,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":736,"tokens_out":449,"duration_ms":4878,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:26:43.159492+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the PaperBench load sweep at equal billed cost: compute the API-billed input-plus-output tokens for the whole sharded panel (each call re-sends the full context) and cap the full-budget solo call at that same total token bill, then see whether the pooled kappa gap of +0.142 the paper reports persists or collapses. The paper's own cost appendix reports input-token costs differing by roughly a factor of 23 between group sizes, so this is a directly checkable quantity, and if the gap vanishes at equal billing the load-only conclusion fails.","supporting_citations":[{"cited_title":"SWE-Bench Pro : Can AI agents solve long-horizon software engineering tasks?, 2025","cited_arxiv_id":null,"evidence_quote":"Supplies SWE-Bench Pro as the direct-reading control where bundling many patches does not overload the verifier."}],"review_version":1}