REVIEW 2 major objections 3 minor 28 references
When Policies Change Probabilities: Modular Decision-Making for LLM Code Review
T0 review · 2 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Changing the decision costs shown in an LLM code-review prompt changes the reported patch-failure probability, even when the patch and evidence are fixed, and separating risk elicitation from the action rule restores stable, reusable scores
desk verdict Solid, well-controlled audit of LLM reviewer probability stability; the policy-dependence result holds up, and the paper's own limitation notes cover most of what worried me. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a matched policy-block intervention: for the same candidate patch and evidence, swap an equal-cost prompt block for a 10:1 false-accept block that also prints the derived threshold, then measure how the reported probability moves. The Bayes decision rule $\tau(C)=C_{\mathrm{FR}}/(C_{\mathrm{FA}}+C_{\mathrm{FR}})$ maps cost asymmetry to an action threshold—$1/2$ under equal costs, roughly .091 under 10:1—and the paper tests whether that threshold changes the probability as well as the action. A repeat-control comparison subtracts call-to-call variation, and a fixed-rule comparison applies the same coded threshold to probabilities from both prompts to isolate the proba
What would settle it
Rerun the full executable harness for all 720 candidate patches and recompute the matched probability-change and fixed-threshold loss; if most of the probability movement or the loss reduction disappears when labels are corrected, the policy-dependence claim collapses. Alternatively, run a factorial prompt variant that changes only the displayed threshold or only the cost semantics; if the probability movement vanishes when the numeric threshold is not printed, the effect is anchoring or copying rather than genuine policy dependence.
Extended reading notes
Core claim
The central claim is that the numeric failure probability produced by an LLM code reviewer is not a stable property of the patch: it moves when only the downstream cost-and-threshold block changes. The authors verify this with matched calls on 720 patches, fixing the issue, patch, code context, monitor evidence, and every other prompt byte; replacing equal costs with a 10:1 false-accept cost changes mean reported probabilities by .136 to .169, and averaged Brier error worsens by .0323. The decision loss is also policy-driven: model-returned actions under the high-cost prompt have loss 1.365 to 3.740 per issue, all above the reject-all loss of 1, while applying the identical cost-10 threshold
Load-bearing premise
The load-bearing premise is that the archived test-harness pass/fail result, used as the label for every probability and loss calculation, correctly describes whether the patch would fail; the authors did not rerun the full harness for all 720 candidates.
Editorial extensions
If this is right
- A one-prompt reviewer score is not a reusable software interface: if the elicitation policy is folded into the number, changing a downstream deployment's costs silently changes what the score means.
- Decision loss should be computed by applying a deterministic cost rule to policy-free probabilities, not by trusting the model-returned action; the fixed-rule comparison shows that the probability source itself contributes to high-cost loss.
- Separating reviewer, monitor, and controller makes negative results visible: at a 10:1 false-accept penalty, no tested pipeline can support approvals with these scores, so reject-all is the correct policy at that operating point.
- System evaluation should report reviewer-only, monitor-only, and fused probability error, since the monitor largely substitutes for weaker reviewers and only weakly complements the strongest.
- The evaluation recipe—vary the downstream policy while holding evidence fixed, compare with repeated calls, then apply the rule outside the model—can serve as a standard test for any LLM-generated probability consumed by a downstream decision.
Reading between the lines
- The same leakage likely occurs wherever LLM probabilities feed downstream decisions with their own costs—safety judging, triage, fraud screening, reward modeling, deferral—so the paper's matched policy-change test is a cheap way to check those domains too.
- The threshold-copying behavior of one reviewer suggests that printed numerical boundaries act as anchors; a factorial prompt that separates cost semantics from displayed threshold text could reveal whether hiding the number removes most of the effect.
- A natural next experiment would run the modular pipeline on natural failure rates and both-pass/both-fail candidate sets, since the balanced 50 percent benchmark may exaggerate or damp the zero-acceptance result at high costs.
- If policy-free probabilities prove stable under prompt variation, cached and cross-team risk scores become more trustworthy; that is a testable consequence of the paper's design even where the authors do not claim it.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a controlled set of experiments on four deployed LLM code-review interfaces. For 720 patches drawn from 360 SWE-rebench issues, each with one archived harness pass and one archived harness failure, the authors hold patch, code context, monitor card, system message, and all other prompt bytes fixed, and replace an equal-cost policy block with a 10:1 false-accept cost/threshold block. They report that this changes elicited failure probabilities by 0.136–0.169 on average; that model-returned actions under the high-cost prompt have higher loss than rejecting all; and that applying the same high-cost decision rule to equal-cost-elicited probabilities gives lower loss than using cost-10-elicited probabilities. They then evaluate a modular pipeline in which a risk-only reviewer score is combined with an independent monitor score and a coded decision rule, reporting improved Brier score and equal-cost loss but zero acceptance at 10:1. The paper includes extensive robustness work: repeat panels, threshold-precision audits, repository-cluster bootstrap and intersection-union tests, frozen calibration, nested refits, and a public reproducible release.
Significance. If the empirical claims hold, the paper makes a significant contribution to LLM-based code review and to any system that consumes LLM probability reports under different costs. The matched-call design is careful: it fixes essentially every user-message byte, uses exact 12-decimal thresholds, freezes calibration before evaluation, and reports repository-cluster intervals with intersection-union tests for all-reviewer claims. The public release with a verification script is a strength. The central qualitative finding—that a downstream cost/threshold block can move reported probabilities for fixed evidence—is robust to the main label concern. The quantitative decision-impact claims (Brier differences, loss comparisons, 'worse than reject-all') are conditional on the archived harness label being the target Y, and that is the point that needs further support before the paper's strongest conclusions can be accepted at face value.
major comments (2)
- [§4.1/A.2; Eq. (1)/(7)] The target Y in Eq. (1) is defined as failure on the archived evaluation harness, but Appendix A.2 states that the full harness was not rerun for the 720 candidates; only structural eligibility checks were performed. Appendix I.3 documents concrete cases where the archived harness is narrower than an intuitive semantic judgment. Every Brier-score and decision-loss result in §5.1–§5.3, Table 3, and Tables 11/16/18 is computed against this archived label. The qualitative 'policy moves probabilities' finding does not depend on Y, but the decision-impact claims do. If label errors are correlated with prompt condition, reviewer, or patch content, the effect sizes—including 'worse than reject-all' and the fixed-rule loss reductions—could reverse. The authors should rerun the frozen harness on all 720 candidates and report label changes, or provide a label-noise sensitivity analysis (e.g., flip
- [§5.2/Table 3] The fixed-rule comparison uses loss per issue where reject-all has loss 1 because the benchmark is balanced by construction (one pass and one fail per issue). This is acknowledged in §7.2 and §8, but the abstract and conclusion state the 'worse than rejecting all' and loss-reduction results without this caveat. The paper should either state these claims as benchmark-relative in the abstract/conclusion or provide a sensitivity analysis over failure prevalence, since reject-all loss in production depends on prevalence and costs.
minor comments (3)
- [§4.1 / B.1 / Table 6] The call inventory is internally inconsistent. Section 4.1 lists 720 risk-only calls plus 40 repeat measurements, and Table 6 shows the same. But B.1 says there are 720 risk-only calls: 160 on CAL_A, 160 on CAL_B, 400 on EVAL, and 40 repeats, which sums to 760. Table 8 also sums the risk-only rows to 760. Please clarify whether the 40 repeats are included in the 720 or additional.
- [Figure 3] The x-axis of panel (a) reads '1 2 3 5 10 200' while panel (b) reads '1 2 3 5 10 20'; presumably both should be the same cost sequence.
- [Abstract] 'worse than rejecting all patches' should carry the 'on the balanced benchmark' qualifier that appears in the contribution bullet and §7.2.
Circularity Check
No significant circularity: the central claims are empirical matched comparisons with calibration fits frozen before test evaluation; the only normative input is the explicit definition of a reusable probability, not a fitted or self-cited result.
full rationale
Walked the paper's derivation chain. The paper defines the reusable risk estimate pπ(X,S) in Eq. (1) and the Bayes threshold τ(C) in Eq. (2); these are explicit normative definitions, not quantities fit to data. The policy-dependence finding is an observed difference between matched prompt conditions (A vs C), quantified via Δabs, Δsigned, and ΔBrier in Eqs. (8)-(10); none of these expressions forces a sign, and the repeat-control comparison in §5.1 and Table 12 subtracts same-prompt repeat movement, so the excess-movement claim is not an artifact of metric construction. The fixed-rule loss comparison (Table 3, §D.1) holds the threshold and implementation identical and changes only the source of the elicited probability; the loss reduction is empirical, not guaranteed by construction. The modular-pipeline results use calibration fits (Eq. (14)) selected on CAL_A/CAL_B and frozen before EVAL, with repository-fold out-of-fold model selection and a nested refit bootstrap (§D.2, §G.1), so the Brier and loss improvements are out-of-sample. There are no load-bearing self-citations; references to SWE-rebench, OpenHands, Qwen, and model providers are independent external sources. The appendix limitations (A.2, I.3) about the archived harness label are data-quality/external-validity concerns, not circularity: a stale or narrow label would corrupt the outcome Y for all comparisons, but it does not make any comparison equivalent to its own input by definition. Overall, the paper's derivation chain is self-contained and its claims stand or fall on the empirical validity of the data and the controlled comparisons, not on circular reasoning.
Assumptions & free parameters
free parameters (2)
- Logistic combination coefficients beta0, betaR, betaM (per reviewer, Eq. 14) =
Fitted on pooled CAL; specific values not listed in text
- WARNING/CLEAR cutoff on M1 monitor score =
Threshold at least .10, selected by Youden's J on CAL_A
assumptions (4)
- standard math Bayes decision rule: reject when posterior reaches tau = CFR / (CFA + CFR)
- domain assumption The probability in Eq. (1) conditions on X, S, and pi but excludes decision costs
- domain assumption The archived executable-harness outcome is ground-truth Y
- domain assumption Reviewer responses come from a stable interface distribution with only repeat-level variation
Cite this review
Pith. "Pith review of When Policies Change Probabilities: Modular Decision-Making for LLM Code Review." pith.science (2026). https://pith.science/paper/BVSESOZA
@misc{pith2026260802677,
author = {Pith},
title = {Pith review of: When Policies Change Probabilities: Modular Decision-Making for LLM Code Review},
year = {2026},
howpublished = {\url{https://pith.science/paper/BVSESOZA}},
note = {Machine review of arXiv:2608.02677}
}
read the original abstract
LLM code reviewers often estimate patch risk and make approval decisions in one prompt. A probability should depend on evidence; costs should determine the action taken from it. We test whether four deployed reviewer interfaces preserve this separation using 15,792 responses on 720 candidate patches, with one that passed and one that failed an archived test harness for each of 360 repository issues. In matched calls with the patch and monitor evidence fixed, replacing an equal-cost policy with a 10:1 false-accept policy changes reported failure probabilities by 13.6 to 16.9 percentage points on average. For every reviewer, the actions returned under the high-cost prompt are worse than rejecting all patches. Applying the same high-cost rule to probabilities elicited under equal costs reduces loss for all four systems, showing that probability elicitation itself contributes to the excess loss. We also evaluate a modular pipeline that elicits risk without policy information, combines an independent monitor score, and applies costs in code. Relative to calibrated reviewer-only scores, the pipeline improves average probability accuracy and, at equal costs, reduces mean loss by .073 per issue while accepting 58 to 68% of patches. At 10:1, it accepts none and matches reject-all. Downstream policy can therefore change the probability it is meant to use, motivating separate evaluation of risk, outside evidence, and action.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Automating Low-Risk Code Review at Meta: RADAR, Risk Calibration, and Review Efficiency
Adams, Chris and Banga, Arjun Singh and Bansal, Parveen and Bhattacharya, Souvik and Bhuptani, Payal and Cao, Rujin and Canahuati, Pedro and others , year =. Automating Low-Risk Code Review at. 2605.30208 , archivePrefix=
-
[2]
Badertdinov, Ibragim and Golubev, Alexander and Nekrashevich, Maksim and Shevtsov, Anton and Karasik, Simon and Andriushchenko, Andrei and Trofimova, Maria and Litvintseva, Daria and Yangel, Boris , year =. 2505.20411 , archivePrefix=
-
[3]
Proceedings of the AAAI Conference on Artificial Intelligence , volume =
Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2025 , doi =
work page 2025
-
[4]
Proceedings of the 34th International Conference on Machine Learning , pages =
On Calibration of Modern Neural Networks , author =. Proceedings of the 34th International Conference on Machine Learning , pages =
-
[5]
2022 , eprint =
Language Models (Mostly) Know What They Know , author =. 2022 , eprint =
2022
-
[6]
and Wang, Zifan and others , year =
Kale, Neil and Zhang, Chen Bo Calvin and Zhu, Kevin and Aich, Ankit and Rodriguez, Paula and Knight, Christina Q. and Wang, Zifan and others , year =. Reliable Weak-to-Strong Monitoring of. 2508.19461 , archivePrefix=
-
[7]
Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration
Kim, Hankyeol and Kang, Pilsung , year =. Asking Is Not Enough: Protocol Sensitivity in. 2605.27752 , archivePrefix=
-
[8]
LLM-as-a-Verifier: A General-Purpose Verification Framework
Kwok, Jacky and Li, Shulu and Atreya, Pranav and Liu, Yuejiang and Jiang, Yixing and Finn, Chelsea and Pavone, Marco and Stoica, Ion and Mirhoseini, Azalia , year =. 2607.05391 , archivePrefix=
Show all 28 references
-
[9]
Landesberg, Eddie , year =. When. 2603.12520 , archivePrefix=
-
[10]
Proceedings of the AAAI Conference on Artificial Intelligence , volume =
Calibrating Large Language Models with Sample Consistency , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =. 2025 , doi =
2025
-
[11]
Papamarkou, Theodore and Alquier, Pierre and Bauer, Matthias and Buntine, Wray and Davison, Andrew and Dziugaite, Gintare Karolina and Filippone, Maurizio and Foong, Andrew Y. K. and Fortuin, Vincent and others , year =. Position: Agentic. 2605.00742 , archivePrefix=
-
[12]
and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and others , booktitle =
Wang, Xingyao and Li, Boxuan and Song, Yufan and Xu, Frank F. and Tang, Xiangru and Zhuge, Mingchen and Pan, Jiayi and others , booktitle =
-
[13]
Wang, Jiawei and Zhou, Yanfei and Devic, Siddartha and Fu, Deqing , year =. Are. 2601.07767 , archivePrefix=
-
[14]
2607.06065 , archivePrefix=
Wang, Ruoyu and Chen, Jierun and Wang, Shaowei and Tao, Chaofan and Yang, Sidi and Jiang, Yuxin and Yap, Kim-Hui and Shang, Lifeng and Li, Xiaohui and Bai, Haoli , year =. 2607.06065 , archivePrefix=
-
[15]
Beyond Accuracy: Policy Invariance as a Reliability Test for
Weng, Shihao and Feng, Yang and Xie, Xiaofei , year =. Beyond Accuracy: Policy Invariance as a Reliability Test for. 2605.06161 , archivePrefix=
-
[16]
Uncertainty Propagation in
Xia, Boming and Zhu, Liming and Gao, Erdun and Lu, Qinghua and Xue, Minhui and Sejdinovic, Dino , year =. Uncertainty Propagation in. 2604.23505 , archivePrefix=
-
[17]
When Agents Say One Thing and Do Another: Validating Elicited Beliefs from
Yamin, Khurram and Tang, Jingjing and Cortes-Gomez, Santiago and Sharma, Amit and Horvitz, Eric and Wilder, Bryan , year =. When Agents Say One Thing and Do Another: Validating Elicited Beliefs from. 2602.06286 , archivePrefix=
-
[18]
On Verbalized Confidence Scores for
Yang, Daniel and Tsai, Yao-Hung Hubert and Yamada, Makoto , year =. On Verbalized Confidence Scores for. 2412.14737 , archivePrefix=
-
[19]
2026 , eprint =
Code Review Agent Benchmark , author =. 2026 , eprint =
2026
-
[20]
1985 , doi =
Statistical Decision Theory and Bayesian Analysis , author =. 1985 , doi =
1985
-
[21]
and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R
Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R. , booktitle =
-
[22]
Calibrate-Then-Act: Cost-Aware Exploration in
Ding, Wenxuan and Tomlin, Nicholas and Durrett, Greg , year =. Calibrate-Then-Act: Cost-Aware Exploration in. 2602.16699 , archivePrefix =
-
[23]
2026 , howpublished =
2026
-
[24]
Grok on Amazon Bedrock , year =
-
[25]
Mistral Medium 3.5 , year =
-
[26]
2026 , howpublished =
Introducing. 2026 , howpublished =
2026
-
[27]
Gemini 3.5: Frontier Intelligence with Action , year =
-
[28]
2025 , howpublished =
2025
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.