{"id":"76f37082-5093-4223-88c0-13b3238ef7d4","arxiv_id":"2508.21787","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Selecting the candidate that maximizes the sum of reasoning log-likelihood and answer log-likelihood improves best-of-n accuracy over majority voting and self-consistency on reasoning benchmarks.","lead":"The paper presents PiCSAR, a training-free way to pick the best answer from several LLM or reasoning-model attempts by scoring both the reasoning chain and the final answer with their log-likelihoods. On math benchmarks it often beats majority-vote baselines while using fewer sampled answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PiCSAR-N result (89.67%) exceeds the oracle upper bound (84.00%) on MATH500 for Qwen3-8B in Table 6, an impossible outcome that undermines the empirical evidence for the central claim.","rationale":"The reader's verdict was CONDITIONAL, citing the internal inconsistency and missing code. I agree with that verdict, but I identify the internal inconsistency as the single most load-bearing concern about the central claim. The theoretical concern about p(y|⟨a⟩,r,x) approximating p(y|r,x) is real but secondary: even if the score is not a true MAP objective, the method could still be a useful heuristic, and the empirical results would remain the primary evidence. The impossible PiCSAR-N result cannot be explained by the theoretical framing; it directly contradicts the paper's own upper-bound definition. If the result is not a typo, then the experimental pipeline is unreliable, which would invalidate the headline accuracy claims. If it is a typo, the paper must correct it, and the absence of code/data makes verification impossible. Therefore, the condition for acceptance should be a public correction and reproducible release, not merely a theoretical clarification. The verdict stays CONDITIONAL because the method is plausible and most other reported numbers are directionally consistent, but the anomaly must be resolved before full acceptance.","tokens_in":31996,"tokens_out":3613,"duration_ms":39079,"concrete_test":"Re-run Qwen3-8B (thinking mode) on MATH500 with k=6 using the paper's protocol: sample the same candidate pool, compute the pass@k upper bound, and evaluate PiCSAR-N accuracy on the identical pool with identical answer extraction. If PiCSAR-N accuracy exceeds the recomputed upper bound, the reported Table 6 entries are invalid; also inspect the raw logs for this row to check for a typo or an evaluation mismatch (e.g., different candidate sets or answer formats).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that PiCSAR's joint confidence score selects better reasoning chains than baselines. The paper's own oracle upper bound (pass@k) is the maximum accuracy attainable given the sampled candidates. In Table 6 (LRMs), for Qwen3-8B on MATH500 with k=6, PiCSAR-N is reported at 89.67±0.37 while the upper bound is 84.00±0.12. A selection method cannot exceed the oracle; this is an internal contradiction in the headline results. Either the PiCSAR-N number, the upper-bound computation, or the evaluation protocol is wrong. Because the claimed gains (e.g., +10.18 on MATH500, +9.81 on AIME2025) rest on these measurements, this impossible result makes the empirical evidence unreliable absent a correction and code/data release. It is more load-bearing than the prompted-answer approximation concern because it directly invalidates the accuracy numbers rather than only the theoretical framing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PiCSAR, a training-free best-of-n selection rule for reasoning chains. For each sampled chain r and extracted answer y, the method scores the pair as log p(r | x) + log p(y | <a>, r, x), where the second term is computed by appending an instruction prompt <a> after the chain. The authors report substantial accuracy gains over Self-Consistency and other baselines across five LLMs and three LRMs on MATH500, GSM8K, SVAMP, GPQA-Diamond, AIME2024, and AIME2025, and claim sample efficiency (e.g., k=6 outperforming k=16/32 baselines). They also present an information-plane analysis, decoupled-generator/evaluator experiments, calibration regressions, and a sentence-level peak analysis. The central claim is that jointly maximizing reasoning confidence and answer confidence is a better selection heuristic than answer-frequency or single-term confidence.","tokens_in":32274,"tokens_out":5038,"duration_ms":55624,"significance":"If the empirical results are correct, the method is simple, parameter-free, and broadly applicable, and the paper would offer a useful inference-time tool. The evaluation is broad (multiple model families, multiple benchmarks, ablations, error bars), and the decoupled-confidence analysis is a valuable practical contribution. However, the current manuscript contains an internal contradiction in a headline result: in Table 6 (and Table 2), PiCSAR-N for Qwen3-8B on MATH500 is reported at 89.67%, exceeding the reported oracle pass@k upper bound of 84.00%. A selection method cannot beat the oracle over the same candidate set. This makes the reported gains unreliable as printed. The paper also contains a suspicious exact duplication of numbers across two model rows in Table 1. These issues are load-bearing for the empirical claim, so the manuscript needs correction and verification before the significance can be credited.","major_comments":[{"comment":"For Qwen3-8B on MATH500, PiCSAR-N is reported as 89.67±0.37 while the Upper Bound is 84.00±0.12. The Upper Bound is the oracle pass@k accuracy over the same candidate set; no selection rule can exceed it. This is an internal contradiction in the headline results. Since the abstract’s claimed gains (e.g., +10.18 on MATH500) and the LRM superiority claims rest on these numbers, the empirical evidence is unreliable until the evaluation is corrected, re-run, or explained with code/data release.","section":"Section 4, Table 6; also Table 2"},{"comment":"The Llama-3.1-70b PiCSAR-N row duplicates the Qwen3-8b PiCSAR-N row exactly in the SVAMP and GPQA-Diamond columns (e.g., 94.44±0.11 / 94.56±0.59 and 47.98±1.01 / 44.95±0.58). Such an exact match across different models is implausible and suggests a transcription or table-generation error. The full table needs to be audited, and any corrected values should be supplied.","section":"Section 4, Table 1; Appendix C, Table 5"},{"comment":"The paper correctly notes that log p(y | r, x) is not well-defined because the model can continue generating after the reasoning chain, then replaces it with log p(y | <a>, r, x). No proof or calibration is provided that this prompted estimate faithfully approximates the conditional in Eq. (1). As a result, Eq. (3) is not, strictly speaking, the joint MAP objective claimed in Eq. (1); it is a heuristic score. This does not necessarily invalidate the empirical method, but the theoretical framing overstates what is established. The authors should either reframe the method as a heuristic or provide evidence (e.g., sensitivity to the wording of <a>) that the substitution is stable.","section":"Section 2.2, Eq. (3)"}],"minor_comments":[{"comment":"The text says \"PiCSAR and PiCSAR-N outperform all baselines in 37/40 comparisons,\" but Table 1 shows a counterexample: Qwen3-32b on MATH500 at k=16/32, where Self-Consistency (76.27) exceeds both PiCSAR (75.93) and PiCSAR-N (75.13). The aggregate claim needs to be recomputed or qualified.","section":"Section 4"},{"comment":"The benchmark name \"SVAMP\" is consistently typeset as \"SV AMP\" throughout the paper.","section":"Section 3"},{"comment":"Several figure captions appear mislabeled: Figures 10–12 are all captioned \"Gemma-2-9b on GSM8K\" but show different statistics, and Figures 18–20 are all captioned \"DS-Distilled-Llama-8b on AIME2024.\" These captions should be corrected.","section":"Appendix D"},{"comment":"The method name is inconsistently capitalized: Section 5 begins with \"PICSAR\" instead of \"PiCSAR.\" Also, Appendix C.1 contains the phrase \"similar to LRM\" which appears to be a typo.","section":"Section 5"},{"comment":"The claimed gains \"+10.18 on MATH500\" and \"+9.81 on AIME2025\" are not tied to a specific model and baseline in the abstract. Please make the comparison explicit.","section":"Abstract and Section 1"},{"comment":"The phrase \"at least 2x fewer samples\" is imprecise: k=6 versus k=16/32 is a 2.7x/5.3x reduction. Please state the exact comparison protocol and verify the 16-out-of-20 count.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The impossible upper-bound result in Table 6 is likely a typo or an evaluation-pipeline bug, but it is serious enough that the paper should not be accepted without a corrected table and ideally code/data release. If the authors can rerun the affected experiments, the method may still be sound; if the discrepancy cannot be explained, the empirical contribution is in question. The exact duplication of numbers across model rows in Table 1 further suggests a data-handling issue that needs full auditing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes PiCSAR, a training-free best-of-n scoring rule: select the reasoning chain maximizing log p(r|x) + log p(y|<a>,r,x), i.e., reasoning log-likelihood plus a separately prompted answer log-likelihood. The idea is simple, and the paper is worth reading because it shows this joint score helps across a range of LLMs and LRMs, with ablations indicating both terms matter. The information-plane analysis is a nice way to visualize why high joint confidence correlates with accuracy. The method has no fitted parameters and is not tuned to the benchmarks, so the central claim is not circular.\n\nThe main soft spot is worse than the reader's flagged concern about the prompted-answer approximation. In Table 6, for Qwen3-8B on MATH500 with k=6, PiCSAR-N is 89.67±0.37 while the paper's own upper bound is 84.00±0.12. A selection method cannot exceed the oracle on the same candidate set. This is an internal contradiction in the headline results. Either the PiCSAR-N number, the upper-bound computation, or the evaluation protocol is wrong. Because the paper's key claims rest on these measurements, the error is load-bearing, not a minor typo. A correction and code/data release are needed before the empirical claims can be trusted.\n\nThe prompted-answer approximation is a genuine theoretical soft spot: p(y|r,x) is not well-defined as a conditional for a model that can keep generating, and the paper substitutes p(y|<a>,r,x) without a calibration argument. In practice the approximation is reasonable and the ablations support the method, but it should be acknowledged more clearly as a modeling choice rather than a well-defined MAP objective.\n\nMinor issues: no significance testing for several headline gains (though error bars are shown and many differences are directionally consistent), and no code/data release.\n\nWho is this for? Researchers working on inference-time selection for reasoning; it is a useful incremental contribution to best-of-n selection. It deserves a serious referee: the method is plausible, the experiments are broad, and the inconsistency is fixable. But I would not accept it without a corrected table and a release of the evaluation code.","headline":"PiCSAR has a plausible training-free scoring rule and broad experiments, but an impossible result in Table 6 undermines the headline numbers until corrected.","tokens_in":32770,"tokens_out":2674,"would_cite":false,"duration_ms":29866,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ranking a reasoning chain by its joint probability with the final answer — not by majority vote — picks more correct solutions, with gains up to 10.18 points on MATH500 and 9.81 on AIME2025.","keywords":["best-of-n sampling","reasoning chain selection","self-consistency","joint log-likelihood scoring","answer confidence","chain-of-thought","large reasoning models","inference-time scaling"],"falsifier":"Prompt-swap test: re-run PiCSAR on MATH500 with two or three different phrasings of the instruction ⟨a⟩, plus a no-instruction condition that scores the answer's forced-continuation probability immediately after the chain. If selection accuracy shifts substantially across phrasings, or diverges from the no-instruction estimate, the answer-confidence term is measuring the instruction artifact rather than p(y|r,x), and the joint-MAP justification is unsupported.","tokens_in":31913,"feed_emoji":"🎯","tokens_out":11758,"duration_ms":108720,"temperature":0.7,"pith_summary":"PiCSAR claims that which reasoning chain you trust matters as much as how many chains you sample, and that the right choice is the candidate with the highest joint probability of chain and final answer. The paper decomposes that joint criterion into two computable signals — reasoning confidence (how likely the chain is given the prompt) and answer confidence (how likely the answer is given the chain) — and uses their sum as a training-free scoring rule for best-of-n sampling. Across five LLMs and three LRMs on math and science benchmarks, the rule reports consistent gains over majority-vote Self-Consistency, prompt-consensus USC, and KL-divergence Self-Certainty, with the largest improvements on hard sets: +10.18 on MATH500 and +9.81 on AIME2025, often while using fewer samples than the baselines. If the rule works as claimed, strong reasoning selection needs no trained reward model — and, the paper's decoupling analysis suggests, not even the same model that produced the chains.","feed_headline":"Joint likelihood of chain and answer beats majority voting","feed_subtitle":"Training-free scoring lifts hard math benchmarks by up to ~10 points, often with half the samples.","key_machinery":"The central object is the scoring rule Score(r,y) = log p(r|x) + log p(y|⟨a⟩,r,x): a reasoning-confidence term (the chain's token-level log-probability given the prompt, optionally length-normalised as PiCSAR-N) plus an answer-confidence term (the probability of the extracted answer when the chain resumes after an instruction prompt ⟨a⟩). The decomposition mirrors the chain rule p(r,y|x) = p(r|x)·p(y|r,x), so the rule is a sampling-based approximation to MAP decoding over (chain, answer) pairs. Its supporting empirical device is the 'information plane', a 2D scatter of the two terms whose quadrant analysis shows correct answers in the high-high region; raising thresholds to the 75th percenti","core_discovery":"Central claim: picking among k sampled reasoning chains should be approximate MAP decoding — choose the candidate maximising log p(r|x) + log p(y|r,x), the chain's likelihood plus the answer's likelihood given the chain. Since p(y|r,x) is undefined, the paper estimates it with an appended instruction prompt ⟨a⟩ requesting the final answer. The empirical payload is an 'information plane': correct answers cluster where both terms are high; reasoning confidence is the stronger predictor (t ≈ 9.1 vs ≈ 4.8), answer confidence the fine-grained tie-breaker. The paper argues this joint rule beats Self-Consistency, USC, and Self-Certainty in 37/40 LLM and all 18 LRM comparisons, with +10.18 on MATH50","pith_inferences":["The paper's own ablations show answer confidence alone occasionally beats the joint rule (GPQA-Diamond at k = 32 for several models); a natural extension is to test whether the fixed unit-weight sum can be improved by a tuned weighting or by combining the two signals as a committee.","If the prompted estimate of p(y|r,x) is faithful, the same chain-rule decomposition transfers to other structured generations — code, proofs, plans — where a verifiable artifact plays the role of the answer, giving a training-free verifier signal.","The decoupling results suggest distilling the answer-confidence term into a tiny dedicated evaluator, decoupling generation cost from evaluation cost more aggressively than the paper's cross-pollination experiments.","The peak-to-sentence dynamics point to a stop-time policy: halt generation when the running answer-confidence sequence peaks, converting PiCSAR from a select-among-k rule into a controller for test-time compute."],"forward_implications":["Sample efficiency: with only k = 6 candidates, PiCSAR beats Self-Consistency and Self-Certainty running on k = 16 or 32 in 16 of 20 comparisons, implying better selection can substitute for a larger sampling budget.","Portable confidence: answer confidence computed by a different, smaller model preserves most of the accuracy, enabling cheap offloading of evaluation to a local model while a large model generates.","Confidence as filter: thresholding the two confidence terms at the 75th percentile isolates near-perfect-accuracy subsets (100% on AIME2025 for DS-Distill-Qwen-2.5-7B), supporting abstention and selective answering.","Intra-model, not inter-model: confidence scores predict correctness within a model (positive logistic-regression slopes across all sizes) but are not comparable across models, so cross-model ranking by score is not valid.","Density over length: higher peak-to-sentence ratios of high-confidence steps correlate with accuracy, and the lowest-ranked chains are the longest — extra reasoning length does not monotonically help."],"supporting_citations":[{"why":"Self-Consistency — the primary majority-vote baseline that PiCSAR's joint score is designed to outperform across every benchmark.","marker":"Wang et al., 2023b"},{"why":"Universal Self-Consistency — the prompt-based consensus baseline; the paper argues it discards the reasoning signal PiCSAR exploits.","marker":"Chen et al., 2023"},{"why":"Self-Certainty — the KL-divergence-confidence baseline that PiCSAR must beat in head-to-head comparisons.","marker":"Kang et al., 2025"},{"why":"The prior result that log-likelihood alone fails as a scoring rule, which motivates PiCSAR's added answer-confidence term.","marker":"Lewkowycz et al., 2022"},{"why":"Introduces best-of-N sampling, the selection framework within which PiCSAR's scoring rule operates.","marker":"Stiennon et al., 2020"},{"why":"DeepSeek-R1 distilled reasoning models — the LRMs where PiCSAR reports its largest gains.","marker":"Guo et al., 2025"},{"why":"Qwen3 models and their thinking-mode sampling settings, which supply both LRM results and generation hyperparameters.","marker":"Yang et al., 2025a"},{"why":"MATH500 — the benchmark of one headline gain (+10.18) and the source of the information-plane analyses.","marker":"Hendrycks et al., 2021"}],"fun_headline_variants":["Score reasoning chains by joint likelihood, not votes","Joint chain and answer likelihood beats majority voting","Pick chains via MAP: reasoning plus answer likelihood","Reasoning confidence + answer confidence selects winners","Training-free scoring: joint likelihood lifts maths by ~10"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The rule's load-bearing premise is that the probability the model assigns to the extracted answer after an appended 'output the final answer' instruction faithfully measures its true confidence in that answer given the reasoning chain — even though the instruction was absent during generation and the model could have continued writing after the answer.","fun_headline_variants_meta":{"raw":{"variants":["Score reasoning chains by joint likelihood, not votes","Joint chain and answer likelihood beats majority voting","Pick chains via MAP: reasoning plus answer likelihood","Reasoning confidence + answer confidence selects winners","Training-free scoring: joint likelihood lifts maths by ~10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1129,"prompt_tokens":710,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":348}},"tokens_in":454,"tokens_out":419,"duration_ms":5611,"temperature":1.0,"reasoning_tokens":348,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:55:40.179461+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Prompt-swap test: re-run PiCSAR on MATH500 with two or three different phrasings of the instruction ⟨a⟩, plus a no-instruction condition that scores the answer's forced-continuation probability immediately after the chain. If selection accuracy shifts substantially across phrasings, or diverges from the no-instruction estimate, the answer-confidence term is measuring the instruction artifact rather than p(y|r,x), and the joint-MAP justification is unsupported.","supporting_citations":[],"review_version":2}