Reasoning or Fluency? Dissecting Probabilistic Confidence in Best-of-N Selection

Hojin Kim; Jaehyung Kim

REVIEW 5 major objections 6 minor 10 references

Reasoning or Fluency? Dissecting Probabilistic Confidence in Best-of-N Selection

T0 review · 5 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Probability-based scoring in Best-of-N selection measures fluency, not the causal chain of reasoning — and a contrastive metric restores some of that signal.

desk verdict Good diagnostic idea and a plausible negative result for Qwen/Llama, but the evidence is undermined by a Table 1 data error and a probe too coarse to support the strong 'insensitivity to causality' claim. read the letter →

arxiv 2601.13735 v2 pith:ULDMHZU7 submitted 2026-01-20 cs.AI

Hojin Kim , Jaehyung Kim This is my paper

classification cs.AI

keywords Best-of-Nselectionprobabilisticconfidencechain-of-thoughtinter-stepcausalitylocalfluencycontrastivemetricattentionmaskingaccuracy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the probability-based scores used to pick the best of N LLM-generated reasoning traces are not actually tracking the logical links between reasoning steps. It shows that when those links are disrupted — by attention masking, shuffling, truncating, paraphrasing, or scoring with a much smaller model — selection accuracy barely changes, on average by less than 1% and sometimes improving. That implies the scores are mostly responding to local fluency and answer-format priors rather than reasoning coherence. The paper then proposes a contrastive metric that subtracts a masked score from the original score, which restores some sensitivity to inter-step structure and improves selection modestly. A sympathetic reader would care because inference-time selection is widely relied on, and this claim redefines what those confidence scores actually measure.

What carries the argument

The central object is the contrastive causality metric, Rcausal(y) = R(y) − α·R̂(y), where R is a standard probabilistic confidence score (e.g., self-certainty) and R̂ is the same score computed under attention-level masking that removes cross-step conditioning. The difference isolates the signal attributable to inter-step causal dependencies, with α controlling how much local fluency is subtracted away. The diagnostic machinery is the three-tier perturbation scheme — attention-level, parameter-level, and data-level disruptions — which provides the test for whether a metric's rankings actually depend on the structure it claims to measure.

What would settle it

Run the same disruption protocol with at least 20 independent seeds per condition and report per-question rank correlation between each confidence metric and the correctness of each candidate. If the rank correlation collapses when steps are masked, shuffled, or truncated — or if bootstrapped confidence intervals exclude zero difference from the unchanged condition — the paper's insensitivity claim would be refuted; if the rank correlations remain flat, the claim is supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that current probabilistic confidence metrics — self-certainty, log-likelihood, and entropy — rank Best-of-N candidates by surface fluency and prior likelihood rather than by the causal coherence of the reasoning trace. The evidence comes from three classes of disruptions that weaken the dependency between reasoning steps while keeping local text fluent: hard attention masks that prevent later steps from attending to earlier ones, evaluation by a much smaller model, and text-level changes such as shuffling, paraphrasing, and truncation. Across five reasoning benchmarks and three LLM families, selection accuracy degrades by less than 1% on average under these disr

Load-bearing premise

The claim rests on the assumption that a single binary correctness score per question, computed once over ten candidates, is sensitive enough to reveal whether the ranking signal was disrupted — differences of about a percentage point with no repeated runs or error bars could equally be noise.

Editorial extensions

If this is right

If the paper is right, Best-of-N pipelines using self-certainty, log-likelihood, or entropy are effectively selecting for fluent, format-typical text rather than logically sound reasoning.
The proposed contrastive metric restores measurable sensitivity to inter-step structure: it improves selection accuracy on average and degrades under shuffling or small-model evaluation, unlike the baseline metrics.
The truncation results imply that a short, locally fluent prefix of a reasoning trace carries most of the selection signal — competitive accuracy is already reached at 5k–10k tokens.
The finding reframes confidence scores: reliability of probability-based selection is not evidence of reasoning fidelity, so downstream applications should include perturbation checks before trusting the selected output.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same perturbation audit could be applied to learned reward models; if their rankings also survive shuffling and masking, the phenomenon would generalize beyond probability metrics to reward-based selection.
Editorial inference: the contrastive metric can be read as approximating the mutual information between a reasoning step and its predecessors; a testable variant would replace the masked score with a score computed on randomly reordered predecessor steps.
Editorial inference: the truncation results suggest a practical token budget for Best-of-N selection — if a short prefix ranks as well as the full trace, evaluation cost can be cut substantially without losing selection quality.
Editorial inference: the paper's own limitation section points to the natural next test — multi-hop and commonsense reasoning, where inter-step structure should matter more; a negative result there would falsify the generality of the insensitivity claim.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First: the core negative finding — that Best-of-N selection accuracy for Qwen and Llama barely moves when you mask cross-step attention during scoring — is real and worth knowing. The three-level perturbation design (attention, parameter, data) is a clean way to ask what probabilistic confidence metrics actually depend on. The new bit is applying this to confidence-based selection and the contrastive metric in Eq. 7. The Qwen/Llama evidence across Tables 1–3 is internally consistent, and the attention-masking result in particular is a meaningful strike against the idea that these metrics capture cross-step conditioning.

But the paper overstates what the data show, and there's a concrete data error that needs fixing. Table 1 lists Phi's 'original' column as identical to Qwen's numbers (42.93 on GPQA-D, 36.13 on GSM8K, etc.). Tables 2 and 8 give different Phi full-model values. So the Phi-specific degradation narrative in §4.2 and Figure 1 are artifacts of a copy-paste error, not a real result. That removes one of three model families from the empirical base and makes you wonder about the rest of the tables.

Even with the corrected tables, the evidence is weaker than the abstract suggests. The probe is top-1 selection accuracy after argmax over N=10 candidates. That is a thresholded outcome; it can stay constant even when the underlying scores change a lot but preserve the winner. The stress-test note is right: sub-1% accuracy changes do not discriminate between 'metric doesn't use cross-step context' and 'metric does, but the winner doesn't change.' Some baselines sit near chance (GPQA-D ~25%), so there's little selection signal to lose. No error bars, no seeds, no repeated runs, so the 'less than 1%' claim isn't statistically grounded.

The contrastive metric is a reasonable idea — subtract the masked score to isolate inter-step signal — but the empirical support is thin. The average gain is +0.39% (Table 4), negative for Qwen and Llama at alpha=0.5 according to Table 6, and the sensitivity story rests largely on one GPQA-D cell. That's not enough to claim 'more faithful output selection.'

The causal language also overreaches. Masking cross-step attention changes the conditional distribution of the scoring pass; that's a statistical intervention, not an identified causal effect on reasoning. The Limitations section concedes the conclusions are behavioral, not theoretical, so the abstract's 'inter-step causality' phrasing should be tempered.

Bottom line: the Qwen/Llama result is a useful caution for anyone using self-certainty or log-likelihood as a selection signal. It deserves a serious referee, but only after the Table 1 error is fixed, variance is reported, and the claims are scaled back to what a coarse argmax probe actually supports. If you work on inference-time compute or CoT faithfulness, read it — with the Table 1 correction in mind.

Referee Report

5 major / 6 minor

Summary. This paper studies whether probabilistic confidence metrics used for Best-of-N selection (self-certainty, normalized log-likelihood, entropy) capture inter-step causal structure in chain-of-thought reasoning. The authors introduce three classes of disruptions—attention-level masking of cross-step conditioning, parameter-level downscaling of the evaluator model, and data-level shuffling/truncation/paraphrasing—and measure selection accuracy on five reasoning benchmarks across Qwen, Llama, and Phi. They report that selection accuracy changes by less than 1% on average, concluding that current confidence metrics are largely insensitive to logical structure and instead act as proxies for local fluency and priors. They then propose a contrastive causality metric R_causal = R − α R̂ and report a small average accuracy improvement and apparent sensitivity to perturbations.

Significance. If substantiated, the central negative result would be important: it challenges the assumption that probability-based confidence is a faithful proxy for reasoning quality in inference-time scaling, and it would redirect research toward contrastive or verifier-based metrics. The study is well-scoped, uses publicly available benchmarks and models, and the three disruption families are clearly operationalized. The paper explicitly acknowledges its empirical-diagnostic nature and lists limitations. However, the evidence as presented is not yet strong enough to support the abstract's universal claim. The selection-accuracy probe is insensitive to score-level changes, the Phi results in Table 1 are internally inconsistent with later tables, and the reported differences are not accompanied by uncertainty estimates. The proposed metric's advantage is within the noise floor. With additional analyses and corrected tables, the work could make a solid contribution.

major comments (5)

[Table 1, §4.1–4.2] The central insensitivity claim rests on selection accuracy, a single binary outcome per question after argmax over N=10 candidates (defined in §4.1). This is a thresholded statistic: large perturbations to the underlying score distributions can leave the argmax unchanged if the winner's margin is large or rank order is preserved. Sub-1% average accuracy differences therefore do not discriminate between 'the metric does not use cross-step context' and 'the metric uses it but the winner is unaffected.' Please report score-level evidence: e.g., Spearman/Kendall rank correlation between original and masked candidate scores, the fraction of items where the selected candidate changes, and the distribution of margins. Without such measures, the conclusion in §4.5/Abstract that metrics are 'largely insensitive to logical structure' is not established.
[Table 1, Phi column; §4.2] The reported 'original' accuracies for Phi are numerically identical to the Qwen column (GPQA-D 42.93, GSM8K 36.13, LogiQA 48.37, AR-LSAT 21.73), yet Tables 2, 8, and 9 report Phi's original accuracies as 24.93, 31.63, 35.63, and 22.47 respectively (Table 2) and 0.217/0.306/0.316/0.252 (Table 8). This internal contradiction invalidates the Phi-specific degradation narrative in §4.2 ('6.9% decrease under masking'); with the corrected baseline, Phi's masking effect is much smaller and in some cases positive. The abstract's claim across 'three LLM families' therefore loses one of its three supports. Please correct all tables and re-evaluate the aggregate claims.
[§4.1, Tables 1–4] No variance estimates, multiple seeds, or replicate runs are reported. For a benchmark like GPQA-Diamond (198 questions, Table 5), a 1% change is about 2 questions; AR-LSAT (230) similarly. Many reported differences (e.g., Qwen AR-LSAT +2.90% under masking; Llama GSM8K −1.53% under q-masking) are within the sampling noise of a binomial proportion. Please provide bootstrap confidence intervals or at least standard errors, and ideally repeat selection with multiple N=10 samples (different random seeds) to establish that the <1% claim is not simply noise.
[§3.2, Eqs. (3)–(6) and §4.5] The paper equates removing autoregressive conditioning on prior steps with disrupting 'inter-step causal dependencies.' In an autoregressive LM, conditional dependence is statistical association, not an identified causal relation; attention masking removes the model's ability to condition on a token, which may affect local fluency signals (e.g., referential coherence) as much as logical reasoning. The conclusion that the metrics do not capture 'logical structure' or 'reasoning quality' therefore overreaches the operational intervention. A concrete test would be to compare the attention-masking effect with a control that preserves the conditional dependence structure but destroys logical content (e.g., replacing reasoning steps with semantically identical-but-non-sequitur statements) to see whether accuracy is sensitive to the latter.
[§5, Eq. (7) and Table 4] The proposed contrastive metric's advantage is not demonstrated. The average gain over self-certainty is +0.39 percentage points (38.72→39.11), well within the noise floor given the issues above, and several benchmark-level entries are flat or negative (e.g., GSM8K 33.07→32.97, AR-LSAT 19.73→19.27). The claim that the metric 'exhibits sensitivity to disruptions' is based on accuracy drops under shuffling/small-eval; but a metric that performs worse under perturbation could simply be less robust, not more sensitive to causal structure. Please report a direct measure of the contrast term's predictive value (e.g., correlation between R−αR̂ and correctness, or the fraction of selections that change when inter-step dependencies are perturbed) and provide uncertainty intervals.

minor comments (6)

[§4.2] The sentence 'Phi’s average accuracy of 23.9% on GPQA-Diamond' does not match any number in Table 1; please clarify.
[Appendix B.1] The heading 'αsensitivity' should read 'α sensitivity'.
[Table 6] Values such as .518 and .324 lack leading zeros and are inconsistently formatted with Table 7's 0.954 format.
[References] The author lists are malformed for some entries (e.g., 'Marah Abdin and 1 others', 'Jyoti Aneja Marah Abdin'); please fix.
[Figure 2] The caption refers to '(i) unaltered evaluation,' '(ii) attention-level disruption,' etc., but the figure itself is not labeled with (i)–(iv); consider adding labels.
[Figure 1 and §4.3] The term 'downscale' used in Figure 1 is not defined; the text refers to 'parameter-level disruption' using a smaller evaluator. Also, Abstract and §4.5 state 'less than 1% average selection accuracy loss,' but Table 1 shows an average increase of 0.75% for Qwen under masking; please phrase as 'absolute change' or specify the direction.

Circularity Check

1 steps flagged · score 2.0 of 10

Central insensitivity claim is externally benchmarked and non-circular; only the Eq. 7 contrastive metric's 'isolates causality' attribution is definitional (mild, not load-bearing).

self definitional [Section 5 (Eq. 7), building on Section 3.2 (Eqs. 3–6)]
"we introduce the contrastive causality metric: Rcausal(y) = R(y) − α· bR(y). (7) ... masked metrics are designed to explicitly remove inter-step causal effects. By taking the difference between these two quantities, we aim to decouple local fluency from the contribution of inter-step causal dependencies."

Section 3.2 defines the causal signal as exactly what the masking removes: removing conditioning on prior steps 'breaks inter-step causality (i.e., s_{k-1} not-> s_k)' and makes the masked score reflect 'the average local fluency of each individual step.' The proposed metric is therefore, by construction, the gap between the full score and the 'fluency-only' masked score, and the Abstract's claim that it 'explicitly isolates inter-step causal dependencies' is true only because 'causal contribution' was defined in Eqs. 3-6 as that gap. This is a labeling/construction circularity, not a fitted prediction: the metric's external validation (Table 4 selection accuracy) is independent and does not reduce to its definition.

full rationale

The headline finding -- that probabilistic confidence metrics are largely insensitive to inter-step structure -- is not circular. The probe is genuine: masked evaluation (Eqs. 3-6) removes cross-step conditioning from the score's computable inputs, and whether top-1 selection accuracy survives that change is an empirical fact measured against fixed external benchmarks (MATH-500, GSM8K, GPQA-D, LogiQA, AR-LSAT). The conclusion that metrics mostly rank by local fluency/priors is an inference from that observation, not an input to it. There are no load-bearing self-citations: no reference is authored by the present authors, and the self-certainty baseline (Kang et al.) is external prior work. The proposed metric's +0.39% improvement (Table 4) is likewise externally evaluated. The single mild circular element is Eq. 7: because Section 3.2 equates 'inter-step causality' with cross-step autoregressive conditioning, R - alpha*R-hat trivially 'isolates' that gap as a matter of definition; this attribution is self-definitional but does not carry the paper's central empirical claim. Separate correctness risks, which I flag but do not count as circularity: (i) selection accuracy is a single thresholded arg-max outcome per question over N=10 candidates, with no seeds or error bars, and several MCQ cells sit at or below chance, so sub-1% stability does not establish score-level insensitivity; (ii) Table 1's Phi 'original' column duplicates Qwen values on GPQA-D, GSM8K, LogiQA, and AR-LSAT and contradicts Tables 2/8/9, so the Section 4.2 Phi 6.9%-degradation narrative is unsupported by the paper's own data; (iii) the 'causal' language (s_{k-1} not-> s_k) overstates what is statistical conditioning. These points weaken empirical support but do not make the derivation circular.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the design choice of what counts as a 'causality disruption,' the coarse correctness-labeled selection-accuracy probe, and tooling assumptions (GPT-4o-mini paraphrases, punctuation segmentation, small-model priors). The only fitted constant is α=0.5, which is robustly ablated. No new physical or model entities are introduced; the contrastive metric is a method, not an entity.

free parameters (3)

α (contrastive metric weight) = 0.5
Hand-set 'for simplicity' in Eq. 7; robustness shown across [0.1,0.9] in Table 6, but the headline accuracy of the proposed metric depends on this choice, and α=1.0 sharply degrades Qwen.
Step segmentation boundary = punctuation
Reasoning traces are cut at punctuation into 'steps' (§4.1); this determines what attention-masking, shuffling, and paraphrasing perturb, and is a design choice that shapes all results.
Truncation retention lengths = 1k-30k tokens
Truncation experiment sweeps retention length; a diagnostic sweep rather than a fit, but the near-flat accuracy curve is defined over these chosen cutoffs only.

assumptions (5)

domain assumption Removing cross-step conditioning during scoring is equivalent to removing inter-step causal dependencies
§3.2, Eqs. 3-6: the paper equates attention masking / step-local scoring with breaking causality; conditional dependence in an autoregressive LM is a statistical association, not an identified causal relation.
domain assumption Selection accuracy on correctness-labeled benchmarks measures whether the metric captures reasoning quality
§4.1: accuracy is the fraction of questions where the selected output is correct; the paper infers reliance on reasoning fidelity from shifts in this coarse binary outcome over N=10 candidates.
domain assumption Shuffling/paraphrase/truncation preserve local fluency while destroying global coherence
§3.2, Appendix A.5: relies on GPT-4o-mini paraphrase quality (temperature/seed unspecified) and punctuation-based segmentation preserving sentence-level fluency; no validation that paraphrases retain logical content.
domain assumption The small evaluator (Qwen2.5-0.5B) retains fluency priors but loses long-range dependency modeling
§3.2: invoked via cited small-model surveys (Wang et al. 2024a; Ramezanali et al. 2025); the interpretability of the parameter-level perturbation depends on this premise.
ad hoc to paper Observed accuracy differences (0.2-1.3%) are attributable to treatment rather than sampling noise
Implicit in §4.2-4.4 and Figure 1; no error bars, seeds, or replicate runs are reported, so 'an average selection accuracy loss of less than 1%' has no statistical support.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Reasoning or Fluency? Dissecting Probabilistic Confidence in Best-of-N Selection." pith.science (2026). https://pith.science/paper/ULDMHZU7

@misc{pith2026260113735,
  author       = {Pith},
  title        = {Pith review of: Reasoning or Fluency? Dissecting Probabilistic Confidence in Best-of-N Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULDMHZU7}},
  note         = {Machine review of arXiv:2601.13735}
}

read the original abstract

Probabilistic confidence metrics are increasingly adopted as proxies for reasoning quality in Best-of-N selection, under the assumption that higher confidence reflects higher reasoning fidelity. In this work, we challenge this assumption by investigating whether these metrics truly capture inter-step causal dependencies necessary for valid reasoning. We introduce three classes of inter-step causality perturbations that systematically disrupt dependencies between reasoning steps while preserving local fluency. Surprisingly, across diverse model families and reasoning benchmarks, we find that selection accuracy degrades only marginally under these disruptions. Even severe interventions, such as applying hard attention masks that directly prevent the model from attending to prior reasoning steps, do not substantially reduce selection performance. These findings provide strong evidence that current probabilistic metrics are largely insensitive to logical structure, and primarily capture surface-level fluency or in-distribution priors instead. Motivated by this gap, we propose a contrastive causality metric that explicitly isolates inter-step causal dependencies, and demonstrate that it yields more faithful output selection than existing probability-based approaches.

Figures

Figures reproduced from arXiv: 2601.13735 by the authors.

**Figure 1.** Summary of our main results. Accuracy with Best-of-N selection using three probabilistic confidence metrics (Self-certainty, Log-likelihood, and Entropy) is insensitive to causal disruptions across a number of reasoning benchmarks, implying these metrics might not be capturing inter-step causality in CoT traces, but rather local fluency and prior likelihood. boost the performance of LLMs across diverse reasoning… view at source ↗

**Figure 2.** [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Paraphrased vs. original text selection accu [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

10 extracted references · 4 linked inside Pith

[1]

Preserveall mathematical meaningand symbols
[2]

Keep logical relationships intact
[3]

Make the wordingformal and clear
[4]

InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33759–33780

seqbench: A tunable benchmark to quantify sequential reasoning limits of llms. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33759–33780. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level...

arXiv 2025
[5]

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empiri- cal methods in natural language processing, pages 236...

arXiv 2018
[9]

Change phrasing, syntax, and structureas much as possible
[10]

Outputonly one rewritten sentence. Sentence: B Additional Results B.1αsensitivity of the contrastive causality metric We present an ablation study on the weighting pa- rameter α used in the contrastive causality metric. Recall that the metric interpolates between stan- dard self-certainty and attention-masked signal as defined in Eq. 7. In particular, whe...
[2021]

Qiwei Di, Kaixuan Ji, Xuheng Li, Heyang Zhao, and Quanquan Gu

Training verifiers to solve math word prob- lems.arXiv preprint arXiv:2110.14168. Qiwei Di, Kaixuan Ji, Xuheng Li, Heyang Zhao, and Quanquan Gu. 2025. Best-of-majority: Minimax- optimal strategy for pass@k inference scaling.arXiv preprint arXiv:2510.03199. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akh...

arXiv 2025

Show all 10 references

[2023]

InThe Twelfth Inter- national Conference on Learning Representations

Let’s verify step by step. InThe Twelfth Inter- national Conference on Learning Representations. Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading compre- hension with logical reasoning.arXiv prepri...

2020 arXiv
[2025]

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa

Scalable best-of-n selection for large lan- guage models via self-certainty.arXiv preprint arXiv:2502.18581. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners.Advances in neural informatio...

2022

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Preserveall mathematical meaningand symbols

[2] [2]

Keep logical relationships intact

[3] [3]

Make the wordingformal and clear

[4] [4]

InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33759–33780

seqbench: A tunable benchmark to quantify sequential reasoning limits of llms. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33759–33780. David Rein, Betty Li Hou, Asa Cooper Stickland, Jack- son Petty, Richard Yuanzhe Pang, Julien Dirani, Ju- lian Michael, and Samuel R Bowman. 2024. Gpqa: A graduate-level...

arXiv 2025

[5] [5]

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empiri- cal methods in natural language processing, pages 236...

arXiv 2018

[6] [9]

Change phrasing, syntax, and structureas much as possible

[7] [10]

Outputonly one rewritten sentence. Sentence: B Additional Results B.1αsensitivity of the contrastive causality metric We present an ablation study on the weighting pa- rameter α used in the contrastive causality metric. Recall that the metric interpolates between stan- dard self-certainty and attention-masked signal as defined in Eq. 7. In particular, whe...

[8] [2021]

Qiwei Di, Kaixuan Ji, Xuheng Li, Heyang Zhao, and Quanquan Gu

Training verifiers to solve math word prob- lems.arXiv preprint arXiv:2110.14168. Qiwei Di, Kaixuan Ji, Xuheng Li, Heyang Zhao, and Quanquan Gu. 2025. Best-of-majority: Minimax- optimal strategy for pass@k inference scaling.arXiv preprint arXiv:2510.03199. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akh...

arXiv 2025