Logical Reasoning with Outcome Reward Models for Test-Time Scaling

Ehsan Shareghi; Ramya Keerthy Thatikonda; Wray Buntine

REVIEW 3 major objections 5 minor 23 references

Logical Reasoning with Outcome Reward Models for Test-Time Scaling

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Outcome reward models trained on deliberately wrong chain-of-thought outperform majority voting on deductive reasoning.

desk verdict A useful, honest empirical paper on ORMs for logical reasoning; the echo-negative idea is genuinely new, but the manuscript needs variance estimates and a cleaner validation split before the central claim is fully convincing. read the letter →

arxiv 2508.19903 v1 pith:Y2SNHPSK submitted 2025-08-27 cs.CL cs.AI

Ramya Keerthy Thatikonda , Wray Buntine , Ehsan Shareghi This is my paper

classification cs.CLcs.AI

keywords outcomerewardmodellogicalreasoningtest-timescalingchain-of-thoughtechogenerationBest-of-Nselectiondeductivetraining

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether outcome reward models—scorers that judge a whole reasoning chain by whether its final answer is correct—can make test-time scaling work for deductive logical reasoning. It shows that training such a scorer on multiple chain-of-thought samples, plus a new source of hard negatives called echo chain-of-thought, lets Best-of-N selection beat majority voting on FOLIO, JustLogic, and ProverQA across four different LLMs. The echo trick deliberately whispers an incorrect answer into the prompt, records the resulting flawed but plausible rationale, keeps only those errors the model's own judge fails to flag, and trains the reward model to reject them. The largest gains appear on ProverQA and JustLogic; on FOLIO, where reasoners already get nearly 90% of rationales right, there is little headroom.

What carries the argument

Echo Chain-of-Thought (EcCoT) data generation is the central mechanism: the generator is prompted with a predetermined answer label (True, False, or Uncertain), producing reasoning that 'echoes' that label even when it is wrong; an LLM judge then filters out errors that are too obvious, keeping only hard negatives. These are combined with standard multi-sample CoT traces to train an outcome reward model, and at inference the ORM's logits rank N sampled reasonings (Best-of-N).

What would settle it

Compute the distribution of errors the four reasoners actually make at test time on ProverQA and JustLogic and measure its overlap with the echo-hard negatives in the training set (for example, by n-gram or semantic similarity). If the two sets are largely disjoint—for instance, if echoed errors all follow the 'answer is X' template—the ORM would be learning to reject a synthetic artifact, and the reported Best-of-N gains would not transfer to new reasoners or datasets. A second check: verify the LLM judge's Correct/Incorrect flags against gold labels on a small annotated sample.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that outcome supervision works for logical reasoning when the training data is diverse enough, and that the cheapest source of that diversity is the model's own echoic obedience. Prompting a generator with 'Given the answer is True/False/Uncertain' coerces it into producing reasoning that unjustifiably supports that answer; a second judge stage keeps only those erroneous trajectories the model itself cannot recognize as wrong. Folded into an ORM training set along with ordinary CoT candidates, these echo-hard negatives make the reward model better at ranking candidate reasonings at test time, pushing Best-of-N accuracy above majority vote. The p

Load-bearing premise

Echoed rationales that survive the model's own judge filter match the kinds of mistakes reasoners actually make at inference, so training an ORM to reject them transfers to Best-of-N decoding.

Editorial extensions

If this is right

Best-of-N selection with an ORM now plausibly beats majority voting on deductive-reasoning benchmarks, so compute budget at test time can be spent on sampling plus ranking instead of fine-tuning the reasoner.
Smaller reasoners benefit most: the Gemma 1B ablation gains roughly 30 points on ProverQA when ranked by the EcCoT ORM.
On JustLogic the CoT ORM already matches the Highest Threshold, meaning the verification bottleneck has been removed and further gains must come from the reasoner itself.
The echo technique yields more diverse training traces (lower self-BLEU) than simply sampling more CoT, which the 40-versus-10 FOLIO ablation suggests is the operative factor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If echo-hard negatives are actually close to real reasoner error modes, the same trick could supply negative data for process reward models, which inspect intermediate steps rather than final answers.
The paper's own BLEU-based resampling suggests diversity, not volume, drives ORM quality; a stronger test would compare two training sets matched for size, one echo-augmented and one size-matched CoT, across multiple reasoners.
The judge-filter stage is the fragile link: because the judge is the same model that generated the echoes, its mistakes set a ceiling on which negatives survive; using a stronger or external judge could either help by better filtering or hurt by removing useful hard negatives.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a decent, honest empirical paper. The genuinely new piece is the echo-generation tactic—prompting the generator with a deliberately wrong answer, then keeping only the flawed rationales that the generator's own judge fails to flag as incorrect. That is a real twist on hard-negative mining for reward models, and it is not in the cited prior work.

What the paper does well: the evaluation is more thorough than the typical ORM paper. Four reasoners, three datasets, a controlled comparison of echo data against simply adding more CoT samples (Appendix C), and a Gemma size sweep that shows the gains concentrate on smaller models. The authors also report an honest ceiling analysis: FOLIO reasoners already produce correct rationales in roughly 90% of samples, so the ORM has little room to improve there. That is the right way to present a near-ceiling result.

The soft spots are real but addressable. First, the abstract overclaims FOLIO; the body itself explains why the gains are small. That should be fixed before publication. Second, there are no confidence intervals or significance tests anywhere, and several reported gaps are small enough that noise could matter. Third, and most important: the JustLogic echo resampling size was chosen by downstream benchmark performance, and the split used in that search is not disclosed. If the test set leaked into that choice, the JustLogic results are selection-on-test. The authors need to say which split they used and, ideally, do the selection on a held-out validation set. Fourth, the self-judge filter is never validated against gold labels, and the paper never measures whether echoed errors resemble the errors reasoners actually make at inference. That is the load-bearing assumption behind the method. It is not a refutation—the ProverQA gains and the multiple-reasoner consistency make the central effect plausible—but the current evidence is only indirect (self-BLEU diversity, Appendix D).

My overall read: the central argument holds up conditionally. The contribution is contained but genuine, and the weaknesses are the kind a careful revision can fix. I would send this to peer review rather than desk reject. If I worked on reward models or test-time scaling, I would cite it, especially for the echo strategy. I would also bring it to a reading group: the method is easy to explain and the limitations discussion is worth debating.

Referee Report

3 major / 5 minor

Summary. The paper proposes Outcome Reward Models (ORMs) for deductive logical reasoning, trained on Chain-of-Thought (CoT) data and on a novel 'Echo Chain-of-Thought' (EcCoT) augmentation. In EcCoT, the generator is prompted with a supplied answer (True/False/Uncertain) to elicit rationales that may be forced, and an LLM judge then discards echoes it can identify as incorrect. ORMs are trained on Qwen2.5-7B with LoRA and evaluated by Best-of-N re-ranking of N samples from four reasoners (Qwen2.5-7B, Qwen3-8B, LLaMA-3.1-8B, GPT-4o) on FOLIO, JustLogic, and ProverQA. The paper reports that ORMs trained on CoT and EcCoT data improve over majority voting, with EcCoT-trained ORMs generally best, and includes ablations on training sample size and reasoner size.

Significance. Test-time scaling for logical reasoning is under-explored, and outcome reward models offer a practical verification mechanism. The paper's strengths include evaluation on external test splits of three public benchmarks, a reasoner set spanning four LLMs, comparisons against majority voting and highest-threshold ceilings, and two useful ablations: controlling for the number of CoT samples and sweeping reasoner size on Gemma. The code is released. However, the main novelty, EcCoT, is not yet convincingly validated as a source of hard negatives: the judge-filter accuracy is unmeasured, the overlap between echo-generated errors and inference-time reasoner errors is not analyzed, and the JustLogic echo-sample size is selected by downstream performance without specifying the split. These issues bear directly on the strength and generalizability of the central claim.

major comments (3)

[Sec. 2, EcCoT data generation; Appendix D] The mechanism behind EcCoT is that self-judged 'hard' echo errors resemble errors reasoners actually make at inference. The paper never validates the LLM judge's flags against gold labels, nor does it measure whether retained echoes occur in the reasoners' unassisted CoT outputs. The only evidence offered is self-BLEU diversity (Table 5), which is indirect and internally inconsistent: for FOLIO 40, EcCoT self-BLEU is 0.93 vs CoT's 0.92, so EcCoT is less diverse, contradicting the claim that 'EcCoT consistently produces more diverse reasoning paths.' Please add (i) judge accuracy on a labeled sample, (ii) an overlap analysis between retained echo errors and real reasoner error modes, and (iii) an ablation that removes the judge filter. Without these, the improvement attributed to filtered echo data is not causally supported.
[Appendix B, Figure 5] The JustLogic EcCoT configuration uses 10,000 resampled echoes, selected because 10k outperformed 20k and 30k in an empirical analysis. If this selection was made on the test split, the reported JustLogic results are optimistic and the method has an additional free parameter tuned on the evaluation set. The manuscript must state which split was used for this selection. If no validation split was used, the JustLogic EcCoT results and the abstract claim of improved performance on JustLogic are not unbiased. This is load-bearing because the sample-size choice is part of the reported EcCoT model.
[Results and Discussion, Figs. 2-7] No error bars, confidence intervals, or significance tests are reported. Given that FOLIO is small and the paper's own analysis shows reasoners are already correct in roughly 90% of samples, the FOLIO EcCoT improvement could be within sampling noise. The claim that EcCoT 'consistently outperforms' both majority voting and CoT ORM needs variance support. Please report per-seed variance or bootstrap intervals over test instances, state the number of test examples per benchmark, and report the number of independent ORM training runs.

minor comments (5)

[Sec. 2, Figure 1 and Tables 1-4] The text says the Echo prompt uses 'Given the answer is True' for all reasoning questions, but Tables 1-4 report Echo modes for True, False, and Uncertain. Please clarify whether all three labels were used as misleading answers.
[Appendix C, Figure 6 caption/text] The sentence 'increasing the number of ORM samples does not outperform the Echo samples for both LLaMA and Qwen reasoners, where the performance was lower compared to CoT' is ambiguous. Please clarify which configuration is lower than which.
[Sec. 4.2, Figure 4 caption] The caption says 'FOLIO achieves close to 32 correct answers on average,' while the text defines the metric as the average number of correct rationales across N samples. Align the caption with the definition and report the corresponding accuracy or proportion.
[Appendix D, Table 5] The FOLIO 40 row shows EcCoT self-BLEU (0.93) worse than CoT self-BLEU (0.92), which contradicts the claim that EcCoT consistently produces more diverse reasoning paths. Correct the claim or explain why this row is an exception.
[Sec. 3, Experimental Setup] Please report the LoRA rank/alpha, the number of training examples after filtering for each ORM, and the train/validation split used for ORM training and for the sample-size selection in Appendix B. This information is needed for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central ORM claims are empirical, evaluated on held-out test splits, and no prediction reduces by construction to a fitted input or self-citation.

full rationale

The paper's central claims are empirical and evaluated against held-out test splits of FOLIO, JustLogic, and ProverQA. The ORM is a separately trained Qwen 2.5 7B model, distinct from the reasoners in the main experiments, and no fitted constant is later relabeled as a prediction. The EcCoT data-augmentation procedure is a construction method, not a definitional identity: echo samples are labeled by their final-answer match to gold labels, and the judge filter selects LLM-missed errors; whether these transfer to inference-time errors is an empirical question that the paper addresses only indirectly via self-BLEU diversity (Appendix D). That is a limitation of evidence, not circularity. The self-citations (Lam et al. 2024; Thatikonda et al. 2024) are related-work background and not load-bearing. The Section 6 limitation (outcome supervision may ignore intermediate errors) is a substantive weakness but does not make the derivation circular. The Appendix B resampling choice (10k samples tuned by downstream performance) is a model-selection risk but not a circular step, since the reported test accuracy is still an external benchmark measurement. Therefore no load-bearing step reduces by construction or by self-citation to its own inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three domain assumptions: outcome supervision suffices (flagged by the authors in Limitations), echo errors transfer to inference-time reasoner errors, and the self-judge filter is accurate. Free parameters are mostly data-generation choices: sample counts per instance, the 10k echo resampling size selected by downstream performance, and the hand-set resampling weights alpha=0.8, beta=0.2; the LoRA rank and alpha are unspecified. No invented entities are introduced; the echo is a data-generation mode, not a new mechanism.

free parameters (4)

JustLogic echo resampling size = 10,000
Chosen from empirical comparison of 10k/20k/30k subsets on downstream ORM performance (Figure 5); the paper does not state whether the comparison used a validation split or the test set, so this may be tuned to the benchmark.
Resampling weights alpha and beta = 0.8 / 0.2
Hand-set linear combination weights for the BLEU-diversity and frequency terms in Appendix B.
CoT samples per instance = 8 (ProverQA), 8 (JustLogic), 10 (FOLIO)
Chosen sample counts for ORM training data generation, Section 3. The FOLIO count is later compared against 40 CoT samples in the ablation.
LoRA rank and alpha = unspecified
The paper states 'LoRA-based PEFT configuration' without rank or alpha values, so the exact training configuration is under-specified.

assumptions (3)

domain assumption Matching the parsed final answer to the gold label is a sufficient reward signal for reasoning quality
Used in Section 2 to label all training trajectories as positive or negative; the paper's Limitations (Section 6) concedes this overlooks intermediate errors that still reach the correct final answer.
domain assumption Echo rationales that survive the self-judge filter approximate the error distribution of reasoners at inference time
This is what makes EcCoT data useful; Section 2 assumes transfer rather than measuring overlap between echo errors and reasoner errors, and Appendix D only reports self-BLEU diversity.
domain assumption The generator's self-judge flags are accurate enough to discard obvious errors and retain hard ones
Section 2 filtering step; judge agreement with gold labels is never reported, so the composition of the retained hard-negative set is unquantified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Logical Reasoning with Outcome Reward Models for Test-Time Scaling." pith.science (2026). https://pith.science/paper/Y2SNHPSK

@misc{pith2026250819903,
  author       = {Pith},
  title        = {Pith review of: Logical Reasoning with Outcome Reward Models for Test-Time Scaling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y2SNHPSK}},
  note         = {Machine review of arXiv:2508.19903}
}

read the original abstract

Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), as it reflects their ability to derive valid conclusions from given premises. While the combination of test-time scaling with dedicated outcome or process reward models has opened up new avenues to enhance LLMs performance in complex reasoning tasks, this space is under-explored in deductive logical reasoning. We present a set of Outcome Reward Models (ORMs) for deductive reasoning. To train the ORMs we mainly generate data using Chain-of-Thought (CoT) with single and multiple samples. Additionally, we propose a novel tactic to further expand the type of errors covered in the training dataset of the ORM. In particular, we propose an echo generation technique that leverages LLMs' tendency to reflect incorrect assumptions made in prompts to extract additional training data, covering previously unexplored error types. While a standard CoT chain may contain errors likely to be made by the reasoner, the echo strategy deliberately steers the model toward incorrect reasoning. We show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO, JustLogic, and ProverQA datasets across four different LLMs.

Figures

Figures reproduced from arXiv: 2508.19903 by the authors.

**Figure 2.** Performance of the ORM trained on GPT-4o-generated data using Chain-of-Thought (CoT) and Echoed [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Performance comparison of ORM-trained models based on training data size, data generator, and reasoning generator for ProverQA dataset. generated per question. As the number of samples increases from 1 to 8, the ORM’s performance surpasses that of majority voting. This result suggests that generating multiple samples per question, while keeping the training set and questions fixed, can yield more effective trainin… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Average majority vote performance across [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Performance of the ORM trained on GPT-4o-generated Echoed data with varying sample sizes. Sampling [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Performance of FOLIO with 10 samples using [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

**Figure 7.** Figure 7: Performance of Gemma-based models of varying sizes on the ProverQA dataset shows a significant [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 2 canonical work pages

[1]

Bradley C. A. Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \' e , and Azalia Mirhoseini. 2024. https://doi.org/10.48550/ARXIV.2407.21787 Large language monkeys: Scaling inference compute with repeated sampling . CoRR, abs/2407.21787

arXiv doi:10.48550/arxiv.2407.21787 2024
[2]

Michael K Chen, Xikun Zhang, and Dacheng Tao. 2025. Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models. arXiv preprint arXiv:2501.14851

arXiv 2025
[3]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur \' e lien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozi \` e...

arXiv doi:10.48550/arxiv.2407.21783 2024
[4]

Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle - Solano, Hannah Szab \' o , Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, W...

work page 2024
[5]

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

2022
[6]

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. https://doi.org/10.48550/ARXIV.2409.12186 Qwen2.5-coder technical report . CoRR, abs/2409.12186

arXiv doi:10.48550/arxiv.2409.12186 2024
[7]

Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \' e , Morgane Rivi \` e re, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean - Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Ga \" e l Liu, Francesco Visin, Kathleen Kenealy, Luca...

arXiv doi:10.48550/arxiv.2503.19786 2025
[8]

Long Hei Matthew Lam, Ramya Keerthy Thatikonda, and Ehsan Shareghi. 2024. https://arxiv.org/abs/2406.00284 A closer look at logical reasoning with llms: The choice of tool matters . Preprint, arXiv:2406.00284

arXiv 2024

Show all 23 references

[9]

Chengpeng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Xiang Wang, Bowen Yu, Binyuan Hui, Junyang Lin, and Dayiheng Liu. 2025. https://doi.org/10.48550/ARXIV.2503.04625 START: self-taught reasoner with tools . CoRR, abs/2503.04625

2025 doi arXiv
[10]

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

2024
[11]

Zhang, Armando Solar - Lezama, Joshua B

Theo Olausson, Alex Gu, Benjamin Lipkin, Cedegao E. Zhang, Armando Solar - Lezama, Joshua B. Tenenbaum, and Roger Levy. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.313 LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logi...

2023 doi
[12]

Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.248 Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning . In Findings of the Association for Computational Linguist...

2023 doi
[13]

Chengwen Qi, Ren Ma, Bowen Li, He Du, Binyuan Hui, Jinwang Wu, Yuanjun Laili, and Conghui He. 2025. https://doi.org/10.48550/ARXIV.2502.06563 Large language models meet symbolic provers for logical reasoning evaluation . CoRR, abs/2502.06563

2025 doi arXiv
[14]

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. https://doi.org/10.48550/ARXIV.2408.03314 Scaling LLM test-time compute optimally can be more effective than scaling model parameters . CoRR, abs/2408.03314

2024 doi arXiv
[15]

Buntine, and Ehsan Shareghi

Ramya Keerthy Thatikonda, Jiuzhou Han, Wray L. Buntine, and Ehsan Shareghi. 2024. https://doi.org/10.48550/ARXIV.2409.16461 Strategies for improving nl-to-fol translation with llms: Data generation, incremental fine-tuning, and verification . CoRR, abs/2409.16461

2024 doi arXiv
[16]

Francis Song, Noah Y

Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. https://doi.org/10.48550/ARXIV.2211.14275 Solving math word problems with process- and outcome-based feedback . CoRR, abs/2211.14275

2022 doi arXiv
[17]

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.510 Math-shepherd: Verify and reinforce llms step-by-step without human annotations . In Proceedings of the 62nd Annual Meeting ...

2024 doi
[18]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[19]

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

2025 arXiv
[20]

Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/8e9c7d4a48bdac81a58f983a64aaf42b-Abstract-Conference.html Satlm: Satisfiability-aided language models using declarative prompting . In Advances in Neural Information Pr...

2023
[21]

Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301

2025 arXiv
[22]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[23]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Bradley C. A. Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \' e , and Azalia Mirhoseini. 2024. https://doi.org/10.48550/ARXIV.2407.21787 Large language monkeys: Scaling inference compute with repeated sampling . CoRR, abs/2407.21787

arXiv doi:10.48550/arxiv.2407.21787 2024

[2] [2]

Michael K Chen, Xikun Zhang, and Dacheng Tao. 2025. Justlogic: A comprehensive benchmark for evaluating deductive reasoning in large language models. arXiv preprint arXiv:2501.14851

arXiv 2025

[3] [3]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur \' e lien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozi \` e...

arXiv doi:10.48550/arxiv.2407.21783 2024

[4] [4]

Fabbri, Wojciech Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle - Solano, Hannah Szab \' o , Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, W...

work page 2024

[5] [5]

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

2022

[6] [6]

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. https://doi.org/10.48550/ARXIV.2409.12186 Qwen2.5-coder technical report . CoRR, abs/2409.12186

arXiv doi:10.48550/arxiv.2409.12186 2024

[7] [7]

Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram \' e , Morgane Rivi \` e re, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean - Bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Ga \" e l Liu, Francesco Visin, Kathleen Kenealy, Luca...

arXiv doi:10.48550/arxiv.2503.19786 2025

[8] [8]

Long Hei Matthew Lam, Ramya Keerthy Thatikonda, and Ehsan Shareghi. 2024. https://arxiv.org/abs/2406.00284 A closer look at logical reasoning with llms: The choice of tool matters . Preprint, arXiv:2406.00284

arXiv 2024