Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

CC-LEARN: Cohort-based Consistency Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training an LLM to answer six surface variants of a question with a single program improves reasoning consistency over pretrained and supervised baselines.

desk verdict The cohort-consistency idea is real, but the evaluation is coupled to the training reward and contaminated by the retriever, and the paper's own HotpotQA numbers contradict the 'consistently outperforms' claim. read the letter →

arxiv 2506.15662 v1 pith:46QXR7IS submitted 2025-06-18 cs.CL

Xiao Ye , Shaswat Shrivastava , Zhaonan Li , Jacob Dineen , Shijie Lu , Avneet Ahuja , Ming Shen , Zhikun Xu
show 1 more author
Ben Zhou
This is my paper
classification cs.CL
keywords cohort-basedconsistencylearningreinforcementlargelanguagemodelreasoningprogrammaticretrievalrejectionfilteringGRPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models often answer a question correctly but fail on a paraphrased or logically equivalent variant, a sign of brittle reasoning. CC-LEARN claims this inconsistency can be reduced by training on cohorts: an original question abstracted into a masked template plus five automatically generated variants that share the same reasoning path but differ in facts. A single executable program of atomic lookups must succeed across the cohort, and a reinforcement learning loop (Group Relative Policy Optimization) maximizes a composite reward that combines cohort accuracy, retrieval-usage bonuses, and penalties for rejected lookups. Across five reasoning benchmarks, the paper reports that this cohort-level training beats both the pretrained model and supervised fine-tuning under lenient (at least four of six correct) and strict (at least five of six correct) consistency criteria.

What carries the argument

The load-bearing object is the cohort, built from an abstraction step that masks an original question into a template with typed parameters (for example, 'Are Film1 and Film2 both DocumentaryType films that involve LocationX?') and instantiates five factual variants of it. The reasoning path is the executable program the policy writes against that template: a Python function using only atomic retrieve calls and control flow, which runs unchanged on all six questions. The retriever is a separate 7B instruction-tuned model with a rejection-prompt filter that refuses multi-step or invalid lookups with 'idk', so the policy cannot shortcut by re-asking a hard question. Group Relative Policy Optimization (GRPO) then maximizes a composite cohort-level reward combining accuracy, retrieval usage, and rejection penalties, channeling the model toward uniform, verifiable reasoning across variants.

What would settle it

Run the trained policy on paraphrases written independently of the paper's masked-template pipeline (by humans or a different generator) that follow the same reasoning structures, with a retriever whose atomic-lookup errors differ from the 7B model used in training. If the consistency advantage over vanilla and SFT disappears, or if a program that scored five of six on the paper's cohorts now scores far lower, the reported consistency is an artifact of the generated cohorts or the specific retriever rather than a generalizable reasoning procedure.

Watch

Extended reading notes

Core claim

The central claim is that reasoning consistency across surface variants is directly trainable: the model must produce one program that solves all six questions in a cohort, so a correct answer can no longer be a lucky accident of an incorrect reasoning path. The program is a small Python function that issues only atomic retrieve(...) calls, and the same function runs unchanged on every variant, which means any output change comes from different facts rather than different reasoning steps. The training reward has three parts: cohort accuracy (credit for the program answering most of the group), a retrieval-usage bonus favoring decomposition into more than one lookup, and a rejection penalty for retrieval calls the retriever classifies as multi-step or invalid. On ARC-Challenge the cohort-trained model reaches 29.8 percent lenient accuracy versus 19.0 for the vanilla checkpoint and 19.8 for SFT, and 22.0 percent strict accuracy versus 13.6 and 14.4; on HotpotQA the lenient result is slightly below SFT but the strict result is above it. The paper takes these results as evidence that enforcing a group-level reward threshold makes the learned reasoning procedure more generalizable.

Load-bearing premise

The load-bearing premise is that the automatically generated similar questions truly share the original question's reasoning path and that the 7B retriever answers the atomic lookups correctly on both the original and the variants, so a program that answers five of six cohort questions is evidence of consistent reasoning rather than of retrieval artifacts or train-test leakage.

Editorial extensions

If this is right

  • On four of five benchmarks and under both consistency criteria, cohort-level RL improves over the vanilla pretrained and SFT baselines, with absolute gains of roughly five to ten points under lenient criteria and three to eight points under strict criteria; on HotpotQA the lenient result trails SFT by one point, while the strict result is above it.
  • The Cohort variant, which pays accuracy credit only when the program answers at least four of six questions, matches or beats the per-question Normal variant on strict consistency across all five datasets, suggesting the group-level threshold itself is valuable.
  • Training on original questions alone instead of full cohorts degrades performance, most clearly on ARC-Challenge, so the similar-variant questions carry the consistency signal.
  • A stronger retriever amplifies the gains: using a 32B retriever in both training and evaluation lifts lenient accuracy on StrategyQA from 16.0 to 34.8 percent and on HotpotQA from 45.8 to 54.4 percent.
  • An upper-bound estimate with pass@128 sampling and manual program correction reaches 88 to 100 percent on the benchmarks, indicating the protocol is feasible and current results leave substantial headroom.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test, but its logic predicts that the learned programs should generalize to paraphrases written independently of its masked-template pipeline, for example by humans; if the advantage disappears there, the consistency is specific to the generated cohort family.
  • The rejection-prompt filter discards 46 percent of simple questions in SimpleQA, so some clean lookups are lost; a better-calibrated rejection rule could strengthen the reward signal, consistent with the large gains seen when a stronger retriever is used.
  • Because training and evaluation couple the policy with one retriever, an informative extension is to report how much cohort-accuracy survives a retriever swap, separating learned reasoning from lookup accuracy.
  • A natural stress test is to make cohorts adversarial, generating variants that flip the answer or change a relation, to see whether the program generalizes the reasoning structure rather than fitting the template family.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. CC-LEARN proposes a reinforcement learning framework for improving reasoning consistency by training a policy to generate executable programs that run unchanged over cohorts of similar questions. Each program issues atomic retrieve calls to a separate retriever model, and GRPO optimizes a composite reward combining cohort accuracy, retrieval-usage bonuses, and rejection penalties. The method is evaluated on ARC-Challenge, ARC-Easy, CommonsenseQA, StrategyQA, and HotpotQA, reporting lenient and strict consistency accuracy (thresholds of at least 4/6 and 5/6 correct on six-question cohorts). The abstract and conclusion claim consistent gains over vanilla and SFT baselines in both accuracy and reasoning stability.

Significance. The idea of training on cohorts of similar questions sharing a programmatic abstraction is a plausible and potentially useful direction for consistency-oriented RL. The paper includes meaningful infrastructure components: a 70B-based variant generation pipeline, cross-model validation, a rejection mechanism for retrieval, and a composite reward design. It also reports a human study and attempts an upper-bound analysis. However, the main empirical claims are not established as presented: the evaluation metric is closely aligned with the training objective, the reported 'consistency' is heavily dependent on the retrieval model, and one of the five benchmarks directly contradicts the claim of consistent improvement. If the evaluation were rebuilt around independent metrics and retrieval confounds were controlled, the framework could still be of interest, but the current evidence does not support the paper's central claims.

major comments (4)
  1. [Section 4.4, Table 1] The central claim that CC-LEARN 'consistently outperforms both vanilla and SFT baselines' is directly contradicted by the HotpotQA lenient-accuracy row: SFT achieves 46.8, Cohort achieves 45.8, and Normal achieves 45.6. The text itself concedes 'SFT holds a narrow edge,' yet the abstract and the concluding sentence of Section 4.4 retain the unqualified 'consistently outperforms' claim. This requires either removing the claim or adding an explicit exception, and it undermines the abstract's 'boosts both accuracy and reasoning stability' statement.
  2. [Sections 3.3-3.4 and 4.4] The evaluation metric is essentially the training objective. Racc = 0.2 * ncorrect with ncorrect in {0, ..., 6} is optimized by GRPO over six-question cohorts, and the reported lenient/strict accuracy are thresholds (>=4/6 and >=5/6) on the same cohort structure generated by the same 70B pipeline. No standard single-question accuracy is reported anywhere, despite the protocol claiming to report it and the abstract claiming improved 'accuracy.' Consequently, the observed gains over SFT may simply reflect that the RL policy optimizes the exact cohort-consistency objective, rather than a generalizable improvement in reasoning accuracy or stability.
  3. [Section 4.5, Tables 7-8; Section 3.3] The reported consistency metric is confounded by retriever quality. Every program's cohort score depends on the 7B retriever answering atomic retrieve calls; Table 3 reports a 46% rejection rate on SimpleQA, and Appendix A.1 shows the retriever returning a wrong answer on a simple true/false lookup. The 32B-retriever ablation in Tables 7-8 changes lenient accuracy by up to +18.8 points (StrategyQA) and +8.6 points (HotpotQA), which is larger than most of the claimed improvements over baselines. The paper does not report per-lookup accuracy or rejection rates on the test cohorts, so the 'reasoning stability' gains may be retrieval artifacts rather than more consistent reasoning by the policy.
  4. [Section 3.3] The definition of the accuracy reward is internally inconsistent. The reward components define Racc = 0.2 * ncorrect for all ncorrect, but the 'RL Variants' paragraph states that Cohort Accuracy grants accuracy reward only if at least 4 of 6 questions are answered correctly. It is therefore unclear which reward specification produced the 'Cohort' models in Tables 1-2, and the ambiguity prevents faithful reproduction of the main result.
minor comments (5)
  1. [Section 4.4] The text states that the protocol 'enables us to report not only standard accuracy but also the critical generalization-across-variants metrics,' but no standard accuracy results appear in any table or figure.
  2. [Section 4.7, Table 9] The human-study win-rate presentation is confusing: 'Cohort RL's reasoning paths are preferred 47% of the time over the left 2 models' could be read as a majority preference, but 47% is below 50%, and the tie convention and pairwise comparison protocol are not specified.
  3. [Appendix A.1] The rejection-prompt example shows the retriever answering 'Is the Eiffel Tower located in Paris, France?' with false, which is a factual error on a simple lookup; the paper should acknowledge and analyze such errors rather than only reporting rejection rates.
  4. [Tables 1-8] The error bars (e.g., +-4.0) are never defined as standard deviation, standard error, or a confidence interval, and no significance tests are reported; several pairwise differences (e.g., HotpotQA) are within these ranges.
  5. [NeurIPS Checklist] Several checklist answers are inconsistent with the manuscript: Q4/Q5 answer 'Yes' based on a future code release rather than provided artifacts, and Q12 answers 'NA' while the paper uses Qwen, LLaMA, and multiple public datasets.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training reward and evaluation metric both measure cohort accuracy, but evaluation uses held-out cohorts and the reward includes additional retrieval terms, so the reported gains are empirical rather than forced by construction.

full rationale

The paper's central claim is empirical, not a derivation. The closest candidate for a circularity concern is the alignment between the training reward and the evaluation protocol: Racc is defined as 0.2 * ncorrect over a six-question cohort, and the lenient/strict evaluation thresholds are ncorrect >= 4 and ncorrect >= 5. However, this is an ordinary case of optimizing a training objective and then measuring the same objective on a held-out test set. The reward also includes retrieval-usage and rejection-penalty terms that do not appear in the evaluation, and the evaluation is performed on 2,500 test questions that are not the training examples. An RL-trained model could fail to improve on held-out cohorts due to overfitting or distribution shift, so the reported gains are not logically forced by the definition of the reward. The sensitivity of results to retriever quality (the 32B-retriever ablation) is an experimental confound and a validity concern, not a circularity, because the retriever is a fixed off-the-shelf component and is not fitted to the test set. The self-citations to prior work by co-author Ben Zhou appear only as background motivation and related work; they are not used to justify a uniqueness theorem, to import an unverified ansatz, or to forbid alternative methods. The upper-bound analysis and ablations are sanity checks rather than circular reductions. Therefore, no circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the reliability of the cohort generation pipeline and the retriever, plus the validity of the cohort-accuracy metric. The reward coefficients and cohort size are hand-selected. No genuinely new physical or conceptual entities are introduced beyond the methodological constructs themselves.

free parameters (5)
  • reward accuracy coefficient = 0.2
    Determines the contribution of each correct cohort answer to the reward; chosen by hand, not tuned.
  • retrieval usage reward thresholds = -0.6, 0, +0.6 for ncalls 0, 1, >1
    Encourages at least two retrieve calls; hand-chosen.
  • rejection penalty coefficient = -0.1
    Penalty per rejected retrieve call; hand-chosen.
  • cohort size = 6 (1 original + 5 variants)
    Number of questions per cohort; chosen for the lenient/strict thresholds.
  • lenient and strict thresholds = 4/6 and 5/6
    Definition of consistency; chosen without external justification.
assumptions (3)
  • domain assumption The similar questions generated by LLaMA-3.3-70B preserve the same reasoning path as the original question.
    Section 3.2; validated only on a sample of 150 questions in Section 4.7.
  • domain assumption The Qwen-2.5-7B-Instruct retriever answers atomic retrieve calls accurately and consistently.
    Section 3.3; the paper itself shows high rejection rates for complex questions, and a stronger retriever changes results substantially in Section 4.6.
  • domain assumption Cohort accuracy (4 or 5 of 6) is a valid measure of reasoning consistency.
    Used as the evaluation metric in Section 3.4; no external validation of this metric is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CC-LEARN: Cohort-based Consistency Learning." pith.science (2026). https://pith.science/paper/46QXR7IS

@misc{pith2026250615662,
  author       = {Pith},
  title        = {Pith review of: CC-LEARN: Cohort-based Consistency Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46QXR7IS}},
  note         = {Machine review of arXiv:2506.15662}
}
read the original abstract

Large language models excel at many tasks but still struggle with consistent, robust reasoning. We introduce Cohort-based Consistency Learning (CC-Learn), a reinforcement learning framework that improves the reliability of LLM reasoning by training on cohorts of similar questions derived from shared programmatic abstractions. To enforce cohort-level consistency, we define a composite objective combining cohort accuracy, a retrieval bonus for effective problem decomposition, and a rejection penalty for trivial or invalid lookups that reinforcement learning can directly optimize, unlike supervised fine-tuning. Optimizing this reward guides the model to adopt uniform reasoning patterns across all cohort members. Experiments on challenging reasoning benchmarks (including ARC-Challenge and StrategyQA) show that CC-Learn boosts both accuracy and reasoning stability over pretrained and SFT baselines. These results demonstrate that cohort-level RL effectively enhances reasoning consistency in LLMs.

Figures

Figures reproduced from arXiv: 2506.15662 by the authors.

Figure 1
Figure 1. An illustrative example of cohort-based consistency learning(See Appendix C). (Top) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the CC-Learn pipeline. Data Generation: we abstract each question into a masked template with parameters, generate cohorts of similar variants using a 70B LLM, and validate answers via cross-model agreement. RL Training: we synthesize executable programs, use a Coder￾Instruct policy and a 7B-Instruct retriever with rejection prompts, and compute a composite reward from accuracy, retrieval usage, and reje… view at source ↗
Figure 3
Figure 3. The screenshot for the example. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Implicit Reasoning Steering via Concept Chaining

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Reinforcement-learning-optimized concept-chain paragraphs covertly steer language-model multiple-choice preferences after continued pretraining, with far lower detectability than direct paraphrases.

  2. Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    Vocabulary dropout prevents diversity collapse in LLM co-evolution by masking proposer logits, yielding average +4.4 point solver gains on mathematical reasoning benchmarks at 8B scale.

Reference graph

Works this paper leans on

32 extracted references · 18 canonical work pages · cited by 2 Pith papers

  1. [1]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    arXiv 2018

  2. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  3. [3]

    Measuring and improving consistency in pretrained language models.Transactions of the Association for Computational Linguistics, 9:1012–1031, 2021

    Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models.Transactions of the Association for Computational Linguistics, 9:1012–1031, 2021

    2021

  4. [4]

    Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks.Nature Machine Intelligence, 2(11):665–673, 2020

    2020

  5. [5]

    Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies

    Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021

    2021

  6. [6]

    Abstraction- of-thought makes language models better reasoners, 2024

    Ruixin Hong, Hongming Zhang, Xiaoman Pan, Dong Yu, and Changshui Zhang. Abstraction- of-thought makes language models better reasoners, 2024

    work page 2024

  7. [7]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186, 2024

    arXiv 2024

  8. [8]

    Becel: Benchmark for consistency evaluation of language models

    Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. Becel: Benchmark for consistency evaluation of language models. InProceedings of the 29th International Conference on Compu- tational Linguistics (COLING), pages 3680–3696. International Committee on Computational Linguistics, 2022

    work page 2022

Show all 32 references
  1. [9]

    Famicom: Further demystifying prompts for language models with task-agnostic performance estimation, 2024

    Bangzheng Li, Ben Zhou, Xingyu Fu, Fei Wang, Dan Roth, and Muhao Chen. Famicom: Further demystifying prompts for language models with task-agnostic performance estimation, 2024

    2024

  2. [10]

    Deceptive semantic shortcuts on reasoning chains: How far can models go without hallucination?, 2024

    Bangzheng Li, Ben Zhou, Fei Wang, Xingyu Fu, Dan Roth, and Muhao Chen. Deceptive semantic shortcuts on reasoning chains: How far can models go without hallucination?, 2024

    2024

  3. [11]

    Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.arXiv preprint arXiv:1902.01007, 2019

    R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.arXiv preprint arXiv:1902.01007, 2019

    1902 arXiv

  4. [12]

    Learning to utilize shaping rewards: A new approach of potential-based reward shaping

    Hoang Nguyen, Mario Araya, and Finale Doshi-Velez. Learning to utilize shaping rewards: A new approach of potential-based reward shaping. InNeurIPS 2020, 2020

    2020

  5. [13]

    Knowledge-based consis- tency testing of large language models

    SaiSathiesh Rajan, Ezekiel Soremekun, and Sudipta Chattopadhyay. Knowledge-based consis- tency testing of large language models. InFindings of EMNLP 2024, 2024. to appear

    2024

  6. [14]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Yankai Li, Yufei Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024

    2024

  7. [15]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297. ACM, March 2025

    2025

  8. [16]

    Masked language modeling and the distributional hypothesis: Order word matters pre-training for little

    Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Proce...

    2021

  9. [17]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of t...

    2019

  10. [18]

    The llama 3 herd of models, 2024

    LLaMA Team. The llama 3 herd of models, 2024

    2024

  11. [19]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  12. [20]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025

    2025

  13. [21]

    F2rl: Fac- tuality and faithfulness reinforcement learning framework for claim-guided evidence-supported counterspeech generation

    Haiyang Wang, Yuchen Pan, Xin Song, Xuechen Zhao, Minghao Hu, and Bin Zhou. F2rl: Fac- tuality and faithfulness reinforcement learning framework for claim-guided evidence-supported counterspeech generation. InEMNLP 2024, 2024

    2024

  14. [22]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, and Aleksei Gusev. Self-consistency improves chain-of-thought reasoning in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  15. [23]

    Measuring short-form factuality in large language models, 2024

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models, 2024

    2024

  16. [24]

    Chain-of-thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.arXiv preprint arXiv:2201.11903, 2022

    2022 arXiv

  17. [25]

    Direct reasoning optimiza- tion: Llms can reward and refine their own reasoning for open-ended tasks

    Yifei Xu, Tusher Chakraborty, Srinagesh Sharma, and et al. Direct reasoning optimiza- tion: Llms can reward and refine their own reasoning for open-ended tasks. arXiv preprint arXiv:2506.13351, 2025

    2025 arXiv

  18. [26]

    Qwen2 technical report.arXiv preprint arXiv:2407.10671, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  19. [27]

    Cohen, Ruslan Salakhut- dinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsu- jii, editor...

    2018

  20. [28]

    Reasonagain: Using extractable symbolic programs to evaluate mathematical reasoning, 2024

    Xiaodong Yu, Ben Zhou, Hao Cheng, and Dan Roth. Reasonagain: Using extractable symbolic programs to evaluate mathematical reasoning, 2024

    2024

  21. [29]

    Calibrate before use: Improving few-shot performance of language models

    Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. InInternational Conference on Machine Learning, pages 12697–12706. PMLR, 2021

    2021

  22. [30]

    Self-supervised analogical learning using language models, 2025

    Ben Zhou, Sarthak Jain, Yi Zhang, Qiang Ning, Shuai Wang, Yassine Benajiba, and Dan Roth. Self-supervised analogical learning using language models, 2025

    2025

  23. [31]

    ‘json "answer

    Ben Zhou, Hongming Zhang, Sihao Chen, Dian Yu, Hongwei Wang, Baolin Peng, Dan Roth, and Dong Yu. Conceptual and unbiased reasoning in language models, 2024. 12 A Prompts A.1 Rejection Prompt Prompt forFact-Lookup Assistant System:You are a fact-lookup assistant. For each user ...

    2024

  24. [32]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.