Pith. sign in

REVIEW 3 major objections 5 minor 38 references

TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read TimeHC-RL lifts a 7B model to frontier social-intelligence scores, matching DeepSeek-R1 and nearly matching OpenAI-O3 on average, according to the paper.

desk verdict A useful systematic post-training comparison with a shaky central mechanism: the temporal reward fires largely by chance at G=8, so the causal advantage over HC-RL is not established. read the letter →

arxiv 2505.24500 v1 pith:C5YZ3HP5 submitted 2025-05-30 cs.CL cs.AI

Guiyang Hou , Xing Gao , Yuchuan Wu , Xiang Huang , Wenqi Zhang , Zhe Zheng , Yongliang Shen , Jialu Du
show 3 more authors
Fei Huang Yongbin Li Weiming Lu
This is my paper
classification cs.CLcs.AI
keywords temporal-awarereinforcementlearningsocialintelligencetheoryofmindhierarchicalcognitionSystem1and2post-trainingGRPOtest-timescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that social intelligence in LLMs cannot be trained the way math and coding are: it needs both fast intuitive responses and deliberate reasoning, and it needs explicit sensitivity to the order in which social events and conversations unfold. To test this, it introduces TimeHC-RL, a rule-based reinforcement-learning post-training method that lets a 7B model choose among three cognitive modes and rewards it extra when it answers better from temporally ordered event lines than from shuffled ones. On eight social benchmarks, the method reports an in-domain average of 80.0%, matching DeepSeek-R1 (79.0%) and nearly matching OpenAI-O3 (81.0%), starting from a backbone that scored 51.0%. The paper also reports that RL generalizes where SFT memorizes, that budget-forcing test-time compute does not build social-situation cognition, and that different cognitive modes suit different data types.

What carries the argument

TimeHC-RL runs group relative policy optimization (GRPO) with the composite reward $r = r_{\text{format}} + r_{\text{accuracy}} + r_{\text{temporal}}$. The temporal component is a contrastive reward: for each question the model answers both the ordered social-event lines or conversation and a shuffled version, and if the ordered accuracy $p$ exceeds $\mu \cdot \tilde{p}$ with $\mu = 0.9$, correct responses in that question's group receive an extra bonus $\alpha = 0.4$. The hierarchical component is a set of three response formats the model learns to select: <answer>...</answer> for intuitive System 1 answers, <social context understanding>...</social context understanding> plus an answer for surface-level analysis, and <think>...</think> plus an answer for deliberate System 2 reasoning.

What would settle it

Ablate the temporal reward: train the same pipeline on the same data with $r = r_{\text{format}} + r_{\text{accuracy}}$ and measure the out-of-distribution average; if the reported +3-point OOD gain over HC-RL does not reproduce, the temporal component is not carrying the claimed benefit.

Watch

Extended reading notes

Core claim

The central claim is that a single post-training recipe can give a 7B backbone frontier-level social intelligence, and that two design choices drive the gain: a temporal reward that contrasts performance on ordered versus shuffled social-event lines, and a hierarchical cognition framework in which the model chooses among System 1 (<answer>), surface-level understanding (<social context understanding>), and System 2 (<think>) responses. The reported evidence is a +29-point in-domain improvement over the backbone, a +2-point edge over System 2-only RL in-domain, and a +3-point out-of-distribution edge, with similar or better scores than the two frontier references on most individual datasets. The authors also establish, on their benchmarks, that direct SFT and long-thought SFT degrade out-of-distribution performance while RL improves it, and that sequential test-time scaling helps interpersonal reasoning but not social-situation cognition.

Load-bearing premise

The temporal reward assumes that shuffling event lines or conversation turns is a clean negative control that removes temporal information without introducing other distortions, and that the hand-set thresholds (bonus 0.4, ratio 0.9) are reasonable for every dataset, including situation-based benchmarks where temporal order is not clearly the deciding factor.

Editorial extensions

If this is right

  • A 7B post-trained model can be competitive with frontier models on current social-intelligence benchmarks, at a fraction of the scale and inference cost.
  • Rule-based RL is a viable post-training route for social cognition; SFT-based methods that work for reasoning tasks can reduce social performance.
  • Reasoning-depth extrapolation (training on depths 1-2, evaluating on 3-4) is more effective through RL than through SFT.
  • Test-time budget forcing is not a substitute for training-time social-situation coverage; sequential scaling helps interpersonal reasoning but not situation cognition.
  • Teaching models to select their own cognitive mode beats forcing one mode; System 1-style RL even outperforms System 2 on situation-cognition data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the contrastive temporal reward works by forcing the model to attend to order, a similar reward could transfer to other order-sensitive tasks such as stepwise instruction following, procedural planning, and temporal commonsense QA; the paper does not test those.
  • The hierarchical tags might act partly as a compute-allocation mechanism, letting the model spend more tokens where needed; comparing against a uniform mixture of the three modes would separate adaptive selection from the mode definitions themselves.
  • The finding that System 1 RL beats System 2 RL on situation benchmarks hints that social-situation cognition is largely pattern recognition; if so, scaling diverse situation data may matter more than longer reasoning traces.
  • Because the temporal reward is applied uniformly to all datasets, a stricter variant restricted to datasets with explicit event lines could isolate when temporal understanding is actually learnable; this is an ablation the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces TimeHC-RL, a GRPO-based post-training method for social intelligence that combines a hierarchical cognitive tagging framework (System 1, surface-level thinking, System 2) with a contrastive "temporal reward" that rewards higher accuracy on temporally ordered inputs than on shuffled inputs. The authors train a Qwen2.5-7B-Instruct-1M backbone on five social/ToM datasets, evaluate in-domain on held-out splits and on three OOD datasets, and compare against five post-training paradigms and two test-time intervention strategies. The headline claims are that the method improves the backbone by 29 points in-domain and 7 points OOD, that it reaches 80.0% in-domain average versus 79.0% for DeepSeek-R1 and 81.0% for OpenAI-O3, and that the temporal reward contributes +1.0 point in-domain and +3.0 points OOD over the identical framework without it.

Significance. If the temporal reward is valid and the results are reproducible, the paper would be a notable contribution: it would show that a 7B model can reach frontier-level performance on social cognition benchmarks through post-training alone, and it would offer a concrete, reusable design (hierarchical cognitive modes plus temporal contrast) for social-intelligence training. The paper's strengths are its systematic experimental coverage (five post-training paradigms, two test-time interventions, eight datasets), the controlled use of a single backbone, the HiToM depth-generalization split, and the reporting of in-domain and OOD setups. However, the central quantitative claims currently rest on single-run evaluations with small OOD sets and on a temporal reward whose triggering mechanism is not shown to be selective for temporal understanding; the significance is therefore conditional on additional controls and statistical reporting.

major comments (3)
  1. [Section 3.2.3] The temporal reward compares two groups of G=8 responses (Appendix A.3). Under the null hypothesis that shuffling is inert, the condition p > 0.9·p̃ is met by chance with high probability: for a true accuracy of p=0.5, P(p > 0.9·p̃) ≈ 0.69, and for p=0.8, P ≈ 0.63. Thus the α=0.4 bonus fires on a majority of questions even when the model has no temporal sensitivity, and because it is applied only to correct responses in the ordered group, it functions as an episodic random amplification of the outcome reward rather than a measurement of temporal logic. The paper's claim that the +1.0/+3.0 differences between TimeHC-RL and HC-RL (Tables 2 and 3) are due to temporal awareness is therefore not established. Please add a null-permutation control (e.g., use a second ordered group as the "shuffled" baseline, or randomize the bonus assignment) and report the empirical firing rate of the temporal reward during training.
  2. [Tables 1-3] The evaluation lacks error bars or repeated-seed statistics. Several OOD sets are very small: ToMATO has 50 samples, OpenToM 85, and SimpleToM 120 (Table 1). The TimeHC-RL vs HC-RL differences are +0.01 in-domain average and +0.03 OOD average; on ToMATO (First), the reported +0.08 corresponds to four samples. With these sizes, the aggregate deltas and the headline comparison of 0.80 vs 0.79 vs 0.81 (Section 4.4) are within sampling noise. Please report means and standard deviations over at least three RL seeds and per-dataset bootstrap confidence intervals, or clearly state that the observed differences are not statistically significant and adjust the claims accordingly.
  3. [Section 3.2.3 and Table 1] The temporal contrast is applied uniformly to all datasets, including ToMBench and SocialIQA, whose inputs are single social-situation paragraphs with no canonical event-line order. The per-dataset results show no consistent temporal benefit (ToMBench 0.82 vs 0.81, SocialIQA 0.78 vs 0.79), yet Section 4.4 presents the temporal reward as uniformly advantageous. Please either restrict the temporal reward to data with an explicit temporal line or conversation structure (e.g., ToMi, HiToM, ExploreToM, ToMATO), or justify what shuffling means for situation paragraphs and show that the reward is beneficial there.
minor comments (5)
  1. [Appendix A.4, Figure 4] The System 2 example's reasoning ends with "the location of the peas is the blue_bathtub" but the answer tag reads "L. green_basket"; as printed, this is internally inconsistent and should be corrected or clarified.
  2. [Section 4.1.1] Direct SFT drops on SocialIQA from 0.77 (backbone) to 0.25, below the 0.33 chance level for three-choice questions; the paper does not explain this, and the "SFT memorizes" interpretation may conflate answer-format effects with memory capacity.
  3. [Section 3.2.3 and Appendix A.3] The paper should state explicitly that the group size G in the temporal reward is the same G=8 used for GRPO; the current notation uses both G and G̃ but never defines G̃.
  4. [Tables 2 and 3] Some rows are visually merged and the "AVG" header is split ("A VG"); please reformat for readability.
  5. [Abstract and Section 4.4] The abstract claims that the method "gives the 7B backbone model wings" and "rival[s] the performance of advanced models like DeepSeek-R1 and OpenAI-O3"; the latter depends on single-run averages, so please qualify with the statistical caveats requested in the major comments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TimeHC-RL's claims rest on held-out accuracy, and the temporal reward is a training signal rather than a fitted or self-defined prediction.

full rationale

The paper's central claims — that TimeHC-RL improves social intelligence and that the temporal reward adds a small gain over HC-RL — are supported by accuracy on held-out in-domain and OOD benchmarks, not by a derivation that reduces to its own inputs. The temporal reward in Section 3.2.3 is defined as rtemporal = α if p > μ·p̃, comparing the model's own correct-response proportions on ordered versus shuffled inputs. This is a reinforcement-learning reward-shaping signal, not a fitted parameter used to produce the reported evaluation numbers. The hyperparameters α=0.4 and μ=0.9 are hand-set and lack sensitivity analysis, which is a robustness concern but not circularity: the paper does not state that they were tuned on the evaluation splits. The only self-citations (Hou et al. 2024a, 2024b) appear in related-work or background context and are not load-bearing for the proposed method or its evaluation. No equation in the paper equates a predicted quantity to an input by construction, and no cited uniqueness theorem or ansatz is imported from the authors' prior work to force the design. The skeptical observation that the G=8 ordered-versus-shuffled contrast can fire under a null model is a statistical validity issue, not a circularity issue; it would require a permutation ablation to assess, but it does not make the derivation circular. The limitations section acknowledges the single-model-size scope and the absence of behavioral intelligence experiments, but those are scope limitations rather than admissions of circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim relies on hand-set reward hyperparameters and several domain assumptions about cognition and temporal order, but introduces no unfalsifiable entities. The method is an empirical recipe rather than a derivation, so the ledger is short.

free parameters (3)
  • alpha (temporal reward weight) = 0.4
    Hand-chosen in Section 3.2.3; scales the extra reward for temporally sensitive correct answers; no sensitivity analysis.
  • mu (temporal reward threshold ratio) = 0.9
    Hand-chosen in Section 3.2.3; correct ordered responses earn reward only if p > 0.9 * p_shuffled.
  • format and outcome reward weights = +1/-1 and +2/-1.5
    Hand-designed in Section 3.2.3; these reward shaping values are set without a tuning analysis.
assumptions (3)
  • domain assumption The social domain requires a mix of System 1, surface-level, and System 2 cognitive modes.
    Motivates the hierarchical tag framework in Section 3.2.1; based on cited psychology rather than measured LLM cognitive states.
  • domain assumption Shuffling social-event lines removes temporal information that is necessary for correct inference.
    Underlies the temporal reward in Section 3.2.3; if shuffled inputs are merely harder in an unrelated way, the reward is not measuring temporal logic.
  • ad hoc to paper The three tag-based response formats correspond to distinct and trainable cognitive modes.
    The tags are author-designed (Section 3.2.1); no evidence these induce different internal processes beyond surface format.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence." pith.science (2026). https://pith.science/paper/C5YZ3HP5

@misc{pith2026250524500,
  author       = {Pith},
  title        = {Pith review of: TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C5YZ3HP5}},
  note         = {Machine review of arXiv:2505.24500}
}
read the original abstract

Recently, Large Language Models (LLMs) have made significant progress in IQ-related domains that require careful thinking, such as mathematics and coding. However, enhancing LLMs' cognitive development in social domains, particularly from a post-training perspective, remains underexplored. Recognizing that the social world follows a distinct timeline and requires a richer blend of cognitive modes (from intuitive reactions (System 1) and surface-level thinking to deliberate thinking (System 2)) than mathematics, which primarily relies on System 2 cognition (careful, step-by-step reasoning), we introduce Temporal-aware Hierarchical Cognitive Reinforcement Learning (TimeHC-RL) for enhancing LLMs' social intelligence. In our experiments, we systematically explore improving LLMs' social intelligence and validate the effectiveness of the TimeHC-RL method, through five other post-training paradigms and two test-time intervention paradigms on eight datasets with diverse data patterns. Experimental results reveal the superiority of our proposed TimeHC-RL method compared to the widely adopted System 2 RL method. It gives the 7B backbone model wings, enabling it to rival the performance of advanced models like DeepSeek-R1 and OpenAI-O3. Additionally, the systematic exploration from post-training and test-time interventions perspectives to improve LLMs' social intelligence has uncovered several valuable insights.

Figures

Figures reproduced from arXiv: 2505.24500 by the authors.

Figure 1
Figure 1. Top: Real-world social events following a clear timeline and character temporal dynamics, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Training template for a model to adaptively choose among three cognitive modes: intuition, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of SFT-based and RL-based methods on interpersonal reasoning depth extrapolation. SFT memorizes, and has limited memory capacity (Direct SFT), while RL general￾izes. Both direct SFT and long-thought SFT reduce the original performance of the backbone model in OOD evaluation. In contrast, the RL paradigm still provides more or less gains to the backbone model. Furthermore, as shown in [PITH_FU… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: LLM adaptively employs different cognitive modes to address social situation cognition [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Sample example from HiToM. Data format: (Social-Event Lines, Question, Choices). [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Sample example from ExploreToM. Data format: (Social-Event Lines, Question). [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Sample example from ToMbench. Data format: (Social Situation, Question, Choices). [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Sample example from SimpleToM, SocialIQA. Data format: (Social Situation, Question, [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Sample example from SimpleToM, SocialIQA. Data format: (Social Situation, Question, [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 11
Figure 11. Figure 11: Sample example from ToMATO (First order). Data format: (Conversation, Question, [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Sample example from ToMATO (Second order). Data format: (Conversation, Question, [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 7 canonical work pages

  1. [2]

    Sft or rl? an early investigation into training r1-like reasoning large vision-language models

    Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1-like reasoning large vision-language models. arXiv preprint arXiv:2504.11468,

    arXiv

  2. [3]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161,

    arXiv

  3. [5]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  4. [7]

    Progressive tuning: Towards generic sentiment abilities for large language models

    Guiyang Hou, Yongliang Shen, and Weiming Lu. Progressive tuning: Towards generic sentiment abilities for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 14392–14402, 2024a. 10 Guiyang Hou, Wenqi Zhang, Yongliang Shen, Linjuan Wu, and Weiming Lu. Timetom: Temporal space is the key to unlocking the door of...

    arXiv 2024

  5. [8]

    A notion of complexity for theory of mind via discrete world models

    X Huang, Emanuele La Malfa, Samuele Marro, Andrea Asperti, Anthony Cohn, and Michael Wooldridge. A notion of complexity for theory of mind via discrete world models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2964–2983, 2024a. Xiang Huang, Sitao Cheng, Shanshan Huang, Jiayu Shen, Yong Xu, Chaoyun Zhang, and Yuzhong Qu....

    arXiv 2024

  6. [9]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720,

    arXiv

  7. [10]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516,

    arXiv

  8. [11]

    Perceptions to beliefs: Exploring precursory inferences for theory of mind in large language models

    Chani Jung, Dongkwan Kim, Jiho Jin, Jiseon Kim, Yeon Seonwoo, Yejin Choi, Alice Oh, and Hyunwoo Kim. Perceptions to beliefs: Exploring precursory inferences for theory of mind in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19794–19809,

    work page 2024

Show all 38 references
  1. [14]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393,

    arXiv

  2. [17]

    Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning

    Melanie Sclar, Jane Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning. arXiv preprint arXiv:2412.12175,

    arXiv

  3. [18]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  4. [19]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256,

    arXiv

  5. [20]

    Tomato: Verbalizing the mental states of role-playing llms for benchmarking theory of mind

    Kazutoshi Shinoda, Nobukatsu Hojo, Kyosuke Nishida, Saki Mizuno, Keita Suzuki, Ryo Masumura, Hiroaki Sugiyama, and Kuniko Saito. Tomato: Verbalizing the mental states of role-playing llms for benchmarking theory of mind. arXiv preprint arXiv:2501.08838,

    arXiv

  6. [22]

    R1-searcher: Incentivizing the search capability in llms via reinforcement learning

    Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592,

    arXiv

  7. [23]

    Climbing the ladder of reasoning: What llms can-and still can’t-solve after sft? arXiv preprint arXiv:2504.11741,

    Yiyou Sun, Georgia Zhou, Hao Wang, Dacheng Li, Nouha Dziri, and Dawn Song. Climbing the ladder of reasoning: What llms can-and still can’t-solve after sft? arXiv preprint arXiv:2504.11741,

  8. [24]

    Helpsteer2: Open-source dataset for training top-performing reward models

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673,

    arXiv

  9. [25]

    Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449,

    arXiv

  10. [26]

    Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models

    Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi-tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10691–10706,

    2023

  11. [27]

    Gui-r1: A generalist r1-style vision-language action model for gui agents

    Xiaobo Xia and Run Luo. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458,

    arXiv

  12. [28]

    Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning

    Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768,

    arXiv

  13. [29]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115,

    arXiv

  14. [30]

    Limo: Less is more for reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387,

    arXiv

  15. [31]

    Z1: Efficient test-time scaling with code

    Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. Z1: Efficient test-time scaling with code. arXiv preprint arXiv:2504.00810,

    arXiv

  16. [32]

    Scaling of search and learning: A roadmap to reproduce o1 from reinforcement learning perspective

    Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Bo Wang, Shimin Li, Yunhua Zhou, Qipeng Guo, Xuanjing Huang, and Xipeng Qiu. Scaling of search and learning: A roadmap to reproduce o1 from reinforcement learning perspective. arXiv preprint arXiv:2412.14135,

    arXiv

  17. [33]

    Autotom: Automated bayesian inverse planning and model discovery for open-ended theory of mind

    Zhining Zhang, Chuanyang Jin, Mung Yao Jia, and Tianmin Shu. Autotom: Automated bayesian inverse planning and model discovery for open-ended theory of mind. arXiv preprint arXiv:2502.15676,

  18. [34]

    Sotopia: Interactive evaluation for social intelligence in language agents

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667,

    arXiv

  19. [35]

    Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks

    Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks. arXiv preprint arXiv:2503.15478,

    arXiv

  20. [36]

    The comparison results are shown in Table

    A Technical Appendices and Supplementary Material A.1 Preliminary Experiments——DeepSeek-R1’s Evaluation Performance on ToMBench We evaluate the performance of the DeepSeek-R1 model on ToMBench, and compare it with models from the GPT-4 [OpenAI, 2023] series, Claude series [Ant...

    2023

  21. [37]

    What LLMs Can—and Still Can’t—Solve after SFT?

    13 Table 5: DeepSeek-R1’s evaluation performance on ToMBench, where UOT represents Unexpected Outcome Test, SIT represents Scalar Implicature Task, PST represents Persuasion Story Task, FBT represents False Belief Task, AST represents Ambiguous Story Task, HT represents Hintin...

    2024

  22. [38]

    budget forcing

    Parallel, which relies on multiple solution attempts generated in parallel and selecting the best via majority voting or reward model (process-based or outcome-based) [Snell et al., 2024, Brown et al., 2024, Liu et al., 2024, Huang et al., 2024b, Wang et al., 2024, Zeng et al....

    2024

  23. [1996]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314,

    arXiv

  24. [2011]

    Revisiting the evaluation of theory of mind through question answering

    Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pr...

    2019

  25. [2013]

    Video-r1: Reinforcing video reasoning in mllms

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025a. Zhaopeng Feng, Shaosheng Cao, Jiahan Ren, Jiayuan Su, Ruizhe Chen, Yan Zhang, Zh...

    arXiv

  26. [2019]

    Relation-r1: Cognitive chain-of-thought guided reinforcement learning for unified relational comprehension

    Lin Li, Wei Chen, Jiahui Li, and Long Chen. Relation-r1: Cognitive chain-of-thought guided reinforcement learning for unified relational comprehension. arXiv preprint arXiv:2504.14642, 2025a. Ming Li, Jike Zhong, Shitian Zhao, Yuxiang Lai, and Kaipeng Zhang. Think or not think...

  27. [2020]

    Social iqa: Common- sense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura...

    2019

  28. [2023]

    11 Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang

    Large language model, November 6 version. 11 Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, and Mao Yang. Mutual reasoning makes smaller llms stronger problem-solvers. arXiv preprint arXiv:2408.06195,

    arXiv

  29. [2024]

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini

    Large language model, October 22 version. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787,

    arXiv

  30. [2025]

    Gencls++: Pushing the boundaries of generative classification in llms through comprehensive sft and rl studies across diverse datasets

    Mingqian He, Fei Zhao, Chonggang Lu, Ziyan Liu, Yue Wang, and Haofu Qian. Gencls++: Pushing the boundaries of generative classification in llms through comprehensive sft and rl studies across diverse datasets. arXiv preprint arXiv:2504.19898,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.