Pith. sign in

REVIEW 5 major objections 7 minor 74 references

Socially intelligent LLMs need coordinated progress in measurement, training, and deployment-time grounding—not bigger models alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 14:19 UTC pith:5PWTYDI6

load-bearing objection A real three-part systems stack for social LLM work—broad benchmark, staged post-training, typed harness—with honest tables and a proxy gap the framing oversells. the 5 major comments →

arxiv 2607.23740 v1 pith:5PWTYDI6 submitted 2026-07-26 cs.CL

Zing: Social Mind for LLMs

Zing Team , Ao Xiang , Bi Jingping , Chen Jiahui , Chen Lehan , Chen Yilin , Cheng Xueqi , Fan Yixing
show 51 more authors
Gan Kairong Gao Haowen Gao Jinhua Gao Shuxuan Gong Chang Guo Jiafeng Guo Ruijie Han Zhouyu He Guangfu He Yichun Jiang Shuo Jing Shaoling Jing Ya Lei Chenhao Lei Yan Li Anqi Li Chengao Li Haoyu Li Shitian Liang Xinjian Liu Zhaoge Lyu Xingyu Nie Zhuwei Pang Liang Quan Zeping Shan Shiguang Shen Huawei Tang Xinran Tian Feng Wang Qian Wang Ruiping Wang Xiaohong Xia Zaiyu Xiao Yi Xu Jiayuan Xu Kehan Xu Qianqian Xu Tianyu Xu Yongjun Yang Haoming Yang Jun Yao Di Yu Xiaoming Zhang Futong Zhang Jie Zhang Shixuan Zhang Yuxuan Zhao Xinyu Zhao Zhuoran Zhong Yunfei Zhu Shengyu
This is my paper
classification cs.CL
keywords social intelligencetheory of mindLLM evaluationSoMBenchdiagnosis-driven trainingreinforcement learningagent harnessdeployment-time grounding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

As language models shift from one-shot task solvers to long-term collaborators in human settings, the paper argues they need a distinct layer of social intelligence: tracking mental states, relations, and norms, then acting under incomplete, changing context. It treats that layer as something that must be measured, built into model weights, and supported at inference time—not left to surface pattern matching. Measurement comes from SoMBench, a psychology-grounded taxonomy of 3 primary and 17 secondary dimensions across 3,481 expert-verified items; current models top out at about 72% and leave clear headroom on every secondary dimension. Internalization comes from Zing, a diagnosis-driven recipe of staged fine-tuning, on-policy distillation, and rubric-based reinforcement that lifts matched base models across five social benchmarks. Grounding comes from Actio, a harness that selectively injects procedural skills, runtime mental-state memory, reusable experience, and gated social knowledge. The combined picture is that usable social mind capability is a three-front problem.

Core claim

The paper’s central claim is that social intelligence in LLMs is not a single skill that emerges from scale or prompting, but a layered capability that only advances when evaluation, parametric training, and deployment-time support move together. SoMBench shows substantial remaining headroom and diagnostic structure; Zing shows social reasoning can be internalized as a more stable model ability through capability-diagnosis-driven staged training; Actio shows typed, gated runtime supports systematically improve frozen bases. Together these establish a technical route from isolated task execution toward social-mind models.

What carries the argument

Zing (知境) as a three-line stack: SoMBench (capability-grounded shared-scenario benchmark with controlled evidence, perspective, and metadata), Zing training (FLARE diagnosis flywheel plus Stage-1 ToM foundations and Stage-2 specialized SFT/OPD/rubric-GRPO), and Actio (harness-gated routing of PRISM skills, Starling state memory, SAGE experience, and gated social RAG into a frozen base model).

Load-bearing premise

That scores on controlled, expert-verified story-and-question scenarios are a good enough stand-in for the paper’s real target—long-term, open social environments—so benchmark and harness gains mean usable social-mind behavior in deployment.

What would settle it

Run the same base models, Zing-trained checkpoints, and Actio harness in interactive multi-turn human or multi-agent settings with evolving relationships and norms; if gains on SoMBench and the five static social benchmarks disappear or reverse while humans still rate the systems as socially brittle, the central coordination claim fails for real use.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Social evaluation should keep shared scenarios and metadata so failures can be traced to belief tracking, norm conflict, or perspective leakage rather than a single leaderboard score.
  • Post-training for social ability should be diagnosis-driven and staged: broad mental-state foundations before specialized affect, strategy, and norm reasoning.
  • At deployment, social support should be typed and gated—procedure, attributed state, experience, external norms—rather than one undifferentiated memory or retrieval dump.
  • Current frontier models still leave large headroom on fine-grained social dimensions, so social mind remains an open capability target rather than a solved side effect of scale.
  • Module-level Actio results imply selective activation can beat uniform context expansion when supports are complementary.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If taxonomy-labeled vignettes systematically miss long-horizon trust and repair dynamics, the next bottleneck may be process-level trajectory data rather than more static items.
  • Harness traces that record gate, path, and validation failures could become a natural feedback signal for the same diagnosis flywheel used in training.
  • Smaller models gaining more from Zing suggests social post-training may be a cheaper path to usable collaborators than waiting for frontier-scale emergence alone.
  • Norm and culture dependence flagged in the limitations implies cross-frame calibration tests will matter as much as average accuracy for real multi-user products.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. This technical report presents a three-part framework for "social mind" LLMs. (1) SoMBench: a psychology-grounded benchmark with 3 primary / 17 secondary dimensions and 71 task paradigms, built via taxonomy-guided seed construction, adversarial context/question rewriting with failure-maximizing model-based filtering, and dual-reviewer human verification, yielding 3,481 verified items over 284 shared scenarios; evaluation of 20 LLMs shows the best model (claude-opus-4-8) at 72.08% overall, with single-choice items near-saturated (94.47%) but multi-select (55.60%), judgment (64.06%), and open-ended (62.86%) items well below ceiling. (2) Zing: a staged post-training recipe (SFT → RL with process/verifiable rewards in Stage 1; capability-oriented SFT → joint on-policy distillation + mixed-reward GRPO in Stage 2) driven by FLARE, a diagnosis-driven data flywheel; Zing models improve over matched Qwen bases across five benchmarks, with Zing-27B-Stage2 achieving the best average (79.80). (3) Actio: a harness-controlled inference architecture wrapping a frozen base LLM with four typed supports (PRISM procedural skills, Starling mental-state memory, SAGE experience memory, gated RAG), improving 14/15 model–benchmark pairs across three benchmarks. The claim is that social intelligence requires coordinated progress on evaluation, parametric internalization, and deployment-time grounding.

Significance. If the results hold, this is a substantive contribution. SoMBench is one of the more carefully engineered social-cognition benchmarks: a pre-registered construct specification (the paradigm quadruple of target construct, required evidence, distractor logic, expected failure mode), shared-context item schema isolating reasoning demand from scenario content, dual-reviewer human verification with blind solving, and metadata-preserving evaluation are all genuine methodological strengths. The Zing training claims are corroborated on four public external benchmarks (ToMBench, EmoBench, FANToM, HiToM) under a unified open-sourced harness (SoMEval), which is exactly the kind of cross-check that anchors a self-built-benchmark paper. Actio provides falsifiable, module-level evidence (repair/damage diagnostics, order-wise Starling gains concentrated at orders 3–4, SAGE fallback ablation) rather than only end-to-end numbers. The coordinated evaluation–training–deployment framing is well supported structurally, and the paper is unusually forthcoming about its own negative results (Stage-2 regressions, PRISM damage cases). The main caveat is that the strongest headline numbers (72.08% headroom,

major comments (5)
  1. [§4.3 (FLARE) and §3 (SoMBench)] Training/evaluation coupling is not decoupled at the item level. FLARE's diagnostic pool is built from 'taxonomy-guided seed cases and benchmark-inspired synthetic cases' on the same SocialMind taxonomy that defines SoMBench, and Stage-2 synthesis is explicitly conditioned on the same capability dimensions SoMBench scores. The LSH deduplication in §4.3.2 is described only against the accumulated diagnostic pool and within the candidate pool; nowhere is an explicit decontamination pass against the 3,481 SoMBench test items stated. The external-benchmark gains (ToMBench, EmoBench, FANToM, HiToM in Table 6) and the fact that Zing-27B-Stage2 does not top SoMBench (67.46 vs GPT-5.5's 68.69 and claude-opus-4-8's 72.08) mitigate contamination risk but do not close it, because shared-taxonomy supervision can inflate in-family scores without item overlap. This is load-bearing for the 'internaliza
  2. [§3.4 (variant filtering) and §3.6 (headline headroom)] The headline headroom claim is partly a function of construction and scoring choices rather than pure capability gap. §3.4's variant filtering explicitly 'rewards target-item errors' and selects four variants per seed by failure-biased softmax sampling, and §3.6 shows single-choice items are already near-saturated (94.47% mean) while the aggregate 72.08% is dragged down by all-or-nothing multi-select scoring (55.60%) and judge-thresholded open-ended items (62.86%). The paper discloses each component but the abstract and §3.6 framing present 72.08% as unqualified capability headroom. A concrete, cheap fix: report the best-of-20 score separately on the 1,704 unfiltered seed instances versus the 1,777 adversarially filtered variants (the metadata already distinguish construction source), and report per-format headroom alongside the aggregate in the abstract and §3.6. If seed-instance head
  3. [§3.6 and §4.6.1 (open-ended scoring)] The scoring of the 481 open-ended items is under-specified in ways that affect all reported Q4 and aggregate numbers. The text says 'two automated judges produce a continuous quality score, which is converted to a binary correctness judgment using the fixed pass threshold used in this release.' Neither the judge models, the threshold value, nor any human-validation of judge agreement (e.g., correlation with expert ratings on a subsample) is reported. Q4 shows the widest model spread (34.47–82.20%), so the judge configuration materially shapes the ranking. Please (i) name the judge models and threshold, (ii) report judge–human agreement on a verified subsample, and (iii) provide judge-swap robustness for the headline ranking (e.g., re-score with a different judge family and report rank correlation).
  4. [§4.6.1 and Table 6 (evaluation protocol); Table 10] Headline quantitative claims rest on single-run numbers with no uncertainty estimates. Zing evaluation uses one sampled response per item at temperature 0.6 / top-p 0.95, so run-to-run variance is non-trivial, yet claims such as 'Zing-27B-Stage2 achieving the best average score' (79.80 vs GPT-5.5's 78.44, a 1.36-point margin) and the Actio '14 of 15 pairs improved' result (mean gain 3.70 pp, several individual gains <1 pp in Table 10) are reported without confidence intervals or significance tests. At minimum, report per-model run variance (e.g., 3 seeds) or bootstrap CIs over items for Table 6 and Table 10, and temper claims whose margins fall within noise.
  5. [§5.7 (Actio harness / offline optimizer)] The offline optimizer updates the runtime policy bundle θ from accumulated execution traces, and the RAG module uses a held-out validation set V for credit assignment, but the provenance of the trace store is not stated. If traces are collected on the evaluation benchmarks themselves, the gating/path policy has been adapted on test data, which would inflate Table 10's system-level gains. SAGE is curated only on the ToMBench training split (good), and V is described as disjoint from evaluation data, but the analogous guarantee for the θ-optimizing traces is missing. Please state explicitly which data the offline optimizer's traces come from for each benchmark in Table 10, and if any test-item traces were used, report a clean condition where θ is fixed from training/validation traces only.
minor comments (7)
  1. [§3.6, Table 4 vs. prose] Two numerical inconsistencies between text and tables: (a) the text says conflict-escalation path prediction has 'a mean of 49.2%' while Table 4 (row 2.4.2) and the Figure 12 discussion give 44.2%; (b) the text says dialogue presupposition recognition 'displays the largest gap, i.e., 42.1 points' while Table 4 (row 2.1.4) reports a 31.4-point gap.
  2. [Abstract] The abstract opens with 'This report presents Zhijing, an integrated framework...' while the rest of the report names the framework Zing (知境). Unify the naming.
  3. [§3.6 (Figure 8) vs. Table 6] GPT-5.5's SoMBench score is 69.09% in §3.6/Figure 8 but 68.69% in Table 6. Presumably different evaluation harnesses/decoding, but the discrepancy should be reconciled or footnoted.
  4. [§5.3.3, Eq. (6)] Eq. (6) defines repair and damage rates, but §5.8.3 reports only family-hit (94.4%) and strict-hit (74.4%) numerically; the actual repair/damage rates are never given despite being the more decision-relevant quantities for the boundary-control design. Please report them.
  5. [§5.8 (Actio evaluation)] Actio's per-item inference cost is not reported. The runtime invokes a router, skill workflows, memory extraction, retrieval, drafting, validation, and possible repair — i.e., multiple LLM calls per item at a 32k token budget. Given that 'deployment-time grounding' is a central claim, latency/cost overhead relative to a single base-model call should be quantified.
  6. [§5.8.3 (diagnostic solver)] The diagnostic-solver model naming is confusing: §5.8.3 and Figure 23 use 'Qwen3.5-27B' while the main evaluation uses Qwen3.6-27B and Figure 24 mentions Qwen3.5-27B as solver; clarify whether these are distinct models and why the diagnostic solver differs from the main base-model pool.
  7. [Typos / references] Reference [19] misspells 'Foucault' as 'Foucoult'. Also: 'SoMBench f is organized' (§3.1), 'the new questions are not not paraphrases' (§3.4), a double period in §3.1 ('exercises..'), and 'reasons about norms' should be 'reason about norms' (§3.3). Several 2026 preprints are cited ([26], [34], [68]); where published versions exist, please update.

Circularity Check

1 steps flagged

Empirical systems paper with mild taxonomy coupling, not by-construction circular derivation.

specific steps
  1. other [§3.4 Variant filtering; §4.3 FLARE diagnostic pool]
    "The filtering score accordingly rewards target-item errors and penalizes excessive non-target instability. From the scored pool we select four variants per seed using temperature-controlled softmax sampling... FLARE's iteration starts from a capability-labeled diagnostic pool constructed from our own data resources: taxonomy-guided seed cases and benchmark-inspired synthetic cases... without reusing their items."

    SoMBench difficulty and FLARE supervision are both driven by the same capability taxonomy and by error-seeking construction (filter for target failures; synthesize to diagnosed gaps). This couples measurement and training infrastructure, so part of 'headroom' and part of 'internalization' live in one team-defined construct space. It is not equation-level circularity: items are human-verified as answerable, training claims not to reuse benchmark items, and gains are also reported on external public benchmarks and frozen-base Actio settings.

full rationale

Zing is an empirical ML report (benchmark + post-training + inference harness), not a first-principles derivation chain. The three headline results are measured outcomes on held-out or external tasks: SoMBench scores over 20 models, Zing gains vs matched Qwen bases on five social-cognition benchmarks (including public ToMBench/EmoBench/FANToM/HiToM), and Actio gains on frozen bases. No fitted parameter is renamed a prediction; no uniqueness theorem is imported from overlapping authors; no ansatz is smuggled in via self-citation. The only mild circularity-adjacent risk is methodological coupling: SoMBench and FLARE both anchor on the same SocialMind taxonomy, and variant filtering explicitly prefers items that induce model errors—so reported headroom and some training signal share a construction family. That does not make the reported accuracies tautological: FLARE states it does not reuse benchmark items; Zing does not lead SoMBench versus GPT-5.5; external-benchmark and frozen-base Actio comparisons supply independent content. Score 2 reflects that residual coupling without elevating it to load-bearing by-construction circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 4 invented entities

This is an empirical systems paper. Load-bearing premises are domain operationalizations and evaluation/training design choices, not free physical constants. The ledger records those premises and the main engineered entities introduced.

free parameters (5)
  • Open-ended Q4 pass threshold / LLM-judge scoring rule = threshold not numerically specified in main text
    Binary correctness for open-ended SoMBench items depends on automated judges and a fixed pass threshold; this choice directly affects reported accuracy and headroom.
  • FLARE filtering thresholds (LSH Jaccard 0.6/0.85, Pass@k categories, repair limits) = Jaccard θ=0.6 external / 0.85 internal; other cutoffs procedural
    Data retained for training depends on hand-set similarity, difficulty, shortcut, and repair rules that shape the supervision distribution.
  • Stage-2 OPD strength λ_OPD and outcome/rubric reward schedule = not fully numeric in main text
    Joint OPD+GRPO behavior and specialization gains depend on regularization weight and progressive reward emphasis chosen by authors.
  • Actio gate/policy bundle θ and module activation rules = policy object; values not released as a single public config in-text
    Harness gains depend on learned/heuristic gating over paths and supports; offline optimizer updates θ from traces.
  • Evaluation decoding settings (temp 0.6 / top-p 0.95 for Zing; temp 0 direct-answer for Actio) = Zing: T=0.6, p=0.95, max 32768; Actio: T=0, thinking off
    Reported scores are protocol-dependent; different decoding regimes are used across sections.
axioms (6)
  • domain assumption Social intelligence for LLMs can be decomposed into a fixed taxonomy of 3 primary / 17 secondary / 71 paradigm constructs with predefined evidence, distractors, and failure modes.
    §3.1–3.3 make this the measurement specification before item writing; all diagnosis and much training routing inherit it.
  • domain assumption Shared-scenario multi-format items with controlled perspective/length isolate social-cognitive operations rather than surface story confounds.
    Core SoMBench design principle in §3.1 and §3.4; interprets format and dimension gaps as capability structure.
  • domain assumption Expert dual review (psychology-background annotators) plus metadata yields sufficiently gold labels for diagnostic leaderboards.
    §3.5 verification workflow underwrites the 3,481-instance claims.
  • domain assumption Diagnosis-driven synthetic data + teacher CoT + rubric/process rewards internalize stable social reasoning rather than benchmark-specific patterns.
    §4 training philosophy; needed to read Zing gains as capability internalization.
  • domain assumption Typed external supports (procedure, state, experience, norms) can be selectively gated without replacing narrative evidence as the primary ground truth.
    §5 Actio design; underwrites interpretation of harness gains as grounding rather than answer leakage.
  • standard math Standard supervised fine-tuning, on-policy distillation, and group-relative policy optimization are valid optimization backbones for these objectives.
    Uses common LLM post-training machinery (SFT, OPD, GRPO/RLVR) as given tools.
invented entities (4)
  • SoMBench taxonomy and shared-context item schema independent evidence
    purpose: Define and measure social-mind capabilities with diagnostic metadata.
    New benchmark artifact; independent of any single base model, but constructed by the authors.
  • FLARE (Failure-Loop Augmented Refinement Engine) no independent evidence
    purpose: Close the loop from capability-labeled failures to synthesized/filtered training data.
    Author-built data engine; value shown only via downstream Zing metrics.
  • Zing staged training recipe / model family independent evidence
    purpose: Internalize social reasoning via Stage-1 ToM foundation and Stage-2 specialized RL/distillation.
    New trained artifacts and recipe; external benchmark gains provide partial independent handle.
  • Actio harness with PRISM, Starling, SAGE, gated social-mental RAG no independent evidence
    purpose: Provide selective deployment-time supports for social reasoning over a frozen LLM.
    System invention; module ablations give internal evidence, not external physical prediction.

pith-pipeline@v1.2.0-grok45-kimik3 · 53843 in / 4119 out tokens · 80234 ms · 2026-07-30T14:19:04.955145+00:00 · methodology

0 comments
read the original abstract

As large language models move from isolated task solving toward long-term service in human environments, they require social intelligence: the ability to infer mental states, track social relations, reason over norms, and adapt behavior under context. This report presents Zhijing, an integrated framework for measuring, internalizing, and grounding social intelligence. For measurement, we introduce SoMBench, a psychology-grounded benchmark spanning 3 primary dimensions, 17 secondary dimensions, and 71 task paradigms. It controls question format, narrative perspective, and context length across 284 shared scenarios and 3,481 expert-verified instances. Evaluation of 20 representative LLMs reveals substantial headroom: the best model achieves only 72.08% overall accuracy, and none of the 17 secondary dimensions reaches the 90% near-ceiling band. For internalization, we develop Zing, a diagnosis-driven training recipe combining supervised fine-tuning, on-policy distillation, and rubric-based reinforcement learning. Across five social-cognition benchmarks, Zing consistently outperforms its base models, with Zing-27B-Stage2 achieving the best average score and Zing-32B-Stage2 remaining competitive with DeepSeek-V4-Pro. For deployment-time grounding, we build Actio, a harness-controlled inference architecture that routes four typed supports into reasoning: PRISM for procedural guidance, Starling for runtime mental-state representation, SAGE for reusable experience, and gated RAG for external social and normative knowledge. Across five base models and three benchmarks, the full harness improves 14 of 15 model-benchmark pairs and is best or tied for best in 8, demonstrating the effectiveness of typed runtime support. Together, these results show that socially intelligent LLMs require coordinated advances in evaluation, parametric internalization, and deployment-time grounding.

Figures

Figures reproduced from arXiv: 2607.23740 by Ao Xiang, Bi Jingping, Cheng Xueqi, Chen Jiahui, Chen Lehan, Chen Yilin, Fan Yixing, Gan Kairong, Gao Haowen, Gao Jinhua, Gao Shuxuan, Gong Chang, Guo Jiafeng, Guo Ruijie, Han Zhouyu, He Guangfu, He Yichun, Jiang Shuo, Jing Shaoling, Jing Ya, Lei Chenhao, Lei Yan, Liang Xinjian, Li Anqi, Li Chengao, Li Haoyu, Li Shitian, Liu Zhaoge, Lyu Xingyu, Nie Zhuwei, Pang Liang, Quan Zeping, Shan Shiguang, Shen Huawei, Tang Xinran, Tian Feng, Wang Qian, Wang Ruiping, Wang Xiaohong, Xiao Yi, Xia Zaiyu, Xu Jiayuan, Xu Kehan, Xu Qianqian, Xu Tianyu, Xu Yongjun, Yang Haoming, Yang jun, Yao Di, Yu Xiaoming, Zhang Futong, Zhang Jie, Zhang Shixuan, Zhang Yuxuan, Zhao Xinyu, Zhao Zhuoran, Zhong Yunfei, Zhu Shengyu, Zing Team.

Figure 1
Figure 1. Figure 1: Technical overview of Zing (知境) across evaluation, training, and deployment-time grounding, highlighting the main artifacts exposed by each implementation line. 2 Technical Overview The following sections develop the three components of Zing (知境) introduced above as the main technical body of this report. The evaluation line starts from the need to keep the evaluated capability identifiable throughout benc… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the SoMBench benchmark workflow. The workflow starts from a capabil [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A three-level overview of the SoMBench taxonomy. The figure presents the complete structure, in which 3 first-level dimensions are refined into 17 second-level dimensions and further into 71 task paradigms. Under each second-level dimension, the figure lists the fine-grained task paradigms it contains. Mentalizing corresponds to the micro intra-mental level. It captures the ability to infer un￾observable m… view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the SoMBench data-construction pipeline. First, taxonomy￾grounded specification derives construction constraints from fine-grained paradigms. Second, a shared-context seed case is constructed, where the same social context supports single-choice, multiple-choice, judgment, and open-ended questions. Third, controlled variant construction produces two types of candidates. Finally, model-based fil… view at source ↗
Figure 5
Figure 5. Figure 5: Coverage comparison across social-cognition benchmarks. Columns corre￾spond to the second-level dimensions in the SoMBench taxonomy and are grouped into Men￾talizing, Social Interaction, and Social Norms. Dark blue marks primary coverage, light blue marks secondary coverage, and blank cells indicate that the dimension is not a central target of the benchmark. SoMBench provides systematic primary coverage a… view at source ↗
Figure 6
Figure 6. Figure 6: Evaluation-depth comparison across social-cognition benchmarks. (a) Each point is one reported metric or dimension, and its value is the best score among the evaluated models rather than the score of a particular model. The grey band marks the 90–100% near￾ceiling region. (b) Fraction of reported dimensions whose best-of-pool score reaches at least 90%. (c) Median remaining headroom to 100%, computed as 1 … view at source ↗
Figure 7
Figure 7. Figure 7: Difficulty and discrimination on shared dimensions. (a) Difficulty, defined as 100−accuracy, for six dimensions shared by SoMBench and public social-cognition benchmarks. (b) Discrimination, defined as the best-minus-worst score across the model pool. SoMBench values are equal-weight means over the four question types; public values are equal-weight means over the contributing benchmark-level metrics. The … view at source ↗
Figure 8
Figure 8. Figure 8: Overall accuracy of the 20 evaluated LLMs on SoMBench. Models are ordered by [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Accuracy by question type. Single-choice items are near-saturated for many models, [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Accuracy across the three primary dimensions of SoMBench: Mentalizing, Strategic [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Secondary-dimension radar plots. The radial axis is truncated to emphasize model [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Selected fine-grained task-paradigm scores. are selected over all 20 evaluated models: (a) the eight paradigms with the lowest cross-model means and (b) the eight with the largest best-minus-worst gaps. Columns show 13 representative models ordered by overall accuracy. Perspective and context effects [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Perspective and context-length effects on SoMBench. (a) Per-model perspec￾tive gaps, computed as third-person minus first-person accuracy. (b) Perspective gaps averaged by secondary dimension. (c) Per-model context-length gaps, computed as long minus short accuracy. (d) Context-length gaps averaged by secondary dimension. Green indicates that the first condition in each difference is easier; red indicates… view at source ↗
Figure 14
Figure 14. Figure 14: Training Pipeline Overview. 4.2 Training Pipeline Overview Zing implements the two principles above through a diagnosis-driven data flywheel and a stage￾wise post-training path, instantiated on Qwen-family backbones. The data flywheel adapts su￾pervision to the model’s current capability gaps, while the stage-wise path organizes capability internalization from broad ToM foundations to more specialized soc… view at source ↗
Figure 15
Figure 15. Figure 15: Capability diagnosis-driven data iteration framework. [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Experimental results of the general Social Mind ability enhancement in Stage 1. [PITH_FULL_IMAGE:figures/full_fig_p036_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Experimental results of the specialized Social Mind ability enhancement in Stage 2. [PITH_FULL_IMAGE:figures/full_fig_p038_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: System overview of Actio. A frozen base LLM is wrapped by a harness; four support [PITH_FULL_IMAGE:figures/full_fig_p040_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Overview of PRISM. (A) The library is two-level: 20 macro skills are scene-level entry points (false belief, faux pas, persuasion, . . . ) that route a story to a recognizable scenario; micro skills are atomic capabilities organized into layers L0–L7, deployed as 56 skills across eight semantic families (one per layer) that separate easily conflated capabilities, for a 76-skill routing pool. (B) A dedicat… view at source ↗
Figure 20
Figure 20. Figure 20: PRISM layer library (L0–L7): each row states a layer’s core capability and typical [PITH_FULL_IMAGE:figures/full_fig_p043_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Overview of SAGE. The offline loop distills and consolidates reusable reasoning [PITH_FULL_IMAGE:figures/full_fig_p045_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Overall architecture of the Actio harness: an online runtime (inner loop) gates, [PITH_FULL_IMAGE:figures/full_fig_p049_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Task-level accuracy changes produced by PRISM on ToMBench, using Qwen3.5-27B [PITH_FULL_IMAGE:figures/full_fig_p052_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Routing diagnostics on ToMBench using Qwen3.5-27B ( [PITH_FULL_IMAGE:figures/full_fig_p053_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: HiToM accuracy by reasoning order under the same-model-in-the-loop protocol. Bars [PITH_FULL_IMAGE:figures/full_fig_p054_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Component ablation of SAGE on the ToMBench test split. Dimension routing [PITH_FULL_IMAGE:figures/full_fig_p055_26.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

74 extracted references · 7 linked inside Pith

  1. [1]

    Basic Books, 1984

    Robert Axelrod.The Evolution of Cooperation. Basic Books, 1984

    1984

  2. [2]

    Mt-bench-101: A fine-grained benchmark for evaluatinglargelanguagemodelsinmulti-turndialogues

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. Mt-bench-101: A fine-grained benchmark for evaluatinglargelanguagemodelsinmulti-turndialogues. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7421–7454, 2024

    2024

  3. [3]

    MIT press, 1997

    Simon Baron-Cohen.Mindblindness: An essay on autism and theory of mind. MIT press, 1997

    1997

  4. [4]

    Beauchamp

    Cindy Beaudoin, Élizabel Leblanc, Charlotte Gagner, and Miriam H. Beauchamp. Sys- tematic review and inventory of theory of mind measures for young children.Frontiers in Psychology, 10:2905, 2020

    2020

  5. [5]

    Reward and guidance through rubrics: Promoting exploration to improve multi-domain reasoning

    Baolong Bi, Shenghua Liu, Yiwei Wang, Siqian Tong, Lingrui Mei, Yuyao Ge, Yilong Xu, Jiafeng Guo, and Xueqi Cheng. Reward and guidance through rubrics: Promoting exploration to improve multi-domain reasoning. InInternational conference on machine learning, 2026

    2026

  6. [6]

    Cambridge University Press, Cambridge, 1987

    PenelopeBrownandStephenC.Levinson.Politeness: Some Universals in Language Usage. Cambridge University Press, Cambridge, 1987

    1987

  7. [7]

    Martin Buber.Ich und Du. 1923. English translation:I and Thou, trans. Ronald Gregor Smith, T. & T. Clark, 1937. 56 Zing (知境) Technical Report

    1923

  8. [8]

    Socialbench: Sociality evaluation of role- playing conversational agents

    Hongzhan Chen, Hehong Chen, Ming Yan, Wenshen Xu, Gao Xing, Weizhou Shen, Xiaojun Quan, Chenliang Li, Ji Zhang, and Fei Huang. Socialbench: Sociality evaluation of role- playing conversational agents. InFindings of the Association for Computational Linguistics ACL 2024, pages 2108–2126. Association for Computational Linguistics, 2024

    2024

  9. [9]

    Tombench: Bench- marking theory of mind in large language models

    Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. Tombench: Bench- marking theory of mind in large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15959–15983. Association...

    2024

  10. [10]

    Social sycophancy: A broader understanding of llm sycophancy.arXiv preprint arXiv:2505.13995, 2025

    Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Juraf- sky. Social sycophancy: A broader understanding of llm sycophancy.arXiv preprint arXiv:2505.13995, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Cialdini and Melanie R

    Robert B. Cialdini and Melanie R. Trost. Social influence: Social norms, conformity, and compliance. In Daniel T. Gilbert, Susan T. Fiske, and Gardner Lindzey, editors,The Handbook of Social Psychology, volume 2, pages 151–192. McGraw-Hill, 4th edition, 1998

    1998

  12. [12]

    Goemotions: Adatasetoffine-grainedemotions

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, andSujithRavi. Goemotions: Adatasetoffine-grainedemotions. InProceedings of the 58th annual meeting of the association for computational linguistics, pages 4040–4054, 2020

    2020

  13. [13]

    Cambridge University Press, 1986

    Willem Doise.Levels of Explanation in Social Psychology. Cambridge University Press, 1986

    1986

  14. [14]

    Eikelboom, Esther van den Berg, Miriam H

    Willem S. Eikelboom, Esther van den Berg, Miriam H. Beauchamp, Katherine O. Bray, Fiona Kumfor, Sarah E. MacPherson, Skye McDonald, Jacoba M. Spikman, and Roy P. C. Kessels. Providing a taxonomy for social cognition: how to bridge the gap between expert opinion, empirical data, and theoretical models.Journal of Psychiatry and Neuroscience, 50(3):E157–E161, 2025

    2025

  15. [15]

    Hwang, Maxwell Forbes, and Yejin Choi

    Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 698–718. Association for Computational Linguistics, 2021

    2021

  16. [16]

    Hwang, Maxwell Forbes, and Yejin Choi

    Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 698–718, nov 2021

    2021

  17. [17]

    A theory of social comparison processes.Human Relations, 7(2):117–140, 1954

    Leon Festinger. A theory of social comparison processes.Human Relations, 7(2):117–140, 1954

    1954

  18. [18]

    Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi

    Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. Social chemistry 101: Learning to reason about social and moral norms. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653–670. Association for Computational Linguistics, 2020

    2020

  19. [19]

    Discipline and punish.A

    Michel Foucoult. Discipline and punish.A. Sheridan, Tr., Paris, FR, Gallimard, 1975

    1975

  20. [20]

    John R. P. French and Bertram Raven. The bases of social power. In Dorwin Cartwright, editor,Studies in Social Power, pages 150–167. Institute for Social Research, Ann Arbor, MI, 1959. 57 Zing (知境) Technical Report

    1959

  21. [21]

    Frith and Uta Frith

    Chris D. Frith and Uta Frith. The neural basis of mentalizing.Neuron, 50(4):531–534, 2006

    2006

  22. [22]

    Violence, peace, and peace research.Journal of Peace Research, 6(3):167– 191, 1969

    Johan Galtung. Violence, peace, and peace research.Journal of Peace Research, 6(3):167– 191, 1969

    1969

  23. [23]

    H. P. Grice. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors,Syntax and Semantics, Volume 3: Speech Acts, pages 41–58. Academic Press, 1975

    1975

  24. [24]

    James J. Gross. The emerging field of emotion regulation: An integrative review.Review of General Psychology, 2(3):271–299, 1998

    1998

  25. [25]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Towards realistic personalization: Evalu- ating long-horizon preference following in personalized user-llm interactions.arXiv preprint arXiv:2603.04191, 2026

    Qianyun Guo, Yibo Li, Yue Liu, and Bryan Hooi. Towards realistic personalization: Evalu- ating long-horizon preference following in personalized user-llm interactions.arXiv preprint arXiv:2603.04191, 2026

    arXiv 2026

  27. [27]

    The emotional dog and its rational tail: A social intuitionist approach to moral judgment.Psychological Review, 108(4):814–834, 2001

    Jonathan Haidt. The emotional dog and its rational tail: A social intuitionist approach to moral judgment.Psychological Review, 108(4):814–834, 2001

    2001

  28. [28]

    Cook, and Geoffrey Bird

    Francesca Happé, Jennifer L. Cook, and Geoffrey Bird. The structure of social cognition: In(ter)dependence of sociocognitive processes.Annual Review of Psychology, 68(1):243– 267, 2017

    2017

  29. [29]

    Aligning ai with shared human values

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. InInternational Conference on Learning Representations, 2021

    2021

  30. [30]

    Si-bench: Benchmarking social intelligence of large language models in human-to-human conversations.arXiv preprint arXiv:2510.23182, 2025

    Shuai Huang, Wenxuan Zhao, and Jun Gao. Si-bench: Benchmarking social intelligence of large language models in human-to-human conversations.arXiv preprint arXiv:2510.23182, 2025

    arXiv 2025

  31. [31]

    Tenenbaum, and Yejin Choi

    Hyunwoo Kim, Melanie Sclar, Tan Zhi-Xuan, Lance Ying, Sydney Levine, Yang Liu, Joshua B. Tenenbaum, and Yejin Choi. Hypothesis-driven theory-of-mind reasoning for large language models.arXiv preprint arXiv:2502.11881, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Fantom: A benchmark for stress-testing machine theory of mind in interac- tions

    Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. Fantom: A benchmark for stress-testing machine theory of mind in interac- tions. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14397–14413. Association for Computational Linguistics, 2023

    2023

  33. [33]

    Stage and sequence: The cognitive-developmental approach to social- ization.Handbook of Socialization Theory and Research, pages 347–480, 1969

    Lawrence Kohlberg. Stage and sequence: The cognitive-developmental approach to social- ization.Handbook of Socialization Theory and Research, pages 347–480, 1969

    1969

  34. [34]

    Deepbias: Adaptive in-depth probing of social biases in lvlms.arXiv preprint arXiv:2607.11228, 2026

    Anqi Li, Jie Zhang, Zhongqi Wang, Songkai Xue, Jiahao Wang, Shiguang Shan, and Xilin Chen. Deepbias: Adaptive in-depth probing of social biases in lvlms.arXiv preprint arXiv:2607.11228, 2026

    Pith/arXiv arXiv 2026

  35. [35]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, volume2024, pages39578–39601, 2024. 58 Zing (知境) Technical Report

    2024

  36. [36]

    Yiwei Liu, Emma Jane Pretty, Jiahao Huang, and Saku Sugawara. Tactfultom: Do llms have the theory of mind ability to understand white lies? InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25043–25061, 2025

    2025

  37. [37]

    On-policy distillation.Thinking Machines Lab: Connectionism, 2025

    Kevin Lu and Thinking Machines Lab. On-policy distillation.Thinking Machines Lab: Connectionism, 2025. https://thinkingmachines.ai/blog/on-policy-distillation

    2025

  38. [38]

    Advancingsocialintelligencein ai agents: Technical challenges and open questions

    LeenaMathur, PaulPuLiang, andLouis-PhilippeMorency. Advancingsocialintelligencein ai agents: Technical challenges and open questions. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20541–20560. Association for Computational Linguistics, 2024

    2024

  39. [39]

    Eval- uating theory of mind in question answering

    Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. Eval- uating theory of mind in question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2392–2400, 2018

    2018

  40. [40]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedbac...

    2022

  41. [41]

    Bernstein

    Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technol- ogy, pages 1–22. ACM, 10 2023

    2023

  42. [42]

    Does the chimpanzee have a theory of mind?Behav- ioral and Brain Sciences, 1(4):515–526, 1978

    David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind?Behav- ioral and Brain Sciences, 1(4):515–526, 1978

    1978

  43. [43]

    Emobench: Evaluating the emotional in- telligence of large language models

    Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, and Minlie Huang. Emobench: Evaluating the emotional in- telligence of large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5986–6004. Association for Compu...

    2024

  44. [44]

    Social iqa: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Con- ference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473. Association for Computatio...

    2019

  45. [45]

    Schaafsma, Donald W

    Sara M. Schaafsma, Donald W. Pfaff, Robert P. Spunt, and Ralph Adolphs. Deconstructing and reconstructing theory of mind.Trends in Cognitive Sciences, 19(2):65–72, 2015

    2015

  46. [46]

    Tholen, Lara Maliske, Daniel S

    Matthias Schurz, Joaquim Radua, Matthias G. Tholen, Lara Maliske, Daniel S. Margulies, Rogier B. Mars, Jerome Sallet, and Philipp Kanske. Toward a hierarchical model of social cognition: A neuroimaging meta-analysis and integrative review of empathy and theory of mind.Psychological Bulletin, 147(3):293–327, 2021

    2021

  47. [47]

    Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning

    Melanie Sclar, Jane Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning. InInternational Conference on Learning Representations, 2025. 59 Zing (知境) Technical Report

    2025

  48. [48]

    Scott.Weapons of the Weak: Everyday Forms of Peasant Resistance

    James C. Scott.Weapons of the Weak: Everyday Forms of Peasant Resistance. Yale University Press, 1985

    1985

  49. [49]

    John R. Searle. Indirect speech acts. In Peter Cole and Jerry L. Morgan, editors,Syntax and Semantics, Volume 3: Speech Acts, pages 59–82. Academic Press, 1975

    1975

  50. [50]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Mindgames: Targeting theory of mind in large lan- guage models with dynamic epistemic modal logic

    Damien Sileo and Antoine Lernould. Mindgames: Targeting theory of mind in large lan- guage models with dynamic epistemic modal logic. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 4570–4577. Association for Computa- tional Linguistics, 2023

    2023

  52. [52]

    Soto, Christopher M

    Christopher J. Soto, Christopher M. Napolitano, Madison N. Sewell, Hee J. Yoon, and Brent W. Roberts. An integrative framework for conceptualizing and assessing social, emotional, and behavioral skills: The bessi.Journal of Personality and Social Psychology, 123(1):192–222, 2022

    2022

  53. [53]

    Pub: A pragmatics understanding benchmark for assessing llms’ pragmat- ics capabilities

    Settaluri Sravanthi, Meet Doshi, Pavan Tankala, Rudra Murthy, Raj Dabre, and Pushpak Bhattacharyya. Pub: A pragmatics understanding benchmark for assessing llms’ pragmat- ics capabilities. InFindings of the Association for Computational Linguistics ACL 2024, pages 12075–12097. Association for Computational Linguistics, 2024

    2024

  54. [54]

    MIT Press, 2009

    Michael Tomasello.Why We Cooperate. MIT Press, 2009

    2009

  55. [55]

    Cogtom: A comprehensive theory of mind benchmark inspired by human cognition for large language models.arXiv preprint arXiv:2601.15628, 2026

    Haibo Tong, Zeyang Yue, Feifei Zhao, Erliang Lin, Lu Jia, Ruolin Chen, Yinqian Sun, Qian Zhang, and Yi Zeng. Cogtom: A comprehensive theory of mind benchmark inspired by human cognition for large language models.arXiv preprint arXiv:2601.15628, 2026

    arXiv 2026

  56. [56]

    Cambridge University Press, 1983

    Elliot Turiel.The Development of Social Knowledge: Morality and Convention. Cambridge University Press, 1983

    1983

  57. [57]

    Decodingtrust: A compre- hensive assessment of trustworthiness in GPT models

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, et al. Decodingtrust: A compre- hensive assessment of trustworthiness in GPT models. InAdvances in Neural Information Processing Systems 36 (Datasets and Benchmarks Track), 2023

    2023

  58. [58]

    Asurvey on large language model based autonomous agents.Frontiers of Computer Science, 18(6), 12 2024

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, JiakaiTang, XuChen, YankaiLin, WayneXinZhao, ZheweiWei, andJirongWen. Asurvey on large language model based autonomous agents.Frontiers of Computer Science, 18(6), 12 2024

    2024

  59. [59]

    Sotopia-π: Interactive learning of socially intelligent language agents

    Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Yonatan Bisk, Gra- ham Neubig, and Hao Zhu. Sotopia-π: Interactive learning of socially intelligent language agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12912–12940. Association for Computational Linguistics, 2024

    2024

  60. [60]

    Reinforcement learning op- timization for large-scale learning: An efficient and user-friendly scaling library.arXiv preprint arXiv:2506.06122, 2025

    Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, et al. Reinforcement learning op- timization for large-scale learning: An efficient and user-friendly scaling library.arXiv preprint arXiv:2506.06122, 2025. 60 Zing (知境) Technical Report

    Pith/arXiv arXiv 2025

  61. [61]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  62. [62]

    Scaling of theory-of-mind tasks.Child Development, 75(2):523–541, 3 2004

    Henry M Wellman and David Liu. Scaling of theory-of-mind tasks.Child Development, 75(2):523–541, 3 2004

    2004

  63. [63]

    Beliefs about beliefs: Representation and constrain- ing function of wrong beliefs in young children’s understanding of deception.Cognition, 13(1):103–128, 1983

    Heinz Wimmer and Josef Perner. Beliefs about beliefs: Representation and constrain- ing function of wrong beliefs in young children’s understanding of deception.Cognition, 13(1):103–128, 1983

    1983

  64. [64]

    Long- memeval: Benchmarking chat assistants on long-term interactive memory

    Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Long- memeval: Benchmarking chat assistants on long-term interactive memory. InInternational Conference on Learning Representations, 2025

    2025

  65. [65]

    Hi- tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models

    Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi- tom: A benchmark for evaluating higher-order theory of mind reasoning in large language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10691–10706. Association for Computational Linguistics, 2023

    2023

  66. [66]

    The rise and potential of large language model based agents: a survey.Science China Information Sciences, 68(2), 2 2025

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, Qi Zhang, and Tao Gui. Th...

    2025

  67. [67]

    Large language models as theory of mind aware generative agents with counterfactual reflection.arXiv preprint arXiv:2501.15355, 2025

    Bo Yang, Jiaxian Guo, Yusuke Iwasawa, and Yutaka Matsuo. Large language models as theory of mind aware generative agents with counterfactual reflection.arXiv preprint arXiv:2501.15355, 2025

    Pith/arXiv arXiv 2025

  68. [68]

    Benchpres: A benchmark for context-aware personalized preference selectivity of persistent-memory llms.arXiv preprint arXiv:2603.16557, 2026

    Sangyeon Yoon, Sunkyoung Kim, Hyesoo Hong, Wonje Jeung, Yongil Kim, Wooseok Seo, Heuiyeen Yeen, and Albert No. Benchpres: A benchmark for context-aware personalized preference selectivity of persistent-memory llms.arXiv preprint arXiv:2603.16557, 2026

    arXiv 2026

  69. [69]

    Agentic context engineering: Evolving contexts for self-improving language models

    Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsid- har Kamanuru, Jay Rainton, Chen Wu, Mengmeng Ji, Hanchen Li, Urmish Thakker, James Zou, and Kunle Olukotun. Agentic context engineering: Evolving contexts for self-improving language models. 2026

    2026

  70. [70]

    Safetybench: Evaluating the safety of large language models

    Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15537–15553, 2024

    2024

  71. [71]

    AutoToM: Automated bayesian inverse planning and model discovery for open-ended theory of mind

    Zhining Zhang, Chuanyang Jin, Mung Yao Jia, and Tianmin Shu. AutoToM: Automated bayesian inverse planning and model discovery for open-ended theory of mind. 2025

    2025

  72. [72]

    Socialeval: Evaluating social intelligence of large language models

    Jinfeng Zhou, Yuxuan Chen, Yihan Shi, Xuanming Zhang, Leqi Lei, Yi Feng, Zexuan Xiong, Miao Yan, Xunzhi Wang, Yaru Cao, Jianing Yin, Shuai Wang, Quanyu Dai, Zhen- hua Dong, Hongning Wang, and Minlie Huang. Socialeval: Evaluating social intelligence of large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Ling...

    2025

  73. [73]

    Sotopia: Interactive evaluation for social intelligence in language agents

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis- Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. Sotopia: Interactive evaluation for social intelligence in language agents. InInternational Conference on Learning Representations, 2024

    2024

  74. [74]

    A believes X

    Caleb Ziems, Jane Dwivedi-Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. Normbank: A knowledge bank of situational social norms. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7756–7776. Association for Computational Linguistics, 2023. 62 Zing (知境) Technical Report A Authors This pro...

    2023