Pith. sign in

REVIEW 5 major objections 5 minor 51 references

This report sets out to establish that an 8-billion-parameter language model can be an on-device robot brain: it can keep the general intelligence of a foundation LLM, gain high-level embodied interaction skills, and answer in far fewer tok

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:47 UTC pith:HC2UCGYQ

load-bearing objection Solid system report with a genuinely interesting merge chapter; the big embodied numbers come from in-domain training, so the 'general robot brain' framing oversells the evidence. the 5 major comments →

arxiv 2607.18985 v2 pith:HC2UCGYQ submitted 2026-07-21 cs.AI

Athena-Brain Technical Report: An Efficient Robot Brain for General Intelligence and Embodied Interaction

Jialian Li , Junhong Liu , Yuchen Cao , Weiran Guo , Jiaming Song , Xutao Wang , Yi Zhao , Jiangpin Liu
show 1 more author
Jie Chen
This is my paper
classification cs.AI
keywords embodied AIrobot brainmodel mergingreinforcement learningtoken efficiencyLLM post-trainingon-device language modeltext-based interaction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This report tries to establish that an 8-billion-parameter language model can serve as an on-device robot brain: it can preserve the general reasoning, coding, and tool-use abilities of a foundation LLM while gaining high-level embodied interaction skills, and it can do so concisely. The authors argue this is possible through a four-stage post-training recipe — general supervised fine-tuning, general reinforcement learning with a token-budget reward, embodied expert reinforcement learning in interactive Museum and Supermarket environments, and a lineage-aware two-stage parameter merge — rather than through a larger model or a single specialized objective. On the paper's in-domain embodied benchmarks, the final model reaches 58.5% overall success versus 6.98% for its SFT-only checkpoint, roughly matches a same-scale thinking model on general benchmarks, and needs only 24.25 generated tokens per interaction step. A sympathetic reader would care because it points to a concrete way to get both generality and embodied competence out of a small deployable model, where prior work typically sacrificed one for the other.

Core claim

The paper's central claim is that the tension between general intelligence and embodied competence in a compact model is not a trade-off but a composition problem. Starting from an open 8B base, the authors build a general foundation with supervised fine-tuning, sharpen reasoning and penalize verbosity with reinforcement learning, specialize a branch for closed-loop interaction in two executable environments, and then merge the branches. The resulting single checkpoint, Athena-Brain-8B, outperforms all same-scale open models on the in-house Museum and Supermarket interaction suites (58.52% overall success, compared with 6.98% for the SFT-only intermediate), roughly matches or exceeds its sam

What carries the argument

The load-bearing mechanism is the staged parametrization of the problem. First, a token-budget reward inside GRPO — a reinforcement-learning update that compares groups of rollouts by relative reward — pulls the model toward concise reasoning without sacrificing correctness. Then embodied expert training creates a specialized branch through supervised warm-up and GRPO on executable interactions, with a curriculum of four embodiment levels ranging from zone-level graph planning to egocentric view-cone navigation. Finally a lineage-aware merge combines same-origin RL experts with TIES, a merging method that keeps only parameter updates whose signs agree across experts, before a low-weight line

Load-bearing premise

The 58.52% in-domain success is measured on held-out instances of the same two environment families (Museum and Supermarket) used for embodied training, while comparison models were evaluated zero-shot; if those test instances share the action parser, observation templates, or map priors with training, the result reflects in-distribution adaptation rather than a general robot brain.

What would settle it

Run Athena-Brain-8B zero-shot on a third environment family built with the same interaction protocol but new semantics, observation formats, and map layouts, or on held-out maps within the existing environments after checking for map overlap. If overall success falls to the level of the 6.98% SFT checkpoint or to the 7–14% range of zero-shot same-scale baselines, the paper's generalization claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A compact 8B model can match or beat same-scale thinking models on general math, code, reasoning, and tool-use benchmarks while generating substantially fewer tokens, making response efficiency itself a trainable objective rather than a post-hoc constraint.
  • Embodied competence can be added to a general LLM without destroying its generality: the merged model retains most of the embodied expert's in-domain success (58.52% vs 60.11%) while improving on most general benchmarks relative to the best single expert.
  • Interaction-policy errors are recoverable: the ablation shows that truncating rollouts only on format violations, using mild reward penalties, and training on easier maps yields the best final policy, because it preserves recovery trajectories after wrong but executable actions.
  • The skills acquired in two text environments transfer to a third, unseen environment: zero-shot success on ALFWorld rises from 11.94% (SFT checkpoint) to 36.57%.
  • Token cost per unit of embodied performance can be cut by an order of magnitude: 0.41 for Athena-Brain-8B versus 11.96 for its SFT checkpoint and 36.81 for a same-scale thinking baseline, which matters for latency-sensitive on-device deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The ALFWorld transfer result, not the 58.52% in-domain score, is the strongest evidence for a general robot brain; the in-domain comparison is confounded by training on the same two environment families. A fair test would hold out an entire environment family or a substantially different action grammar.
  • Because the mild-reward/format-only-truncation configuration prevents RL from over-optimizing protocol avoidance, one testable prediction is that the same configuration will help other tool-use agents — code repair, computer use — where recoverable errors carry useful signal.
  • The lineage-aware merge principle (TIES among same-origin updates, cautious interpolation across lineages) could generalize to any multi-skill post-training stack, such as merging math, code, and tool-use RL experts and later adding a new SFT lineage.
  • The reported 24.25 tokens per step suggests the model has internalized the action schema; a testable extension is to measure whether token efficiency degrades gracefully as observations become more ambiguous, since E3 success rates of 25% (Museum) and 15% (Supermarket) show headroom.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Athena-Brain-8B, an 8B model built from Qwen3-8B-Base through four post-training stages: General SFT, General RL with a token-budget reward, Embodied Expert RL in two text-based environments (Museum, Supermarket), and a two-stage TIES/linear model merge. The authors report that the final model is competitive with Qwen3-8B-Thinking on general benchmarks while generating far fewer tokens, and that it reaches 58.52% average success on in-domain Museum/Supermarket tasks versus 6.98% for the SFT-only checkpoint, requiring only 24.25 generated tokens per interaction. They also report zero-shot ALFWorld transfer and two intermediate probes.

Significance. The training recipe is well structured and the paper contains genuinely useful empirical content: detailed ablations of the General RL rollout budget (§5.2), the embodied RL truncation/reward/map-sampling axes (§5.3), and a lineage-aware merge analysis (§5.1). The token-efficiency results, 24.25 tokens per interaction step, are striking. However, the load-bearing evidence for the embodied half of the central claim is in-domain; if the authors can add a convincing held-out environment evaluation or clearly reframe the contribution as in-domain capability acquisition, the work would be a useful technical report. As it stands, the significance is conditional on that fix.

major comments (5)
  1. [§4.2.2 / Table 5] The central embodied claim is based on an in-distribution comparison. The manuscript states: 'Athena-Brain-8B is trained on the Museum and Supermarket environment families, while external baselines are evaluated zero-shot without environment-specific training.' Because the embodied RL reward is task completion in these exact environments (§3.3.2), the 58.52% overall success and the 51.5-point gain over Athena-SFT are in-distribution measures. They do not establish that the model has acquired a general robot brain. To support the generalization claim, the authors should either evaluate on a genuinely held-out environment family, train the comparison models on the same environment families, or explicitly limit the claim to in-domain capability acquisition. Note also that §3.4 selects merge coefficients using 'overall capability balance across both general and embodied evaluations,' so the
  2. [§4.2.3 / Table 7] ALFWorld is the only out-of-domain control, but the comparison is incomplete. Athena-Brain-8B's 36.57% is below MiniCPM4.1-8B's 40.30%, and Qwen3-8B-Thinking—the strongest reasoning baseline in Table 5—is not reported in Table 7. Single-run results with no variance are reported. Please add the missing baseline, report multiple runs and standard deviations, and specify how the 'unseen split' of ALFWorld is defined. Without this, the transfer evidence does not support the claim that the learned interaction skills generalize.
  3. [§4.1 / Table 4] Most general benchmark results are single-run evaluations; only AIME 2024/2025 are averaged over eight runs. Several headline differences are small (e.g., Athena-Brain vs Qwen3-8B: 60.85 vs 56.70 overall; GPQA-Diamond 59.60 vs 61.62). Without variance estimates or repeated decoding, the claim that Athena-Brain-8B maintains general capabilities comparable to the base model is not statistically supported. Add error bars or multiple seeds, at least for the category averages.
  4. [§4.2.4 / Tables 8–10] The intermediate probes are generated from successful trajectories 'on the test split of our in-domain evaluation environments' and are therefore in-domain as well. While they provide useful diagnostic information about short-horizon goal completion and spatial recovery, they do not break the in-domain circularity. If the intention is to use these probes as evidence of general embodied ability, they must be run in held-out environments.
  5. [§5.3 / Table 12] The best ablation configuration (partial maps, mild reward, format-only-stop) achieves 0.6000 overall, yet the final Athena-Brain-8B in Table 5 scores 58.52. The report does not explain why the final model does not adopt the best configuration found in the ablation. Please clarify whether the final model uses a different configuration, or whether the evaluations differ, so readers can reconcile the two numbers.
minor comments (5)
  1. [Abstract / §1] The phrase 'for embodied intelligence' is duplicated in the abstract ('an 8B LLM designed to serve as an on-device brain for embodied intelligence for embodied intelligence'). Fix the typo.
  2. [Figure 3] The figure contains an unresolved editorial note in the diagram ('RL/SFT, not Athena SFT / pick one'). This should be cleaned up before publication.
  3. [Table 4] The model name is spelled 'Mimo-Embodied-7B' in the table header but 'MiMo-Embodied-7B' elsewhere. Standardize the spelling.
  4. [Table 6] Token Cost is defined as average generated tokens divided by overall score, where overall score is a percentage; this makes the absolute values difficult to interpret. Use tokens per successful episode or clarify the denominator.
  5. [§4.2.1] The evaluation protocol does not state whether Table 5/7 results are averaged over multiple episodes or runs. Reporting the number of evaluation episodes and variance would strengthen the quantitative claims.

Circularity Check

1 steps flagged

Merge coefficients are tuned against the same general/embodied evaluation suites that are later reported as headline results, making the embodied success partly a selection outcome; the general benchmarks and the ALFWorld transfer probe provide independent evidence, so the circularity is partial.

specific steps
  1. fitted input called prediction [Section 3.4 (Model Merging), evaluation in Section 4.2.2 Table 5]
    "the merging coefficients are selected based on overall capability balance across both general and embodied evaluations, rather than optimizing for a specific task or environment."

    The merge weights are the final free parameters of Athena-Brain-8B. Selecting them 'based on ... embodied evaluations' and then reporting Table 5 (58.52% overall) as evidence means the headline number is the value of the selection objective, not an independent validation. The comparison to zero-shot baselines in the same section makes the selected score look like a prediction. The circularity is partial because the general benchmark table and the unseen ALFWorld probe (36.57%) are not part of this coefficient-selection loop for the embodied claim.

full rationale

The paper's pipeline is otherwise a standard post-training report: general SFT/RL, embodied expert RL, and TIES/linear merge are described with training curves and ablations, and the ALFWorld zero-shot result is an out-of-distribution probe. The explicit statement that 'Athena-Brain-8B is trained on the Museum and Supermarket environment families, while external baselines are evaluated zero-shot' is an evaluation-limitation, not a definitional circularity; in-domain evaluation alone does not make the derivation equivalent to its inputs. Self-citations ([9], [34], [50]) are not load-bearing: RynnBrain is a comparison baseline, DeepInsight is an evaluation platform, and the Engagement Process reference is future work. The one substantive circular step is the model-merge coefficient selection: the coefficients are chosen 'based on overall capability balance across both general and embodied evaluations,' and the same evaluation outcomes are then presented as the verified result. This is a fitted-input-called-prediction pattern, but it is not the whole story: the general-language improvement over Qwen3-8B and the ALFWorld transfer result are independent of the merge selection, and the embodied RL gain over Athena-SFT is not purely a merge artifact. I therefore score the circularity as partial (5/10).

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper's claims rest primarily on empirical training choices and on the assumption that text-simulator interaction measures embodied intelligence. No new physical or formal entities are introduced; the main unstated burden is that the embodied evaluation shares distribution with training.

free parameters (5)
  • Merge coefficients (TIES voting threshold/rescale factor; linear interpolation weight)
    Chosen by 'overall capability balance across both general and embodied evaluations' (Section 3.4); values not reported, so the final model is tuned to the evaluation suite.
  • Token-budget reward coefficient in General RL
    Not specified; directly controls the claimed response-length reduction.
  • Embodied RL reward penalties = punitive: -100 truncation, -10 tool error; mild: -0.1 tool error
    Selected after ablation (Section 5.3); final model uses mild reward, format-only-stop, partial maps.
  • Rollout budget in General RL = 16K tokens
    Chosen from {4K, 8K, 16K} by downstream performance (Section 5.2).
  • Embodied supervised warm-up steps = 4K steps
    Reported in Section 3.3.2 without sensitivity analysis.
axioms (5)
  • domain assumption Text-based tool-mediated environments are a valid proxy for real robot-brain interaction
    The paper trains and evaluates in text simulators and states they are a 'scalable proxy' (Section 6), not physical robots.
  • domain assumption Held-out test instances of the same environment family measure generalization beyond training
    Used to claim strong embodied capabilities (Section 4.2.2); if test shares distributional priors with training, the claim is weaker.
  • standard math TIES merging of same-origin task vectors preserves capabilities
    Borrowed from prior work [16]; the paper extends it to a heterogeneous lineage without formal proof (Section 5.1).
  • domain assumption Same-origin RL updates have limited principal-spectrum drift
    Asserted in Section 5.1 based on prior work [37]; no quantitative verification is provided in this paper.
  • domain assumption Response length is a hardware-independent measure of inference efficiency
    Stated in Section 1 to justify token-cost comparisons.

pith-pipeline@v1.3.0-alltime-deepseek · 24503 in / 15646 out tokens · 152616 ms · 2026-08-01T13:47:47.499666+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities in language understanding, reasoning, and world knowledge. As embodied agents become increasingly capable, there is a growing demand for compact models that can serve as an on-device brain, preserving the broad general intelligence of LLMs while enabling effective high-level interaction with embodied environments. Existing approaches, however, often prioritize either general-purpose intelligence or specialized embodied capabilities, making it challenging to satisfy both requirements within a single model. We present \textbf{Athena-Brain-8B}, an 8B LLM designed to serve as an on-device brain for embodied intelligence for embodied intelligence. Through a multi-stage post-training pipeline consisting of General Supervised Fine-Tuning, General Reinforcement Learning, Embodied Expert training, and Model Merge, Athena-Brain-8B maintains strong general capabilities while acquiring strong high-level embodied interaction capabilities and generating concise responses for efficient embodied interaction. Experimental results demonstrate the effectiveness of Athena across both general and embodied evaluations. Compared with the corresponding Qwen3-8B thinking model, Athena-Brain-8B achieves comparable performance on general language and reasoning benchmarks while generating substantially shorter responses. On in-domain embodied benchmarks, Athena-Brain-8B consistently outperforms models of similar scale and surpasses several substantially larger frontier models evaluated zero-shot, demonstrating that compact language models can effectively integrate strong general intelligence with embodied capabilities.

Figures

Figures reproduced from arXiv: 2607.18985 by Jialian Li, Jiaming Song, Jiangpin Liu, Jie Chen, Junhong Liu, Weiran Guo, Xutao Wang, Yi Zhao, Yuchen Cao.

Figure 1
Figure 1. Figure 1: Overall comparison of Athena-Brain-8B with representative open-source models across general [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representative performance of Athena-Brain-8B on selected general language and embodied [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the Athena-Brain-8B post-training pipeline. Starting from an open-weight base [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics during General Reinforcement Learning. The first three subfigures show the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representative interactive scenarios used for embodied expert training. The Museum and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Illustration of the four embodiment levels in the supermarket environment, ranging from [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics during embodied expert training. Both the average interaction reward and [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Average generated tokens of different models across representative benchmark categories. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Two intermediate probes for embodied interaction. State-Goal evaluates near-future goal [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Overview of the lineage-aware merge used to construct Athena-Brain-8B. Same-origin RL [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Effect of the maximum rollout budget (4K, 8K, 16K tokens) during General RL. (a) Train [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Qualitative examples of embodied task solving. Left: the agent uses local visual feedback to [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 16 linked inside Pith

  1. [1]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, ...

    Pith/arXiv arXiv 2025

  2. [2]

    GPT-5.5 system card

    OpenAI. GPT-5.5 system card. https://openai.com/index/gpt-5-5-system-card/ , April

  3. [3]

    Introducing Claude Sonnet 5

    Anthropic. Introducing Claude Sonnet 5. https://www.anthropic.com/news/ claude-sonnet-5, June 2026. Accessed: 2026-07-10

    2026

  4. [4]

    Qwen3.7-Max: The agent frontier

    Qwen Team. Qwen3.7-Max: The agent frontier. https://qwen.ai/blog?id=qwen3.7, May

  5. [5]

    Zhipu AI. GLM-5.2. https://docs.bigmodel.cn/cn/update/new-releases, June 2026. Ac- cessed: 2026-07-10

    2026

  6. [6]

    Accessed: 2026-07-10

    2026

  7. [7]

    Palm-e: an embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: an embodied multimodal language model. InProceedings of the 40th International Conference on Machine Learning, pages 8469–8488, 2023. 25

    2023

  8. [8]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. InConference on Robot Learning, pages 2165–2183. PMLR, 2023

    2023

  9. [9]

    Rynnbrain: Open embodied foundation models.arXiv preprint arXiv:2602.14979, 2026

    Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng, Kehan Li, Xin Li, Jiangpin Liu, Yunxuan Mao, Zhikai Wang, Yuqian Yuan, et al. Rynnbrain: Open embodied foundation models.arXiv preprint arXiv:2602.14979, 2026

    arXiv 2026

  10. [10]

    Mimo-embodied: X-embodied foundation model technical report, 2025

    Xiaomi Embodied Intelligence Team. Mimo-embodied: X-embodied foundation model technical report, 2025. URLhttps://arxiv.org/abs/2511.16518

    Pith/arXiv arXiv 2025

  11. [11]

    Introducing ChatGPT Agent: Bridging research and action

    OpenAI. Introducing ChatGPT Agent: Bridging research and action. https://openai.com/ index/introducing-chatgpt-agent/, July 2025. Accessed: 2026-07-10

    2025

  12. [12]

    Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

    Anthropic. Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku. https:// www.anthropic.com/news/3-5-models-and-computer-use , October 2024. Accessed: 2026- 07-10

    2024

  13. [13]

    Minicpm4: Ultra-efficient llms on end devices.arXiv preprint arXiv:2506.07900, 2025

    Team MiniCPM. Minicpm4: Ultra-efficient llms on end devices.arXiv preprint arXiv:2506.07900, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    UI-TARS: Pioneering automated GUI interaction with native agents, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. UI-TARS: Pioneering automated GUI interaction with native agents, 2025

    2025

  15. [15]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Qwen3-Max: Just scale

    Qwen Team. Qwen3-Max: Just scale. https://qwen.ai/blog?id=qwen3-max, September 2025. Accessed: 2026-07-10

    2025

  17. [17]

    Gemini 3.1 Pro model card

    Google DeepMind. Gemini 3.1 Pro model card. https://deepmind.google/models/ model-cards/gemini-3-1-pro/, February 2026. Accessed: 2026-07-10

    2026

  18. [18]

    Ties-merging: Resolving interference when merging models.arXiv preprint arXiv:2306.01708, 2023

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.arXiv preprint arXiv:2306.01708, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    Meet Kimi K2.6: Advancing open-source coding

    Moonshot AI. Meet Kimi K2.6: Advancing open-source coding. https://forum.moonshot.ai/ t/meet-kimi-k2-6-advancing-open-source-coding/369 , April 2026. Accessed: 2026-07- 10

    2026

  20. [20]

    DeepSeek-V4 Preview: Towards affordable million-token context

    DeepSeek-AI. DeepSeek-V4 Preview: Towards affordable million-token context. https:// api-docs.deepseek.com/news/news260424, April 2026. Accessed: 2026-07-10

    2026

  21. [21]

    Beyondaime: Advancing math reasoning evaluation beyond high school olympiads

    ByteDance-Seed. Beyondaime: Advancing math reasoning evaluation beyond high school olympiads. https://huggingface.co/datasets/ByteDance-Seed/BeyondAIME, 2025

    2025

  22. [22]

    Aime problems and solutions

    Art of Problem Solving. Aime problems and solutions. https://artofproblemsolving.com/ wiki/index.php/AIME_Problems_and_Solutions, 2025. Accessed: 2026-07-09

    2025

  23. [23]

    LiveCodeBench: Holistic and contamination free eval- uation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and contamination free eval- uation of large language models for code. InInternational Conference on Learning Representations, 2025. 26

    2025

  24. [24]

    Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024

    2024

  25. [25]

    From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939, 2024

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline.arXiv preprint arXiv:2406.11939, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Zebralogic: Benchmarking the logical rea- soning ability of language models, 2024

    Bill Yuchen Lin, Ronan Le Bras, and Yejin Choi. Zebralogic: Benchmarking the logical rea- soning ability of language models, 2024. URL https://huggingface.co/spaces/allenai/ ZebraLogic

    2024

  27. [27]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark. InInternational Conference on Machine Learning, 2024

    2024

  29. [29]

    SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines.arXiv preprint arXiv:2502.14739, 2025

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, Dehua Ma, Yuansheng Ni, Haoran Que, Qiyao Wang, Zhoufutu Wen, Siwei Wu, Tianshun Xing, Ming Xu, Zhenzhu Yang, Zekun Mo...

    Pith/arXiv arXiv 2025

  30. [30]

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, and Pasquale Minervini. Are we done with MMLU? InProceedings of the 2025 Conference of the Nations of ...

    2025

  31. [31]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    LiveBench: A challenging, contamination-limited LLM benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. LiveBench: A challenging, contamination-limited LLM benchmark. InInternation...

    2025

  33. [33]

    URL https://arxiv.org/abs/2506

    Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ2-bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv.org/abs/2506. 07982

    2025

  34. [34]

    Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pa...

    2025

  35. [35]

    ALFWorld: Aligning text and embodied environments for interactive learning

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In International Conference on Learning Representations, 2021. URL https://openreview.net/ forum?id=0IOX0YcCdTn

    2021

  36. [36]

    Deepinsight: A unified evaluation infrastructure across the physical ai stack, 2026

    Siyi Li, Chunyu Sun, Jiahao Zhang, Yuchen Kang, Wuliang Wang, Yu Qiu, Rui Jiang, Haitao Cui, and Jie Chen. Deepinsight: A unified evaluation infrastructure across the physical ai stack, 2026. URLhttps://arxiv.org/abs/2606.17574. 27

    Pith/arXiv arXiv 2026

  37. [37]

    The path not taken: Rlvr provably learns off the principals.arXiv preprint arXiv:2511.08567, 2025

    Hanqing Zhu, Zhenyu Zhang, Hanxian Huang, et al. The path not taken: Rlvr provably learns off the principals.arXiv preprint arXiv:2511.08567, 2025

    arXiv 2025

  38. [38]

    To mix or to merge: Toward multi-domain reinforcement learning for large language models.arXiv preprint arXiv:2602.12566, 2026

    Haoqing Wang, Xiang Long, Ziheng Li, Yilong Xu, Tingguang Li, and Yehui Tang. To mix or to merge: Toward multi-domain reinforcement learning for large language models.arXiv preprint arXiv:2602.12566, 2026

    arXiv 2026

  39. [39]

    Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  40. [40]

    Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730– 27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730– 27744, 2022

    2022

  41. [41]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  42. [42]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.Advances in Neural Information Processing Systems, 35:24824–24837, 2022

    2022

  43. [43]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  44. [44]

    Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  46. [46]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Do as i can, not as i say: Grounding language in robotic affordances

    Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. InConference on robot learning, pages 287–318. PMLR, 2023

    2023

  48. [48]

    Openhands: An open platform for ai software developers as generalist agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. InInternational Conference on Learning Representations, volume 2025, pages 65882–65919, 2025

    2025

  49. [49]

    Textworld: A learning environment for text-based games.Computer Games, page 41

    Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. Textworld: A learning environment for text-based games.Computer Games, page 41

  50. [50]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source vision-language-action model. InConference on Robot Learning, pages 2679–2713. PMLR, 2025. 28

    2025

  51. [52]

    Engagement process: Rethinking the temporal interface of action and observation, 2026

    Jialian Li, Yuchen Cao, Junhong Liu, Weiran Guo, Xutao Wang, Jiaming Song, Jiahao Zhang, and Jie Chen. Engagement process: Rethinking the temporal interface of action and observation, 2026. URLhttps://arxiv.org/abs/2605.11484. 29

    Pith/arXiv arXiv 2026