Pith. sign in

REVIEW 3 major objections 6 minor 147 references

7B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancement

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Moxin 7B claims a fully open pipeline—pretraining, post-training, GRPO-based reasoning, and a VLM—matches several leading 7B/8B open models on standard benchmarks.

desk verdict A genuinely open 7B model family with a working GRPO recipe, but the headline base-model numbers are contaminated by training on the eval benchmarks themselves. read the letter →

arxiv 2412.06845 v6 pith:WSEXPNFO submitted 2024-12-08 cs.CL cs.AIcs.LG

Pu Zhao , Xuan Shen , Zhenglun Kong , Yixin Shen , Sung-En Chang , Arash Akbari , Timothy Rupprecht , Lei Lu
show 11 more authors
Enfu Nan Changdi Yang Yumei He Weiyan Shi Xingchen Xu Yu Huang Wei Jiang Wei Wang Yue Chen Yong He Yanzhi Wang
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords largelanguagemodelopen-sourceLLMreinforcementlearningGRPOchain-of-thoughtreasoningvisionopennessframework7B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a 7B model built end-to-end from fully open components—pretraining data, training code, post-training recipes, and reinforcement learning—can compete with widely used 7B/8B open models while releasing everything needed for reproduction and audit. Its evidence is a family of models: a base model averaging 75.44% on five zero-shot tasks, an instruct model averaging 75.92% on the same suite, a reasoning model trained with GRPO that averages 43.2% Pass@1 on four competition-math benchmarks, and a vision-language model on the same backbone that outperforms comparable VLM baselines. If the results hold, they matter because most 'open' releases still withhold training data or code; Moxin claims the highest completeness level of the Model Openness Framework, so every stage of the pipeline can be inspected and rebuilt. The released base model, however, is the capability-enhanced version, produced by partial fine-tuning on the training data of the evaluation benchmarks.

What carries the argument

Two mechanisms carry the argument. The first is the three-phase pretraining schedule whose final 'capability enhancement' phase mixes high-quality data, including training splits of evaluation benchmarks such as MMLU and HellaSwag, into the corpus; this step converts Moxin-7B-Original into Moxin-7B-Enhanced, and the authors release the Enhanced checkpoint as the base model. The second is GRPO (Group Relative Policy Optimization), a reinforcement-learning algorithm that uses an outcome reward model—returning 1 if an answer passes LaTeX/Sympy checks and 0 otherwise—and optimizes a group-relative advantage rather than a learned value model. The reasoning pipeline first SFTs the instruct model on DeepSeek-R1-distilled chain-of-thought data (OpenThoughts and OpenR1-Math-220k), then applies GRPO through the DeepScaleR framework. The VLM uses the Prismatic architecture with DINOv2 and SigLIP visual features fused and Moxin-7B as the language backbone.

What would settle it

Re-run Moxin-7B-Original (the pretrained-only checkpoint) on the same lm-evaluation-harness/opencompass pipeline and compare its zero-shot and few-shot averages—the paper reports 67.21% and 64.74%—against the released base's 75.44% and 70.55%. If the gap is almost fully accounted for by the partial fine-tune on evaluation-benchmark training data, or if retraining any baseline with the same capability-enhancement step closes the gap, the 'superior performance' claim for the base model is falsified. For the reasoning claim, a falsifier is a 7B run that keeps the CoT SFT but omits GRPO; if its competition-math scores stay near 43.2%, the claim that reinforcement learning provides the gain is falsified.

Watch

Extended reading notes

Core claim

The paper's own claim is that a completely open 7B pipeline can be strong, not just transparent. On the zero-shot suite, the released base model reaches 75.44% average accuracy, above all listed 7B/8B baselines including Gemma-7B (73.83%), LLaMA 3.1-8B (73.79%), and Mistral-7B (73.29%). The reasoning model reaches 43.2% average Pass@1 on MATH-500, AMC 2023, Minerva Math, and OlympiadBench, above Qwen2.5-Math-7B-Base (33.55%) and Llama-3.1-70B-Instruct (40.48%), which the authors present as evidence that GRPO-based RL works at 7B scale. The VLM averages 64.68 across seven visual benchmarks, above LLaVA-v1.5-7B and above the same Prismatic framework with Llama-2 or Mistral backbones. An important framing note: the released base model is Moxin-7B-Enhanced, created by partially fine-tuning the pretrained model on the training splits of the evaluation benchmarks; the paper also reports the pretrained-only Moxin-7B-Original, which averages 67.21% on the same zero-shot suite.

Load-bearing premise

The benchmark comparisons rest on the assumption that fine-tuning the base model on the training splits of the evaluation benchmarks does not materially inflate its test scores, so comparisons against baselines that did not receive that fine-tuning remain fair.

Editorial extensions

If this is right

  • If the results hold, a fully open 7B pipeline can match or beat several popular 7B/8B open models on standard zero-shot and few-shot benchmarks, so openness need not cost competitive performance.
  • The GRPO result would extend evidence for outcome-reward reinforcement learning from 1.5B and 67B scales to a 7B model, strengthening the case that RL is a viable reasoning booster for small LLMs.
  • The released checkpoint names mean that users of 'Moxin-7B-Base' are actually getting the capability-enhanced model, not the raw pretrained weights.
  • The VLM result would imply that the Moxin-7B backbone transfers well to multimodal tasks in the Prismatic-style architecture, outperforming Llama-2 and Mistral backbones trained identically.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: The paper's own tables show the pretrained-only Moxin-7B-Original averaging 67.21% zero-shot and 64.74% few-shot, below most listed baselines; readers should attribute the 'superior' base-model claim to the released enhanced checkpoint, not to the raw pretrained weights.
  • Editorial: The reasoning pipeline does not ablate the GRPO stage from the preceding CoT SFT, so the marginal contribution of reinforcement learning itself is not isolated; a no-RL control run would settle whether the 43.2% average comes from the reward optimization or from the distilled reasoning traces.
  • Editorial: The VLM comparison is confined to one architecture family and one evaluation suite; testing the released VLM on broader multimodal benchmarks would show whether the advantage generalizes.
  • Editorial: Because the paper releases intermediate checkpoints, an independent audit could rerun the capability-enhancement phase and the GRPO run to check whether the reported scores reproduce within noise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Moxin 7B, a fully open-source 7B language model that releases pretraining code, training data, intermediate checkpoints, and final weights, together with Instruct, Reasoning, and VLM variants. The authors report zero-shot and few-shot base-model comparisons, post-trained instruct-model evaluations, GRPO-based chain-of-thought reinforcement learning results on math benchmarks, and VLM evaluations. The central claims are that the base model achieves superior or competitive performance against stronger 7B/8B baselines, the reasoning model demonstrates the effectiveness of RL for small LLMs, and the VLM outperforms comparable open VLM baselines.

Significance. If the empirical results were clean, the paper would be a valuable demonstration that a fully open 7B model can be competitive with widely used open-weight models and that GRPO-based RL can transfer from smaller and larger scales to 7B. The release of code, data, checkpoints, and configurations, including the reported $160,000 pretraining cost, is a genuine contribution to reproducibility and aligns with the Model Openness Framework. However, the base-model evaluation is compromised because both reported base models were trained on the training splits of the evaluation benchmarks themselves, so the headline 'superior performance' claim for the base model is not supported by the evidence as presented.

major comments (3)
  1. [§7.2, Tables 2 and 3; §3.3.3; Table 9] The base-model comparisons in Tables 2 and 3 cannot support the claimed 'superior performance' because both Moxin-7B-Original and Moxin-7B-Enhanced were trained on the training splits of the evaluation benchmarks. Section 3.3.3 states that the Capability Enhancement phase uses 'the training datasets of various evaluation benchmarks such as MMLU and HellaSwag,' so even Moxin-7B-Original has already seen these benchmark training examples before the additional fine-tuning described in Section 7.2. Section 7.2 then defines Moxin-7B-Enhanced as the result of 'subsequent partial fine-tuning of Moxin-7B-Original on the training data of the evaluation datasets' and Table 9 confirms that Moxin-7B-Enhanced is released as Moxin-7B-Base. The baseline models in Tables 2 and 3 were not trained on those benchmark training splits. Consequently, the reported gains (e.g., 75.44 vs. the 73.83 best baseline in Table 2, and 70.55 vs. 71.57 in Table 3) may reflect benchmark familiarity or memorization rather than general capability. The manuscript provides no decontamination analysis, no per-task overlap statistics, and no error bars, so the magnitude of any inflation cannot be bounded.
  2. [§7.2 (definition of Moxin-7B-Original) vs. §3.3.3] There is an internal inconsistency in the definition of the models being compared. Section 7.2 calls Moxin-7B-Original 'the foundation model before fine-tuning on the training data of the evaluation datasets,' but Section 3.3.3 describes training on those same benchmark training datasets as part of the pretraining Capability Enhancement phase. The paper should clarify the exact chronology and composition of the training data, and it should either report evaluations for a truly clean foundation model or explicitly state that all reported base-model variants have seen evaluation-benchmark training data, in which case the comparisons against clean baselines must be reframed.
  3. [Tables 4, 5, 7, and 8] All performance claims are based on single-point accuracy numbers with no error bars, confidence intervals, or significance tests. Several margins are small, for example Table 8 reports Ours at 64.68 vs. Mistral v0.1 7B at 62.83, and Table 5 reports Moxin-7B-DPO at 71.66 vs. Mistral 8B Instruct at 71.62. Because the 'outperforms' claims in the abstract and introduction depend on these differences, the paper should report variance estimates from multiple evaluation runs or otherwise justify that the evaluation protocol is deterministic and the differences are reliable.
minor comments (6)
  1. [§7.2.2] The text says 'Table 3 presents our zero-shot evaluation results,' but Table 3 is in the Few-Shot Evaluation section and uses few-shot settings; the sentence should say 'few-shot.'
  2. [§7.3] The phrase 'few show evaluation' should be corrected to 'few-shot evaluation.'
  3. [§5 and Table 7] The RL framework name is spelled inconsistently as 'AReal' in the text and 'AReaL' in the caption of Table 7; the authors should pick one spelling and use it consistently.
  4. [Table 5] The average for Moxin-7B-II is reported as 71.095 with three decimal places while all other entries have two; this formatting inconsistency should be fixed.
  5. [§7.1, MMLU bullet] The MMLU benchmark is cited as reference [141], which is an AI Sandbagging paper; this appears to be an incorrect reference, and the intended MMLU citation is likely [107].
  6. [Table 8] The claim that 'our model outperforms all other VLM baselines' is based on the average score, but Ours is lower than several baselines on individual tasks (e.g., VizWiz: 54.08 vs. 56.39 for Llama-2 Chat 7B); the claim should be qualified to 'on average' or a per-task breakdown should be discussed.

Circularity Check

2 steps flagged · score 7.0 of 10

Base-model benchmark superiority is a fitted-input evaluation: the released Moxin-7B-Base is Moxin-7B-Enhanced, fine-tuned on the training data of the same benchmarks reported in Tables 2 and 3, so the headline zero-/few-shot gains are not independent predictions.

  1. fitted input called prediction [Section 3.3.3 'Capability Enhancement', pre-training data]
    "In our study, we collect open-source data from HuggingFace, primarily utilizing the training datasets of various evaluation benchmarks such as MMLU [107] and HellaSwag [108]. These data are used experimentally to investigate the relationship between high-quality, capability-focused training data and model performance."

    The pre-training 'Capability Enhancement' phase feeds the training splits of MMLU and HellaSwag into the model, and both benchmarks are subsequently reported in the few-shot and zero-shot evaluation tables. Consequently, even Moxin-7B-Original has already seen the training distribution of those evaluation tasks. Its Table 2 and Table 3 scores therefore partly reflect memorization or distribution matching of the evaluation-benchmark training data, not an independent measurement of capability. The comparisons against baselines (Mistral, LLaMA, Gemma, Qwen, etc.) that were not trained on those exact benchmark training splits are not controlled, and no decontamination analysis or error bars are provided.

  2. fitted input called prediction [Section 7.2.1, paragraph before Table 2; Section 7.2 first paragraph; Table 9]
    "We name the initial model as Moxin-7B-Original, which presents the foundation model before fine-tuning on the training data of the evaluation datasets. After subsequent partial fine-tuning of Moxin-7B-Original on the training data of the evaluation datasets, we developed Moxin-7B-Enhanced... We release our Moxin-7B-Enhanced model as Moxin-7B-Base model. ... After training with the training data of evaluation tasks, our Moxin-7B-Enhanced can achieve superior performance compared with state-of-the-art (SOTA) baselines."

    The released Moxin-7B-Base is identical to Moxin-7B-Enhanced (Table 9 maps Moxin-7B-Enhanced to Moxin-7B-LLM). The zero-shot and few-shot tables report scores of Moxin-7B-Enhanced on ARC, HellaSwag, PIQA, Winogrande, and MMLU after the paper explicitly says it was partially fine-tuned on the training data of those evaluation datasets. The claim 'superior performance compared with SOTA baselines' is therefore a fitted-input statement: the model was trained on the evaluation benchmarks' training splits and then evaluated on the same benchmarks, while the comparison baselines did not receive that benchmark training. No decontamination bound is provided, so the reported 75.44 average in Table 2 and the few-shot scores cannot be interpreted as evidence of independent base-model generalization.

full rationale

The paper's central base-model claim is not self-contained against external benchmarks because of its own disclosed training protocol. Section 3.3.3 states that the final pre-training phase uses 'the training datasets of various evaluation benchmarks such as MMLU and HellaSwag', and Section 7.2 states that Moxin-7B-Enhanced was produced by 'partial fine-tuning of Moxin-7B-Original on the training data of the evaluation datasets' and is released as Moxin-7B-Base. The zero-shot and few-shot evaluations in Tables 2 and 3 then report scores on exactly those benchmarks, with Moxin-7B-Enhanced achieving the headline 'superior performance compared with state-of-the-art (SOTA) baselines'. This is a textbook fitted-input-called-prediction pattern: the evaluated quantity is constructed from the evaluation's training distribution, and the comparison baselines were not trained on those splits. The GRPO/reasoning claim in Table 7 is less directly affected because MATH-500, AMC, Minerva, and OlympiadBench are not listed as capability-enhancement or fine-tuning data, although the reasoning model inherits the tainted base foundation. The VLM comparison is also a separate, externally benchmarked claim. I therefore assign a score of 7 rather than 8 or 10: the central base-model superiority claim is substantially undermined by construction, but meaningful independent content remains in the reasoning and VLM results, and the paper is transparent about the training protocol.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claims rest on standard deep learning assumptions (that training on large public corpora produces a capable base model, and that RL on competition math transfers to held-out math benchmarks) plus one paper-specific assumption: that fine-tuning on evaluation-benchmark training data does not invalidate the reported base-model scores. No new physical or mathematical entities are introduced.

free parameters (4)
  • Model depth (n_layers) = 36 (vs 32 in Mistral)
    Chosen via empirical ablation and grid search (Section 3.1); directly affects the reported performance.
  • Pretraining learning rate = 2e-6
    Set via tuning, with cosine decay to 10% of max; load-bearing for stable 2T-token training (Section 3.4).
  • SFT and DPO learning rates = 5e-6 for SFT (2 epochs), 5e-7 for DPO (1 epoch)
    Adopted from Tülu 3 defaults; affects instruction-following quality (Section 4).
  • Use of evaluation-benchmark training data = ARC, HellaSwag, PIQA, Winogrande, and MMLU train splits
    Moxin-7B-Enhanced is fine-tuned on these splits and released as the base model (Section 7.2); this ad hoc choice inflates the reported zero-shot and few-shot numbers and is central to the 'superior performance' claim.
assumptions (3)
  • ad hoc to paper Fine-tuning on the training splits of evaluation benchmarks does not materially inflate the reported test scores, so comparisons against baselines without such fine-tuning remain valid.
    Required for the zero-shot and few-shot comparisons in Tables 2 and 3 to be meaningful; Section 7.2 discloses the fine-tuning yet the paper still presents the numbers as a basis for 'superior performance.'
  • domain assumption RL training on 40k competition math problems with an outcome reward model generalizes to held-out benchmarks (MATH-500, AMC 2023, Minerva Math, OlympiadBench) without reward hacking.
    The GRPO training (Section 5) optimizes LaTeX and Sympy answer correctness on DeepScaleR's competition-problem distribution; the assumption that this transfers to the evaluation benchmarks is load-bearing for Table 7.
  • domain assumption The Tülu 3 SFT/DPO recipes and Prismatic VLM framework are correctly applied and their default hyperparameters are suitable for a 7B model.
    The paper adopts defaults from Tülu 3 (Section 4) and Prismatic (Section 6) without re-tuning; incorrect integration would skew reported instruct and VLM results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 7B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancement." pith.science (2026). https://pith.science/paper/WSEXPNFO

@misc{pith2026241206845,
  author       = {Pith},
  title        = {Pith review of: 7B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSEXPNFO}},
  note         = {Machine review of arXiv:2412.06845}
}
read the original abstract

Recently, Large Language Models (LLMs) have undergone a significant transformation, marked by a rapid rise in both their popularity and capabilities. Leading this evolution are proprietary LLMs like GPT-4 and GPT-o1, which have captured widespread attention in the AI community due to their remarkable performance and versatility. Simultaneously, open-source LLMs, such as LLaMA, have made great contributions to the ever-increasing popularity of LLMs due to the ease to customize and deploy the models across diverse applications. Although open-source LLMs present unprecedented opportunities for innovation and research, the commercialization of LLMs has raised concerns about transparency, reproducibility, and safety. Many open-source LLMs fail to meet fundamental transparency requirements by withholding essential components like training code and data, which may hinder further innovations on LLMs. To mitigate this issue, we introduce Moxin 7B, a fully open-source LLM developed, adhering to principles of open science, open source, open data, and open access. We release the pre-training code and configurations, training and fine-tuning datasets, and intermediate and final checkpoints, aiming to make continuous commitments to fully open-source LLMs. After pre-training the base model, we finetune the Moxin Base model with SOTA post-training framework and instruction data to obtain Moxin Instruct model. To improve the reasoning capability, we further finetune our Instruct model with chain-of-thought data distilled from DeepSeek R1, and then use Group Relative Policy Optimization (GRPO) following DeepSeek R1 to finetune our model, leading to the Moxin Reasoning model. Moreover, we develop our vision language model based on our Moxin model. Experiments show that our models achieve superior performance in various evaluations such as zero-shot evaluation, few-shot evaluation, and CoT evaluation.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

147 extracted references · 4 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku. https: //www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf

  3. [3]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  4. [4]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  5. [5]

    Thomas Prest, Pierre-Alain Fouque, Jeffrey Hoffstein, Paul Kirchner, Vadim Lyubashevsky, Thomas Pornin, Thomas Ricosset, Gregor Seiler, William Whyte, and Zhenfei Zhang. Falcon. Post-Quantum Cryptography Project of NIST, 2020

    2020

  6. [6]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  7. [7]

    The foundation model transparency index

    Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. The foundation model transparency index. arXiv preprint arXiv:2310.12941, 2023

    arXiv 2023

  8. [8]

    On the societal impact of open foundation models

    Sayash Kapoor, Rishi Bommasani, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Peter Cihon, Aspen Hopkins, Kevin Bankston, Stella Biderman, Miranda Bogen, et al. On the societal impact of open foundation models. arXiv preprint arXiv:2403.07918, 2024

    arXiv 2024

Show all 147 references
  1. [9]

    The model openness framework: Promoting completeness and openness for reproducibility, transparency and usability in ai

    Matt White, Ibrahim Haddad, Cailean Osborne, Ahmed Abdelmonsef, Sachin Varghese, et al. The model openness framework: Promoting completeness and openness for reproducibility, transparency and usability in ai. arXiv preprint arXiv:2403.13784, 2024

    2024 arXiv

  2. [10]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  3. [11]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  4. [12]

    Fedkd-hybrid: Federated hybrid knowledge distillation for lithography hotspot detection

    Yuqi Li, Xingyou Lin, Kai Zhang, Chuanguang Yang, Zhongliang Guo, Jianping Gou, and Yanli Li. Fedkd-hybrid: Federated hybrid knowledge distillation for lithography hotspot detection. arXiv preprint arXiv:2501.04066, 2025

    2025 arXiv

  5. [13]

    Advancing model pruning via bi-level optimization

    Yihua Zhang, Yuguang Yao, Parikshit Ram, et al. Advancing model pruning via bi-level optimization. NeurIPS, 2022

    2022

  6. [14]

    A generic layer pruning method for signal modulation recognition deep learning models

    Yao Lu, Yutao Zhu, Yuqi Li, Dongwei Xu, Yun Lin, Qi Xuan, and Xiaoniu Yang. A generic layer pruning method for signal modulation recognition deep learning models. IEEE TCCN, 2024

    2024

  7. [15]

    Cross-layer graph knowledge distillation for image recognition

    Jiaming Chu, Yanzhuo Xiang, Yuqi Li, Chuanguang Yang, Zhulin An, and Yongjun Xu. Cross-layer graph knowledge distillation for image recognition. In ICASSP, 2025

    2025

  8. [16]

    Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training

    Zhenglun Kong, Haoyu Ma, Geng Yuan, Mengshu Sun, Yanyue Xie, Peiyan Dong, Xin Meng, Xuan Shen, Hao Tang, Minghai Qin, et al. Peeling the onion: Hierarchical reduction of data redundancy for efficient vision transformer training. In Proceedings of the AAAI Conference on Artific...

    2023

  9. [17]

    Pruning foundation models for high accuracy without retraining

    Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, and Xue Lin. Pruning foundation models for high accuracy without retraining. In Findings of EMNLP 2024, pages 9681–9694. ACL, November 2024. 18

    2024

  10. [18]

    Sparse learning for state space models on mobile

    Xuan Shen, Hangyu Zheng, Yifan Gong, Zhenglun Kong, Changdi Yang, Zheng Zhan, Yushu Wu, Xue Lin, Yanzhi Wang, Pu Zhao, and Wei Niu. Sparse learning for state space models on mobile. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  11. [19]

    Numerical pruning for efficient autoregressive models

    Xuan Shen, Zhao Song, Yufa Zhou, Bo Chen, Jing Liu, Ruiyi Zhang, Ryan A Rossi, Hao Tan, Tong Yu, Xiang Chen, et al. Numerical pruning for efficient autoregressive models. In AAAI, 2025

    2025

  12. [20]

    Lazydit: Lazy learning for the acceleration of diffusion transformers

    Xuan Shen, Zhao Song, Yufa Zhou, Bo Chen, Yanyu Li, Yifan Gong, Kai Zhang, Hao Tan, Jason Kuen, Henghui Ding, et al. Lazydit: Lazy learning for the acceleration of diffusion transformers. In AAAI, 2025

    2025

  13. [21]

    Toward adaptive large language models structured pruning via hybrid-grained weight importance assessment

    Jun Liu, Zhenglun Kong, Pu Zhao, Changdi Yang, Hao Tang, Xuan Shen, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, Dong Huang, and Yanzhi Wang. Toward adaptive large language models structured pruning via hybrid-grained weight importance assessment. In AAAI, 2025

    2025

  14. [22]

    Search for efficient large language models

    Xuan Shen, Pu Zhao, Yifan Gong, Zhenglun Kong, Zheng Zhan, Yushu Wu, Ming Lin, Chao Wu, Xue Lin, and Yanzhi Wang. Search for efficient large language models. In NeurIPS, 2024

    2024

  15. [23]

    Pruning parameterization with bi-level optimization for efficient semantic segmentation on the edge

    Changdi Yang, Pu Zhao, Yanyu Li, et al. Pruning parameterization with bi-level optimization for efficient semantic segmentation on the edge. In CVPR, 2023

    2023

  16. [24]

    Comae: Comprehensive attribute exploration for zero-shot hashing

    Yuqi Li, Qingqing Long, Yihang Zhou, Ning Cao, Shuai Liu, Fang Zheng, Zhihong Zhu, Zhiyuan Ning, Meng Xiao, Xuezhi Wang, et al. Comae: Comprehensive attribute exploration for zero-shot hashing. arXiv preprint arXiv:2402.16424, 2024

    2024 arXiv

  17. [25]

    Quartdepth: Post-training quantization for real-time depth estimation on the edge

    Xuan Shen, Weize Ma, Jing Liu, et al. Quartdepth: Post-training quantization for real-time depth estimation on the edge. In CVPR, 2025

    2025

  18. [26]

    Fast and memory-efficient video diffusion using streamlined inference

    Zheng Zhan, Yushu Wu, Yifan Gong, et al. Fast and memory-efficient video diffusion using streamlined inference. In NeurIPS, 2024

    2024

  19. [27]

    Compiler-aware neural architecture search for on- mobile real-time super-resolution

    Yushu Wu, Yifan Gong, Pu Zhao, et al. Compiler-aware neural architecture search for on- mobile real-time super-resolution. In ECCV, pages 92–111. Springer, 2022

    2022

  20. [28]

    Achieving on-mobile real-time super- resolution with neural architecture and pruning search

    Zheng Zhan, Yifan Gong, Pu Zhao, Geng Yuan, et al. Achieving on-mobile real-time super- resolution with neural architecture and pruning search. In ICCV, pages 4821–4831, 2021

    2021

  21. [29]

    Towards real-time segmentation on the edge

    Yanyu Li, Changdi Yang, Pu Zhao, et al. Towards real-time segmentation on the edge. AAAI’23/IAAI’23/EAAI’23, 2023

    2023

  22. [30]

    Pruning-as-search: Efficient neural architecture search via channel pruning and structural reparameterization

    Yanyu Li, Pu Zhao, Geng Yuan, Xue Lin, Yanzhi Wang, and Xin Chen. Pruning-as-search: Efficient neural architecture search via channel pruning and structural reparameterization. arXiv preprint arXiv:2206.01198, 2022

    2022 arXiv

  23. [31]

    Exploring token pruning in vision state space models

    Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zichong Meng, Hangyu Zheng, Xuan Shen, Stratis Ioannidis, Wei Niu, Pu Zhao, and Yanzhi Wang. Exploring token pruning in vision state space models. In NeurIPS, 2024

    2024

  24. [32]

    Rethinking token reduction for state space models

    Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang, Yifan Gong, Xuan Shen, Xue Lin, Pu Zhao, and Yanzhi Wang. Rethinking token reduction for state space models. In EMNLP, pages 1686–1697, Miami, Florida, USA, nov 2024. ACL

    2024

  25. [33]

    Spvit: Enabling faster vision transformers via latency-aware soft token pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In European conference on computer vision, pages 620–640. Springer, 2022

    2022

  26. [34]

    Efficient reasoning with hidden thinking

    Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, and Jiuxiang Gu. Efficient reasoning with hidden thinking. arXiv preprint arXiv:2501.19201, 2025

    2025 arXiv

  27. [35]

    Enabling flexible multi-llm integration for scalable knowledge aggregation

    Zhenglun Kong, Zheng Zhan, Shiyue Hou, Yifan Gong, Xin Meng, Pengwei Sui, Peiyan Dong, Xuan Shen, Zifeng Wang, Pu Zhao, et al. Enabling flexible multi-llm integration for scalable knowledge aggregation. arXiv preprint arXiv:2505.23844, 2025

    2025 arXiv

  28. [36]

    Fastcar: Cache attentive replay for fast auto-regressive video generation on the edge

    Xuan Shen, Weize Ma, Yufa Zhou, Enhao Tang, Yanyue Xie, Zhengang Li, Yifan Gong, Quanyi Wang, Henghui Ding, Yiwei Wang, et al. Fastcar: Cache attentive replay for fast auto-regressive video generation on the edge. arXiv preprint arXiv:2505.14709, 2025

    2025 arXiv

  29. [37]

    Taming diffusion for dataset distillation with high representativeness

    Lin Zhao, Yushu Wu, Xinru Jiang, Jianyang Gu, Yanzhi Wang, Xiaolin Xu, Pu Zhao, and Xue Lin. Taming diffusion for dataset distillation with high representativeness. arXiv preprint arXiv:2505.18399, 2025. 19

    2025 arXiv

  30. [38]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  31. [39]

    Jamba-1.5: Hybrid transformer- mamba models at scale

    Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, et al. Jamba-1.5: Hybrid transformer- mamba models at scale. arXiv preprint arXiv:2408.12570, 2024

    2024 arXiv

  32. [40]

    Neural machine translation of rare words with subword units

    Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015

    2015 arXiv

  33. [41]

    tiktoken, 2022

    OpenAI Team. tiktoken, 2022

    2022

  34. [42]

    Sentencepiece: A simple and language independent subword tokenizer and detok- enizer for neural text processing

    T Kudo. Sentencepiece: A simple and language independent subword tokenizer and detok- enizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018

    2018 arXiv

  35. [43]

    Xlnet: Generalized autoregressive pretraining for language understanding

    Zhilin Yang. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019

    1906 arXiv

  36. [44]

    Summary of the tokenizers

    Hugging Face Team. Summary of the tokenizers. 2024

    2024

  37. [45]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv pr...

    2024 arXiv

  38. [46]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  39. [47]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  40. [48]

    Mm-llms: Recent advances in multimodal large language models

    Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601, 2024

    2024 arXiv

  41. [49]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  42. [50]

    Language models are few-shot learners

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sas- try, A Askell, S Agarwal, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 1, 2020

    2005 arXiv

  43. [51]

    The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cap- pelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv ...

    2023 arXiv

  44. [52]

    Scaling language models: Methods, analysis & insights from training gopher

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021

    2021 arXiv

  45. [53]

    Ccnet: Extracting high quality monolingual datasets from web crawl data

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019

    1911 arXiv

  46. [54]

    mt5: A massively multilingual pre-trained text-to-text transformer

    L Xue. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020

    2010 arXiv

  47. [55]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  48. [56]

    Cross-lingual language model pretraining

    Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Advances in neural information processing systems, 32, 2019

    2019

  49. [57]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. 20

    2021 arXiv

  50. [58]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  51. [59]

    How to train data-efficient llms

    Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient llms. arXiv preprint arXiv:2402.09668, 2024

    2024 arXiv

  52. [60]

    A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity

    Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv...

    2023 arXiv

  53. [61]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pages 5547–5569. PMLR, 2022

    2022

  54. [62]

    Deduplicating training data makes language models better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021

    2021 arXiv

  55. [63]

    Url normal- ization for de-duplication of web pages

    Amit Agarwal, Hema Swetha Koppula, Krishna P Leela, Krishna Prasad Chitrapura, Sachin Garg, Pavan Kumar GM, Chittaranjan Haty, Anirban Roy, and Amit Sasturkar. Url normal- ization for de-duplication of web pages. In Proceedings of the 18th ACM conference on information and kno...

    1987

  56. [64]

    Datacomp-lm: In search of the next generation of training sets for language models

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.11794, 2024

    2024 arXiv

  57. [65]

    Efficient online data mixing for language model pre-training

    Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. In R0-FoMo: Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023

    2023

  58. [66]

    Slimpajama-dc: Under- standing data combinations for llm training

    Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Zhengzhong Liu, Hongyi Wang, Bowen Tan, Joel Hestness, Natalia Vassilieva, Daria Soboleva, et al. Slimpajama-dc: Under- standing data combinations for llm training. arXiv preprint arXiv:2309.10818, 2023

    2023 arXiv

  59. [67]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  60. [68]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1– 113, 2023

    2023

  61. [69]

    Dolma: An open corpus of three trillion tokens for language model pretraining research

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024

    2024 arXiv

  62. [70]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlíˇcek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024

    2024 arXiv

  63. [71]

    Llm-datasets: An open framework for pretraining datasets of large language models

    Malte Ostendorff, Pedro Ortiz Suarez, Lucas Fonseca Lage, and Georg Rehm. Llm-datasets: An open framework for pretraining datasets of large language models

  64. [72]

    Starcoder 2 and the stack v2: The next generation

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  65. [73]

    T \" ulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T \" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. 21

    2024 arXiv

  66. [74]

    Infinity Instruct

    BAAI Team. Infinity Instruct. https://huggingface.co/datasets/BAAI/Infinity-Instruct, January 2025

    2025

  67. [75]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  68. [76]

    Open Thoughts

    OpenThoughts Team. Open Thoughts. https://open-thoughts.ai, January 2025

    2025

  69. [77]

    OpenR1-Math-220k

    OpenR1 Team. OpenR1-Math-220k. https://huggingface.co/datasets/open-r1/OpenR1-Math- 220k, January 2025

    2025

  70. [78]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y . Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. https://pretty-radio-b75 .notion.site/De...

    2025

  71. [79]

    Areal: Ant reasoning rl.https://github.com/inclusionAI/AReaL, 2025

    Ant Research RL Lab. Areal: Ant reasoning rl.https://github.com/inclusionAI/AReaL, 2025

    2025

  72. [80]

    Realhf: Optimized rlhf training for large language models through parameter reallocation

    Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. Realhf: Optimized rlhf training for large language models through parameter reallocation. arXiv preprint arXiv:2406.14088, 2024

    2024 arXiv

  73. [81]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, et al. Gemini: A family of highly capable multimodal models, 2024

    2024

  74. [82]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  75. [83]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  76. [84]

    Baichuan 2: Open large-scale language models

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023

    2023 arXiv

  77. [85]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    2024 arXiv

  78. [86]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In Intern...

    2023

  79. [87]

    Gpt-neox-20b: An open-source autoregressive language model

    Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022

    2022 arXiv

  80. [88]

    Olmo: Accelerating the science of language models

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838, 2024

    2024 arXiv

  81. [89]

    Llm360: Towards fully transparent open-source llms

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source llms. arXiv preprint arXiv:2312.06550, 2023

    2023 arXiv

  82. [90]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  83. [91]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245, 2023. 22

    2023 arXiv

  84. [92]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    2004 arXiv

  85. [93]

    SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://cerebras.ai/blog/slimpajama-a-627b-token-cleaned-and- deduplicated-version-of-redpajama , 2023

    2023

  86. [94]

    Redpajama: an open dataset for training large language models

    Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, et al. Redpajama: an open dataset for training large language models. arXiv preprint arXiv:2411.12372, 2024

    2024 arXiv

  87. [95]

    Semd- edup: Data-efficient learning at web-scale through semantic deduplication

    Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semd- edup: Data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540, 2023

    2023 arXiv

  88. [96]

    Large-scale near-deduplication behind bigcode, 2023

    2023

  89. [97]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019

    1904 arXiv

  90. [98]

    Mining of massive datasets, cambridge university press, cambridge, 2014

    J Leskovec, A Rajaraman, and JD Ullman. Mining of massive datasets, cambridge university press, cambridge, 2014

    2014

  91. [99]

    Introduction to common crawl datasets

    Jay M Patel and Jay M Patel. Introduction to common crawl datasets. Getting structured data from the internet: running web crawlers/scrapers on a big data production scale, pages 277–324, 2020

    2020

  92. [100]

    On the resemblance and containment of documents

    Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21–29. IEEE, 1997

    1997

  93. [102]

    Color-filter: Conditional loss reduction filtering for targeted language model pre- training

    David Brandfonbrener, Hanlin Zhang, Andreas Kirsch, Jonathan Richard Schwarz, and Sham Kakade. Color-filter: Conditional loss reduction filtering for targeted language model pre- training. arXiv preprint arXiv:2406.10670, 2024

    2024 arXiv

  94. [103]

    Data filtering networks

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425, 2023

    2023 arXiv

  95. [104]

    The stack: 3 tb of permissively licensed source code

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533, 2022

    2022 arXiv

  96. [105]

    GLM-130b: An open bilingual pre-trained model

    Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model...

    2023

  97. [106]

    Nemotron-4 15b technical report

    Jupinder Parmar, Shrimai Prabhumoye, Joseph Jennings, Mostofa Patwary, Sandeep Subrama- nian, Dan Su, Chen Zhu, Deepak Narayanan, Aastha Jhunjhunwala, Ayush Dattagupta, et al. Nemotron-4 15b technical report. arXiv preprint arXiv:2402.16819, 2024

    2024 arXiv

  98. [107]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021

    2021

  99. [108]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  100. [109]

    Colossal-ai: A unified deep learning system for large-scale parallel training

    Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. Colossal-ai: A unified deep learning system for large-scale parallel training. In Proceedings of the 52nd International Conference on Parallel Processing, pages 766–775, 2023

    2023

  101. [110]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  102. [111]

    Smith, Yejin Choi, and Hannaneh Hajishirzi

    Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. The Art of Saying No: Contextual Noncomp...

    2024

  103. [112]

    Finetuned language models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations

  104. [113]

    Wildchat: 1m chatGPT interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024

    2024

  105. [114]

    Openmathinstruct-2: Accelerating ai for math with massive open-source instruc- tion data

    Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruc- tion data. arXiv preprint arXiv:2410.01560, 2024

    arXiv

  106. [115]

    Numinamath

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https: //huggingface.co/AI-MO/NuminaMat...

    2024

  107. [116]

    Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024

    2024

  108. [117]

    Sciriff: A resource to enhance language model instruction-following over scientific literature

    David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, et al. Sciriff: A resource to enhance language model instruction-following over scientific literature. arXiv preprint arXiv:2406.07835, 2024

    2024

  109. [118]

    Table-gpt: Table-tuned gpt for diverse table tasks

    Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, and Surajit Chaudhuri. Table-gpt: Table-tuned gpt for diverse table tasks. arXiv preprint arXiv:2310.09263, 2023

    2023 arXiv

  110. [119]

    Hybridflow: A flexible and efficient rlhf framework

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv:2409.19256, 2024

    2024 arXiv

  111. [120]

    Simple Reinforcement Learning for Reasoning

    hkust-nlp Team. Simple Reinforcement Learning for Reasoning. https://github.com/hkust- nlp/simpleRL-reason, January 2025

    2025

  112. [121]

    7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient

    Weihao Zeng, Yuzhen Huang, Wei Liu, Keqing He, Qian Liu, Zejun Ma, and Junxian He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. https://hkust-nlp.notion.site/simplerl-reason, 2025. Notion Blog

    2025

  113. [122]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    J Bai, S Bai, S Yang, S Wang, S Tan, P Wang, J Lin, C Zhou, and J Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arxiv 2023. arXiv preprint arXiv:2308.12966

    2023 arXiv

  114. [123]

    Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition.arXiv preprint arXiv:2309.15112, 2023

    Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. Internlm-xcomposer: A vision- language large model for advanced text-image comprehension and composition.arXiv preprint arXiv:2309.15112, 2023

    2023 arXiv

  115. [124]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  116. [125]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  117. [126]

    Prismatic vlms: Investigating the design space of visually-conditioned language models

    Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. In International Conference on Machine Learning (ICML), 2024

    2024

  118. [127]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  119. [128]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 24

    2023

  120. [129]

    Lerf: Language embedded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19729–19739, 2023

    2023

  121. [130]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  122. [131]

    Pali-3 vision language models: Smaller, faster, stronger

    Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul V oigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023

    arXiv

  123. [132]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2018

  124. [133]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2024

    2024

  125. [134]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  126. [135]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. InEuropean Conference on Computer Vision, pages 370–387. Springer, 2024

    2024

  127. [136]

    Lm evaluation harness, 2024

    LM Evaluation Harness Team. Lm evaluation harness, 2024. Accessed: Summer 2024

    2024

  128. [137]

    Open compass, 2024

    Open Compass Team. Open compass, 2024. Accessed: Summer 2024

    2024

  129. [138]

    Autogptq: An user-friendly llms quantization package, 2024

    AutoGPTQ Team. Autogptq: An user-friendly llms quantization package, 2024. Accessed: Spring 2024

    2024

  130. [139]

    Deep- speed4science initiative: Enabling large-scale scientific discovery through sophisticated ai system technologies

    Shuaiwen Leon Song, Bonnie Kruft, Minjia Zhang, Conglong Li, Shiyang Chen, Chengming Zhang, Masahiro Tanaka, Xiaoxia Wu, Jeff Rasley, Ammar Ahmad Awan, et al. Deep- speed4science initiative: Enabling large-scale scientific discovery through sophisticated ai system technologies...

    arXiv

  131. [140]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    2018 arXiv

  132. [141]

    Ai sandbagging: Language models can strategically underperform on evaluations

    Teun van der Weij, Felix Hofstätter, Ollie Jaffe, Samuel F Brown, and Francis Rhys Ward. Ai sandbagging: Language models can strategically underperform on evaluations. arXiv preprint arXiv:2406.07358, 2024

    2024 arXiv

  133. [142]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

    2021

  134. [143]

    Piqa: Reasoning about physical commonsense in natural language, 2019

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019

    2019

  135. [144]

    Vizwiz: nearly real-time answers to visual questions

    Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interf...

    2010

  136. [145]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

  137. [146]

    Ocid-ref: A 3d robotic dataset with embodied language for clutter scene grounding

    Ke-Jyun Wang, Yun-Hsuan Liu, Hung-Ting Su, Jen-Wei Wang, Yu-Siang Wang, Winston H Hsu, and Wen-Chin Chen. Ocid-ref: A 3d robotic dataset with embodied language for clutter scene grounding. arXiv preprint arXiv:2103.07679, 2021

    2021 arXiv

  138. [147]

    Tallyqa: Answering complex counting questions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8076–8084, 2019. 25

    2019

  139. [148]

    Query-relevant images jailbreak large multi-modal models

    Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, and Yu Qiao. Query-relevant images jailbreak large multi-modal models. arXiv preprint arXiv:2311.17600, 7:14, 2023. 26

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.