Pith. sign in

REVIEW 4 major objections 5 minor 14 references

This paper claims that log-likelihood-based evaluation systematically underestimates the teacher–student quality gap in cross-architecture distillation, and can even reverse the ranking of design choices.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 17:15 UTC pith:Q7UOJGPT

load-bearing objection Plausible and useful paper, but the headline evaluation-gap claim is confounded by harness, prompt, and task-version differences, not just scoring. the 4 major comments →

arxiv 2603.26556 v2 pith:Q7UOJGPT submitted 2026-03-27 cs.CL cs.AI

When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

Juan Gabriel Kostelec , Qinghai Guo This is my paper
classification cs.CL cs.AI MSC 68T5068T07
keywords knowledge distillationcross-architecture distillationhybrid sequence modelslinear attentionperplexity evaluationgeneration-based evaluationKimi Delta AttentionKV cache efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that perplexity-based evaluation is a systematically misleading yardstick for cross-architecture distillation, where a pretrained Transformer teacher is converted into a more efficient hybrid student. The headline evidence: a 7B-parameter distilled model that appears to match its teacher within 0.2 percentage points under log-likelihood scoring falls 20.8 points behind when it must generate answers autoregressively. Using a 0.6B-parameter controlled testbed, the authors ablate six design axes (objective, masking, duration, dataset, freezing, architecture) and find the same pattern everywhere: likelihood-based evaluation compresses differences and can even reverse the ranking of competing choices. They then propose a hybrid architecture and a multi-stage distillation pipeline, and report that their best student retains 86–90% of teacher accuracy on knowledge benchmarks while cutting KV cache memory by up to 75% and improving time-to-first-token by 2–4× at long contexts. The practical conclusion: generation-based evaluation should be the primary protocol for judging distilled models.

Core claim

The central claim is that in cross-architecture distillation, log-likelihood-based scoring consistently underestimates the teacher–student quality gap and can reverse design-choice rankings. The paper quantifies this with a 7B distilled model that scores within 0.2pp of its teacher under perplexity yet trails by 20.8pp under generation-based evaluation; it also demonstrates the divergence across every design axis in its 0.6B ablations, including objective (KD vs SFT), masking (full vs completion-only), training duration, dataset, parameter freezing, and architecture choice. The authors attribute the gap to a model being able to rank correct answers highly without being able to generate them.

What carries the argument

Two mechanisms carry the argument. The first is the evaluation-pair contrast: log-likelihood ranking of multiple-choice candidates versus autoregressive generation scoring, which the paper shows can disagree qualitatively about which model is better. The second is the Hybrid-KDA student architecture and the GenDistill pipeline built around it. Hybrid-KDA is a Transformer with 7 of 28 attention layers preserved and 21 replaced by KDA (Kimi Delta Attention), a linear-attention mixer whose recurrent state is updated by a delta rule with per-channel forget gates; the mixture preserves retrieval and compositional abilities while shrinking the KV cache. GenDistill is a three-stage pipeline—mixer-l

Load-bearing premise

The headline 0.2pp-vs-20.8pp figure assumes the published perplexity numbers for the 7B distilled model were produced under the same evaluation conditions as this paper's generation-based runs; if the harnesses, prompts, few-shot settings, or task versions differ, part of the gap is an artifact.

What would settle it

Run the same 7B distilled checkpoint under both protocols with identical prompts, few-shot settings, and harness versions; if the teacher–student gap under matched log-likelihood scoring is already large (e.g., above 10pp), the claim that perplexity underestimates the gap is weakened. Alternatively, in the 0.6B testbed, find one design choice where perplexity and generation ranking agree—the paper claims divergence across all six axes, so a single counterexample would falsify that claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Published distillation results that report only perplexity or log-likelihood scores likely overstate student quality; re-evaluating those checkpoints under generation would be the direct test.
  • Generation-based evaluation should become the primary protocol in cross-architecture distillation, with perplexity at best as a cheap complement.
  • The recipe—KD with completion-only masking, teacher-aligned instruction data, and frozen attention layers—gives the best generation quality in every ablation, so future distillation pipelines should adopt these as defaults.
  • Hybrid architectures with a minority of retained attention layers are essential: pure linear models collapse on multi-step reasoning and in-context retrieval, while hybrids keep most teacher knowledge.
  • The divergence persists for architecture comparisons, so conclusions about which mixer or hybrid design is better should be drawn from generation metrics, not perplexity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If perplexity is systematically optimistic here, likelihood-based validation during training may select checkpoints that generate poorly; a cheap fix would be to guide early stopping and checkpoint selection with a small generation-based validation set.
  • The 7B example is striking but mixes two evaluation setups; a matched-harness replication at 7B would settle whether the quantitative 20.8pp gap is real or partly an artifact, and whether the recipe recommendations hold at scale.
  • The 'teacher-aligned instruction data' principle suggests a testable extension: hold the pipeline fixed and vary only the teacher used to generate the instruction responses, predicting that same-family teachers produce the best generation quality.
  • The long-context results (about 58% of teacher accuracy on LongBench) imply the efficiency gains at 128K are not yet matched by quality; adding a long-context distillation stage is the obvious next step, and one could test whether the PPL-vs-generation divergence also appears on long-context benchmarks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that log-likelihood/perplexity-based evaluation systematically understates the gap between a Transformer teacher and a distilled hybrid student, and can even reverse the ranking of design choices. It proposes GenDistill, a multi-stage distillation pipeline into a Hybrid-KDA architecture, and reports systematic 0.6B-scale ablations over objective, masking, duration, dataset, freezing, and architecture, evaluated under both LM Evaluation Harness (perplexity) and EvalScope (generation). A headline 7B example (QRWKV6-7B-Instruct vs Qwen2.5-7B-Instruct) shows a 0.2pp teacher--student gap under perplexity but a 20.8pp gap under generation. The paper concludes that generation-based evaluation should be the primary protocol for cross-architecture distillation.

Significance. If the central measurement claim is correct, the paper addresses a real and timely problem: many cross-architecture distillation pipelines report only likelihood-based scores, which may overstate student quality. The internal 0.6B ablations are extensive and mostly well controlled within each protocol: freezing attention layers (Table 6), objective/masking/duration (Table 5), and architecture (Tables 2/19) are each varied systematically, and the paper reports efficiency gains, long-context limitations, and negative results (Appendix L) transparently. The strengths are the breadth of ablations, the explicit two-protocol evaluation design, and the promised release of code and checkpoints. However, as detailed below, the headline cross-scale demonstration and even the internal protocol comparisons are confounded by differences beyond the scoring rule, so the paper's central conclusion is not yet established at the strength claimed.

major comments (4)
  1. [Table 1 / Section 5 / footnote 4] The caption claims 'only the evaluation protocol differs,' but this is false. The PPL column is taken from the original QRWKV6 paper (LM Evaluation Harness), while the generation column is the authors' EvalScope runs; these differ in prompt formatting, few-shot counts, decoding parameters, and task versions. MMLU is used for PPL and MMLU-Redux for generation, and ARC/HellaSwag/WinoGrande prompts are not matched across harnesses. The teacher's own ARC-Challenge numbers (55.0 PPL vs 87.7 generation) are too large to attribute to scoring alone. The 0.2pp-vs-20.8pp headline therefore mixes protocol effects with harness, prompt, and task-version effects. Please either rerun PPL evaluation with the same prompts/task versions used in generation, or explicitly rephrase the claim to acknowledge the confound.
  2. [Section 5; Tables 5, 6, 19] The same confound affects the internal 0.6B ablations that are the controlled core of the paper. Perplexity-based results always come from LM-Eval (raw prompts, different few-shot settings), while generation-based results come from EvalScope (chat-style templates, different few-shot settings). A student trained with completion-only chat-format KD may do well under chat-formatted generation prompts but poorly under raw LM-Eval likelihood prompts, creating an apparent 'compression' or 'ranking reversal' without isolating likelihood vs generation per se. Please provide at least one controlled comparison where the prompt format and task version are identical, with only the scoring rule (likelihood ranking vs autoregressive generation) varied.
  3. [Section 6.4 / Section 7] The claim that PPL 'can invert rankings' is stated broadly, but the evidence is not robust to the confound above. For example, the specific statement in Section 7 that 'SFT achieves lower perplexity than KD despite 8–11pp worse generation accuracy' is not consistently supported by Table 5: the SFT-vs-KD perplexity relationship varies by task and token budget (e.g., KD C 500M has lower LAMBADA PPL than SFT C 500M, while SFT P+C 750M has lower LAMBADA PPL than KD P+C 750M). Please identify the precise configuration being cited and clarify that the inversion is conditional on task, masking, and duration. The broader qualitative conclusion may survive, but the current phrasing overstates the consistency.
  4. [Section 4.1.1 / Abstract] The paper states that generation-based evaluation is used 'throughout' to guide design decisions, but the attention-layer selection procedure in Section 4.1.1 selects layers by perplexity on three datasets (AR, C-EVAL, FineWeb-EDU). This is not a fatal contradiction—layer selection is an architecture search step, not a final-quality measurement—but it should be acknowledged explicitly, since the manuscript's own protocol is not generation-only at every stage. Please clarify the role of perplexity in the search and whether a generation-based selection criterion was considered.
minor comments (5)
  1. [Abstract / Section 6.1] The abstract says the best student retains 86–90% of teacher accuracy on knowledge benchmarks, but Section 6.1 says 86–99% on knowledge and common-sense tasks. Please make the numbers consistent and specify which benchmark subset each range refers to.
  2. [Section 2] Typo: 'lm-evalutaion harness' should be 'lm-evaluation harness'.
  3. [Appendix D / References] The text cites 'PostNAS (Gu et al., 2025)', but the reference list entry appears as 'Jet-Nemotron' (Gu et al., 2025). Please align the cited name with the reference entry.
  4. [Table 18] The column header 'HEval' is used inconsistently with 'HumanEval' elsewhere; please standardize. Also, the greedy-decoding parentheticals are useful, but a sentence explaining why sampling and greedy results differ would help.
  5. [Appendix J] The note that BBH/GSM8K/IFEval are excluded because scores are trivially low under LM-Eval is informative but should appear in Section 5 as well, since it affects the interpretation of the two-protocol comparison.

Circularity Check

0 steps flagged

No circularity: central claims are empirical measurements, not derivations; the only self-citation is background and not load-bearing.

full rationale

The paper's central claims—that perplexity-based evaluation compresses teacher–student gaps and can reverse design-choice rankings—are established by direct measurement on identical checkpoints (Tables 5, 6, 19) and by an external cross-harness comparison (Table 1). No constant or parameter is fitted and then re-announced as a prediction; the design recipe was selected using generation-based evaluation, and the recommendation that generation-based evaluation be primary is a methodological conclusion from those measurements, not a reduction of the conclusion to its premises. The only self-citation (Kostelec & Guo, 2025) appears in a related-work list about early linearization methods and supports no load-bearing step. The Table 1 comparison is confounded by harness/prompt/task-version differences (LM Evaluation Harness with standard MMLU vs EvalScope with MMLU-Redux), but this is a measurement-validity concern, not circularity; the internal 0.6B ablations use matched harnesses and carry the argument. No quoted step satisfies the standard of Eq. X = Eq. Y by construction, so no circular step is reported.

Axiom & Free-Parameter Ledger

7 free parameters · 7 axioms · 1 invented entities

The paper's contribution is empirical, so the ledger is dominated by hyperparameters chosen by hand/ablation and by assumptions about evaluation comparability. The free parameters (token budgets, LR, layer selection, sampling config) do not fit the central claim but do define the recipe. The most consequential assumptions are the comparability of third-party PPL numbers (Table 1) and the extension of dual-protocol findings from multiple-choice-style tasks to generation-heavy tasks.

free parameters (7)
  • Stage 3a token budget = 500M (ablated: 100M/250M/500M)
    Chosen as the default for all main experiments; Table 3 shows downstream gains are modest past 250M, so the choice is partially ad hoc.
  • Stage 3b token budget = 500M (ablated: 250M/500M/750M)
    Default for all main results unless stated; Table 5 shows mixed per-benchmark trends with extended training.
  • Retained attention layers and indices = 7 of 28, indices {0,2,6,11,13,18,21}
    Selected by beam search minimizing average perplexity on AR + C-Eval + FineWeb-EDU; the 7-layer count matches a 1:3 attention-to-SSM ratio from prior hybrid models.
  • Stage 3b learning rate = 2.5e-5 (1e-5 ablated)
    Kept because the smaller LR slows gains on knowledge benchmarks (Appendix F).
  • Generation-eval sampling configuration = temperature 0.7, top-p 0.8, top-k 20, seed 42
    All generation-based scores depend on this sampling; greedy results in Appendix I differ modestly and mostly preserve rankings.
  • KD loss hyperparameters = tau=1.0, alpha_CE=0.0
    Appendix E shows any positive CE weight degrades knowledge benchmarks; full-vocabulary KD is used.
  • Pretraining data filter = FineWeb-EDU score>=4 subset
    Data curation choice for Stages 1-3a; the filter threshold is not ablated.
axioms (7)
  • domain assumption Teacher forward-KL logits are an effective and sufficient distillation target for student generation quality
    Stage 3 uses forward KL with tau=1.0 and alpha_CE=0.0 (Appendix B); if teacher soft targets were uninformative, the recipe's ranking claims would be undermined.
  • domain assumption EvalScope sampled generation (temperature 0.7, top-p 0.8) is a valid operationalization of 'generation quality'
    All generation-based numbers rest on this configuration; greedy results (Appendix I, parentheses) mostly preserve rankings, lending partial support.
  • domain assumption PPL numbers from Lin et al. (2025) for QRWKV6-7B are directly comparable to the authors' own EvalScope runs
    Table 1's 0.2pp-to-20.8pp headline relies on this; harness, prompt formatting, and few-shot settings are not controlled by the authors.
  • domain assumption Dual-protocol conclusions from multiple-choice-style tasks extend to generation-heavy reasoning and code tasks
    GSM8K, HumanEval, BBH, and IFEval are never scored under the likelihood protocol in the 0.6B ablations (footnote 5, Appendix J); the claim that likelihood compresses gaps there relies on the external Table 1 comparison.
  • standard math KDA recurrence (Eq. 4) and MOHAWK-style staged alignment are sound background
    Equations 1-4 from prior work are used as-is; no new math is derived.
  • domain assumption Qwen3-0.6B teacher, AM-Qwen3-Distilled, and FineWeb-EDU are used as-is from third parties
    Student quality is measured relative to a third-party teacher; dataset provenance and quality are external inputs.
  • domain assumption Design-choice rankings and the protocol gap observed at 0.6B transfer to 1B-7B scales
    Stated in Section 7.1 as needing re-tuning at larger scales; Table 1 is the only cross-scale support and is internally mixed-source.
invented entities (1)
  • Hybrid-KDA student (7/28 attention layers, KDA mixers elsewhere) independent evidence
    purpose: Efficient hybrid sequence model that retains retrieval-critical attention layers while using linear-time KDA for most layers
    Not a physical entity but a new architectural configuration; its claims are falsifiable via public benchmarks (Tables 2, 19-21) and the reported efficiency curves, though code/checkpoints are not yet released.

pith-pipeline@v1.3.0-alltime-deepseek · 28287 in / 19583 out tokens · 193597 ms · 2026-08-02T17:15:14.382492+00:00 · methodology

0 comments
read the original abstract

Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference costs. However, achieving high-quality generation in distilled models requires careful joint design of both the student architecture and the distillation process. Many prior distillation works evaluate downstream multiple-choice benchmarks by ranking candidate answers with log-likelihood rather than requiring autoregressive generation, which can obscure important differences in model quality. For example, on overlapping benchmarks, we show that a 7B distilled model that nearly matches its teacher to within 0.2 pp under log-likelihood scoring falls behind by 20.8 pp when it must generate answers autoregressively. We investigate this phenomenon with GenDistill, a multi-stage pipeline we designed for distilling a pretrained Transformer into an efficient Hybrid Kimi Delta Attention (Hybrid-KDA) student. Using it as a controlled testbed on Qwen3-0.6B, we systematically ablate six design axes (training objective, loss masking, training duration, dataset selection, parameter freezing, and architecture choice) and evaluate every choice under both log-likelihood and generation-based protocols. We find that log-likelihood-based evaluation consistently underestimates the gap between teacher and student, and can in some cases reverse the ranking of design choices, so conclusions drawn from perplexity-only evaluation may be misleading. Among the factors we study, dataset selection, completion-only masking, and freezing attention layers during post-training have the largest impact on generation quality. Our best distillation recipe, using a Hybrid-KDA model as the student, retains 86-90% of teacher accuracy on knowledge benchmarks while reducing KV cache memory by up to 75% and improving time-to-first-token by 2-4x at 128K-token contexts.

Figures

Figures reproduced from arXiv: 2603.26556 by Juan Gabriel Kostelec, Qinghai Guo.

Figure 1
Figure 1. Figure 1: Architecture overview. Left: The two block types in the hybrid transformer: an Attention block (7 of 28 layers, frozen from the teacher) and a KDA block (21 of 28 layers). MLPs and layer norms are initialized from the teacher and trained only during end-to-end distillation (Stage 3). Right: Internal structure of the KDA layer. Fill colors indicate the earliest distillation stage in which each component beg… view at source ↗
Figure 2
Figure 2. Figure 2: Inference efficiency comparison (single GPU, batch size 1). [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Held-out KL loss during Stage 3b for [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Evaluation loss curves for attention layer selection methods during Stage 3b. Greedy Learned [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: (a) Theoretical cache memory vs. sequence length. Attention cache grows linearly with context; SSM state is constant. (b) Peak throughput (sweeping batch size to OOM) at four context lengths. An￾notations show relative throughput vs. the Teacher. At 32K+ tokens, SSM variants achieve 1.9–2.5× the teacher’s peak throughput by fitting larger batches. sequences of at most 4K tokens. The resulting gap relative … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 6 linked inside Pith

  1. [1]

    Using a Sigmoid gate underperforms SiLU

    SiLU✗VO 24.42 Random init SiLU✗Random 24.88 2025). Using a Sigmoid gate underperforms SiLU. We hypothesize this is partly because Sigmoid does not admit an identity-style initialization for the gate during distillation, which may hinder optimization. Adding a post-SSM normalization layer is slightly worse in perplexity and, in our downstream evaluation, s...

    2025

  2. [3]

    URLhttps://zenodo.org/records/10256836. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Gardent, Minghao Wu, Alham Fikri Aji, Harish Tayyar Madabushi, and Pasquale Minervini. Are we done with mmlu? InProceedings of the 202...

    arXiv 2025

  3. [5]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi

    URLhttps: //arxiv.org/abs/2503.14456. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.Communications of the ACM, 64(9):99–106,

    Pith/arXiv arXiv

  4. [7]

    URLhttps://arxiv.org/abs/2510.26692. MiniCPM Team, Wenhao An, Yingfa Chen, Yewei Fang, Jiayi Li, Xin Li, Yaohui Li, Yishan Li, Yuxuan Li, Biyuan Lin, Chuan Liu, Hezi Liu, Siyuan Liu, Hongya Lyu, Yinxu Pan, Shixin Ren, Xingyu Shen, Zhou Su, Haojun Sun, Yangang Sun, Zhen Leng Thai, Xin Tian, Rui Wang, Xiaorong Wang, Yudong Wang, Bo Wu, Xiaoyue Xu, Dong Xu, ...

    Pith/arXiv arXiv

  5. [8]

    URLhttps: //arxiv.org/abs/2602.11761. Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants,

    arXiv

  6. [9]

    org/abs/2505.14464

    URLhttps://arxiv. org/abs/2505.14464. 16 Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. InAdvances in Neural Information Processing Systems 37 (NeurIPS),

    Pith/arXiv arXiv

  7. [10]

    URLhttps://arxiv.org/ abs/2504.10449. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang ...

    Pith/arXiv arXiv

  8. [11]

    URLhttps://arxiv.org/abs/2507.22448. 17 This appendix provides supplementary details and ablation studies: •Appendix A: Model Architecture Details •Appendix B: Hyperparameter Details •Appendix C: Architecture Design Choices •Appendix D: Attention Layer Selection Ablation •Appendix E: Top-K Sparsification and CE Loss Weight •Appendix F: Learning Rate Ablat...

    Pith/arXiv arXiv

  9. [12]

    Specifically, we investigate: (i) the output gate activation function (SiLU vs

    C Architecture Design Choices We conduct an ablation study to analyze the impact of key architectural choices on the distilled hybrid model. Specifically, we investigate: (i) the output gate activation function (SiLU vs. Sigmoid), (ii) the presence of a post-SSM normalization layer, and (iii) the initialization strategy for the mixer parameters (VO initia...

    2048

  10. [14]

    Hybrid” models retain 7 attention layers; “Pure

    (synthetic needle-in-a-haystack retrieval, 4K–128K tokens). Importantly, no dedicated long-context extension training was performed: all students were distilled on 25 2K 8K 32K 128K Prefill Length 0 2000 4000 6000 8000Peak Throughput (tok/s) 1.0x 1.0x 1.0x 1.0x 1.26x 1.73x 2.52x 5.16x 0.87x 1.16x 1.66x 3.44x 1.09x 1.43x 1.86x 2.62x 0.83x 1.12x 1.45x 2.21x...

    2000

  11. [2021]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. InFindings of the Association for Computational Linguistics: ACL 2023, pp. 13003–13051,

    2023

  12. [2024]

    URL https://arxiv.org/abs/2402.19427. Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih- Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, and Pavlo Molchanov. Hymba: A hybrid-head architecture for small language models. InInternational Conference on Learning Representatio...

    Pith/arXiv arXiv

  13. [2025]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin

    URLhttps://arxiv.org/abs/2511.00576. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: Measuring massive multitask language understanding in Chinese. InFindings of the Association for Computational Linguistics: ACL 2024, pp. 11260–11285,

    arXiv 2024

  14. [2026]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord

    URLhttps://arxiv.org/abs/2601.22156. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge,

    arXiv