Pith. sign in

REVIEW 4 major objections 4 minor 47 references

By gating on token-level disagreement between a science-specialist and a general reasoning LLM, Divergence Decoding fuses their strengths and outperforms either model alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:40 UTC pith:ACK6XXY5

load-bearing objection Clever and useful inference-time fusion idea, but the proof doesn't cover the main experiments and the empirical claims need more rigor. the 4 major comments →

arxiv 2607.27248 v1 pith:ACK6XXY5 submitted 2026-07-28 cs.AI cs.LG

Divergence Decoding: Training-Free Capability Fusion

Yimi Wang , Hao Li , Shuo Yang , He Cao , Dechen Zhang , Ziang Wu , Zhiyuan Yan , Fanyang Mo
show 1 more author
Li Yuan
This is my paper
classification cs.AI cs.LG
keywords divergence decodingtraining-free capability fusiontoken-level routingJensen-Shannon divergencespeculative decodingLLM collaborationscientific reasoninginference-time alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to prove that two large language models—one specialized in a scientific domain, one strong at general reasoning—can be fused at inference time without any training. Its central claim is that the Jensen-Shannon divergence between their next-token distributions is a reliable signal of where each model is trustworthy: when the two models largely agree, the specialist's token is kept; when they sharply diverge, control switches to the generalist. The paper gives a theorem stating that under assumptions about the oracle target and the two models' errors, this JS-gated routing has strictly lower expected risk than either constituent model. It also reports that the method outperforms both its specialist and generalist backbones on ChemCoTBench, ChemBench, and GPQA, calling this the A+B > A or B effect. If correct, this establishes a general, training-free paradigm for composing heterogeneous LLM capabilities.

Core claim

Divergence Decoding reconstructs the 'draft-and-verify' skeleton of speculative decoding into an adaptive routing rule. The specialist model A drafts a block of candidate tokens; the general reasoning model B produces teacher-forced next-token distributions for those positions. At each token, the decoder computes the Jensen-Shannon divergence (a symmetric, bounded measure of how different two probability distributions are) between A's and B's predictions. If the divergence is below a threshold τ, the specialist's token is accepted; otherwise a replacement token is sampled from B and verification of the current block stops. The theoretical result, Theorem A.1, proves that under Assumption A.1

What carries the argument

The carrying mechanism is a hard token-level gate g_t = 1[JS(p_A(·|h_t), p_B(·|h_t)) > τ] combined with block drafting and teacher-forced verification. JS divergence is chosen because the square root of JS divergence is a true metric: the proof uses the triangle and reverse-triangle inequalities to guarantee that low-divergence states force the specialist into its in-domain subspace and high-divergence states force the specialist farther from the oracle than the generalist. The top-10 vocabulary JS approximation and sequence-level alignment of mismatched tokenizers are the engineering details that make the signal computable across model families.

Load-bearing premise

The proof assumes an unobservable 'oracle' target distribution r_t together with bounds ε_A and ε_B such that the generalist is everywhere within ε_B of the oracle and the specialist is strictly better only in low-divergence states; if real states violate these inequalities, the theorem's guarantee can fail even if the empirical trick still works.

What would settle it

Find or construct decoding states where the JS divergence between A and B is below τ yet the specialist is closer to a known oracle target than the generalist (or above τ yet the generalist is farther), then show the routed policy's expected risk is not strictly below min(L(A), L(B)). Concretely, on a hand-labeled or verifiable dataset of molecule edits, record s_t, L(A), L(B), and L(τ) for a grid of τ values and test whether L(τ) < min(L(A), L(B)) holds uniformly.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the claim holds, any specialist-generalist pair can be composed at decode time without router training, giving a cheap way to recover reasoning ability lost during domain fine-tuning.
  • The theorem states routed risk is strictly below min(L(A), L(B)); this is a testable prediction for any new model pair, not just the reported benchmarks.
  • Because the gate is asymmetric (specialist default, generalist fallback), the method preserves domain priors while adding robustness, suggesting application to biomedicine, physics, and materials science.
  • The blockwise draft-and-verify design keeps overhead near single-model decoding (overall latency 0.96× in the paper), so the fusion is practical under inference-cost constraints.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The empirical thresholds differ per benchmark (τ=0.65 for ChemCoTBench, τ≈0.2 for ChemBench), which hints that the router is not fully parameter-free: a fixed threshold would likely not generalize across tasks without tuning.
  • The token analysis shows about 77% of fallbacks occur in the first 0–25% of the trajectory and are mostly connectives and natural-language words; I infer the generalist's real role is fixing early reasoning structure rather than injecting domain facts, a testable extension being to compare against a cheap 'reasoning-only' model that cannot supply domain knowledge.
  • The theory relies on an unobservable oracle; a practical extension would be to estimate ε_A and ε_B from observed agreement patterns, turning the sufficient conditions into a calibration procedure.
  • The same gating idea could be applied to more than two models—for example, a cascade of specialists—with divergence against a common generalist selecting which expert to trust at each token.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Divergence Decoding, a training-free token-level routing method that combines a domain-specialized LLM (A) with a general reasoning LLM (B). At each decoding step, the decoder computes the Jensen–Shannon divergence between the two models' next-token distributions; if the divergence is below a threshold τ, the specialist's token is accepted, otherwise the decoder falls back to the generalist. The authors claim a theoretical guarantee (Theorem A.1) that under Assumptions A.1/A.2 the routed policy has strictly lower expected risk than either model. They evaluate the method on ChemCoTBench, ChemBench, and GPQA using Qwen- and Llama-based pairs, reporting improvements over both constituent models on most tasks, and include ablations comparing divergence types, sensitivity to threshold and temperature, and efficiency.

Significance. If the empirical and theoretical claims hold, the paper would provide a simple, training-free paradigm for fusing complementary capabilities of existing LLMs at inference time. The paper clearly distinguishes its objective from speculative decoding and provides a proof-of-concept across multiple benchmarks and model families. Its strengths include a formal conditional theorem, an anonymous code link, and sensitivity/efficiency analyses. However, the theoretical guarantee is disconnected from the empirical operating point, the assumptions are unverified and arguably encode the conclusion, and the empirical results lack statistical rigor (single runs, no error bars or significance tests). The central 'A+B > A or B' claim is therefore not yet convincingly established.

major comments (4)
  1. [Theorem A.1 and Appendix B] The theorem's condition ε_A > √τ + ε_B is impossible for the main experimental setting. Since D_JS ≤ ln2 ≈ 0.693, √D_JS ≤ 0.8326. For τ = 0.65 (ChemCoTBench, Appendix B), √τ ≈ 0.806, so ε_A > 0.806 + ε_B requires ε_B < 0.0264. Assumption A.1 then forces the generalist to be within D_JS(p_B, r_t) ≤ 0.0007 of the oracle at every state—an extraordinarily strong, unverified, and unrealistic condition. Moreover, the guaranteed improvement over B is at most δ² ≤ ε_B² ≤ 0.0007, far below the empirical gains in Tables 1–3. The theory thus does not apply to the headline experiments; the paper must either verify these constants at the operating point or prove a version that applies to realistic ε_B values.
  2. [Section A.2, Assumptions A.1/A.2] The assumptions essentially assert the separation the router is designed to exploit. A.1 posits the generalist is uniformly within ε_B of the oracle; A.2 posits the specialist is at least ε_A away from the oracle outside the in-domain subspace. The proof's key step—that s_t ≤ τ implies h_t ∈ H_in—uses ε_A > √τ + ε_B to force low-divergence states to be in-domain. But this is an assumption about the relationship between observable disagreement and model error, not a derived property. The paper gives no calibration or empirical evidence that these constants hold for ChemDFM-R / R1-Distill-Qwen or Chem-R / R1-Distill-Llama. As stated, the theorem is a valid conditional statement but does not establish that the routing criterion is justified for the tested models.
  3. [Tables 1–3] All reported results are single-run point estimates with no error bars, confidence intervals, or significance tests. The threshold τ is tuned per benchmark (0.65 for ChemCoTBench, 0.2 for ChemBench; Appendix B), and Table 5 shows that performance is sensitive to this choice (e.g., Ring-sys varies from 0.56 at τ=0.58 to 0.67 at τ=0.64). Without repeated evaluations and statistical testing, the claim that Divergence Decoding 'outperforms both the domain-specialized and general-purpose models' is not robustly supported. At minimum, the authors should provide multiple seeds and paired significance tests, and report performance across a range of thresholds rather than a single tuned value.
  4. [Section 4 and Related Work] The paper cites existing token-level routing methods (ETR, CITER, FusionRoute) in Section 2.2 but does not compare against any of them in the experiments. The ablation in Figure 3 compares only different divergence signals and standard speculative sampling, not alternative routing or ensemble strategies. To support the claim of a new 'training-free capability fusion paradigm,' the evaluation should include at least one strong existing token-level fusion baseline under the same evaluation protocol.
minor comments (4)
  1. [Table 1] The abstract and Section 4.1 state that Divergence Decoding 'outperforms both' constituent models, but Table 1 shows several exceptions, e.g., on the Llama backbone, SMILES Equivalence is 0.55 vs. 0.61 for R1-distill-Llama-70B, and Edit:Sub is 58 vs. 63 for Chem-R. The wording should be adjusted to 'outperforms on most tasks.'
  2. [Appendix B] The text says 'the theoretical upper bound of the JS divergence is around 0.70.' The exact bound is ln 2 ≈ 0.693; please state the precise value, especially since the vacuity concern in Major Comment 1 depends on it.
  3. [Section 4.3] The claim that 'injecting reasoning tokens under a broad range of JS-divergence thresholds brings consistent gains' is contradicted by Table 5: at τ=0.58, Ring-sys is 0.56, which is below the ChemDFM-R baseline of 0.58 in Table 1. Please qualify this statement or present a statistical summary across thresholds.
  4. [Figure 4] In panel (a), the x-axis labels include both '0%' and '0-25%', which is confusing. Since the text says 77% of resampling occurs in the 0–25% segment, clarify that '0%' corresponds to the first token position, not the first quartile.

Circularity Check

0 steps flagged

No construction-level circularity: Theorem A.1 is a conditional implication with unverified assumptions, and empirical gains are tuned rather than predicted from the theorem.

full rationale

The paper's central derivation is Theorem A.1 (Appendix A.2): under Assumptions A.1 and A.2 plus the threshold inequalities, the JS-gated policy has strictly lower expected JS-risk than either constituent. The proof uses the metric property of sqrt(D_JS) and the triangle/reverse-triangle inequalities to relate the observable disagreement s_t to the unobservable target distances. The assumptions are strong and are not verified—in particular, A.1 requires the generalist to be uniformly within epsilon_B of the oracle, and A.2 asserts the specialist is good only in an in-domain subspace and bad outside. But the theorem does not define H_in in terms of s_t, and the separation s_t <= tau => h_t in H_in is derived, not assumed. This is a conditional result, not a tautology; its vacuity at the empirical operating point (tau=0.65, with D_JS <= ln2 forcing epsilon_B < 0.026) is a real correctness/verification gap but not circularity. The empirical thresholds are chosen per benchmark (Appendix B: 'we follow the setting of [44] and set the JS threshold to 0.65'; for ChemBench 'we lower the JS threshold to around 0.2'), so the reported gains are partly tuned; however, the paper does not present these gains as numerical predictions of Theorem A.1. Self-citations appear (e.g., ChemCoTBench [24] and [44] with overlapping authorship), but they are used as benchmark/hyperparameter sources, not as the load-bearing justification for the method's validity. The Limitations section itself concedes reliance on complementarity and on hyperparameter calibration, which aligns with this assessment. Therefore no step reduces by construction to its input; the appropriate circularity score is low.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central claim rests on unverified assumptions about an oracle target distribution and error bounds for both models. The per-benchmark threshold and other decoding hyperparameters are chosen by hand, so the empirical result is not a parameter-free prediction. The paper introduces no new physical or architectural entities.

free parameters (4)
  • JS threshold τ = 0.65 (ChemCoTBench), ~0.2 (ChemBench)
    Set per benchmark in Section B; controls fallback frequency; sensitivity analysis in Section C shows results vary with τ; no held-out selection is reported.
  • Block size n = 10
    Fixed to 10 for all experiments (Section B); no ablation or sensitivity analysis for this hyperparameter is provided.
  • Temperature = 1.0
    Chosen in Section B to 'mitigate the influence of model-specific decoding configurations'; temperature changes distribution sharpness and therefore JS values.
  • Top-k support for JS = 10
    Top-10 vocabulary used for JS computation; ablation in Section 4.3 shows top-10 is empirically best, but this is a tuned choice rather than a derived one.
axioms (6)
  • domain assumption Assumption A.1: the generalist B is uniformly robust, with sqrt(D_JS(p_B, r_t)) ≤ ϵ_B for every decoding state h_t.
    Unverified bound involving an unobservable oracle target r_t; the entire high-divergence routing argument depends on it.
  • domain assumption Assumption A.2: the specialist A is δ-better than B on an in-domain subspace H_in and at least ϵ_A from the target outside H_in.
    Unverified; this encodes the exact separation between reliable and unreliable specialist states that the router is supposed to exploit.
  • domain assumption An oracle target next-token distribution r_t exists and expected JS risk L(i) is the right objective.
    Needed to define the theorem; r_t is not observable and is not grounded in any training data or benchmark.
  • domain assumption The task is non-trivial: P(s_t > τ) > 0 and P(s_t ≤ τ) > 0.
    Stated in Appendix A.2; without this, the strict inequality L(τ) < min(L(A), L(B)) is vacuous.
  • standard math sqrt(D_JS) is a metric and satisfies the triangle inequality.
    Standard mathematical property, used in the proof of Theorem A.1.
  • domain assumption The JS divergence between next-token distributions, after sequence alignment, is a reliable reliability signal for token routing.
    Empirical premise of the method; the sequence-alignment procedure in Section B is approximate and can itself affect the computed divergence.

pith-pipeline@v1.3.0-alltime-deepseek · 17240 in / 11500 out tokens · 110363 ms · 2026-08-01T02:40:33.485447+00:00 · methodology

0 comments
read the original abstract

While large language models excel in reasoning, these generalists often lack knowledge for specialized scientific domains. Conversely, domain models~(specialists), while knowledgeable, suffer from specialization side-effects including diminished logic and reduced robustness.To address this dilemma, we introduce Divergence Decoding, a training-free framework for capability fusion. It reconstructs the "draft-and-verify" skeleton of speculative decoding into an adaptive routing mechanism. The core is using Jensen-Shannon divergence to monitor the distributional disagreement between the two models at each token. When the specialist exhibits significant divergence, our method identifies it as a potential reasoning risk and instantaneously routes control to the generalist. This allows the dynamic injection of general reasoning while preserving domain expertise, achieving inference-time policy composition of the generalist and the specialist.We evaluate Divergence Decoding across diverse model families (Qwen and Llama series) on challenging scientific benchmarks (GPQA, ChemBench, and ChemCoTBench). Experimental results demonstrate that Divergence Decoding outperforms both the domain-specialized and general-purpose models, effectively surpassing the performance of most single-model baseline. This suggests that Divergence Decoding provides a general, training-free paradigm for fusing diverse LLM capabilities through adaptive inference-time collaboration.

Figures

Figures reproduced from arXiv: 2607.27248 by Dechen Zhang, Fanyang Mo, Hao Li, He Cao, Li Yuan, Shuo Yang, Yimi Wang, Zhiyuan Yan, Ziang Wu.

Figure 1
Figure 1. Figure 1: Illustration of the proposed Divergence Decoding framework. Left: Scientific tasks often demand both fine-grained domain knowledge and long-chain reasoning, which are typically bifurcated between specialized and generalist LLMs. Right: Our method bridges this gap through a state-dependent routing policy. By measuring the distribution divergence between the two models at each step t, the decoder dynamically… view at source ↗
Figure 2
Figure 2. Figure 2: Divergence Decoding framework. (Step 1) At each iteration, the domain-specialized model drafts a block of candidate tokens and records the corresponding next-token distributions. (Step 2) The reasoning model then evaluates the same draft prefixes to obtain its predictive distributions. (Step 3) The decoder computes the JS divergence between the two distributions at each position. (Step 4) If the divergence… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of different divergence methods during Divergence Decoding. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Resampling Analysis: Resampling patterns by position, uncertainty, and token type. Token Analysis during Resampling: In [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Sensitivity Analysis:For visualization only, metrics with different scales and optimization directions are transformed into task-wise normalized performance scores(higher-is-better). The y-axis reflects normalized relative trends. Hyperparameter Analysis: We also conduct a sensitivity analysis on key hyperparameters. The results show that both the JS-divergence threshold and the temperature affect downstre… view at source ↗
Figure 6
Figure 6. Figure 6: Case study: the divergence decoding between ChemDFM and R1-Distill Qwen affects the reasoning process in the molecule understanding task. injecting reasoning tokens under a broad range of JS-divergence thresholds brings consistent gains by facilitating the integration of domain knowledge and reasoning ability. Detailed number is presented in Appendix 6. Efficiency Analysis. We also compare the end-to-end g… view at source ↗
Figure 7
Figure 7. Figure 7: Case study: divergence decoding between ChemDFM and R1-Distill Qwen affects the reasoning process in the function group counting task. F Code Availability The anonymous source code for reproducing our experiments is available at: https://github. com/wyattxuanyang/Divergence-Decoding G Limitations and Broader Impacts Limitations. Although Divergence Decoding provides a training-free way to fuse domain exper… view at source ↗
Figure 8
Figure 8. Figure 8: Case study: divergence decoding between ChemDFM and R1-Distill Qwen affects the reasoning process in the molecule editing task. Question: You are given a SMILES representation of a molecule, please count the number of the primary sulfonamides in the Molecule. SMILES: NS(=O)(=O)c1ccc(NC(=O)CN2CCN(c3cc ccc3)CC2)cc1 Okay, to determine how many primary sulfonamide groups are present, I first analyze the SMILES… view at source ↗
Figure 9
Figure 9. Figure 9: Case study: divergence decoding between ChemDFM and R1-Distill Qwen affects the reasoning process in the function group counting task. part of the overhead, Divergence Decoding still requires loading and querying two LLMs, leading to higher memory requirements than single-model decoding. Broader Impacts. Divergence Decoding may have positive impacts by enabling stronger scientific reasoning systems without… view at source ↗
Figure 10
Figure 10. Figure 10: Case study: divergence decoding between ChemDFM and R1-Distill Qwen affects the reasoning process in the function group counting task. At the same time, stronger scientific LLM systems may also introduce risks. Users may over-rely on generated reasoning traces or final answers, especially in high-stakes scientific, biomedical, or engineering settings where incorrect conclusions can have real-world consequ… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 22 linked inside Pith

  1. [1]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    A Yang Qwen, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengpeng Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5

    2026

  5. [5]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Mistral 7b

    Albert Q Jiang, A Sablayrolles, A Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. Mistral 7b. arxiv.arXiv preprint arXiv:2310.06825, 10: 3, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback....

    Pith/arXiv arXiv 2022

  10. [10]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Model card and evaluations for claude models, 2023

    Anthropic. Model card and evaluations for claude models, 2023. URL https://www. anthropic.com/news/claude-2

    2023

  12. [12]

    Introducing the next generation of claude, 2024

    Anthropic. Introducing the next generation of claude, 2024. URL https://www.anthropic. com/news/claude-3-family

    2024

  13. [13]

    Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team Google. Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Gemma open models, 2024

    Google. Gemma open models, 2024. URLhttps://ai.google.dev/gemma

    2024

  15. [15]

    Chemdfm: A large language foundation model for chemistry

    Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Hongshen Xu, Zichen Zhu, Su Zhu, Shuai Fan, et al. Chemdfm: A large language foundation model for chemistry. InNeurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, 2024

    2024

  16. [16]

    Chemdfm-r: A chemical reasoning llm enhanced with atomized chemical knowledge.arXiv preprint arXiv:2507.21990, 2025

    Zihan Zhao, Bo Chen, Ziping Wan, Lu Chen, Xuanze Lin, Shiyang Yu, Situo Zhang, Da Ma, Zichen Zhu, Danyang Zhang, et al. Chemdfm-r: A chemical reasoning llm enhanced with atomized chemical knowledge.arXiv preprint arXiv:2507.21990, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesis.arXiv preprint arXiv:2601.17687, 2026

    Hao Li, He Cao, Shenyao Peng, Zijing Liu, Bin Feng, Yu Wang, Zhiyuan Yan, Yonghong Tian, Yu Li, and Li Yuan. Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesis.arXiv preprint arXiv:2601.17687, 2026. 11

    arXiv 2026

  18. [18]

    Biomistral: A collection of open-source pretrained large language models for medical domains

    Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. Biomistral: A collection of open-source pretrained large language models for medical domains. InFindings of the association for computational linguistics: acl 2024, pages 5848–5864, 2024

    2024

  19. [19]

    Intern-s1: A scientific multimodal foundation model.arXiv preprint arXiv:2508.15763, 2025

    Lei Bai, Zhongrui Cai, Yuhang Cao, Maosong Cao, Weihan Cao, Chiyu Chen, Haojiong Chen, Kai Chen, Pengcheng Chen, Ying Chen, et al. Intern-s1: A scientific multimodal foundation model.arXiv preprint arXiv:2508.15763, 2025

    arXiv 2025

  20. [20]

    A comprehensive survey of scientific large language models and their applications in scientific discovery

    Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, and Jiawei Han. A comprehensive survey of scientific large language models and their applications in scientific discovery. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8783–8817, 2024

    2024

  21. [21]

    A survey of scientific large language models: From data foundations to agent frontiers.arXiv preprint arXiv:2508.21148, 2025

    Ming Hu, Chenglong Ma, Wei Li, Wanghan Xu, Jiamin Wu, Jucheng Hu, Tianbin Li, Guohang Zhuang, Jiaqi Liu, Yingzhou Lu, et al. A survey of scientific large language models: From data foundations to agent frontiers.arXiv preprint arXiv:2508.21148, 2025

    arXiv 2025

  22. [22]

    Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    Beyond chemical qa: Evaluating llm’s chemical reasoning with modular chemical operations.arXiv preprint arXiv:2505.21318, 2025

    Hao Li, He Cao, Bin Feng, Yanjun Shao, Xiangru Tang, Zhiyuan Yan, Li Yuan, Yonghong Tian, and Yu Li. Beyond chemical qa: Evaluating llm’s chemical reasoning with modular chemical operations.arXiv preprint arXiv:2505.21318, 2025

    arXiv 2025

  25. [25]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.Advances in Neural Information Processing Systems, 37:95266–95290, 2024

    2024

  26. [26]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  27. [27]

    The sciqa scientific question answering benchmark for scholarly knowledge.Scientific Reports, 13(1):7240, 2023

    Sören Auer, Dante AC Barone, Cassiano Bartz, Eduardo G Cortes, Mohamad Yaser Jaradeh, Oliver Karras, Manolis Koubarakis, Dmitry Mouromtsev, Dmitrii Pliukhin, Daniil Radyush, et al. The sciqa scientific question answering benchmark for scholarly knowledge.Scientific Reports, 13(1):7240, 2023

    2023

  28. [28]

    Moleculeqa: A dataset to evaluate factual accuracy in molecular comprehension

    Xingyu Lu, He Cao, Zijing Liu, Shengyuan Bai, Leqing Chen, Yuan Yao, Hai-Tao Zheng, and Yu Li. Moleculeqa: A dataset to evaluate factual accuracy in molecular comprehension. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3769–3789, 2024

    2024

  29. [29]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InInternational Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  30. [30]

    Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling.arXiv preprint arXiv:2302.01318, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Are large language models superhuman chemists?arXiv preprint arXiv:2404.01475, 2024

    Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-García, Benedict Emoek- abu, Aswanth Krishnan, Tanya Gupta, Mara Schilling-Wilhelmi, Macjonathan Okereke, Anagha Aneesh, et al. Are large language models superhuman chemists?arXiv preprint arXiv:2404.01475, 2024. 12

    Pith/arXiv arXiv 2024

  32. [32]

    Decoupled peak property learning for efficient and interpretable electronic circular dichroism spectrum prediction.Nature Computational Science, 5(3):234–244, 2025

    Hao Li, Da Long, Li Yuan, Yu Wang, Yonghong Tian, Xinchang Wang, and Fanyang Mo. Decoupled peak property learning for efficient and interpretable electronic circular dichroism spectrum prediction.Nature Computational Science, 5(3):234–244, 2025

    2025

  33. [33]

    Prollama: A protein large language model for multi-task protein language processing.IEEE Transactions on Artificial Intelligence, 2025

    Liuzhenghao Lv, Zongying Lin, Hao Li, Yuyang Liu, Jiaxi Cui, Calvin Yu-Chian Chen, Li Yuan, and Yonghong Tian. Prollama: A protein large language model for multi-task protein language processing.IEEE Transactions on Artificial Intelligence, 2025

    2025

  34. [34]

    Weakly-supervised 3d spatial reasoning for text-based visual question answering.IEEE Transactions on Image Processing, 32:3367–3382, 2023

    Hao Li, Jinfa Huang, Peng Jin, Guoli Song, Qi Wu, and Jie Chen. Weakly-supervised 3d spatial reasoning for text-based visual question answering.IEEE Transactions on Image Processing, 32:3367–3382, 2023

    2023

  35. [35]

    Developing chemdfm as a large language foundation model for chemistry

    Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, Zichen Zhu, Su Zhu, et al. Developing chemdfm as a large language foundation model for chemistry. Cell Reports Physical Science, 6(4), 2025

    2025

  36. [36]

    Txgemma: Efficient and agentic llms for therapeutics

    Eric Wang, Samuel Schmidgall, Paul F Jaeger, Fan Zhang, Rory Pilgrim, Yossi Matias, Joelle Barral, David Fleet, and Shekoofeh Azizi. Txgemma: Efficient and agentic llms for therapeutics. arXiv preprint arXiv:2504.06196, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077, 2024

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Eagle-2: Faster inference of language models with dynamic draft trees, 2024b.URL https://arxiv

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees, 2024b.URL https://arxiv. org/abs/2406.16858, 1(2), 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Mixture-of-experts with expert choice routing.Advances in Neural Information Processing Systems, 35:7103–7114, 2022

    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing.Advances in Neural Information Processing Systems, 35:7103–7114, 2022

    2022

  41. [41]

    An expert is worth one token: Synergizing multiple expert llms as generalist via expert token routing

    Ziwei Chai, Guoyin Wang, Jing Su, Tianjie Zhang, Xuanwen Huang, Xuwu Wang, Jingjing Xu, Jianbo Yuan, Hongxia Yang, Fei Wu, et al. An expert is worth one token: Synergizing multiple expert llms as generalist via expert token routing. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 113...

    2024

  42. [42]

    Citer: Collaborative inference for efficient large language model decoding with token-level routing.arXiv preprint arXiv:2502.01976, 2025

    Wenhao Zheng, Yixiao Chen, Weitong Zhang, Souvik Kundu, Yun Li, Zhengzhong Liu, Eric P Xing, Hongyi Wang, and Huaxiu Yao. Citer: Collaborative inference for efficient large language model decoding with token-level routing.arXiv preprint arXiv:2502.01976, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Token-level llm collaboration via fusionroute.arXiv preprint arXiv:2601.05106, 2026

    Nuoya Xiong, Yuhang Zhou, Hanqing Zeng, Zhaorun Chen, Furong Huang, Shuchao Bi, Lizhu Zhang, and Zhuokai Zhao. Token-level llm collaboration via fusionroute.arXiv preprint arXiv:2601.05106, 2026

    Pith/arXiv arXiv 2026

  44. [44]

    Sparse but critical: A token-level analysis of distributional shifts in rlvr fine-tuning of llms.arXiv preprint arXiv:2603.22446, 2026

    Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, and Jingren Zhou. Sparse but critical: A token-level analysis of distributional shifts in rlvr fine-tuning of llms.arXiv preprint arXiv:2603.22446, 2026

    arXiv 2026

  45. [45]

    Raman-01: Compact rl-enhanced physics solver

    Sai Praneeth Diddigam. Raman-01: Compact rl-enhanced physics solver. https:// huggingface.co/think-a-tron/raman-01-1.7B , 2025. RL (GRPO) finetuned on Qwen3- 1.7B

    2025

  46. [46]

    Unlocking on-policy distillation for any model family, 2025

    Carlos Miguel Patiño, Kashif Rasul, Quentin Gallouédec, Ben Burtenshaw, Sergio Paniego, Vaibhav Srivastav, Thibaud Frere, Ed Beeching, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Unlocking on-policy distillation for any model family, 2025

    2025

  47. [47]

    Freestyleret: retrieving images from style-diversified queries

    Hao Li, Yanhao Jia, Peng Jin, Zesen Cheng, Kehan Li, Jialu Sui, Chang Liu, and Li Yuan. Freestyleret: retrieving images from style-diversified queries. InEuropean Conference on Computer Vision, pages 258–274. Springer, 2024. 13 A Theoretical Analysis In this section, we provide sufficient conditions under which a JS-thresholded routing policy outper- form...

    2024