Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Matching fine-tune data proportions to a model's own pretrained knowledge improves multi-domain performance; VersaTune reports a 35.21% gain over uniform weights.

desk verdict VersaTune is a practical, well-tested SFT data-mixing framework whose headline numbers rest on an unvalidated knowledge-detection step; worth refereeing after major revision. read the letter →

arxiv 2411.11266 v5 pith:L547C2RH submitted 2024-11-18 cs.CL

Keer Lu , Keshi Zhao , Zhuoran Zhang , Zheng Liang , Da Pan , Shusen Zhang , Xin Wu , Guosheng Dong
show 3 more authors
Bin Cui Tengjiao Wang Wentao Zhang
This is my paper · ORCID
classification cs.CL
keywords VersaTunedatacompositionsupervisedfine-tuningmulti-domaincapabilityknowledgedistributiondetectioncatastrophicforgettinglearnablepotentialdomainexpansion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VersaTune is a recipe for deciding how to mix data from different domains during supervised fine-tuning of a large language model. It claims that a model benefits most when the share of fine-tuning data from each domain matches the domain-knowledge distribution the model already has from pretraining. The framework estimates that distribution by generating free-form samples from a bare $\langle BOS\rangle$ token, classifying each sample into domains with a fine-tuned classifier, and averaging over repeated runs. Initial domain weights are set to the detected distribution, then adjusted during training according to each domain's learnable potential and forgetting degree. The paper reports a 35.21% average multi-domain improvement over uniform weights, a Qwen-2.5-32B model that beats GPT-4o, Claude3.5-Sonnet and DeepSeek-V3 on its benchmarks, and a 38.77% reduction in non-target-domain degradation during flexible domain expansion.

What carries the argument

The load-bearing object is the domain-knowledge distribution $\vec{P} = (P_j)_{j=1}^k$, estimated in Algorithm 1 by generating $N_S$ samples from the base model seeded with only a $\langle BOS\rangle$ token, asking a fine-tuned classifier $M_P$ for per-sample domain probability vectors, and aggregating over $T$ iterations. This distribution sets the initial SFT data proportions via $P_{SFT}(x) \approx P_{knowledge}(x)$. Training then reweights domains with two feedback signals: learnable potential $\gamma_j^{(t)} = \max\{(\ell_j^{\theta(t)} - \ell_j^{ref})/\ell_j^{\theta(t)}, 0\}$ and forgetting degree $\varphi_j^{(t)} = \max\{(\ell_j^{\theta(t)} - \ell_j^{\theta(t-1)})/\ell_j^{\theta(t-1)}, 0\}$, where $\ell_j^{ref}$ is the minimum loss a small reference model reaches when fine-tuned on domain $j$ alone. The update $P_j^{(t)'} = P_j^{(t-1)}(1+\sigma\gamma_j^{(t)})$ followed by renormalization carries the adaptivity. The mechanism matters because it turns the abstract 'match the model's knowledge' principle into concrete, trainable per-domain weights, and the paper's ablation shows the dynamic component outperforms fixed matched weights.

What would settle it

Run VersaTune's knowledge probe on a model pretrained on a corpus with a known domain mixture, then compare the estimated distribution to the true mixture; if the estimate is systematically off, or if matching it does not beat both uniform and oracle-matched proportions on held-out domain benchmarks, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is stated as Statement 1: an LLM fine-tuned with domain-specific data proportions $P_{SFT}(x)$ that align with its pretrained output distributions $P_{knowledge}(x)$ will exhibit enhanced and balanced performance across these domains, compared with a model fine-tuned with a non-matching data distribution. The paper's evidence is that aligning proportions to the detected distribution and then dynamically reweighting by learnable potential and forgetting degree consistently beats uniform and inverse distributions across seven open-source models at 7B-32B scale. The headline results are a 35.21% improvement in overall multi-ability performance versus uniform domain weights, and Qwen-2.5-32B + VersaTune surpassing GPT-4o, Claude3.5-Sonnet, and DeepSeek-V3 by 0.86%, 4.76%, and 4.60% on the paper's benchmarks. For flexible domain expansion, Statement 2 says that increasing one domain while shrinking the rest proportionally to the base knowledge distribution lets the model expand into a target domain while reducing damage to other domains; the paper measures a 38.77% reduction in non-target performance degradation compared with 100% single-domain fine-tuning.

Load-bearing premise

The load-bearing assumption is that the domain-knowledge distribution estimated by generating free-form samples from a bare $\langle BOS\rangle$ token and classifying them with a fine-tuned proprietary model is an accurate and meaningful prior for how much SFT data each domain should receive.

Editorial extensions

If this is right

  • Matched SFT proportions beat uniform and inverse distributions across every model tested, with the reported overall gain of 35.21% over uniform weights.
  • Dynamic reweighting contributes beyond knowledge matching: VersaTune's adaptive version outperforms a constant-weight version initialized from the same detected distribution in the ablation study.
  • For domain expansion, raising one domain's share while scaling the others down in proportion to detected knowledge reduces non-target degradation by 38.77% compared with training entirely on the target domain, with comparable target-domain gains.
  • On the paper's benchmark suite, Qwen-2.5-32B fine-tuned with VersaTune surpasses GPT-4o, Claude3.5-Sonnet, and DeepSeek-V3 by 0.86%, 4.76%, and 4.60%.
  • The inverse-distribution baseline underperforming uniform supports the paper's knowledge-consistency premise: mismatching the model's existing knowledge distribution hurts multi-domain fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test of the framework is to replace the $\langle BOS\rangle$ probe with task-oriented prompts from the actual evaluation distribution; if the estimated knowledge distribution shifts, the optimal SFT proportions would likely shift with it, which would let practitioners calibrate the probe to their deployment.
  • The same per-sample domain probabilities could be used for fine-grained data selection within a domain, weighting individual examples rather than whole domain blocks, a step the paper does not take.
  • The saturation pattern observed after epoch two or three in domain expansion suggests a per-domain early-stopping signal: stop raising a domain's share when its learnable potential approaches zero, and use that epoch as a checkpoint for final model selection.
  • The frontier-model comparison is only as broad as the benchmark suite; extending the same tuned 32B model to additional general-knowledge and reasoning benchmarks would show whether the measured edge over GPT-4o and Claude3.5-Sonnet generalizes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes VersaTune, a two-phase data-composition framework for supervised fine-tuning (SFT) of LLMs across multiple domains. Phase 1 estimates the base model's domain-knowledge distribution by generating free-form samples from a bare <BOS> token and classifying them with a proprietary Qwen2.5-72B-Instruct model fine-tuned on 120 hand-labeled examples. Phase 2 initializes SFT domain proportions to match this distribution and then dynamically adjusts them during training using 'learnable potential' and 'forgetting degree' signals computed against a lightweight reference model. The paper reports that VersaTune improves overall multi-domain performance by 35.21% over uniform weighting across seven open-source base models, that Qwen-2.5-32B + VersaTune outperforms frontier models by 0.86% to 4.60%, and that it reduces non-target-domain degradation by 38.77% in domain-expansion scenarios.

Significance. If the reported results are reproducible, VersaTune offers a practical and novel answer to the data-mixing question for multi-capability SFT, with the notable strength of being evaluated across seven base models and two distinct training scenarios. The paper includes a useful inverse-distribution baseline and an ablation of the dynamic-weighting component (VersaTune Constant), which are good experimental controls. The manuscript also provides detailed algorithm pseudocode and uses publicly available datasets. However, the central knowledge-distribution estimator is not validated, the headline percentage gains are not reproducible from the provided tables, and the frontier-model comparison is internally inconsistent. These issues currently prevent the paper's quantitative claims from being accepted at face value.

major comments (4)
  1. [Section 2.1.2, Algorithm 1, Appendix C.1] The domain-knowledge prior P is estimated from unconditional <BOS>-generated samples classified by a proprietary model fine-tuned on only 120 examples. No accuracy, calibration, or robustness check of this classifier is reported, and no evidence is provided that the distribution of unconditional generations matches the distribution of the SFT and evaluation data used downstream. Since Algorithms 2 and 3 initialize all data proportions from this estimate, the entire method inherits any bias in it. Section 7 concedes that the classifier cannot guarantee absolute accuracy, but the paper does not quantify this uncertainty. I request validation experiments, such as classification accuracy on a held-out known-mixture test set or a sensitivity analysis of final benchmark scores to perturbations of P.
  2. [Section 3.3, Tables 1 and 5] The headline 35.21% improvement over uniform distribution is not reproducible from the tables because no aggregation formula is given. Direct computation from Table 1 using reasonable choices (per-benchmark relative change averaged across models, or per-domain averages) does not yield 35.21% for any simple scheme. All results are single runs without error bars, so the magnitude of the improvement is not statistically grounded. The paper should state exactly how the 35.21% and 38.77% figures are computed and provide seed variance or confidence intervals.
  3. [Section 3.3 vs. Table 1 and Abstract] The claim that Qwen-2.5-32B + VersaTune surpasses frontier models by 0.86%, 4.76%, and 4.60% is inconsistent with the reported numbers. The text says the improvement is 'under medical scenarios,' but the MedQA and MedMCQA scores in Table 1 for VersaTune (78.72 and 72.36) are lower than those of GPT-4o (81.92 and 74.60) and DeepSeek-V3 (78.82 and 74.30). The abstract omits the qualifier and implies an overall multi-capability advantage. This discrepancy needs to be resolved, and the comparison basis (which benchmarks, which averaging) must be specified.
  4. [Section 2.2.1, Equations (2)-(3), Section 7] The learnable potential and forgetting degree are computed using a lightweight reference model (Sheared-LLaMA-1.3B or Qwen-1.5B) rather than the target model, and the paper acknowledges that this proxy may not reflect the target model's actual performance. Since these quantities drive the dynamic reweighting in Algorithms 2 and 3, the paper should provide evidence that the proxy's loss trends correlate with the target model's downstream benchmark performance, or at least test sensitivity to the choice of reference model.
minor comments (6)
  1. [Section 6] "noval" should be "novel".
  2. [Appendix E] The JSON output format in the prompt contains a typo: "Sciencee" instead of "Science".
  3. [Tables 1 and 5] Some numeric entries are concatenated without spacing (e.g., "75.5071.82 90.56"), making the tables hard to read.
  4. [Algorithms 1, 2, and 3] The symbol T is used for both the number of detection iterations in Algorithm 1 and the number of training steps in Algorithms 2 and 3; distinct symbols would avoid confusion.
  5. [Equation (5), Statement 2] The expression for A(D_j) for non-target domains is typeset in a confusing way and should be written more clearly, for example as a piecewise definition with explicit normalization.
  6. [Title and Section 3.1] The paper claims efficiency but does not report the computational cost of the knowledge-detection phase (40K generated samples, 5 iterations) nor the wall-clock time of the overall pipeline; a brief cost analysis would substantiate the 'efficient' claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core claim is an empirically tested hypothesis and the reported gains are external benchmark comparisons, not consequences of the method's definitions.

full rationale

VersaTune's central statement (Statement 1, Section 2.1.1) is a design hypothesis: fine-tuning with domain proportions aligned to an estimated pretrained knowledge distribution will improve multi-domain performance. The operationalization in Algorithm 1 estimates P_knowledge by generating unconditional <BOS> continuations and classifying them with a separately fine-tuned Qwen2.5-72B-Instruct model. This estimate is an input to Algorithms 2 and 3; it is not derived from the evaluation results. The headline improvements (35.21% overall gain, frontier-model comparisons in Table 1) are measured against uniform and inverse baselines on standard external benchmarks, so the claim is falsifiable and does not reduce by construction to the fitted prior. The learnable-potential update (Equation 2) compares the target model's loss to a reference model's per-domain loss and adjusts weights accordingly; this is a closed-loop reweighting heuristic, not a self-definitional cycle, because the evaluation is performed on held-out benchmarks rather than on the reference-loss values. The unvalidated accuracy of the domain classifier, acknowledged in Section 7 ('it cannot guarantee absolute accuracy in classification'), is a correctness and robustness concern, not a circularity: a biased prior would weaken the empirical claim, but the claim's support still comes from independent benchmark measurements. No load-bearing self-citations or imported uniqueness theorems are used. Overall, no circular step was found.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework introduces no new physical entities. Its load-bearing assumptions are empirical: the knowledge-consistency hypothesis, the validity of the detection pipeline, and the use of small-model losses as ceilings.

free parameters (5)
  • sigma (adjustment magnitude) = 0.5
    Set after experimenting with [0.1, 0.3, 0.5, 0.8, 1.0]; no results shown for the sweep, so this is chosen by hand.
  • delta (target domain increment) = 10% per training step
    Chosen based on overall domain weight distribution; not justified by a sweep.
  • epsilon (variation threshold) = 1
    Trade-off parameter in Algorithm 3, assigned without sensitivity analysis.
  • NS (generated samples) = 40,000
    Sample count for knowledge detection; chosen for statistical robustness.
  • T (detection iterations) = 5
    Number of repetitions for detection; variance reported as under 1.874%.
assumptions (5)
  • domain assumption Statement 1: P_SFT(x) approximately P_knowledge(x) improves multi-domain performance
    This is the core hypothesis, asserted without derivation in Section 2.1.1.
  • domain assumption Free-form generation from <BOS> reveals the model's domain knowledge distribution
    Algorithm 1 assumes that unconditional samples and their classifier scores represent the model's knowledge; no validation against a known mixture.
  • domain assumption The proprietary classifier MP (Qwen2.5-72B-Instruct fine-tuned on 120 hand-labeled samples) gives accurate domain probabilities
    The entire detection phase depends on this classifier, but its accuracy is not measured.
  • domain assumption A small reference model's per-domain loss is a valid mastery ceiling for the larger target model
    Section 2.2.1 defines learnable potential using reference losses; no justification that 1.3B-1.5B losses bound 7B-32B model losses.
  • domain assumption Domains are disjoint and comprehensive
    Six domains plus 'other'; the Alpaca general set is filtered to avoid overlap, but overlap may remain (e.g., code in science).

how reviews work

0 comments
Cite this review

Pith. "Pith review of VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMs." pith.science (2026). https://pith.science/paper/L547C2RH

@misc{pith2026241111266,
  author       = {Pith},
  title        = {Pith review of: VersaTune: An Efficient Data Composition Framework for Training Multi-Capability LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L547C2RH}},
  note         = {Machine review of arXiv:2411.11266}
}
read the original abstract

As demonstrated by the proprietary Large Language Models (LLMs) such as GPT and Claude series, LLMs have the potential to achieve remarkable proficiency across a wide range of domains, including law, medicine, finance, science, code, etc., all within a single model. These capabilities are further augmented during the Supervised Fine-Tuning (SFT) phase. Despite their potential, existing work mainly focuses on domain-specific enhancements during fine-tuning, the challenge of which lies in catastrophic forgetting of knowledge across other domains. In this study, we introduce **VersaTune**, a novel data composition framework designed for enhancing LLMs' overall multi-domain capabilities during training. We begin with detecting the distribution of domain-specific knowledge within the base model, followed by the training data composition that aligns with the model's existing knowledge distribution. During the subsequent training process, domain weights are dynamically adjusted based on their learnable potential and forgetting degree. Experimental results indicate that VersaTune is effective in multi-domain fostering, with an improvement of 35.21\% in the overall multi-ability performances compared to uniform domain weights. Furthermore, we find that Qwen-2.5-32B + VersaTune even surpasses frontier models, including GPT-4o, Claude3.5-Sonnet and DeepSeek-V3 by 0.86\%, 4.76\% and 4.60\%. Additionally, in scenarios where flexible expansion of a specific domain is required, VersaTune reduces the performance degradation in other domains by 38.77\%, while preserving the training efficacy of the target domain.

Figures

Figures reproduced from arXiv: 2411.11266 by the authors.

Figure 1
Figure 1. Overview of VersaTune. We begin by probing the knowledge distribution within the base model Mθ, utilizing a proprietary model MP to estimate the probability of sequences generated by Mθ belonging to various domains. Throughout the efficient fine-tuning process, we dynamically adjust the data domain ratios in response to Mθ’s real-time performance feedback, with learnable potential and forgetting degree serving as ev… view at source ↗
Figure 2
Figure 2. Performances of Qwen-2-7B on versatile tasks across different domains for multi-ability fostering. monitoring the learnable potential within each do￾main. In contrast, training with fixed domain weights exhibits certain fluctuations. A key rea￾son for this phenomenon is the distribution of do￾main knowledge mastered by the model changes during training, and the learning efficiency varies among domains. Therefore, dy… view at source ↗
Figure 3
Figure 3. Domain expansion for medicine domain. We evaluated checkpoints from each epoch. Left (a) presents the grouped stacked bar chart showing the growth or loss of capabilities in non-target domains compared to the pre-fine-tuning state. Within each group, the left, center, and right bars represent: (1) 100% specific domain fine-tuning, (2) domain increase with uniform distribution of remainder, and (3) VersaTune implemen… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: The average scores of models’ performances [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the LLMs training workflow. In [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Illustration of variations in models’ performance on non-target domain tasks when trained on a single [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: An illustration of the domain knowledge distribution among models. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Performances of different models on versatile benchmarks related to various domains under the scenario [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Performance comparison between Qwen-2.5-32B + Ver￾saTune and frontier models acorss multiple domains. Comparison with Frontier Mod￾els. Furthermore, to demonstrate the efficacy of VersaTune across diverse domain tasks, we conducted a com￾parative analysis between Qwen-…
Figure 10
Figure 10. Figure 10: Domain expansion results for the law domain, including non-target domains (a) and target domain (b). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Domain expansion results for finance domain, including non-target domains (a) and target domain (b). Domain Increase with Uniform [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Domain expansion results for science domain, including non-target domains (a) and target domain (b). Domain Increase with Uniform [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Domain expansion results for the code domain, including non-target domains (a) and target domain (b). 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Line chart of the multi-task performances of models across different domains during the domain [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A knowledge-graph-guided method that scores an LLM's knowledge gaps and generates atomic, aggregated, and multi-hop QA pairs, improving closed-book QA after fine-tuning.

Reference graph

Works this paper leans on

70 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, and 1 others. 2024. A survey on data selection for language models. arXiv preprint arXiv:2402.16827

    arXiv 2024

  3. [3]

    Anthropic. 2024. https://www.anthropic.com/ Claude . Accessed: 2024-06-27

    work page 2024

  4. [4]

    Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. 2024. Generalization vs memorization: Tracing language models' capabilities back to pretraining data. arXiv preprint arXiv:2407.14985

    arXiv 2024

  5. [5]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  6. [6]

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2023. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631

    arXiv 2023

  7. [7]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  8. [8]

    Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676

    arXiv 2019

Show all 70 references
  1. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  2. [10]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  3. [11]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  4. [12]

    Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092

    2023 arXiv

  5. [13]

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale s Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3...

    2021

  6. [14]

    Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  7. [15]

    Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. 2022. https://doi.org/10.1109/CVPR52688.2022.00707 Source-free domain adaptation via distribution estimation . In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7202--7212

    2022

  8. [16]

    Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. 2023. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492

    2023 arXiv

  9. [17]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  10. [18]

    Yogesh K Dwivedi, Laurie Hughes, Elvira Ismagilova, Gert Aarts, Crispin Coombs, Tom Crick, Yanqing Duan, Rohita Dwivedi, John Edwards, Aled Eirug, and 1 others. 2021. Artificial intelligence (ai): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda...

    2021

  11. [19]

    Simin Fan, Matteo Pagliardini, and Martin Jaggi. 2023. Doge: Domain reweighting with generalization estimation. arXiv preprint arXiv:2310.15393

    2023 arXiv

  12. [20]

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289

    2023 arXiv

  13. [21]

    Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. 2024. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904

    2024 arXiv

  14. [22]

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher R \'e , Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2024. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. A...

    2024

  15. [23]

    Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. 2024. Data mixture inference: What do bpe tokenizers reveal about their training data? arXiv preprint arXiv:2407.16607

    2024 arXiv

  16. [24]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  17. [25]

    Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. 2022. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(11s):1--37

    2022

  18. [26]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  19. [27]

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081

    2020 arXiv

  20. [28]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, and 1 others. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  21. [29]

    Prakhar Kaushik, Alex Gain, Adam Kortylewski, and Alan Yuille. 2021. Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping. arXiv preprint arXiv:2102.11343

    2021 arXiv

  22. [30]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664

    2023 arXiv

  23. [31]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processin...

    2022

  24. [32]

    Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. 2023 a . Trocr: Transformer-based optical character recognition with pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37,...

    2023

  25. [33]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023 b . Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance, pages 374--382

    2023

  26. [34]

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

    2024 arXiv

  27. [35]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024 b . Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36

    2024

  28. [36]

    Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896

    2023 arXiv

  29. [37]

    Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109--165. Elsevier

    1989

  30. [38]

    Xiaonan Nie, Yi Liu, Fangcheng Fu, Jinbao Xue, Dian Jiao, Xupeng Miao, Yangyu Tao, and Bin Cui. 2023. Angel-ptm: A scalable and economical large-scale pre-training system in tencent. Proceedings of the VLDB Endowment, 16(12):3781--3794

    2023

  31. [39]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248--260. PMLR

    2022

  32. [40]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  33. [41]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  34. [42]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, and 1 others. 2021. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207

    2021 arXiv

  35. [43]

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and 1 others. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172--180

    2023

  36. [44]

    Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. 2024. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19053--19061

    2024

  37. [45]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  38. [46]

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085

    2022 arXiv

  39. [47]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  40. [48]

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine, 29(8):1930--1940

    2023

  41. [49]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  42. [50]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  43. [51]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions

    2022

  44. [52]

    Zige Wang, Wanjun Zhong, Yufei Wang, Qi Zhu, Fei Mi, Baojun Wang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Data management for large language models: A survey. arXiv preprint arXiv:2312.01700

    2023 arXiv

  45. [53]

    Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. 2025. Grit: A generative region-to-text transformer for object understanding. In European Conference on Computer Vision, pages 207--224. Springer

    2025

  46. [54]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564

    2023 arXiv

  47. [55]

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint arXiv:2310.06694

    2023 arXiv

  48. [56]

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, and 1 others. 2024 a . The finben: An holistic financial benchmark for large language models. arXiv preprint arXiv:2402.12659

    2024 arXiv

  49. [57]

    Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy S Liang, Quoc V Le, Tengyu Ma, and Adams Wei Yu. 2024 b . Doremi: Optimizing data mixtures speeds up language model pretraining. Advances in Neural Information Processing Systems, 36

    2024

  50. [58]

    Zhuoyan Xu, Zhenmei Shi, and Yingyu Liang. 2024. Do large language models have compositional ability? an investigation into limitations and scalability. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models

    2024

  51. [59]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, and 1 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  52. [60]

    Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. 2022. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093--3106

    2022

  53. [61]

    Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2022. Hype: Better pre-trained language model fine-tuning with hidden representation perturbation. arXiv preprint arXiv:2212.08853

    2022 arXiv

  54. [62]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  55. [63]

    Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024. Sciglm: Training scientific language models with self-reflective instruction annotation and tuning. arXiv preprint arXiv:2401.07950

    2024 arXiv

  56. [64]

    Liwen Zhang, Weige Cai, Zhaowei Liu, Zhi Yang, Wei Dai, Yujie Liao, Qianru Qin, Yifei Li, Xingyu Liu, Zhiqiang Liu, and 1 others. 2023. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models. arXiv preprint arXiv:2308.09975

    2023 arXiv

  57. [65]

    Xuanyu Zhang and Qing Yang. 2023. Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters. In Proceedings of the 32nd ACM international conference on information and knowledge management, pages 4435--4439

    2023

  58. [66]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others. 2023 a . A survey of large language models. arXiv preprint arXiv:2303.18223

    2023 arXiv

  59. [67]

    Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Chong Meng, Shuaiqiang Wang, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2023 b . Knowing what llms do not know: A simple yet effective self-detection method. arXiv preprint arXiv:2310.17918

    2023 arXiv

  60. [68]

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364

    2023 arXiv

  61. [69]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  62. [70]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.