Pith. sign in

REVIEW 4 major objections 6 minor 4 cited by

Baichuan4-Finance Technical Report

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A finance-specialized LLM beats GPT-4o on Chinese certification exams by using a KL-constrained training objective that keeps its general knowledge intact.

desk verdict A competent engineering report whose headline financial benchmark numbers are undermined by an unaddressed contamination risk, since the pretraining corpus explicitly includes exam questions and the benchmarks are the same kind of certification exams. read the letter →

arxiv 2412.15270 v2 pith:OZJBRPAJ submitted 2024-12-17 cs.CL cs.AIcs.CEcs.CYcs.LG

Hanyu Zhang , Boyu Qiu , Yuhao Feng , Shuqi Li , Qian Ma , Xiyuan Zhang , Qiang Ju , Dong Yan
show 1 more author
Jian Xie
This is my paper · ORCID
classification cs.CLcs.AIcs.CEcs.CYcs.LG
keywords financiallargelanguagemodelcontinualpre-trainingdomainself-constraintscalinglawdatamixtureRLHFFinanceIQFLAMEcertificationbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports the development of Baichuan4-Finance, a pair of financial large language models built by continuing pre-training a general-purpose base model on a 500-billion-token mixture of general and financial data. The central technical claim is a 'domain self-constraint' training objective: on general documents the model is trained to keep its next-token distribution close to the original base model, while on financial documents it combines that constraint with standard language-modeling loss. The authors claim this lets the base model absorb financial knowledge without losing general ability, and that the instruction-tuned chat model sets new state-of-the-art accuracy on two Chinese financial certification benchmarks, FinanceIQ and FLAME-Cer, beating GPT-4o and strong open-source models. The result matters because it offers a general recipe for injecting specialized knowledge into an LLM while preserving its general competence, which is exactly the trade-off that usually makes domain adaptation costly.

What carries the argument

The central object is the domain self-constraint continual pre-training objective. For a training document $x$, if the document is general data, the loss is the per-token KL divergence between the new model's next-token distribution $P_{\theta_{\mathrm{fin}}}(x_t \mid x_{<t})$ and the reference base model's distribution $P_{\theta_{\mathrm{ref}}}(x_t \mid x_{<t})$; if the document is financial data, the loss is $\alpha \cdot \mathrm{KL} + \mathcal{L}_{\mathrm{lm}}$, where $\mathcal{L}_{\mathrm{lm}}$ is the standard negative log-likelihood. This lets the model memorize and reason about financial material while actively resisting drift of the general-language distribution. A second piece of machinery is the two-stage scaling-law procedure (D-CPT Law plus a loss-to-accuracy mapping) used to pick the 37-source financial data mixture ratio at limited training cost.

What would settle it

Conduct a near-duplicate search (for example, 13-gram overlap after normalization) between the FinanceIQ and FLAME-Cer question sets and the financial pre-training corpus. If a non-trivial fraction of benchmark questions appear verbatim or near-verbatim in the corpus, the accuracy numbers are invalid as a comparison of reasoning ability. A second check: compare accuracy on questions first published after the corpus was collected with accuracy on older questions; a large gap would indicate memorization of training data rather than learned financial competence.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that continual pre-training for a specialized domain does not have to trade away general knowledge. The proposed domain self-constraint objective applies a KL-divergence penalty between the fine-tuned model and the frozen reference base model on general-data documents, and applies the same penalty together with the ordinary log-likelihood loss on financial documents. Trained this way, Baichuan4-Finance-Base outperforms nearly all baselines on the FinanceIQ and FLAME financial benchmarks while scoring comparably to its backbone on C-Eval, CMMLU, MMLU, GSM8k-ZH, and HumanEval. After supervised fine-tuning and a reward-robust PPO alignment stage, the chat model Baichuan4-Finance reaches 93.62 average accuracy on FLAME-Cer and 79.23 average on FinanceIQ, both above GPT-4o and the open-source models evaluated.

Load-bearing premise

The reported accuracy comparisons are only meaningful if the FinanceIQ and FLAME-Cer exam questions were not part of the 100-billion-token financial pre-training corpus, even though the corpus is described as containing examination questions.

Editorial extensions

If this is right

  • Domain-specific continual pre-training can be done with a KL constraint on general data and a mixed objective on domain data, reducing the catastrophic-forgetting penalty that normally comes with adding specialized knowledge.
  • The reported results on FinanceIQ and FLAME-Cer indicate that a chat model trained this way can outperform general-purpose proprietary and open-source models on Chinese financial certification exams.
  • The scaling-law-based data mixture selection provides a cost-effective way to set the training data ratio for many domain sources, rather than relying on heuristic weights.
  • The PPO ablation suggests that reinforcement learning moves the model's per-inference hit rate toward what the model can already achieve with multiple samples, rather than adding entirely new knowledge.
  • The FLAME-Sce results suggest the same recipe carries over to real-world financial applications such as compliance, document generation, and risk control, not just exam questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the KL-constrained objective works as described, it should transfer to other high-stakes domains like medicine or law, where preserving general language competence during domain adaptation is equally important; a testable extension would be applying the same objective to an English legal or medical corpus and measuring general benchmarks.
  • The reported accuracy gap between Baichuan4-Finance and GPT-4o is on Chinese-language certification benchmarks; on English or multilingual financial benchmarks the ordering may differ, since the training corpus is heavily Chinese.
  • The contamination concern cuts both ways: a near-duplicate scan between the financial corpus and the exam benchmarks would settle whether the gains reflect genuine financial reasoning or memorization, and the paper does not report such a scan.
  • The scaling-law mixture selection was tuned for validation loss and benchmark accuracy; it might be extended to predict other properties such as calibration or robustness, which matter for real financial deployment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports the development of Baichuan4-Finance, a financial LLM series built by continual pre-training of Baichuan4-Turbo on a mixed general/financial corpus, using a proposed 'domain self-constraint' training objective intended to preserve general knowledge. The resulting base and chat models are evaluated on general benchmarks (C-Eval, MMLU, GSM8K, etc.) and on two Chinese financial certification benchmarks, FinanceIQ and FLAME. The central claim is that Baichuan4-Finance-Base surpasses competitive baselines on financial tasks by significant margins without losing general capability, and that the chat model Baichuan4-Finance achieves state-of-the-art accuracy on FinanceIQ and FLAME-Cer, outperforming GPT-4o and strong open-source models.

Significance. If the reported results are trustworthy, the paper offers a useful engineering recipe for financial LLM continual pre-training, including a data-quality pipeline, a scaling-law-based mixture-ratio selection, and an alignment pipeline with human and AI feedback. The domain self-constraint objective is a plausible mechanism for avoiding catastrophic forgetting. However, the central empirical claims are currently not credible because the pre-training corpus explicitly includes examination questions (Section 3.3) while the headline benchmarks are certification exams (Section 5.1), and no contamination analysis is reported. The paper also relies on externally quoted baseline numbers for FLAME without verification. These issues undermine the validity of the claimed SOTA margins.

major comments (4)
  1. [Section 3.3, Section 5.1, Table 1] The pre-training data composition described in Section 3.3 includes 'examination questions' among the 37 financial data sources, while the main benchmarks FinanceIQ and FLAME-Cer consist of Chinese financial certification exam questions (Section 5.1). The paper reports no n-gram or semantic overlap analysis, no deduplication against evaluation data, and no held-out exam set. Consequently, the large margins reported in Table 1 (e.g., FLAME-Cer average 93.62 vs. 81.17 for Qwen2.5-72B-Instruct) could be inflated by memorization rather than genuine capability. This is a load-bearing issue for the abstract's claim of 'significant margins' and must be addressed with a rigorous contamination study before the results can be interpreted.
  2. [Section 5.3.1, Tables 1 and 2] The baselines for the FLAME benchmark are not evaluated by the authors; Section 5.3.1 states that the paper 'presents the evaluation results of the official institute' for GPT-4o, ERNIE-4.0-Turbo-128K, GLM-4-PLUS, Qwen2.5-72B-Instruct, and XuanYuan3-70B. No details are given about the evaluation protocol, prompting format, or how these numbers were obtained, making them unverifiable and potentially inconsistent with the zero-shot protocol used for Baichuan4-Finance. The authors should either run these baselines themselves under identical conditions or provide the exact evaluation setup and release the raw outputs for verification.
  3. [Section 5.2, Tables 1-3] All evaluation results are reported as single accuracy numbers without error bars, confidence intervals, or significance tests. Some differences are small (e.g., Table 3, CAA: Baichuan4-Finance and GPT-4o both score 37.50; Table 2, 'Analysis and Research': Baichuan4-Finance 45.45 vs. GPT-4o 45.45), so the sweeping claim of 'significant margins' is not statistically supported. The authors should provide multiple evaluation runs (e.g., different seeds or sample-based bootstrap) and report significance or at least variance, especially for the certification subcategories with small sample sizes.
  4. [Section 3.3, Eq. (2)] The scaling-law model in Eq. (2) has eight free parameters per source ({E_i, A_i, B_i, C_i, alpha_i, beta_i, gamma_i, eta_i}) and is applied to n=37 sources, yielding 296 parameters. The paper does not report how many small-model runs were used to fit these parameters, nor does it validate the fitted law on held-out mixture ratios or at the actual target model scale. Without such validation, the predicted optimal mixture ratio is at risk of overfitting, and the claim that the law 'predicts the downstream performance of arbitrary data mixture ratios' is not established.
minor comments (6)
  1. [Table 3] The row labeled 'FindPQ' should be 'FundPQ' (Fund Practitioner Qualification) to be consistent with Section 5.1 and Table 1.
  2. [Table 1] 'Ecomonist' is a misspelling of 'Economist' in the last certification row.
  3. [Section 3.4] The sentence 'we sample 200 probabilities of the reference model for each token' is unclear; it likely means sampling 200 tokens or using a Monte Carlo estimate of the KL divergence, but the wording should be clarified.
  4. [Section 5.2.2] The pass@5 evaluation for multiple-choice questions is ambiguous: if the model generates a single answer per run, then pass@5 over five independent runs is a different metric than typical pass@k for code generation. Please define how multiple answers are aggregated and whether the benchmark protocol supports this.
  5. [Figures 2-5] The radar charts and bar figures lack axis labels and legends in the text; this makes it difficult to interpret the ablation results, especially the quantitative scale of the differences in Figure 5.
  6. [References] The citation for RoPE (Su et al., 2023) is incomplete and appears as a mixture of DOI and title; please provide the full reference.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the training-evaluation chain is not self-referential, though the unexamined overlap between pretraining 'examination questions' and public certification benchmarks is a correctness risk.

full rationale

The report's derivation chain is empirical: a base model is continually pretrained with a KL-regularized objective, aligned with SFT/RLHF, and then evaluated on public benchmarks. No claimed result is an input to its own derivation by construction. The scaling-law mixture selection (Sec. 3.3) fits per-source parameters to small-model losses and benchmark accuracies and then uses the fitted surface to choose the data ratio; the final model's scores are measured after training, not read off the fitted curve, so this is a hyperparameter-selection procedure rather than a circular prediction. The domain self-constraint objective (Eqs. 3-5) combines a KL term with a standard LM loss; its effectiveness is demonstrated by ablations, not assumed. Self-citations (Yan et al. 2024; Li et al. 2024b) supply training methods but are not load-bearing for the headline performance claims. The strongest validity threat is benchmark contamination: Sec. 3.3 lists 'examination questions' among the 37 financial data sources while the headline results use public certification-exam benchmarks (FinanceIQ, FLAME-Cer), and no contamination analysis is reported. That is a serious external-validity risk, but it is not a circularity of the derivation chain because the paper never states that the specific benchmark items are in the pretraining corpus, nor does any equation reduce the reported accuracy to a training input. Under the hard rule requiring a quoted reduction, this does not qualify as a circular step.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The main claim depends on benchmark cleanliness, external baseline comparability, and scaling-law extrapolation. The first is likely violated by the inclusion of exam questions in the training data, and the others are unverified.

free parameters (3)
  • alpha (Eq. 4)
    Hyperparameter balancing KL loss and language modeling loss in the domain self-constraint objective; value not reported.
  • Scaling law parameters {E_i, A_i, B_i, C_i, alpha_i, beta_i, gamma_i, eta_i} per source = Fitted values not reported
    37 financial data sources each have 8 fitted parameters to predict validation loss; these are fit to small-scale experiments and extrapolated to the full model.
  • Pareto alpha in Eq. (1)
    Shape parameter for the Pareto distribution used in quality filtering; value not reported.
assumptions (4)
  • domain assumption The finance benchmarks (FinanceIQ, FLAME) are clean and not present in the pre-training corpus.
    The paper does not perform a contamination check, and Section 3.3 lists 'examination questions' as part of the training data, making this assumption likely false.
  • domain assumption The FLAME baseline results quoted from the 'official institute' were computed under the same zero-shot evaluation setting as Baichuan4-Finance.
    Section 5.3.1 says the baseline results are from the official FLAME evaluation, not run by the authors, so comparability is unverified.
  • domain assumption Baichuan4-Turbo is a capable base model, and the continual pre-training does not degrade general capabilities.
    The paper's general benchmark results show comparable performance, but only on a small set of tasks (C-Eval, CMMLU, MMLU, GSM8K ZH, HumanEval, math subsets).
  • ad hoc to paper The scaling laws accurately predict performance at full scale.
    The scaling law models (Eq. 2) are fit to small models with many free parameters; there is no published validation that the extrapolation holds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Baichuan4-Finance Technical Report." pith.science (2026). https://pith.science/paper/OZJBRPAJ

@misc{pith2026241215270,
  author       = {Pith},
  title        = {Pith review of: Baichuan4-Finance Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OZJBRPAJ}},
  note         = {Machine review of arXiv:2412.15270}
}
read the original abstract

Large language models (LLMs) have demonstrated strong capabilities in language understanding, generation, and reasoning, yet their potential in finance remains underexplored due to the complexity and specialization of financial knowledge. In this work, we report the development of the Baichuan4-Finance series, including a comprehensive suite of foundational Baichuan4-Finance-Base and an aligned language model Baichuan4-Finance, which are built upon Baichuan4-Turbo base model and tailored for finance domain. Firstly, we have dedicated significant effort to building a detailed pipeline for improving data quality. Moreover, in the continual pre-training phase, we propose a novel domain self-constraint training strategy, which enables Baichuan4-Finance-Base to acquire financial knowledge without losing general capabilities. After Supervised Fine-tuning and Reinforcement Learning from Human Feedback and AI Feedback, the chat model Baichuan4-Finance is able to tackle various financial certification questions and real-world scenario applications. We evaluate Baichuan4-Finance on many widely used general datasets and two holistic financial benchmarks. The evaluation results show that Baichuan4-Finance-Base surpasses almost all competitive baselines on financial tasks by significant margins without sacrificing performance on general LLM benchmarks. At the same time, Baichuan4-Finance demonstrates even more impressive performance on financial application scenarios, showcasing its potential to foster community innovation in the financial LLM field.

Figures

Figures reproduced from arXiv: 2412.15270 by the authors.

Figure 1
Figure 1. The composition of financial data for Baichuan4-Finance-Base pre-training. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Here we put a case on a 1B model to explore the validation of the proposed domain [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison across Baichuan4-Finance-Base and its backbone Baichuan4- [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: An experiment to demonstrate the effectiveness of PPO focusing on models’ mathematical [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: An experiment to explore the mechanism of how PPO takes effect. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MedCollab: IBIS-Guided Multi-Agent Collaboration with Hierarchical Disease Relation Chains for Clinical Diagnosis

    cs.MA 2026-03 conditional novelty 6.0 of 10

    A structured multi-agent framework with IBIS argumentation and disease relation chains outperforms standard LLM and medical-agent baselines on full-cycle clinical diagnosis.

  2. Baichuan-M2: Scaling Medical Capability with Large Verifier System

    cs.LG 2025-09 conditional novelty 5.0 of 10

    Baichuan-M2, a 32B medical LLM trained with a patient simulator and a clinical rubric generator as RL verifiers, reports state-of-the-art HealthBench scores (60.1 overall, 34.7 hard), ahead of all open-source models.

  3. Agentar-Fin-R1: Enhancing Financial Intelligence through Domain Expertise, Training Efficiency, and Advanced Reasoning

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Agentar-Fin-R1, an 8B and 32B financial LLM family, reports top scores on FinEval, FinanceIQ, and a new Finova benchmark while keeping general reasoning near its Qwen3 base.

  4. RiverEcho: Real-Time Interactive Digital System for Ancient Yellow River Culture

    cs.MM 2025-06 conditional novelty 4.0 of 10

    The authors built a voice-interactive digital human system for ancient Yellow River culture with a curated 20,000-segment knowledge base and showed that RAG improves answer quality.

Reference graph

Works this paper leans on

25 extracted references · 22 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [3]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609,

    arXiv

  3. [4]

    On the resemblance and containment of documents

    Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pp. 21–29. IEEE,

    1997

  4. [8]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  5. [9]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300,

    arXiv 2009

  6. [10]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  7. [11]

    A survey of large language models in finance (finllms)

    Jean Lee, Nicholas Stevens, Soyeon Caren Han, and Minseok Song. A survey of large language models in finance (finllms). arXiv preprint arXiv:2402.02315,

    arXiv

  8. [12]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 2024a. Jialian Li, Yipin Zhang, Wei Shen, Yuzi Yan, Jian Xie, and Dong Yan. ...

    arXiv

Show all 25 references
  1. [14]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  2. [16]

    org/10.1016/j

    DOI: https://doi. org/10.1016/j. neucom,

    doi

  3. [17]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

    arXiv

  4. [18]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi`ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    arXiv

  5. [19]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    arXiv

  6. [22]

    Pixiu: A large language model, instruction data and evaluation benchmark for finance

    Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443,

    arXiv

  7. [23]

    Reward-robust rlhf in llms

    Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, and Yuan Shen. Reward-robust rlhf in llms. arXiv preprint arXiv:2409.15360,

    arXiv

  8. [24]

    Baichuan 2: Open large-scale language models

    Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305, 2023a. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng...

    arXiv

  9. [25]

    Fingpt: Open-source financial large language models

    16 Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023b. Yi Yang, Yixuan Tang, and Kar Yan Tam. Investlm: A large language model for investment using financial domain instruction tunin...

  10. [1997]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374,

    arXiv

  11. [2017]

    When flue meets flang: Benchmarks and large pre-trained language model for financial domain

    Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. When flue meets flang: Benchmarks and large pre-trained language model for financial domain. arXiv preprint arXiv:2211.00083,

    arXiv

  12. [2019]

    Bloomberggpt: A large language model for finance

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhan- jan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564,

    arXiv

  13. [2020]

    Ccnet: Extracting high quality monolingual datasets from web crawl data

    Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzm´an, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359,

    1911 arXiv

  14. [2021]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132,

    arXiv

  15. [2022]

    D-cpt law: Domain-specific continual pre-training scaling law for large language models

    Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, et al. D-cpt law: Domain-specific continual pre-training scaling law for large language models. arXiv preprint arXiv:2406.01375,

    arXiv

  16. [2023]

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr´on, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245,

    arXiv

  17. [2024]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.