Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Continual Pre-Training is (not) What You Need in Domain Adaption

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Domain-adaptive continual pre-training of legal LLMs improves some tasks, hurts others.

desk verdict Honest, expensively produced negative-result paper whose title overclaims: no ablation isolates the legal CPT stage, so the causal claim outruns the design, but the descriptive results and benchmark are worth a serious look. read the letter →

arxiv 2504.13603 v1 pith:YKEKPBNF submitted 2025-04-18 cs.CL

Pin-Er Chen , Da-Chen Lian , Shu-Kai Hsieh , Sieh-Chuen Huang , Hsuan-Lei Shao , Jun-Wei Chiu , Yang-Hsien Lin , Zih-Ching Chen
show 3 more authors
Cheng-Kuang Eddie TC Huang Simon See
This is my paper · ORCID
classification cs.CL
keywords domain-adaptivecontinualpre-traininglegallargelanguagemodelsTaiwaneselawinstructiontuningpreferenceoptimizationDPOORPOreasoningbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether Domain-Adaptive Continual Pre-Training (DACP) — continued pre-training of a general LLM on legal text — improves legal reasoning. Testing on a new Traditional-Chinese legal benchmark with four task types from Taiwan's legal framework, the authors find that DACP increases some domain-specific knowledge but does not uniformly beat the base model, and can lower performance on prompt-based and general tasks. They also report that preference optimization via DPO and ORPO failed to improve their target tasks, with Task C accuracy falling from 53.07 to 43.94 and 45.67 respectively. The paper's contribution is empirical evidence that the common assumption that more domain pre-training is unconditionally beneficial does not hold for legal LLMs.

What carries the argument

The engine of the study is a three-stage training pipeline: first, domain-adaptive continual pre-training on Taiwanese legal texts, German legal texts (reflecting the Continental-law influence on Taiwan), and self-curated reasoning data; second, full-parameter instruction tuning on general and legal instruction data; third, preference alignment via DPO or ORPO. The evaluation is a new benchmark with four tasks: single/multiple-choice legal questions (Tasks A and B), argument-based decision-making in legal symposia (Task C), and essay questions graded by GPT-4o with a Gutachten-style rubric (Task D). The load-bearing comparison is Table 3, which pits the LLAWA models against their base models and two LoRA-tuned baselines.

What would settle it

Train the same base model (Llama-3-TAIDE-LX-8B-Chat-Alpha1) with and without the legal continual pre-training stage, then apply the identical instruction-tuning recipe to both; if the version without legal CPT matches or beats the version with it on Tasks A–C, the paper's central claim about DACP's trade-off would be falsified. A weaker check: run DPO with alternative preference data and hyperparameters to see whether the accuracy drop on Task C persists.

Watch

Extended reading notes

Core claim

The central claim is that DACP is a trade-off, not a free lunch: it may improve certain types of domain-specific reasoning while simultaneously diminishing performance on prompt-based tasks and other generalization capabilities. In the reported experiments, the instruction-tuned LLAWA-TCxYZL-Instruct model performs only slightly better than its TAIDE base on Task A (28.55 vs 24.92) and equally on Task B (33.47 vs 33.47), and preference alignment makes things worse: DPO drops Task C accuracy from 53.07 to 43.94 and ORPO to 45.67. The paper concludes that hybrid approaches combining DACP with task-specific fine-tuning or meta-learning may be needed, and that evaluation benchmarks should capture task-specific trade-offs rather than a single aggregate score.

Load-bearing premise

The paper treats Table 3 as evidence about the effect of legal continual pre-training, but the design never isolates that stage: the main model differs from every comparison in base model, tuning method, and presence of the legal-CPT stage at once, so something else could be driving the flat results.

Editorial extensions

If this is right

  • If the claim holds, organizations cannot assume that adding legal-domain pre-training will improve legal AI; task-specific evaluation is required before deployment.
  • Preference optimization methods like DPO and ORPO need careful validation on legal tasks, because in these experiments they actively reduced multiple-choice accuracy.
  • The choice of base model matters as much as the domain pre-training: TAIDE's prior Traditional-Chinese pre-training may already supply much of the benefit.
  • The observed trade-off between domain-specific reasoning and prompt-based generalization should be a first-class consideration in any domain-adaptation pipeline.
  • Benchmarks that include argumentation and essay tasks, not just multiple choice, reveal effects that simpler benchmarks miss.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's title claim is broader than what the experiment can prove: no run isolates the legal continual-pre-training stage, because the base model, tuning method, and dataset all differ between the LLAWA row and the comparison rows. An ablation that removes only the legal-CPT step from an otherwise identical recipe would be needed to confirm that DACP itself causes the observed pattern.
  • A plausible reading of the DPO/ORPO failure is that the preference dataset construction (ground truth as chosen, model output as rejected) and hyperparameters rather than preference optimization per se; a different recipe might show gains.
  • The GPT-4o-based essay grading, with its acknowledged LLM-judge bias, could be masking true legal-quality differences; human expert scoring on a subset would be a cheaper robustness check.
  • If the trade-off is real, a testable extension is to vary the mixture ratio of legal to general corpus during DACP, as the paper itself notes is unexplored.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper examines whether Domain-Adaptive Continual Pre-Training (DACP) improves legal reasoning for Taiwanese-Mandarin legal tasks. The authors train LLAWA by continually pre-training the TAIDE Llama-3-8B base on legal corpora, then instruction-tuning it; they additionally train LoRA-based baselines (BLLAWA from Meta-Llama-3-8B-Instruct and BLAWSTRAL from Mistral-Nemo-Instruct-2407) and experiment with DPO and ORPO. They construct four evaluation tasks from Taiwanese bar and judicial exam materials and a legal symposium dataset, reporting accuracy on Tasks A–C and LLM-judged essay scores on Task D. The main conclusion is that while DACP may improve some domain-specific abilities, it does not uniformly improve all legal tasks, and preference optimization actually degrades performance on the targeted tasks. The manuscript argues that continual pre-training is therefore 'not what you need' for all legal tasks, and it discusses trade-offs between domain knowledge and generalization.

Significance. If the central claim were fully supported, the paper would be a valuable counterpoint to the prevailing assumption that domain-adaptive continual pre-training reliably helps legal LLMs, particularly in a non-Anglo-American legal system. The authors contribute a new benchmark for Taiwanese legal reasoning, transparent reporting of training hyperparameters and data sources, and a candid discussion of evaluation pitfalls such as LLM-judge bias and data contamination. The paper also makes a useful empirical observation that DPO and ORPO, in their setup, reduced accuracy on Tasks A–C. However, the headline causal claim about DACP is not yet established because the experimental design never isolates the DACP stage from other variables, so the significance is conditional on a missing control experiment. The work is publishable as a contribution to the empirical literature only after that design gap is addressed.

major comments (4)
  1. [Sections 3.2–3.4, Table 3] The paper's central claim that DACP 'does not uniformly improve performance across all legal tasks' is not supported by the current comparisons. LLAWA-TCxYZL-Instruct differs from every other row in Table 3 in at least two of three variables: base model, presence of the legal CPT stage, and instruction-tuning recipe. Compared with Llama-3-TAIDE-LX-8B-Chat-Alpha1, it changes both the legal CPT stage and the one-stage instruction tuning; compared with BLLAWA or Blawstral, it changes the base model and the full-parameter versus LoRA recipe. Without a same-base, same-recipe ablation that removes only the legal CPT stage, the flat or slightly improved Task A–C numbers cannot be causally attributed to DACP; they could equally reflect the choice of the TAIDE base or the full-parameter tuning recipe. The authors should add a control model trained from the TAIDE base with the identical instruction-tuning recipe (Taiwan Chat plus legal data, two epochs) but without the legal pre-training stage, and report it in Table 3.
  2. [Section 4.3, Table 3 footnote, Figure 2] The Task D evaluation is described inconsistently and its validity is load-bearing for the paper's generalization trade-off claim. Section 4.3 states that GPT-4-Turbo segments answers into four chunks and that 'these segmented answers are then evaluated by legal experts from law school,' but the following paragraph and the Table 3 footnote say that GPT-4o evaluates similarity and legal rationality against the golden segmented answers. Moreover, the reference answers were produced by GPT-4-Turbo, one of the evaluated models, and Section 6 acknowledges that using LLMs to evaluate other LLMs 'introduces potential biases.' This makes the Task D scores self-referential in part and uninterpretable as a measure of legal quality. The authors should clarify the exact pipeline (who segments, who grades, and with what prompt) and either provide human-expert scores or explicitly label Task D as an exploratory LLM-as-judge experiment rather than a claim about legal reasoning ability.
  3. [Abstract, Sections 4 and 5] The abstract and conclusion assert that DACP 'enhances domain-specific knowledge,' but the paper never directly measures domain knowledge. No knowledge probe, fact-retrieval test, or legal-terminology assessment is reported; the only evidence is task accuracy, which is also the variable used to show the lack of uniform improvement. Since the paper's takeaway is a trade-off between domain-specific gains and generalization or prompt-based ability, the positive side of that trade-off needs an independent measurement. Without a direct knowledge probe, the claim that DACP improved legal knowledge while hurting other skills is an inference from a design that, as noted above, does not isolate DACP.
  4. [Section 6, Tasks C and D] The limitation section acknowledges potential data contamination but does not attempt to measure overlap between the evaluation data and the training corpora. This matters especially for Tasks C and D, whose public sources (Taiwan High Court website, bar and judicial exams) are exactly the kind of web-scraped legal text that plausibly appears in pretraining or in the legal CPT corpus. The authors should report a contamination analysis (e.g., n-gram overlap between the evaluation instances and the pretraining data) or, if overlap is found, re-evaluate on a fresh or time-shifted sample. This is not a reason to reject the paper, but it is a required check for the benchmark claims.
minor comments (6)
  1. [Title and Abstract] The title uses 'Domain Adaption' while the text uses 'Adaptation,' and the abstract contains a stray space in 'Y ou'; these should be corrected.
  2. [Section 3.3] The model name is written inconsistently as 'LLawa,' 'Llawa,' and 'LLAWA'; the authors should pick one spelling and use it throughout.
  3. [Section 4.3] There is a typo in 'Eergebnis' (should be 'Ergebnis'), and the four German chunk labels should be given uniformly with consistent capitalization.
  4. [Table 3] Task D entries for LLAWA-TC-YZL-Instruct and LLAWA-TCxYZL-Instruct are missing without explanation; a footnote should state why these models were not evaluated on the essay task.
  5. [Section 4.2] Task C outputs are free-form final arguments, but the paper writes that accuracy 'focuses solely on comparing the final conclusion' without specifying how the conclusion was extracted from each model's output or how duplicate or partial conclusions were scored; this should be operationalized.
  6. [Section 3.4] The preference optimization data are described only as 'the ground truth as the preferred output and the model's generated output as the rejected one'; the authors should report the number of preference pairs, the source of the generated outputs, and any filtering or format normalization applied.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper's claims rest on empirical training comparisons, not on equations, fitted parameters, or author-supplied uniqueness theorems.

full rationale

This is an empirical study, not a derivation. No equation-level reduction or fitted-input-as-prediction is present: the models are trained with standard continual pre-training, instruction tuning, and preference optimization, and then evaluated on independently constructed tasks. The central claim about DACP is supported by comparisons such as LLAWA-TCxYZL-Instruct against its TAIDE base, which are direct training-effect comparisons. The skeptical concern that no run removes only the legal CPT stage while holding base model and recipe fixed is a confound in experimental design, not circularity: it does not make the conclusion equivalent to its inputs by construction. The Task D evaluation is partially self-referential because reference answers were produced by GPT-4-Turbo, one of the evaluated models, and graded by GPT-4o; however, this is a benchmark-validity limitation, and the paper itself concedes LLM-judge bias in Section 6. Task D is also not used in the DACP argument, since the LLAWA instruction-tuned variants are not scored on it. No load-bearing self-citation, imported uniqueness theorem, or ansatz smuggled via citation appears. Therefore, no circular step meets the required evidence bar, and the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

This is an empirical study, so the ledger lists hand-chosen training hyperparameters and evaluation assumptions rather than fitted scientific constants. The headline numbers are not derived from any formula, so no parameter is 'fitted to data' in the usual sense; the listed hyperparameters are the ones the paper's conclusions most directly depend on, and the axioms are the premises without which the comparisons collapse. No new particles, forces, or theoretical constructs are introduced; the burden of the paper is empirical and its central weakness is the missing control condition, listed as axiom 3.

free parameters (5)
  • DPO beta = 0.01
    Hand-chosen (Section 3.4). The DPO degradation on Task C (53.07 to 43.94) may be an artifact of this beta and the training schedule; the paper itself lists suboptimal hyperparameters as a possible cause.
  • ORPO beta = 0.1
    Hand-chosen (Section 3.4), and ORPO is applied after instruction tuning, an order the authors adopted to limit confounds. The negative result is task-dependent and could change with beta or epoch count.
  • LoRA rank and alpha = rank 64, alpha 128, learning rate 5e-5
    Hand-chosen (Section 3.3.2). The conclusion that LoRA routes beat or trail full fine-tuning rests on these values and three training epochs.
  • Training epochs = 1 for CPT; 2 for instruction tuning; 3 for LoRA and alignment
    Hand-chosen with early stopping on validation loss (Sections 3.2, 3.3, 3.4). The paper's own overfitting hypothesis for the BLLAWA and DPO/ORPO regressions implicates epoch choices.
  • Maximum input token length with right truncation = 7192
    Evaluation protocol (Section 4). It interacts with the Task D question-length analysis, since prompts exceeding the limit are truncated on the right.
assumptions (4)
  • domain assumption GPT-4o grading of essays against gold answers segmented by GPT-4-Turbo is a valid measure of legal reasoning quality.
    Task D scores in Table 3 depend on it. Section 4.3 says law school experts evaluated the segmented answers while the Table 3 caption says GPT-4o evaluated them; Section 6 concedes that LLM-based assessment 'raises concerns about objectivity and comprehensiveness.'
  • domain assumption Evaluation questions do not overlap with the training corpora of any evaluated model.
    The paper itself flags this as a contamination risk in Section 6. If overlap exists, scores are inflated and the cross-model comparisons are biased.
  • ad hoc to paper Cross-model differences in Table 3 can be attributed to the presence or absence of DACP.
    No same-base, same-recipe ablation exists: base model, tuning method, and the legal CPT stage vary together (Sections 3.2 to 3.4). The paper's causal reading of the title claim presupposes this attribution.
  • domain assumption Task C accuracy is well-defined for free-form outputs with many acceptable answer shapes.
    Section 4.2 says correct answers can be partial adoption, reservation, referral, or agreement with review opinions, but a single accuracy percentage is reported without specifying the scoring rule or the 'final conclusion' comparison method used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Continual Pre-Training is (not) What You Need in Domain Adaption." pith.science (2026). https://pith.science/paper/YKEKPBNF

@misc{pith2026250413603,
  author       = {Pith},
  title        = {Pith review of: Continual Pre-Training is (not) What You Need in Domain Adaption},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKEKPBNF}},
  note         = {Machine review of arXiv:2504.13603}
}
read the original abstract

The recent advances in Legal Large Language Models (LLMs) have transformed the landscape of legal research and practice by automating tasks, enhancing research precision, and supporting complex decision-making processes. However, effectively adapting LLMs to the legal domain remains challenging due to the complexity of legal reasoning, the need for precise interpretation of specialized language, and the potential for hallucinations. This paper examines the efficacy of Domain-Adaptive Continual Pre-Training (DACP) in improving the legal reasoning capabilities of LLMs. Through a series of experiments on legal reasoning tasks within the Taiwanese legal framework, we demonstrate that while DACP enhances domain-specific knowledge, it does not uniformly improve performance across all legal tasks. We discuss the trade-offs involved in DACP, particularly its impact on model generalization and performance in prompt-based tasks, and propose directions for future research to optimize domain adaptation strategies in legal AI.

Figures

Figures reproduced from arXiv: 2504.13603 by the authors.

Figure 1
Figure 1. The correlation between model performance and question length for each of the grading criteria and the [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Model performance varies as question length increases for many of the models. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Report

    cs.CR 2025-04 reject novelty 4.0 of 10

    A continued-pretrained 8B cybersecurity LLM claims to match GPT-4o-mini and Llama 3.1-70B on certain cyber threat intelligence benchmarks, but the decisive benchmark overlaps with its training corpus.

Reference graph

Works this paper leans on

43 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    Almeida, José Luiz Nunes, Neele Engelmann, Alex Wiegmann, and Marcelo de Araújo

    Guilherme F.C.F. Almeida, José Luiz Nunes, Neele Engelmann, Alex Wiegmann, and Marcelo de Araújo. 2024. https://doi.org/10.1016/j.artint.2024.104145 Exploring the psychology of llms’ moral and legal reasoning . Artificial Intelligence, 333:104145

    arXiv 2024

  2. [2]

    Axolotl AI . 2024. https://github.com/axolotl-ai-cloud/axolotl axolotl-ai-cloud/axolotl

    work page 2024

  3. [3]

    Giorgio Bongiovanni, Gerald Postema, Antonino Rotolo, Giovanni Sartor, Chiara Valentini, and Douglas Walton. 2018. Handbook of legal reasoning and argumentation. Springer

    work page 2018

  4. [4]

    Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.634 Recall and learn: Fine-tuning deep pretrained language models with less forgetting . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7870--7881, Online. Association fo...

    doi:10.18653/v1/2020.emnlp-main.634 2020

  5. [5]

    Daixuan Cheng, Shaohan Huang, and Furu Wei. 2023. Adapting large language models via reading comprehension. arXiv preprint arXiv:2309.09530

    arXiv 2023

  6. [6]

    Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Sofia Morgado, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, and Michael Desa. 2024. https://arxiv.org/abs/2407.19584 Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain . Preprint, arXiv:2407.19584

    arXiv 2024

  7. [7]

    Yiming Cui, Ziqing Yang, and Xin Yao. 2024. https://arxiv.org/abs/2304.08177 Efficient and effective text encoding for chinese llama and alpaca . Preprint, arXiv:2304.08177

    arXiv 2024

  8. [8]

    Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, and Hao Wang. 2024. https://arxiv.org/abs/2310.05620 Laiw: A chinese legal large language models benchmark . Preprint, arXiv:2310.05620

    arXiv 2024

Show all 43 references
  1. [9]

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. https://arxiv.org/abs/2309.16289 Lawbench: Benchmarking legal knowledge of large language models . Preprint, arXiv:2309.16289

    2023 arXiv

  2. [10]

    Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. https://doi.org/10.1145/3458754 Domain-specific language model pretraining for biomedical natural language processing . ACM Transactions on Computi...

    2021 doi

  3. [11]

    Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N

    Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat...

    2023 arXiv

  4. [12]

    Zhen Guo and Yining Hua. 2023. https://arxiv.org/abs/2311.00204 Continuous training and fine-tuning for domain-specific language models in medical question answering . Preprint, arXiv:2311.00204

    2023 arXiv

  5. [13]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.acl-main.740 Don ' t stop pretraining: Adapt language models to domains and tasks . In Proceedings of the 58th Annual Meeting o...

    2020 doi

  6. [14]

    Michael Han and Daniel Han. 2024. Unsloth. https://github.com/unslothai/unsloth

    2024

  7. [15]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

    2021 arXiv

  8. [16]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. https://arxiv.org/abs/2403.07691 Orpo: Monolithic preference optimization without reference model . Preprint, arXiv:2403.07691

    2024 arXiv

  9. [17]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  10. [18]

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. https://arxiv.org/abs/2305.08322 C-eval: A multi-level multi-discipline chinese evaluation suite for fo...

    2023 arXiv

  11. [19]

    Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2022. https://doi.org/10.18653/v1/2022.bigscience-1.1 Lifelong pretraining: Continually adapting language models to emerging corpora . In Proceedings of BigScience Episode ...

    2022 doi

  12. [20]

    Jinqi Lai, Wensheng Gan, Jiayang Wu, Zhenlian Qi, and Philip S. Yu. 2023. https://arxiv.org/abs/2312.03718 Large language models in law: A survey . Preprint, arXiv:2312.03718

    2023 arXiv

  13. [21]

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. https://doi.org/10.1093/bioinformatics/btz682 Biobert: a pre-trained biomedical language representation model for biomedical text mining . Bioinformatics, 36(4):1234–1240

    2019 doi

  14. [22]

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. https://arxiv.org/abs/2306.09212 Cmmlu: Measuring massive multitask language understanding in chinese . Preprint, arXiv:2306.09212

    2024 arXiv

  15. [23]

    Yen - Ting Lin and Yun - Nung Chen. 2023. https://doi.org/10.48550/ARXIV.2311.17487 Taiwan LLM: bridging the linguistic divide with a culturally aligned language model . CoRR, abs/2311.17487

    2023 doi arXiv

  16. [24]

    Shirong Ma, Shen Huang, Shulin Huang, Xiaobin Wang, Yangning Li, Hai-Tao Zheng, Pengjun Xie, Fei Huang, and Yong Jiang. 2023. https://arxiv.org/abs/2312.15696 Ecomgpt-ct: Continual pre-training of e-commerce large language models with semi-structured data . Preprint, arXiv:2312.15696

    2023 arXiv

  17. [25]

    Meta. 2024. https://ai.meta.com/blog/meta-llama-3/ Introducing Meta Llama 3: The most capable openly available LLM to date . Accessed: 2024-08-15

    2024

  18. [26]

    Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. 2024. https://arxiv.org/abs/2406.01375 D-cpt law: Domain-specific continual pre-training...

    2024 arXiv

  19. [27]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. https://arxiv.org/abs/2305.18290 Direct preference optimization: Your language model is secretly a reward model . Preprint, arXiv:2305.18290

    2024 arXiv

  20. [28]

    Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. 2024. https://arxiv.org/abs/2404.16789 Continual learning of large language models: A comprehensive survey . Preprint, arXiv:2404.16789

    2024 arXiv

  21. [29]

    Zhi-Rui Tam, Ya-Ting Pai, Yen-Wei Lee, Sega Cheng, and Hong-Han Shuai. 2024. An improved traditional chinese evaluation suite for foundation model. arXiv preprint arXiv:2403.01858

    2024 arXiv

  22. [30]

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl

    2020

  23. [31]

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. https://arxiv.org/abs/2303.17564 Bloomberggpt: A large language model for finance . Preprint, arXiv:2303.17564

    2023 arXiv

  24. [32]

    Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan Fang Li, Guilin Qi, and Gholamreza Haffari. 2022. https://openreview.net/group?id=ICLR.cc/2022/Conference, https://iclr.cc/Conferences/2022 Pretrained language model in continual learning: A comparative study . In International Conf...

    2022

  25. [33]

    Yong Xie, Karan Aggarwal, and Aitzaz Ahmad. 2023. https://arxiv.org/abs/2311.08545 Efficient continual pre-training for building domain specific large language models . Preprint, arXiv:2311.08545

    2023

  26. [34]

    Qimin Yang, Rongsheng Wang, Jiexin Chen, Runqi Su, and Tao Tan. 2024. https://arxiv.org/abs/2407.11536 Fine-tuning medical language models for enhanced long-contextual understanding and domain expertise . Preprint, arXiv:2407.11536

    2024 arXiv

  27. [35]

    Shengbin Yue, Wei Chen, Siyuan Wang, Bingxuan Li, Chenchen Shen, Shujun Liu, Yuxuan Zhou, Yao Xiao, Song Yun, Xuanjing Huang, and Zhongyu Wei. 2023. https://arxiv.org/abs/2309.11325 Disc-lawllm: Fine-tuning large language models for intelligent legal services . Preprint, arXiv...

    2023 arXiv

  28. [36]

    Alex Zhang. 2019. https://www.nyulawglobal.org/globalex/Taiwan1.html Taiwan Legal Research . Accessed: 2024-08-15

    2019

  29. [37]

    Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. 2024. https://arxiv.org/abs/2308.10792 Instruction tuning for large language models: A survey . Preprint, arXiv:2308.10792

    2024

  30. [38]

    Xuanyu Zhang, Qing Yang, and Dongliang Xu. 2023. https://arxiv.org/abs/2305.12002 Xuanyuan 2.0: A large chinese financial chat model with hundreds of billions parameters . Preprint, arXiv:2305.12002

    2023 arXiv

  31. [39]

    Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. https://arxiv.org/abs/2401.01055 Llama beyond english: An empirical study on language capability transfer . Preprint, arXiv:2401.01055

    2024 arXiv

  32. [40]

    Ziqiang Zheng, Jipeng Zhang, Tuan-Anh Vu, Shizhe Diao, Yue Him Wong Tim, and Sai-Kit Yeung. 2023. https://arxiv.org/abs/2310.13596 Marinegpt: Unlocking secrets of ocean to the public . Preprint, arXiv:2310.13596

    2023 arXiv

  33. [41]

    Çağatay Yıldız, Nishaanth Kanna Ravichandran, Prishruit Punia, Matthias Bethge, and Beyza Ermis. 2024. https://arxiv.org/abs/2402.17400 Investigating continual pretraining in large language models: Insights and implications . Preprint, arXiv:2402.17400

    2024 arXiv

  34. [42]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.