SOAEsV2-7B/72B: Full-Pipeline Optimization for State-Owned Enterprise LLMs via Continual Pre-Training, Domain-Progressive SFT and Distillation-Enhanced Speculative Decoding

Jingyang Deng; Jinwen Ma; Jo-Ku Cheng; Ran Chen

REVIEW 3 major objections 5 minor 28 references

SOAEsV2-7B/72B: Full-Pipeline Optimization for State-Owned Enterprise LLMs via Continual Pre-Training, Domain-Progressive SFT and Distillation-Enhanced Speculative Decoding

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Domain LLM keeps 99.8% of general skill, runs 1.39-1.52x faster

desk verdict A reasonable industrial recipe that deserves refereeing, but the unverified self-derived test set makes the headline numbers uncheckable. read the letter →

arxiv 2505.04723 v1 pith:RZS6LB5V submitted 2025-05-07 cs.CL

Jingyang Deng , Ran Chen , Jo-Ku Cheng , Jinwen Ma This is my paper · ORCID

classification cs.CL

keywords LargeLanguageModelsContinualPre-TrainingDomain-ProgressiveSFTCurriculumLearningKnowledgeDistillationSpeculativeDecodingState-OwnedEnterprisesChineseNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a complete three-phase recipe for turning a general-purpose large language model into a specialist for Chinese state-owned assets and enterprises (SOAEs) at both 7B and 72B scale. The pipeline continually pre-trains on a curated 17-billion-token domain corpus, then runs a two-stage curriculum fine-tune that moves from loosely related conversational data to expert-annotated SOAEs data, and finally distills the 72B model's logits into a 7B draft model used for speculative decoding. The authors report that the 72B model keeps 99.8% of its general Chinese ability on CMMLU and C-EVAL while improving domain Rouge-1 by 1.08x and BLEU-4 by 1.17x, and that the distilled draft model accelerates inference by 1.39x under top-p sampling and 1.52x under greedy decoding with no quality loss. A sympathetic reader would take the paper's point to be that full-pipeline coordination, not any single trick, is what makes domain specialization affordable.

What carries the argument

The machinery is the three-phase pipeline itself, with its load-bearing components: continual pre-training on a 17B-token expert-filtered SOAEs corpus; domain-progressive SFT, a curriculum that orders data by domain relevance (stage-1 general dialog from Infinity-Instruct and LongWriter, stage-2 roughly 33k expert-annotated and GLM4-generated samples); and speculative decoding with a logit-distilled draft model, where the training objective $L = \alpha L_{KL} + (1-\alpha)L_{SFT}$ aligns the 7B draft's softmax logits with the 72B target's at temperature $\tau$. The decisive mechanism is the distributional alignment between draft and target: it makes token acceptance high enough (75.78%) that parallel verification outweighs the cost of drafting.

What would settle it

Take the same three-stage pipeline and evaluate it on a publicly documented, held-out SOAEs test set with a fixed generation protocol; if the staged-vs-joint SFT advantage (2.2% Rouge-1) and the CPT advantage (8.4% Rouge-1) shrink or invert on that set, the central claim fails. A simpler check: ask the authors to release the test queries and document how they were split from the 33k SFT samples.

Watch

Extended reading notes

Core claim

The paper's central claim is that a three-phase pipeline—continual pre-training on 17B curated SOAEs tokens, two-stage domain-progressive SFT, and logit-distillation-enhanced speculative decoding—produces a 72B SOAEs model (SOAEsV2-72B-Chat) that retains 99.8% of Qwen2.5-72B's average CMMLU/C-EVAL performance (90.1 vs 90.3), improves domain Rouge-1 to 43.08 and BLEU-4 to 25.11, and runs 1.39x faster under top-p sampling and 1.52x faster under greedy decoding. The paper further claims that each stage is load-bearing: CPT contributes roughly 8% to 17% over non-CPT fine-tuned baselines, staged SFT beats single-stage joint SFT by 4.6% in Rouge-1 and 9.9% in BLEU-4, and a distilled 7B draft model beats prompt-lookup decoding by a 24% relative speedup (1.39x vs 1.12x at block size 3).

Load-bearing premise

The domain performance numbers rest on an evaluation set that the paper never names or describes, so the reported Rouge/BLEU gains could reflect a small, leaked, or unrepresentative sample rather than genuine domain mastery.

Editorial extensions

If this is right

If the pipeline works as reported, domain specialization no longer forces a choice between expertise and generality: a 72B model can be specialized to an enterprise domain while keeping roughly 99.8% of base Chinese-language capability.
The staged curriculum result implies that mixing domain-adjacent and expert data in one fine-tune round is worse than ordering them, so future domain-adaptation recipes should treat data ordering as a design axis, not a detail.
The distillation result implies that speculative decoding for very large models can be made substantially more effective by training the draft model with logit-level alignment to the target rather than reusing an off-the-shelf smaller model.
The scaling comparison (7B retaining 98.8% vs 72B retaining 99.8%) suggests that catastrophic forgetting during domain pre-training shrinks as model scale grows.
The speedups under both sampling modes mean the acceleration applies to the deployment setting that matters (top-p controlled randomness) as well as deterministic greedy generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the reported Rouge/BLEU gains survive evaluation on a named, held-out benchmark, the same curriculum (weak-relevance dialog first, expert task data second) is likely to transfer to other regulated Chinese-language domains such as finance, law, or healthcare, where similar report-generation and Q&A tasks dominate.
The 99.8% retention figure is measured on CMMLU and C-EVAL only; a reasonable extension would be to test whether the pipeline preserves instruction-following, code, and reasoning abilities, which may not be covered by those benchmarks.
The distillation objective could be probed further: since $\alpha = 0.5$ and $\tau = 2.0$ were optimal on one domain test set, a natural testable extension is whether the same constants hold for other teacher-student pairs or whether they should be tuned per domain.
The reported 1.39x speedup was measured at block size 3 with single concurrency under vLLM; real deployments with batched serving or longer block sizes may shift the speedup, so the headline number should be re-measured in the target serving configuration.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this is an applied domain-adaptation paper whose real content is a staged-SFT ablation wrapped in a three-phase pipeline. The components are all standard, but the staged-versus-joint SFT comparison is a genuine empirical question, and the distillation-enhanced speculative decoding setup is a sensible application of known ideas. There is no new scientific principle, and the paper does not oversell that; it is framed as an engineering recipe.

What it does well: the training setup is described concretely (data sources, hyperparameters, one epoch for all phases), and the internal logic is consistent. The finding that relevance-staged SFT beats both stage-2-only and joint training is plausible and worth testing. The distillation hyperparameter sweep (alpha, tau) shows a coherent pattern, and the 1.39–1.52x speedup numbers are in line with what speculative decoding typically gives.

The soft spots are real and they sit exactly where the reader's stress-test says. The domain evaluation test set is never identified. Section 3.2 says only that ~20% of the ~33k SFT samples were reserved for development and testing, but the paper never states how many test examples there are, which tasks, or whether they are disjoint from the 17B-token pre-training corpus. The only cited SOAEs benchmark is the same group's SOAEs-DataSuite [4]. That makes Tables 2–4, and especially the small relative gains (1.02x Rouge-1, 1.06x BLEU-4 for staging), effectively unverifiable. Modest test-set bias or leakage could reverse those conclusions.

Also missing: any error bars or significance tests (all numbers are single-run), and any quality measurement for the accelerated outputs. The claim of acceleration 'without quality loss' is supported only by the fact that the draft model matches the teacher on the same test set — but that test set is the same undescribed one. The speedup itself is modest and measured under single-concurrency vLLM, which is fine, but it is not a breakthrough.

Who is this for? Practitioners building Chinese SOAE-specialized models will find the recipe useful. For a research venue, the paper is thin on evidence as written. That said, the flaws are fixable: a described held-out test set, error bars, and a quality comparison for accelerated outputs would address most of my concerns.

Recommendation: send to peer review. The topic is active, the staged-SFT question is legitimate, and the evaluation can be substantially improved. A serious referee should not desk-reject it, but should require major revision.

Referee Report

3 major / 5 minor

Summary. The paper presents SOAEsV2-7B/72B, a domain-specialized Chinese LLM series for State-Owned Assets and Enterprises, built through a three-phase pipeline: (1) continual pre-training on a 17B-token domain corpus derived from the authors' SOAEs-DataSuite, (2) a two-stage curriculum-driven SFT that first uses domain-adjacent conversational data and then expert-annotated SOAEs data, and (3) logit-distillation-enhanced speculative decoding in which a 7B draft model accelerates the 72B target. The authors report that the 72B model retains 99.8% of the base Qwen2.5-72B score on CMMLU and C-EVAL, improves domain Rouge-1 and BLEU-4 by 1.08x and 1.17x respectively, and achieves 1.39-1.52x decoding speedup without quality loss. Ablations compare staged vs. joint SFT, distillation hyperparameters, and SPD vs. prompt-lookup decoding.

Significance. If the empirical claims hold, the paper offers a practical, fully described recipe for domain-adapting a 72B-scale model while preserving general capabilities, and it demonstrates a promising use of distillation-aligned speculative decoding to reduce inference cost at that scale. The method section is concrete: exact hyperparameters, optimizer settings, and data sources are given, and the general-capability retention claim rests on the public CMMLU and C-EVAL benchmarks. The paper also reports several controlled ablations (distillation weights, temperature, block size, greedy vs. top-p) that are useful. However, the core domain-performance claims depend on an evaluation set that is never adequately described or made public, and the speedup's 'without quality loss' assertion is not directly verified. These issues limit the independent verifiability of the paper's central results.

major comments (3)

[§3.2, §4.2 (Tables 2–4)] The domain evaluation test set is never defined. Section 3.2 states that about 80% of the ~33k stage-2 samples were used for training and the remainder 'reserved ... for subsequent development and testing,' but the paper does not state how many examples are in the test set, the split between the ~1k report-generation tasks and the short Q&A tasks, or whether the test set is disjoint from the 17B-token continual pre-training corpus and from the stage-1 data. The only SOAEs benchmark in the references is the authors' own SOAEs-DataSuite [4]. Since the headline gains (1.08x Rouge-1, 1.17x BLEU-4) and the staged-vs-joint SFT comparison (1.02x Rouge-1, 1.06x BLEU-4) are small relative differences, an unidentified or leaked test set could reverse the conclusions. The empirical core of the paper is not independently checkable as written.
[§4.2, Table 4] The claim that speculative decoding is 'without quality loss' is not directly tested. Table 4 reports acceptance rate and throughput for the 7B draft model, and Table 3 reports the 7B student's standalone quality, but no evaluation is given of the actual outputs produced by the 72B target under SPD versus autoregressive decoding. The authors use top-p sampling, and their rejection step (Eq. 4) is intended to preserve the target distribution, but no empirical verification (e.g., text similarity, downstream task scores, or distributional tests) is shown. Without such a check, the 'without quality loss' claim in the abstract and Section 4.2 is unsupported.
[§4.2, Tables 2–4] All reported scores are single-run numbers with no error bars, significance tests, or repeated evaluations. Given the small effect sizes that drive the paper's main conclusions (e.g., 1.02x Rouge-1 for progressive vs. joint SFT, 1.08x overall domain improvement), run-to-run variance in SFT or evaluation could plausibly change the ordinal conclusions. The authors should report standard deviations over at least a few independent runs, or a bootstrap confidence interval, to establish that the improvements are not noise.

minor comments (5)

[§4.1, §4.2, Table 3] There are several typos: 'Exprimental' in §4.1, 'Resutls' in §4.2, and 'V arying' in the Table 3 caption. These should be corrected.
[Figure 1] The caption of Figure 1 reads 'Pipeline of our document layout analysis algorithm,' which appears to be a leftover from another document; the figure actually shows the SOAEsV2 pipeline. The caption and in-figure labels should be aligned with the content.
[§3.3, Eq. (4)] Equation (4) defines a residual distribution for rejection sampling, but it is not made explicit whether this distribution is used for every rejected position or only for the first rejected token, and the conditional dependence on the preceding context is omitted in the notation. Please clarify.
[References] Reference [14] for Infinity-Instruct cites a HuggingFace URL with an 'accessed' date but no version or proper dataset citation details; provide a stable identifier. Also, the paper claims to be the 'largest 72B-parameter LLM in the SOAEs domain' but does not provide a comparison table of other SOAEs models beyond the 7B baseline.
[§4.2, Table 1] The comparison to DeepSeek-V3 base is informative, but the source [24] is a pre-print; please clarify whether the comparison numbers are for the base or instruct model, and whether the comparison is meaningful given different training paradigms.

Circularity Check

0 steps flagged · score 2.0 of 10

No construction-level circularity; domain evaluation is an undescribed in-house holdout, while general-capability and speedup claims rely on external benchmarks and standard mechanisms.

full rationale

The paper's load-bearing derivations do not reduce to their inputs by any exhibited equation. Continual pre-training is evaluated on external CMMLU and C-EVAL (Table 1), and the speculative-decoding quality-preservation claim follows from the standard rejection-sampling identity in Eqs. (3)-(4), not from a fitted parameter. The domain-progressive SFT versus joint-SFT comparison (Table 2) is an empirical ordering comparison on a held-out split, not a definitional equivalence. The main concern is that Section 3.2 ('We randomly selected ∼80% of the data as the training set and reserved the remaining data for subsequent development and testing') never specifies the domain test set used in Tables 2-4, and the only SOAEs benchmark cited is the authors' own SOAEs-DataSuite [4]; this is a verifiability and potential in-sample-selection issue, but the paper does not explicitly derive the domain results from [4]'s definitions and the split is a legitimate holdout. The distillation hyperparameters α and τ in Table 3 are tuned on the same downstream metrics, a mild selection-on-test loop, but the paper presents this as optimization rather than as an independent prediction. These issues warrant a small score but not a circularity finding.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No code or external artifact is shipped; the central claims rely on a self-created domain benchmark, a curated corpus whose composition is not released, and standard speculative decoding theory. The paper introduces no new theoretical entities.

free parameters (6)

distillation weight alpha = 0.5
Table 3 scans alpha in {0.2, 0.5, 0.8, 1.0}; 0.5 is chosen as the peak on the in-house Rouge-1 and BLEU-4 scores and used in the final pipeline.
distillation temperature tau = 2.0
Table 3 scans tau in {1.0, 2.0, 3.0, 5.0}; 2.0 is chosen as optimal on the same evaluation.
speculative decoding block size n = 3
Table 5 compares n=3 and 5; n=3 is chosen because it gives the highest measured speedup.
learning rates = 1e-5 (CPT), 1e-6 (stage 1), 3.5e-6 (stage 2 and distillation)
Chosen by hand; no sweep is reported. Listed in Section 4.1.
batch sizes = 64 (CPT, stage 1), 8 (stage 2, distillation)
Chosen by hand; no sweep is reported. Listed in Section 4.1.
top-p sampling probability = 0.7
Used for candidate generation in speculative decoding; fixed in Section 4.1 with no ablation.

assumptions (5)

domain assumption The SOAEs domain test set used in Tables 2-4 is representative and not contaminated by training data.
The paper never names the benchmark or its construction; the only SOAEs benchmark in the references is the authors' SOAEs-DataSuite [4].
domain assumption The 17B-token filtered corpus from SOAEs-DataSuite is sufficient and representative for continual pre-training.
Section 3.1 describes expert-guided filtering but reports no data composition, quality metrics, or leakage checks.
standard math Speculative decoding with the rejection-sampling rule preserves the target distribution exactly.
Section 3.3 relies on the standard guarantee from [17]; the 'no quality loss' claim depends on this theorem holding in the implementation.
domain assumption Logit distillation with alpha=0.5 and tau=2.0 aligns draft and target distributions sufficiently for high acceptance without hurting downstream quality.
This is an empirical tuning assumption; the paper tests a small grid on its own benchmark but does not establish robustness.
domain assumption GLM4-generated synthetic data, after human validation, is factually correct and useful for domain SFT.
Section 3.2 uses LLM-generated Q&A but provides no audit or accuracy statistics for the generated data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SOAEsV2-7B/72B: Full-Pipeline Optimization for State-Owned Enterprise LLMs via Continual Pre-Training, Domain-Progressive SFT and Distillation-Enhanced Speculative Decoding." pith.science (2026). https://pith.science/paper/RZS6LB5V

@misc{pith2026250504723,
  author       = {Pith},
  title        = {Pith review of: SOAEsV2-7B/72B: Full-Pipeline Optimization for State-Owned Enterprise LLMs via Continual Pre-Training, Domain-Progressive SFT and Distillation-Enhanced Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RZS6LB5V}},
  note         = {Machine review of arXiv:2505.04723}
}

abstract

This study addresses key challenges in developing domain-specific large language models (LLMs) for Chinese state-owned assets and enterprises (SOAEs), where current approaches face three limitations: 1) constrained model capacity that limits knowledge integration and cross-task adaptability; 2) excessive reliance on domain-specific supervised fine-tuning (SFT) data, which neglects the broader applicability of general language patterns; and 3) inefficient inference acceleration for large models processing long contexts. In this work, we propose SOAEsV2-7B/72B, a specialized LLM series developed via a three-phase framework: 1) continual pre-training integrates domain knowledge while retaining base capabilities; 2) domain-progressive SFT employs curriculum-based learning strategy, transitioning from weakly relevant conversational data to expert-annotated SOAEs datasets to optimize domain-specific tasks; 3) distillation-enhanced speculative decoding accelerates inference via logit distillation between 72B target and 7B draft models, achieving 1.39-1.52$\times$ speedup without quality loss. Experimental results demonstrate that our domain-specific pre-training phase maintains 99.8% of original general language capabilities while significantly improving domain performance, resulting in a 1.08$\times$ improvement in Rouge-1 score and a 1.17$\times$ enhancement in BLEU-4 score. Ablation studies further show that domain-progressive SFT outperforms single-stage training, achieving 1.02$\times$ improvement in Rouge-1 and 1.06$\times$ in BLEU-4. Our work introduces a comprehensive, full-pipeline approach for optimizing SOAEs LLMs, bridging the gap between general language capabilities and domain-specific expertise.

Figures

Figures reproduced from arXiv: 2505.04723 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 12 canonical work pages

[4]

In: 2024 IEEE 17th International Conference on Signal Processing (ICSP)

Huang, J., Deng, J., Ma, J.: Soaes-datasuite: Tailored pre-training corpus, fine- tuning dataset and benchmark for large language models in the state-owned assets and enterprises domain. In: 2024 IEEE 17th International Conference on Signal Processing (ICSP). pp. 253–257. IEEE (2024)

work page 2024
[1]

In: International Conference on Services Computing

Zhang, L.J., Chen, H., He, S., Li, C., Chen, J., Zhang, H., Du, W.: Cosis: An ai- enabled digital transformation framework integrating large language models and SOAEsV2-7B/72B 13 key performance indicators. In: International Conference on Services Computing. pp. 74–99. Springer (2024)

work page 2024
[2]

Computer Stan- dards & Interfaces p

Shaikh, T.A., Rasool, T., Mir, W.A.: Fields of the future: Digital transformation in smart agriculture with large language models and generative ai. Computer Stan- dards & Interfaces p. 104005 (2025)

work page 2025
[3]

Song, Z., Yan, B., Liu, Y., Fang, M., Li, M., Yan, R., Chen, X.: Injecting domain- specific knowledge into large language models: A comprehensive survey (2025), https://arxiv.org/abs/2502.10708

arXiv 2025
[5]

In: ICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP)

Deng, J., Zhang, Z., Cheng, J.K., Ma, J.: Enhancing large language models on domain-specific tasks: A novel training strategy via domain adaptation and prefer- ence alignment. In: ICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). pp. 1–5. IEEE (2025)

work page 2025
[6]

Zhong, M., An, C., Chen, W., Han, J., He, P.: Seeking neural nuggets: Knowledge transfer in large language models from a parametric perspective (2024),https: //arxiv.org/abs/2310.11451

arXiv 2024
[7]

Zhou, Z., Ning, X., Hong, K., Fu, T., Xu, J., Li, S., Lou, Y., Wang, L., Yuan, Z., Li, X., Yan, S., Dai, G., Zhang, X.P., Dong, Y., Wang, Y.: A survey on efficient inference for large language models (2024),https://arxiv.org/abs/2404.14294

arXiv 2024
[8]

In: Proceedings of the fourth ACM international conference on AI in finance

Li, Y., Wang, S., Ding, H., Chen, H.: Large language models in finance: A survey. In: Proceedings of the fourth ACM international conference on AI in finance. pp. 374–382 (2023)

2023

Show all 28 references

[9]

Health Care Science 2(4), 255–263 (2023)

Yang, R., Tan, T.F., Lu, W., Thirunavukarasu, A.J., Ting, D.S.W., Liu, N.: Large language models in health care: Development, applications, and challenges. Health Care Science 2(4), 255–263 (2023)

2023
[10]

In: Pro- ceedings of the 26th annual international conference on machine learning

Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Pro- ceedings of the 26th annual international conference on machine learning. pp. 41–48 (2009)

2009
[11]

In: Fu, X., Fleisig, E

Naïr, M., Yamani, K., Lhadj, L., Baghdadi, R.: Curriculum learning for small code language models. In: Fu, X., Fleisig, E. (eds.) Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume 4: Stu- dent Research Workshop). pp. 390–401. Ass...

2024
[12]

In: Al-Onaizan, Y., Bansal, M., Chen, Y.N

Hu, S., Zhou, L., Liu, S., Chen, S., Meng, L., Hao, H., Pan, J., Liu, X., Li, J., Sivasankaran, S., Liu, L., Wei, F.: WavLLM: Towards robust and adaptive speech large language model. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Find- ings of the Association for Computatio...

2024
[13]

Liu, Y., Liu, J., Shi, X., Cheng, Q., Huang, Y., Lu, W.: Let’s learn step by step: Enhancing in-context learning ability with curriculum learning (2024),https:// arxiv.org/abs/2402.10738

2024 arXiv
[14]

https:// huggingface.co/datasets/BAAI/Infinity-Instruct (2024), accessed: 2024-04- 08

Infinity-instruct: A large-scale high-quality instruction dataset. https:// huggingface.co/datasets/BAAI/Infinity-Instruct (2024), accessed: 2024-04- 08

2024
[15]

arXiv preprint arXiv:2408.07055 (2024) 14 Jingyang Deng †, Ran Chen†, Jo-Ku Cheng, and Jinwen Ma∗

Bai, Y., Zhang, J., Lv, X., Zheng, L., Zhu, S., Hou, L., Dong, Y., Tang, J., Li, J.: Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055 (2024) 14 Jingyang Deng †, Ran Chen†, Jo-Ku Cheng, and Jinwen Ma∗

2024 arXiv
[16]

arXiv preprint arXiv:2406.12793 (2024)

GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., Zhao, H., et al.: Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793 (2024)

2024 arXiv
[17]

In: International Conference on Machine Learning

Leviathan, Y., Kalman, M., Matias, Y.: Fast inference from transformers via spec- ulative decoding. In: International Conference on Machine Learning. pp. 19274– 19286. PMLR (2023)

2023
[18]

https://github.com/apoorvumang/ prompt-lookup-decoding/ (November 2023), accessed: 2024-04-08

Saxena, A.: Prompt lookup decoding. https://github.com/apoorvumang/ prompt-lookup-decoding/ (November 2023), accessed: 2024-04-08

2023
[19]

5 technical report

Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al.: Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 (2024)

2024 arXiv
[20]

In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining

Rasley, J., Rajbhandari, S., Ruwase, O., He, Y.: Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In: Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 3505–3506 (2020)

2020
[21]

arXiv preprint arXiv:1604.06174 (2016)

Chen, T., Xu, B., Zhang, C., Guestrin, C.: Training deep nets with sublinear mem- ory cost. arXiv preprint arXiv:1604.06174 (2016)

2016 arXiv
[22]

arXiv preprint arXiv:2307.08691 (2023)

Dao, T.: Flashattention-2: Faster attention with better parallelism and work par- titioning. arXiv preprint arXiv:2307.08691 (2023)

2023 arXiv
[23]

arXiv preprint arXiv:2107.02027 (2021)

Krell, M.M., Kosec, M., Perez, S.P., Fitzgibbon, A.: Efficient sequence packing without cross-contamination: Accelerating large language models without impact- ing performance. arXiv preprint arXiv:2107.02027 (2021)

2021 arXiv
[24]

arXiv preprint arXiv:2412.19437 (2024)

Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al.: Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 (2024)

2024 arXiv
[25]

arXiv preprint arXiv:2306.09212 (2023)

Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., Baldwin, T.: Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212 (2023)

2023 arXiv
[26]

Advances in Neural Information Processing Systems 36, 62991–63010 (2023)

Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Fu, Y., et al.: C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems 36, 62991–63010 (2023)

2023
[27]

In: Proceedings of the 29th Symposium on Operating Systems Principles

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C.H., Gonzalez, J., Zhang, H., Stoica, I.: Efficient memory management for large language model serving with pagedattention. In: Proceedings of the 29th Symposium on Operating Systems Principles. pp. 611–626 (2023)

2023
[4572]

Association for Computational Linguistics, Miami, Florida, USA (Nov 2024), https://aclanthology.org/2024.findings-emnlp.263/

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [4]

In: 2024 IEEE 17th International Conference on Signal Processing (ICSP)

Huang, J., Deng, J., Ma, J.: Soaes-datasuite: Tailored pre-training corpus, fine- tuning dataset and benchmark for large language models in the state-owned assets and enterprises domain. In: 2024 IEEE 17th International Conference on Signal Processing (ICSP). pp. 253–257. IEEE (2024)

work page 2024

[2] [1]

In: International Conference on Services Computing

Zhang, L.J., Chen, H., He, S., Li, C., Chen, J., Zhang, H., Du, W.: Cosis: An ai- enabled digital transformation framework integrating large language models and SOAEsV2-7B/72B 13 key performance indicators. In: International Conference on Services Computing. pp. 74–99. Springer (2024)

work page 2024

[3] [2]

Computer Stan- dards & Interfaces p

Shaikh, T.A., Rasool, T., Mir, W.A.: Fields of the future: Digital transformation in smart agriculture with large language models and generative ai. Computer Stan- dards & Interfaces p. 104005 (2025)

work page 2025

[4] [3]

Song, Z., Yan, B., Liu, Y., Fang, M., Li, M., Yan, R., Chen, X.: Injecting domain- specific knowledge into large language models: A comprehensive survey (2025), https://arxiv.org/abs/2502.10708

arXiv 2025

[5] [5]

In: ICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP)

Deng, J., Zhang, Z., Cheng, J.K., Ma, J.: Enhancing large language models on domain-specific tasks: A novel training strategy via domain adaptation and prefer- ence alignment. In: ICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). pp. 1–5. IEEE (2025)

work page 2025

[6] [6]

Zhong, M., An, C., Chen, W., Han, J., He, P.: Seeking neural nuggets: Knowledge transfer in large language models from a parametric perspective (2024),https: //arxiv.org/abs/2310.11451

arXiv 2024

[7] [7]

Zhou, Z., Ning, X., Hong, K., Fu, T., Xu, J., Li, S., Lou, Y., Wang, L., Yuan, Z., Li, X., Yan, S., Dai, G., Zhang, X.P., Dong, Y., Wang, Y.: A survey on efficient inference for large language models (2024),https://arxiv.org/abs/2404.14294

arXiv 2024

[8] [8]

In: Proceedings of the fourth ACM international conference on AI in finance

Li, Y., Wang, S., Ding, H., Chen, H.: Large language models in finance: A survey. In: Proceedings of the fourth ACM international conference on AI in finance. pp. 374–382 (2023)

2023