{"id":"0a2a05c4-9247-4037-b87e-64b3dbd20d23","arxiv_id":"2505.04723","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A full training pipeline for SOAE-specific LLMs combines continual pre-training, two-stage curriculum SFT, and distilled speculative decoding; the authors report retained general ability, domain gains, and 1.39-1.52x speedup.","lead":"This paper builds a 72-billion-parameter Chinese language model for state-owned enterprises using continual pre-training, a two-stage curriculum fine-tuning scheme, and a small draft model for faster inference. It reports that the model keeps most general abilities, improves domain-specific scores, and speeds up generation by about 1.4 to 1.5 times.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Domain results rest on an undescribed, possibly self-derived test set; Tables 2–4 cannot be independently verified as written.","rationale":"The reader's weakest-assumption analysis identified the same load-bearing gap: the evaluation test set is never described, so the domain-performance results cannot be audited. My stress-test review of the manuscript confirms this: Section 3.2 mentions only a random 80/20 split of the ~33k SFT samples and leaves the test set anonymous; Tables 2–4 give no sample size, task breakdown, or overlap check; the only cited SOAEs benchmark is the authors' own SOAEs-DataSuite [4]. Because the claimed effects are modest (e.g., 1.02× Rouge-1 for progressive SFT over single-stage, 1.08× overall), even mild test-set leakage or small-sample noise could change the conclusions. The paper's methodology is otherwise standard and the three-stage recipe is plausible, so a conditional verdict with a request for evaluation transparency is appropriate. No change to the reader's verdict is needed.","tokens_in":10743,"tokens_out":3829,"duration_ms":43200,"concrete_test":"Ask the authors to release or fully specify the test set used for Tables 2–4: source document IDs, number of examples per task, and the exact split procedure with deduplication against the 17B-token CPT corpus and the 33k SFT samples. Then re-run the main comparisons (CPT+stage-1+stage-2 vs. CPT+stage-2 vs. CPT+joint SFT) on an independently constructed held-out sample from SOAEs-DataSuite or on a public Chinese domain QA/report-generation benchmark, and check whether the 1.08×/1.17× gains and the ordering of SFT strategies persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central domain-performance claims in §4.2–4.3 are measured on a test set that the paper never defines. Section 3.2 says only that ~80% of the ~33k stage-2 samples were used for training and the remaining data was 'reserved ... for subsequent development and testing'; it does not state how many test examples there are, which tasks (report generation vs. short QA) are included, or whether the test set is disjoint from the 17B-token continual pre-training corpus and from all SFT data. The only cited SOAEs benchmark, SOAEs-DataSuite [4], is the authors' own prior work, so the test set may be small, non-public, or overlapping with training data. This matters because the headline improvements (1.08× Rouge-1, 1.17× BLEU-4) and the staged-vs-joint SFT comparison are small relative differences; a modest test-set bias or leakage could reverse the conclusions. Without a specified held-out evaluation benchmark, the empirical core of the paper is not independently checkable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SOAEsV2-7B/72B, a domain-specialized Chinese LLM series for State-Owned Assets and Enterprises, built through a three-phase pipeline: (1) continual pre-training on a 17B-token domain corpus derived from the authors' SOAEs-DataSuite, (2) a two-stage curriculum-driven SFT that first uses domain-adjacent conversational data and then expert-annotated SOAEs data, and (3) logit-distillation-enhanced speculative decoding in which a 7B draft model accelerates the 72B target. The authors report that the 72B model retains 99.8% of the base Qwen2.5-72B score on CMMLU and C-EVAL, improves domain Rouge-1 and BLEU-4 by 1.08x and 1.17x respectively, and achieves 1.39-1.52x decoding speedup without quality loss. Ablations compare staged vs. joint SFT, distillation hyperparameters, and SPD vs. prompt-lookup decoding.","tokens_in":10965,"tokens_out":3485,"duration_ms":33462,"significance":"If the empirical claims hold, the paper offers a practical, fully described recipe for domain-adapting a 72B-scale model while preserving general capabilities, and it demonstrates a promising use of distillation-aligned speculative decoding to reduce inference cost at that scale. The method section is concrete: exact hyperparameters, optimizer settings, and data sources are given, and the general-capability retention claim rests on the public CMMLU and C-EVAL benchmarks. The paper also reports several controlled ablations (distillation weights, temperature, block size, greedy vs. top-p) that are useful. However, the core domain-performance claims depend on an evaluation set that is never adequately described or made public, and the speedup's 'without quality loss' assertion is not directly verified. These issues limit the independent verifiability of the paper's central results.","major_comments":[{"comment":"The domain evaluation test set is never defined. Section 3.2 states that about 80% of the ~33k stage-2 samples were used for training and the remainder 'reserved ... for subsequent development and testing,' but the paper does not state how many examples are in the test set, the split between the ~1k report-generation tasks and the short Q&A tasks, or whether the test set is disjoint from the 17B-token continual pre-training corpus and from the stage-1 data. The only SOAEs benchmark in the references is the authors' own SOAEs-DataSuite [4]. Since the headline gains (1.08x Rouge-1, 1.17x BLEU-4) and the staged-vs-joint SFT comparison (1.02x Rouge-1, 1.06x BLEU-4) are small relative differences, an unidentified or leaked test set could reverse the conclusions. The empirical core of the paper is not independently checkable as written.","section":"§3.2, §4.2 (Tables 2–4)"},{"comment":"The claim that speculative decoding is 'without quality loss' is not directly tested. Table 4 reports acceptance rate and throughput for the 7B draft model, and Table 3 reports the 7B student's standalone quality, but no evaluation is given of the actual outputs produced by the 72B target under SPD versus autoregressive decoding. The authors use top-p sampling, and their rejection step (Eq. 4) is intended to preserve the target distribution, but no empirical verification (e.g., text similarity, downstream task scores, or distributional tests) is shown. Without such a check, the 'without quality loss' claim in the abstract and Section 4.2 is unsupported.","section":"§4.2, Table 4"},{"comment":"All reported scores are single-run numbers with no error bars, significance tests, or repeated evaluations. Given the small effect sizes that drive the paper's main conclusions (e.g., 1.02x Rouge-1 for progressive vs. joint SFT, 1.08x overall domain improvement), run-to-run variance in SFT or evaluation could plausibly change the ordinal conclusions. The authors should report standard deviations over at least a few independent runs, or a bootstrap confidence interval, to establish that the improvements are not noise.","section":"§4.2, Tables 2–4"}],"minor_comments":[{"comment":"There are several typos: 'Exprimental' in §4.1, 'Resutls' in §4.2, and 'V arying' in the Table 3 caption. These should be corrected.","section":"§4.1, §4.2, Table 3"},{"comment":"The caption of Figure 1 reads 'Pipeline of our document layout analysis algorithm,' which appears to be a leftover from another document; the figure actually shows the SOAEsV2 pipeline. The caption and in-figure labels should be aligned with the content.","section":"Figure 1"},{"comment":"Equation (4) defines a residual distribution for rejection sampling, but it is not made explicit whether this distribution is used for every rejected position or only for the first rejected token, and the conditional dependence on the preceding context is omitted in the notation. Please clarify.","section":"§3.3, Eq. (4)"},{"comment":"Reference [14] for Infinity-Instruct cites a HuggingFace URL with an 'accessed' date but no version or proper dataset citation details; provide a stable identifier. Also, the paper claims to be the 'largest 72B-parameter LLM in the SOAEs domain' but does not provide a comparison table of other SOAEs models beyond the 7B baseline.","section":"References"},{"comment":"The comparison to DeepSeek-V3 base is informative, but the source [24] is a pre-print; please clarify whether the comparison numbers are for the base or instruct model, and whether the comparison is meaningful given different training paradigms.","section":"§4.2, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claims are plausible but rest on an unidentified in-house test set and single-run evaluations. The authors should be asked to release or fully describe the test set, demonstrate disjointness from training data, and add repeated evaluations or significance measures. The speculative decoding 'no quality loss' claim also needs direct empirical support. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this is an applied domain-adaptation paper whose real content is a staged-SFT ablation wrapped in a three-phase pipeline. The components are all standard, but the staged-versus-joint SFT comparison is a genuine empirical question, and the distillation-enhanced speculative decoding setup is a sensible application of known ideas. There is no new scientific principle, and the paper does not oversell that; it is framed as an engineering recipe.\n\nWhat it does well: the training setup is described concretely (data sources, hyperparameters, one epoch for all phases), and the internal logic is consistent. The finding that relevance-staged SFT beats both stage-2-only and joint training is plausible and worth testing. The distillation hyperparameter sweep (alpha, tau) shows a coherent pattern, and the 1.39–1.52x speedup numbers are in line with what speculative decoding typically gives.\n\nThe soft spots are real and they sit exactly where the reader's stress-test says. The domain evaluation test set is never identified. Section 3.2 says only that ~20% of the ~33k SFT samples were reserved for development and testing, but the paper never states how many test examples there are, which tasks, or whether they are disjoint from the 17B-token pre-training corpus. The only cited SOAEs benchmark is the same group's SOAEs-DataSuite [4]. That makes Tables 2–4, and especially the small relative gains (1.02x Rouge-1, 1.06x BLEU-4 for staging), effectively unverifiable. Modest test-set bias or leakage could reverse those conclusions.\n\nAlso missing: any error bars or significance tests (all numbers are single-run), and any quality measurement for the accelerated outputs. The claim of acceleration 'without quality loss' is supported only by the fact that the draft model matches the teacher on the same test set — but that test set is the same undescribed one. The speedup itself is modest and measured under single-concurrency vLLM, which is fine, but it is not a breakthrough.\n\nWho is this for? Practitioners building Chinese SOAE-specialized models will find the recipe useful. For a research venue, the paper is thin on evidence as written. That said, the flaws are fixable: a described held-out test set, error bars, and a quality comparison for accelerated outputs would address most of my concerns.\n\nRecommendation: send to peer review. The topic is active, the staged-SFT question is legitimate, and the evaluation can be substantially improved. A serious referee should not desk-reject it, but should require major revision.","headline":"A reasonable industrial recipe that deserves refereeing, but the unverified self-derived test set makes the headline numbers uncheckable.","tokens_in":11512,"tokens_out":1771,"would_cite":false,"duration_ms":19861,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Domain LLM keeps 99.8% of general skill, runs 1.39-1.52x faster","keywords":["Large Language Models","Continual Pre-Training","Domain-Progressive SFT","Curriculum Learning","Knowledge Distillation","Speculative Decoding","State-Owned Enterprises","Chinese NLP"],"falsifier":"Take the same three-stage pipeline and evaluate it on a publicly documented, held-out SOAEs test set with a fixed generation protocol; if the staged-vs-joint SFT advantage (2.2% Rouge-1) and the CPT advantage (8.4% Rouge-1) shrink or invert on that set, the central claim fails. A simpler check: ask the authors to release the test queries and document how they were split from the 33k SFT samples.","tokens_in":10543,"feed_emoji":"⚡","tokens_out":7564,"duration_ms":61310,"temperature":0.7,"pith_summary":"This paper proposes a complete three-phase recipe for turning a general-purpose large language model into a specialist for Chinese state-owned assets and enterprises (SOAEs) at both 7B and 72B scale. The pipeline continually pre-trains on a curated 17-billion-token domain corpus, then runs a two-stage curriculum fine-tune that moves from loosely related conversational data to expert-annotated SOAEs data, and finally distills the 72B model's logits into a 7B draft model used for speculative decoding. The authors report that the 72B model keeps 99.8% of its general Chinese ability on CMMLU and C-EVAL while improving domain Rouge-1 by 1.08x and BLEU-4 by 1.17x, and that the distilled draft model accelerates inference by 1.39x under top-p sampling and 1.52x under greedy decoding with no quality loss. A sympathetic reader would take the paper's point to be that full-pipeline coordination, not any single trick, is what makes domain specialization affordable.","feed_headline":"Domain LLM keeps 99.8% of general skill, runs 1.39-1.52x faster","feed_subtitle":"Adds state-owned-enterprise expertise without losing general ability, and cuts cost.","key_machinery":"The machinery is the three-phase pipeline itself, with its load-bearing components: continual pre-training on a 17B-token expert-filtered SOAEs corpus; domain-progressive SFT, a curriculum that orders data by domain relevance (stage-1 general dialog from Infinity-Instruct and LongWriter, stage-2 roughly 33k expert-annotated and GLM4-generated samples); and speculative decoding with a logit-distilled draft model, where the training objective $L = \\alpha L_{KL} + (1-\\alpha)L_{SFT}$ aligns the 7B draft's softmax logits with the 72B target's at temperature $\\tau$. The decisive mechanism is the distributional alignment between draft and target: it makes token acceptance high enough (75.78%) that parallel verification outweighs the cost of drafting.","core_discovery":"The paper's central claim is that a three-phase pipeline—continual pre-training on 17B curated SOAEs tokens, two-stage domain-progressive SFT, and logit-distillation-enhanced speculative decoding—produces a 72B SOAEs model (SOAEsV2-72B-Chat) that retains 99.8% of Qwen2.5-72B's average CMMLU/C-EVAL performance (90.1 vs 90.3), improves domain Rouge-1 to 43.08 and BLEU-4 to 25.11, and runs 1.39x faster under top-p sampling and 1.52x faster under greedy decoding. The paper further claims that each stage is load-bearing: CPT contributes roughly 8% to 17% over non-CPT fine-tuned baselines, staged SFT beats single-stage joint SFT by 4.6% in Rouge-1 and 9.9% in BLEU-4, and a distilled 7B draft model beats prompt-lookup decoding by a 24% relative speedup (1.39x vs 1.12x at block size 3).","pith_inferences":["If the reported Rouge/BLEU gains survive evaluation on a named, held-out benchmark, the same curriculum (weak-relevance dialog first, expert task data second) is likely to transfer to other regulated Chinese-language domains such as finance, law, or healthcare, where similar report-generation and Q&A tasks dominate.","The 99.8% retention figure is measured on CMMLU and C-EVAL only; a reasonable extension would be to test whether the pipeline preserves instruction-following, code, and reasoning abilities, which may not be covered by those benchmarks.","The distillation objective could be probed further: since $\\alpha = 0.5$ and $\\tau = 2.0$ were optimal on one domain test set, a natural testable extension is whether the same constants hold for other teacher-student pairs or whether they should be tuned per domain.","The reported 1.39x speedup was measured at block size 3 with single concurrency under vLLM; real deployments with batched serving or longer block sizes may shift the speedup, so the headline number should be re-measured in the target serving configuration."],"forward_implications":["If the pipeline works as reported, domain specialization no longer forces a choice between expertise and generality: a 72B model can be specialized to an enterprise domain while keeping roughly 99.8% of base Chinese-language capability.","The staged curriculum result implies that mixing domain-adjacent and expert data in one fine-tune round is worse than ordering them, so future domain-adaptation recipes should treat data ordering as a design axis, not a detail.","The distillation result implies that speculative decoding for very large models can be made substantially more effective by training the draft model with logit-level alignment to the target rather than reusing an off-the-shelf smaller model.","The scaling comparison (7B retaining 98.8% vs 72B retaining 99.8%) suggests that catastrophic forgetting during domain pre-training shrinks as model scale grows.","The speedups under both sampling modes mean the acceleration applies to the deployment setting that matters (top-p controlled randomness) as well as deterministic greedy generation."],"supporting_citations":[{"why":"Supplies the 17B-token pre-training corpus and the preprocessing methodology the continual pre-training phase builds on.","marker":"[4]"},{"why":"Provides the prior 7B SOAEs model (SOAEsV1-7B) and the sequential domain-adaptation baseline this work extends.","marker":"[5]"},{"why":"Stage-1 SFT dialog data source for domain-relevant general conversation priming.","marker":"[14]"},{"why":"Stage-1 SFT long-generation data source used to build foundational competencies.","marker":"[15]"},{"why":"LLM used to generate the synthetic Q&A pairs that populate stage-2 expert-specialization data.","marker":"[16]"},{"why":"Defines speculative decoding, the framework the acceleration method modifies with a distilled draft model.","marker":"[17]"},{"why":"Baseline prompt-lookup decoding that the ablation compares against the distilled-draft approach.","marker":"[18]"},{"why":"Base Qwen2.5-7B/72B models used for initialization and as the general-purpose baselines.","marker":"[19]"},{"why":"CMMLU benchmark used to measure retained general Chinese understanding after continual pre-training.","marker":"[25]"},{"why":"C-EVAL benchmark used alongside CMMLU to quantify general capability retention.","marker":"[26]"}],"fun_headline_variants":["SOAEsV2: 72B LLM keeps 99.8% general skill, 1.39-1.52x speedup","Domain LLM: 99.8% general skill, 1.39-1.52x faster","SOAEsV2: 99.8% general ability, 1.39-1.52x faster","Distilled 7B draft speeds 72B model 1.39-1.52x, no loss","Domain-progressive SFT: +4.6% Rouge-1, +9.9% BLEU-4 vs single-stage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The domain performance numbers rest on an evaluation set that the paper never names or describes, so the reported Rouge/BLEU gains could reflect a small, leaked, or unrepresentative sample rather than genuine domain mastery.","fun_headline_variants_meta":{"raw":{"variants":["SOAEsV2: 72B LLM keeps 99.8% general skill, 1.39-1.52x speedup","Domain LLM: 99.8% general skill, 1.39-1.52x faster","SOAEsV2: 99.8% general ability, 1.39-1.52x faster","Distilled 7B draft speeds 72B model 1.39-1.52x, no loss","Domain-progressive SFT: +4.6% Rouge-1, +9.9% BLEU-4 vs single-stage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000841,"raw_usage":{"total_tokens":3764,"prompt_tokens":1146,"completion_tokens":2618,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":762,"completion_tokens_details":{"reasoning_tokens":2464}},"tokens_in":762,"tokens_out":2618,"duration_ms":17555,"temperature":1.0,"reasoning_tokens":2464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:22:48.527261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same three-stage pipeline and evaluate it on a publicly documented, held-out SOAEs test set with a fixed generation protocol; if the staged-vs-joint SFT advantage (2.2% Rouge-1) and the CPT advantage (8.4% Rouge-1) shrink or invert on that set, the central claim fails. A simpler check: ask the authors to release the test queries and document how they were split from the 33k SFT samples.","supporting_citations":[{"cited_title":"In: 2024 IEEE 17th International Conference on Signal Processing (ICSP)","cited_arxiv_id":null,"evidence_quote":"Supplies the 17B-token pre-training corpus and the preprocessing methodology the continual pre-training phase builds on."},{"cited_title":"In: ICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP)","cited_arxiv_id":null,"evidence_quote":"Provides the prior 7B SOAEs model (SOAEsV1-7B) and the sequential domain-adaptation baseline this work extends."},{"cited_title":"https:// huggingface.co/datasets/BAAI/Infinity-Instruct (2024), accessed: 2024-04- 08","cited_arxiv_id":null,"evidence_quote":"Stage-1 SFT dialog data source for domain-relevant general conversation priming."},{"cited_title":"https://github.com/apoorvumang/ prompt-lookup-decoding/ (November 2023), accessed: 2024-04-08","cited_arxiv_id":null,"evidence_quote":"Baseline prompt-lookup decoding that the ablation compares against the distilled-draft approach."},{"cited_title":"Advances in Neural Information Processing Systems 36, 62991–63010 (2023)","cited_arxiv_id":null,"evidence_quote":"C-EVAL benchmark used alongside CMMLU to quantify general capability retention."}],"review_version":1}