{"id":"1eedae66-d3dd-4958-8c8e-2c8b6add34fd","arxiv_id":"2412.15891","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For Llama-2-7B on telecommunications tasks, instruction tuning on a telco-generated dataset suffices; continuing pretraining on raw telco text adds little (max +0.03 accuracy).","lead":"The authors built a 800M-token corpus of telecommunications text and 80K instructions, then adapted a Llama-2-7B model using six combinations of continued pretraining and instruction tuning. They report that instruction tuning alone matches the full two-step pipeline on telco tasks and can approach GPT-3.5-level answers, potentially cutting the cost of domain adaptation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'DAPT unnecessary' claim rests on a telco MCQ benchmark generated from the same paragraphs used in IAPT training, so the score likely reflects memorization; this must be tested before the claim is accepted.","rationale":"The reader's weakest assumption and my own converge on the same load-bearing point: the main comparison is made on a benchmark whose test items were generated from the same source documents used to build the training data. The paper is transparent about this in Section 3.1.3, but transparency does not remove the circularity. The IAPT training set includes paragraph-derived instructions and TeleQnA questions from the same corpus, so the IAPT-only model can score well by retrieving training passages rather than by acquiring transferable domain knowledge. Consequently, the observed difference between IAPT-only and DAPT+IAPT (at most +0.03) may reflect that both models have already memorized the test-generating passages, not that continued pretraining is genuinely unnecessary. The Nokia subset provides a partial OOD check, but the authors do not report the DAPT-vs-IAPT gap there, limiting its value for adjudicating the claim. The proposed ablation directly tests whether the benchmark measures memorization; until it is run, the central claim is conditional rather than established. I therefore keep the reader's CONDITIONAL verdict, with the condition being the outcome of this experiment. No code or data release is mentioned, which further supports the cautious stance, but the decisive scientific issue is the benchmark validity.","tokens_in":28257,"tokens_out":5148,"duration_ms":46551,"concrete_test":"Remove from the DAPT and IAPT training corpora every paragraph from ATIS/3GPP/ETSI that was selected for GPT-4 MCQ generation, retrain the IAPT-only and DAPT+IAPT variants from identical checkpoints/hyperparameters, and re-evaluate on the telco test MCQs. If IAPT-only accuracy falls substantially (e.g., >5 points) or the DAPT gain grows well beyond +0.03, the original benchmark measured memorization of training passages and the 'DAPT unnecessary' claim is not supported for held-out knowledge. If accuracy is stable, the circularity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 5 and abstract) that IAPT alone suffices, with DAPT adding at most +0.03 accuracy, is based chiefly on the telco MCQ benchmark introduced in Section 3.1.3. That benchmark was constructed by prompting GPT-4 to generate MCQs from ATIS, 3GPP, and ETSI paragraphs that are explicitly said to be 'knowledges present in the DAPT training set.' The IAPT training set (Section 3.1.2) was generated from the same raw corpus, including 11,235 paragraph-derived instruction/output pairs and 4,000 TeleQnA questions, so models can answer test MCQs by retrieving memorized passages rather than by possessing generalizable telco knowledge. The paper acknowledges this overlap in Section 3.1.3 but still anchors the main conclusion on it. The out-of-distribution Nokia MCQs, whose answers are not present in adaptation data, are reported as 'notably less favorable' (Section 4.2.1), yet the DAPT-vs-IAPT gap on that subset is not reported. Without an OOD measure, the +0.03 bound on DAPT's benefit is not established for real telco knowledge; it may only show that IAPT memorized the test-generating paragraphs. The absence of error bars or multiple seeds further weakens the small observed difference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"TelcoLM reports an empirical study of adapting Llama-2-7B to the telecommunications domain. The authors collect a raw-text corpus of about 803M tokens and an instruction set of about 80K examples, then compare continued pretraining (DAPT), instruction tuning (IAPT), and their combinations under different data mixtures. Evaluation covers telco MCQs, open QA on TeleQnA, abstract generation, and general-domain benchmarks. The paper's main claim is that IAPT alone suffices for telco adaptation and that DAPT adds at most +0.03 accuracy, with the best adapted models approaching GPT-3.5.","tokens_in":28455,"tokens_out":5919,"duration_ms":50474,"significance":"If the central claim held, it would be practically valuable: skipping DAPT substantially lowers compute and simplifies adaptation pipelines. The paper also provides a transparent description of data collection and a benchmark construction process, and it checks general-domain forgetting. However, the main quantitative evidence is currently hard to verify: the result tables contain no numerical values in the submitted text, and the in-domain telco MCQ benchmark is generated from the same paragraphs used in adaptation training. The 'DAPT unnecessary' conclusion therefore needs out-of-distribution support and uncertainty quantification before it can be accepted.","major_comments":[{"comment":"The main conclusion that DAPT adds at most +0.03 accuracy is based on a telco MCQ set generated by GPT-4 from ATIS/3GPP/ETSI paragraphs that are explicitly part of the DAPT training set, while the IAPT training set contains 11,235 instructions generated from the same raw corpus plus converted TeleQnA questions. As the paper itself acknowledges, these test questions concern 'knowledges present in the DAPT training set,' so high accuracy on them can be explained by memorization of training text rather than by domain competence. The result does not establish that DAPT is unnecessary for general telco knowledge; the Nokia MCQs, whose answers are not in the adaptation data, are reported only as 'notably less favorable' (Section 4.2.1), and the DAPT-vs-IAPT comparison on that subset is not given. Please report the per-condition accuracy on Nokia and other out-of-distribution questions, and use that, not the in-distribution synthetic MCQs, to support the 'DAPT unnecessary' claim.","section":"§3.1.3 and Table 6"},{"comment":"In the submitted manuscript, Tables 5 through 9 are present only as captions; no numerical perplexities, accuracies, ROUGE/METEOR scores, MOS scores, or abstract lengths are shown. The abstract, Section 4, and Section 5 make quantitative claims (e.g., '+0.03 maximum,' '+0.09 on average,' 'best METEOR/MOS') that cannot be checked against the provided text. Please provide the full numerical tables, including sample sizes, and state whether scores are averaged over repeated runs; without this, the central empirical comparisons are not verifiable.","section":"Tables 5–9"},{"comment":"The design has no statistical support for the 'no gain' conclusion. The observed DAPT gain is at most +0.03 accuracy, but no confidence intervals, standard errors, or multiple-seed results are reported anywhere; on test sets of the sizes described (e.g., 632 Nokia items, 900 TeleQnA items), a 0.03 difference may be within sampling variability. Even after the numerical tables are restored, the central negative claim requires an explicit uncertainty estimate or at least a multiple-seed comparison.","section":"§4.2.1 and §5"}],"minor_comments":[{"comment":"There are typos and duplicated text, including 'dicarding' in the abstract and 'on a wide range of tasks, on a wide range of tasks' in the first paragraph of Section 1.","section":"Abstract and §1"},{"comment":"The sentence 'we have selected some paragraphs having fewer than 256 words or more than 3092 words (as described in 3.1.2)' appears to invert the filtering rule stated in Section 3.1.2; please clarify which paragraphs were actually used to generate the evaluation MCQs.","section":"§3.1.3"},{"comment":"The description of transforming MCQs into instructions says 'we added a prefix' and then quotes a full instruction; please clarify whether the quoted text is the complete instruction template and how the input/output fields are structured.","section":"§3.1.2"},{"comment":"Several references are incomplete or inconsistently formatted, e.g., the in-text citation for 'Smith, 2020' is listed with a garbled author string, and the entry for 'Haxo ZXhang' should be checked; please harmonize with the journal's reference style.","section":"References"},{"comment":"There is a typo 'intereting' in the abstract-generation section; please proofread the final text.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript describes a solid engineering pipeline, but the main negative claim ('DAPT unnecessary') is currently supported only by an in-distribution benchmark with acknowledged overlap, and the central tables are not populated in the submitted text. I would welcome a revision that adds the actual numerical results, reports the Nokia out-of-distribution split per condition, and includes error bars or multiple-seed results. The scope is appropriate for cs.CL."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for its engineering substance: it collects a large new telco corpus (803M tokens, 80K instructions), compares six DAPT/IAPT recipes in a clean head-to-head, and includes a useful OOD set (Nokia MCQs) plus general-domain checks for catastrophic forgetting. The authors are also transparent about the train/eval overlap, which is more than many papers manage. That transparency is the main reason the central conclusion still feels overreached.\n\nThe soft spot is load-bearing. The telco MCQ test set is generated by GPT-4 from ATIS/3GPP/ETSI paragraphs, and those same paragraphs were used to build both the DAPT corpus and the IAPT instructions. So the model can score well by retrieving memorized passages, not by exhibiting generalizable telco knowledge. The paper acknowledges this in Section 3.1.3 but still anchors the \"DAPT is unnecessary\" claim on that benchmark. The +0.03 upper bound on DAPT's benefit comes almost entirely from that in-distribution set; the Nokia OOD results are reported as \"notably less favorable,\" but the DAPT-vs-IAPT gap on that subset is never given. Until that gap is reported, the claim that IAPT alone suffices is not supported for real telco knowledge.\n\nTwo more issues compound the problem. The actual numeric tables appear to be missing from the text (captions are there, values are not), so the quantitative evidence can't be checked by reading. And there are no error bars, no significance tests, and no released code or data. The comparison may well be internally consistent, but as presented it is not verifiable.\n\nThe upside is real. The corpus, the recipe comparison, and the inclusion of an OOD split are all useful contributions for telco NLP practitioners, and the paper is a good case study in how easy it is to create a circular benchmark. I'd like to see this go through peer review, but only conditionally: the authors need to release artifacts, report the Nokia DAPT-vs-IAPT gap, and add some estimate of variance. As written, the headline claim should be read as an interesting hypothesis, not a demonstrated result.","headline":"A serious data-and-benchmark contribution whose central \"skip DAPT\" claim is undermined by a test set generated from the same corpus used for training, so the headline result is not yet established.","tokens_in":29066,"tokens_out":1766,"would_cite":false,"duration_ms":17890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adapting Llama-2-7B to the telecommunications domain needs only a single instruction-tuning step, not continued pretraining on raw domain text, and the adapted model competes with GPT-3.5.","keywords":["telecommunications","domain adaptation","instruction tuning","continued pretraining","Llama-2","MCQA benchmark","language model evaluation","domain-specific data"],"falsifier":"A human-curated set of telco MCQs drawn from documents and exam sources not present in the DAPT/IAPT corpus, administered to the best IAPT-only model and GPT-3.5; if the IAPT-only model's accuracy drops near the unadapted Llama-2 baseline while GPT-3.5 holds, the claim that IAPT alone imparts usable domain knowledge is weakened.","tokens_in":28043,"feed_emoji":"📡","tokens_out":9197,"duration_ms":64210,"temperature":0.7,"pith_summary":"The paper tries to establish that adapting a general-purpose LLM (Llama-2-7B) to the telecommunications domain does not require continued pretraining on raw domain text (DAPT). Instead, a single instruction-tuning step (IAPT) on domain-specific and general instructions is enough to reach competitive accuracy on telco question-answering, and the best adapted models match GPT-3.5. This matters because instruction-tuning is much cheaper and simpler than continued pretraining, making domain adaptation feasible for organizations with limited compute. The claim rests on a newly collected 803M-token telco corpus and 80K instructions, plus a telco evaluation suite.","feed_headline":"Instruction tuning alone adapts Llama-2 to telecom","feed_subtitle":"Adapting a 7B model to telecom costs one fine-tuning step and still rivals GPT-3.5.","key_machinery":"The central object is the comparison of six adaptation recipes: DAPT-only, IAPT-only, and DAPT+IAPT with telco-only, general-only, or blended instructions. The load-bearing result is the near-tie between IAPT-only and DAPT+IAPT, at most +0.03 accuracy difference on telco MCQs, which makes the continued-pretraining step dispensable. The data machinery comprises an 803M-token raw corpus built from standards documents, scientific paper repositories, web corpora, and DSIR-selected sources, plus an 80K-instruction set generated from those documents and existing benchmark data. Evaluation uses telco MCQA, open QA, and abstract generation tasks.","core_discovery":"The central claim is that a single instruction-tuning step on a blend of domain-specific and general instructions is sufficient to adapt Llama-2-7B to the telecommunications domain; adding continued pretraining on raw telco text yields at most +0.03 accuracy on the telco MCQ benchmark. The adapted models reach accuracy comparable to GPT-3.5 on telco knowledge questions while remaining much smaller and cheaper to deploy.","pith_inferences":["If the same benchmark-building logic is applied elsewhere, the 'instruction tuning suffices' result is likely to hold for any technical domain where domain knowledge can be densely converted into instruction-output pairs, but this extrapolation is not tested here.","The synthetic test MCQs share source documents with the training data, so part of the measured accuracy may reflect retrieval of memorized text; a human-curated held-out exam set would quantify how much.","The paper's open-QA examples suggest adapted models also answer more concisely; a focused study could separate conciseness from factual correctness."],"forward_implications":["Telco-specific instruction tuning alone is a viable, far cheaper alternative to continued pretraining for adapting open 7B-scale models.","Blending domain and general instructions gives the best adaptation recipe, outperforming telco-only and general-only mixes.","The adapted Llama-2-7B models reach GPT-3.5-level accuracy on telco MCQ knowledge, suggesting in-house open models can replace proprietary APIs for niche domains.","Gains concentrate on knowledge explicitly present in the adaptation corpus; on out-of-domain exam MCQs, adaptation gains shrink."],"supporting_citations":[{"why":"Supplies the Llama-2-7B base model used in all adaptation runs.","marker":"(Touvron et al., 2023)"},{"why":"Supplies the TeleQnA dataset used as the telco MCQ and open-QA benchmark.","marker":"(Maatouk et al., 2023a)"},{"why":"Prior evidence that DAPT-only degrades Llama-2 prompting ability, which the paper confirms and builds on.","marker":"(Cheng et al., 2023b)"},{"why":"Argued DAPT+TAPT is the best combination; the paper's IAPT-only result directly challenges this.","marker":"(Smith, 2020)"},{"why":"Provides the GPT-3.5/GPT-4 generalist baselines and the GPT-4 used to generate synthetic instructions and MCQs.","marker":"(openai, 2023)"},{"why":"DSIR importance resampling used to select telco-relevant raw text for the DAPT corpus.","marker":"(Xie et al., 2023)"}],"fun_headline_variants":["Llama-2-7B matches GPT-3.5 on telecom with only instruction tuning","Single instruct-tuning pass replaces continued pretraining for telco LLM","One-step instruction tuning adapts Llama-2 to telecom, rivals GPT-3.5","Telco LLM: instruction tuning alone, no continued pretraining, rivals GPT-3.5","Llama-2: one instruct-tune step, no pretraining, matches GPT-3.5 on telco"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The in-domain telco benchmark is valid as a measure of domain knowledge rather than memorization, even though the synthetic test MCQs were generated from the same technical documents used to build the training sets.","fun_headline_variants_meta":{"raw":{"variants":["Llama-2-7B matches GPT-3.5 on telecom with only instruction tuning","Single instruct-tuning pass replaces continued pretraining for telco LLM","One-step instruction tuning adapts Llama-2 to telecom, rivals GPT-3.5","Telco LLM: instruction tuning alone, no continued pretraining, rivals GPT-3.5","Llama-2: one instruct-tune step, no pretraining, matches GPT-3.5 on telco"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001292,"raw_usage":{"total_tokens":5199,"prompt_tokens":791,"completion_tokens":4408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":407,"completion_tokens_details":{"reasoning_tokens":4287}},"tokens_in":407,"tokens_out":4408,"duration_ms":28727,"temperature":1.0,"reasoning_tokens":4287,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:59:56.177490+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A human-curated set of telco MCQs drawn from documents and exam sources not present in the DAPT/IAPT corpus, administered to the best IAPT-only model and GPT-3.5; if the IAPT-only model's accuracy drops near the unadapted Llama-2 baseline while GPT-3.5 holds, the claim that IAPT alone imparts usable domain knowledge is weakened.","supporting_citations":[],"review_version":1}