{"id":"d5a54e48-6463-47a9-922a-79286188723a","arxiv_id":"2504.19856","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Concatenating k-nearest-neighbor context from domain and in-domain text during continual pretraining improves German process-industry semantic search at roughly one quarter of the GPU cost of standard DAPT.","lead":"This paper proposes a cheaper way to teach a German language model the vocabulary of the chemical process industry: it retrieves a few similar documents for each training example and trains the model on those short contexts. The authors report that this method beats standard domain-adaptive pretraining on a semantic search benchmark while using about a quarter of the GPU time.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The admitted overlap between the ID/target training data and the seven-shift-book test collection means ICL-APT's 28.7% advantage could be leakage, not method.","rationale":"The paper's central claim is that ICL-APT's best configuration beats DAPT by 7.87 points at roughly one quarter of the GPU time, showing that retrieval-augmented context can substitute for large-scale continual pretraining in this low-resource domain. The most load-bearing condition for that claim is a clean evaluation where the training data does not contain the test distribution. The paper itself explicitly undermines this condition by admitting that portions of the test collection may appear in both target and source datasets. This is not merely a matter of statistical rigor: ICL-APT is the one method that retrieves kNN context per target seed, so test documents in the source corpus are disproportionately likely to be selected as neighbors and then seen verbatim during MLM. That would directly inflate the retrieval/ranking metrics without improving generalizable domain learning. The DAPT and TAPT baselines also train on some of the same source data, but they do not perform this targeted selection of near-test passages, so the leakage is asymmetric. Because the paper also selects the best configuration from Table 1 and gives no error bars, the headline number is additionally fragile, but the overlap issue is the single decisive threat. I agree with the reader's weakest assumption. The proposed concrete test would settle the concern: quantify the overlap and rerun the best configuration with disjoint ID books. If the gap persists after removing leaked records, the central claim would be substantially supported; if it collapses, the rejection is warranted. Given the paper's own admission and the absence of a disjoint evaluation, the reader's REJECT verdict remains appropriate.","tokens_in":11066,"tokens_out":2783,"duration_ms":30926,"concrete_test":"Compute exact and near-duplicate overlap between the 330K test documents plus 205 queries and the target 10K logs, the ID 3.2M logs, and the DR 10.3GB corpus (e.g., normalized exact match with MinHash and 8-gram overlap), and report the fraction of test records appearing in the ICL-APT training set under the best configuration (DR:ID = 10:20, cos. dist. = 0.7). Then rerun the best ICL-APT configuration with ID data restricted to shift books disjoint from the seven test shift books, keeping all other hyperparameters fixed. If the mean score drops materially from 35.28 and approaches or falls below DAPT's 27.41, leakage is the explanation. A secondary check is to log whether test documents are actually returned as kNN neighbors for target seeds under the same retrieval settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states: 'Some portions of the text data from this collection may appear in both the target and source datasets.' The ID source comprises 3.2M logs from twenty shift books, while the test collection is 330K documents and 205 queries from seven shift books. ICL-APT retrieves kNN context from ID/DR data for target seed logs, then trains MLM on the concatenated text. If test documents appear in ID data, they can be retrieved and memorized during pretraining, so the retrieval evaluation rewards exposure to the evaluation corpus rather than learned domain semantics. The paper provides no disjointness guarantee, no overlap quantification, and no experiment holding out the test shift books. DAPT and TAPT baselines also train on some of these data, but ICL-APT is the configuration that actively selects nearest neighbors of the target data, so leakage would asymmetrically inflate ICL-APT's scores. The load-bearing premise of the central claim is therefore that the augmentation method, not test-set exposure, explains the 7.87-point improvement over DAPT; that premise is currently unsupported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ICL-APT, a domain-adaptive continual pretraining method for German process-industry text. ICL-APT uses k-nearest-neighbor retrieval from domain-related (DR) and in-domain (ID) sources to augment a seed set of target text logs with additional context, and then performs masked language modeling on the augmented texts. The authors report that their best configuration outperforms DAPT on a German semantic-search benchmark by 28.7% (7.87 points) in mean MAP@10/MRR/nDCG@10 while requiring about a quarter of the GPU hours, and they include ablations and a fine-tuned bi-encoder comparison. The central claim is that retrieval-based context augmentation with small curated data can replace large-scale continual pretraining in this domain.","tokens_in":11258,"tokens_out":5285,"duration_ms":51337,"significance":"If the reported gains are real, the work has clear practical value for low-resource industrial domains: it promises strong domain adaptation at a fraction of the compute cost of DAPT, and it integrates kNN retrieval and in-context learning into the pretraining data pipeline in a way that is simple and likely portable. The paper also deserves credit for including multiple standard baselines (DAPT, cTAPT, TAPT, DAPT+combinations), for reproducing the Gururangan et al. training setup, and for being transparent about a data-overlap risk that many papers would omit. However, the significance is currently undermined by three load-bearing concerns: the admitted potential overlap between training and test data, the selection of the best configuration from the test table without a validation split, and the absence of any error bars or significance testing. Because these issues directly affect the headline 7.87-point improvement, the practical significance is not yet established.","major_comments":[{"comment":"The paper admits that \"some portions of the text data from this collection may appear in both the target and source datasets.\" Since the ID source comprises 3.2M logs from twenty shift books and the test collection is based on seven shift books, any overlap means test-corpus documents may have been present in the continual-pretraining data. For ICL-APT, the risk is asymmetric: kNN retrieval explicitly selects nearest neighbors of each target seed, so if a test document is in the ID index, it can be retrieved and included in the MLM training context. This would allow the model to memorize test text during pretraining rather than learning generalizable domain semantics. The manuscript does not quantify the overlap, does not guarantee disjointness, and does not run a version with the seven test shift books held out. The central claim of a 7.87-point improvement over DAPT therefore rests on an unsupported and explicitly acknowledged leakage risk.","section":"§4.1, Semantic search task"},{"comment":"The headline result is obtained by selecting the best of five ICL-APT configurations (10:10 with cos. dist. 0.7 and 0.8, 10:10/0.7 at 10 and 15 epochs, 20:10/0.7, 10:20/0.7) after inspecting the test-set means. No held-out validation split is used, and no correction for multiple comparisons is reported. As a result, the reported 35.28 is the maximum over a small grid, not a verified prediction. Furthermore, no error bars, number of random seeds, or significance tests are reported for any of Tables 1, 2, or 3. Without these, it is impossible to determine whether the 7.87-point gap over DAPT, or the 1.42-point gap over TAPT(80ep), is outside the run-to-run noise.","section":"Table 1"},{"comment":"The baseline set omits aTAPT, the kNN-based retrieval-augmented pretraining method of Gururangan et al. (2020) from which ICL-APT is directly derived. aTAPT is the natural state-of-the-art competitor for the retrieval component of ICL-APT; without it, the paper cannot support the claim that its retrieval-plus-context-concatenation design, rather than the kNN selection mechanism alone, is responsible for the gains. The authors should run aTAPT under the same compute budget and report it alongside DAPT and ICL-APT.","section":"§4.1, Baselines"},{"comment":"The ablation table is internally inconsistent with the accompanying claim that \"each component positively impacts\" ICL-APT. The row labeled \"–ID\" reports a delta of +3.48, which means removing in-domain data improves the mean, yet the text states that every component contributes positively. Either the sign convention is inverted, the label is incorrect, or the narrative misrepresents the data. As printed, the ablation contradicts the method's own motivation and must be corrected and re-interpreted.","section":"Table 2"}],"minor_comments":[{"comment":"The claim \"almost 4 times less GPU-computing time\" is numerically inaccurate for the headline comparison: DAPT uses 22 GPU-hours and the best ICL-APT configuration uses 4.5 GPU-hours, a factor of 4.9. The text should say \"about 5 times less\" or report the exact factor.","section":"Abstract, §1"},{"comment":"The metric name \"normalized commutative normalized discounted cumulative gain\" contains a typo; it should be \"normalized discounted cumulative gain (nDCG@10).\"","section":"§4.1, Metrics"},{"comment":"The statement that ICL-APT \"requires less than 80 times the training data for all training steps\" is vague. Please specify what is being factored (e.g., total tokens, number of documents, or GPU-hours) and give the concrete multiplier.","section":"§4.2, Table 3 discussion"},{"comment":"The paper does not state how many random seeds or independent runs were used for the continual-pretraining experiments. Please report this, and if only one run was performed, say so explicitly and add at least bootstrap confidence intervals over the test queries.","section":"§4.1, Implementation details"},{"comment":"Because the evaluation benchmark is the authors' own prior work (reference [31]), the paper should describe whether the seven test shift books are disjoint from the twenty ID shift books and from the 10K target logs, and if they are not, give the exact overlap statistics.","section":"§4.1, Semantic search task"}],"recommendation":"major_revision","confidential_remarks":"The reader's report recommends rejection, and I agree that the current evidence is not sufficient. However, the flaws are experimental in nature and could in principle be fixed within the scope of a revision: the authors could re-run with a disjoint test split, quantify or eliminate the overlap, add aTAPT as a baseline, and report significance. I therefore recommend major_revision rather than outright rejection, but the revision must be substantive: the headline claim is not currently supported. Please ask the authors to explicitly state whether any test shift book appears in the ID source, and to provide a held-out evaluation before any configuration selection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a sensible, incremental extension of aTAPT and cross-document LM pretraining, and the efficiency numbers are probably real in direction. But the headline claim—28.7% over DAPT with 4x less GPU—is not reliable yet, because the evaluation setup admits possible overlap between the ID pretraining source and the test collection, and the best configuration is picked from the test table.\n\nWhat's actually new: ICL-APT retrieves nearest neighbors from both domain-related and in-domain data, concatenates them to target seed logs, and creates multiple masked versions for MLM. That combination is new for German process-industry logs. The paper does a reasonably thorough sweep over cosine thresholds, masking ratios, and epochs, and the ablation (Table 2) shows each component helps. The GPU-hour comparison is useful and concrete: 0.12GB of augmented data yields competitive retrieval scores at a fraction of DAPT's training time. That efficiency result is the paper's real value, if the contamination issue can be settled.\n\nSoft spots, in order. First, Section 4.1 explicitly says 'Some portions of the text data from this collection may appear in both the target and source datasets.' Given the ID source contains 3.2M logs from twenty shift books and the test collection is seven shift books, the overlap could be substantial. ICL-APT is the only setup that actively retrieves neighbors of target records; if test documents are in the source, the retrieval step preferentially exposes the model to the evaluation distribution. That asymmetry makes the 7.87-point advantage over DAPT uninterpretable. Second, the best ICL-APT row is selected from the same table where it wins; there is no holdout configuration choice. Third, no error bars or significance tests, and no release of data or code, so we cannot estimate variance or re-run. Fourth, the evaluation benchmark is the authors' own prior work; that's acceptable if the data were independent, but combined with the overlap it leaves the main comparison without an independent anchor. The public bi-encoder baselines in Table 3 help, but they do not test the central DAPT comparison.\n\nNone of this is fatal to the method. The paper is honest about the overlap, the experimental sweep is systematic, and the efficiency argument is plausible. The problem is that the headline number is not yet evidenced. The authors need a disjoint holdout of shift books, a pre-registered configuration, and ideally a few seeds.\n\nBottom line: this deserves a serious referee, but it should go back for a clean evaluation before the efficiency-vs-DAPT claim is accepted. I'd rather see the revised version than cite the current number.","headline":"Useful efficiency recipe, but the headline 28.7% gain over DAPT is not trustworthy until the admitted train/test overlap is ruled out and the configuration is chosen on a holdout.","tokens_in":11796,"tokens_out":3135,"would_cite":false,"duration_ms":32073,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retrieval-augmented continual-pretraining method, ICL-APT, outperforms state-of-the-art DAPT on German process-industry semantic search by 28.7% while using roughly one-quarter of the GPU time.","keywords":["domain-adaptive continual pretraining","in-context learning","k-nearest neighbors","masked language modeling","semantic search","low-resource German","process industry","data augmentation"],"falsifier":"Re-run the best ICL-APT configuration after removing from the domain-related and in-domain source corpora every document that shares text with the seven evaluation shift books, and compare the semantic-search score to 35.28; if the advantage over DAPT largely disappears, the central claim is refuted.","tokens_in":10853,"feed_emoji":"🔍","tokens_out":6504,"duration_ms":59647,"temperature":0.7,"pith_summary":"This paper claims that a small, carefully augmented pretraining set can outperform large-scale domain-adaptive continual pretraining in a low-resource domain. The proposed method, ICL-APT, retrieves the nearest domain-related and in-domain texts for each target log, concatenates them as context, and runs masked language modeling on the augmented records. On German process-industry shift-log semantic search, the best configuration scores 35.28, beating DAPT's 27.41 by 28.7% while using roughly one-quarter of the GPU time. If correct, this means costly gigabyte-scale domain corpora can be replaced by lighter retrieval-augmented data selection.","feed_headline":"Context-augmented pretraining beats DAPT at one-quarter GPU cost","feed_subtitle":"In German process-industry search, adding retrieved context beats standard domain pretraining at one-quarter of the GPU cost.","key_machinery":"The load-bearing object is the context-augmented training record: each short shift-log text, the seed, is expanded with its nearest neighbors from a 10.3 GB domain-related corpus and a 1.51 GB in-domain corpus, selected by cosine similarity in a multilingual sentence-embedding space and concatenated within the 512-token window. Multiple randomly masked versions of each augmented record are produced so that several domain terms can be learned within the same context. This pipeline moves data preparation to the CPU and keeps GPU training on a small augmented dataset, which is what allows the claimed fourfold GPU savings.","core_discovery":"The central discovery is that giving a BERT-size language model extra textual context during continual pretraining, context retrieved by similarity from both broadly domain-related and in-domain sources, lets it learn dense domain vocabulary from far fewer training steps and far less data. Concretely, for each seed log the authors retrieve the $K=3$ nearest neighbors under a cosine-distance threshold, concatenate them to the seed, randomly mask tokens with 10 or 20 variations, and continue pretraining GBERT-base on this augmented set. The best configuration, with 10 masking variations for domain-related text, 20 for in-domain text, and a cosine threshold of 0.7, reaches a mean of 35.28 across MAP@10, MRR, and nDCG@10, outperforming DAPT, TAPT at 80 epochs, and DAPT+TAPT while using about 4.5 GPU-hours versus 22 for DAPT. The authors interpret this as evidence that context and data quality, not corpus size or epoch count, drive effective domain adaptation in this setting.","pith_inferences":["If the source overlap concern is resolved, the transferable idea is that similarity-based context concatenation can substitute for corpus scale in other jargon-heavy low-resource domains, such as legal or engineering German.","The strong effect of in-domain masking variation suggests that learning term identity, not just topical similarity, is what drives the gain; one testable extension is to weight retrieval toward sentences containing rare domain tokens.","Because the comparison uses a single GPU and fixed hyperparameters, the fourfold GPU-time claim is tied to that setup; replicating on smaller accelerators or with gradient accumulation would show whether the savings hold under tighter constraints."],"forward_implications":["In this domain, retrieval-augmented pretraining records can replace full-domain DAPT: the best ICL-APT configuration beats DAPT by 7.87 points at about one-quarter of the GPU hours.","Stricter retrieval distance improves performance: a cosine threshold of 0.7 outperforms 0.8 even when trained for fewer epochs.","Masking variation on in-domain text matters more than masking variation on domain-related text: the 10:20 configuration beats 20:10 by 3.15 points.","After fine-tuning as a retrieval encoder, the ICL-APT model outperforms GBERT-base, TAPT at 80 epochs, DAPT+TAPT, and the 560-million-parameter multilingual E5 encoder despite being much smaller.","The ablation shows every component contributes: removing the in-domain retrieval component costs 5.35 points, removing the added context costs 1.46 points, and reducing token-masking variation costs 4.05 points."],"supporting_citations":[{"why":"Supplies the DAPT, TAPT, cTAPT, and aTAPT baselines and the masked-language-modeling continual-pretraining setup that ICL-APT extends.","marker":"[16]"},{"why":"Motivates pretraining on concatenated related documents so the model learns term semantics from context, which is the core of the ICL idea.","marker":"[4]"},{"why":"Provides retrieval-augmented in-context learning for domain adaptation, which ICL-APT adapts to the pretraining stage.","marker":"[21]"},{"why":"Provides the multilingual sentence-embedding model used for kNN retrieval and for mean-pooled document encoding.","marker":"[25]"},{"why":"Supplies the German version of the MS MARCO training collection used to fine-tune retrieval encoders for the semantic search evaluation.","marker":"[24]"},{"why":"Defines the zero-shot semantic-search evaluation setup and the metric conventions used for reporting.","marker":"[28]"},{"why":"Supplies the seven-shift-book test collection with 205 queries over 330K documents used for the evaluation.","marker":"[31]"}],"fun_headline_variants":["Context-augmented pretraining beats DAPT by 29% using 4x less GPU","German process-industry NLP: retrieved context beats standard pretraining","kNN-augmented pretraining cuts GPU hours 4x for German domain models","Less compute, better results: context retrieval boosts German NLP","ICL-APT: 29% better than DAPT with one-quarter GPU cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the source texts retrieved for augmentation do not overlap with the seven shift books used for evaluation, since the paper notes that some portions of the test collection may appear in both datasets.","fun_headline_variants_meta":{"raw":{"variants":["Context-augmented pretraining beats DAPT by 29% using 4x less GPU","German process-industry NLP: retrieved context beats standard pretraining","kNN-augmented pretraining cuts GPU hours 4x for German domain models","Less compute, better results: context retrieval boosts German NLP","ICL-APT: 29% better than DAPT with one-quarter GPU cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000854,"raw_usage":{"total_tokens":3730,"prompt_tokens":984,"completion_tokens":2746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":2653}},"tokens_in":600,"tokens_out":2746,"duration_ms":22280,"temperature":1.0,"reasoning_tokens":2653,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:41:33.139830+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the best ICL-APT configuration after removing from the domain-related and in-domain source corpora every document that shares text with the seven evaluation shift books, and compare the semantic-search score to 35.28; if the advantage over DAPT largely disappears, the central claim is refuted.","supporting_citations":[{"cited_title":"In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics","cited_arxiv_id":null,"evidence_quote":"Supplies the DAPT, TAPT, cTAPT, and aTAPT baselines and the masked-language-modeling continual-pretraining setup that ICL-APT extends."},{"cited_title":"In: Besold, T.R., Bordes, A., d’Avila Garcez, A.S., Wayne, G","cited_arxiv_id":null,"evidence_quote":"Supplies the German version of the MS MARCO training collection used to fine-tune retrieval encoders for the semantic search evaluation."},{"cited_title":"In: Vanschoren, J., Yeung, S","cited_arxiv_id":null,"evidence_quote":"Defines the zero-shot semantic-search evaluation setup and the metric conventions used for reporting."},{"cited_title":"In: Hettiarachchi, H., Ranasinghe, T., Rayson, P., Mitkov, R., Gaber, M., Premasiri, D., Tan, F.A., Uyangodage, L","cited_arxiv_id":null,"evidence_quote":"Supplies the seven-shift-book test collection with 205 queries over 330K documents used for the evaluation."}],"review_version":1}