Pith. sign in

REVIEW 4 major objections 5 minor 34 references

Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Language

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A retrieval-augmented continual-pretraining method, ICL-APT, outperforms state-of-the-art DAPT on German process-industry semantic search by 28.7% while using roughly one-quarter of the GPU time.

desk verdict Useful efficiency recipe, but the headline 28.7% gain over DAPT is not trustworthy until the admitted train/test overlap is ruled out and the configuration is chosen on a holdout. read the letter →

arxiv 2504.19856 v3 pith:HSCQ3LEY submitted 2025-04-28 cs.CL

Anastasia Zhukova , Christian E. Matt , Bela Gipp This is my paper · ORCID
classification cs.CL
keywords domain-adaptivecontinualpretrainingin-contextlearningk-nearestneighborsmaskedlanguagemodelingsemanticsearchlow-resourceGermanprocessindustrydataaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a small, carefully augmented pretraining set can outperform large-scale domain-adaptive continual pretraining in a low-resource domain. The proposed method, ICL-APT, retrieves the nearest domain-related and in-domain texts for each target log, concatenates them as context, and runs masked language modeling on the augmented records. On German process-industry shift-log semantic search, the best configuration scores 35.28, beating DAPT's 27.41 by 28.7% while using roughly one-quarter of the GPU time. If correct, this means costly gigabyte-scale domain corpora can be replaced by lighter retrieval-augmented data selection.

What carries the argument

The load-bearing object is the context-augmented training record: each short shift-log text, the seed, is expanded with its nearest neighbors from a 10.3 GB domain-related corpus and a 1.51 GB in-domain corpus, selected by cosine similarity in a multilingual sentence-embedding space and concatenated within the 512-token window. Multiple randomly masked versions of each augmented record are produced so that several domain terms can be learned within the same context. This pipeline moves data preparation to the CPU and keeps GPU training on a small augmented dataset, which is what allows the claimed fourfold GPU savings.

What would settle it

Re-run the best ICL-APT configuration after removing from the domain-related and in-domain source corpora every document that shares text with the seven evaluation shift books, and compare the semantic-search score to 35.28; if the advantage over DAPT largely disappears, the central claim is refuted.

Watch

Extended reading notes

Core claim

The central discovery is that giving a BERT-size language model extra textual context during continual pretraining, context retrieved by similarity from both broadly domain-related and in-domain sources, lets it learn dense domain vocabulary from far fewer training steps and far less data. Concretely, for each seed log the authors retrieve the $K=3$ nearest neighbors under a cosine-distance threshold, concatenate them to the seed, randomly mask tokens with 10 or 20 variations, and continue pretraining GBERT-base on this augmented set. The best configuration, with 10 masking variations for domain-related text, 20 for in-domain text, and a cosine threshold of 0.7, reaches a mean of 35.28 across MAP@10, MRR, and nDCG@10, outperforming DAPT, TAPT at 80 epochs, and DAPT+TAPT while using about 4.5 GPU-hours versus 22 for DAPT. The authors interpret this as evidence that context and data quality, not corpus size or epoch count, drive effective domain adaptation in this setting.

Load-bearing premise

The load-bearing premise is that the source texts retrieved for augmentation do not overlap with the seven shift books used for evaluation, since the paper notes that some portions of the test collection may appear in both datasets.

Editorial extensions

If this is right

  • In this domain, retrieval-augmented pretraining records can replace full-domain DAPT: the best ICL-APT configuration beats DAPT by 7.87 points at about one-quarter of the GPU hours.
  • Stricter retrieval distance improves performance: a cosine threshold of 0.7 outperforms 0.8 even when trained for fewer epochs.
  • Masking variation on in-domain text matters more than masking variation on domain-related text: the 10:20 configuration beats 20:10 by 3.15 points.
  • After fine-tuning as a retrieval encoder, the ICL-APT model outperforms GBERT-base, TAPT at 80 epochs, DAPT+TAPT, and the 560-million-parameter multilingual E5 encoder despite being much smaller.
  • The ablation shows every component contributes: removing the in-domain retrieval component costs 5.35 points, removing the added context costs 1.46 points, and reducing token-masking variation costs 4.05 points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the source overlap concern is resolved, the transferable idea is that similarity-based context concatenation can substitute for corpus scale in other jargon-heavy low-resource domains, such as legal or engineering German.
  • The strong effect of in-domain masking variation suggests that learning term identity, not just topical similarity, is what drives the gain; one testable extension is to weight retrieval toward sentences containing rare domain tokens.
  • Because the comparison uses a single GPU and fixed hyperparameters, the fourfold GPU-time claim is tied to that setup; replicating on smaller accelerators or with gradient accumulation would show whether the savings hold under tighter constraints.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ICL-APT, a domain-adaptive continual pretraining method for German process-industry text. ICL-APT uses k-nearest-neighbor retrieval from domain-related (DR) and in-domain (ID) sources to augment a seed set of target text logs with additional context, and then performs masked language modeling on the augmented texts. The authors report that their best configuration outperforms DAPT on a German semantic-search benchmark by 28.7% (7.87 points) in mean MAP@10/MRR/nDCG@10 while requiring about a quarter of the GPU hours, and they include ablations and a fine-tuned bi-encoder comparison. The central claim is that retrieval-based context augmentation with small curated data can replace large-scale continual pretraining in this domain.

Significance. If the reported gains are real, the work has clear practical value for low-resource industrial domains: it promises strong domain adaptation at a fraction of the compute cost of DAPT, and it integrates kNN retrieval and in-context learning into the pretraining data pipeline in a way that is simple and likely portable. The paper also deserves credit for including multiple standard baselines (DAPT, cTAPT, TAPT, DAPT+combinations), for reproducing the Gururangan et al. training setup, and for being transparent about a data-overlap risk that many papers would omit. However, the significance is currently undermined by three load-bearing concerns: the admitted potential overlap between training and test data, the selection of the best configuration from the test table without a validation split, and the absence of any error bars or significance testing. Because these issues directly affect the headline 7.87-point improvement, the practical significance is not yet established.

major comments (4)
  1. [§4.1, Semantic search task] The paper admits that "some portions of the text data from this collection may appear in both the target and source datasets." Since the ID source comprises 3.2M logs from twenty shift books and the test collection is based on seven shift books, any overlap means test-corpus documents may have been present in the continual-pretraining data. For ICL-APT, the risk is asymmetric: kNN retrieval explicitly selects nearest neighbors of each target seed, so if a test document is in the ID index, it can be retrieved and included in the MLM training context. This would allow the model to memorize test text during pretraining rather than learning generalizable domain semantics. The manuscript does not quantify the overlap, does not guarantee disjointness, and does not run a version with the seven test shift books held out. The central claim of a 7.87-point improvement over DAPT therefore rests on an unsupported and explicitly acknowledged leakage risk.
  2. [Table 1] The headline result is obtained by selecting the best of five ICL-APT configurations (10:10 with cos. dist. 0.7 and 0.8, 10:10/0.7 at 10 and 15 epochs, 20:10/0.7, 10:20/0.7) after inspecting the test-set means. No held-out validation split is used, and no correction for multiple comparisons is reported. As a result, the reported 35.28 is the maximum over a small grid, not a verified prediction. Furthermore, no error bars, number of random seeds, or significance tests are reported for any of Tables 1, 2, or 3. Without these, it is impossible to determine whether the 7.87-point gap over DAPT, or the 1.42-point gap over TAPT(80ep), is outside the run-to-run noise.
  3. [§4.1, Baselines] The baseline set omits aTAPT, the kNN-based retrieval-augmented pretraining method of Gururangan et al. (2020) from which ICL-APT is directly derived. aTAPT is the natural state-of-the-art competitor for the retrieval component of ICL-APT; without it, the paper cannot support the claim that its retrieval-plus-context-concatenation design, rather than the kNN selection mechanism alone, is responsible for the gains. The authors should run aTAPT under the same compute budget and report it alongside DAPT and ICL-APT.
  4. [Table 2] The ablation table is internally inconsistent with the accompanying claim that "each component positively impacts" ICL-APT. The row labeled "–ID" reports a delta of +3.48, which means removing in-domain data improves the mean, yet the text states that every component contributes positively. Either the sign convention is inverted, the label is incorrect, or the narrative misrepresents the data. As printed, the ablation contradicts the method's own motivation and must be corrected and re-interpreted.
minor comments (5)
  1. [Abstract, §1] The claim "almost 4 times less GPU-computing time" is numerically inaccurate for the headline comparison: DAPT uses 22 GPU-hours and the best ICL-APT configuration uses 4.5 GPU-hours, a factor of 4.9. The text should say "about 5 times less" or report the exact factor.
  2. [§4.1, Metrics] The metric name "normalized commutative normalized discounted cumulative gain" contains a typo; it should be "normalized discounted cumulative gain (nDCG@10)."
  3. [§4.2, Table 3 discussion] The statement that ICL-APT "requires less than 80 times the training data for all training steps" is vague. Please specify what is being factored (e.g., total tokens, number of documents, or GPU-hours) and give the concrete multiplier.
  4. [§4.1, Implementation details] The paper does not state how many random seeds or independent runs were used for the continual-pretraining experiments. Please report this, and if only one run was performed, say so explicitly and add at least bootstrap confidence intervals over the test queries.
  5. [§4.1, Semantic search task] Because the evaluation benchmark is the authors' own prior work (reference [31]), the paper should describe whether the seven test shift books are disjoint from the twenty ID shift books and from the 10K target logs, and if they are not, give the exact overlap statistics.

Circularity Check

2 steps flagged · score 6.0 of 10

Admitted overlap between ICL-APT's in-domain training source and the seven-shift-book test collection, evaluated on the authors' own benchmark, makes the headline 28.7% gain over DAPT not an independent measurement.

  1. fitted input called prediction [Section 4.1, 'Semantic search task' paragraph; pipeline in Section 3]
    "We use an ID test collection based on seven shift books, resulting in 205 search queries over a collection of 330K documents [31]. Some portions of the text data from this collection may appear in both the target and source datasets."

    ICL-APT constructs its pretraining set by retrieving kNN neighbors from the ID source (3.2M logs from twenty shift books) for each target seed and training MLM on the concatenated text. If any of the seven-shift-book test documents occur in that ID source, the retrieval step can place them into the training context, so the model is pretrained on the very collection on which it is later scored. The paper's own sentence concedes such overlap is possible, and no disjointness guarantee, overlap quantification, or hold-out experiment is reported. The claimed 7.87-point advantage over DAPT may therefore measure exposure to the evaluation corpus rather than the augmentation method.

  2. self citation load bearing [Section 4.1, 'Semantic search task' paragraph]
    "We use an ID test collection based on seven shift books, resulting in 205 search queries over a collection of 330K documents [31]."

    The only in-domain evaluation benchmark used for the headline comparison is the authors' own prior work [31]. The paper does not validate this benchmark against an independent, externally constructed test set, and it simultaneously admits that the benchmark documents may overlap with the training sources. Every retrieval and ranking score, including the DAPT baseline and the ICL-APT configurations, is measured on this self-cited collection, so the central claim is not grounded in an independent evaluation artifact.

full rationale

The ICL-APT method itself is a combination of known building blocks (kNN data selection from [16], concatenated-document pretraining from [4], MLM on GBERT-base) and is not circular by construction. The circularity is in the evaluation chain. Section 4.1 explicitly states that the ID test collection, built from seven shift books, may overlap with the target and source datasets used for continual pretraining; the ICL-APT pipeline then retrieves neighbors from that source and trains on the concatenated text. If the overlap is real, the model has seen evaluation documents during pretraining, so the reported 28.7% improvement over DAPT is not a clean out-of-sample result. This is compounded by the fact that the benchmark is the authors' own prior publication [31] and is not validated against any independent test collection. Because the paper includes public bi-encoder baselines and zero-shot comparisons, there is some independent content, but the central claim of superiority rests on a possibly contaminated, self-cited evaluation. Score 6 reflects a partial circularity: the prediction is not shown to be independent of its training input, and the evaluation artifact is self-cited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

This is an empirical machine learning paper with no formal derivation. The central claim depends on several tuned hyperparameters, listed above, and on the unverified validity of the authors' own benchmark and the assumption that retrieved context improves masked-language pretraining.

free parameters (4)
  • K, number of nearest neighbors = 3
    Retrieval neighborhood size, fixed without a reported sensitivity analysis.
  • Cosine distance threshold D = 0.7 for the best configuration, 0.8 also tested
    Threshold for accepting retrieved neighbors; the value used in the headline result is selected from the test benchmark results in Table 1.
  • Masking variations (DR : ID) = 10 : 20 for the best configuration
    Number of masked versions per seed; this changes the mean score from 32.13 to 35.28 across configurations and was tuned on the test table.
  • Training epochs = 10 for the best configuration, 20 for the initial configuration
    Epoch count is a chosen low-compute setting; results show 15 epochs gives 35.04 and 10 epochs gives 35.28, so the best epoch count is selected from test performance.
assumptions (4)
  • domain assumption The seven-shift-book benchmark [31] is a valid measure of process-industry semantic search.
    The evaluation collection comes from a prior paper by the same authors, and this preprint provides no external validation of its quality or representativeness.
  • domain assumption kNN retrieval with paraphrase-multilingual-MiniLM-L12-v2 selects context that improves MLM learning rather than adding noise.
    This is the central mechanism of ICL-APT and is only indirectly tested through the ablation study, not independently verified.
  • domain assumption GBERT-base is representative of the practical CPU-deployable model setting for this industry.
    The choice to exclude LLMs and use GBERT-base is motivated by deployment constraints, not by a direct comparison of model families.
  • domain assumption Proprietary in-domain shift logs are representative of the target data distribution.
    The ID data comes from twenty shift books and is used as both training source and, partly, as the evaluation domain, without a demonstrated distribution match.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Language." pith.science (2026). https://pith.science/paper/HSCQ3LEY

@misc{pith2026250419856,
  author       = {Pith},
  title        = {Pith review of: Efficient Domain-adaptive Continual Pretraining for the Process Industry in the German Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HSCQ3LEY}},
  note         = {Machine review of arXiv:2504.19856}
}
read the original abstract

Domain-adaptive continual pretraining (DAPT) is a state-of-the-art technique that further trains a language model (LM) on its pretraining task, e.g., masked language modeling (MLM), when common domain adaptation via LM fine-tuning is not possible due to a lack of labeled task data. Although popular, MLM requires a significant corpus of domain-related data, which is difficult to obtain for specific domains in languages other than English, such as the process industry in the German language. This paper introduces an efficient approach called ICL-augmented pretraining or ICL-APT that leverages in-context learning (ICL) and k-nearest neighbors (kNN) to augment target data with domain-related and in-domain texts, significantly reducing GPU time while maintaining strong model performance. Our results show that the best configuration of ICL-APT performed better than the state-of-the-art DAPT by 28.7% (7.87 points) and requires almost 4 times less GPU-computing time, providing a cost-effective solution for industries with limited computational capacity. The findings highlight the broader applicability of this framework to other low-resource industries, making NLP-based solutions more accessible and feasible in production environments.

Figures

Figures reproduced from arXiv: 2504.19856 by the authors.

Figure 1
Figure 1. The proposed ICL-APT methodology retrieves data for continual pretraining using target data (light green) from the domain-related document collection (orange) and in-domain data (dark green) to create augmented target data for pretraining. The target data consists of the text logs from the German process industry, which contain domain-specific terminology that demands specialized knowledge of the field and a thoroug… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages

  1. [1]

    In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

    Bai, F., Ritter, A., Xu, W.: Pre-train or annotate? domain adaptation with a constrained bud- get. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 5002–5015. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (2021). https://doi.org/10.18653/v1/2021.emnlp- main.409

    doi:10.18653/v1/2021.emnlp- 2021

  2. [2]

    Beltagy, I., Lo, K., Cohan, A.: SciBERT: A pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP). pp. 3613–3618. Association for Computational Linguistics, Hong Kong, China (2019). ...

    doi:10.18653/v1/d19-1371 2019

  3. [3]

    In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakan- tan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-V oss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCan- dlish, S., Radford,...

    work page 2020

  4. [4]

    In: Findings of the Association for Computational Linguis- tics: EMNLP 2021

    Caciularu, A., Cohan, A., Beltagy, I., Peters, M., Cattan, A., Dagan, I.: Cdlm: Cross- document language modeling. In: Findings of the Association for Computational Linguis- tics: EMNLP 2021. pp. 2648–2662. Association for Computational Linguistics, Punta Cana, Dominican Republic (2021). https://doi.org/10.18653/v1/2021.findings- emnlp.225

    doi:10.18653/v1/2021.findings- 2021

  5. [5]

    In: Proceedings of the 28th International Conference on Computational Linguistics

    Chan, B., Schweter, S., M ¨oller, T.: German’s next language model. In: Proceedings of the 28th International Conference on Computational Linguistics. pp. 6788–6796. International Committee on Computational Linguistics, Barcelona, Spain (Online) (2020). https:// doi.org/10.18653/v1/2020.coling-main.598

    doi:10.18653/v1/2020.coling-main.598 2020

  6. [6]

    Computers in Biology and Medicine 154, 106581 (Mar 2023)

    Chizhikova, M., L ´opez- ´Ubeda, P., Collado-Monta˜nez, J., Mart´ın-Noguerol, T., D´ıaz-Galiano, M.C., Luna, A., Ure˜na-L´opez, L.A., Mart´ın-Valdivia, M.T.: CARES: A corpus for classifica- tion of spanish radiological reports. Computers in Biology and Medicine 154, 106581 (Mar 2023). https://doi.org/10.1016/j.compbiomed.2023.106581

    arXiv 2023

  7. [7]

    In: Bender, E.M., Derczynski, L., Isabelle, P

    Chu, C., Wang, R.: A survey of domain adaptation for neural machine translation. In: Bender, E.M., Derczynski, L., Isabelle, P. (eds.) Proceedings of the 27th International Conference on Computational Linguistics. pp. 1304–1319. Association for Computational Linguistics, Santa Fe, New Mexico, USA (Aug 2018), https://aclanthology.org/C18-1111

    work page 2018

  8. [8]

    https://data.dnb.de/ FreieOnlineHochschulschriften/ (2024), data retrieved from the source on 2024-05-06

    DNB: Free online university publications. https://data.dnb.de/ FreieOnlineHochschulschriften/ (2024), data retrieved from the source on 2024-05-06

    work page 2024

Show all 34 references
  1. [9]

    https: //register.dpma.de/DPMAregister/pat/basis (2024), data retrieved from the source on 2024-01-10

    DPMAregister: DPMAregister - Amtliche Publikations- und Registerdatenbank. https: //register.dpma.de/DPMAregister/pat/basis (2024), data retrieved from the source on 2024-01-10

    2024

  2. [10]

    https://echa.europa.eu/de/regulations/ reach/legislation (2024), data retrieved from the source on 2023-07-23

    ECHA: Rechtsvorschriften. https://echa.europa.eu/de/regulations/ reach/legislation (2024), data retrieved from the source on 2023-07-23

    2024

  3. [11]

    https://eur- lex.europa.eu/legal-content/EN/ALL/?uri=celex%3A32006L0042 (2019), data retrieved from the source on 2023-07-26 Efficient Domain-adaptive Continual Pretraining 11

    EUR-Lex: Directive 2006/42/EC of the European Parliament and of the Council of 17 May 2006 on machinery, and amending Directive 95/16/EC. https://eur- lex.europa.eu/legal-content/EN/ALL/?uri=celex%3A32006L0042 (2019), data retrieved from the source on 2023-07-26 Efficient Doma...

    2019

  4. [12]

    European Comission: Legal framework governing medicinal products for human use in the EU. https://health.ec.europa.eu/medicinal-products/ legal-framework-governing-medicinal-products-human-use- eu en?etrans=de&prefLang=de (2024), data retrieved from the source on 2023-07- 24

    2024

  5. [13]

    https: //doi.org/10.7802/2423 (2022), data retrieved from the source on 2023-08-12

    GESIS: Patentdaten der Deutschen Demokratischen Republik (DDR) (1949-1990). https: //doi.org/10.7802/2423 (2022), data retrieved from the source on 2023-08-12

    2022 doi

  6. [14]

    Garg, S., Moghaddam, R.Z., Sundaresan, N.: RAPGen: An approach for fixing code ineffi- ciencies in zero-shot (Jul 2024), https://arxiv.org/abs/2306.17077

    2024 arXiv

  7. [15]

    Gunasekar, S., Zhang, Y ., Aneja, J., Mendes, C.C.T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H.S., Wang, X., Bubeck, S., Eldan, R., Kalai, A.T., Lee, Y .T., Li, Y .: Textbooks are all you need (Oct 2023)

    2023

  8. [16]

    In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

    Gururangan, S., Marasovi ´c, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., Smith, N.A.: Don’t stop pretraining: Adapt language models to domains and tasks. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 8342–

  9. [17]

    In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cot- terell, R., Chakraborty, T., Zhou, Y

    Hedderich, M.A., Lange, L., Adel, H., Str ¨otgen, J., Klakow, D.: A survey on recent approaches for natural language processing in low-resource scenarios. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-Tur, D., Beltagy, I., Bethard, S., Cot- terell, R., Chakraborty...

    2021

  10. [18]

    In: Findings of the Association for Computational Linguistics: EMNLP 2023

    Jiang, G., Jiang, C., Xue, S., Zhang, J., Zhou, J., Lian, D., Wei, Y .: Towards anytime fine- tuning: Continually pre-trained language models with hypernetwork prompts. In: Findings of the Association for Computational Linguistics: EMNLP 2023. pp. 12081–12095. Associ- ation fo...

    2023

  11. [19]

    In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J

    Joshi, P., Santy, S., Budhiraja, A., Bali, K., Choudhury, M.: The state and fate of lin- guistic diversity and inclusion in the NLP world. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Com- putation...

    2020 doi

  12. [20]

    In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023

    Ke, Z., Shao, Y ., Lin, H., Konishi, T., Kim, G., Liu, B.: Continual pre-training of language models. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net (2023), https://openreview.net/ forum?id=m GDIItaI3o

    2023

  13. [21]

    In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    Long, Q., Wang, W., Pan, S.: Adapt in contexts: Retrieval-augmented domain adaptation via in-context learning. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 6525–6542. Association for Computational Linguistics, Singapore (2023)...

    2023 doi

  14. [22]

    Monajatipoor, M., Yang, J., Stremmel, J., Emami, M., Mohaghegh, F., Rouhsedaghat, M., Chang, K.W.: Llms in biomedicine: A study on clinical named entity recognition (Apr 2024)

    2024

  15. [23]

    in-context learning: A fair comparison and evaluation

    Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., Elazar, Y .: Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Findings of the Association for Computational Linguistics: ACL

  16. [24]

    In: Besold, T.R., Bordes, A., d’Avila Garcez, A.S., Wayne, G

    Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., Deng, L.: MS MARCO: A human generated machine reading comprehension dataset. In: Besold, T.R., Bordes, A., d’Avila Garcez, A.S., Wayne, G. (eds.) Proceedings of the Workshop on Cogni- tive Computation: Int...

    2016

  17. [25]

    In: Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP)

    Reimers, N., Gurevych, I.: Making monolingual sentence embeddings multilingual us- ing knowledge distillation. In: Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP). pp. 4512–4525. Association for Compu- tational Linguistics, Onli...

    2020 doi

  18. [26]

    In: Proceedings of the 4th Clinical Natural Language Processing Workshop

    Rojas, M., Dunstan, J., Villena, F.: Clinical Flair: A pre-trained language model for spanish clinical natural language processing. In: Proceedings of the 4th Clinical Natural Language Processing Workshop. pp. 87–92. Association for Computational Linguistics, Seattle, W A (202...

    2022 doi

  19. [27]

    In: Findings of the Association for Computational Linguistics: EMNLP 2020

    Tai, W., Kung, H.T., Dong, X., Comiter, M., Kuo, C.F.: exBERT: Extending pre-trained models with domain-specific vocabulary under constrained training resources. In: Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 1433–1439. Associa- tion for Computa...

    2020 doi

  20. [28]

    In: Vanschoren, J., Yeung, S

    Thakur, N., Reimers, N., R ¨uckl´e, A., Srivastava, A., Gurevych, I.: Beir: A hetero- geneous benchmark for zero-shot evaluation of information retrieval models. In: Vanschoren, J., Yeung, S. (eds.) Proceedings of the Neural Information Processing Systems Track on Datasets and...

    2021

  21. [29]

    In: Proceedings of the 37th International Conference on Neural Information Pro- cessing Systems

    Xie, S.M., Santurkar, S., Ma, T., Liang, P.: Data selection for language models via importance resampling. In: Proceedings of the 37th International Conference on Neural Information Pro- cessing Systems. NIPS ’23, Curran Associates Inc., Red Hook, NY , USA (2023)

    2023

  22. [30]

    In: Knowledge Science, Engineer- ing and Management: 16th International Conference, KSEM 2023, Guangzhou, China, August 16–18, 2023, Proceedings, Part IV

    Yang, J., Hu, X., Huang, W., Yuan, H., Shen, Y ., Xiao, G.: Advancing domain adap- tation of BERT by learning domain term semantics. In: Knowledge Science, Engineer- ing and Management: 16th International Conference, KSEM 2023, Guangzhou, China, August 16–18, 2023, Proceedings...

    2023 doi

  23. [31]

    In: Hettiarachchi, H., Ranasinghe, T., Rayson, P., Mitkov, R., Gaber, M., Premasiri, D., Tan, F.A., Uyangodage, L

    Zhukova, A., Matt, C.E., Gipp, B.: Automated collection of evaluation dataset for seman- tic search in low-resource domain language. In: Hettiarachchi, H., Ranasinghe, T., Rayson, P., Mitkov, R., Gaber, M., Premasiri, D., Tan, F.A., Uyangodage, L. (eds.) Proceedings of the Fir...

    2025

  24. [32]

    Knowledge and In- formation Systems 66, 7859–7889 (September 2024)

    Zhukova, A., von Sperl, L., Matt, C.E., Gipp, B.: Generative user-experience research for developing domain-specific natural language processing applications. Knowledge and In- formation Systems 66, 7859–7889 (September 2024). https://doi.org/10.1007/ s10115-024-02212-5 , http...

    2024 doi

  25. [2023]

    12284–12314

    pp. 12284–12314. Association for Computational Linguistics, Toronto, Canada (Jul 12 Anastasia Zhukova, Christian E. Matt, and Bela Gipp 2023). https://doi.org/10.18653/v1/2023.findings-acl.779, https:// aclanthology.org/2023.findings-acl.779/

    2023 doi

  26. [8360]

    https://doi.org/ 10.18653/v1/2020.acl-main.740

    Association for Computational Linguistics, Online (2020). https://doi.org/ 10.18653/v1/2020.acl-main.740

    2020 doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.