REVIEW 3 major objections 5 minor 31 references
An Effective Domain Adaptive Post-Training Method for BERT in Response Selection
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Post-training BERT on the target dialogue corpus, then fine-tuning only its top layers, sets new state-of-the-art results on two response-selection benchmarks.
desk verdict Domain post-training helps on response-selection benchmarks, but the paper must specify what exactly was in the post-training corpus before the SOTA numbers can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is domain post-training (DPT): resuming BERT's two unsupervised objectives, masked language modeling and next sentence prediction, on the target dialogue corpus, with the addition of a special [EOT] boundary token at the end of each utterance. This step adapts the model's contextualized representations to domain terminology and to the sequential structure of dialogue. Two further components sharpen the gain: variable fine-tuning (VFT), which freezes lower layers and updates only the top $T$ layers ($T=4$ for Ubuntu, $T=6$ for Advising), and data augmentation that increases negative samples from 1:1 to 1:4 with resampling each epoch. The paper attributes the largest share of the improvement to MLM, with NSP and [EOT] contributing smaller additional gains.
What would settle it
Re-run the Ubuntu V1 experiment with post-training restricted to the official training split only, and the Advising experiment likewise, so that no validation or test context is seen during the unsupervised step. If R10@1 on the held-out test drops back toward the BERT-base level of 0.817, then the reported 0.858 is partly an artifact of post-training having seen the evaluation conversations.
Extended reading notes
Core claim
The central claim is that a continuing-pretraining step on in-domain dialogue text substantially improves BERT for response selection. Starting from the standard uncased BERT-base checkpoint, the authors continue training with masked language modeling and next sentence prediction on Ubuntu Corpus V1 and Advising Corpus, appending an [EOT] token after each utterance so the model learns where turns end. They then fine-tune only the top four (Ubuntu) or top six (Advising) transformer layers, and train with a 1:4 ratio of positive to randomly resampled negative responses. This yields a new state of the art on both benchmarks, improving R10@1 from 0.800 to 0.858 on Ubuntu V1 and R100@1 from 0.214 to 0.274 on Advising Corpus. The paper argues that the next-sentence objective is especially suited to response selection because it directly asks whether one piece of text follows another.
Load-bearing premise
The paper's reported gains rest on the unstated assumption that the domain post-training corpus consists only of the training split of each benchmark, not the validation or test contexts that are later used for evaluation.
Editorial extensions
If this is right
- Post-training on in-domain dialogue text can yield larger gains than switching model architecture: BERT-DPT beats the previous state-of-the-art MSN by 5.1 points on Ubuntu R10@1.
- Freezing most of BERT's layers is not just cheaper; it is better: partial fine-tuning outperforms full fine-tuning on both datasets.
- The next-sentence prediction objective transfers almost directly to response selection, suggesting that other tasks with a natural 'nextness' prediction could benefit from the same post-training objective.
- Resampling negative responses each epoch at a 1:4 ratio improves over 1:1 training, pointing to a simple data-augmentation baseline for retrieval-based dialogue models.
- The recipe transfers across domains: gains appear on both a technical support corpus (Ubuntu) and an academic advising corpus.
Reading between the lines
- The paper does not state which portion of each corpus is used for post-training; a careful replication should confirm that only the training splits were used, because if validation or test contexts entered the unsupervised step, the reported gains would partly reflect information from the evaluation conversations rather than generalizable learning.
- The same post-training trick could be tested on other retrieval tasks where BERT-like models are applied, such as question answering or fact verification, where the 'response follows context' signal is also central.
- The finding that partial fine-tuning beats full fine-tuning suggests that for small in-domain adaptation sets, early-layer representations are already well-formed and further tuning hurts; this could guide a general rule for when to freeze layers.
- The positive effect of appending a turn-boundary token suggests that other structural markers, such as speaker identities or time gaps, might further help post-training on conversational data.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an effective domain adaptive post-training method for BERT in multi-turn response selection. The method consists of three components: (1) domain post-training (DPT) on the target dialog corpus with masked language modeling and next sentence prediction, inserting an [EOT] token at the end of each utterance; (2) variable fine-tuning (VFT), where only the top T layers of BERT are fine-tuned; and (3) data augmentation (DA), where the number of negative training samples is increased from 1 to 4. The authors evaluate on Ubuntu Corpus V1 and the Advising Corpus, reporting new state-of-the-art results: R10@1 of 0.858 on Ubuntu V1 and R100@1 of 0.274 on Advising, improvements over the previous best of 0.800 and 0.214, respectively. They also include ablations comparing MLM and NSP objectives, with and without the [EOT] token.
Significance. If the reported results are sound, the paper provides a simple and general recipe for adapting BERT to domain-specific response selection, and it is the first to apply BERT base to the Ubuntu V1 benchmark. The controlled comparisons among BERT_base, BERT-DPT, BERT-VFT, and BERT-VFT(DA) are a strength, and the two-benchmark evaluation lends credibility to the core idea of domain post-training. The paper also demonstrates the utility of variable fine-tuning and data augmentation for this task. However, the central state-of-the-art claim depends on an underspecified experimental condition: the exact data split used for domain post-training is never stated. If the post-training corpus includes the validation or test contexts, the reported gains would be inflated by transductive leakage and would not transfer to a properly held-out setting. In addition, the paper's own ablation in Table 5 does not support the claim that NSP is especially important, and no error bars or significance tests are reported, making the smaller differences among the proposed variants difficult to assess.
major comments (3)
- [§4.2] The data split used for domain post-training is not specified. The text says only 'We post-train the model more on Ubuntu Corpus V1 and Advising Corpus, 200,000 steps and 100,000 steps, respectively.' Because DPT is an unsupervised step performed on the same benchmark corpora, it is essential to state whether it uses only the training split or the full corpus including the validation and test contexts. If the full corpora are used, the test contexts are seen during MLM/NSP training, and the downstream classifier then evaluates on those same contexts; this is transductive leakage that could plausibly account for part or all of the reported gains (e.g., 0.817 to 0.851 for BERT-DPT on Ubuntu). The authors must specify the exact split, and ideally verify that no post-training example overlaps with any validation or test context. This condition is load-bearing for the paper's central state-of-the-art claim.
- [§4.8 and Table 5] The paper's claim that NSP is 'especially considered as an important task' (Introduction) is contradicted by its own ablation. Table 5 shows that NSP-only post-training with or without [EOT] yields R10@1 of 0.816 or 0.819, which is no better than BERT_base at 0.817, whereas MLM-only yields 0.834 or 0.838. The text in §4.8 even states 'it does not influence much in improvement of performance.' This inconsistency should be resolved: either the abstract/introduction claim should be revised to reflect that MLM is the main contributor, or the authors should explain what specific evidence supports the NSP-importance claim. As written, the claim is not supported by the reported experiments.
- [Tables 2 and 3] No error bars, confidence intervals, or significance tests are reported for any of the main results. The differences among the proposed variants are small in several cases: on Ubuntu, BERT-VFT (0.855) to BERT-VFT(DA) (0.858) is 0.3 percentage point, and on Advising, BERT-VFT and BERT-VFT(DA) tie at R100@1 with 0.274. Without multiple runs or statistical testing, it is unclear whether the improvements attributed to VFT and DA are reliable or within run-to-run variance. This is especially relevant because the selection of the number of top layers T is based on MRR in Table 4, yet the headline results are R10@1/R100@1, so the relationship between the chosen T and the reported gains should be made explicit and accompanied by variance estimates.
minor comments (5)
- [Abstract] The phrase 'performance improvement by 5.9% and 6% on R@1' is ambiguous because the two benchmarks use different metrics (R10@1 and R100@1). Please state the metric names explicitly.
- [§4.4] In the sentence 'domain-specific optimized BERT-VPT model shows performance improvement by 8.8%', 'BERT-VPT' appears to be a typo for 'BERT-VFT'.
- [§4.3] The sentence 'BERT-VFT(DA)performs data augmentation technique' is missing a space between '(DA)' and 'performs'.
- [§4.7] The description of data augmentation ('Negative responses are randomly resampled for every epoch from response candidates pool') does not specify whether the candidate pool is restricted to the training split. Given the transductive-leakage concern in §4.2, this should be clarified.
- [§4.5] When comparing to Vig and Remea, the authors say 'different implementation frameworks and hyperparameters' but do not provide implementation details of their own base BERT fine-tuning, such as learning rate schedule and number of fine-tuning epochs. These details would help reproducibility.
Circularity Check
No significant circularity: the SOTA numbers are empirical benchmark results, not fitted constants; the only flagged risk is an unspecified DPT split, which is a data-handling ambiguity rather than an in-equation circularity.
full rationale
The paper's derivation chain is empirical rather than algebraic: DPT continues masked-LM/NSP training on domain text (Eq. 1), fine-tuning trains a pointwise classifier on labeled context-response triples (Eqs. 2-3), and the reported R10@1/R100@1/MRR numbers are measured on the standard held-out splits of Ubuntu Corpus V1 and Advising Corpus (Section 4.1, Tables 2-3). No fitted constant is renamed as a prediction, and no equation reduces to its own input by construction. The variable fine-tuning depth T and the 1:4 negative-sample ratio are selected via validation-based experiments and reported as ablations, not as derived predictions. There are no load-bearing self-citations: the cited prior works (BERT, Houlsby et al., Xu et al.) are external sources of general techniques or baselines. The one legitimate concern is that Section 4.2 says 'We post-train the model more on Ubuntu Corpus V1 and Advising Corpus' without stating whether only the training splits are used; if the full corpora including test contexts were used for DPT, the test numbers would be inflated by transductive leakage. That is an experimental-validity ambiguity and a missing experimental detail, not a demonstrated circular reduction, so it should be resolved by the authors but does not by itself make the derivation circular.
Assumptions & free parameters
free parameters (3)
- number of negative samples for data augmentation =
4
- number of top layers fine-tuned (T) =
4 (Ubuntu), 6 (Advising)
- post-training steps =
200,000 (Ubuntu), 100,000 (Advising)
assumptions (4)
- domain assumption BERT_base provides transferable general-purpose contextual representations.
- domain assumption MLM and NSP objectives on in-domain text improve domain-specific representation quality.
- domain assumption The domain post-training corpus excludes validation and test contexts.
- standard math Standard deep learning infrastructure (TensorFlow, AdamW, gradient computation) behaves as expected.
Cite this review
Pith. "Pith review of An Effective Domain Adaptive Post-Training Method for BERT in Response Selection." pith.science (2026). https://pith.science/paper/IIDGX56Y
@misc{pith2026190804812,
author = {Pith},
title = {Pith review of: An Effective Domain Adaptive Post-Training Method for BERT in Response Selection},
year = {2026},
howpublished = {\url{https://pith.science/paper/IIDGX56Y}},
note = {Machine review of arXiv:1908.04812}
}
read the original abstract
We focus on multi-turn response selection in a retrieval-based dialog system. In this paper, we utilize the powerful pre-trained language model Bi-directional Encoder Representations from Transformer (BERT) for a multi-turn dialog system and propose a highly effective post-training method on domain-specific corpus. Although BERT is easily adopted to various NLP tasks and outperforms previous baselines of each task, it still has limitations if a task corpus is too focused on a certain domain. Post-training on domain-specific corpus (e.g., Ubuntu Corpus) helps the model to train contextualized representations and words that do not appear in general corpus (e.g., English Wikipedia). Experimental results show that our approach achieves new state-of-the-art on two response selection benchmarks (i.e., Ubuntu Corpus V1, Advising Corpus) performance improvement by 5.9% and 6% on R@1.
Figures
Reference graph
Works this paper leans on
-
[1]
An Effective Domain Adaptive Post-Training Method for BERT in Response Selection
Introduction Human computer conversation system aims to have natural and consistent conversation. In general, dialog systems can be cate- gorized into task-oriented dialog systems and non-task-oriented dialog systems ( i.e., chatbot systems). In this paper, we focus on one of non-task-oriented dialog systems, especially retrieval- based dialog systems, in...
work page Pith review arXiv 1908
-
[2]
Related Work Lowe et al. [1] proposed a new response selection benchmark dataset, Ubuntu Corpus V1, with a dual encoder baseline model. They utilized a RNN based models, specifically a vanilla RNN, a long short-term memory (LSTM), and a bi-directional LSTM. The sequential representations of a dialog context and a re- sponse are encoded by the RNN-based enc...
work page 2018
-
[3]
Our Approach 3.1. Domain Post-Training BERT is designed to be easily applied to other nlp tasks with a fine-tuning manner. Since it is pre-trained on general corpus (e.g., Wikipedia Corpora), it is insufficient to have enough su- pervision of task-specific words and phrases during the period of fine-tuning. To alleviate this issue, we post-train BERT on our t...
-
[4]
Experiments 4.1. Datasets and Training Setup We evaluate our model on two multi-turn dyadic data sets, Ubuntu IRC (Internet Relay Chat) Corpus V1[1] and Advis- ing Corpus1 [17]. For the Ubuntu Corpus, training set is com- posed of 0.5M dialog context containing positive and negative response with the ratio of 1:1. Each validation and test set con- tains 5...
-
[5]
Conclusion In this paper, a highly effective post-training method for a multi- turn response selection is proposed and evaluated. Our ap- proach achieved new state-of-the-art results for two response selection benchmark data sets, Ubuntu Corpus V1 and Advis- ing Corpus. For future work, we will utilize external domain knowledge, such as ubuntu manual desc...
-
[6]
Multi-turn response selection for chatbots with deep attention matching network,
X. Zhou, L. Li, D. Dong, Y . Liu, Y . Chen, W. X. Zhao, D. Yu, and H. Wu, “Multi-turn response selection for chatbots with deep attention matching network,” in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 1118–1127
2018
-
[7]
R. Lowe, N. Pow, I. Serban, and J. Pineau, “The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems,” in Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 2015, pp. 285–294
work page 2015
-
[8]
Improved deep learn- ing baselines for ubuntu corpus dialogs,
R. Kadlec, M. Schmid, and J. Kleindienst, “Improved deep learn- ing baselines for ubuntu corpus dialogs,” Computing Research Repository, vol. arXiv:1510.03753, 2015
arXiv 2015
Show all 31 references
-
[9]
Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots,
Y . Wu, W. Wu, C. Xing, M. Zhou, and Z. Li, “Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots,” inProceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), 201...
2017
-
[10]
Modeling multi-turn conversation with deep utterance aggregation,
Z. Zhang, J. Li, P. Zhu, H. Zhao, and G. Liu, “Modeling multi-turn conversation with deep utterance aggregation,” in Proceedings of the 27th International Conference on Computational Linguistics . Association for Computational Linguistics, 2018, pp. 3740–3752
2018
-
[11]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the Advances in Neural Information Processing Systems, 2017, pp. 5998–6008
2017
-
[12]
Xlnet: Generalized autoregressive pretraining for lan- guage understanding,
Z. Yang, Z. Dai, Y . Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V . Le, “Xlnet: Generalized autoregressive pretraining for lan- guage understanding,” in Proceedings of the Advances in Neural Information Processing Systems, 2019, pp. 5753–5763
2019
-
[13]
One time of interaction may not be enough: Go deep with an interaction- over-interaction network for response selection in dialogues,
C. Tao, W. Wu, C. Xu, W. Hu, D. Zhao, and R. Yan, “One time of interaction may not be enough: Go deep with an interaction- over-interaction network for response selection in dialogues,” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,...
2019
-
[14]
Multi-hop selector network for multi-turn response selection in retrieval-based chatbots,
C. Yuan, W. Zhou, M. Li, S. Lv, F. Zhu, J. Han, and S. Hu, “Multi-hop selector network for multi-turn response selection in retrieval-based chatbots,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conf...
2019
-
[15]
Comparison of transfer-learning ap- proaches for response selection in multi-turn conversations,
J. Vig and K. Ramea, “Comparison of transfer-learning ap- proaches for response selection in multi-turn conversations,” in 7th Edition of the Dialog System Technology Challenges at AAAI 2019, 2019
2019
-
[16]
Deep contextualized word representations,
M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, V...
2018
-
[17]
BERT: Pre- training of deep bidirectional transformers for language under- standing,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language under- standing,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Techn...
2019
-
[18]
Multi-view response selection for human-computer con- versation,
X. Zhou, D. Dong, H. Wu, S. Zhao, D. Yu, H. Tian, X. Liu, and R. Yan, “Multi-view response selection for human-computer con- versation,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016, pp. 372–381
2016
-
[19]
Improv- ing response selection in multi-turn dialogue systems by incorpo- rating domain knowledge,
D. Chaudhuri, A. Kristiadi, J. Lehmann, and A. Fischer, “Improv- ing response selection in multi-turn dialogue systems by incorpo- rating domain knowledge,” inProceedings of the 22nd Conference on Computational Natural Language Learning. Association for Computational Linguisti...
2018
-
[20]
Bert post-training for re- view reading comprehension and aspect-based sentiment anal- ysis,
H. Xu, B. Liu, L. Shu, and P. S. Yu, “Bert post-training for re- view reading comprehension and aspect-based sentiment anal- ysis,” Computing Research Repository , vol. arXiv:1904.02232, 2019
1904 arXiv
-
[21]
Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks,
J. Phang, T. F ´evry, and S. R. Bowman, “Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks,” Computing Research Repository, vol. arXiv:1811.01088, 2018
2018 arXiv
-
[22]
Superglue: A stickier bench- mark for general-purpose language understanding systems,
A. Wang, Y . Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “Superglue: A stickier bench- mark for general-purpose language understanding systems,” in Proceedings of the Advances in Neural Information Processing Systems, 2019, pp. 3266–3280
2019
-
[23]
DSTC7 task 1: Noetic end-to-end response selec- tion,
C. Gunasekara, J. K. Kummerfeld, L. Polymenakos, and W. Lasecki, “DSTC7 task 1: Noetic end-to-end response selec- tion,” inProceedings of the First Workshop on NLP for Conversa- tional AI at ACL 2019, 2019, pp. 60–67
2019
-
[24]
Sequential attention-based network for noetic end-to-end response selection,
Q. Chen and W. Wang, “Sequential attention-based network for noetic end-to-end response selection,” in7th Edition of the Dialog System Technology Challenges at AAAI 2019, 2019
2019
-
[25]
Enhanced LSTM for natural language inference,
Q. Chen, X. Zhu, Z.-H. Ling, S. Wei, H. Jiang, and D. Inkpen, “Enhanced LSTM for natural language inference,” inProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 1657–1668
2017
-
[26]
Multi- representation fusion network for multi-turn response selection in retrieval-based chatbots,
C. Tao, W. Wu, C. Xu, W. Hu, D. Zhao, and R. Yan, “Multi- representation fusion network for multi-turn response selection in retrieval-based chatbots,” in Proceedings of the Twelfth ACM In- ternational Conference on Web Search and Data Mining. ACM, 2019, pp. 267–275
2019
-
[27]
The trec-8 question answering track re- port
E. M. V oorhees et al., “The trec-8 question answering track re- port.” inTrec, vol. 99, 1999, pp. 77–82
1999
-
[28]
Tensorflow: A system for large-scale machine learning,
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for large-scale machine learning,” inProceedings of the 12th USENIX Symposium on Operating Systems Design and Im- plementation (OSDI 16), 2016, pp. 265–283
2016
-
[29]
Distributed representations of words and phrases and their com- positionality,
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their com- positionality,” inProceedings of the Advances in Neural Informa- tion Processing Systems, 2013, pp. 3111–3119
2013
-
[31]
Parameter- efficient transfer learning for nlp,
N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Larous- silhe, A. Gesmundo, M. Attariyan, and S. Gelly, “Parameter- efficient transfer learning for nlp,” in Proceedings of the 36th In- ternational Conference on Machine Learning, 2019
2019
-
[2017]
Zhou et al
achieved significant performance improvements on var- ious NLP tasks; it was also adapted for the response selection task [6, 7, 8]. Zhou et al. [6] used self-attention and cross- attention simultaneously so as to capture token-level dependen- cies as well as relevant segment p...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.