REVIEW 4 major objections 5 minor 23 references
A lightweight contrastive denoising autoencoder on frozen BERT keeps sentence vectors more similar under synonym, mask, and dropout noise than raw BERT or SimCSE.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-31 13:41 UTC pith:5722SZHZ
load-bearing objection Small, clear post-hoc head that raises clean–perturbed cosine under lexical noise; the result is real but circular, and “preserves semantics” is untested. the 4 major comments →
CDAE: Enhancing Perturbation Robustness in Pretrained Language Models with Contrastive Denoising
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The authors show that jointly optimizing an InfoNCE contrastive loss and a denoising-plus-identity reconstruction loss on top of frozen mean-pooled BERT embeddings produces a 128-dimensional latent space whose clean-versus-perturbed cosine similarity is consistently higher than that of raw BERT and of SimCSE, across synonym replacement, masking, and word dropout at strengths 0.1–0.9, with the advantage growing under stronger perturbation.
What carries the argument
CDAE (Contrastive Denoising Autoencoder): a small MLP encoder–decoder (768→512→256→128 and reverse) trained on frozen BERT vectors so the encoder is shaped by both InfoNCE alignment of clean/perturbed pairs and MSE reconstruction, while the decoder sees only reconstruction; the 128-d bottleneck is the final sentence representation.
Load-bearing premise
Higher cosine similarity between clean and perturbed vectors is treated as enough proof that semantic content is preserved, without checking any downstream similarity, retrieval, or classification task.
What would settle it
Run standard STS, retrieval, or clustering benchmarks on the same 128-d CDAE codes versus BERT and SimCSE; if CDAE wins on clean–perturbed cosine yet loses or ties on those semantic tasks, the claim that it preserves semantic information while gaining stability fails.
If this is right
- Sentence embeddings used in search and recommenders can be made more stable to everyday edits without fine-tuning the underlying language model.
- The same frozen-backbone plus lightweight CDAE pattern can be dropped on other embedding architectures, not only BERT.
- Stronger natural perturbations hurt less once the representation is forced through the joint contrastive–denoising bottleneck.
- Layer-wise follow-up work can locate where transformer stacks lose stability under semantic-preserving noise and guide where to attach such refiners.
Where Pith is reading between the lines
- Because only the autoencoder is trained, the method is a cheap post-hoc stabilizer that could be stacked on already-deployed embedding APIs without re-indexing costs from backbone changes.
- The widening gap at high perturbation strength suggests the reconstruction term is doing real denoising work rather than merely shrinking the embedding space.
- If downstream STS scores hold, the same objective might also reduce sensitivity to OCR noise, typos, and mild paraphrase in production pipelines.
- Absence of a memory bank or momentum encoder means the method stays simple but may scale poorly to very large negative sets compared with full SimCSE-style training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CDAE, a lightweight contrastive denoising autoencoder trained on top of frozen bert-base-uncased mean-pooled embeddings. For each SNLI premise it builds a perturbed view via synonym replacement, masking, or word dropout, encodes both views, and jointly optimizes a symmetric InfoNCE loss on 128-d latent codes plus an MSE reconstruction loss through a mirrored MLP decoder. The sole reported evaluation is mean cosine similarity between clean and perturbed representations across the three perturbation types at strengths 0.1–0.9, comparing CDAE latents to raw BERT and SimCSE; Figure 4 shows CDAE retaining higher similarity, with larger gaps at high strength. The authors conclude that CDAE improves perturbation stability while preserving semantic information.
Significance. If the invariance gains were shown to leave semantic utility intact, a frozen-backbone refinement module with few trainable parameters and public code would be a useful, easily adoptable contribution for retrieval and embedding pipelines that face natural lexical noise. The systematic multi-strength, multi-strategy robustness sweep and the explicit gradient-flow split between encoder and decoder are clear methodological strengths. As written, however, significance is limited: the headline metric largely recapitulates the training objective, and no STS, retrieval, NLI, clustering, or MTEB-style result demonstrates that the 128-d codes remain useful sentence embeddings. The work is therefore incremental until semantic fidelity is independently measured.
major comments (4)
- [Abstract; §4–§5] Abstract and §5 claim that CDAE enhances stability “while preserving semantic information,” yet §4 reports only mean clean–perturbed cosine (Eqs. 8–9, Figure 4). No STS, retrieval, NLI, clustering, or MTEB evaluation is provided. Without an independent semantic task, the stronger claim is unsupported; at minimum the abstract/conclusion language must be narrowed, or standard embedding benchmarks must be added for CDAE vs BERT/SimCSE.
- [§3.2.4–§3.2.5; §4; Figure 4] Training maximizes agreement between clean and perturbed latents via InfoNCE (Eqs. 2–3) plus reconstruction, using the same perturbation families (synonym/mask/dropout) at ρ=0.7 (§4). Evaluation then reports mean cos(clean, perturbed) on those families. Figure 4 therefore largely confirms that optimization succeeded and generalized across strengths, not that the latents are better sentence embeddings. A held-out metric orthogonal to the training signal is needed for the central robustness-quality claim.
- [§3.2; Eq. (4)] §3.2 prose states the decoder recovers the clean backbone embedding from a perturbed latent, but Eq. (4) is identity reconstruction: ‖ẑ_p − z_p‖² + ‖ẑ_o − z_o‖² (both targets match their own inputs). There is no term ‖ẑ_p − z_o‖². Either the equation or the denoising narrative must be corrected; as written the “denoising” story is inconsistent with the loss.
- [§4; Eqs. (8)–(9)] The authors note that comparing raw cosine in 128-d latent space vs 768-d BERT/SimCSE space is uncalibrated (§4). Dimensionality and training-induced isotropy can inflate apparent gaps. Report a calibrated comparison (e.g., same-dimension baselines, linear probes, or rank-based agreement) or restrict claims to within-space degradation curves rather than cross-space Δ.
minor comments (5)
- [Abstract] Abstract and title block contain repeated grammatical errors (“embedding remain,” “as framework effectively,” missing words in the abstract’s last sentence). A full copy-edit is needed.
- [§3.2.5; §4] λ_recon, λ_contrast, and temperature τ appear in Eqs. (2)–(5) but are never given numeric values or ablated in §4.
- [Figure 4] Figure 4 bottom row y-axis is labeled only “Gap”; specify that it is Δ cosine (CDAE − baseline) and which baseline each curve uses.
- [§2.2–§2.3] Related work cites TSDAE and RobustSentEmbed but does not clearly position CDAE against them (frozen vs fine-tuned backbone; natural vs adversarial perturbations).
- [throughout] Typos: “Pre-retrained,” “visializes,” “Institude,” inconsistent capitalization in the title line.
Circularity Check
Clean–perturbed cosine largely recapitulates the InfoNCE training signal; “preserves semantic information” is not independently tested.
specific steps
-
fitted input called prediction
[§3.2.4 Eqs. 2–3 (train); §4 Eqs. 8–9 and Figure 4 (eval)]
"Lcontrast = −1/2 [ (1/B) Σ log exp(logits_ii)/Σ exp(logits_ij) + (perturbed→original) ] ... ¯c_{m,ρ} = (1/N) Σ cos(x^o_i, x^p_i), (x^o_i, x^p_i) ∈ {(z^o_i, z^p_i), (¯z^o_i, ¯z^p_i)} ... Δ_{m,ρ} = ¯c^CDAE_{m,ρ} − ¯c^BERT_{m,ρ}"
InfoNCE is trained to maximize agreement between clean and perturbed latents under synonym/mask/dropout. Evaluation then reports mean clean–perturbed cosine under the same three families. Success on Figure 4 is therefore largely that optimization worked and generalized across strengths, not an independent robustness discovery. Comparison to BERT/SimCSE shows a relative gain, but the absolute claim “preserves higher embedding similarity under perturbations” is the training target renamed as the result.
-
self definitional
[Abstract; §5 Conclusion]
"CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings."
“Preserving semantic information” is asserted solely from elevated clean–perturbed cosine under the training perturbations. Semantic fidelity is thereby defined as invariance to those perturbations, with no external semantic task (STS, retrieval, NLI, clustering) that could falsify collapse or loss of discriminative content. The conclusion restates the training objective’s success as evidence of semantic preservation by construction of the metric.
-
other
[§3.2 prose vs Eq. 4; §4 note on uncalibrated cross-space cosine]
"a reconstruction loss that trains a decoder to recover the clean 768-dimensional backbone embedding from the latent code of a perturbed input ... L_recon = ∥ẑ_p − z_p∥²_2 + ∥ẑ_o − z_o∥²_2 ... We note that this comparison uses raw, uncalibrated cosine similarity across the two spaces. Because the latent space is 128-dimensional and the raw BERT space is 768-dimensional."
Prose sells a denoising story (recover clean embedding from perturbed latent), but Eq. 4 is identity reconstruction of each view onto itself (ẑ_p→z_p, ẑ_o→z_o), so the advertised denoising signal is not what is optimized. Gaps vs 768-d BERT are also acknowledged as uncalibrated across dimensions, which can mechanically inflate Δ. These do not fully force the result but tighten the loop between stated objective, actual loss, and reported superiority.
full rationale
CDAE is trained with a symmetric InfoNCE loss that explicitly pulls together ℓ2-normalized latent codes of a sentence and its synonym/mask/dropout perturbation (Eqs. 2–3), plus a reconstruction term (Eqs. 4–5), using those same perturbation families at strength 0.7. The sole reported success metric is again mean cosine similarity between clean and perturbed representations in the learned 128-d space versus raw BERT and SimCSE (Eqs. 8–9, Figure 4). Held-out SNLI sentences and a sweep over strengths 0.1–0.9 give some generalization beyond pure train/test identity, and outperforming frozen BERT/SimCSE is a real empirical observation that the encoder internalized the invariance objective. Nonetheless, the central robustness claim reduces largely to confirming that the quantity optimized at train time remains high at test time on the same perturbation types. The stronger Abstract/§5 language that CDAE enhances stability “while preserving semantic information” equates that invariance with semantic fidelity without any independent STS, retrieval, NLI, clustering, or MTEB-style check that different meanings remain separated. No self-citation chain or uniqueness import is involved; the circularity is objective–metric alignment plus an untested semantic-preservation gloss. Score 6 reflects partial circularity of the headline result, not total vacuity of the method.
Axiom & Free-Parameter Ledger
free parameters (5)
- λ_recon and λ_contrast (loss weights) =
unspecified in text
- InfoNCE temperature τ =
unspecified in text
- Training perturbation strength ρ =
0.7
- Latent dimension and MLP widths =
128-d latent; 512/256 hidden
- Dropout p and optimization hyperparameters =
p=0.3, lr=1e-4, wd=1e-5, bs=512, 20 epochs
axioms (5)
- domain assumption Mean-pooled final-layer bert-base-uncased vectors are fixed, sufficiently informative sentence features for refinement without updating BERT.
- domain assumption Synonym replacement (WordNet), content-word dropout, and mask-token substitution at rate ρ are semantic-preserving perturbations.
- domain assumption In-batch negatives without a memory bank are adequate for InfoNCE sentence representation learning here.
- ad hoc to paper Higher cosine similarity between clean and perturbed embeddings indicates improved perturbation-invariant sentence representation quality.
- standard math Standard autograd / InfoNCE / MSE training dynamics apply; encoder receives both losses, decoder only reconstruction.
invented entities (1)
-
CDAE refinement module (contrastive denoising AE on frozen BERT)
no independent evidence
read the original abstract
Pre-trained language models have significantly improved sentence representation learning, yet their embedding remain sensitive to semantic preserving textual perturbations such as synonym substitution, masking and word dropout. This work proposes a lightweight Contrastive Denoising Autoencoder (CDAE) that refines pre-trained BERT embedding by jointly optimizing contrastive and reconstruction objective to learn perturbation-invariant representation. We evaluate the proposed framework using multiple perturbation strategies with varying strengths and compare it against the original BERT embeddings and SimCSE. Experimental results show that CDAE consistently preserves higher embedding similarity under perturbations, with the improvements becoming more pronounced as framework effectively enhances representation stability while preserving semantic information, highlighting perturbation-invariant learning as a promising direction for improving sentence embeddings. The source code is publicly available at: https://github.com/ComputationIASBS/CDAE
Figures
Reference graph
Works this paper leans on
-
[1]
Peters et al
Matthew E. Peters et al. Deep contextualized word representations. InProceedings of NAACL-HLT, 2018
2018
-
[2]
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings 7 APREPRINT- JULY31, 2026 of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human L...
2026
-
[3]
Llms are also effective embedding models: An in-depth overview.arXiv preprint arXiv:2412.12591, 2024
Chongyang Tao, Tao Shen, Shen Gao, Junshuo Zhang, Zhen Li, Kai Hua, Wenpeng Hu, Zhengwei Tao, and Shuai Ma. Llms are also effective embedding models: An in-depth overview.arXiv preprint arXiv:2412.12591, 2024
Pith/arXiv arXiv 2024
-
[4]
Sentence-BERT: Sentence embeddings using Siamese BERT-networks
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, ...
2019
-
[5]
Bowman, Gabor Angeli, Christopher Potts, and Christopher D
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors,Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal, September 2015. Association fo...
2015
-
[6]
Llm applications: Current paradigms and the next frontier.arXiv preprint arXiv:2503.04596, 2025
Xinyi Hou, Yanjie Zhao, and Haoyu Wang. Llm applications: Current paradigms and the next frontier.arXiv preprint arXiv:2503.04596, 2025
arXiv 2025
-
[7]
Interpreting the robustness of neural NLP models to textual perturbations
Yunxiang Zhang, Liangming Pan, Samson Tan, and Min-Yen Kan. Interpreting the robustness of neural NLP models to textual perturbations. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors,Findings of the Association for Computational Linguistics: ACL 2022, pages 3993–4007, Dublin, Ireland, May 2022. Association for Computational Linguistics
2022
-
[8]
Multilingual e5 text embeddings: A technical report, 2024
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Multilingual e5 text embeddings: A technical report, 2024
2024
-
[9]
SimCSE: Simple contrastive learning of sentence embeddings
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors,Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic, November 2021. Assoc...
2021
-
[10]
Robustsentembed: Robust sentence embeddings using adversarial self-supervised contrastive learning
Javad Rafiei Asl, Prajwal Panzade, Eduardo Blanco, Daniel Takabi, and Zhipeng Cai. Robustsentembed: Robust sentence embeddings using adversarial self-supervised contrastive learning. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 3795–3809, 2024
2024
-
[11]
Text embeddings by weakly-supervised contrastive pre-training.arXiv preprint arXiv:2212.03533, 2022
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training.arXiv preprint arXiv:2212.03533, 2022
Pith/arXiv arXiv 2022
-
[12]
Mteb: Massive text embedding benchmark
Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, 2023
2014
-
[13]
Tsdae: Using transformer-based sequential denoising auto- encoderfor unsupervised sentence embedding learning
Kexin Wang, Nils Reimers, and Iryna Gurevych. Tsdae: Using transformer-based sequential denoising auto- encoderfor unsupervised sentence embedding learning. InFindings of the association for computational linguistics: EMNLP 2021, pages 671–688, 2021
2021
-
[14]
George A. Miller. WordNet: A lexical database for English. InSpeech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992, 1992
1992
-
[15]
Representation learning with contrastive predictive coding, 2019
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019
2019
-
[16]
Pytorch: An imperative style, high-performance deep learning library, 2019
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performa...
2019
-
[17]
Transformers: State-of-the-art natural language processing
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art na...
2020
-
[18]
Decoupled weight decay regularization, 2019
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 8 APREPRINT- JULY31, 2026
2019
-
[19]
Datasets: A community library for natural language processing
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger,...
2021
-
[20]
Data Structures for Statistical Computing in Python
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors,Proceedings of the 9th Python in Science Conference, pages 56 – 61, 2010
2010
-
[21]
Harris, K
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Shepp...
2020
-
[22]
NLTK: The natural language toolkit
Steven Bird and Edward Loper. NLTK: The natural language toolkit. InProceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain, July 2004. Association for Computational Linguistics. 9
2004
-
[2019]
Association for Computational Linguistics
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.