REVIEW 4 major objections 5 minor 24 references
"Mask and Infill" : Applying Masked Language Model to Sentiment Transfer
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper claims that sentiment transfer is best recast as localization-plus-infill: mask the sentiment-bearing tokens and let a bidirectional masked language model conditioned on the target sentiment fill the gaps, which improves…
desk verdict A genuinely early application of BERT-style MLM to sentiment transfer via mask-and-infill; the empirical case is decent, but the core masking assumption is never directly tested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a two-stage pipeline. The mask stage fuses a frequency-ratio n-gram vocabulary with an attention-based sentiment classifier: each candidate marker word or phrase gets a salience score $s(u,a)=s_c(u,a)p$, the classifier probability filtering out fake markers, and tokens above a threshold are masked. The infill stage converts a pretrained masked language model into an Attribute Conditional Masked Language Model (AC-MLM) by replacing segment embeddings with attribute embeddings, then trains it with reconstruction loss on the original masked words plus a discriminative loss from a pretrained classifier on the transferred sentence, back-propagated through soft-sampling. The objective $L=L_{\text{rec}}+\eta L_{\text{acc}}$ lets the two losses trade attribute accuracy against content preservation.
What would settle it
A reader could test whether the mask step genuinely isolates sentiment by annotating sentiment-bearing spans in, say, 500 Yelp or Amazon sentences, running the fusion masker, and measuring recall of those spans; if recall is low, or if sentences whose sentiment is carried by negation or syntax ("not bad at all") transfer poorly, the central assumption is falsified.
Extended reading notes
Core claim
The central discovery is that sentiment transfer can be reduced to finding and masking the sentiment-bearing tokens and then letting a masked language model, conditioned on the target sentiment, fill in what was removed. The paper reports that its full model, AC-MLM-SS with the fusion masker, reaches 97.3% transfer accuracy and 14.4 BLEU on Yelp and 84.5% accuracy with 28.5 BLEU on Amazon under the standard evaluation tools, the best joint accuracy-content scores among the compared systems. The base AC-MLM already produces the highest BLEU in the comparison, but its accuracy is poor; adding a pretrained classifier as a discriminator, trained through soft-sampling, pushes accuracy up sharply at a small BLEU cost. The paper reads this as evidence that bidirectional pretrained language models can replace RNN encoder-decoders for this kind of controlled generation.
Load-bearing premise
The load-bearing premise is that sentiment lives in a small set of maskable words or phrases, so that cutting them out leaves a content template that can be refilled for the opposite sentiment.
Editorial extensions
If this is right
- Non-parallel style transfer can be built on pretrained bidirectional models instead of RNN encoder-decoders, since the generator only needs to fill gaps in an otherwise preserved template.
- A fusion masker that combines frequency-ratio dictionaries with attention filtering should beat either masking method alone, especially on noisier datasets where the dictionary contains false markers.
- The reconstruction-versus-discrimination loss gives an explicit weighting parameter ($\eta$) for tuning the trade-off between target-attribute accuracy and content preservation.
- Because the approach explicitly copies unmodified content tokens and only generates at masked positions, it inherits the pointer-generator-style benefit of high content overlap without training a copy mechanism.
- The framework is expected by the authors to carry over to fine-grained (more than two) sentiment attributes and to other natural language generation tasks.
Reading between the lines
- A stress test for the approach would be sentences where sentiment is carried by syntax or negation rather than local words (e.g., "not bad at all", "the movie was good, which surprised me"); the paper does not report how the masker handles such cases.
- The frequency-ratio times classifier-probability scoring at the heart of the fusion masker could be reused as a cheap way to build attribute lexicons for low-resource style or register transfer.
- The reported BLEU gains may partly reflect that unmasked source tokens are copied verbatim; a content-preservation metric based on meaning rather than token overlap could change how large the apparent gain is.
- A testable extension is to control the mask set externally—for example, mask words supplied by a human editor or a different attribute detector—and use the same AC-MLM infill stage, which would isolate how much of the gain comes from masking versus infilling.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-stage approach to non-parallel sentiment transfer. In the mask stage, sentiment attribute markers are identified by a fusion of a frequency-ratio method and an attention-based classifier, and those markers are masked out. In the infill stage, a pre-trained BERT model is adapted into an Attribute Conditional Masked Language Model (AC-MLM) that fills the masked positions conditioned on the context and a target sentiment label; a variant called AC-MLM-SS additionally uses a pre-trained discriminator and soft-sampling to improve attribute compatibility. Experiments on Yelp and Amazon report automatic accuracy/BLEU scores, qualitative examples, and a three-annotator human evaluation, with the fusion-mask AC-MLM-SS variant achieving the best combined accuracy/BLEU on both datasets. The main claim is that the proposed Mask and Infill framework improves state-of-the-art sentiment transfer performance.
Significance. If the results hold, the paper makes a useful contribution by showing that pre-trained masked language models can be adapted to generative style transfer via a simple mask-and-infill mechanism, achieving high content preservation (BLEU) while maintaining competitive attribute accuracy. The fusion masking method is a reasonable synthesis of existing attention- and frequency-based marker identification, and the AC-MLM with soft-sampling is a clean way to inject discriminator guidance into discrete token generation. The paper also provides an instructive analysis of the trade-off between reconstruction and discrimination losses. However, the lack of mask-level validation, absent significance testing, and unclear independence between the training discriminator and evaluation classifiers materially weaken the confidence in the state-of-the-art claim.
major comments (4)
- [Section 3.2, Tables 2 and 3] The core mechanism—that masking the identified attribute markers leaves a sentiment-neutral content template—is never directly validated. The paper reports no precision or recall of the marker vocabulary V against human-annotated sentiment markers, no measurement of residual sentiment in the masked templates, and no ablation over the thresholds γ, γ_c, or the attention cutoff. This is load-bearing because if sentiment-bearing tokens remain unmasked, the infill step cannot correct them, and the reported advantage of fusion over attention on Amazon (75.9 to 84.5 accuracy at identical BLEU in Table 2) cannot be attributed to mask quality. Please add a mask-level evaluation (e.g., human-annotated markers on a held-out sample, coverage/precision, and sentiment-classifier accuracy on masked templates) and a threshold sensitivity analysis.
- [Section 4.4, Tables 2 and 3] The state-of-the-art claim rests on single-run comparisons without variance or significance testing. Several key differences are small (Yelp: fusion vs. attention accuracy 97.3 vs. 97.3 and BLEU 14.4 vs. 14.1; Table 3: accuracy 97.3 vs. 98.5 and BLEU 20.7 vs. 20.5). Without multiple seeds, standard deviations, or paired significance tests, readers cannot judge whether these differences are meaningful. Please report variance over at least three runs or provide paired bootstrap tests on the test sets.
- [Sections 4.3 and 4.4, Eq. (9)] The relationship between the CNN discriminator used to compute Lacc in Eq. (9) and the classifiers used to report ACC in Tables 2 and 3 is not specified. If the discriminator and the ACC evaluator share architecture or training data, the accuracy gains from AC-MLM-SS could partly reflect overfitting to the evaluation classifier. Please clarify whether the evaluation classifiers are independent of the training discriminator, and if not, report accuracy with a held-out classifier not used during training.
- [Table 5] The human evaluation uses only three annotators, reports no inter-annotator agreement statistics, and compares only against DeleteAndRetrieval rather than all strong baselines. The claim that fusion-method AC-MLM-SS is best on grammaticality, content preservation, and attribute match is therefore under-supported. Please report agreement statistics (e.g., Krippendorff's alpha or Cohen's kappa) and expand the comparison or temper the claim accordingly.
minor comments (5)
- [Section 4.3] The hyperparameters selected by grid search (η, τ, and the various thresholds) are never reported, which hampers reproducibility; please include the selected values or a link to code.
- [Table 4] The example output contains formatting artifacts ('textcolorbluelove') and the human reference drops the second clause of the source, making the example less informative; please clean and choose a more representative example.
- [Algorithm 1] The condition 'if S is too short or S is the same as S' is ambiguous, since 'same as S' likely means no tokens were masked; please clarify the intended fallback conditions.
- [Equation (1)] The smoothed frequency-ratio in Eq. (1) uses a denominator that sums only over a' ≠ a, which is an unusual choice; please justify this definition or clarify that it is intentional.
- [Introduction and Figure 3] Figure 3 is referenced in the analysis but the figure itself is not discussed in detail; please describe how the trade-off curve was generated and what hyperparameter values produced it.
Circularity Check
No significant circularity: the central performance claims are evaluated on held-out test sets with external evaluation tools, and no derived parameter is used as its own prediction.
full rationale
The paper's derivation chain is self-contained with respect to its empirical claims. The Mask module builds an attribute-marker vocabulary from training-set frequency ratios and an attention classifier (Eq. 1 and Eq. 6), and the Infill module trains AC-MLM with a reconstruction loss (Eq. 7) plus a discriminator constraint (Eq. 9). Neither of these ingredients is defined in terms of the paper's outcome variables: the automatic metrics (accuracy and BLEU) are computed with the external tools of Li et al. (2018) and Yang et al. (2018) on held-out test sets, and the human evaluation is performed by three non-author annotators. The discriminator used during training is constrained on the training split, which is standard practice and does not make the test-set evaluation tautological. The paper contains no self-citations that carry a load-bearing premise, and its 'Mask and Infill' framing is a new modeling combination rather than a renaming of an existing result. The untested assumption that sentiment is carried by a sparse set of maskable tokens is a legitimate correctness or robustness risk, but it is not a circularity: failing to validate that assumption would make the method potentially ineffective, not its reported results true by construction.
Assumptions & free parameters
free parameters (6)
- smoothing parameter lambda =
1
- candidate vocabulary threshold gamma_c =
not specified
- marker vocabulary threshold gamma =
not specified
- balancing parameter eta =
grid-selected
- soft-sampling temperature tau =
grid-selected
- minimum content token count =
5
assumptions (4)
- domain assumption Sentiment is carried by a small set of surface tokens that can be masked out, leaving a sentiment-neutral template.
- domain assumption BERT's masked language model provides a strong enough linguistic prior to generate attribute-appropriate words at masked positions.
- domain assumption Training with reconstruction of originally masked words from the same sentence's content and attribute is sufficient to learn infilling for unseen target attributes.
- domain assumption The attention-based classifier trained on the same data gives a reliable measure of how 'sentimental' a word is.
Cite this review
Pith. "Pith review of "Mask and Infill" : Applying Masked Language Model to Sentiment Transfer." pith.science (2026). https://pith.science/paper/O2LA2A4N
@misc{pith2026190808039,
author = {Pith},
title = {Pith review of: "Mask and Infill" : Applying Masked Language Model to Sentiment Transfer},
year = {2026},
howpublished = {\url{https://pith.science/paper/O2LA2A4N}},
note = {Machine review of arXiv:1908.08039}
}
read the original abstract
This paper focuses on the task of sentiment transfer on non-parallel text, which modifies sentiment attributes (e.g., positive or negative) of sentences while preserving their attribute-independent content. Due to the limited capability of RNNbased encoder-decoder structure to capture deep and long-range dependencies among words, previous works can hardly generate satisfactory sentences from scratch. When humans convert the sentiment attribute of a sentence, a simple but effective approach is to only replace the original sentimental tokens in the sentence with target sentimental expressions, instead of building a new sentence from scratch. Such a process is very similar to the task of Text Infilling or Cloze, which could be handled by a deep bidirectional Masked Language Model (e.g. BERT). So we propose a two step approach "Mask and Infill". In the mask step, we separate style from content by masking the positions of sentimental tokens. In the infill step, we retrofit MLM to Attribute Conditional MLM, to infill the masked positions by predicting words or phrases conditioned on the context1 and target sentiment. We evaluate our model on two review datasets with quantitative, qualitative, and human evaluations. Experimental results demonstrate that our models improve state-of-the-art performance.
Figures
Reference graph
Works this paper leans on
-
[1]
Neural machine translation by jointly learning to align and translate
[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings,
work page 2015
-
[4]
Bert: Pre-training of deep bidirectional transformers for language understand- ing
[Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805,
arXiv 2018
-
[6]
Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering
[He and McAuley, 2016] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. pages 507– 517,
work page 2016
-
[9]
Toward controlled generation of text
[Hu et al., 2017] Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. Toward controlled generation of text. pages 1587–1596,
work page 2017
-
[11]
Convolutional neural networks for sentence classification
[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Process- ing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL , pages 1746–1751,
2014
-
[12]
Delete, retrieve, generate: a simple approach to sentiment and style transfer
[Li et al., 2018] Juncen Li, Robin Jia, He He, and Percy Liang. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volu...
work page 2018
-
[13]
Content preserving text generation with attribute controls
[Logeswaran et al., 2018] Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. Content preserving text generation with attribute controls. pages 5108–5118,
work page 2018
-
[14]
Bleu: a method for auto- matic evaluation of machine translation
[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for auto- matic evaluation of machine translation. pages 311–318,
work page 2002
Show all 24 references
-
[16]
Improving language understanding with unsupervised learning
[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. Improving language understanding with unsupervised learning. Technical re- port, Technical report, OpenAI,
2018
-
[17]
Liu, and Christo- pher D
[See et al., 2017] Abigail See, Peter J. Liu, and Christo- pher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL 2017, Vancouver, Canada, July 30 - Aug...
2017
-
[18]
Style transfer from non-parallel text by cross-alignment
[Shen et al., 2017] Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. Style transfer from non-parallel text by cross-alignment. pages 6830–6841,
2017
-
[19]
Sequence to sequence learning with neural networks
[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. pages 3104–3112,
2014
-
[20]
cloze procedure
[Taylor, 1953] Wilson L Taylor. “cloze procedure”: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433,
1953
-
[22]
Unpaired sentiment-to-sentiment translation: A cycled re- inforcement learning approach
[Xu et al., 2018] Jingjing Xu, Sun Xu, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. Unpaired sentiment-to-sentiment translation: A cycled re- inforcement learning approach. pages 979–988,
2018
-
[23]
Unsupervised text style transfer using language models as discrimina- tors
[Yang et al., 2018] Zichao Yang, Zhiting Hu, Chris Dyer, Eric P Xing, and Taylor Berg-Kirkpatrick. Unsupervised text style transfer using language models as discrimina- tors. pages 7298–7309,
2018
-
[24]
Text infilling
[Zhu et al., 2019] Wanrong Zhu, Zhiting Hu, and Eric Xing. Text infilling. CoRR, abs/1901.00158, 2019
2019 arXiv
-
[1953]
Bow- man
[Wang et al., 2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bow- man. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Proceedings of the Workshop: Analyzing and Interpreting Neural Net- works ...
2018
-
[1997]
Universal language model fine-tuning for text clas- sification
[Howard and Ruder, 2018] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text clas- sification. 1:328–339,
2018
-
[2002]
[Prabhumoye et al., 2018] Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W. Black. Style transfer through back-translation. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics, ACL 2018, Melbourne, Australia, July 15...
2018
-
[2014]
Semi- supervised sequence learning
[Dai and Le, 2015] Andrew M Dai and Quoc V Le. Semi- supervised sequence learning. pages 3079–3087,
2015
-
[2015]
Learning phrase representations using RNN encoder-decoder for statistical machine translation
[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, C ¸ aglar G¨ulc ¸ehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Confer-...
2014
-
[2016]
Long short-term memory
[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and J¨urgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780,
1997
-
[2017]
Arbitrary style transfer in real-time with adaptive instance normalization
[Huang and Belongie, 2017] Xun Huang and Serge Be- longie. Arbitrary style transfer in real-time with adaptive instance normalization. pages 1501–1510,
2017
-
[2018]
Style transfer in text: Ex- ploration and evaluation
[Fu et al., 2018] Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Ex- ploration and evaluation. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI- 18), New Orleans, Louisiana, USA, February 2-7, 2018 ,...
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.