Pith. sign in

REVIEW 5 major objections 5 minor 33 references

Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer

T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A Transformer-based delete-and-generate pipeline outperforms prior style-transfer systems on five datasets, and the paper proposes GLEU as a metric that tracks human preferences better than BLEU.

desk verdict A genuinely useful BERT-deletion + GPT-generation combination for non-parallel style transfer, but the headline claim overstates what the paper's own tables show. read the letter →

arxiv 1908.09368 v1 pith:SZCZYLFG submitted 2019-08-25 cs.CL cs.LG

classification cs.CLcs.LG
keywords textstyletransferdelete-retrieve-generateattention-basedattributedeletiongenerativetransformerGLEUmetricsentimentgenderpoliticalslant
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that text style transfer without parallel corpora can be done reliably by separating style words from content with a Transformer's own attention weights, then regenerating the sentence with a pre-trained generative Transformer. The authors build a Delete Transformer, which scores each token by how much attention the classifier's [CLS] token pays to it, deletes the top-scoring fraction, and retrieves target-style material from a corpus. A Generative Style Transformer then rewrites the remaining content in the target style, either blind to specific target attributes (B-GST) or guided by retrieved ones (G-GST). In human evaluations on sentiment, gender, and political-slant datasets, B-GST is preferred over the previous best systems for content, fluency, style match, and overall quality, and the paper proposes GLEU as an automatic metric that aligns with human judgments better than BLEU.

What carries the argument

The load-bearing components are three. The Delete Transformer is a pre-trained Transformer text classifier whose per-head, per-layer attention weights onto its [CLS] token serve as importance scores for each input token; the paper selects the single head-layer pair that, after removing the top $\gamma|x|$ tokens, most raises the classifier's confusion about the sentence's style (Eqs. 2 and 5). The Generative Style Transformer (GST) is a decoder-only Transformer language model, pre-trained on a large text corpus, that takes content, with or without retrieved target attributes, and the target style as input and generates the output sentence token by token. The two variants, B-GST (blind to target attributes) and G-GST (guided by retrieved attributes), share this machinery but differ in whether a Retrieve step supplies explicit target attribute tokens. A third component, the GLEU metric, scores outputs by n-gram overlap among source, reference, and generated sentence, letting it penalize wrongly changed content words while rewarding successful style edits.

What would settle it

Run B-GST on the Yelp sentiment task while replacing the chosen head-layer attention scores with random token scores that delete the same fraction of tokens; if human raters do not noticeably prefer the attention-based deletions for content preservation and style match, then the claim that attention identifies style attributes is not doing the work.

Watch

Extended reading notes

Core claim

The central discovery is that a Transformer classifier's per-head attention onto its [CLS] token can act as a deletion policy for style transfer. The Delete Transformer identifies style-bearing words by choosing the head-layer pair whose attention, when the highest-attended tokens are removed, most confuses the classifier about the sentence's style; the remaining tokens are treated as content. A decoder-only Transformer language model, pre-trained on a large unlabeled corpus, is then fine-tuned to reconstruct sentences from their own content, or from content plus retrieved or noisy attributes, using the target style as a prompt. The paper reports that in human evaluations on five datasets covering sentiment, gender, and political slant, its blind variant B-GST is preferred over the previous best systems for content preservation, fluency, style match, and overall quality, and that GLEU, an n-gram metric that compares source, reference, and output, correlates better with those human judgments than BLEU does.

Load-bearing premise

The method assumes that the attention weights from one chosen head-layer pair of the style classifier mark style words, not content words; if that fails, the delete step either throws away content or leaves style in place, and the pipeline cannot do both.

Editorial extensions

If this is right

  • On the Yelp, Amazon, and Captions datasets, human raters prefer B-GST over the previous Delete-and-Retrieve system for content, fluency, style match, and overall quality.
  • On the Political and Gender datasets, human raters prefer B-GST over back-translation for content and fluency.
  • Because B-GST keeps output length closely correlated with the source sentence length, the generated rewrites are less prone to the drift seen in earlier systems.
  • GLEU scores for B-GST are consistently the highest across datasets, which the paper reads as evidence that the model makes only necessary edits.
  • G-GST offers controllability: at inference the user can supply desired target attributes directly, bypassing the Retrieve step, which is useful when the target corpus lacks similar sentences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the head-layer attention selection generalizes, the same delete mechanism could be applied to formality, toxicity, or dialect rewriting without retraining a classifier on each new style pair.
  • The gap between B-GST and G-GST points to the Retrieve component as the bottleneck; a learned attribute predictor or retrieval over latent attribute vectors could restore controllability without sacrificing quality.
  • GLEU's fit with human ratings on these five datasets suggests style-transfer evaluation can follow grammatical-error-correction practice; a systematic correlation study across more models and human-judgement setups could settle which metric should be standard.
  • The selected head-layer pair and the deletion-rate parameter are tuned per dataset; a sensitivity analysis of these choices would tell whether the method's gains are robust or concentrated in a narrow operating point.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a non-parallel text style transfer system built on a Delete-Retrieve-Generate (DRG) framework. The Delete component uses attention weights from a fine-tuned BERT-based classifier (Delete Transformer) to identify and remove style attributes; the Retrieve component selects a similar target-style sentence via TF-IDF; the Generate component is a decoder-only GPT-based Transformer (GST) fine-tuned with a reconstruction objective, with two variants (B-GST and G-GST). The authors report human and automatic evaluations on Yelp, Amazon, Captions, Political, and Gender datasets, claiming state-of-the-art performance, and they propose GLEU as an automatic evaluation metric.

Significance. The idea of using attention from a single head-layer of a Transformer as style-importance scores is novel and potentially useful, and the use of a pre-trained GPT model for generation is a sensible extension of the DRG framework. The authors make code and data available, and they include a human evaluation of the delete mechanism. However, the significance is currently limited by evaluation weaknesses: key claims of style-transfer superiority are not consistently supported by the reported data, and the statistical basis of the human evaluation is not documented.

major comments (5)
  1. [Abstract and Section 4.3, Table 2] The abstract and Section 4.3 claim that the models 'outperform state-of-art systems across 5 datasets on sentiment, gender and political slant transfer.' Table 2 shows that on CAPTIONS, D&R is preferred over B-GST on target style match (47.75% vs 34%), so B-GST is not superior on the style-transfer axis for that dataset. For POLITICAL and GENDER, Section 4.1 states that annotators only rated content and fluency because target style strength is hard to judge; the only style signal for POLITICAL is the automatic accuracy in Table 5, where B-GST scores 71.2% versus BT's 88.0%. Thus the paper's own evidence contradicts the style-transfer superiority part of the claim, and the claim should be revised or additional style evaluations supplied.
  2. [Section 4.1, Tables 2 and 3] The human evaluation results in Tables 2 and 3 are reported as pairwise preference percentages with no information about the number of annotators, confidence intervals, or significance tests. For example, the 34% vs 47.75% difference on CAPTIONS style match cannot be assessed for reliability without such information. Please report annotator counts and at least approximate confidence intervals or a significance test for the main comparisons.
  3. [Abstract and Section 4.3] The abstract states that GLEU 'compare[s] better with human ratings than the predominantly used BLEU score,' but Section 4.3 explicitly says 'a detailed statistical correlation study is left for future work.' This is an unsupported claim; it should be backed by a correlation analysis (e.g., system-level or segment-level Spearman correlation with human judgments) or presented as a hypothesis rather than a finding.
  4. [Sections 2.1.1 and 2.3.2] The Delete Transformer is used both to select the head-layer pair for deletion (Eq. 5) and, during inference, to re-rank beam outputs by target-style match (Section 2.3.2). This means the automatic style accuracy in Tables 4 and 5 measures the model against the same classifier that selects its outputs, which can inflate style scores. Additionally, the deletion proportion gamma and smoothing parameter lambda are tuned per dataset, but no sensitivity analysis is provided, and the validity of the single head-layer assumption is only tested on 200 sentences. Please include a sensitivity analysis and clarify the consequences of the beam-selection procedure for the automatic style metric.
  5. [Section 4.2] The FastText style classifiers used for automatic style accuracy have accuracies of 80% on CAPTIONS and 82% on GENDER (Section 4.2). For a binary task, 80% accuracy means the classifier is wrong on 20% of examples, so automatic style accuracy on these datasets is a noisy measure. Since the POLITICAL and GENDER style comparisons rely on this metric, its limitations should be discussed and, where possible, supplemented with human style judgments.
minor comments (5)
  1. [Throughout] The paper repeatedly uses 'state-of-art' instead of 'state-of-the-art' (e.g., abstract, Section 1, Section 3.2); please fix this typo consistently.
  2. [Table 6] Table 6 refers to colored attributes, but the printed text does not show colors; use bold or underline to mark attributes.
  3. [Equation (2)] Equation (2) uses softmax over w to define α_{h,l}(w), which is a distribution over tokens; if this normalized value is used as an importance score, please state this explicitly and clarify that the score is not a raw attention weight.
  4. [Section 2.2] Section 2.2 states that TF-IDF retrieval gives 'best retrieval results' without presenting a comparison; please add a small table or a quantitative justification for this choice.
  5. [Section 4.3, Figure 2] Figure 2 is referenced in Section 4.3 but the actual plot is not visible in the manuscript text; please ensure the figure is included and legible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim rests on external human evaluation and independent FastText metrics; no prediction reduces to a fit.

full rationale

Walking the derivation chain, the paper's components are not defined in terms of the claimed outputs. The Delete Transformer assigns token importance scores from BERT attention (Eq. 2), selects a head-layer pair by minimizing classifier confusion on a validation set (Eqs. 4-5), and deletes the top-gamma tokens; gamma and the head-layer pair are hyperparameters selected on validation data, not fitted to the reported test outcomes. Both GST variants are trained with a reconstruction objective (Eqs. 7-8), which is a standard non-parallel training device rather than a way of encoding the target-style transfer result. The headline claim of outperforming state-of-the-art systems is supported by human preference judgments in Tables 2-3 and by automatic metrics from separately trained FastText classifiers and a separate GPT-2 perplexity model in Tables 4-5. The only self-referential element is that the Delete Transformer is also used at inference to re-rank the beam (Section 2.3.2, 'The output beam ... that obtains the highest target-style match score using the Delete Transformer described earlier, is chosen as the output sentence'), but this is not the paper's evidence for its claims; the central comparison uses independent human and FastText evaluations. The skeptical observation that the paper's own tables do not uniformly support the abstract's 'outperform across 5 datasets' claim (e.g., D&R wins on CAPTIONS target style match, and BT has higher automatic style accuracy on POLITICAL) is a correctness or evaluation-interpretation issue, not circularity. No uniqueness theorem is imported from the authors' prior work, no ansatz is smuggled in via self-citation, and no known empirical pattern is merely renamed. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The approach relies on two empirically justified inductive biases: style words are sparse (Li et al. 2018) and attention can explain classifier decisions (Feng et al. 2018). The paper's own contribution is mostly the specific integration, not new assumptions. We flag gamma and lambda as fitted hyperparameters whose values are not reported.

free parameters (4)
  • Deletion proportion gamma = Not reported; tuned per dataset
    Section 2.1.1: gamma controls how many top tokens are deleted as style attributes; the paper says it is tuned to each dataset but does not report the values.
  • Smoothing parameter lambda = Not reported
    Equation 4 uses lambda in the score z; the paper never states its value or tuning schedule.
  • G-GST noise rates = 10% / 5% / 5%
    Section 2.3.3: random attributes replace the original for 10% of examples, split 5% source style and 5% target style; chosen by hand.
  • Beam width and look-left window = 5 / 1
    Section 2.3.2: inference uses beam width 5 and look-left window 1; these affect output quality and are not tuned systematically.
assumptions (6)
  • standard math Transformer attention equations (Vaswani et al. 2017) and BERT pretraining assumptions hold
    Used in Equations 2 and 3 for attention weights and in the Delete Transformer and GST.
  • domain assumption Style attributes are localized to a small subset of words
    From Li et al. (2018), used to justify deleting top attention tokens rather than editing the whole sentence.
  • ad hoc to paper Attention weights of a single head-layer pair act as reliable style importance scores
    Section 2.1.1, Equation 5 selects the pair by minimizing a confusion score on a validation set; this is the core inductive step of the delete mechanism and is assumed to transfer to new sentences.
  • domain assumption Input reduction (Feng et al. 2018) is a valid interpretation method for attention
    The paper extends input reduction to BERT [CLS] attention, assuming that removing high-attention tokens lowers style classifier confidence.
  • domain assumption Target style corpora contain sentences whose retrieved attributes are compatible with source content
    Retrieve via TF-IDF (Equation 6) assumes proximity in TF-IDF space implies compatible attributes; the paper reports this is the weak point of G-GST.
  • domain assumption Style labels in the datasets are reliable enough for training classifiers and deletion
    Gender labels come from user markers and political labels from politician pages; sentiment from review scores; errors would propagate to deletion and generation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer." pith.science (2026). https://pith.science/paper/SZCZYLFG

@misc{pith2026190809368,
  author       = {Pith},
  title        = {Pith review of: Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SZCZYLFG}},
  note         = {Machine review of arXiv:1908.09368}
}
read the original abstract

Text style transfer is the task of transferring the style of text having certain stylistic attributes, while preserving non-stylistic or content information. In this work we introduce the Generative Style Transformer (GST) - a new approach to rewriting sentences to a target style in the absence of parallel style corpora. GST leverages the power of both, large unsupervised pre-trained language models as well as the Transformer. GST is a part of a larger `Delete Retrieve Generate' framework, in which we also propose a novel method of deleting style attributes from the source sentence by exploiting the inner workings of the Transformer. Our models outperform state-of-art systems across 5 datasets on sentiment, gender and political slant transfer. We also propose the use of the GLEU metric as an automatic metric of evaluation of style transfer, which we found to compare better with human ratings than the predominantly used BLEU score.

Figures

Figures reproduced from arXiv: 1908.09368 by the authors.

Figure 1
Figure 1. Our architecture, with an example from the Yelp dataset for the task of sentiment transfer [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Correlation of B-GST (ours, left) with in [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 12 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. http://dl.acm.org/citation.cfm?id=2969239.2969370 Scheduled sampling for sequence prediction with recurrent neural networks . In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pages 1171--1179, Cambridge, MA, USA. MIT Press

  4. [4]

    John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil

    Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. http://arxiv.org/abs/1803.11175 Universal sentence encoder

  5. [5]

    Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. http://arxiv.org/abs/1905.05621 Style transformer: Unpaired text style transfer without disentangled latent representation

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  7. [7]

    Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018. https://www.aclweb.org/anthology/D18-1407 Pathologies of neural models make interpretations difficult . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3719--3728, Brussels, Belgium. Association for Computa...

  8. [8]

    Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. In Thirty-Second AAAI Conference on Artificial Intelligence

Show all 33 references
  1. [9]

    Amir Globerson and Sam Roweis. 2006. https://doi.org/10.1145/1143844.1143889 Nightmare at test time: Robust learning by feature deletion . In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, pages 353--360, New York, NY, USA. ACM

  2. [10]

    Hongyu Gong, Suma Bhat, Lingfei Wu, Jinjun Xiong, and Wen mei Hwu. 2019. http://arxiv.org/abs/1903.10671 Reinforcement learning based text style transfer without parallel training corpus

  3. [11]

    Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. http://proceedings.mlr.press/v70/hu17e.html Toward controlled generation of text . In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine...

  4. [12]

    Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2018. Disentangled representation learning for non-parallel text style transfer

  5. [13]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427--431. Ass...

  6. [14]

    Jiwei Li, Will Monroe, and Daniel Jurafsky. 2016. Understanding neural networks through representation erasure. CoRR, abs/1612.08220

  7. [15]

    Juncen Li, Robin Jia, He He, and Percy Liang. 2018. https://doi.org/10.18653/v1/N18-1169 Delete, retrieve, generate: a simple approach to sentiment and style transfer . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Li...

  8. [16]

    Remi Mir, Bjarke Felbo, Nick Obradovich, and Iyad Rahwan. 2019. http://arxiv.org/abs/1904.02295 Evaluating style transfer for text

  9. [17]

    Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. https://doi.org/10.3115/v1/P15-2097 Ground truth for grammatical error correction metrics . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Internat...

  10. [18]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In ACL

  11. [19]

    Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style transfer through back-translation. In Proc. ACL

  12. [20]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training

  13. [21]

    Sudha Rao and Joel Tetreault. 2018. https://doi.org/10.18653/v1/N18-1012 Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association ...

  14. [22]

    Sravana Reddy and Kevin Knight. 2016. https://doi.org/10.18653/v1/W16-5603 Obfuscating gender in social media writing . In Proceedings of the First Workshop on NLP and Computational Social Science , pages 17--26, Austin, Texas. Association for Computational Linguistics

  15. [23]

    Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830--6841

  16. [24]

    Sandeep Subramanian, Guillaume Lample, Eric Michael Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2018. http://arxiv.org/abs/1811.00552 Multiple-attribute text style transfer

  17. [25]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf Attention is all you need . In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fe...

  18. [26]

    Jesse Vig. 2019. http://arxiv.org/abs/1904.02679 Visualizing attention in transformer-based language representation models

  19. [27]

    Rob Voigt, David Jurgens, Vinodkumar Prabhakaran, Dan Jurafsky, and Yulia Tsvetkov. 2018. RtGender : A corpus for studying differential responses to gender. In Proc. LREC

  20. [28]

    Williams and David Zipser

    Ronald J. Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks

  21. [29]

    Jingjing Xu, Xu SUN, Qi Zeng, Xiaodong Zhang, Xuancheng Ren, Houfeng Wang, and Wenjie Li. 2018. https://www.aclweb.org/anthology/P18-1090 Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach . In Proceedings of the 56th Annual Meeting of the As...

  22. [30]

    Cheng Yang, Maosong Sun, Xiaoyuan Yi, and Wenhao Li. 2018. https://www.aclweb.org/anthology/D18-1430 Stylistic C hinese poetry generation via unsupervised style disentanglement . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3...

  23. [31]

    Ye Zhang, Nan Ding, and Radu Soricut. 2018 a . https://doi.org/10.18653/v1/N18-1138 SHAPED : Shared-private encoder-decoder for text style adaptation . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human ...

  24. [32]

    Zhirui Zhang, Shuo Ren, Shujie Liu, Jianyong Wang, Peng Chen, Mu Li, Ming Zhou, and Enhong Chen. 2018 b . Style transfer as unsupervised machine translation. CoRR, abs/1808.07894

  25. [33]

    Ganbin Zhou, Ping Luo, Rongyu Cao, Fen Lin, Bo Chen, and Qing He. 2017. Mechanism-aware neural machine for dialogue response generation. In Thirty-First AAAI Conference on Artificial Intelligence

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.