SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation

Micha{\l} Pere{\l}kiewicz; Rafa{\l} Po\'swiata; S{\l}awomir Dadas

REVIEW 4 major objections 7 minor 89 references

SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that prepending a sentence-level semantic embedding as the initial token in autoregressive causal language modeling lets a GPT-2-sized model learn paraphrase generation from unpaired sentences, matching supervised systems…

desk verdict SMCLM is a clean, useful self-supervised paraphrase method with released artifacts, but the headline SOTA claim rests on unvalidated, test-tuned metrics, so the evaluation needs repair, not the core idea. read the letter →

arxiv 2507.03415 v1 pith:4WZ4MBAY submitted 2025-07-04 cs.CL

Micha{l} Pere{l}kiewicz , S{l}awomir Dadas , Rafa{l} Poświata This is my paper

classification cs.CL

keywords paraphrasegenerationsemanticembeddingsautoregressivelanguagemodelscausalmodelingself-supervisedlearningtextBERT-iBLEUevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces SMCLM, a training procedure that injects a sentence-level semantic embedding as the first token when training an autoregressive language model with the standard causal language modeling objective. The claim is that this simple change lets a GPT-2-sized model produce paraphrases without any aligned sentence pairs, and that in experiments it beats unsupervised baselines and is comparable to supervised ones. The paper also argues that common metrics like BLEU, ROUGE, and BERTScore misjudge paraphrase quality, and proposes a set of metrics, especially BERT-iBLEU and SBERT-iBLEU, that balance semantic similarity with lexical divergence. If true, it offers a data-efficient route to paraphrase generation and a stronger evaluation standard.

What carries the argument

The load-bearing object is the sentence-semantic embedding $h(s)$ that is prepended to the token sequence, replacing the embedding of the first special token during training and generation. Concretely, the training sequence becomes $\{h(s), x_1, \ldots, x_T\}$ and the model learns $P(x_t \mid h(s), x_{<t})$, so the autoregressive model conditions on the meaning of the whole sentence rather than on its surface form. This lets the standard causal language modeling objective (next-token prediction on unpaired text) teach the model to generate different wordings for the same meaning. The evaluation machinery consists of the BERT-iBLEU and SBERT-iBLEU measures, which combine a semantic similarity term with an inverse lexical similarity term $1 - \text{BLEU}$ under a scaling factor $\beta = 2$ to penalize copying the source phrase.

What would settle it

A human study in which annotators rate the semantic equivalence and fluency of paraphrases from SMCLM and the baselines would settle the issue: if SMCLM's ranking under BERT-iBLEU and SBERT-iBLEU disagrees with human preference, the central claim is falsified. Alternatively, computing the correlation of these metrics against human judgments on a held-out set would directly test their validity.

Watch

Extended reading notes

Core claim

The central claim is that an autoregressive language model can learn paraphrase generation from plain unpaired sentences by prepending the sentence's semantic embedding, computed by a pretrained sentence encoder, to the token sequence and training with the usual next-token objective. At generation, the same semantic embedding is used as the initial context, so the model is conditioned on meaning rather than on a particular surface form. According to the experiments, this makes the model generate paraphrases that are semantically faithful and lexically diverse, outperforming state-of-the-art unsupervised methods and reaching levels comparable to supervised approaches. The paper also claims that BLEU, ROUGE, and BERTScore are unreliable for paraphrase evaluation and that the proposed BERT-iBLEU and SBERT-iBLEU metrics are a better standard.

Load-bearing premise

The evaluation metrics used to judge the models are assumed to be faithful to human judgment of paraphrase quality, but the paper never validates them against human ratings.

Editorial extensions

If this is right

An autoregressive language model trained with SMCLM can generate paraphrases from a plain sentence corpus, without any aligned paraphrase pairs.
On QQP, MSCOCO, and CNN News, SMCLM achieves higher BERT-iBLEU and SBERT-iBLEU scores than the tested unsupervised methods, including back-translation and CGMH.
The best SMCLM model is comparable to supervised baselines and to large language model references on semantic similarity and fluency.
BLEU and ROUGE scores can mislead in paraphrase evaluation, because they reward lexical overlap with the source even when the output is not a genuine paraphrase.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same prepended-embedding mechanism could be transferred to other conditional text generation tasks, such as style transfer or keyphrase generation, provided a suitable semantic or style embedding exists.
Because the method is self-supervised, the amount of paraphrase quality it achieves likely scales with the diversity and size of the unpaired corpus, so a larger corpus than the ten million sentences used could improve results further.
If the proposed metrics were validated against human ratings, the paper's claim that BLEU, ROUGE, and BERTScore misjudge paraphrase quality would have broad implications for how paraphrase systems are compared in the literature.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this one with the evaluation section flagged, not the method. The actual training idea is simple and sound: prepend a frozen sentence embedding from a semantic encoder to the token sequence, train GPT-2 with the standard CLM objective, and the model learns to paraphrase from unaligned text. That is a genuinely useful contribution for data augmentation and low-resource settings, and the authors did the right thing by releasing code, a pretrained model, and a 10M-sentence corpus. They also reproduced several baselines rather than quoting numbers, which deserves credit.

Where the paper gets wobbly is the evaluation layer, and the stress-test note has it right. The claim that SMCLM beats unsupervised SOTA and matches supervised methods is carried entirely by BERT-iBLEU and SBERT-iBLEU. Those metrics are never validated against human judgments of paraphrase adequacy or fluency, which matters because the paper simultaneously argues that BLEU/ROUGE/BERTScore are unreliable. You cannot make that argument with another unvalidated metric.

The more concrete problem is leakage in the metric construction. The beta=2 scaling factor is fit on the reference pairs of the very same QQP, MSCOCO, and CNN News sets used for evaluation (Appendix E). Table 8 shows the headline scores move by about two points when beta goes from 1 to 5, so the choice is not immaterial. On top of that, Section IV-C selects the best of five candidates per test sentence using the headline metric itself, so the reported numbers are upper envelopes of that metric. That is a real circularity, and it directly weakens the SOTA and metric-reliability claims. It does not kill the method, though; the training procedure is self-contained and the paraphrases in Tables 9 and 10 look plausible, with the usual caveats about named entities and numbers, which the authors themselves acknowledge in the limitations.

Minor but worth saying: no error bars anywhere, and the CNN News references are ChatGPT-generated, which is fine for evaluation but should be stated more prominently as a potential confound.

Overall: the core method is a keeper, and the paper deserves a serious referee. I would ask for a human-evaluation correlation study for the proposed metrics, beta chosen on a held-out set or fixed a priori, and candidate selection decoupled from the headline metric. If those changes come back, the practical case for SMCLM stands on its own. Send it out.

Referee Report

4 major / 7 minor

Summary. SMCLM is a self-supervised training procedure for autoregressive paraphrase generation. The method prepends a frozen sentence embedding (computed by a Sentence-Transformer) as the initial token representation in a causal language model (GPT-2) and trains on plain text with the standard next-token objective. At inference, the same embedding is fed to the model, which generates paraphrases via diverse beam search. The paper evaluates three SMCLM variants (trained on the per-dataset corpora, on a new 10M-sentence corpus, and with fine-tuning) on QQP, MSCOCO, and CNN News, comparing with supervised (DiPS, QCPG, GPT-2, T5 v1.1) and unsupervised (BackTransl, CorruptLM, CGMH) baselines plus ChatGPT reference outputs. The authors report a battery of metrics, including two new lexically-dependent semantic similarity measures, BERT-iBLEU and SBERT-iBLEU, and use them as the basis for their principal claims: SMCLM achieves state-of-the-art results among unsupervised methods and is comparable to supervised approaches. The paper also proposes these new metrics as a standard for paraphrase evaluation and argues that BLEU, ROUGE, and BERTScore are unreliable. Code, model weights, and a 10M-sentence corpus are released.

Significance. If the empirical claims hold, SMCLM would be a simple, data-efficient route to paraphrase generation that does not require aligned data, with clear practical value for data augmentation and low-resource languages. The method itself is plausible and the training procedure is genuinely self-contained: the model generates from a frozen embedding, and the causal LM objective is used without circular use of evaluation data. The authors also contribute reproducible artifacts (code, model checkpoint, 10M corpus) and reproduce comparison methods. The principal weakness is that the headline claims are carried by two new metrics that are (i) calibrated on the same test sets used for evaluation and (ii) never validated against human judgments. Without a meta-evaluation showing that BERT-iBLEU/SBERT-iBLEU track human quality, the superiority claims and the dismissal of BLEU/ROUGE/BERTScore are not yet supported.

major comments (4)

[IV-D, Appendix E] Section IV-D and Appendix E: the scaling factor beta=2 in BERT-iBLEU/SBERT-iBLEU is derived from the average ratio of BERT/SBERT to iBLEU computed over the QQP, MSCOCO, and CNN News reference pairs (Table 7), which are the same test sets used for the evaluation in Table 3. Table 8 shows that the metric is sensitive to this choice: SBERT-iBLEU for SMCLM-10M-ft ranges from 81.61 at beta=5 to 83.87 at beta=1. Because the paper does not show that the ranking of methods in Table 3 is stable across beta, the headline claim that SMCLM outperforms existing unsupervised methods rests on a test-calibrated metric. The authors should set beta on a separate validation set or demonstrate ranking stability over a range of beta values.
[IV-D, VIII] The paper states that BLEU, ROUGE, and BERTScore are unreliable (Section IV-D) and proposes BERT-iBLEU/SBERT-iBLEU as a new standard (Section VIII), but it never validates these metrics against human judgments of paraphrase quality. The entire comparative evaluation in Table 3 and the central claims of state-of-the-art unsupervised performance and parity with supervised methods depend on these unvalidated measures. The authors should conduct a human evaluation on a sample of outputs (e.g., adequacy, fluency, lexical diversity) and report the correlation of each automatic metric with the human ratings, as is standard in NLG meta-evaluation.
[IV-C] In Section IV-C, the best of five generated candidates is selected using the same SBERT-iBLEU score that is subsequently used as the headline evaluation metric. The reported results are therefore upper envelopes of the proposed metric, and the selection procedure and the evaluation are conflated: if SBERT-iBLEU is not a faithful proxy for human quality, both the selected candidates and the final scores could be biased. The paper should either report aggregate scores over all five candidates, or use a selection criterion independent of the evaluation metric (e.g., the generation model's average log-probability), and verify that the conclusions are robust to the choice of selection method.
[Table 3, Section V] Table 3 reports no error bars, confidence intervals, or significance tests for any of the eleven metrics across the three datasets and eighteen models. Given that some headline differences are modest (e.g., BERT-iBLEU on QQP: 84.52 for SMCLM-10M-ft vs 87.67 for supervised GPT-2), the paper cannot establish that the observed rankings are reliable. The authors should provide variance estimates over multiple runs or bootstrap resampling and apply paired significance tests for the comparisons that support the main claims.

minor comments (7)

[Figure 1] The label 'Semantinc encoding and tokenization' contains a typo; it should read 'Semantic encoding and tokenization'.
[II-A] The sentence beginning 'A novel formulation of the encoder-decoder model learning problem...' describing DiPS appears twice in consecutive paragraphs; remove the duplicate.
[IV-A] The phrase 'We unsqueezed the splits into simple text corpora' is unclear; if the intended word is 'unsplit' or 'unlabeled', please rephrase.
[Header] The header line 'Digital Object Identifier 10.1 109/ACCESS.2024.0429000' appears to contain a malformed DOI with a space after '10.1'; if this is a template placeholder, remove or correct it.
[Table 10] In Table 10, the column header for the CorruptLM model is misspelled as 'CorrputLM'.
[Abstract vs. Conclusion] The abstract says the method is 'competitive with the supervised method' while Section VIII says 'comparable to the supervised approaches'; unify the phrasing.
[IV-D] The metric names oriBLEU and oriSBERT are introduced without defining the prefix 'ori' (presumably 'original'); please define them as source-oriented similarity measures at first use.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; SMCLM training is self-contained, though the evaluation layer carries external-validity risks that are not circular reductions.

full rationale

SMCLM's training procedure is self-contained: a frozen sentence encoder produces a semantic embedding h(s), that embedding is prepended to the token sequence, and the model is trained with the standard causal language modeling objective in Eq. 4. At inference the same encoder conditions generation, but the model is not trained to optimize any of the evaluation metrics, so the generated text is not constructed to equal the inputs. The comparison in Table 3 is based on reproduced baselines, including CopyInput and ChatGPT references, and the reported BERT-iBLEU/SBERT-iBLEU values are empirical outputs of the chosen metric, not algebraic consequences of the fitted beta. Two evaluation weaknesses are correctness risks rather than circularity: (i) beta=2 is calibrated on the same QQP, MSCOCO, and CNN News reference pairs used for evaluation (Appendix E), making the metric test-set-tuned and unvalidated against human judgments; (ii) the best candidate is selected by SBERT-iBLEU and then scored on the same metric family, so the reported values are upper envelopes over candidates. Neither forces the SMCLM ranking: the ranking is an empirical outcome under that metric and would need re-computation under a different beta or under a human-validated metric to be falsified. The paper's self-citations (e.g., [62], [65]) are related-work references and are not load-bearing for the central derivation.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method itself adds no new theoretical entities. The free parameters are mostly standard hyperparameters, but the beta coefficient and beam search settings are tuned on or near the evaluation data. The key axioms are domain assumptions about embedding semantics and the conditioning mechanism, plus reliance on external pretrained models.

free parameters (5)

beta in BERT-iBLEU/SBERT-iBLEU = 2
Set to 2 based on average ratios of BERT/SBERT to iBLEU computed over all input-reference pairs from QQP, MSCOCO, and CNN News (Appendix E, Table 7). This is fitted to the evaluation data and affects the reported headline scores.
diverse beam search hyperparameters = penalty=0.6, beams=5, groups=5, no_repeat_ngram=2
Selected through experiments described in Appendix B; the paper does not state clearly whether selection used validation or test data.
training hyperparameters (SMCLM) = lr=5e-6, epochs=8, batch=32
Chosen from tested ranges (Table 4) based on best results; standard model selection but still tuned hyperparameters.
training hyperparameters (GPT-2 supervised) = lr=1e-5, epochs=20, batch=16
Chosen from tested ranges (Table 4) for the supervised comparison model.
training hyperparameters (T5 v1.1) = lr=1e-6, epochs=10, batch=16
Chosen from tested ranges (Table 4) for the supervised comparison model.

assumptions (5)

domain assumption Sentences similar in meaning have similar sentence embeddings under cosine similarity.
Stated in Section III-B as the basis for the method.
domain assumption The pretrained paraphrase-mpnet-base-v2 encoder provides semantically meaningful representations without any fine-tuning.
The entire method relies on a frozen off-the-shelf encoder (Section IV-B, Appendix A).
domain assumption Prepending the semantic embedding as the first token (summed with positional embedding) effectively conditions autoregressive generation on sentence meaning.
Central mechanism of SMCLM; empirically validated only through the reported results (Section III-B, Appendix A).
domain assumption The external fluency model (parrot_fluency_model) gives valid fluency scores.
Fluency results in Table 3 depend on this model without in-paper validation (Section IV-D).
domain assumption BLEU-3, ROUGE-L, BERTScore, and SBERT cosine similarity are adequate operationalizations of their respective constructs.
These measures are used both as evaluation and, for BLEU/ROUGE/BERTScore, as targets of the paper's criticism, creating a tension (Section IV-D).

how reviews work

0 comments

Cite this review

Pith. "Pith review of SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation." pith.science (2026). https://pith.science/paper/4WZ4MBAY

@misc{pith2026250703415,
  author       = {Pith},
  title        = {Pith review of: SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4WZ4MBAY}},
  note         = {Machine review of arXiv:2507.03415}
}

read the original abstract

This article introduces semantically meaningful causal language modeling (SMCLM), a selfsupervised method of training autoregressive models to generate semantically equivalent text. Our approach involves using semantically meaningful text representation as an initial embedding in the autoregressive training and generation processes. The extensive empirical study demonstrates that the SMCLM approach makes autoregressive models capable of learning robust and high-quality paraphrase generation. The proposed method is competitive with the supervised method and achieves state-of-the-art results in unsupervised approaches. This article also presents a comprehensive set of automatic metrics that cover a wide range of autogenerated paraphrase evaluation aspects. Simultaneously, this article highlights the low reliability of the metrics that are widely used in paraphrase generation evaluation, including BLEU, ROUGE, and BERTScore.

Figures

Figures reproduced from arXiv: 2507.03415 by the authors.

**Figure 2.** shows an implementation scheme of our GPT2 model-based SMCLM approach. The semantic embedding of the sentence, encoded by the SentenceTransformers paraphrase-mpnet-base-v2 encoder, is passed to the GPT2 model. It replaces the first token’s embedding from the GPT-2 embedding layer - in the case of our model, the token <|endoftext|>, which is added during tokenization. The size of the sentence encoding embedding mu… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

89 extracted references · 73 canonical work pages

[1]

A. Syed, F. Gaol, and T. Matsuo, ‘‘A Survey of the State- of-the-Art Models in Neural Abstractive Text Summa- rization,’’ IEEE Access , vol. 9, pp. 13 248–13 265, 01 2021

2021
[2]

S. Wu, B. Chen, C. Xin, X. Han, L. Sun, W. Zhang, J. Chen, F. Y ang, and X. Cai, ‘‘From paraphrasing to semantic parsing: Unsupervised semantic parsing via synchronous semantic decoding,’’ in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V...

2021
[3]

Barancíková and A

P . Barancíková and A. Tamchyna, ‘‘Machine translation within one language as a paraphrasing technique online,’’ in Conference on Theory and Practice of Information Technologies, 2014. [Online]. Available: https://api.semanticscholar.org/CorpusID:15595460

2014
[4]

E. Okur, S. Sahay, and L. Nachman, ‘‘Data augmentation with paraphrase generation and entity extraction for multimodal dialogue system,’’ in Proceedings of the Thirteenth Language Resources and Evaluation Conference , N. Calzolari, F. Béchet, P . Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, a...

2022
[5]

X. Li, C. Tao, J. Gao, and H. Guo, ‘‘A review of quality assurance research of dialogue systems,’’ in 2022 IEEE International Conference On Artificial Intelligence Testing (AITest), 2022, pp. 87–94

2022
[6]

Zhou and S

J. Zhou and S. Bhat, ‘‘Paraphrase generation: A survey of the state of the art,’’ in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 5075–5086. [Online]. Available: https: //aclanthology.org/2021.emnlp-main.414

2021
[7]

Madnani and B

N. Madnani and B. J. Dorr, ‘‘Generating phrasal and sentential paraphrases: A survey of data-driven methods,’’ Computational Linguistics , vol. 36, no. 3, pp. 341–387, Sep. 2010. [Online]. Available: https: //aclanthology.org/J10-3003

2010
[8]

K. R. McKeown, ‘‘Paraphrasing questions using given and new information,’’ American Journal of Computational Linguistics, vol. 9, no. 1, pp. 1–10, 1983. [Online]. Available: https://aclanthology.org/J83-1001

1983

Show all 89 references

[9]

Lin and P

D. Lin and P . Pantel, ‘‘Discovery of inference rules for question-answering,’’ Natural Language Engineering , vol. 7, no. 4, p. 343–360, 2001

2001
[10]

Barzilay and L

R. Barzilay and L. Lee, ‘‘Learning to paraphrase: An unsupervised approach using multiple-sequence alignment,’’ in Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, 2003, pp. 16–23. [Onl...

2003
[11]

Koehn, H

P . Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst, ‘‘Moses: Open source toolkit for statistical machine translation,’’ in Proceedings of the 45th Annual Meeting of t...

2007
[12]

Prakash, S

A. Prakash, S. A. Hasan, K. Lee, V . Datla, A. Qadir, J. Liu, and O. Farri, ‘‘Neural paraphrase generation with stacked residual LSTM networks,’’ in Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. Osaka, Japan: The ...

2016
[13]

Gupta, A

A. Gupta, A. Agarwal, P . Singh, and P . Rai, ‘‘A deep gen- erative framework for paraphrase generation,’’ Proceed- ings of the AAAI Conference on Artificial Intelligence , vol. 32, 09 2017

2017
[14]

Kumar, S

A. Kumar, S. Bhattamishra, M. Bhandari, and P . Talukdar, ‘‘Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation,’’ in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu...

2019
[15]

Hosking, H

T. Hosking, H. Tang, and M. Lapata, ‘‘Hierarchical sketch induction for paraphrase generation,’’ in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers). Dublin, Ireland: Association for Compu- tational Linguistics, M...

2022
[16]

Huang and K.-W

K.-H. Huang and K.-W. Chang, ‘‘Generating syntacti- cally controlled paraphrases without using annotated parallel pairs,’’ in Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2021

2021
[17]

Iyyer, J

M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer, ‘‘Adversarial example generation with syntactically con- trolled paraphrase networks,’’ 01 2018, pp. 1875–1885

2018
[18]

Goyal and G

T. Goyal and G. Durrett, ‘‘Neural syntactic preordering 18 VOLUME 11, 2023 Michał Perełkiewicz et al.: SMCLM for Autoregressive Paraphrase Generation for controlled paraphrase generation,’’ in Proceedings of the 58th Annual Meeting of the Association for Computational Linguist...

2023
[19]

Z. Li, X. Jiang, L. Shang, and H. Li, ‘‘Paraphrase generation with deep reinforcement learning,’’ in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsujii, Eds. Brussels, Belgium: Associatio...

2018
[20]

Y ang, Z

Q. Y ang, Z. Huo, D. Shen, Y . Cheng, W. Wang, G. Wang, and L. Carin, ‘‘An end-to-end generative architecture for paraphrase generation,’’ in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Na...

2019
[21]

N. Miao, H. Zhou, L. Mou, R. Y an, and L. Li, ‘‘Cgmh: Constrained sentence generation by metropolis-hastings sampling,’’ Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6834–6842, 07 2019

2019
[22]

Xianggen, M

L. Xianggen, M. Lili, M. Fandong, Z. Hao, Z. Jie, and S. Song, ‘‘Unsupervised paraphrasing by simulated annealing,’’ in ACL, 2020

2020
[23]

Lewis, Y

M. Lewis, Y . Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V . Stoyanov, and L. Zettlemoyer, ‘‘BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,’’ in Proceedings of the 58th Annual Meeting of the Associa...

2020
[24]

Raffel, N

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P . J. Liu, ‘‘Exploring the limits of transfer learning with a unified text-to-text transformer,’’ 2023

2023
[25]

Radford, J

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, ‘‘Language models are unsupervised multitask learners,’’ 2019. [Online]. Available: https: //api.semanticscholar.org/CorpusID:160025533

2019
[26]

T. Niu, S. Y avuz, Y . Zhou, N. S. Keskar, H. Wang, and C. Xiong, ‘‘Unsupervised paraphrasing with pretrained language models,’’ in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. ...

2021
[27]

Kiros, Y

R. Kiros, Y . Zhu, R. Salakhutdinov, R. Zemel, A. Tor- ralba, R. Urtasun, and S. Fidler, ‘‘Skip-thought vectors,’’ Advances in Neural Information Processing Systems , vol. 28, 06 2015

2015
[28]

Pagliardini, P

M. Pagliardini, P . Gupta, and M. Jaggi, ‘‘Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features,’’ in NAACL 2018 - Conference of the North American Chapter of the Association for Computational Linguistics, 2018

2018
[29]

Conneau, D

A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes, ‘‘Supervised learning of universal sentence representations from natural language inference data,’’ in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , M. Palmer, R. Hwa, and ...

2017
[30]

D. Cer, Y . Y ang, S. yi Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. Guajardo-Cespedes, S. Y uan, C. Tar, Y .-H. Sung, B. Strope, and R. Kurzweil, ‘‘Uni- versal sentence encoder,’’ 2018

2018
[31]

Reimers and I

N. Reimers and I. Gurevych, ‘‘Sentence-bert: Sentence embeddings using siamese bert-networks,’’ in Conference on Empirical Methods in Natural Language Processing , 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:201646309

2019
[32]

T. Gao, X. Y ao, and D. Chen, ‘‘SimCSE: Simple contrastive learning of sentence embeddings,’’ in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Online and Punta Cana, Dominican R...

2021
[33]

K. Wang, N. Reimers, and I. Gurevych, ‘‘TSDAE: using transformer-based sequential denoising auto- encoderfor unsupervised sentence embedding learning,’’ in Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16...

2021 doi
[34]

Z. Li, X. Zhang, Y . Zhang, D. Long, P . Xie, and M. Zhang, ‘‘Towards general text embeddings with multi-stage contrastive learning,’’ 2023

2023
[35]

L. Wang, N. Y ang, X. Huang, B. Jiao, L. Y ang, D. Jiang, R. Majumder, and F. Wei, ‘‘Text embeddings by weakly- supervised contrastive pre-training,’’ 2022

2022
[36]

F. Feng, Y . Y ang, D. Cer, N. Arivazhagan, and W. Wang, ‘‘Language-agnostic bert sentence embedding,’’ 2020. VOLUME 11, 2023 19 Michał Perełkiewicz et al.: SMCLM for Autoregressive Paraphrase Generation

2020
[37]

Reimers and I

N. Reimers and I. Gurevych, ‘‘Making monolingual sentence embeddings multilingual using knowledge distillation,’’ in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2020. [Online]. Available...

2020 arXiv
[38]

J. Chen, S. Xiao, P . Zhang, K. Luo, D. Lian, and Z. Liu, ‘‘Bge m3-embedding: Multi-lingual, multi- functionality, multi-granularity text embeddings through self-knowledge distillation,’’ 2024. [Online]. Available: https://arxiv.org/abs/2402.03216

2024 arXiv
[39]

Günther, J

M. Günther, J. Ong, I. Mohr, A. Abdessalem, T. Abel, M. K. Akram, S. Guzman, G. Mastrapas, S. Sturua, B. Wang, M. Werk, N. Wang, and H. Xiao, ‘‘Jina embeddings 2: 8192-token general-purpose text embeddings for long documents,’’ 2024. [Online]. Available: https://arxiv.org/abs/...

2024 arXiv
[40]

J. Liu, D. Zhu, Z. Bai, Y . He, H. Liao, H. Que, Z. Wang, C. Zhang, G. Zhang, J. Zhang, Y . Zhang, Z. Chen, H. Guo, S. Li, Z. Liu, Y . Shan, Y . Song, J. Tian, W. Wu, and Z. Zhang, ‘‘A comprehensive survey on long context language modeling,’’ 03 2025

2025
[41]

Trinquier, G

J. Trinquier, G. Uguzzoni, A. Pagnani, F. Zamponi, and M. Weigt, ‘‘Efficient generative modeling of protein sequences using simple autoregressive models,’’ Nature Communications, vol. 12, 10 2021

2021
[42]

I. A. Bolshakov and A. Gelbukh, ‘‘Synonymous para- phrasing using wordnet and internet,’’ in Natural Lan- guage Processing and Information Systems , F. Meziane and E. Métais, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2004, pp. 312–323

2004
[43]

Kauchak and R

D. Kauchak and R. Barzilay, ‘‘Paraphrasing for automatic evaluation,’’ in Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, R. C. Moore, J. Bilmes, J. Chu-Carroll, and M. Sanderson, Eds. New Y ork City, USA: Association for Computational Li...

2006
[44]

Wubben, A

S. Wubben, A. van den Bosch, and E. Krahmer, ‘‘Paraphrase generation as monolingual translation: Data and evaluation,’’ in Proceedings of the 6th International Natural Language Generation Conference, J. Kelleher, B. M. Namee, and I. v. d. Sluis, Eds. Association for Computatio...

2010
[45]

Bowman, L

S. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio, ‘‘Generating sentences from a continuous space,’’ 01 2016, pp. 10–21

2016
[46]

L. Qian, L. Qiu, W. Zhang, X. Jiang, and Y . Y u, ‘‘Exploring diverse expressions for paraphrase generation,’’ in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (E...

2019
[47]

S. Garg, S. Prabhu, H. Misra, and G. Srinivasaraghavan, ‘‘Unsupervised contextual paraphrase generation using lexical control and reinforcement learning,’’ ArXiv, vol. abs/2103.12777, 2021. [Online]. Available: https: //api.semanticscholar.org/CorpusID:232335591

2021 arXiv
[48]

Cao and X

Y . Cao and X. Wan, ‘‘DivGAN: Towards diverse paraphrase generation via diversified generative adversarial network,’’ in Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, T. Cohn, Y . He, and Y . Liu, Eds. Online: Association for Computational Linguistic...

2020
[49]

W. John, M. Jonathan, and G. Kevin, ‘‘Learning para- phrastic sentence embeddings from back-translated bi- text,’’ in Proceedings of Empirical Methods in Natural Language Processing, 2017

2017
[50]

V aswani, N

A. V aswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, ‘‘Attention is all you need,’’ CoRR, vol. abs/1706.03762, 2017. [Online]. Available: http://arxiv.org/abs/1706.03762

2017 arXiv
[51]

S. Wang, R. Gupta, N. Chang, and J. Baldridge, ‘‘A task in a suit and a tie: paraphrase generation with semantic augmentation,’’ 2018

2018
[52]

Roy and D

A. Roy and D. Grangier, ‘‘Unsupervised paraphrasing without translation,’’ in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2...

2019
[53]

van den Oord, O

A. van den Oord, O. Vinyals, and K. Kavukcuoglu, ‘‘Neu- ral discrete representation learning,’’ in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17. Red Hook, NY , USA: Curran Associates Inc., 2017, p. 6309–6318

2017
[54]

Bandel, R

E. Bandel, R. Aharonov, M. Shmueli-Scheuer, I. Shnayderman, N. Slonim, and L. Ein-Dor, ‘‘Quality controlled paraphrase generation,’’ in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers) . Dublin, Ireland: Associati...

2022
[55]

J. Liu, J. Shi, J. Qi, L. Hou, J. Li, and Q. Tian, ‘‘ParaMac: A general unsupervised paraphrase generation framework leveraging semantic constraints and diversifying mechanisms,’’ in Findings of the Association for Computational Linguistics: EMNLP 2022 , Y . Goldberg, Z. Kozar...

2022
[56]

H. Palivela, ‘‘Optimization of paraphrase generation and 20 VOLUME 11, 2023 Michał Perełkiewicz et al.: SMCLM for Autoregressive Paraphrase Generation identification using language models in natural language processing,’’ International Journal of Information Management Data In...

2023
[57]

Radford and K

A. Radford and K. Narasimhan, ‘‘Improving language understanding by generative pre-training,’’ 2018. [Online]. Available: https://api.semanticscholar.org/ CorpusID:49313245

2018
[58]

Brown, B

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. ...

2020
[59]

Witteveen and M

S. Witteveen and M. Andrews, ‘‘Paraphrasing with large language models,’’ in Proceedings of the 3rd Workshop on Neural Generation and Translation . Hong Kong: Association for Computational Linguistics, Nov. 2019, pp. 215–220. [Online]. Available: https: //aclanthology.org/D19-5623

2019
[60]

Mikolov, K

T. Mikolov, K. Chen, G. S. Corrado, and J. Dean, ‘‘Efficient estimation of word representations in vector space,’’ in International Conference on Learning Representations, 2013. [Online]. Available: https://api. semanticscholar.org/CorpusID:5959482

2013
[61]

Iyyer, V

M. Iyyer, V . Manjunatha, J. Boyd-Graber, and H. Daumé III, ‘‘Deep unordered composition rivals syntactic methods for text classification,’’ inProceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on ...

2015
[62]

S. Dadas, ‘‘Training effective neural sentence encoders from automatically mined paraphrases,’’ in 2022 IEEE International Conference on Systems, Man, and Cyber- netics (SMC), 2022, pp. 371–378

2022
[63]

L. Xu, H. Xie, Z. Li, F. L. Wang, W. Wang, and Q. Li, ‘‘Contrastive learning models for sentence representations,’’ ACM Trans. Intell. Syst. Technol. , vol. 14, no. 4, jun 2023. [Online]. Available: https: //doi.org/10.1145/3593590

2023 doi
[64]

Muennighoff, N

N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, ‘‘MTEB: Massive text embedding benchmark,’’ in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, A. Vlachos and I. Augenstein, Eds. Dubrovnik, Croatia: Association for...

2023
[65]

Dadas, M

S. Dadas, M. Kozłowski, R. Poświata, M. Perełkiewicz, M. Białas, and M. Grębowiec, ‘‘A support system for the detection of abusive clauses in b2c contracts,’’ Artificial Intelligence and Law , Jun 2024. [Online]. Available: https://doi.org/10.1007/s10506-024-09408-8

2024 doi
[66]

W. Wang, G. Chen, H. Wang, Y . Han, and Y . Chen, ‘‘Multilingual sentence transformer as a multilingual word aligner,’’ ArXiv, vol. abs/2301.12140, 2023. [Online]. Available: https://api.semanticscholar.org/ CorpusID:256390411

2023 arXiv
[67]

C.-C. Lin, A. Jaech, X. Li, M. R. Gormley, and J. Eisner, ‘‘Limitations of autoregressive models and their alternatives,’’ in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, K. Tout...

2021
[68]

X. Shen, Z. Song, Y . Zhou, B. Chen, J. Liu, R. Zhang, R. A. Rossi, H. Tan, T. Y u, X. Chen, Y . Zhou, T. Sun, P . Zhao, Y . Wang, and J. Gu, ‘‘Numerical pruning for efficient autoregressive models,’’ Proceedings of the AAAI Conference on Artificial Intelligence , vol. 39, no....

2025
[69]

X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, ‘‘Self-supervised learning: Generative or contrastive,’’IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 1, pp. 857–876, 2023

2023
[70]

Lee, ‘‘A mathematical interpretation of autoregressive generative pre-trained transformer and self-supervised learning,’’ Mathematics, vol

M. Lee, ‘‘A mathematical interpretation of autoregressive generative pre-trained transformer and self-supervised learning,’’ Mathematics, vol. 11, no. 11, 2023. [Online]. Available: https://www.mdpi.com/2227-7390/ 11/11/2451

2023
[71]

T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P . Perona, D. Ramanan, P . Dollár, and C. L. Zitnick, ‘‘Microsoft coco: Common objects in context,’’ in Computer Vision – ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuyte- laars, Eds. Cham: Springer International Publishing,...

2014
[72]

Karpathy and L

A. Karpathy and L. Fei-Fei, ‘‘Deep visual-semantic alignments for generating image descriptions,’’ IEEE Transactions on Pattern Analysis and Machine Intelli- gence, vol. 39, pp. 664–676, 2014. [Online]. Available: https://api.semanticscholar.org/CorpusID:8517067

2014
[73]

A. See, P . J. Liu, and C. D. Manning, ‘‘Get to the point: Summarization with pointer-generator networks,’’ VOLUME 11, 2023 21 Michał Perełkiewicz et al.: SMCLM for Autoregressive Paraphrase Generation in Proceedings of the 55th Annual Meeting of the Association for Computatio...

2023
[74]

Tiedemann and S

J. Tiedemann and S. Thottingal, ‘‘OPUS-MT — Building open translation services for the World,’’ in Proceedings of the 22nd Annual Conferenec of the European Associa- tion for Machine Translation (EAMT) , Lisbon, Portugal, 2020

2020
[75]

C. V . Hegde and S. Patil, ‘‘Unsupervised paraphrase generation using pre-trained language models,’’ CoRR, vol. abs/2006.05477, 2020. [Online]. Available: https: //arxiv.org/abs/2006.05477

2006 arXiv
[76]

A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. J. Crandall, and D. Batra, ‘‘Diverse beam search: Decoding diverse solutions from neural sequence models,’’ ArXiv, vol. abs/1610.02424,

arXiv
[77]

Papineni, S

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, ‘‘Bleu: A method for automatic evaluation of machine translation,’’ in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , ser. ACL ’02. USA: Association for Computational Linguistics, 2002, p. 3...

2002
[78]

Lin, ‘‘ROUGE: A package for automatic evaluation of summaries,’’ in Text Summarization Branches Out

C.-Y . Lin, ‘‘ROUGE: A package for automatic evaluation of summaries,’’ in Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013

2004
[79]

Ng and V

J.-P . Ng and V . Abrecht, ‘‘Better summarization evaluation with word embeddings for ROUGE,’’ in Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing . Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 1925–1930. [On...

2015
[80]

Novikova, O

J. Novikova, O. Dušek, A. Cercas Curry, and V . Rieser, ‘‘Why we need new evaluation metrics for NLG,’’ in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 224...

2017
[81]

A. Chen, G. Stanovsky, S. Singh, and M. Gardner, ‘‘Evaluating question answering evaluation,’’ in Proceedings of the 2nd Workshop on Machine Reading for Question Answering . Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 119–124. [Online]. Availabl...

2019
[82]

Y . Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, and Y . Y u, ‘‘Texygen: A benchmarking platform for text generation models,’’ The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , 2018. [Online]. Available: https://api.semanticsc...

2018
[83]

Zhang, V

T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, ‘‘Bertscore: Evaluating text generation with BERT,’’ in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. [Online]. Available: htt...

2020
[84]

T. Niu, S. Y avuz, Y . Zhou, N. S. Keskar, H. Wang, and C. Xiong, ‘‘Unsupervised paraphrasing with pretrained language models,’’ in EMNLP, 2021

2021
[85]

J. Gu, Z. Lu, H. Li, and V . O. Li, ‘‘Incorporating copying mechanism in sequence-to-sequence learning,’’ in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), K. Erk and N. A. Smith, Eds. Berlin, Germany: Associat...

2016
[252]

Available: https://aclanthology.org/2020

[Online]. Available: https://aclanthology.org/2020. acl-main.22

2020
[2016]

Available: https://api.semanticscholar

[Online]. Available: https://api.semanticscholar. org/CorpusID:44614
[2021]

Available: https://www.sciencedirect

[Online]. Available: https://www.sciencedirect. com/science/article/pii/S2667096821000185
[6910]

Available: https://aclanthology.org/2021

[Online]. Available: https://aclanthology.org/2021. emnlp-main.552

2021

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

A. Syed, F. Gaol, and T. Matsuo, ‘‘A Survey of the State- of-the-Art Models in Neural Abstractive Text Summa- rization,’’ IEEE Access , vol. 9, pp. 13 248–13 265, 01 2021

2021

[2] [2]

S. Wu, B. Chen, C. Xin, X. Han, L. Sun, W. Zhang, J. Chen, F. Y ang, and X. Cai, ‘‘From paraphrasing to semantic parsing: Unsupervised semantic parsing via synchronous semantic decoding,’’ in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V...

2021

[3] [3]

Barancíková and A

P . Barancíková and A. Tamchyna, ‘‘Machine translation within one language as a paraphrasing technique online,’’ in Conference on Theory and Practice of Information Technologies, 2014. [Online]. Available: https://api.semanticscholar.org/CorpusID:15595460

2014

[4] [4]

E. Okur, S. Sahay, and L. Nachman, ‘‘Data augmentation with paraphrase generation and entity extraction for multimodal dialogue system,’’ in Proceedings of the Thirteenth Language Resources and Evaluation Conference , N. Calzolari, F. Béchet, P . Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, a...

2022

[5] [5]

X. Li, C. Tao, J. Gao, and H. Guo, ‘‘A review of quality assurance research of dialogue systems,’’ in 2022 IEEE International Conference On Artificial Intelligence Testing (AITest), 2022, pp. 87–94

2022

[6] [6]

Zhou and S

J. Zhou and S. Bhat, ‘‘Paraphrase generation: A survey of the state of the art,’’ in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 5075–5086. [Online]. Available: https: //aclanthology.org/2021.emnlp-main.414

2021

[7] [7]

Madnani and B

N. Madnani and B. J. Dorr, ‘‘Generating phrasal and sentential paraphrases: A survey of data-driven methods,’’ Computational Linguistics , vol. 36, no. 3, pp. 341–387, Sep. 2010. [Online]. Available: https: //aclanthology.org/J10-3003

2010

[8] [8]

K. R. McKeown, ‘‘Paraphrasing questions using given and new information,’’ American Journal of Computational Linguistics, vol. 9, no. 1, pp. 1–10, 1983. [Online]. Available: https://aclanthology.org/J83-1001

1983