REVIEW 3 major objections 4 minor 49 references
(Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper introduces PASTEL, a parallel corpus of 41,000 sentences and 8,300 stories rewritten across seven persona styles, and shows that parallel text improves style classification and supervised transfer.
desk verdict PASTEL is a real resource worth having, and the denotation study is a genuine methodological contribution, but the controlled-style classification claim is undercut by an annotator-leakage split. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the parallel annotation scheme with a shared denotation: crowd workers describe the same five-image story with local keywords, so every annotation refers to the same events, and each annotator's individual persona is the only free stylistic variable. A preliminary denotation experiment compares input settings by meaning-preservation metrics, such as METEOR and embedding similarity, and style-diversity metrics based on n-gram entropy, selecting images-with-local-keywords as the setting that balances the two. This design is what makes controlled classification and parallel supervised transfer possible.
What would settle it
Compute the same controlled style classification with a leave-one-annotator-out split that holds out every story from a given writer; if accuracy falls to chance for held-out writers, the reported persona signal is mostly individual idiolect rather than demographic style.
Extended reading notes
Core claim
PASTEL is the first large parallel stylistic dataset built around multiple personas in conjunction: gender, age, ethnicity, country, education, political view, and time of writing, with every text tied to a shared visual story so denotation stays constant. The paper's central claim is that with this resource, a classifier can predict one style, such as male versus female, while all other persona variables are fixed, isolating the target style's textual signature, and that a simple attentional sequence-to-sequence model using the parallel text outperforms unsupervised style-transfer baselines that rely on nonparallel corpora. The paper further claims that story-level prompts preserve meaning better and promote greater stylistic diversity than single reference sentences, and that the best input setting is a sequence of images with per-image keywords.
Load-bearing premise
The claim that external style variables are fixed assumes that the random train/test split by story does not let the same annotator's individual writing style appear in both sides, so the persona label, not the person, is what the classifier or transfer model learns.
Editorial extensions
If this is right
- Style classifiers can now be trained and evaluated with all non-target persona variables fixed, giving a cleaner measure of which demographic traits are actually readable in text.
- Supervised style transfer on parallel text beats unsupervised transfer, so nonparallel training data is not a harmless substitute when parallel data exists for the target styles.
- Because each story has multiple annotators, the dataset supports joint modeling of several persona styles at once rather than single-axis transfer.
- The finding that story-level context helps predict age and education but not gender or political view suggests different styles need different amounts of context.
- The word-level and embedding-based evaluation metrics disagree on parts of the style-transfer results, which points to the need for better meaning-preservation evaluation in style transfer.
Reading between the lines
- Editorial inference: an annotator-based split, holding out all stories from a given writer, would test whether the reported classification signal is demographic style or individual idiolect; the paper's split by story leaves this open.
- Editorial inference: the salient content words, such as 'food' and 'love', suggest that style and content are not cleanly separable, so PASTEL could be used to test content-invariant style representations directly.
- Editorial inference: the same denotation design could be extended to non-English or multimodal settings, where parallel style corpora are even scarcer.
- Editorial inference: the residual BLEU gap after supervised transfer, despite good soft-metric scores, may indicate that BLEU is inappropriate for style transfer where many valid target paraphrases exist.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces PASTEL, a crowd-sourced parallel stylistic corpus built from Visual Storytelling image stories. A denotation experiment identifies an input setting (images plus local keywords) that balances meaning preservation and stylistic diversity; annotators then write texts in their own self-reported persona across seven attributes. The authors evaluate two applications: controlled style classification, in which all other persona variables are supposedly fixed while one target is predicted, and supervised style transfer, in which a simple sequence-to-sequence model is compared with retrieval and unsupervised baselines. The paper reports human quality judgments in Appendix B and releases the dataset publicly.
Significance. If the protocol is sound, PASTEL would be a valuable resource: it is among the first multi-persona parallel style corpora, and the combination of a public release, a denotation experiment, and an independent human meaning-preservation study in Appendix B are concrete strengths. The supervised-versus-unsupervised comparison and the controlled classification setup are also potentially useful benchmarks. However, the two headline claims are conditional on resolving the split and control issues below; the resource itself, rather than the specific reported numbers, is the strongest contribution.
major comments (3)
- [§4.1 / §5.1, Table 3, Figure 4] The train/valid/test split is by story, not by annotator. With 501 annotators and an average of 9.97 HITs per annotator, each annotator contributes multiple stories; after a random 0.8/0.1/0.1 story-level split the same writer appears in both training and test with near-certainty. Since annotators are instructed to write in their own persona rather than to impersonate others, each writer's idiolect is a stable cue correlated with every persona label. The controlled classifiers in §5.1 can therefore exploit writer-identity features, inflating the macro-F scores in Figure 4 and contaminating both the difficulty ordering of styles and the feature-salience analysis in Table 5. The paper's own Appendix A shows that annotator behavior is detectable, since careless workers are manually blocked, confirming that writer-level signals are present. Please re-run the classification with an annotator-disjoint split, add annotator identity as a covariate, or otherwise report overlap statistics and show that the results are unchanged.
- [§5.1 Setup / Abstract / §1] The claim that 'other external style variables are controlled' is not supported by the experimental design as described. The setup fixes only gender, age, education, and politics, giving 2^3 = 8 combinations; the remaining three PASTEL persona styles, namely ethnicity, country, and time-of-day, are neither fixed nor mentioned in the classifier. The Abstract and §1 claim that all external variables are controlled. Please either include all seven styles in the controlled subsets or explicitly restrict the claim to the four selected variables and justify why the remaining three can be ignored.
- [§4.1 / Table 8] The reported collection statistics are internally inconsistent. Section 4.1 states that 501 annotators completed an average of 9.97 HITs with three stories per HIT, which implies about 14,985 story annotations; Table 8's category counts sum to 4,273 story annotations for each style; and Section 4.1 also reports 2.63 annotators per story over 8,310 stories, which implies about 21,855 annotations. These numbers cannot all be correct. Please reconcile the counts and clarify whether the reported '41K parallel sentences' refers to source sentences, annotation instances, or something else.
minor comments (4)
- [§1] The phrase 'ethnics' should be 'ethnicity' in the list of persona types.
- [Table 2] The entropy metric E(GM) is not defined; please state how the Gaussian-mixture n-gram entropy is computed.
- [§5.1, Features] The bullet list includes 'number of named entities' under both lexical and syntax features; please remove the duplicate.
- [Appendix B] There is a typo: 'we also conduct add Meaning Preservation human study' should read 'we also conduct a Meaning Preservation human study'.
Circularity Check
No circular derivation; PASTEL is an empirical dataset paper with held-out evaluation; the annotator-overlap issue is a validity concern, not circularity.
full rationale
PASTEL is an empirical resource paper, not a derivation chain whose conclusion is equivalent to its premises. The denotation experiment (Section 3) selects the story(images+local keywords) input setting by comparing automatic meaning-preservation and diversity metrics, and Section 5.2 later reuses overlapping automatic metrics (METEOR, VectorExtrema, Embedding Average) to score style-transfer outputs. That is a consistency of measurement choices, not a fitted parameter renamed as a prediction: no test-set parameter is fit using those metrics, and the supervised transfer model is evaluated on held-out parallel annotations. The controlled style classification claim (Section 5.1) fixes all labeled persona variables except the target by training separate classifiers per combination of the other persona labels, so the labeled style variables are genuinely held fixed. It does not control annotator identity, and because the split is by story rather than annotator (Section 4.1), shared writers across train/test could inflate classification scores; however, this is an external-validity confound, not circularity, since the classifier's output is not defined in terms of the labels it predicts. The paper also provides human quality ratings (Appendix B, Table 9) that are independent of the automatic metrics used in collection. Self-citations such as Hovy (1987) and Lin and Hovy (2003) are contextual only and not load-bearing. No specific circular step can be exhibited under the required standard.
Assumptions & free parameters
assumptions (4)
- domain assumption Crowd workers accurately self-report their demographic and persona attributes in the annotation scheme.
- domain assumption ViST reference sentences and image sequences provide a shared denotation that anchors meaning across annotators.
- domain assumption Automatic metrics (METEOR, VectorExtrema, embedding averaging) are valid proxies for meaning preservation and style quality.
- domain assumption Style can be meaningfully separated from content, and annotators' writing reflects their persona rather than the topic of the images.
Cite this review
Pith. "Pith review of (Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas." pith.science (2026). https://pith.science/paper/WOQONUD7
@misc{pith2026190900098,
author = {Pith},
title = {Pith review of: (Male, Bachelor) and (Female, Ph.D) have different connotations: Parallelly Annotated Stylistic Language Dataset with Multiple Personas},
year = {2026},
howpublished = {\url{https://pith.science/paper/WOQONUD7}},
note = {Machine review of arXiv:1909.00098}
}
read the original abstract
Stylistic variation in text needs to be studied with different aspects including the writer's personal traits, interpersonal relations, rhetoric, and more. Despite recent attempts on computational modeling of the variation, the lack of parallel corpora of style language makes it difficult to systematically control the stylistic change as well as evaluate such models. We release PASTEL, the parallel and annotated stylistic language dataset, that contains ~41K parallel sentences (8.3K parallel stories) annotated across different personas. Each persona has different styles in conjunction: gender, age, country, political view, education, ethnic, and time-of-writing. The dataset is collected from human annotators with solid control of input denotation: not only preserving original meaning between text, but promoting stylistic diversity to annotators. We test the dataset on two interesting applications of style language, where PASTEL helps design appropriate experiment and evaluation. First, in predicting a target style (e.g., male or female in gender) given a text, multiple styles of PASTEL make other external style variables controlled (or fixed), which is a more accurate experimental design. Second, a simple supervised model with our parallel text outperforms the unsupervised models using nonparallel text in style transfer. Our dataset is publicly available.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...
-
[4]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[5]
Shlomo Argamon, Moshe Koppel, James W Pennebaker, and Jonathan Schler. 2009. Automatically profiling the author of an anonymous text. Commun. ACM, 52(2):119--123
work page 2009
-
[6]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473
arXiv 2014
-
[7]
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72
work page 2005
-
[8]
Douglas Biber. 1991. Variation across speech and writing. Cambridge University Press
1991
Show all 49 references
-
[9]
Leonard Bloomfield. 1927. Literate and illiterate speech. American speech, 2(10):432--439
1927
-
[10]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL
2019
-
[11]
Chrysanne DiMarco and Graeme Hirst. 1990. Accounting for style in machine translation. In Third International Conference on Theoretical Issues in Machine Translation, Austin
1990
-
[12]
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018. Style transfer in text: Exploration and evaluation. CoRR, abs/1711.06861
2018 arXiv
-
[13]
Nikesh Garera and David Yarowsky. 2009. M odeling latent biographic attributes in conversational genres. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume ...
2009
-
[14]
Marjan Ghazvininejad, Xing Shi, Jay Priyadarshi, and Kevin Knight. 2017. H afez: an I nteractive P oetry G eneration S ystem. Proceedings of ACL 2017, System Demonstrations, pages 43--48
2017
-
[15]
Mengqiao Han, Ou Wu, and Zhendong Niu. 2017. Unsupervised A utomatic T ext S tyle T ransfer using LSTM . In National CCF Conference on Natural Language Processing and Chinese Computing, pages 281--292. Springer
2017
-
[16]
Leo Apostel
Francis Heylighen and Jean-Marc Dewaele. 1999. Formality of language: definition, measurement and behavioral determinants. Interner Bericht, Center “Leo Apostel”, Vrije Universiteit Br \"u ssel
1999
-
[17]
Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780
1997
-
[18]
Eduard Hovy. 1987. Generating natural language under pragmatic constraints. Journal of Pragmatics, 11(6):689--719
1987
-
[19]
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In International Conference on Machine Learning, pages 1587--1596
2017
-
[20]
Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell
Ting-Hao (Kenneth) Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. 2016. https://www.mi...
2016
-
[21]
Shankar Iyer, Nikhil Dandekar, and Korn \'e l Csernai. 2017. F irst quora dataset release: Q uestion pairs
2017
-
[22]
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daum \'e III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proc. of ACL-IJCNLP, volume 1, pages 1681--1691
2015
-
[23]
Hovy, and Eric Nyberg
Harsh Jhamtani, Varun Gangal, Eduard H. Hovy, and Eric Nyberg. 2017. Shakespearizing modern language using copy-enriched sequence-to-sequence models. CoRR, abs/1707.01161
2017 arXiv
-
[24]
Kingma and Jimmy Ba
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980
2014 arXiv
-
[25]
Moshe Koppel, Jonathan Schler, and Shlomo Argamon. 2009. Computational methods in authorship attribution. Journal of the American Society for information Science and Technology, 60(1):9--26
2009
-
[26]
Moshe Koppel, Jonathan Schler, and Shlomo Argamon. 2011. https://doi.org/10.1007/s10579-009-9111-2 Authorship attribution in the wild . Language Resources and Evaluation, 45:83--94
2011 doi
-
[27]
Moshe Koppel and Yaron Winter. 2014. Determining if two documents are written by the same author. Journal of the Association for Information Science and Technology, 65(1):178--187
2014
-
[28]
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043
2017 arXiv
-
[29]
Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 71--78...
2003
-
[30]
Zachary C Lipton, Sharad Vikram, and Julian McAuley. 2015. G enerative concatenative nets jointly learn to W rite and C lassify reviews. arXiv preprint arXiv:1511.03683
2015 arXiv
-
[31]
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: A n empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023
2016 arXiv
-
[32]
Gonzalo Navarro. 2001. A guided tour to approximate string matching. ACM computing surveys (CSUR), 33(1):31--88
2001
-
[33]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU : a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311--318. Association for Computational Linguistics
2002
-
[34]
Adam Paszke, Sam Gross, Soumith Chintala, and Gregory Chanan. 2017. Pytorch
2017
-
[35]
Fabian Pedregosa, Ga \"e l Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct):2825--2830
2011
-
[36]
Lotem Peled and Roi Reichart. 2017. Sarcasm sign: Interpreting sarcasm with sentiment based monolingual machine translation. In ACL
2017
-
[37]
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. G love: G lobal vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532--1543
2014
-
[38]
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W. Black. 2018. Style transfer through back-translation. In ACL
2018
-
[39]
Tetreault
Sudha Rao and Joel R. Tetreault. 2018. Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer. In NAACL-HLT
2018
-
[40]
Sravana Reddy and Kevin Knight. 2016. Obfuscating gender in social media writing. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 17--26
2016
-
[41]
Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010. Automatic keyword extraction from individual documents. Text Mining: Applications and Theory, pages 1--20
2010
-
[42]
Claude E Shannon. 1951. P rediction and entropy of printed E nglish. Bell Labs Technical Journal, 30(1):50--64
1951
-
[43]
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in Neural Information Processing Systems, pages 6830--6841
2017
-
[44]
Efstathios Stamatatos, Francisco Rangel, Michael Tschuggnall, Benno Stein, Mike Kestemont, Paolo Rosso, and Martin Potthast. 2018. Overview of pan 2018. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 267--285. Springer
2018
-
[45]
Hanna K Ulatowska, Mari M Hayashi, Michael P Cannito, and Susan G Fleming. 1986. Disruption of reference in aging. Brain and language, 28(1):24--41
1986
-
[46]
Ben Verhoeven and Walter Daelemans. 2014. CLiPS S tylometry I nvestigation ( CSI ) corpus: A D utch corpus for the detection of age, gender, personality, sentiment and deception in text. In LREC 2014-NINTH INTERNATIONAL CONFERENCE ON LANGUAGE RESOURCES AND EVALUATION, pages 3081--3085
2014
-
[47]
Adam Vogel and Dan Jurafsky. 2012. H e said, she said: G ender in the ACL anthology. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 33--41. Association for Computational Linguistics
2012
-
[48]
pro-drop
Lydia White. 1985. The “pro-drop” parameter in adult second language acquisition. Language learning, 35(1):47--61
1985
-
[49]
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proc. of NIPS
2015
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.