{"id":"e5c0d2dd-df7d-43ad-9934-178a56c96fd0","arxiv_id":"1909.00098","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new parallel, multi-persona stylistic dataset with human annotations enables controlled style classification and supervised style transfer that outperforms unsupervised baselines.","lead":"PASTEL is a dataset of about 41,000 parallel sentences and 8,300 short stories, each written by a different crowd worker under multiple persona labels such as gender, age, education, and politics. It gives style-transfer and style-classification researchers a way to test models while holding other demographic attributes fixed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Train/test splits are by story, not by annotator; shared writers across splits can inflate the controlled style-classification results that anchor the 'controlled variables' claim.","rationale":"The paper is a useful resource contribution: the parallel multi-persona annotation protocol, the denotation experiment, the human meaning-preservation evaluation, and the public release are all genuine positives. The reader's conditional verdict is appropriate. The single load-bearing concern is that the controlled style-classification result, the application that ostensibly demonstrates the advantage of having multiple persona variables in conjunction, does not actually control all external variables because writer identity is neither split nor modeled. With an average of 9.97 HITs per annotator and a random story-level split, the same writers appear across train and test, so the classifier can exploit per-writer stylistic fingerprints that are correlated with the target labels. This would inflate the reported macro-F scores and invalidate cross-style comparisons and the feature-salience table. The concern is not fatal to the dataset or to the supervised-transfer comparison, but it means the controlled-classification claim needs a caveat or an annotator-disjoint split before it can be taken at face value. The proposed check, annotator-level splitting and annotator-ID prediction, would settle whether the effect is large. My read therefore does not change the reader's CONDITIONAL verdict.","tokens_in":14955,"tokens_out":6854,"duration_ms":72303,"concrete_test":"Using the released PASTEL annotations, group all stories by annotator ID and create a disjoint train/valid/test split (or run leave-one-annotator-out cross-validation). Rerun the Section 5.1 controlled classification with the same feature sets and model families, and compare macro-F to Figure 4. Also train a classifier to predict anonymized annotator ID on held-out stories from the original split. If macro-F drops materially (e.g., more than 5 points) or annotator-ID prediction is well above chance, writer leakage inflates the reported controlled-classification results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central demonstration that PASTEL enables controlled style classification (Section 5.1) rests on the assumption that fixing the labeled persona variables (gender, age, education, politics) also removes confounds. It does not fix writer identity. Per Section 4.1, 501 annotators produced the data, each completing an average of 9.97 HITs, with each HIT containing three stories. The dataset is then split randomly by story (0.8/0.1/0.1), not by annotator. The same writer therefore appears in training and test with near-certainty. Because each annotator writes all their texts in their own self-reported persona and retains stable lexical and syntactic habits, a classifier can learn writer-specific cues that are correlated with every persona label. This inflates macro-F scores and contaminates both the comparison of style difficulty and the feature-salience analysis, since writer identity is an uncontrolled variable that the design was supposed to exclude. The appendix even acknowledges identifiable annotator behavior by manually blocking careless workers after a threshold, confirming that individual writing style is a detectable signal. The fix is straightforward: split at the annotator level or include annotator identity as a covariate, and report overlap statistics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PASTEL, a crowd-sourced parallel stylistic corpus built from Visual Storytelling image stories. A denotation experiment identifies an input setting (images plus local keywords) that balances meaning preservation and stylistic diversity; annotators then write texts in their own self-reported persona across seven attributes. The authors evaluate two applications: controlled style classification, in which all other persona variables are supposedly fixed while one target is predicted, and supervised style transfer, in which a simple sequence-to-sequence model is compared with retrieval and unsupervised baselines. The paper reports human quality judgments in Appendix B and releases the dataset publicly.","tokens_in":15253,"tokens_out":13213,"duration_ms":161530,"significance":"If the protocol is sound, PASTEL would be a valuable resource: it is among the first multi-persona parallel style corpora, and the combination of a public release, a denotation experiment, and an independent human meaning-preservation study in Appendix B are concrete strengths. The supervised-versus-unsupervised comparison and the controlled classification setup are also potentially useful benchmarks. However, the two headline claims are conditional on resolving the split and control issues below; the resource itself, rather than the specific reported numbers, is the strongest contribution.","major_comments":[{"comment":"The train/valid/test split is by story, not by annotator. With 501 annotators and an average of 9.97 HITs per annotator, each annotator contributes multiple stories; after a random 0.8/0.1/0.1 story-level split the same writer appears in both training and test with near-certainty. Since annotators are instructed to write in their own persona rather than to impersonate others, each writer's idiolect is a stable cue correlated with every persona label. The controlled classifiers in §5.1 can therefore exploit writer-identity features, inflating the macro-F scores in Figure 4 and contaminating both the difficulty ordering of styles and the feature-salience analysis in Table 5. The paper's own Appendix A shows that annotator behavior is detectable, since careless workers are manually blocked, confirming that writer-level signals are present. Please re-run the classification with an annotator-disjoint split, add annotator identity as a covariate, or otherwise report overlap statistics and show that the results are unchanged.","section":"§4.1 / §5.1, Table 3, Figure 4"},{"comment":"The claim that 'other external style variables are controlled' is not supported by the experimental design as described. The setup fixes only gender, age, education, and politics, giving 2^3 = 8 combinations; the remaining three PASTEL persona styles, namely ethnicity, country, and time-of-day, are neither fixed nor mentioned in the classifier. The Abstract and §1 claim that all external variables are controlled. Please either include all seven styles in the controlled subsets or explicitly restrict the claim to the four selected variables and justify why the remaining three can be ignored.","section":"§5.1 Setup / Abstract / §1"},{"comment":"The reported collection statistics are internally inconsistent. Section 4.1 states that 501 annotators completed an average of 9.97 HITs with three stories per HIT, which implies about 14,985 story annotations; Table 8's category counts sum to 4,273 story annotations for each style; and Section 4.1 also reports 2.63 annotators per story over 8,310 stories, which implies about 21,855 annotations. These numbers cannot all be correct. Please reconcile the counts and clarify whether the reported '41K parallel sentences' refers to source sentences, annotation instances, or something else.","section":"§4.1 / Table 8"}],"minor_comments":[{"comment":"The phrase 'ethnics' should be 'ethnicity' in the list of persona types.","section":"§1"},{"comment":"The entropy metric E(GM) is not defined; please state how the Gaussian-mixture n-gram entropy is computed.","section":"Table 2"},{"comment":"The bullet list includes 'number of named entities' under both lexical and syntax features; please remove the duplicate.","section":"§5.1, Features"},{"comment":"There is a typo: 'we also conduct add Meaning Preservation human study' should read 'we also conduct a Meaning Preservation human study'.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The dataset is potentially a useful community resource, and the concern about writer leakage is well founded; the controlled-classification numbers in the current version should not be cited without an annotator-level split. The numerical inconsistencies in Section 4.1 and Table 8 also need to be resolved before the data statistics can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing you should know: PASTEL is the first parallel corpus I know of with multiple persona styles annotated in conjunction, and the denotation experiment that selected the input setting is a real methodological contribution. The collection protocol is detailed, and the independent human study in Appendix B gives me confidence that the annotations preserve meaning. The dataset itself is the contribution, not the two applications, and I'd be happy to cite it in style-transfer work.\n\nThe soft spot is the controlled classification experiment, and it's the one the reader flagged. The split is by story, not annotator; with 501 workers averaging about ten HITs each, the same writer appears in train and test nearly by construction. Since annotators were explicitly asked to write in their own persona, stable per-writer habits become a signal correlated with every label. So the \"controlled\" comparison is at least partially measuring writer identity, not persona. The appendix even admits to manually blocking careless workers after a threshold, which confirms that individual writing style is detectable. The fix is straightforward -- split by annotator, or include annotator identity as a covariate and report overlap statistics -- but without it, the numbers in Figure 4 can't be read as clean estimates of style difficulty or feature salience.\n\nThat said, the stress-test note doesn't sink the paper. The transfer experiments are more of a sanity check than a breakthrough: a supervised model beating unsupervised baselines on parallel data is expected, and the comparison is fair. The feature-salience analysis is honestly presented, including the observation that content words like 'food' and 'love' are discriminative, which the authors openly frame as a counterexample to the style/content separation. The central dataset claim -- that PASTEL provides parallel multi-persona text with reasonable meaning preservation -- holds up.\n\nMy take: send this to peer review. The resource deserves referee time, even though the classification section needs revision. With an annotator split or clear overlap statistics, the paper would be solid; without it, the controlled-classification claim should be softened. I'd bring it to reading group mainly to talk about the leakage problem, which is a useful lesson for anyone building crowd-sourced style corpora.","headline":"PASTEL is a real resource worth having, and the denotation study is a genuine methodological contribution, but the controlled-style classification claim is undercut by an annotator-leakage split.","tokens_in":15657,"tokens_out":1794,"would_cite":true,"duration_ms":18819,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces PASTEL, a parallel corpus of 41,000 sentences and 8,300 stories rewritten across seven persona styles, and shows that parallel text improves style classification and supervised transfer.","keywords":["style transfer","parallel corpus","persona","controlled style classification","stylistic variation","crowdsourced dataset","demographic attributes","denotation experiment"],"falsifier":"Compute the same controlled style classification with a leave-one-annotator-out split that holds out every story from a given writer; if accuracy falls to chance for held-out writers, the reported persona signal is mostly individual idiolect rather than demographic style.","tokens_in":14742,"feed_emoji":"🗣️","tokens_out":5865,"duration_ms":51853,"temperature":0.7,"pith_summary":"This paper attempts to establish that a large parallel corpus of text annotated across several persona styles at once is feasible to build, and that it changes what experiments in stylistic language can measure. The authors release PASTEL, roughly 41,000 parallel sentences and 8,300 parallel stories, each written by a crowd worker about the same visual story prompt so the meaning is shared while the writer's own demographic style shows through. On this dataset they argue that style classification should hold every persona variable except the target fixed, and that supervised style transfer trained on parallel text beats unsupervised models trained on nonparallel text. A sympathetic reader would care because style research has lacked a benchmark where meaning is genuinely controlled while multiple personal traits vary together.","feed_headline":"41K parallel sentences tie text style to seven personas","feed_subtitle":"A new benchmark fixes every style variable except the target, making style classification and transfer cleaner to measure.","key_machinery":"The load-bearing mechanism is the parallel annotation scheme with a shared denotation: crowd workers describe the same five-image story with local keywords, so every annotation refers to the same events, and each annotator's individual persona is the only free stylistic variable. A preliminary denotation experiment compares input settings by meaning-preservation metrics, such as METEOR and embedding similarity, and style-diversity metrics based on n-gram entropy, selecting images-with-local-keywords as the setting that balances the two. This design is what makes controlled classification and parallel supervised transfer possible.","core_discovery":"PASTEL is the first large parallel stylistic dataset built around multiple personas in conjunction: gender, age, ethnicity, country, education, political view, and time of writing, with every text tied to a shared visual story so denotation stays constant. The paper's central claim is that with this resource, a classifier can predict one style, such as male versus female, while all other persona variables are fixed, isolating the target style's textual signature, and that a simple attentional sequence-to-sequence model using the parallel text outperforms unsupervised style-transfer baselines that rely on nonparallel corpora. The paper further claims that story-level prompts preserve meaning better and promote greater stylistic diversity than single reference sentences, and that the best input setting is a sequence of images with per-image keywords.","pith_inferences":["Editorial inference: an annotator-based split, holding out all stories from a given writer, would test whether the reported classification signal is demographic style or individual idiolect; the paper's split by story leaves this open.","Editorial inference: the salient content words, such as 'food' and 'love', suggest that style and content are not cleanly separable, so PASTEL could be used to test content-invariant style representations directly.","Editorial inference: the same denotation design could be extended to non-English or multimodal settings, where parallel style corpora are even scarcer.","Editorial inference: the residual BLEU gap after supervised transfer, despite good soft-metric scores, may indicate that BLEU is inappropriate for style transfer where many valid target paraphrases exist."],"forward_implications":["Style classifiers can now be trained and evaluated with all non-target persona variables fixed, giving a cleaner measure of which demographic traits are actually readable in text.","Supervised style transfer on parallel text beats unsupervised transfer, so nonparallel training data is not a harmless substitute when parallel data exists for the target styles.","Because each story has multiple annotators, the dataset supports joint modeling of several persona styles at once rather than single-axis transfer.","The finding that story-level context helps predict age and education but not gender or political view suggests different styles need different amounts of context.","The word-level and embedding-based evaluation metrics disagree on parts of the style-transfer results, which points to the need for better meaning-preservation evaluation in style transfer."],"supporting_citations":[{"why":"Supplies the visual storytelling image-story source used as the shared denotation input for all annotations.","marker":"(Huang et al., 2016)"},{"why":"Provides the RAKE keyword-extraction algorithm used to select the local keywords that keep annotator outputs on the same meaning.","marker":"(Rose et al., 2010)"},{"why":"Provides the attentional sequence-to-sequence architecture used for the supervised style-transfer model.","marker":"(Bahdanau et al., 2014)"},{"why":"Supplies a variational autoencoder baseline that the supervised parallel model is compared against and outperforms.","marker":"(Shen et al., 2017)"},{"why":"Supplies an unsupervised cross-domain and adversarial baseline that the supervised parallel model is compared against and outperforms.","marker":"(Lample et al., 2017)"},{"why":"Provides GloVe embeddings used both as soft semantic-similarity features and as initialization for the transfer model.","marker":"(Pennington et al., 2014)"},{"why":"Provides the formality-transfer corpus used to pretrain the supervised style-transfer model.","marker":"(Rao and Tetreault, 2018)"},{"why":"Supplies the METEOR metric used to measure meaning preservation in both the denotation experiment and style-transfer evaluation.","marker":"(Banerjee and Lavie, 2005)"},{"why":"Supplies the embedding-averaging and vector-extrema soft similarity metrics used to evaluate meaning preservation.","marker":"(Liu et al., 2016)"}],"fun_headline_variants":["PASTEL: 41K parallel sentences, 7 personas","Parallel dataset isolates style by persona","Style transfer gets cleaner with parallel persona dataset","New benchmark controls all but one style variable","41K parallel stories tie style to seven personas"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that external style variables are fixed assumes that the random train/test split by story does not let the same annotator's individual writing style appear in both sides, so the persona label, not the person, is what the classifier or transfer model learns.","fun_headline_variants_meta":{"raw":{"variants":["PASTEL: 41K parallel sentences, 7 personas","Parallel dataset isolates style by persona","Style transfer gets cleaner with parallel persona dataset","New benchmark controls all but one style variable","41K parallel stories tie style to seven personas"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1362,"prompt_tokens":932,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":360}},"tokens_in":548,"tokens_out":430,"duration_ms":9671,"temperature":1.0,"reasoning_tokens":360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T06:01:27.481853+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the same controlled style classification with a leave-one-annotator-out split that holds out every story from a given writer; if accuracy falls to chance for held-out writers, the reported persona signal is mostly individual idiolect rather than demographic style.","supporting_citations":[{"cited_title":"Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell","cited_arxiv_id":null,"evidence_quote":"Supplies the visual storytelling image-story source used as the shared denotation input for all annotations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RAKE keyword-extraction algorithm used to select the local keywords that keep annotator outputs on the same meaning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies a variational autoencoder baseline that the supervised parallel model is compared against and outperforms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides GloVe embeddings used both as soft semantic-similarity features and as initialization for the transfer model."},{"cited_title":"Tetreault","cited_arxiv_id":null,"evidence_quote":"Provides the formality-transfer corpus used to pretrain the supervised style-transfer model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the METEOR metric used to measure meaning preservation in both the denotation experiment and style-transfer evaluation."}],"review_version":1}