{"id":"bdc037e5-44d3-4357-b51c-c1a95bfda733","arxiv_id":"2501.16073","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning on low-level style edits with contrastive learning does not transfer to high-level style classification, questioning the assumption that low-level style changes compose into high-level styles.","lead":"Researchers fine-tuned BERT and RoBERTa on small, low-level style changes to see if the models could learn a general sense of writing style. The fine-tuned embeddings did not beat the original pre-trained ones on high-level styles, and contrastive learning usually made things worse.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never verifies that fine-tuning actually learns low-level StylePTB styles, so the negative high-level results may reflect a failed training recipe rather than the composition assumption.","rationale":"The reader's weakest assumption concerned whether low-level StylePTB changes are compositional building blocks of high-level styles. I agree that is a substantive external-validity question, but the more immediately load-bearing issue is internal: whether the fine-tuned models learned any low-level style signal at all. A negative high-level result can support the paper's conclusion only if the training objective demonstrably shaped the representation toward low-level style distinctions. The paper reports neither StylePTB validation accuracy nor any internal probe; the only internal signal is 'best model based on the loss on the validation dataset' (Sec. 4.1), and a low loss can arise from trivial solutions (e.g., pushing all negative pairs apart) without meaningful style axes. This is not hypothetical: the authors state that with more than two styles per batch the contrastive objective 'resulted in no meaningful representations' (Sec. 4.1), showing the setup is prone to degenerate solutions. Under this uncertainty, the small cross-entropy gains and the contrastive drops in Table 1 cannot be attributed to the composition assumption; they could simply reflect that contrastive fine-tuning with non-parallel StylePTB data collapsed or reorganized the representation in a way that hurts downstream linear probes. A StylePTB probe is a cheap, decisive check. It also addresses the reader's condition: with the probe in hand, the claim can be stated either as 'low-level styles were learned but did not transfer' or as 'the fine-tuning recipe failed to learn low-level styles,' which are different scientific statements. I would keep the verdict at CONDITIONAL: the paper is an honest, clearly written negative result, but this missing internal validation is precisely the kind of condition that should be satisfied before the conclusion is accepted at face value. Because my concern is internal rather than about the composition premise, I only partially agree with the reader's identification of the weakest assumption.","tokens_in":7250,"tokens_out":6126,"duration_ms":54721,"concrete_test":"After fine-tuning exactly as in Sec. 4.1 (same hyperparameters, epochs, samplers), freeze each encoder and train a logistic regression probe on held-out StylePTB style labels—both the 21 individual and 32 compositional style changes. Measure probe accuracy and macro-F1 for the fine-tuned BERT/RoBERTa versus the same probe on the pretrained encoders. If the fine-tuned probe is not clearly above the pretrained probe (e.g., by more than the seed variability estimated with at least 3 runs), then the high-level negative results are confounded by a failure to learn low-level styles. If it is clearly above, the central claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that fine-tuning on low-level StylePTB changes does not improve high-level style classification—requires that the fine-tuned encoders actually capture the low-level style distinctions in StylePTB. Nowhere in Sections 4.1–4.3 is this verified: training is stopped by validation loss (Sec. 4.1), but no StylePTB validation loss, accuracy, or pairwise separation is reported for either the contrastive or cross-entropy models. The text even notes that contrasting more than two styles 'resulted in no meaningful representations' (Sec. 4.1), making it plausible that the two-style setup also failed to learn robust low-level style structure. If the encoder does not separate low-level styles, the observed high-level accuracy drops are explained by a failed training recipe, not by a falsification of Assumption 1 (low-level changes compose high-level styles). The Limitations section acknowledges dependence on StylePTB but does not address this internal validation gap. So the negative result is currently underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper tests two assumptions underlying generic text style embeddings: (A1) low-level stylistic changes can be composed to form high-level styles, and (A2) contrastively contrasting different styles yields style embeddings. The authors fine-tune BERT-Large and RoBERTa-Large on the StylePTB dataset using cross-entropy, contrastive, or combined losses under three batch-sampling strategies, then train a logistic regression probe on the resulting CLS representations and evaluate on Yelp/Amazon sentiment, GYAFC formality, Shakespearean/modern English, and Bible-version classification. Table 1 shows that cross-entropy fine-tuning gives small and inconsistent gains, contrastive loss often lowers accuracy, and the pre-trained encoders are already strong baselines. The paper concludes that these results challenge the assumption that low-level style changes compose high-level styles and that contrastive learning is an effective route to generic style embeddings.","tokens_in":7446,"tokens_out":5711,"duration_ms":51430,"significance":"If fully established, this negative result would be a useful corrective to the common assumption that fine-grained StylePTB transformations are compositional building blocks of high-level style and to the expectation that contrastive objectives readily produce generic style embeddings. The evaluation design is anchored on held-out high-level style datasets, uses a simple linear probe, and compares against pre-trained encoders, which is methodologically clean for a negative-result study. The paper also states its two assumptions explicitly rather than baking them into the training objective. However, the conclusion is currently underdetermined: the absence of repeated runs and error bars, combined with the lack of any verification that the fine-tuned encoders actually learned the low-level StylePTB distinctions, leaves two competing explanations for the results: a failed training recipe or a genuinely falsified assumption. With internal validation and statistical evidence, the paper could make a valuable contribution.","major_comments":[{"comment":"Every number in Table 1 is a single run with no error bars, confidence intervals, or significance tests. The claimed cross-entropy gains are small, for example BERT Shakespeare 85.7 to 86.4 and BERT Bible 50.9 to 52.8, and are within the range of seed-to-seed variation expected when fine-tuning deep transformers. To support the claims that 'incorporating the contrastive loss reduces accuracy compared to cross-entropy fine-tuning' and that cross-entropy gives 'slight accuracy improvements', the authors should report means and standard deviations over at least 3-5 seeds and, where applicable, paired significance tests such as a bootstrap or approximate randomization test.","section":"4.2, Table 1"},{"comment":"The paper never verifies that the fine-tuned encoders actually separate the low-level StylePTB styles used in training, so the negative high-level results may reflect a failed training recipe rather than a falsified composition assumption. Training is stopped using the loss on a validation dataset, but no StylePTB validation loss, validation accuracy, or pairwise embedding separation is reported for either the cross-entropy or contrastive models. The note in Section 4.1 that contrasting more than two styles per batch 'resulted in no meaningful representations' makes it plausible that the two-style setup also failed to learn robust low-level structure. The authors should add a direct low-level evaluation, such as StylePTB held-out classification accuracy or pairwise separation, for each fine-tuned model; if the encoders do not separate low-level styles, the paper must be reframed as a training-recipe study rather than a test of Assumption 1.","section":"4.1-4.3"},{"comment":"Key implementation choices are underspecified: which subset of the 21 individual and 32 compositional StylePTB changes is used, how low-level styles are defined for batch construction, how many candidate sentences form X_cand, and how positive and negative pairs are drawn from the dataset. The notation in Section 3 is also garbled, as 'Xcand = Xs ∪ Xs' should presumably read X_s ∪ X_{\\bar s} to denote same-style and different-style candidates. Without these details, the cross-entropy and contrastive experiments cannot be reproduced, and it is unclear whether the low-level objective actually instantiates the composition operation assumed in Assumption 1.","section":"3, 4.1"},{"comment":"Even if the encoders learn low-level distinctions, the evaluation is only meaningful if the StylePTB transformations align with the high-level target styles. The authors acknowledge in Section 4.3 that 'some low-level changes, like info addition, relate directly to a formality change, others, such as tense changes, do not align with the investigated styles,' but they provide no quantitative analysis of how many training examples are of each transformation type or how much overlap exists between StylePTB transformations and the sentiment, formality, and register dimensions of the evaluation datasets. This leaves the negative result open to the alternative interpretation that the fine-tuning objective is irrelevant to the high-level styles under test. A per-transformation breakdown of downstream accuracy, or an analysis restricted to transformations that plausibly align with each target style, would materially strengthen the conclusion.","section":"4.3"},{"comment":"The paper repeatedly contrasts its findings with StyleDistance (Patel et al., 2024) but never includes a StyleDistance baseline or a close reproduction of its training setup. Since the authors identify synthetic parallel data as the key difference, the statement in Section 5 that the approach 'does not yield the expected results compared to previous work (Patel et al., 2024)' is not directly supported without running a comparable method under the same evaluation protocol. Adding such a baseline, or at least an ablation that uses synthetically constructed low-level parallel examples with the same contrastive loss, would clarify whether the failure is due to the contrastive objective per se or to the use of non-parallel training data.","section":"2, 5"}],"minor_comments":[{"comment":"The table header 'GY AFC' contains a spurious space, and the column structure under Random Sampler, Pairwise Sampler, and Corpus Sampler should be explained in a table note, as it is currently easy to misread which loss corresponds to which column.","section":"Table 1"},{"comment":"The notation X_s and X_{\\bar s} is introduced informally, and the sentence 'we first form pairs of sentences, some equal and others of a different style' is ambiguous: 'equal' should be specified as same sentence, same style, or something else.","section":"3, Eq. (1)-(2)"},{"comment":"The 'RoBERTa's \"CLS\" Tokens Need Fine-Tuning' explanation is imprecise because BERT's CLS token is trained with a next-sentence prediction objective while RoBERTa's is not; the proposed mechanism should be verified empirically or stated more cautiously.","section":"4.3"},{"comment":"The paper does not state whether code and trained models will be released; for a negative-result paper, releasing the fine-tuning and evaluation code would substantially increase its value to the community.","section":"General"},{"comment":"The abbreviation 'i.a.' in the Introduction is uncommon in machine learning venues and should be replaced with 'among others'.","section":"1"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations, and the experimental framing is promising for a negative-result contribution. The key risk is the missing internal validation: unless the authors show that the fine-tuned encoders separate low-level StylePTB styles, the central conclusion is underdetermined. I would encourage the editor to send the revision back with a request for repeated runs with error bars and for the StylePTB validation numbers. A direct comparison or ablation involving StyleDistance's synthetic-parallel-data setup would also sharpen the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis paper is a straightforward, honest negative result: applying the contrastive fine-tuning recipe from StyleDistance to non-parallel low-level StylePTB changes does not improve high-level style classification, and often hurts. The authors carefully frame it as a challenge to the composition assumption (Assumption 1) and the contrastive-learning assumption (Assumption 2) rather than a proof against them. That is worth having on record, since StyleDistance's positive result on synthetic parallel data is the main counterpoint.\n\nThe experimental setup is clean in many respects: two large pre-trained encoders, a simple linear probe on external held-out datasets (Bible, GYAFC, Shakespeare, Yelp, Amazon), and explicit reporting of three sampling strategies. The discussion in Section 4.3 is measured and self-aware, listing sensible possible explanations including low-level changes that do not align with high-level styles.\n\nThe soft spots, in order of importance:\n\nFirst, the load-bearing one: the paper never checks whether the fine-tuned encoders actually learned the low-level StylePTB style distinctions. Training is stopped on validation loss, but no StylePTB validation accuracy, loss, or representation-separation numbers are reported. They even note that contrasting more than two styles per batch produced 'no meaningful representations' (Sec. 4.1). If the two-style contrastive setup also failed to learn low-level structure, then the high-level accuracy drops are explained by a failed training recipe, not by a falsification of the composition assumption. The Limitations section does not address this internal validation gap.\n\nSecond, every accuracy in Table 1 appears to be a single run. Small cross-entropy gains (e.g., BERT Shakespeare 85.7 to 86.4, Bible 50.9 to 52.8) could easily be noise. Adding seeds, error bars, or significance tests would make the conclusions much more robust.\n\nThe paper's core value is empirical: it documents that this contrastive recipe does not transfer from low-level to high-level styles with non-parallel data. That is a useful data point even if the larger claim about the composition assumption remains open.\n\nWho is this for? Researchers working on style representation learning and style transfer. It deserves referee time because the negative result is relevant and the missing low-level validation is fixable with additional experiments. I would send it out, but ask for added low-level evaluations and multiple seeds before acceptance.","headline":"Useful negative result held back by missing low-level validation and single-run accuracy numbers.","tokens_in":7954,"tokens_out":3992,"would_cite":true,"duration_ms":33641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a general-purpose text encoder on low-level style changes does not yield representations that classify high-level styles better than the pre-trained encoder.","keywords":["text style embeddings","contrastive learning","style transfer","compositionality","StylePTB","sentence representations","negative results","BERT fine-tuning"],"falsifier":"A concrete way to test the claim: take a held-out set of high-level style pairs that are explicitly constructed as compositions of StylePTB-style low-level edits (e.g., formality realized through the specific operations in StylePTB's info-addition and word-substitution categories), and check whether contrastive fine-tuning on StylePTB raises classification accuracy on that set relative to the pre-trained encoder. If it does, the paper's negative result is specific to its evaluation datasets rather than to the compositionality assumption; if it does not, the claim that low-level supervision transfers is directly refuted.","tokens_in":7057,"feed_emoji":"🎭","tokens_out":4509,"duration_ms":36546,"temperature":0.7,"pith_summary":"The paper tests a hypothesis: that fine-tuning a general-purpose text encoder on fine-grained, low-level style changes (tense, information addition, and similar edits) will produce generic sentence-level style embeddings that transfer to high-level styles such as sentiment, formality, and author style. Using StylePTB as the training source and BERT/RoBERTa as encoders, it compares contrastive and cross-entropy fine-tuning against the untouched pre-trained encoders. The results are largely negative: contrastive learning consistently lowers classification accuracy on all five high-level style benchmarks, and cross-entropy fine-tuning yields only small, inconsistent improvements. The paper reads this as evidence that the compositionality assumption—low-level style changes build high-level styles—may not hold for representation learning.","feed_headline":"Contrastive style fine-tuning lowers high-level style accuracy","feed_subtitle":"Fine-tuning BERT and RoBERTa on StylePTB gives no gain over pre-trained encoders for sentiment, formality, and Bible style.","key_machinery":"The core mechanism is the contrastive fine-tuning objective over sentence pairs drawn from StylePTB, with similarity computed as the inner product $f(x)^\\top f(\\tilde{x})$ and a softmax over same-style versus different-style candidates (Eq. 1–2). The paper also fine-tunes with standard cross-entropy loss, and trains a logistic regression classifier on the resulting embeddings to test transfer. The load-bearing assumption is that StylePTB's low-level, linguistically motivated transformations compose into the high-level styles found in the evaluation corpora; the dataset is the only available source of such fine-grained style changes, and the non-parallel pairing of sentences is what distinguishes this approach from prior work using synthetic parallel data.","core_discovery":"The paper's central claim is that applying the compositional view of style transfer to representation learning does not deliver the expected generic style embeddings. Fine-tuning BERT and RoBERTa on the 21 individual and 32 compositional low-level style changes in StylePTB, then evaluating with a logistic regression classifier on Yelp/Amazon sentiment, GYAFC formality, Shakespearean versus modern English, and eight Bible versions, shows that the contrastive objective reduces accuracy in every configuration tested, while cross-entropy fine-tuning improves only some datasets (notably Shakespeare and Bible) and by a small margin. The authors attribute the contrastive failure to the objective pushing dissimilar styles too far apart and to training on non-parallel data, and they note that many low-level changes (e.g., tense shifts) do not align with the evaluated high-level styles. The conclusion is that the learned representations do not capture high-level text styles in a way that improves over simply taking the pre-trained CLS representation.","pith_inferences":["The negative result is consistent with the view that style is not a single compositional hierarchy; low-level lexical and syntactic operations and high-level pragmatic or register differences may be orthogonal axes, in which case no amount of low-level supervision will produce a generic high-level style space.","A direct test of the compositional premise would be to evaluate the learned embeddings on the low-level StylePTB task itself; if fine-tuning does not even improve low-level style classification, the failure is in the objective, whereas if it does, the failure is in transfer.","The paper's findings suggest that future style-embedding work should either define style operationally (as StyleDistance does with synthetic parallel data) or incorporate high-level style labels directly, rather than assuming compositionality.","The reliance on English and on a single low-level dataset weakens the generality of the conclusion; a multilingual low-level style corpus could distinguish whether the issue is the compositionality assumption or an artifact of English syntax."],"forward_implications":["Contrastive fine-tuning on low-level edits, as implemented here, is not a reliable route to generic style embeddings; the paper suggests it is too aggressive and pushes styles too far apart.","Cross-entropy fine-tuning gives small gains on some high-level styles, mostly on datasets where the pre-trained CLS token is weak (Shakespeare, Bible), consistent with the claim that RoBERTa's CLS token lacks next-sentence pretraining.","The compositionality assumption needs empirical support: a dataset that maps low-level operations to high-level styles, or evaluation on such constructed styles, is required to test whether the approach can work.","Non-parallel contrastive data may be insufficient; synthetic parallel data (as in the StyleDistance approach) is the alternative that has shown positive results, implying data type, not the objective, may be the deciding factor."],"supporting_citations":[{"why":"Supplies the StylePTB dataset of fine-grained and compositional style changes and the premise that low-level changes compose high-level styles.","marker":"Lyu et al. (2021)"},{"why":"Introduces the contrastive-learning assumption for generic style embeddings and the synthetic-parallel-data alternative that this paper contrasts with.","marker":"Patel et al. (2024)"},{"why":"Provides the BERT encoder and its CLS representation, used as both baseline and fine-tuned model.","marker":"Devlin et al. (2019)"},{"why":"Provides the RoBERTa encoder, the other pre-trained model fine-tuned and used as a baseline.","marker":"Liu et al. (2019)"},{"why":"Provides the GYAFC formality corpus used as one of the high-level style evaluation benchmarks.","marker":"Rao and Tetreault (2018)"},{"why":"Provides the Bible corpus with eight style versions used to evaluate high-level style classification.","marker":"Carlson et al. (2018)"},{"why":"Provides the Shakespearean versus modern English corpus used as another high-level style evaluation benchmark.","marker":"Xu et al. (2012)"}],"fun_headline_variants":["Contrastive style tuning backfires on high-level styles","Fine-tuning on low-level styles doesn't transfer to high-level","Style embeddings: no gain over plain BERT for sentiment, formality","Questioning generic style embeddings: contrastive fails"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the low-level stylistic edits in StylePTB (tense changes, information addition, and similar operations) are the compositional building blocks of the high-level styles used for evaluation (sentiment, formality, Bible version, Shakespearean). If these edits are not the right atoms of high-level style, then the fine-tuning objective is training the encoder on the wrong signal, and the negative result says nothing about contrastive learning for style.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive style tuning backfires on high-level styles","Fine-tuning on low-level styles doesn't transfer to high-level","Style embeddings: no gain over plain BERT for sentiment, formality","Questioning generic style embeddings: contrastive fails"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1179,"prompt_tokens":862,"completion_tokens":317,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":248}},"tokens_in":478,"tokens_out":317,"duration_ms":3400,"temperature":1.0,"reasoning_tokens":248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:44:00.825634+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete way to test the claim: take a held-out set of high-level style pairs that are explicitly constructed as compositions of StylePTB-style low-level edits (e.g., formality realized through the specific operations in StylePTB's info-addition and word-substitution categories), and check whether contrastive fine-tuning on StylePTB raises classification accuracy on that set relative to the pre-trained encoder. If it does, the paper's negative result is specific to its evaluation datasets rather than to the compositionality assumption; if it does not, the claim that low-level supervision transfers is directly refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Bible corpus with eight style versions used to evaluate high-level style classification."}],"review_version":1}