{"id":"fec4181a-426c-44f9-9992-b1daf98ade50","arxiv_id":"2507.21828","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Sentence transformers underperform standard classifiers on the ADEPT adjectival-plausibility benchmark, and unbalanced evaluation inflates apparent accuracy.","lead":"This paper tests whether sentence-transformers can judge how an adjective changes an event's plausibility, for example 'a horse goes away' versus 'a dead horse goes away'. It finds they underperform standard classifiers like RoBERTa, and argues that unbalanced evaluation hides how poorly all models handle this task.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-balanced evaluation's window wrap-around reuses test instances across iterations, inflating the effective sample size behind the ASO test; the 0.024 F1-macro gap claimed for ST underperformance may not be statistically significant.","rationale":"I agree with the reader that the paper is transparent and that the modified benchmark is a legitimate concern, but I find a more load-bearing threat in the evaluation protocol itself. The headline contribution is that sentence transformers underperform transformer classifiers despite conceptual alignment. This claim is based on a small F1-macro gap (0.612 vs. 0.588) in the balanced condition. The cross-balanced evaluation's wrap-around window reuses boundary instances across iterations, so the ASO significance test operates on non-independent predictions with an inflated effective sample size. With only three seeds, no confidence intervals, and a multiple-comparison correction, the reported significance is not credible as-is. The paper even reports under the same cross-balanced protocol that ST-full is the best model when trained on the full imbalanced data, which directly conflicts with the abstract's unqualified statement that STs underperform. This does not mean the paper's empirical contributions are worthless; the dataset analysis and the balanced-evaluation proposal are useful. But the central performance-based claim needs a clean, non-reusing evaluation before it can be accepted. Since the reader already set CONDITIONAL and this is precisely a condition to address, I leave the verdict unchanged. The concrete test above would settle whether the concern lands.","tokens_in":8847,"tokens_out":5381,"duration_ms":68265,"concrete_test":"Recompute the cross-balanced F1-macro and the ASO comparison between RoBERTa-balanced and ST-balanced using a single disjoint balanced test split (e.g., k-fold cross-validation on the test set with each instance scored exactly once) and report the per-seed scores and confidence intervals. If the F1-macro gap shrinks to within noise or ASO no longer reaches significance at alpha=0.05 after Bonferroni correction, then the 'sentence transformers underperform' claim is not established by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central 'sentence transformers underperform' claim rests on the cross-balanced results in Table 2: RoBERTa-balanced and MPNet-balanced reach F1-macro 0.612 while ST-balanced reaches 0.588, a gap of 0.024. This gap is declared statistically significant via ASO (Section 4), but the supporting protocol in Section 3.4 has a structural flaw. Cross-balanced evaluation slides a window of size s along each class and, 'if the window arrives at the end of a class, it wraps back around to the start'. When a class size is not a multiple of s, instances near the boundary appear in two different iterations, so the averaged predictions contain duplicates. The ASO test is then run on a sample whose independence assumption is violated, inflating the effective sample size and potentially manufacturing significance for a small real difference. This is especially fragile with only 3 seeds and no error bars. Furthermore, the abstract's blanket claim of ST underperformance is internally contradicted by Section 4, where ST-full is the best model under cross-balanced evaluation (F1-macro 0.562 vs. DeBERTa-full 0.529 and RoBERTa-full 0.521). The underperformance therefore holds only for the balanced-training condition, and even there the gap is small and statistically fragile. The benchmark modification of Section 3.1 is justified and affects both model families equally, so it is not the main threat; the evaluation protocol's duplicate reuse is.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the task of predicting changes in semantic plausibility triggered by adjectival modification, using the ADEPT benchmark. The authors adapt ADEPT by removing the 'impossible' and 'necessarily true' classes and down-sampling 'equally likely', then fine-tune transformer classifiers (BERT, RoBERTa, DeBERTa, MPNet) and a sentence transformer (ST, MPNet-based) on both balanced and imbalanced versions of the training data. They propose a 'cross-balanced' evaluation protocol that slides a window over each class, wrapping around at the end, and compare results to standard evaluation. The main claims are: (i) both model families struggle with the task; (ii) sentence transformers underperform transformer-based classifiers despite conceptual alignment; and (iii) class imbalance in training and test data distorts reported performance. The paper also compares its best models to previous results from Emami et al. and presents a confusion-matrix analysis for one seed.","tokens_in":9194,"tokens_out":5467,"duration_ms":63895,"significance":"If the central claims held, the paper would provide a useful benchmark modification and evaluation protocol for studying semantic plausibility changes, and a noteworthy negative result about sentence transformers on this task. The experiments are reproducible with public code, and the paper does show a consistent pattern that balanced training and cross-balanced evaluation shift model rankings relative to standard practice. The 'models struggle' finding is credible and consistent with prior work on ADEPT. However, the headline claim that sentence transformers underperform is contradicted by the paper's own cross-balanced results for full-data training (ST-full is best), and the statistical significance of the balanced-condition gap is undermined by a design flaw in the cross-balanced protocol. These issues affect the main conclusions and require revision.","major_comments":[{"comment":"The blanket claim in the abstract and conclusion that sentence transformers underperform is not supported by the paper's own results. In Table 2, under cross-balanced evaluation, ST-full achieves the highest F1-macro (0.562) among models trained on full data, exceeding DeBERTa-full (0.529) and RoBERTa-full (0.521). The underperformance holds only in the balanced-training condition, where ST-balanced (0.588) trails RoBERTa/MPNet-balanced (0.612) by 0.024. The abstract and conclusion must be qualified to specify that the underperformance is observed for the balanced-training setup, not as a general property.","section":"Abstract; Section 4, Table 2"},{"comment":"The cross-balanced evaluation protocol with wraparound reuses test instances when a class size is not a multiple of the window size s. For a class of size n with n mod s != 0, the wrapping causes some instances to appear in two iterations. These duplicated predictions are then averaged into the final balanced score and used in the ASO significance test, violating the i.i.d. assumption that ASO requires and inflating the effective sample size. With only three seeds and no per-seed error bars, the reported statistical significance of the 0.024 F1-macro gap between RoBERTa/MPNet-balanced and ST-balanced is not credible. The authors should report per-seed results, use non-overlapping evaluation folds, correct the significance test for duplicates, or provide bootstrap confidence intervals.","section":"Section 3.4; Section 4"},{"comment":"The paper repeatedly labels cross-balanced evaluation as 'more realistic' without providing a task-based justification. The original ADEPT distribution is heavily skewed by design, and forcing a balanced test distribution changes the evaluation task. The comparison to Emami et al. in Table 3 is confounded by class removals and different evaluation protocols, so it does not demonstrate that imbalance distorts performance; it only shows that different test distributions yield different scores. This claim should either be substantiated with an argument about how balanced evaluation better reflects real-world usage, or it should be tempered to 'alternative evaluation' rather than 'more realistic'.","section":"Section 1; Section 4; Section 5"},{"comment":"The sentence transformer comparison is not parameter-free: the cosine label mapping (0, 0.5, 1) and the decision thresholds (0.33/0.66) are empirically tuned, as the text states. While the paper notes that different thresholds yield similar results, the comparison against classifiers with a learned output layer is not on equal footing. Please add a sensitivity analysis to the main text (e.g., a table showing ST performance across a range of thresholds), and discuss whether the 'conceptual alignment' claim holds if the mapping or thresholds change materially.","section":"Section 3.3"}],"minor_comments":[{"comment":"The dataset name is written as both 'Adept' and 'ADEPT'; please use a single consistent form.","section":"Throughout"},{"comment":"The hyperparameter description is incomplete: please specify batch size, number of training steps or epochs, weight decay, and warmup if used, for all models so that the experiments can be reproduced exactly.","section":"Section 3.2"},{"comment":"Please specify the actual window size s used in the cross-balanced evaluation, and explain how the wrapping is applied across classes of different sizes. This information is necessary for readers to evaluate the duplicate-instance issue.","section":"Section 3.4"},{"comment":"The table compares RoBERTa-full standard accuracy to DeBERTa results with different class setups, and the discussion itself notes that the comparison is not direct. Consider adding a footnote that explicitly lists the class sets and evaluation protocols so the table is self-explanatory.","section":"Table 3"},{"comment":"The caption text appears to contain rendering artifacts in the provided manuscript (the second caption is garbled). Please regenerate the figure and verify that the captions are readable.","section":"Figure 2"},{"comment":"The statement 'performancescoreslagclearlybehindcross-balancedevaluation' has missing spaces; similar spacing inconsistencies appear in several places. A careful proofread is recommended.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid reproducible experimental setup and an interesting evaluation idea, but the main claim of sentence-transformer underperformance is overstated and internally inconsistent across evaluation conditions. The duplicate-reuse flaw in the cross-balanced protocol is not just a presentation issue; it directly affects the statistical significance claim and requires re-analysis. The contradiction between the abstract and Section 4 (ST-full best under cross-balanced) will need to be resolved in the framing. I would like the revised manuscript to explicitly acknowledge these limitations and re-run the significance testing with a corrected protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a transparent, well-written empirical paper that does three useful things. It replicates Emami et al.'s ADEPT baselines, adds sentence transformers as a conceptually different model family, and proposes a balanced evaluation protocol to counter the dataset's heavy skew. The main pattern—models struggle, and sentence transformers are not clearly better—is plausible. But the paper's headline claim that ST underperform is overstated, and the new evaluation protocol has a statistical flaw that undermines the significance test behind the main comparison.\n\nThe novelty is real but modest. Applying sentence transformers to ADEPT is new, and the cross-balanced window idea is a fresh way to report class-balanced performance without throwing away the original splits. The paper is also honest: it acknowledges the thresholds and label mappings for ST were tuned, it releases code, and it compares carefully with prior work's evaluation choices.\n\nNow the soft spots, in proportion. The biggest one is the cross-balanced protocol. When a class size is not a multiple of the window size, wrapping reuses instances near the boundary, so those instances contribute twice to the averaged predictions. The ASO test then runs on a sample with duplicated data, violating independence and inflating the effective sample size. With only three seeds and no error bars, the 0.024 F1-macro gap between ST-balanced and RoBERTa/MPNet-balanced in Table 2 may well be noise. This is fixable—don't wrap, or use a different balancing scheme—but until it's fixed, the significance claim should not be taken at face value.\n\nRelatedly, the abstract's blanket statement that sentence transformers underperform is internally contradicted by Section 4, where ST-full actually achieves the highest cross-balanced F1-macro. The underperformance holds only under balanced training, and even there the gap is small and statistically fragile as argued above.\n\nThe benchmark modification—removing 'impossible' and 'necessarily true' and down-sampling 'equally likely'—is a limitation but not fatal. The authors justify it with low annotator agreement, and it affects both model families equally. Still, conclusions about model struggle are tied to this adapted 3-class setting, not the original 5-class ADEPT.\n\nWho is this for? Researchers working on plausibility or sentence-pair evaluation benchmarks. The imbalance critique, independent of the protocol bug, is a valid reminder that accuracy on skewed data is meaningless and that class-balanced reporting matters.\n\nRecommendation: this deserves peer review, but as a major-revision case. The evaluation protocol needs to be fixed or re-analyzed without wrap-around duplication, the significance claim needs to be re-run, and the conclusion should be carefully qualified. The paper's core message—that imbalance distorts evaluation and that sentence transformers don't automatically win on a similarity-flavored task—can survive those changes.","headline":"Useful negative result and a sensible evaluation critique, but the headline ST-underperformance claim rests on a small gap and a duplicate-reuse bug in the proposed cross-balanced protocol.","tokens_in":9641,"tokens_out":3145,"would_cite":false,"duration_ms":35347,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that sentence transformers underperform transformer classifiers on the ADEPT plausibility-change task, and that class imbalance distorts reported performance.","keywords":["semantic plausibility","adjectival modification","event modification","sentence transformers","transformer classifiers","class imbalance","cross-balanced evaluation","ADEPT benchmark"],"falsifier":"Run the same fine-tuning and evaluation on the original five-class ADEPT with the 'impossible' and 'necessarily true' labels cleaned of the noisy annotations the paper cites; if a sentence transformer then matches or beats RoBERTa and MPNet under cross-balanced evaluation, the paper's underperformance claim falls.","tokens_in":8652,"feed_emoji":"⚖️","tokens_out":5059,"duration_ms":55115,"temperature":0.7,"pith_summary":"The paper asks whether language models can detect how adding one adjective changes whether an event is plausible, for example turning \"a horse goes away\" into \"a dead horse goes away.\" It introduces sentence transformers as a new approach to the ADEPT benchmark and compares them with BERT, RoBERTa, DeBERTa, and MPNet classifiers. The central claim is that sentence transformers, despite being conceptually suited to comparing sentence pairs, do not beat standard transformer classifiers under a balanced evaluation, and that imbalanced training and test data distort earlier-looking conclusions. The authors propose cross-balanced evaluation, which averages scores across windows over all classes, as a more trustworthy estimate of model ability.","feed_headline":"Sentence transformers lose on plausibility-change task","feed_subtitle":"Balanced evaluation shows RoBERTa and MPNet beat them, while imbalanced tests distort scores.","key_machinery":"The central object is the ADEPT benchmark: 16,115 English sentence pairs differing by exactly one adjective, labeled for whether the modified event is less, equally, or more plausible. The paper adapts it to a three-class version and introduces cross-balanced evaluation, where a window of size equal to the smallest class is moved across each class's test instances and scores are averaged, so every instance contributes equally. Sentence transformers are the new modeling device: an MPNet backbone maps each pair to a cosine similarity, with labels converted to 0, 0.5, and 1 and thresholds dividing the cosine range into three bins. This machinery lets the paper compare architectures under controlled class balance.","core_discovery":"On the adapted three-class ADEPT task (less likely, equally likely, more likely), the paper's cross-balanced evaluation shows the best sentence transformer reaches F1-macro 0.59 when trained on balanced data, while RoBERTa and MPNet classifiers reach 0.61; under standard evaluation the gap widens, with the sentence transformer dropping to 0.410 versus RoBERTa's 0.471. When trained on the imbalanced full data, the roles reverse: the sentence transformer reaches 0.562 F1-macro in cross-balanced evaluation, outperforming the transformer classifiers. The paper reads both results as evidence that no model is close to robust performance, and that the training and test balance changes which architecture looks best.","pith_inferences":["A natural extension would be to apply the cross-balanced evaluation method to other imbalanced multi-class NLP benchmarks, where it could expose how much of a reported F1 score is an artifact of class priors.","The sentence transformers' weakness may stem from their training objective rather than their architecture; contrastive fine-tuning on plausibility-change pairs, instead of similarity-score regression, could change the ranking.","The paper leaves open whether the excluded extreme classes ('impossible' and 'necessarily true') carry signal; if those classes are cleaned rather than dropped, a five-class version could show a different model ranking."],"forward_implications":["If the central claim is right, published accuracy numbers on the original imbalanced ADEPT test set overstate model ability, because a majority-class predictor can appear strong.","Balanced training data plus cross-balanced evaluation becomes the default way to compare plausibility-change models, and rankings under standard evaluation should be treated as provisional.","Sentence transformers should not be assumed to be the natural fit for pairwise plausibility change just because they compare sentence pairs; their scalar cosine-similarity output needs stronger calibration or a different training signal.","Future model development should target the errors revealed by balanced confusion matrices, where 'equally likely' instances are split into 'less likely' and 'more likely' rather than collapsed into the majority class."],"supporting_citations":[{"why":"Supplies the ADEPT dataset, its label scheme, and the transformer baseline results this paper replicates and compares against.","marker":"[6]"},{"why":"Defines sentence transformers, the Siamese architecture the paper applies to plausibility-change prediction.","marker":"[16]"},{"why":"Provides the MPNet backbone used both by the sentence transformer and as a transformer classifier.","marker":"[17]"},{"why":"Provides the RoBERTa model, which reaches the best cross-balanced results for transformer classifiers.","marker":"[13]"},{"why":"Provides the DeBERTa model used as one of the transformer classifier baselines.","marker":"[8]"},{"why":"Provides the BERT model used as a transformer classifier baseline.","marker":"[3]"},{"why":"Supplies the Almost Stochastic Order testing method used to determine whether performance differences are statistically significant.","marker":"[18]"}],"fun_headline_variants":["Balanced test flips verdict on plausibility models","Sentence transformers stumble on modified plausibility","RoBERTa beats sentence transformers on plausibility shift","Why balanced data changes who wins plausibility task","Adjectival tweak exposes transformer weakness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the adapted three-class, rebalanced version of ADEPT still measures the same plausibility-change ability as the original five-class benchmark, so conclusions about which models struggle transfer to the original task.","fun_headline_variants_meta":{"raw":{"variants":["Balanced test flips verdict on plausibility models","Sentence transformers stumble on modified plausibility","RoBERTa beats sentence transformers on plausibility shift","Why balanced data changes who wins plausibility task","Adjectival tweak exposes transformer weakness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1422,"prompt_tokens":872,"completion_tokens":550,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":480}},"tokens_in":488,"tokens_out":550,"duration_ms":5647,"temperature":1.0,"reasoning_tokens":480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:19:21.964018+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fine-tuning and evaluation on the original five-class ADEPT with the 'impossible' and 'necessarily true' labels cleaned of the noisy annotations the paper cites; if a sentence transformer then matches or beats RoBERTa and MPNet under cross-balanced evaluation, the paper's underperformance claim falls.","supporting_citations":[{"cited_title":"In: Zong, C., Xia, F., Li, W., Navigli, R","cited_arxiv_id":null,"evidence_quote":"Supplies the ADEPT dataset, its label scheme, and the transformer baseline results this paper replicates and compares against."},{"cited_title":"Advances in neural information processing systems 33, 16857–16867 (2020)","cited_arxiv_id":null,"evidence_quote":"Provides the MPNet backbone used both by the sentence transformer and as a transformer classifier."},{"cited_title":"In: International Conference on Learning Representations (2021), https://openreview.net/forum?id=XPZIaotutsD","cited_arxiv_id":null,"evidence_quote":"Provides the DeBERTa model used as one of the transformer classifier baselines."}],"review_version":1}