{"id":"d2159541-1eae-4a9f-ae2a-d39635879e4b","arxiv_id":"2501.13758","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Stacking unsupervised SimCSE before supervised SimCSE on minBERT yields a dev STS correlation of 0.811, a small gain over supervised SimCSE alone (0.806), but the evidence is single-run.","lead":"This paper stacks two existing SimCSE training stages, unsupervised then supervised, on a small BERT model and reports a small gain on semantic textual similarity. The gain is tiny, based on single runs, and the paper does not release code or compare with the original SimCSE results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"2-Tier gain is confounded: no control for initial STS pretraining, so the 0.005 dev gap over supervised SimCSE cannot be attributed to the two-tier stacking.","rationale":"The reader identified single-run noise as the weakest assumption. I agree that the absence of error bars is a serious issue, but I see a deeper problem: even a perfectly reproduced 0.811 dev score would not demonstrate that the 2-Tier combination works, because the comparison lacks a control for the initial STS pre-training. The supervised SimCSE baseline did not receive STS pre-training, while the 2-Tier model did. The 0.005 dev gap could therefore come entirely from the extra STS data exposure, not from the sequential unsupervised-then-supervised contrastive procedure. This confound directly undermines the paper's central contribution. The concrete test I propose would settle it: run a control that takes the same STS-pretrained minBERT and applies supervised SimCSE directly, omitting the unsupervised tier. If the control matches 2-Tier, the second tier is superfluous. I still view the paper as conditionally acceptable because the missing control is a well-defined, easy addition; hence I retain the reader's CONDITIONAL verdict rather than moving to REJECT. The reader's request for multi-seed evaluation remains necessary but not sufficient—the control ablation must be added for the claim to be interpretable.","tokens_in":8318,"tokens_out":7257,"duration_ms":59901,"concrete_test":"Add a control condition to Table 7: initialize from the same STS-pretrained minBERT used in Section 4.3.6, skip the unsupervised SimCSE step, and directly fine-tune with supervised SimCSE on nli_for_simcse (batch size 24, lr 5e-5, dropout 0.1, 5 epochs, default other params). Also train a second control that applies unsupervised SimCSE on STS but skips the initial STS pretraining. Run each condition (including the full 2-Tier) with at least 5 random seeds and report mean ± std dev Pearson on the STS dev set. If the STS-pretrained + supervised control matches 2-Tier within seed noise, the unsupervised tier contributes nothing and the '2-Tier' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core claim (Section 4.3.6, Table 7) is that the 2-Tier SimCSE model, which applies unsupervised SimCSE on STS followed by supervised SimCSE on NLI, 'surpasses' the single supervised SimCSE (0.811 vs 0.806 dev Pearson). But the comparison is confounded by initialization. The supervised SimCSE baseline (Section 4.3.4) is trained from a single-task minBERT without prior STS fine-tuning. The 2-Tier model instead starts from a minBERT already pre-trained on the STS task, then applies unsupervised SimCSE on STS, then supervised SimCSE on NLI. Thus the 0.005 gap may be entirely due to the extra STS pre-training stage, not to the two-tier stacking of contrastive objectives. No ablation isolates the unsupervised tier: e.g., STS-pretrained minBERT + supervised SimCSE without the unsupervised step is missing. Additionally, all numbers come from a single run without error bars, so the gap is within plausible seed noise. The abstract's 'average test score of 0.742' also mixes the best models per task (SST and paraphrase from the baseline, STS from 2-Tier), which obscures rather than supports the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies SimCSE-style contrastive fine-tuning to the minBERT model for three downstream tasks: sentiment analysis (SST), semantic textual similarity (STS), and paraphrase detection (QQP). It compares single-task and multitask baselines, experiments with standard, curriculum, and adaptive dropout, and proposes a '2-Tier SimCSE Fine-tuning Model' that first applies unsupervised SimCSE on STS and then supervised SimCSE on NLI data. The central empirical claim, in Section 4.3.6 and Table 7, is that this 2-Tier model achieves a dev Pearson score of 0.811 on STS, surpassing the supervised-SimCSE-only baseline (0.806). The paper also reports transfer-learning results on paraphrase and SST tasks, an error analysis, and an ablation of adaptive dropout. The abstract summarizes an average test score of 0.742 across the three tasks, assembled from the best model per task.","tokens_in":8529,"tokens_out":2857,"duration_ms":24934,"significance":"If the 2-Tier result were robust, it would be a simple and potentially useful recipe: sequentially combining unsupervised and supervised SimCSE objectives improves sentence embeddings for STS beyond either stage alone. The paper also provides a thoughtful error analysis and an honest ablation showing that adaptive dropout hurts in the unsupervised SimCSE setting, which is a useful negative result. However, the load-bearing evidence is thin: the headline improvement is a 0.005 dev Pearson difference from a single training run with no error bars or statistical testing, and the comparison is confounded by initialization differences between the 2-Tier model and the supervised-SimCSE baseline. The contribution is incremental relative to the existing SimCSE framework, and the evaluation does not yet establish the claimed superiority.","major_comments":[{"comment":"The central claim of 'superior performance' rests on a single training run: the 2-Tier model yields a dev Pearson score of 0.811 versus 0.806 for the supervised-SimCSE baseline (Section 4.3.4, Table 5). No standard deviations, multiple seeds, or statistical tests are reported anywhere in the paper. A gap of 0.005 on one run is within the range of typical seed-to-seed variation for contrastive fine-tuning, so the paper does not currently provide evidence that the 2-Tier stacking is reliably better than supervised SimCSE alone. The authors should report mean and standard deviation over at least 3-5 random seeds for both models and, ideally, a paired significance test.","section":"Section 4.3.6, Table 7"},{"comment":"The comparison between the 2-Tier model and the supervised-SimCSE baseline is confounded by initialization. The supervised baseline (Section 4.3.4) is described as starting from a single-task minBERT and then fine-tuning on NLI, whereas the 2-Tier model (Section 4.3.6) starts from a minBERT already pre-trained on STS, then applies unsupervised SimCSE on STS, and then supervised SimCSE on NLI. Thus the 0.005 dev gap cannot be attributed to the two-tier stacking of contrastive objectives; it may be entirely due to the extra STS pre-training stage. A missing ablation, such as STS-pretrained minBERT followed by supervised SimCSE without the unsupervised tier (or equivalently, supervised SimCSE initialized from the same checkpoint used for the 2-Tier model), is needed to isolate the effect of the unsupervised tier. Without this control, the paper's specific claim that combining two contrastive stages is beneficial is not established.","section":"Section 4.3.6 vs. Section 4.3.4"},{"comment":"The abstract's 'average test score of 0.742' is not a score of the 2-Tier model: it combines the best test scores from different models per task, namely the single-task baseline with standard dropout on paraphrase and SST, and the 2-Tier model on STS. This is stated explicitly in Section 4.3.7. Reporting a per-task 'best model' average obscures the actual performance of the proposed 2-Tier model, which is worse than the baseline on paraphrase (0.785 dev vs. 0.794) and much worse on SST (0.490 dev vs. 0.534). The average score should either be reported for a single consistent model, or the per-task best scores should be clearly labeled as an upper bound, not as the 2-Tier model's achievement.","section":"Abstract and Section 4.3.7"},{"comment":"The test leaderboard reports a single STS test correlation of 0.788 for the 2-Tier model but provides no test-set numbers for the supervised-SimCSE baseline or any other comparison model. Even if the dev-set gap were accepted as real, the paper does not demonstrate that the improvement transfers to the test set, because the baseline's test correlation is never given. The authors should report test results for all competing models under identical evaluation conditions.","section":"Section 4.3.7"}],"minor_comments":[{"comment":"The section is numbered '5.3.1' in the text, which appears to be a typo; the surrounding sections are numbered 4.3.x. Please renumber.","section":"Section 4.3.1"},{"comment":"The description of the five scaling methods is unclear. For example, the first row ('Sum of two sentence pair embeddings with linear transformation') and the fifth row ('Cross-attention between sentence pair embeddings') report results of 0.381 and 0.338, but the text does not explain what these numbers mean (Pearson correlation on the dev set, presumably). Please define the evaluation protocol for Table 1 in a complete sentence.","section":"Table 1"},{"comment":"The reference to Morerio et al. (2017) is appropriate for curriculum dropout, but the described implementation ('dynamically increasing dropout rates during training') is not precisely the 'standout' method cited; please clarify whether the implementation follows the original paper or a simplified variant.","section":"Section 3.2.1.1"},{"comment":"The reference 'Chuang Zhai, Yeyun Gong, Heng Zhang, Yuqing Kong, Weikang Wang, Guozhong Tang, Longbiao Wang, and Hanghang Tong. 2021. Contrastive learning for unsupervised sentence embedding enhancement.' is missing publication venue or arXiv identifier. The same applies to the Dasgupta et al. (2019) and Yin et al. (2019) entries, which have arXiv identifiers but no 'arXiv:' prefix.","section":"References"},{"comment":"The figures have no captions or axis labels in the text, making the accuracy comparison and the heatmap difficult to interpret. Please add proper captions and label the axes in Figure 3.","section":"Figure 2 and Figure 3"},{"comment":"The sentence 'We observed the best-performing dropout strategies being adaptive dropout (highest performance on STS task) and standard dropout (highest performance on paraphrase and SST tasks)' is contradicted by Table 3, where adaptive dropout gives 0.579 on STS but standard dropout gives 0.502; the sentence should state the values explicitly so the reader can verify the claim.","section":"Section 4.3.2"},{"comment":"The appendix table reports SST and CFIMDB pre-train/finetune results but is never referenced in the main text; either integrate it into the experimental narrative or remove it.","section":"Appendix Table 8"}],"recommendation":"major_revision","confidential_remarks":"This manuscript appears to be a course project report (CS224N), which is not by itself disqualifying, but the framing as a 'novel' architecture is overstated for a sequential application of two existing, published objectives. The main technical barrier is the missing ablation and the absence of repeated-seed statistics; these are fixable within the scope of the paper if the authors have compute access. I recommend major revision rather than rejection because the core idea is plausible and the error analysis is useful, but the current evidence does not support the headline claim. I would also ask the editor to confirm that the test leaderboard scores were obtained with the official held-out test sets and that the reported numbers are not merely dev-set results relabeled."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this one before reading: it is a Stanford CS224N project paper, clearly written and unusually detailed for the genre, but the central claim rests on a 0.005 dev Pearson difference from a single run. The proposed 2-Tier SimCSE—unsupervised SimCSE on STS, then supervised SimCSE on NLI—is not in the original SimCSE paper, so the configuration is new. What the paper does well: the experimental write-up is transparent about hyperparameters, the dropout ablations are sensible, and the error analysis is genuinely thoughtful (the SST and paraphrase examples are well chosen). The authors also report a negative transfer result, which is honest and useful. Credit where due: this is a solid course project. The soft spots are exactly where the stress-test note lands. The 2-Tier baseline is confounded: the supervised SimCSE baseline starts from vanilla minBERT, while 2-Tier starts from an STS-pretrained minBERT, so the 0.005 gap could just be extra task-specific pretraining, not the two-tier stacking. No ablation isolates the unsupervised tier. All numbers come from a single run with no error bars, and the abstract's 0.742 average test score mixes the best model per task, which muddies rather than supports the claim. The paper even says the 2-Tier result was 'better than expected,' which reads as cherry-picking rather than a pre-registered comparison. On the citation pattern: they cite the standard SimCSE, dropout, and NLI sources; nothing looks inflated. The math is simple contrastive learning, correctly stated. The empirical evidence is what it is: a small, possibly real, possibly noise-level gain. For whom is this useful? Someone teaching or learning SimCSE on a small budget, or anyone curious about how a two-stage contrastive recipe behaves on the STS benchmark. It does not deserve a full journal review, but it does deserve a serious referee if submitted to a workshop or as a short paper, because the stacking idea is testable and the failure analysis is above average. My own verdict is skeptical: the load-bearing claim is not established by the evidence presented. If I were advising the authors, the next step would be multi-seed runs, error bars, a clean ablation (STS-pretrained minBERT + supervised SimCSE without the unsupervised tier), and code release. Recommendation: send to peer review only as a workshop-level empirical report; the claim needs to be de-risked before it carries any weight.","headline":"A clear, honest course-project write-up whose single-run 0.005 STS gain is too thin to support the 'superior performance' claim, though the stacking recipe is genuinely new.","tokens_in":9116,"tokens_out":697,"would_cite":false,"duration_ms":7821,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that stacking unsupervised SimCSE followed by supervised SimCSE to fine-tune minBERT yields the best sentence-embedding model for semantic textual similarity, with a dev Pearson score of 0.811 and a test correlation of…","keywords":["sentence embeddings","SimCSE","contrastive learning","semantic textual similarity","minBERT","dropout","transfer learning","2-Tier fine-tuning"],"falsifier":"Run both the supervised-only SimCSE and the 2-Tier SimCSE from the same minBERT initialization for at least 10 random seeds with identical total compute and the paper's hyperparameters; if the mean test Pearson correlation of the 2-Tier model does not exceed that of supervised SimCSE by more than the standard error, the claimed superiority is not established.","tokens_in":8027,"feed_emoji":"📈","tokens_out":6672,"duration_ms":46098,"temperature":0.7,"pith_summary":"The paper tries to establish that combining the two known SimCSE objectives in sequence—unsupervised first, supervised second—produces sentence embeddings from minBERT that are more accurate for semantic textual similarity than either objective alone. If true, it would mean that a cheap unlabeled-data contrastive stage can usefully prime a model before a labeled contrastive stage, without requiring new architectural components. The authors report a development Pearson correlation of 0.811 on STS, against 0.806 for supervised SimCSE alone and 0.716 for unsupervised SimCSE alone, and a test correlation of 0.788. The paper also claims that transfer of STS-trained SimCSE models to paraphrase and sentiment tasks does not help, and that adaptive dropout hurts SimCSE training through added parameters.","feed_headline":"Two-pass SimCSE fine-tuning tops single-stage STS at 0.811","feed_subtitle":"Sequentially applying unsupervised then supervised SimCSE beats either stage alone on semantic similarity.","key_machinery":"The load-bearing mechanism is the 2-Tier SimCSE Fine-tuning Procedure, a sequential fine-tuning schedule: a minBERT model is first trained on the STS dataset, then fine-tuned with the unsupervised SimCSE contrastive objective (two dropout-masked passes of the same sentence as positives, in-batch sentences as negatives), and then fine-tuned again with the supervised SimCSE objective (NLI entailment pairs as positives, contradiction pairs as hard negatives). The contrastive objective is a cross-entropy loss with in-batch negatives, and the procedure's work is to combine a general alignment objective from unlabeled data with a task-aligned, label-driven objective; the paper attributes the STS gain to this stacking.","core_discovery":"The central discovery claimed by the paper is that a two-tier fine-tuning procedure—first unsupervised SimCSE (dropout as augmentation on the STS corpus), then supervised SimCSE (NLI entailment pairs as positives and contradiction pairs as hard negatives)—raises the STS Pearson correlation of minBERT beyond either stage alone, reaching a dev score of 0.811 and a test correlation of 0.788. The paper further reports that transferring this model to paraphrase and sentiment tasks does not improve those tasks, and that removing adaptive dropout from the unsupervised SimCSE single-task model improves its STS score.","pith_inferences":["If the 0.005 dev gain is reproducible across seeds, the two-tier ordering may serve as a general recipe: an unsupervised contrastive pre-fine-tune could align the representation space before a supervised contrastive stage shapes it for a target task.","A testable extension is to apply the same two-tier schedule to other backbone models and other supervised contrastive datasets, where the gap could be larger when the unsupervised stage uses in-domain unlabeled text.","Since the paper reports one run, a fair reading is that the 2-Tier versus supervised-only difference is within seed noise until multi-seed confidence intervals are presented.","The adaptive-dropout ablation suggests parameter-count growth, not the dropout mechanism itself, drives the overfitting; comparing adaptive dropout with matched parameter counts would separate these effects."],"forward_implications":["The 2-Tier SimCSE model achieves a test Pearson correlation of 0.788 on STS and an average test score of 0.742 across SST, paraphrase, and STS.","Removing adaptive dropout from the single-task unsupervised SimCSE model improves its STS dev score from 0.656 to 0.716, indicating the added dropout parameters cause overfitting.","Transfer learning from STS-trained SimCSE models does not improve paraphrase or SST dev accuracy; the 2-Tier transfer reaches 0.785 on paraphrase and 0.490 on SST, near or below the single-task baselines.","The supervised SimCSE stage alone (0.806 dev Pearson) is close to the 2-Tier result (0.811), so the marginal gain of the unsupervised first stage is small on the development set."],"supporting_citations":[{"why":"Supplies the SimCSE unsupervised and supervised contrastive objectives and the nli_for_simcse dataset construction used in both stages.","marker":"Gao et al. (2021)"},{"why":"SNLI corpus, part of the NLI data for supervised SimCSE positives and negatives.","marker":"Bowman et al. (2015)"},{"why":"MNLI corpus, the other NLI source for supervised SimCSE.","marker":"Williams et al. (2018)"},{"why":"Identifies anisotropy and representation degeneration that motivates contrastive fine-tuning.","marker":"Gao et al. (2019)"},{"why":"Curriculum dropout technique used as a comparison in the dropout experiments.","marker":"Morerio et al. (2017)"},{"why":"Adaptive dropout technique and its layer code used in the dropout and ablation experiments.","marker":"Ba and Frey (2013)"}],"fun_headline_variants":["Two-tier SimCSE lifts STS score to 0.811","Sequential SimCSE outdoes single-stage on STS","Unsupervised then supervised SimCSE wins STS","Two-pass SimCSE tops single-stage STS at 0.811","Two-stage SimCSE boosts STS correlation to 0.811"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on a single training run: the 0.005 point advantage of the 2-Tier model over supervised SimCSE alone on the development set is assumed to be a real effect rather than random seed or hyperparameter variation.","fun_headline_variants_meta":{"raw":{"variants":["Two-tier SimCSE lifts STS score to 0.811","Sequential SimCSE outdoes single-stage on STS","Unsupervised then supervised SimCSE wins STS","Two-pass SimCSE tops single-stage STS at 0.811","Two-stage SimCSE boosts STS correlation to 0.811"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000637,"raw_usage":{"total_tokens":2914,"prompt_tokens":903,"completion_tokens":2011,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1938}},"tokens_in":519,"tokens_out":2011,"duration_ms":12852,"temperature":1.0,"reasoning_tokens":1938,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:37:08.930082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run both the supervised-only SimCSE and the 2-Tier SimCSE from the same minBERT initialization for at least 10 random seeds with identical total compute and the paper's hyperparameters; if the mean test Pearson correlation of the 2-Tier model does not exceed that of supervised SimCSE by more than the standard error, the claimed superiority is not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MNLI corpus, the other NLI source for supervised SimCSE."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Curriculum dropout technique used as a comparison in the dropout experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Adaptive dropout technique and its layer code used in the dropout and ablation experiments."}],"review_version":1}