{"id":"b6b223a8-ba6c-4d3e-a07d-3528ae315b57","arxiv_id":"2505.00389","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CSE-SFP places two representation tokens in a two-stage prompt so a decoder-only LLM produces anchor and positive embeddings in one forward pass, improving unsupervised sentence embedding quality and efficiency.","lead":"This paper presents a training scheme that draws two different vector views of one sentence from a decoder-only language model in a single forward pass, and uses them as positive pairs for contrastive learning. The authors report better sentence-embedding quality than three prompt-based baselines on STS and retrieval benchmarks while cutting training time and memory use.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No ablation isolates the single-pass positive-pair mechanism from the two-template prompt: quality gains in Tables 3-4 are confounded, so the causal contribution of the one-pass construction is underdetermined.","rationale":"The reader's weakest assumption concerns whether the prefix and suffix embeddings form a valid positive pair. My concern is adjacent but more specific: the experimental design never varies the template/prompt composition independently of the one-pass mechanism, so the causal role of the headline mechanism in the quality gains is unidentified. The efficiency part of the claim is credible and internally consistent: processing one concatenated sequence rather than two separate sequences reduces token count and activation memory, and Table 5's roughly 40% savings is consistent with such an accounting. Code and checkpoints are promised, but there is no formal verification. I would keep the reader's CONDITIONAL verdict and add the two-pass template-control as an explicit condition for the quality claim; without it, the paper should not be read as demonstrating that the single-pass construction itself improves embeddings. This is not a soundness objection to the method, and I do not see grounds for rejection.","tokens_in":17410,"tokens_out":17227,"duration_ms":197670,"concrete_test":"On Mistral-7B, fix the anchor as Rep2 from the full CSE-SFP prompt (prefix + [Text] + Rep1 + suffix + Rep2) and train with InfoNCE under three positive constructions: (A) CSE-SFP, Rep1 from the same forward pass; (B) Rep1 from a prefix-only forward pass and Rep2 from a second full-prompt forward pass, which by causality should match A and isolates pass count; (C) a dropout/two-pass positive on the same full prompt. Use the same 1M Wiki sentences, QLoRA, batch size 256, truncation length 32, and three seeds; evaluate on the seven STS benchmarks and eight MTEB tasks. If A clearly beats C (more than about 1 point average Spearman and consistent IR gains), the two-stage positive view drives quality; if A is comparable to C, the reported quality gain is attributable to the prompt/template, and the single-pass mechanism is purely an efficiency optimization.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline method changes two variables at once relative to the baselines: (i) the number of forward passes and (ii) the prompt/template design. CSE-SFP's Rep2 is read after a concatenated two-stage prompt that includes Rep1 and the prefix, whereas PromptEOL/PromptSUM/PromptSTH use a single template and their positive view has no access to the other prompted view. Section 4.1 calls the comparison 'an ablation study,' but it does not hold the template fixed while varying the pass structure, or vice versa. Section 3.2 asserts that Rep1 and Rep2 are 'sufficiently diverse' and that the suffix makes Rep2 'distinguishable from Rep1,' but no experiment tests this premise by varying the template combination independently of the single-pass design. Consequently, the reported STS/IR improvements in Tables 3-4 could be driven by the richer two-stage prompt or by the choice of the suffix inference template, rather than by the shared-context single-pass positive-pair construction. This does not invalidate the efficiency claim, which follows directly from processing fewer total tokens, but it leaves the 'higher-quality embeddings' component of the central claim causally unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CSE-SFP, an unsupervised contrastive sentence representation method for decoder-only LLMs. The method concatenates a two-stage prompt with two representation tokens, Rep1 and Rep2, and exploits the causal attention mask so that the prefix-stage embedding (Rep1) and the suffix-stage embedding (Rep2) of the same sentence are produced in a single forward pass and used as a positive pair for InfoNCE. Experiments on seven STS benchmarks and eight MTEB IR tasks with four LLM backbones report that CSE-SFP outperforms PromptEOL, PromptSUM, and PromptSTH in quality while reducing training time by roughly 40% and memory use by several GB. The paper also proposes two ratio-based metrics derived from alignment and uniformity to evaluate embedding spaces.","tokens_in":17668,"tokens_out":6972,"duration_ms":74682,"significance":"If the quality claim holds, the efficiency contribution is practically important: it makes unsupervised contrastive fine-tuning of 7B-scale LLMs substantially cheaper (about 40% less training time and 7-8 GB less GPU memory in Table 5). The method is simple, general across template families and backbones, and the authors release code and checkpoints. The efficiency measurement is well supported by the single-forward-pass design. However, the quality comparison is currently not fully controlled, and the absence of significance testing weakens the claim that CSE-SFP 'produces higher-quality embeddings.' The proposed ratio metrics are an interesting idea but receive only limited validation.","major_comments":[{"comment":"The statement in §4.1 that the comparison with PromptEOL/PromptSUM/PromptSTH 'also functions as an ablation study' is not supported, because CSE-SFP differs from these baselines in two variables at once: the prompt template (two concatenated stages with two representation tokens) and the number of forward passes used to obtain positive pairs. The higher STS and IR scores in Tables 3 and 4 could therefore be caused by the richer two-stage template or by using Rep2 as the anchor, rather than by the single-pass shared-context construction. Please add control experiments, for example a two-pass version of CSE-SFP that computes Rep1 and Rep2 from the same concatenated template in separate forward passes, and single-pass versions of the baseline templates, so that the efficiency property and the template design are disentangled.","section":"§4.1, Tables 3 and 4"},{"comment":"No multiple seeds, confidence intervals, or significance tests are reported for any quality comparison. Several average differences are small (e.g., OPT6.7b: 78.54 vs. 78.26; LLaMA2: 80.12 vs. 79.20), and on individual datasets CSE-SFP sometimes trails a baseline (e.g., STS-15 for LLaMA2: 83.64 vs. 84.49). The abstract's claim that CSE-SFP 'produces higher-quality embeddings' needs repeated training runs and a paired significance test over the benchmark suite before it can be considered empirically established.","section":"Tables 3 and 4"},{"comment":"The central premise that Rep1 and Rep2 are 'sufficiently diverse to support effective contrastive learning' while preserving semantic similarity is asserted rather than demonstrated. Table 6 reports a lower alignment value for CSE-SFP, which is indirect evidence, but no diagnostic varies the template combination (for example, swapping which template serves as prefix vs. suffix, or using the same template for both stages) to show that the quality of the positive pair, rather than the choice of anchor position or suffix instruction, drives the improvement. A small controlled study of template combinations would make the mechanism transparent.","section":"§3.2 and §5.1"}],"minor_comments":[{"comment":"Figure 1 contains duplicated text ('The battle resulted in a Roman victory.') and a 'Copy' label that appears to be a leftover editing artifact; please clean up the figure and its caption.","section":"§1, Figure 1"},{"comment":"The definition of Ratio 2 uses a positive exponent in both numerator and denominator, unlike the standard uniformity formula which uses a negative exponent; the paper should state explicitly why this formulation is preferable and explain the conditions under which lower Ratio 1/Ratio 2 values are guaranteed to reflect a better semantic space.","section":"§5.1, Eq. (9)"},{"comment":"The temperature τ, QLoRA rank, learning rate, and number of training steps are not reported; these details are needed for reproducibility of the quality and efficiency numbers in Tables 3-5.","section":"§4.1"},{"comment":"The memory usage values exceed the capacity of a single RTX 4090 (24 GB), so the paper should state explicitly how GPU memory is aggregated across the four GPUs and whether the reported time includes data loading and evaluation or only training steps.","section":"§4.4, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The efficiency result in Table 5 is solid and likely citable, and the method is simple enough to be reproducible. The main risk is the overclaim in §4.1 that the comparison is an ablation; the requested control experiments and significance testing are straightforward and within the paper's scope. I do not see grounds for rejection. The fact that two of the three template baselines come from the authors' own prior work is worth keeping in mind for the editor, but it is not itself a problem given that the baselines are externally used in the literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the single-forward-pass positive-pair construction is a real trick and the efficiency numbers are convincing; the quality comparison is not, because it changes two things at once.\n\nWhat's new: CSE-SFP uses a two-stage prompt with two representation tokens—Rep1 in the prefix, Rep2 in the suffix—so a causal decoder computes both the anchor and the positive in one forward pass. That's absent from SimCSE, PromptEOL, PromptSUM, and PromptSTH, which all need two passes. Given the cost of fine-tuning 7B models, that is a practically meaningful contribution.\n\nWhat's good: Table 5 is consistent across OPT, LLaMA2, LLaMA3, and Mistral: roughly 40% less training time and about 8GB less memory for the same Wiki-1M contrastive setup. The evaluation also gives a fair head-to-head of the existing prompt templates on four backbones, which is a useful reference. Code and checkpoints are released.\n\nSoft spots: The headline quality claim is underdetermined. Relative to PromptEOL/PromptSUM/PromptSTH, CSE-SFP changes both the number of forward passes and the prompt content—a concatenated two-stage template with a suffix the baselines don't have. Section 4.1 calls this an ablation, but there's no condition holding the template fixed while varying pass structure, or vice versa. So we don't know whether the STS/IR gains in Tables 3 and 4 come from the single-pass sharing or simply from the richer prompt. That's not a fatal flaw for the efficiency story, but it means \"higher-quality embeddings\" is a hypothesis, not a demonstrated conclusion. The other soft spot is statistical: no seeds, no error bars, no significance tests. On STS, differences of one to three points could be noise. Minor: the Ratio 1 and Ratio 2 metrics are simple algebraic ratios of alignment and uniformity; fine as descriptive diagnostics, not a theoretical advance.\n\nBottom line: the paper is a solid efficiency contribution with a suggestive but unproven quality bonus. It deserves a serious referee, and it would have benefited from one that required a proper template-controlled ablation before acceptance. For a reader, the single-pass idea is worth citing and the tables are worth checking; the quality claims should be treated as provisional.","headline":"A genuinely useful efficiency trick for unsupervised contrastive learning on decoder-only LLMs, but the paper's quality gains are confounded with a template change; the efficiency claim is solid, the quality claim is not yet pinned down.","tokens_in":18148,"tokens_out":3447,"would_cite":true,"duration_ms":33302,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CSE-SFP performs unsupervised contrastive sentence representation for decoder-only LLMs with a single forward pass, producing higher-quality embeddings while cutting training time and memory use.","keywords":["sentence representation","contrastive learning","unsupervised learning","decoder-only language model","single forward pass","prompt-based embedding","text embedding","alignment and uniformity"],"falsifier":"Compute the cosine similarity between the Rep1 and Rep2 views for a large sample of sentences before and during contrastive training; if the two views are almost identical (similarity near 1) or nearly orthogonal (similarity near 0) across the sample, the single-pass positive-pair construction would fail to provide the learning signal the method claims. A controlled ablation that keeps the two templates fixed and compares one-pass CSE-SFP against two independent forward passes with the same prefix and suffix templates would also isolate whether the reported gains come from the single-pass sharing or from the template composition alone.","tokens_in":17225,"feed_emoji":"⚡","tokens_out":7245,"duration_ms":65116,"temperature":0.7,"pith_summary":"CSE-SFP is an unsupervised method for learning sentence embeddings with decoder-only large language models that needs only one forward pass per sentence instead of two. It wraps the sentence in a two-stage prompt with a representation token in the prefix and another in the suffix, so the same text produces two distinct embeddings that serve as a positive pair for contrastive learning. The paper argues that this arrangement makes contrastive training both cheaper and more effective, and reports gains over PromptEOL, PromptSUM, and PromptSTH on seven STS benchmarks and eight IR tasks while cutting training time by roughly 40 percent. The paper also introduces two ratio metrics that combine alignment and uniformity into a single evaluation score.","feed_headline":"One forward pass trains better sentence embeddings","feed_subtitle":"Pairing a prefix and suffix prompt in one causal pass cuts contrastive training time by about 40 percent.","key_machinery":"The load-bearing mechanism is the two-stage prompt with two representation tokens, Rep1 and Rep2, combined with the causal attention mask of decoder-only transformers. Because the mask prevents the suffix from influencing the prefix, the same input sentence yields two embeddings computed under different attention scopes and different instruction conditions in a single forward pass: Rep1 reflects the model's encoding capability, while Rep2 reflects its generative capability through the final next-token prediction. The two vectors are both semantically close to the sentence and mutually distinct, which is what makes them usable as a positive pair for the InfoNCE loss.","core_discovery":"The central claim is that a single forward pass suffices for effective unsupervised contrastive learning of sentence embeddings in decoder-only PLMs. CSE-SFP concatenates a prefix prompt and a suffix prompt, each terminated by a representation token; the causal attention mask ensures the prefix embedding is computed independently of the suffix, while the suffix embedding is computed at the end of the sequence and draws on the model's generative next-token prediction. These two vectors, Rep1 and Rep2, are treated as the positive pair and anchor in the InfoNCE loss. Across OPT6.7b, LLaMA2-7b, Mistral-7b, and LLaMA3-8b, the paper reports that CSE-SFP consistently outperforms two-pass baselines on seven STS benchmarks and eight MTEB IR tasks, and that it reduces training time by about 40 percent while consuming less GPU memory.","pith_inferences":["A testable extension is to vary the distance between the sentence and Rep1 or the length of the suffix; the causal-mask account predicts that these changes alter the diversity of the positive pair in a predictable direction, which could be measured with the ratio metrics.","The single-pass trick may transfer to supervised contrastive settings or to asymmetric retrieval, where a query-side prefix and a document-side suffix could be encoded in one forward pass while remaining separate at inference.","The Ratio 1 and Ratio 2 metrics, being cheap to compute, could plausibly serve as a training-time early-stopping signal or as a selection criterion for choosing among prompt templates without needing a full STS evaluation."],"forward_implications":["Unsupervised contrastive tuning of 7B-scale decoder-only models becomes practical on a single multi-GPU node, since training time drops to roughly 60 percent of two-pass methods and memory usage falls by about 2 to 8 GB in the reported settings.","The two-stage prompt acts as a versatile augmentation strategy that can wrap any existing sentence-representation prompt, so future template designs can inherit the single-pass efficiency without changing the training objective.","Because CSE-SFP does not need dropout-based augmentation, it removes a major obstacle to applying unsupervised contrastive learning to generative PLMs such as LLaMA that lack reliable dropout in the intended places.","The ratio-based alignment-uniformity metrics give a single scalar that tracks Spearman rank performance, providing a way to compare semantic spaces that avoids the ambiguity when one encoder wins on alignment and another on uniformity."],"supporting_citations":[{"why":"Establishes the unsupervised contrastive setup, the Wiki-1M training corpus, and the dropout-based positive-pair baseline that CSE-SFP compares against.","marker":"[11]"},{"why":"Supplies the PromptEOL template that CSE-SFP reuses as one stage and a supervised sentence-embedding baseline.","marker":"[15]"},{"why":"Supplies the PromptSTH and PromptSUM templates that CSE-SFP composes into its two-stage prompt.","marker":"[42]"},{"why":"Defines the InfoNCE loss, the contrastive objective that the one-pass positive pairs feed into.","marker":"[23]"},{"why":"Provides the Transformer causal attention structure that lets the two prompt stages compute separate views in one forward pass.","marker":"[32]"},{"why":"Motivates the claim that LLMs possess both encoding and generative capabilities that the two stages exploit.","marker":"[21]"}],"fun_headline_variants":["Single-pass contrastive learning beats two-pass for sentence embeddings","Cut training time 40% with one forward pass for sentence embeddings","CSE-SFP: Single forward pass yields better unsupervised embeddings","Efficient sentence embeddings: one forward pass instead of two","One-pass contrastive learning for better sentence embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the two views of the same sentence produced by the prefix token and the suffix token really are close enough to be a valid positive pair and different enough to teach the contrastive loss something, a behavioral property that the paper verifies only indirectly through downstream performance.","fun_headline_variants_meta":{"raw":{"variants":["Single-pass contrastive learning beats two-pass for sentence embeddings","Cut training time 40% with one forward pass for sentence embeddings","CSE-SFP: Single forward pass yields better unsupervised embeddings","Efficient sentence embeddings: one forward pass instead of two","One-pass contrastive learning for better sentence embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000826,"raw_usage":{"total_tokens":3610,"prompt_tokens":944,"completion_tokens":2666,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":2584}},"tokens_in":560,"tokens_out":2666,"duration_ms":18065,"temperature":1.0,"reasoning_tokens":2584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:43:39.901743+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the cosine similarity between the Rep1 and Rep2 views for a large sample of sentences before and during contrastive training; if the two views are almost identical (similarity near 1) or nearly orthogonal (similarity near 0) across the sample, the single-pass positive-pair construction would fail to provide the learning signal the method claims. A controlled ablation that keeps the two templates fixed and compares one-pass CSE-SFP against two independent forward passes with the same prefix and suffix templates would also isolate whether the reported gains come from the single-pass sharing or from the template composition alone.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PromptSTH and PromptSUM templates that CSE-SFP composes into its two-stage prompt."}],"review_version":1}