{"id":"3d1b17a0-8de2-4142-9c08-183984622e78","arxiv_id":"2508.06781","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"BiXSE trains dense retrievers with binary cross-entropy on LLM-generated graded relevance probabilities, consistently beating InfoNCE and roughly matching pairwise ranking baselines.","lead":"This paper introduces BiXSE, a training loss for retrieval models that uses graded relevance scores (for example, 1 to 5 from an LLM) as probabilistic targets instead of simple relevant or irrelevant labels. Across BEIR, MTEB, and TREC-DL benchmarks, models trained with BiXSE outperform standard contrastive training and come close to much larger LLM rankers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract claim that BiXSE 'consistently matches or exceeds' pairwise baselines is contradicted by the paper's own Appendix: LambdaLoss nDCG-v2 wins most BGE-M3 configurations and several LightBlue benchmarks.","rationale":"The load-bearing condition for the central claim is not only that LLM graded scores are faithful; it is also that BiXSE actually delivers the advertised pairwise comparisons. The faithfulness concern (the reader's stated weakest assumption) is legitimate but would only matter if the empirical comparisons were clean; here they are not. The appendix contains direct disconfirming evidence: on the same LLM-graded LightBlue data, LambdaLoss nDCG-v2 beats BiXSE on MTEB and on TREC 2022/2023 top-100 reranking, and on BGE-M3 it wins 4 of 5 configurations on both BEIR and MTEB retrieval. That is exactly the setting the abstract's 'consistently' was meant to cover. The absence of significance/variance means we cannot know whether even the wins are stable, which makes the unqualified claim weaker still. I therefore agree with the CONDITIONAL verdict: the method is plausible and the InfoNCE comparison is broadly favorable, but the central claim as worded should be qualified or revised. I partially agree with the reader because the reader's stated weakest assumption is LLM-score calibration, whereas my load-bearing concern is an internal inconsistency in the empirical pairwise claim; the reader's rationale does note the overclaim, so the disagreement is only about emphasis.","tokens_in":21628,"tokens_out":10130,"duration_ms":113047,"concrete_test":"Aggregate the appendix pairwise-baseline comparisons into a single win/loss table with per-configuration standard errors: Table 9 (7 LightBlue benchmarks) plus Tables 11/13 (5 BGE-M3 configurations each). If LambdaLoss nDCG-v2 wins at least 3 of 7 LightBlue benchmarks or at least 4 of 5 BGE-M3 configurations with non-overlapping confidence intervals, the abstract's 'consistently matches or exceeds' should be revised to a weaker claim such as 'is competitive with, and sometimes outperformed by.' Specifically, re-run the three closest LightBlue losses (MTEB, TREC 2022, TREC 2023) with 3 seeds and paired bootstrap CIs to determine whether the observed gaps are reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's second conjunct is the least secure. The abstract says BiXSE 'consistently ... matches or exceeds strong pairwise ranking baselines when trained on LLM-supervised data.' The appendix's own LightBlue comparison (Table 9) shows LambdaLoss nDCG-v2 beating BiXSE on MTEB Multilingual v1 (55.62 vs 55.46), TREC 2022 top-100 (37.25 vs 36.85), and TREC 2023 top-100 (38.23 vs 38.04); LambdaLoss nDCG-v1 also beats BiXSE on both TREC top-100 rows, and PairwiseBCE beats BiXSE on TREC 2022. That is only 4 wins for BiXSE out of 7 benchmark families. On BGE-M3 (Tables 11 and 13), LambdaLoss nDCG-v2 wins 4 of 5 hard-negative/batch configurations on both BEIR and MTEB retrieval, often by more than 3 points (e.g., BEIR 15/16: 49.84 vs 43.63). The paper even acknowledges LambdaLoss v2 'performs close or better ... on several benchmarks.' So the unqualified 'consistently matches or exceeds' is unsupported by the paper's own numbers, independent of any concern about LLM-label calibration. The central empirical claim therefore needs qualification: BiXSE is competitive with pairwise losses in some settings and is outperformed by LambdaLoss v2 in others.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BiXSE, a pointwise training objective for dense retrieval that replaces softmax-based InfoNCE with binary cross-entropy (BCE) over graded relevance labels. LLM-produced ordinal judgments are converted to continuous relevance probabilities and used as BCE targets; a scalar logit bias is added to the scoring function to account for the imbalance introduced by in-batch negatives. The authors fine-tune ModernBERT, Llama-3.2, and Qwen 2.5 encoders on English and multilingual mixtures of binary and graded data and evaluate on BEIR, MMTEB, and TREC-DL. They report consistent gains over InfoNCE, favorable comparisons with MarginMSE and Soft InfoNCE, noise-robustness, and competitive performance with pairwise and listwise baselines such as PairwiseBCE and LambdaLoss.","tokens_in":21991,"tokens_out":4603,"duration_ms":55831,"significance":"If supported, the paper would make a useful practical contribution: a simple, scalable, token-efficient way to distill graded LLM relevance judgments into dense encoders, with the notable property that it works with one labeled document per query. The empirical effort is substantial: multiple base models, multilingual and English benchmarks, controlled noise experiments, and a filtering study. The authors also release code. However, the central comparison against pairwise baselines is substantially overstated by the paper's own appendix data, and the main results table contains an arithmetically inconsistent cell. The core pointwise BCE idea is defensible and likely valuable, but the current presentation overclaims its standing relative to LambdaLoss.","major_comments":[{"comment":"The Qwen2.5-0.5B-Instruct row reports InfoNCE BEIR=38.25 and BiXSE=73.91 with '+6.0%'. These values are mutually inconsistent: a +6.0% relative gain over 38.25 is about 40.5, not 73.91. Appendix Table 9 lists 73.91 as the BEIR(short) value for BiXSE, so Table 1 appears to have copied the wrong column into the BEIR column. Because Table 1 is the principal evidence for the claim that BiXSE 'consistently outperforms' InfoNCE, this error must be corrected and every aggregate and percentage in the table rechecked against the per-benchmark appendix results.","section":"Table 1; §5 'Main Results'"},{"comment":"The abstract's claim that BiXSE 'consistently ... matches or exceeds strong pairwise ranking baselines when trained on LLM-supervised data' is not supported by the paper's own tables. In Table 9, LambdaLoss nDCG-v2 beats BiXSE on MTEB Multilingual v1 (55.62 vs. 55.46), TREC 2022 top-100 (37.25 vs. 36.85), and TREC 2023 top-100 (38.23 vs. 38.04); PairwiseBCE also beats BiXSE on TREC 2022. In Tables 11 and 13, LambdaLoss nDCG-v2 wins four of the five hard-negative/batch configurations on both BEIR and MTEB English retrieval, in some cases by more than 3 points. The later acknowledgement that LambdaLoss 'performs close or better' only in the appendix is not enough. The wording should be qualified, e.g., BiXSE is competitive with pairwise losses in some settings and is outperformed by LambdaLoss v2 when ample labeled negatives are available.","section":"Abstract; §5 'Comparison to other training objectives'; Appendix Table 9; Tables 11 and 13"},{"comment":"The comparison against pairwise baselines on BGE-M3 varies the number of hard negatives and the batch size jointly and then reports each method's best configuration. This does not isolate the loss function: BiXSE's best BEIR result uses 0 hard negatives with batch 256, while PairwiseBCE and LambdaLoss nDCG-v2 use 1–3 hard negatives with smaller batches. The reported differences may therefore reflect the supervision budget or the batch-size/negative trade-off rather than the intrinsic merit of the objectives. I recommend reporting matched-resource comparisons or explicitly presenting the Pareto frontier over labeled-negative count and batch size.","section":"Appendix Tables 11 and 13; 'BGE-M3 Results'"},{"comment":"The paper asserts that the transformed LLM scores z := sum_s s * p_LLM(s|q,d) / N are valid relevance probabilities, but it provides no calibration analysis. This premise is load-bearing for interpreting BiXSE as probabilistic graded-relevance distillation: if the teacher's probabilities are systematically miscalibrated or biased across languages/domains, the BCE targets are distorted. This is not circular because the downstream evaluation is held out, but it is an unvalidated assumption. The paper would be strengthened by a calibration plot or by showing that the conclusions are robust to alternative monotone transformations of the teacher scores.","section":"Section 4; 'Lightblue Reranker Distillation Dataset'"}],"minor_comments":[{"comment":"'with 0 being most relevant and 1 absolutely relevant' should read '0 being least relevant and 1 absolutely relevant'.","section":"Section 4, paragraph after Eq. (2)"},{"comment":"For the LightBlue teacher, scores are on 1–5, so the formula z := sum_s s p_LLM(s) / N yields values in [0.2, 1], not [0, 1]. Please clarify the exact affine rescaling used.","section":"Section 4, label transformation"},{"comment":"The method name is typeset inconsistently as both 'BiXSE' and 'BIXSE' (e.g., abstract and figure captions use different spellings). Please standardize.","section":"Throughout"},{"comment":"The statement 'BIXSE consistently matches or outperforms alternatives like Pairwise BCE and LambdaLoss' is repeated in the main text, but the appendix data contradict a strict reading. Please align the main-text summary with the actual per-benchmark tables.","section":"§5, 'Comparison to other training objectives'"},{"comment":"The Deep InfoMax relation is explained well, but the transition from representation learning for images to text retrieval could cite the SigLIP adaptation more explicitly in the main text rather than only in the appendix.","section":"Appendix, 'Extended Related Work'"}],"recommendation":"major_revision","confidential_remarks":"The core contribution is simple and plausibly useful, and the InfoNCE comparisons appear extensive. The main blockers are the Table 1 arithmetic inconsistency and the overstatement of the pairwise comparison. Both are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The authors should also be asked to make the BGE-M3 comparison less confounded or to present it as an efficiency trade-off."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the core idea is good, the experiments are broad enough to take seriously, and the paper deserves a real referee. But the abstract overclaims against pairwise baselines, and there are mechanical errors that undercut trust.\n\nWhat's actually new: adapting the SigLIP-style BCE loss with a logit bias to dense retrieval, using LLM-generated graded relevance scores as probabilistic targets. That specific combination is new, and it makes sense. The logit bias for correcting in-batch negative imbalance is a neat, well-motivated fix. The paper shows consistent gains over InfoNCE across several base models and benchmarks, which is the main empirical claim and it holds up reasonably well. The noise-robustness experiment and the filtering analysis are nice additions. The authors also ship code, which is a plus.\n\nThe soft spots. First, the abstract says BiXSE \"consistently matches or exceeds strong pairwise ranking baselines.\" The appendix's own numbers contradict that. On BGE-M3, LambdaLoss nDCG-v2 wins most configurations, often by 3+ points. On LightBlue, LambdaLoss v2 beats BiXSE on several TREC and MTEB rows. The paper itself admits LambdaLoss v2 \"performs close or better ... on several benchmarks.\" So the unqualified \"consistently\" is not supported. The authors could drop that word, say \"competitive with,\" and the contribution would still stand.\n\nSecond, Table 1 is sloppy. The Qwen 2.5-0.5B BEIR row shows BiXSE at 73.91 with +6.0% vs InfoNCE at 38.25. That's arithmetically nonsense—73.91 is 93% higher than 38.25, not 6%. The appendix memory claim about LambdaLoss exceeding 134 GB for a batch of 16,384 also looks off by an order of magnitude; that's the kind of error that makes a careful reader question other numbers. No significance or variance is reported anywhere, which is a real limitation but not fatal.\n\nThird, the underlying premise—that the LLM's averaged softmax scores are faithful relevance probabilities—is not independently validated. That's a fair caveat, but not a killer, since the method works empirically on held-out benchmarks even if the targets are miscalibrated.\n\nThe central argument holds up: BCE with graded targets is a cheaper, scalable alternative that broadly outperforms InfoNCE and is competitive with pairwise losses. I'd recommend the editor send this to peer review, with the expectation of a revision that fixes the numbers and qualifies the pairwise claim. This is the kind of paper I'd bring to a reading group as an example of a useful method with an overhyped abstract.\n\nFor the record: I'd cite this if I were working on dense retrieval training, and the paper reflects clear, honest engagement with the literature despite the presentation issues.","headline":"BiXSE is a solid, practical idea with broad experiments, but the abstract overstates the pairwise comparison and the manuscript has table and appendix errors that need fixing before the claims are taken at face value.","tokens_in":22468,"tokens_out":1505,"would_cite":true,"duration_ms":19228,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a pointwise binary cross-entropy loss over LLM-generated graded relevance probabilities, with a logit bias to offset in-batch negative imbalance, trains dense retrievers that consistently outperform InfoNCE and match","keywords":["dense retrieval","graded relevance","binary cross-entropy","knowledge distillation","LLM-generated labels","contrastive learning","in-batch negatives","sentence embeddings"],"falsifier":"Train two BiXSE encoders on the same LightBlue query-document pairs, one with the raw teacher probabilities $z$ as targets and one with $z$ passed through a calibration step that maps teacher probabilities onto human graded relevance distributions from TREC-DL qrels. If the calibrated-target model does not improve nDCG@10 on held-out TREC-DL 2019–2023 queries, then the claim that $z$ is a faithful relevance probability is not what carries BiXSE's gains; the gains would instead come from the loss shape or the logit bias.","tokens_in":1922,"feed_emoji":"🎯","tokens_out":1897,"duration_ms":111987,"temperature":0.7,"pith_summary":"BiXSE trains dense retrieval models with binary cross-entropy over graded relevance probabilities instead of softmax contrastive learning. The paper argues that a single LLM-generated graded label per query, converted to a probability target, carries enough supervision to match pairwise and listwise distillation methods that need several labeled documents per query. Experiments across retrieval and sentence-embedding benchmarks show that BiXSE consistently outperforms InfoNCE on both binary and graded data, matches or exceeds pairwise baselines on LLM-supervised data, degrades more gracefully under label noise, and lets models learn from a wider range of relevance scores without aggressive filtering. The practical stake is that as LLMs make graded relevance labels cheap and abundant, this pointwise loss is a scalable, annotation-efficient way to distill that supervision into fast encoders.","feed_headline":"BiXSE: one binary loss beats InfoNCE on graded relevance","feed_subtitle":"One graded LLM label per query trains dense retrievers as well as pairwise baselines, at lower cost.","key_machinery":"The mechanism is the BiXSE loss, $$L_{\\mathrm{BiXSE}} = -\\frac{1}{B}\\sum_{i=1}^{B}\\sum_{j=1}^{B} \\left[ z_{i,j}\\log\\$\\sigma$(s(q_i,d_j)) + (1-z_{i,j})\\log\\$\\sigma$(-s(q_i,d_j)) \\right],$$ with $s(q,d)=\\alpha q^{\\top}d + \\beta$, $z_{i,i}$ equal to the graded relevance probability and $z_{i,j}=0$ for $i\\neq j$. The continuous target $z$ is what lets one graded label act as richer supervision than a binary label; the learnable logit bias $\\beta$ is what lets the loss use all in-batch pairs as negatives without pushing all scores down. Because $\\beta$ is optimized at a higher learning rate than the encoder, it absorbs the marginal label distribution, and the encoder must learn query-document content t","core_discovery":"BiXSE replaces the softmax contrastive objective for dense retrieval with a binary cross-entropy loss over probabilistic graded targets. For each query-document pair, the teacher LLM's token probabilities over discrete ordinal relevance options are averaged into a continuous target $z = \\sum_{s\\in S} s \\, p_{\\mathrm{LLM}}(s|q,d)/N$ in $[0,1]$. The student scores a pair as $s(q,d)=\\alpha q^{\\top} d + \\beta$; the loss then penalizes every pair in the batch against $z_{i,i}$ for the labeled document and $0$ for in-batch negatives. The central claim is that this simple pointwise objective extracts more from a single graded label than InfoNCE does from a binary one, and that the learnable logit b","pith_inferences":["Editorial inference: if graded LLM labels become the default supervision source, the practical center of gravity in dense retrieval training may shift from contrastive formulations to pointwise BCE, making hard-negative mining a second-order concern.","Editorial inference: the logit-bias correction is a general device for any in-batch negative training setup; the same bias-tuned-faster trick could be applied to other BCE-style embedding losses, such as cross-modal alignment, where label imbalance is present.","Editorial inference: the paper's noise-robustness gradient argument implies that BCE's advantage should grow with batch size, since one mislabeled pair affects a smaller fraction of BCE terms than of softmax terms; this is directly testable on datasets with known false-negative rates.","Editorial extension: because the method's targets are teacher probabilities, a natural next step is calibrating LLM relevance probabilities against human graded judgments; BiXSE's gains should increase if the teacher's probabilities are well calibrated and degrade if they are not."],"forward_implications":["If BiXSE is right, dense retrieval training no longer needs multiple labeled comparisons per query: one graded LLM label per query plus in-batch negatives matches pairwise baselines.","Because the pointwise BCE loss scales quadratically in batch size while pairwise losses scale cubically, larger batches and larger datasets become feasible within the same memory budget.","The noise-robustness result implies that mined hard-negative datasets with false negatives are less harmful under BCE than under InfoNCE, so less aggressive negative mining should be needed.","BiXSE narrows the gap to zero-shot LLM rankers: a 3B encoder trained this way lands within about three nDCG@10 points of a 32B ranker, suggesting distillation can substitute for expensive LLM inference at serving time.","Training can use a wider spectrum of graded scores without aggressive low-relevance filtering; peak performance comes at a moderate cutoff rather than the strictest one."],"supporting_citations":[{"why":"Supplies the DPR setup and the contrastive training recipe with hard negatives that BiXSE is compared against and generalizes.","marker":"Karpukhin et al. (2020)"},{"why":"Defines InfoNCE, the softmax contrastive baseline BiXSE claims to consistently outperform.","marker":"Oord et al. (2018)"},{"why":"Provides the large-scale LLM-graded relevance dataset with token-level probabilities that BiXSE uses to construct continuous training targets.","marker":"Lightblue, 2025"},{"why":"Shows that fine-grained relevance labels improve zero-shot LLM rankers, motivating the graded supervision BiXSE exploits.","marker":"Zhuang et al. (2024)"},{"why":"Introduces the BCE formulation with a learnable bias for image-text alignment that BiXSE adapts to text retrieval.","marker":"Zhai et al. (2023)"},{"why":"Provides Soft InfoNCE, a graded-label contrastive baseline that BiXSE compares against and outperforms.","marker":"Cheng et al. (2023)"},{"why":"Provides MarginMSE, a distillation baseline that cannot use in-batch negatives; BiXSE's logit bias is introduced to address this limitation.","marker":"Hofstätter et al. (2021)"},{"why":"Defines RankNet's pairwise BCE, one of the pairwise ranking baselines BiXSE matches while requiring fewer labels.","marker":"Burges et al. (2005)"},{"why":"Defines LambdaLoss, the pairwise baseline in the efficiency comparison; BiXSE matches or exceeds it with fewer labeled negatives.","marker":"Wang et al. (2018)"},{"why":"Supplies the public E5 binary-relevance dataset and task instructions used in the main comparisons and the label-noise experiment.","marker":"Wang et al. (2022)"}],"fun_headline_variants":["Binary loss with graded labels outperforms InfoNCE in dense retrieval","A single graded label replaces pairwise ranking for dense retrievers","BiXSE: pointwise BCE beats contrastive learning on graded relevance","Lower-cost dense retrieval with one LLM-graded label per query","Pointwise BCE: graded relevance without pairwise annotations"],"cache_read_input_tokens":24320,"weakest_assumption_plain":"The load-bearing premise is that LLM-generated graded scores, converted to probabilities by averaging the teacher's softmax over discrete options and rescaling to [0,1], are faithful relevance probabilities for every query-document pair; if these targets are miscalibrated or systematically biased, the BCE loss trains the encoder toward a distorted notion of relevance.","fun_headline_variants_meta":{"raw":{"variants":["Binary loss with graded labels outperforms InfoNCE in dense retrieval","A single graded label replaces pairwise ranking for dense retrievers","BiXSE: pointwise BCE beats contrastive learning on graded relevance","Lower-cost dense retrieval with one LLM-graded label per query","Pointwise BCE: graded relevance without pairwise annotations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000586,"raw_usage":{"total_tokens":2607,"prompt_tokens":775,"completion_tokens":1832,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1746}},"tokens_in":519,"tokens_out":1832,"duration_ms":13974,"temperature":1.0,"reasoning_tokens":1746,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:30:57.246862+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two BiXSE encoders on the same LightBlue query-document pairs, one with the raw teacher probabilities $z$ as targets and one with $z$ passed through a calibration step that maps teacher probabilities onto human graded relevance distributions from TREC-DL qrels. If the calibrated-target model does not improve nDCG@10 on held-out TREC-DL 2019–2023 queries, then the claim that $z$ is a faithful relevance probability is not what carries BiXSE's gains; the gains would instead come from the loss shape or the logit bias.","supporting_citations":[{"cited_title":"Dense passage retrieval for open-domain question an- swering","cited_arxiv_id":null,"evidence_quote":"Supplies the DPR setup and the contrastive training recipe with hard negatives that BiXSE is compared against and generalizes."},{"cited_title":"Improving con- trastive learning of sentence embeddings from AI feedback","cited_arxiv_id":null,"evidence_quote":"Provides Soft InfoNCE, a graded-label contrastive baseline that BiXSE compares against and outperforms."}],"review_version":1}