{"id":"c4294875-19a5-407d-8e0d-face8005914d","arxiv_id":"2507.06895","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A no-fine-tuning, contrastive-learning plus Bayesian kNN system matches or exceeds state-of-the-art relation extraction on several benchmarks at a fraction of the energy cost.","lead":"SCoRE is a relation extraction system that uses a frozen language model, contrastive learning, and a Bayesian k-nearest-neighbor classifier to label relationships between entities without fine-tuning. It matches or beats several larger systems on five benchmarks while using much less energy, and it adds new evaluation metrics and a noisy benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 7's 'flat prior' makes kNN label aggregation a uniform average over neighbors, so the claimed Bayesian, class-prior-aware posterior reduces to standard distance-weighted kNN; the load-bearing assumption is that CL geometry alone makes this work on frozen BERT embeddings, yet the paper's own…","rationale":"The reader's weakest_assumption pointed to the frozen-BERT premise, and that is real but not the most dangerous issue: the paper's own anti-ablation results provide some indirect support for using frozen embeddings, and nothing in the text contradicts the premise as a design choice. The more concrete and falsifiable problem is the Bayesian kNN posterior itself, which is the second half of the method and is what produces the reported scores.\n\nI agree with the reader that the paper is conditionally acceptable in spirit, but the combination of (1) a mis-specified loss whose sign is inconsistent with Eq. 6, (2) a posterior that under the flat prior is not actually prior-corrected or label-frequency-corrected, and (3) main-table results without any variance reporting while Table 6 shows large run-to-run variance, makes the headline 'matches or surpasses SOTA' unsupported as written. The CSD metric is also formally broken because Eq. 13 mixes true and predicted labels in the n01 and n10 counts, making the reported CSD numbers uninterpretable as a distance between two correlation matrices.\n\nThese are not stylistic complaints; they bear directly on whether the central claim survives. A single re-run of the released pipeline with seeds and a small code inspection would determine whether the concern lands. If the authors provide the exact code, seed-averaged results, and fix the loss sign, the paper would be a solid contribution, which is why I moved from CONDITIONAL to a conditional-style REJECT for the current manuscript, with a clear path to resubmission. This is the most honest verdict given the current evidence.","tokens_in":24716,"tokens_out":2141,"duration_ms":20795,"concrete_test":"Run the released SCoRE code on Wiki20D and NYT10D with 10 fixed random seeds, and report mean +/- std for microF1 and macroF1 using the exact UU configuration from Table 6. If the Wiki20D microF1 spread (e.g., 65.1 +/- 2.5) overlaps the reported PARE value or the IC value under a simple t-test, the headline claim of superior performance on Wiki20D loses support. Additionally, inspect the training loop to verify whether the loss is exp(d/tau) as printed in Eq. 4 or exp(-d/tau) as used in Eq. 6; if the code uses the negative sign, the paper's Eq. 4 must be corrected and re-derived before the metric-consistency argument can be evaluated.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that SCoRE matches or surpasses SOTA on five benchmarks at far lower energy, driven by the design choice of a frozen PLM plus CL and Bayesian kNN. The load-bearing concern is the stability and validity of the kNN posterior used for the headline results.\n\nIn Eq. 7 the method sets P(r_h) = 1/2, so the prior and P(r_h) cancel in both numerator and denominator. The posterior then becomes a distance-weighted average of neighbor labels with weights exp(-d/tau), with no class-frequency correction and no Bayesian adjustment. The paper's own Table 6 shows the chosen UU configuration gives microF1 65.1 +/- 2.5 on Wiki20D, while the IC configuration gives 67.2 +/- 1.5 and the IU configuration collapses to 24.5 +/- 3.0. The sensitivity is enormous, but the main results Table 2 are single best-of-five runs with no error bars, so it is unclear whether SCoRE's reported advantage on Wiki20D is within run-to-run variance.\n\nSecond, the CL loss sign is internally inconsistent: Eq. 4 maximizes exp(d/tau), i.e., it pulls apart similar points if d is the standard Euclidean/cosine distance, while Eq. 6 and the text assume exp(-d/tau) for the kNN weights. The mismatch means the metric induced by the loss is not the metric used in Eq. 7, so the 'CL-enforced' geometry claim is unsupported as written. If the implemented loss is the standard supervised contrastive loss with negative distance, then Eq. 4 as printed is wrong; either way, the math and the code need checking.\n\nThird, CSD as defined in Eq. 13 mixes true labels (n01 uses y_p) and predicted labels (n10 uses y_hat_p), so the correlation matrix distance compares an asymmetric, inconsistently labeled matrix against ground truth; the reported CSD numbers are therefore not a well-defined 'correlation matrix distance' as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SCoRE, a sentence-level multi-label relation extraction system. It encodes each sentence once with a frozen BERT-base model, averages the head and tail entity token embeddings, trains a small MLP with a supervised multi-label contrastive loss, and predicts labels at inference using a Bayesian kNN over the training embeddings with a shared probability threshold. The authors evaluate on five benchmarks (NYT10M, NYT10D, DisRex, Wiki20m, Wiki20D), compare with five SOTA systems, report micro/macro-F1 and energy consumption, and propose two new metrics (P@R and CSD) plus a new fully distantly supervised benchmark, Wiki20d. The central empirical claim is that SCoRE matches or surpasses SOTA while consuming far less energy.","tokens_in":24991,"tokens_out":8380,"duration_ms":90340,"significance":"If the claims held, this would be a useful contribution: it would show that a frozen PLM plus contrastive training and non-parametric inference is competitive with heavily fine-tuned systems, with real advantages in modularity and reproducibility. The release of Wiki20d, the explicit energy measurements, and the anti-ablation experiments are concrete strengths. However, the manuscript currently contains a sign inconsistency in the contrastive loss, an over-stated Bayesian interpretation of the kNN posterior, an unsupported claim of outperformance on Wiki20D, and an incorrectly specified new metric. These issues are load-bearing for the headline results and must be resolved before the contribution can be properly assessed.","major_comments":[{"comment":"Equation (4) defines the contrastive loss with exp(d(z_i,z_j)/tau), where d is a distance (Euclidean or cosine), but Equations (6) and (7) and the surrounding text use exp(-d/tau). With a literal distance in Eq. (4), maximizing the numerator would pull positive pairs apart, the opposite of the claimed clustering behavior. If the implementation uses -d, the printed equation is wrong; if it uses +d, the training objective is inconsistent with the metric used in the kNN likelihood of Eq. (6). Because the paper claims that the kNN exploits the metric induced by CL, this sign issue is central and must be corrected and verified in the released code.","section":"3.2, Eq. (4)"},{"comment":"With the stated flat prior P(r_h)=1/2, the prior cancels algebraically in Eq. (7), leaving the posterior as a distance-weighted average of neighbor labels, sum_i y_i^h exp(-d/tau) / sum_i exp(-d/tau). This is standard distance-weighted kNN, with no class-frequency correction and no informative prior. The claims that the flat prior 'focuses on optimizing recall' or 'adopts a dynamic, local neighborhood-dependent class weight' are not derived from Eq. (7). The authors should either provide a concrete Bayesian contribution that remains after the algebra or stop describing the default configuration as class-prior-aware.","section":"3.3, Eq. (7)"},{"comment":"The text states that SCoRE 'outperforms competitors on NYT10D, NYT10M, and Wiki20d', but Table 2 lists PARE at 67.2 and HiCLRE at 68.3 micro-F1 on Wiki20D, both above SCoRE's 66.9. Furthermore, Table 2 reports the best of five runs with no variance, while Table 6 gives the UU configuration on Wiki20D as 65.1 ± 2.5 over 10 runs; the reported 66.9 is therefore within one standard deviation of the mean, and the SOTA gap is not statistically established. Please report means and standard deviations for the main comparison and qualify the 'matches or surpasses' claim accordingly.","section":"7.1, Table 2"},{"comment":"The definition of n_10 in Eq. (13) uses \\hat y_h^j (1 - y_p^j), mixing predicted and true labels inside a single phi coefficient, while n_01 uses (1 - y_h^j) y_p^j. As written, phi is not a well-defined correlation for either the true or the predicted label matrix. If the intended computation applies Eq. (13) separately to the true matrix and to the predicted matrix, the hat in n_10 is a typo and must be removed; otherwise the CSD values in Tables 3 and 6 are not reproducible from the formula given.","section":"4.5, Eq. (13)"},{"comment":"The energy comparison is not on equal footing. The PARE SL and PARE BAG entries appear to include the entire process in one measurement, while SCoRE's 'Train & Test' column excludes the dataset-creation phase reported in a separate column (e.g., 0.63 kWh for NYT10M and 5.75 kWh for Wiki20D). For a fair comparison, the total SCoRE cost (dataset creation plus training and testing) should be compared against the PARE totals; otherwise the headline claim of reducing energy consumption by 'up to three orders of magnitude' is overstated for the smaller datasets.","section":"7.1, Table 4"}],"minor_comments":[{"comment":"There is a duplicated word in 'we estimate testsetrelationtypeprobabilitiesprobabilities right after CL training'; please fix the typo.","section":"3.3"},{"comment":"The baseline SCoRE numbers in Table 5 differ slightly from Table 2 (e.g., NYT10M 77.4 vs 77.6, Wiki20D 66.8 vs 66.9) with no explanation; clarify the relationship between the two tables.","section":"7.1 vs 7.2"},{"comment":"The text calls y_{i,j} a 'one-hot encoding', but for multi-label samples it is a binary label vector with potentially several ones; please use 'binary label vector' to avoid confusion.","section":"3.1"},{"comment":"The margin notation n_{1·} and n_{·1} in Eq. (13) is ambiguous because the text later says phi is computed on the true and predicted matrices separately; please add explicit superscripts to distinguish the two computations.","section":"4.5"},{"comment":"The paragraph states that 'no single setup consistently outperforms the others' but then says UU 'confirms to be the best' with 16 podium positions; make the selection criterion explicit, since podium counts and mean metrics can disagree.","section":"7.3"}],"recommendation":"major_revision","confidential_remarks":"The contribution is potentially publishable in a specialized NLP/data-mining venue, but the sign error in Eq. (4), the flattening of the Bayesian posterior under the flat prior, the contradictory Wiki20D claim, and the ill-defined CSD formula need to be fixed with code verification and rerun statistics before the paper can be recommended for acceptance. The novel dataset and the energy analysis are assets, but the current framing overstates both the Bayesian novelty and the empirical advantage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: SCoRE is a believable, cheap no-finetune pipeline for sentence-level RE, and its most valuable result is negative—adding complexity (CLS tokens, attention, dynamic encoding) hurts when you don't fine-tune. The second thing: the math is sloppy in three places, and the headline advantage on Wiki20D is fragile once you look at their own sensitivity table.\n\nWhat's new: combining multi-label supervised contrastive learning with a Bayesian kNN on frozen BERT embeddings is genuinely new for RE. The paper is honest about competitors it couldn't reproduce, and it ships code and a new benchmark (Wiki20d) for fully distantly supervised settings. The energy comparison is plausible in direction, though the PARE adaptation (sentence-by-sentence inference) inflates PARE's cost—fine for a sentence-level comparison, but not a fair \"system\" comparison.\n\nSoft spots, in proportion. First, Eq. 4 as printed uses exp(d/tau) with d described as a distance; that would pull similar points apart. Eq. 6 uses exp(-d/tau). One is a typo, but as written the paper is internally inconsistent, and the CL-induced metric claim is unsupported. Second, with a flat prior, Eq. 7 collapses to a distance-weighted vote over neighbor labels—the \"Bayesian\" framing is decorative. That's fine, but it should be presented that way. Third, CSD as defined mixes true labels in n01 and predicted labels in n10, so it's not a well-defined correlation matrix distance; the reported numbers need redefinition. Fourth, Table 2 has no error bars, while Table 6 shows Wiki20D microF1 ranging from about 24 (IU) to 67 (IC) depending on kNN configuration. So the reported SCoRE advantage over PARE on Wiki20D (66.9 vs 67.2) is actually a slight loss in micro-F1; the \"surpasses\" claim on that dataset rests on macro-F1, which should be said plainly. The load-bearing assumption that frozen BERT embeddings capture enough relational signal is only supported indirectly by the anti-ablation; an independent probe would strengthen the paper.\n\nWho is this for: people working on efficient or no-finetune approaches to RE, and anyone building benchmarks for noisy supervision. It deserves serious peer review; the method is simple, reproducible in principle, and the negative results are useful. It needs the math cleaned up, error bars added, and the metric definitions fixed before it becomes a reliable reference.","headline":"A believable no-finetune RE system whose most interesting result is negative, but the math is sloppy in three places and the headline numbers need error bars before I'd trust them.","tokens_in":25710,"tokens_out":3294,"would_cite":true,"duration_ms":33506,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SCoRE claims that a frozen-embedding, no-fine-tuning pipeline matches or outperforms state-of-the-art relation extraction on five benchmarks while using orders of magnitude less energy.","keywords":["multi-label relation extraction","distant supervision","knowledge graph enrichment","Bayesian kNN","contrastive learning","frozen pretrained language models","energy-efficient NLP","Wiki20d"],"falsifier":"Construct a test set where the same head–tail entity pair appears in sentences that express different relations (for example, 'X was born in Y' versus 'X lives in Y'), and check whether SCoRE can separate them, since its input uses only the averaged mention embeddings and ignores the rest of the sentence. A second decisive check is to replace the frozen BERT-base encoder with randomly initialized embeddings of the same dimension: if micro-F1 on NYT10M stays near the reported 77.6, the relational signal is not coming from pretrained language knowledge; if it collapses, the frozen-encoder assumption is doing the work.","tokens_in":24373,"feed_emoji":"⚡","tokens_out":11672,"duration_ms":113901,"temperature":0.7,"pith_summary":"This paper proposes SCoRE, a sentence-level relation extraction system that deliberately avoids the usual recipe of fine-tuning a large pretrained language model. Instead, it runs a single frozen BERT-base pass to average the embeddings of the head and tail entity mentions, trains a small MLP with a multi-label supervised contrastive loss, and classifies new mentions with a Bayesian k-nearest-neighbor rule on the learned hypersphere. The authors claim that on five benchmarks—NYT10M, NYT10D, DisRex, Wiki20m, and their new Wiki20d—this minimal pipeline matches or beats state-of-the-art methods while consuming as little as 0.002 kWh for training and testing, orders of magnitude below a fine-tuned competitor. They also argue that extra complexity hurts when fine-tuning is removed: adding the CLS token, full-sentence attention, or dynamic encoding degrades performance. If right, the paper shows that the relational signal needed for extraction already lives in frozen PLM embeddings, and that heavy fine-tuning is not the price of accuracy.","feed_headline":"Frozen BERT plus Bayesian kNN matches fine-tuned relation extractors","feed_subtitle":"Two datasets set new best micro-F1, and all five run at a small fraction of the energy.","key_machinery":"The load-bearing machinery is the metric space built by supervised contrastive learning and then exploited by a non-parametric classifier. During dataset creation the frozen PLM maps each sentence once to token embeddings; the head and tail mentions are averaged and concatenated, $\\mathbf{x}_{i,j}=[\\mathbf{e}_{i,h_j};\\mathbf{e}_{i,t_j}]$. A shallow MLP with normalized output projects these vectors onto the hypersphere under a multi-label supervised contrastive loss in which the pair weight $\\beta_{ij}$ is the normalized number of shared relation labels, so the distance $d(\\cdot,\\cdot)$ in the loss becomes relation-aware. At inference, Bayesian kNN estimates each relation posterior from the $k$ nearest neighbors weighted by $\\exp(-d/\\tau)$ with a flat prior $P(r_h)=1/2$, and a universal threshold produces the final multi-label prediction. The key design point is that the same distance measure is used in training and inference, so the learned geometry and the classifier are one and the same.","core_discovery":"The paper's claim is that sentence-level multi-label relation extraction does not need fine-tuned pretrained language models. SCoRE encodes every sentence once with frozen BERT-base, averages the embeddings of the head and tail entity mentions, concatenates the two averages, and trains only a small MLP under a multi-label supervised contrastive loss; inference is a Bayesian kNN over the learned hypersphere. On the five benchmarks the authors report the best micro-F1 on NYT10M (77.6) and NYT10D (89.2), the best macro-F1 on Wiki20D (23.9), competitive micro-F1 on DisRex (75.4 vs 77.1) and Wiki20m (83.5 vs 83.8), and the lower CSD on every dataset when compared with the strongest fine-tuned baseline, PARE. Training plus testing consumes 0.002–0.06 kWh, versus 2.11–6.20 kWh for bag-level PARE and much more for its sentence-level adaptation, so the paper concludes that accuracy and cost are not in tension the way the field's fine-tuning-centric practice assumes.","pith_inferences":["A natural extension the paper does not run: replace BERT-base with a larger frozen PLM and check whether the same no-fine-tuning pipeline improves long-tail relations, since modularity claims would predict monotonic gains.","The CSD metric could be turned into a training or model-selection signal: if SCoRE's lower CSD is what protects KG structure, one could validate threshold and neighbor choices by CSD rather than F1 alone.","In a human-in-the-loop KG-enrichment workflow, SCoRE's stable P@R across thresholds suggests it is best deployed as a ranker that suggests candidate relations for expert confirmation, rather than as a hard classifier."],"forward_implications":["Because the PLM is used only in a single forward pass during dataset creation, swapping in a newer or larger PLM costs an embedding re-run rather than a full re-training, making the system track the fast-moving PLM ecosystem cheaply.","The anti-ablation results imply that without fine-tuning, additional input tokens (CLS), full-sentence attention, and dynamic encoding actively reduce micro- and macro-F1 while multiplying energy use, so SCoRE's minimal design is not a compromise but a better operating point for the no-fine-tuning regime.","The CSD analysis shows that a fine-tuned model can earn competitive F1 while conflating inverse relations (e.g., 'followed' vs 'followed-by'), so SCoRE's sentence-level, non-parametric inference is claimed to preserve the directed structure of the KG better.","The sensitivity analysis identifies flat prior plus universal threshold as the recommended configuration and argues that @M ranking metrics are unreliable in multi-label RE, with P@R better reflecting real-world recommender use."],"supporting_citations":[{"why":"Supplies the supervised contrastive loss whose multi-label adaptation (via [38]) is SCoRE's training objective.","marker":"[19]"},{"why":"Provides the multi-label adaptation of the CL loss and the nearest-neighbor datastore idea used at inference.","marker":"[38]"},{"why":"Gives the Bayesian posterior formulation and the flat-prior prescription used by the Bayesian kNN classifier.","marker":"[13]"},{"why":"PARE is the main fine-tuned baseline that SCoRE matches or beats on F1, CSD, and energy.","marker":"[29]"},{"why":"Contributes the Wiki20m and NYT10m manually annotated benchmarks and test protocols SCoRE builds on; Wiki20d is its extension.","marker":"[12]"},{"why":"Introduced the NYT10d distantly supervised benchmark that anchors the noisy-DS evaluation.","marker":"[31]"},{"why":"Provides the DisRex multilingual distant-supervision benchmark; SCoRE uses its English portion.","marker":"[2]"},{"why":"Measures energy consumption in kWh, backing the paper's central efficiency claim.","marker":"[8]"}],"fun_headline_variants":["Frozen BERT plus Bayesian kNN matches fine-tuned RE","No fine-tuning needed: SCoRE matches SOTA relation extraction","Frozen embeddings and Bayesian kNN: low-cost relation extraction","SCoRE: fine-tuned accuracy without the fine-tuning cost","Relation extraction without fine-tuning: SCoRE beats energy budget"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that averaging the frozen BERT-base embeddings of just the head and tail entity mentions preserves enough relational information to separate all relation types; if those averages are relationally uninformative, no amount of contrastive learning or Bayesian kNN can recover the signal.","fun_headline_variants_meta":{"raw":{"variants":["Frozen BERT plus Bayesian kNN matches fine-tuned RE","No fine-tuning needed: SCoRE matches SOTA relation extraction","Frozen embeddings and Bayesian kNN: low-cost relation extraction","SCoRE: fine-tuned accuracy without the fine-tuning cost","Relation extraction without fine-tuning: SCoRE beats energy budget"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1628,"prompt_tokens":1050,"completion_tokens":578,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":489}},"tokens_in":666,"tokens_out":578,"duration_ms":6657,"temperature":1.0,"reasoning_tokens":489,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:52:25.905843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a test set where the same head–tail entity pair appears in sentences that express different relations (for example, 'X was born in Y' versus 'X lives in Y'), and check whether SCoRE can separate them, since its input uses only the averaged mention embeddings and ignores the rest of the sentence. A second decisive check is to replace the frozen BERT-base encoder with randomly initialized embeddings of the same dimension: if micro-F1 on NYT10M stays near the reported 77.6, the relational signal is not coming from pretrained language knowledge; if it collapses, the frozen-encoder assumption is doing the work.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-label adaptation of the CL loss and the nearest-neighbor datastore idea used at inference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the Bayesian posterior formulation and the flat-prior prescription used by the Bayesian kNN classifier."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"PARE is the main fine-tuned baseline that SCoRE matches or beats on F1, CSD, and energy."},{"cited_title":"Manual evaluation matters: Reviewing test protocols of distantly supervised relation extraction, in: Zong, C., Xia, F., Li, W., Navigli, R","cited_arxiv_id":null,"evidence_quote":"Contributes the Wiki20m and NYT10m manually annotated benchmarks and test protocols SCoRE builds on; Wiki20d is its extension."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduced the NYT10d distantly supervised benchmark that anchors the noisy-DS evaluation."},{"cited_title":"DiS-ReX: A multilingual dataset for distantly supervised relation extraction, in: Muresan, S., Nakov, P., Villavicencio, A","cited_arxiv_id":null,"evidence_quote":"Provides the DisRex multilingual distant-supervision benchmark; SCoRE uses its English portion."},{"cited_title":"mlco2/codecarbon: v2.4.1","cited_arxiv_id":null,"evidence_quote":"Measures energy consumption in kWh, backing the paper's central efficiency claim."}],"review_version":1}