{"id":"45cdc54a-78c3-4a0a-b328-577b0236c3c3","arxiv_id":"2502.05933","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A BERT model fine-tuned with ranking losses against BARTScore substitutes words to improve that score, outperforming supervised and LLM baselines on BARTScore-based metrics without human labels.","lead":"The paper trains a word-substitution model without human labels, using BARTScore, an automated sentence-quality score, as the training signal and evaluation metric. If the metric matches human judgment, the method could cut annotation costs for writing-assistance tools and beat GPT-4 and LLaMA on its chosen benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'outperforms' claim is supported mainly by metrics that are also the training objective (CS and ABR); the only external check, a 50-case human study, is too small to validate BARTScore as a proxy or to compare against LLMs.","rationale":"The reader's weakest assumption is that BARTScore is a valid proxy for human-perceived substitution quality. I agree this premise is load-bearing and under-supported, but I would sharpen the concern: even if BARTScore were a perfect proxy, the paper's headline comparison would still be weak because CS and ABR are the very objectives used to train MR+AS. The GPTScore results are only weakly independent given the high reported correlation with BARTScore. The human study is the only genuinely external evidence, and it is too small, shows low annotator agreement on replacement decisions, and does not compare MR+AS against any strong baseline. These observations reinforce the reader's conditional verdict rather than overturning it: the method is clearly specified and the label-free training idea is plausible, but the empirical claim of superiority needs an external, human-grounded evaluation. My recommended verdict is therefore unchanged: conditional acceptance pending such an evaluation.","tokens_in":18122,"tokens_out":7705,"duration_ms":78736,"concrete_test":"Run a pre-registered human forced-choice study on 100+ sentences sampled from the SWS test set. For each sentence, mask the target word and show, in randomized order, the top-1 substitution from MR+AS and the top-1 substitution from GPT-4o, without revealing the source. Ask at least five annotators to choose the better substitution or mark a tie. If MR+AS does not achieve a significant win rate over GPT-4o (e.g., 95% CI excluding 50%), the 'outperforms LLMs' claim is unsupported. As a cheaper complementary check, score all models' top-1 substitutions with BLEURT, a metric trained on human judgments and never used during training; if MR+AS does not lead under BLEURT, the current results likely reflect reward overfitting rather than general task quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparative claim, that MR+AS outperforms both MLMs and LLMs, rests on CS and ABR in Tables 2 and 3. But these are the same quantities optimized by the training losses. Eq. (4) (LAS) directly maximizes a softmax-weighted sum of BARTScore, and Eq. (3) (MR) aligns model logits with BARTScore order. At test time, CS measures cosine similarity between logits and BARTScore, and ABR measures average BARTScore of modified sentences; both are essentially the training reward. Thus the benchmark largely measures how well the model fits its own reward, not task quality. The GPTScore tables in the Appendix are presented as independent, but the paper reports Spearman correlations of 0.929 and 0.921 between BARTScore and the two GPTScore variants, so these checks are far from independent. The only direct human evidence is small: 25+25 cases, with replacement-agreement Cohen's kappa of 0.13+/-0.21, 38.8% ties, and in the 25 model/BARTScore disagreement cases annotators sided with BARTScore only 67% of the time. This is too limited to establish that BARTScore is a valid proxy for human judgment, and it never compares MR+AS directly to GPT-4o or LLaMA. Granting for the sake of argument that BARTScore is a reasonable quality proxy, the experiments still do not demonstrate that the model beats baselines on any metric not used as a training signal.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MR+AS, a method for smart word substitution that fine-tunes BERT without human labels by using BARTScore as a reward. The training objective combines a margin ranking loss (Eq. 3) that aligns token logits with the order of BARTScore values and an average-score loss (Eq. 4) that maximizes a softmax-weighted sum of BARTScore. The authors compare MR+AS against MLMs (BERT, BART), a rule-based thesaurus method, and LLMs (GPT-4o, LLaMA) on four datasets, reporting that MR+AS achieves the best CS and ABR metrics, both computed from BARTScore. They also report a model-based p-value statistic (Eq. 2) and a small human study (25+25 cases) as external checks.","tokens_in":18492,"tokens_out":6517,"duration_ms":61682,"significance":"If the claims were established, the approach would be a useful contribution: it removes the need for human annotations in substitution tasks, gives a general recipe for aligning token predictions with a model-based quality score, and would show that a comparatively small BERT model can outperform much larger LLMs on a writing-assistance task. The manuscript is clearly organized, the code is released, and the ablation over MR, DPO variants, MR+BS, and MR+AS is informative. However, the main comparative claim is currently supported only by metrics that are essentially the training objective, and the external human validation is too thin to break the circularity. With additional independent evaluation, the result could be significant, but the present evidence is insufficient.","major_comments":[{"comment":"The headline comparison uses CS and ABR, both defined via the same BARTScore that appears in the training losses. Eq. (3) directly encourages logits to be ordered by BARTScore, and Eq. (4) directly maximizes a softmax-weighted average of BARTScore. Consequently, MR+AS's near-perfect CS (0.98-0.99) and its ABR advantage over baselines are expected by construction; they show that the model fits its reward, not that it produces better substitutions. The Appendix GPTScore tables do not resolve this, because the paper reports Spearman correlations of 0.929 and 0.921 between BARTScore and the two GPTScore variants. An independent evaluation—either a larger human study that pits MR+AS against GPT-4o/LLaMA head-to-head, or a held-out reward model not used in training—is needed before the 'outperforms' claim can be accepted.","section":"Section 4, Evaluation Metrics and Tables 2-3"},{"comment":"The validity of BARTScore as a proxy for human substitution quality is the load-bearing assumption of the whole method, and the paper's own Limitations section concedes that 'we cannot eliminate the possibility that BARTScore may not always accurately quantify the quality of token substitutions.' The only direct check is the human study in Section 4. It uses only 25+25 cases, reports replacement-agreement Cohen's kappa of 0.13±0.21, has 38.8% ties, and in the disagreement subset annotators sided with BARTScore only 67% of the time. This is far too thin to establish BARTScore as a valid reward, and it never compares MR+AS against GPT-4o or LLaMA on human judgment. Without a stronger external validation, the central claim that the method improves sentence quality is not supported.","section":"Section 3.1 and Limitations"},{"comment":"The loss L_BS = max(0, (M(X) - M(\\tilde X_1)) f(s_1)) contains an undefined function f. This makes the MR+BS variant (Eq. 7) unreproducible, since no definition or reference is given for f in the text, equations, or appendix. Either define f and specify its role, remove MR+BS from the ablation, or report the specific choice used to produce the MR+BS results in Table 2.","section":"Eq. (5)"},{"comment":"The DPO* and σDPO* losses are presented as derived from DPO, but the derivation replaces log p_θ(\\tilde X_k) with the token logit s_k and drops the softmax denominator, yielding losses that are no longer DPO under either the Plackett-Luce or Bradley-Terry model. This substitution of token-level logits for sentence-level log-likelihoods is not justified in the paper. Since these variants underperform MR+AS, this does not overturn the main result, but as proposed the derivation is not valid and the name 'DPO*' is misleading.","section":"Section 3.2 and Appendix A"}],"minor_comments":[{"comment":"The sentence defining the sorted order uses 'for i, j = 1, ..., K' but i is never used; the condition should be over pairs k, j.","section":"Eq. (3)"},{"comment":"The quantity in Eq. (2) is described as a p-value and used in a hypothesis test, but it is an empirical proportion over the model's own candidate set. Calling it a p-value and 'rejecting the hypothesis' for p < α is statistically imprecise and could mislead readers about the meaning of Table 4.","section":"Section 3.1, Eq. (2)"},{"comment":"The abstract and contributions state that the method 'defin[es] a distribution for each word substitution,' but the paper actually defines a rank statistic (Eq. 2) and a set of loss functions; there is no distribution over substitutions. Please rephrase to match the content.","section":"Abstract and Section 1"},{"comment":"The second row of Table 1 is misaligned ('Not change14138'), making the counts hard to read; also, the row/column totals should be verified, since 14138 + 969 = 15107 appears to be the row total for 'Not change' tokens.","section":"Table 1"},{"comment":"The human study reports only average agreement percentages without confidence intervals or per-item breakdowns for the 79% and 67% statistics; given the small sample, this makes the strength of the agreement hard to assess.","section":"Section 4, Human Study"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the code release is a plus. The main concern is evaluation circularity: the reported superiority on CS and ABR is largely a measure of how well the model optimizes its own training reward. I believe the result could become publishable if the authors add a meaningful human evaluation that compares MR+AS directly with GPT-4o and LLaMA, or use a held-out reward model that was not part of the training loss. I do not see evidence of misconduct, but the abstract's 'outperforms GPT-4, LLaMA' claim should be softened or supported by such independent evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: this paper has a genuinely new training recipe for word substitution without human labels, but its main evaluation is mostly measuring how well the model fits its own reward. The \"outperforms GPT-4o and LLaMA\" claim should be taken with a large grain of salt.\n\nWhat's new: prior work used BARTScore to rank candidates at inference (ParaLS); this paper uses BARTScore as a training signal, defining a margin ranking loss and a score-improving loss, and trains a BERT model label-free. The loss combination (MR+AS, Eq. 6) is clearly specified, and the code is public. The limitations section honestly acknowledges that BARTScore may not always track human quality. That's more than many papers do.\n\nThe soft spot is the evaluation. CS and ABR are the same quantities the training loss directly maximizes: Eq. (4) is a softmax-weighted average BARTScore, Eq. (3) aligns logits with BARTScore order, and the test metrics are cosine similarity to BARTScore and average BARTScore ratio. So the gains in Tables 2-3 are largely expected by construction. The GPTScore appendix check is not independent: the paper itself reports Spearman correlations of 0.93 and 0.92 between BARTScore and the two GPTScore variants, so that's nearly the same signal. The human study is the only direct check, and it's tiny: 25 cases for ordering, 25 for model-vs-BARTScore disagreement, with a replacement-agreement kappa of 0.13±0.21 and 38.8% ties. In the disagreement cases, annotators sided with BARTScore only 67% of the time. That's too weak to establish BARTScore as a human-quality proxy, and it never compares MR+AS directly to LLMs.\n\nThere's also a small technical flaw: Eq. (5) uses f(s1) without ever defining f. Not fatal, but should be fixed.\n\nNet: the paper is a reasonable contribution to a niche subfield. If you work on writing assistance or on label-free training with automated metrics, it's worth reading and citing. As a referee, I would ask for a larger human study (or an external human-labeled benchmark like LS07 with standard metrics) and an evaluation with a score that isn't the training reward. As submitted, the central comparative claim is not established.\n\nRecommendation: send to peer review, but expect heavy revision.","headline":"A label-free training recipe for word substitution that is clearly specified and code-released, but the evaluation is mostly circular — the gains over GPT-4o/LLaMA are on the metric that the training loss maximizes.","tokens_in":18970,"tokens_out":2916,"would_cite":false,"duration_ms":27265,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BARTScore can replace human labels for word-substitution training and evaluation.","keywords":["smart word substitution","BARTScore","model-based evaluation","margin ranking loss","preference optimization","BERT fine-tuning","learning without human labels","lexical substitution"],"falsifier":"Take a few thousand substitution pairs, have a diverse panel of annotators rank them by preference, and compute the rank correlation between human preference and BARTScore; if the correlation is near zero or negative, the premise that BARTScore can stand in for human quality judgment collapses.","tokens_in":17934,"feed_emoji":"📝","tokens_out":8187,"duration_ms":72521,"temperature":0.7,"pith_summary":"Smart word substitution—deciding which words to replace and what to replace them with—has relied on human-annotated datasets that are costly and inconsistent across annotators. This paper claims that a purely model-based score, BARTScore, can serve as both the training signal and the evaluation metric, eliminating the need for human labels while preserving substitution quality. The authors fine-tune BERT with a loss that couples margin ranking with an average-score term, and report that it outperforms masked language models (BERT, BART) and large language models (GPT-4o, LLaMA) on score alignment and quality across four datasets. If true, this points to a general recipe for refining text without manual annotation: take a trainable model, a frozen evaluator, and a ranking objective that aligns the two.","feed_headline":"BARTScore replaces human labels for word-substitution training","feed_subtitle":"Fine-tuned BERT with a ranking-plus-score loss beats BERT, BART, GPT-4o, and LLaMA on substitution quality.","key_machinery":"The central machinery is BARTScore, a model-based evaluator defined as the conditional log-likelihood of a sentence under a pre-trained BART model. It plays three roles: it defines the reference distribution for a significance statistic of substitution quality, it supplies the target that the margin-ranking loss aligns the model's logits to, and it is itself the evaluation metric (CS and ABR). The proposed objective combines the margin ranking loss $L_{\\text{MR}}$ with an average-score loss $L_{\\text{AS}}$, so the model is trained to rank candidates like BARTScore and to raise the average BARTScore of its own candidates.","core_discovery":"The paper's central claim is that smart word substitution can be learned and evaluated without any human annotations, using BARTScore as a stand-in for human judgment. The authors define the quality of a candidate substitution as the BARTScore of the sentence with that substitution, $M(\\tilde{X}_k) = \\sum_{n=1}^N \\log \\hat{p}_\\theta(\\tilde{x}_n \\mid \\tilde{x}_{