{"id":"a7c35380-8ff7-494f-a140-3059582a8da4","arxiv_id":"2505.19722","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A closed-source LLM's re-ranking labels are distilled into a locally deployable open-source LLM, producing small but consistent Acc@1 gains in low-resource biomedical entity linking on two datasets.","lead":"RPDR is a three-step framework for biomedical entity linking that uses a closed-source large language model to generate training labels from unannotated text, then fine-tunes a small open-source model to re-rank candidate entities locally. The paper reports small accuracy gains on a Chinese eye-hospital dataset and an English patient dataset, while arguing for lower cost and more stable deployment than calling commercial APIs directly.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported Acc@1 gains are within a few test instances; without paired significance tests the central improvement claim is not established.","rationale":"The reader's weakest_assumption was that an unmeasured retrieval recall bounds the re-ranking gains. That concern is partly defused by Table 2 itself: final Acc@5 values (0.933 on Aier, 0.871 on Ask A Patient) imply retrieval recall at k=6 is at least that high, since re-ranking only permutes retrieved candidates. Thus the retrieval ceiling is high and the re-ranker has room to operate; recall is not the most load-bearing issue. The more fundamental weakness is statistical: the reported differences are small relative to the sample sizes, especially on Aier where the headline comparison to Raw LLM is a single instance. The reader did note 'no significance testing is provided' in the rationale, so there is partial agreement, but the identified weakest_assumption differs. A paired significance test is the decisive, low-cost check that would settle whether the observed improvements are real or within sampling noise. Because the reader's verdict is already CONDITIONAL and the required analysis would be a reasonable condition, the verdict should remain unchanged.","tokens_in":9238,"tokens_out":14370,"duration_ms":152620,"concrete_test":"Obtain the per-instance predictions behind Table 2 (or re-run the pipeline) and apply McNemar's exact test to the paired binary outcomes for (i) Our Method vs Raw LLM and (ii) Our Method vs the best baseline on both datasets. Also compute exact 95% confidence intervals for the Acc@1 differences. If on Aier the p-value exceeds 0.05 (expected, since the gap is 1-2 instances), the Aier result should be reported as inconclusive; if on Ask A Patient the Raw LLM comparison is also non-significant, the paper's claim of surpassing the teacher is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that RPDR improves Acc@1 rests on Table 2. On Aier (n=104), Our Method reaches 0.740 versus Raw LLM 0.731 and the best baseline BioCEI/CMTN 0.721: these are differences of 1 and 2 correct instances. The standard error of a proportion near 0.73 with n=104 is about 0.044, so the headline gains are far below one standard error. On Ask A Patient (n=867), Our Method 0.734 versus Raw LLM 0.724 is a 9-instance difference, and versus SapBERT 0.698 is a 31-instance difference; the latter is suggestive, but the paper reports no paired test, no confidence intervals, and no error bars. Because the paper's contribution is empirical and the 'student surpasses teacher' distillation claim is based on the Raw LLM comparison, the possibility that these differences are sampling noise directly undermines the central claim. The missing significance testing is not a cosmetic omission; it is the difference between 'distillation works' and 'no measurable effect on this test set.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RPDR, a framework for low-resource biomedical entity linking. It first retrieves k candidate entities with a bi-encoder (SapBERT or a fine-tuned Chinese encoder), then prompts a closed-source LLM (GPT-3.5 Turbo or DeepSeek V3) to re-rank the candidates for unlabeled mentions, and finally fine-tunes an open-source LLM (BenTsao or Llama2-7B) with LoRA on the generated re-rankings. The resulting local model is evaluated on the private Aier Chinese dataset and the public Ask A Patient English dataset, reporting Acc@1/Acc@5 improvements over baselines and a cost comparison in Table 6. The core claim is that the distilled open-source model slightly outperforms the raw closed-source LLM (0.740 vs 0.731 on Aier; 0.734 vs 0.724 on Ask A Patient) while avoiding per-query API costs.","tokens_in":9390,"tokens_out":6646,"duration_ms":65794,"significance":"If the central empirical claim were established, RPDR would be a useful practical recipe: it replaces repeated closed-source API calls with a locally deployable model and shows cross-lingual applicability. The paper is clearly written, the prompt design is sensible, and the evaluation against held-out human labels avoids the circularity that would arise from evaluating on teacher-generated labels. The cost numbers in Table 6 are a concrete practical contribution. However, the headline gains are small relative to test-set sizes, and the comparison against baselines on Ask A Patient is not fully controlled. The framework itself is not fundamentally new—it combines candidate retrieval, LLM-generated training data, and LoRA distillation—but the combination and the biomedical application are useful. The lack of released code or checkpoints limits reproducibility.","major_comments":[{"comment":"The central claim that RPDR improves Acc@1 over the raw LLM and baselines is not supported by statistical evidence. On Aier (n=104), the reported 0.740 vs 0.731 vs 0.721 differences correspond to one or two changed predictions; the standard error of a proportion near 0.73 is about 0.044. On Ask A Patient (n=867), the 0.734 vs 0.724 gap is 9 instances. The paper reports no confidence intervals, no paired significance test (e.g., McNemar or bootstrap), and no repeated runs. Because this improvement claim is the main contribution, the authors should either provide significance evidence or substantially weaken the 'superiority' language.","section":"Table 2, §4.5"},{"comment":"The comparison between human-labeled and GPT-3.5-generated training data is confounded by dataset size and composition: the Aier training set has only 309 human-labeled instances (Table 1), while the generated-data condition uses 1,000 mention-candidate pairs. The 0.740 vs 0.712 gap may reflect the larger training set or the selection of unlabeled mentions rather than the teacher's labels. The authors should equalize the number of training instances or provide a learning-curve comparison before claiming that generated data is superior to human labels.","section":"Table 4, §4.6"},{"comment":"On Ask A Patient, the supervised baselines (BioSyn, BioCEI) are evaluated with zero-shot checkpoints trained on NCBI-Disease, while the proposed method is fine-tuned on 2,000 generated samples. This is not an apples-to-apples comparison for the claim that RPDR is superior in low-resource settings. The relevant comparison against Raw LLM is more controlled, but that gap is small and statistically unquantified. The authors should either fine-tune baselines on comparable in-domain data or explicitly frame the comparison as zero-shot adaptation.","section":"Section 4.3, Table 2"},{"comment":"The paper never reports recall@k of the candidate retrieval stage, even though the re-ranking step can only succeed when the golden entity is among the k retrieved candidates. Without recall@k, the reader cannot tell whether the observed Acc@1 gains are bounded by retrieval failures, and the framework's contribution to re-ranking is hard to isolate. Reporting recall@k for both datasets would make the headroom explicit and strengthen the interpretation of Table 2.","section":"Sections 3.1 and 4.4"}],"minor_comments":[{"comment":"There is a typo: 'representationss' should be 'representations'.","section":"Section 3.1"},{"comment":"The section headings contain a stray space: 'T raining' should be 'Training'.","section":"Sections 3.2 and 3.3"},{"comment":"The caption should identify which LLM is 'Raw LLM' for each dataset, since Section 4.4 indicates GPT-3.5 Turbo for Aier and DeepSeek V3 for Ask A Patient.","section":"Table 2"},{"comment":"The claim that the authors are 'the first to apply knowledge distillation in biomedical entity linking' is too strong without a more thorough related-work search; several earlier works use distillation for biomedical NLP, so this novelty claim should be softened or supported.","section":"Section 1"},{"comment":"DeepSeek V3 is described as a 'closed-source' LLM, but the model weights are openly released; this characterization should be corrected or qualified.","section":"Sections 1 and 4.4"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is within scope for an applied NLP venue, but the empirical core needs strengthening. The main risk is that the headline improvements are within sampling noise; a rejection would be premature if the authors can supply paired significance tests and a more controlled baseline comparison. I would not recommend acceptance on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my bottom line: the paper is a clean engineering combination—retrieve candidates, prompt GPT-3.5/DeepSeek to re-rank them, and fine-tune a small open-source LLM with LoRA on those rankings—and the cost comparison in Table 6 makes the deployment motivation concrete. The ablation in Table 4, where GPT-generated training data beats human-annotated data on Aier, is the most interesting result in the paper.\n\nThe problem is the statistical basis of the central claim. On Aier, the distilled student's 0.740 versus the raw teacher's 0.731 is a difference of one instance out of 104. On Ask A Patient, 0.734 versus 0.724 is nine instances out of 867. There are no confidence intervals, no paired tests, no repeated runs. The stress-test note is correct: on these numbers, 'distillation works' and 'no measurable effect' are not distinguishable. The comparison against SapBERT on Ask A Patient is more favorable, but it is also apples-to-oranges: SapBERT is applied zero-shot, while the student is fine-tuned on teacher-generated data for that same dataset. And the paper never reports retrieval recall@k, so we cannot see how much headroom the re-ranker actually has.\n\nTwo minor issues. The claim to be 'the first to apply knowledge distillation in biomedical entity linking' is too strong; there is prior work on distillation for biomedical concept normalization, so that sentence should be softened or supported with citations. And with Aier being private, the only reproducible experiment is Ask A Patient, where the comparison is against a weak baseline.\n\nThe idea itself is not discredited. The pipeline is plausible and the authors report their hyperparameters. But the paper as written does not establish the headline improvement. The fixes are straightforward: test on a public Chinese biomedical EL benchmark with multiple seeds and paired significance tests, and on Ask A Patient include a control model trained on the same generated data without teacher re-ranking, to isolate the distillation contribution.\n\nMy recommendation: send it to peer review, but the referee should make these experiments a condition of acceptance. This is a serious paper, not a desk-reject, but its main empirical claim needs to be demonstrated, not asserted.","headline":"A sensible distillation pipeline whose headline accuracy gains are within sampling noise; the framework is worth reviewing, but the central claim needs significance testing and fair baselines.","tokens_in":9988,"tokens_out":2798,"would_cite":false,"duration_ms":28400,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small open-source model, trained on a closed-source LLM's rankings, can beat that teacher's Acc@1 on biomedical entity linking.","keywords":["biomedical entity linking","knowledge distillation","candidate re-ranking","LLM prompting","low-resource learning","LoRA fine-tuning","closed-source LLM","medical concept normalization"],"falsifier":"Compute recall@6 of the retriever on the Aier and Ask A Patient test sets; if the golden entity is missing from the candidate list for a substantial fraction of mentions, the reported Acc@1 improvements are bounded by that fraction, and the distillation claim could be tested by comparing re-ranker behavior only on mentions where recall succeeds.","tokens_in":8988,"feed_emoji":"🧬","tokens_out":9277,"duration_ms":83368,"temperature":0.7,"pith_summary":"Biomedical entity linking maps a nonstandard mention in clinical text to its standard knowledge-base entry. The paper proposes RPDR, a three-step framework in which a retriever proposes six candidate entities, a closed-source LLM is prompted to rank them, and those rankings are used to fine-tune a small open-source LLM that performs the same re-ranking locally. The distilled model improves Acc@1 over the raw closed-source teacher on both tested datasets (0.740 vs 0.731 on Aier, 0.734 vs 0.724 on Ask A Patient), using only a small amount of generated training data. The appeal is practical: a locally deployed open-source model avoids API costs, rate limits, and the risk that a vendor discontinues or changes the service, while still getting much of the teacher's cross-language transfer ability.","feed_headline":"Distilled local LLM beats closed-source teacher on entity linking","feed_subtitle":"One prompted labeling pass lifts Acc@1 by 0.019 on Aier and 0.036 on Ask A Patient.","key_machinery":"The engine is a distillation loop. A fixed prompt template asks a closed-source LLM to understand the mention and output the six retrieved candidates in likelihood order; the output, generated at temperature 0, becomes training data. A small open-source LLM is fine-tuned with LoRA, a low-rank adaptation that trains only a small parameter increment on top of the pretrained weights, to reproduce that ordering. The retriever is the supporting equipment: a bi-encoder initialized from a BERT-based model (SapBERT on English, a Chinese RoBERTa on the Aier set) whose top-six list bounds what re-ranking can fix.","core_discovery":"The paper claims that RPDR, which reformulates entity linking from two steps into three (retrieval, prompting-based training-data generation, and distillation for re-ranking), lets a locally deployable open-source LLM reproduce and slightly exceed a closed-source LLM's re-ranking quality. On the Chinese Aier dataset, the distilled model reaches 0.740 Acc@1 versus 0.731 for the raw teacher, and on the English Ask A Patient dataset it reaches 0.734 versus 0.724. The paper reads these results as evidence that the teacher's knowledge has been transferred into a model that can run without the teacher, and that the transfer generalizes across a private Chinese clinical corpus and a public English consumer-health corpus.","pith_inferences":["The reported gains sit on top of retrieval, so a natural next experiment is to swap in a stronger retriever and test whether the distilled re-ranker's advantage grows with retrieval recall.","The same teacher-label-plus-LoRA-distillation recipe should transfer to other ranking-heavy tasks, such as passage retrieval or product search, where a strong closed model can rank but cannot be called at scale.","The peak-and-decline curve in Fig. 4 suggests teacher-generated labels contain noise, so a confidence filter or agreement-based selection on teacher outputs could raise the usable data ceiling.","Because the student copies the teacher's rankings rather than verified answers, systematic teacher errors will be inherited; combining the local model with cheap lexical or rule signals could catch cases where the teacher is biased."],"forward_implications":["A provider can run re-ranking entirely on local hardware; the paper's cost table puts local Llama-2 inference at $0.038 versus $0.294 for GPT-3.5 Turbo on the Ask A Patient evaluation set.","The pipeline stops depending on a vendor's uptime or version changes at inference time, because the closed-source model is called once to create labels and never per query.","The framework transfers across English and Chinese and across private and public data, so unannotated mentions plus teacher-generated labels can substitute for extensive human annotation.","Teacher-generated data can outperform human-labeled data at the same size; the Aier ablation reports 0.740 Acc@1 with generated data versus 0.712 with human labels."],"supporting_citations":[{"why":"It supplies the bi-encoder candidate retriever whose top-six lists the re-ranker reorders, and its training setup is used for the Aier dataset.","marker":"[32]"},{"why":"It provides SapBERT, the pretrained biomedical encoder used without fine-tuning to retrieve candidates on the English Ask A Patient dataset.","marker":"[15]"},{"why":"It provides the Llama2-7B base that is LoRA fine-tuned as the local re-ranking student on Ask A Patient.","marker":"[24]"},{"why":"It provides BenTsao, the Chinese medical LLM base that is LoRA fine-tuned as the local re-ranking student on Aier.","marker":"[25]"},{"why":"It supplies the LoRA method used to fine-tune the open-source student models with a small trainable parameter increment.","marker":"[9]"},{"why":"It supplies the general knowledge-distillation principle that a student learns by imitating a teacher's output.","marker":"[8]"},{"why":"It provides the Ask A Patient dataset used for the English evaluation.","marker":"[14]"}],"fun_headline_variants":["Local distilled LLM edges out closed-source teacher on entity linking","Open-source model beats closed-source teacher after one labeling pass","Distillation makes local LLM rival and beat closed-source teacher","Cheap local model surpasses closed-source teacher on biomedical linking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The re-ranking step can only help when the retriever has already placed the golden entity among the six candidates, and the paper never reports that retrieval recall.","fun_headline_variants_meta":{"raw":{"variants":["Local distilled LLM edges out closed-source teacher on entity linking","Open-source model beats closed-source teacher after one labeling pass","Distillation makes local LLM rival and beat closed-source teacher","Cheap local model surpasses closed-source teacher on biomedical linking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001111,"raw_usage":{"total_tokens":4623,"prompt_tokens":937,"completion_tokens":3686,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":3616}},"tokens_in":553,"tokens_out":3686,"duration_ms":93999,"temperature":1.0,"reasoning_tokens":3616,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:07:31.282469+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute recall@6 of the retriever on the Aier and Ask A Patient test sets; if the golden entity is missing from the candidate list for a substantial fraction of mentions, the reported Acc@1 improvements are bounded by that fraction, and the distillation claim could be tested by comparing re-ranker behavior only on mentions where recall succeeds.","supporting_citations":[{"cited_title":"13869–13877 (2023)","cited_arxiv_id":null,"evidence_quote":"It supplies the bi-encoder candidate retriever whose top-six lists the re-ranker reorders, and its training setup is used for the Aier dataset."},{"cited_title":"In: International Conference on Learning Representations (2021)","cited_arxiv_id":null,"evidence_quote":"It supplies the LoRA method used to fine-tune the open-source student models with a small trainable parameter increment."},{"cited_title":"In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (volume 1: long papers)","cited_arxiv_id":null,"evidence_quote":"It provides the Ask A Patient dataset used for the English evaluation."}],"review_version":1}