Pith. sign in

REVIEW 4 major objections 6 minor 19 references

BIBERT-Pipe shows that a standard biomedical encoder, wrapped with two-stage retrieval-ranking, learnable boundary cues, and dataset augmentation, can reach third place in multilingual nested biomedical entity linking.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A two-stage entity linking pipeline with boundary-cue tokens and data augmentation achieves third place on the BioNNE 2025 multilingual nested entity linking task.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A legitimate shared-task system paper with a real third-place result, but the boundary-cue ablation runs on the test set, so the evidence for one of the three central modifications is weaker than the abstract suggests. the 4 major comments →

arxiv 2509.09725 v1 pith:FRWKUZF6 submitted 2025-09-10 cs.CL

BIBERT-Pipe on Biomedical Nested Named Entity Linking at BioASQ 2025

Chunyu Li , Xindi Zheng , Siqi Liu This is my paper
classification cs.CL
keywords biomedical entity linkingnested named entitiesmultilingual entity linkingdense retrievalcontrastive learningboundary cuesdataset augmentationBioNNE shared task
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that competitive multilingual, nested biomedical entity linking does not require a new architecture or a generative model. The authors build BIBERT-Pipe as a two-stage retrieval-ranking system that reuses the same pretrained biomedical encoder for both stages, marks each mention with learnable start and end tokens ([Ms]/[Me]), and expands the ranking training data with existing English and Russian corpora. On the BioNNE 2025 shared task, this system ranks third in the bilingual (English/Russian) track, with best top-1 accuracy of 0.6497 on Russian, 0.6370 on English, and 0.6342 on the bilingual track. The paper argues that the three modifications carry the improvement: boundary cues alone add 6.6 percentage points on Russian in the ablation, contrastive per-candidate ranking beats listwise ranking, and data augmentation consistently helps. A reader should care because nested and non-English mentions are common in real biomedical text and existing English-only, flat-mention systems handle them poorly.

Core claim

On its own terms, the paper's central claim is that the hard parts of nested and multilingual entity linking can be handled by task-aligned modifications around an unchanged encoder, rather than by a purpose-built model. Retrieval first narrows the knowledge base to ten candidate concepts using cosine similarity from a pretrained biomedical encoder; the ranking stage then re-scores each candidate independently with a contrastively fine-tuned encoder. Two details carry the argument: wrapping the mention span in learned [Ms]/[Me] tokens tells the encoder exactly where the entity starts and ends, which the ablation shows matters most for Russian, and re-ranking candidates as independent binary

What carries the argument

The load-bearing mechanism is the two-stage candidate pipeline. In retrieval, a pretrained biomedical encoder scores a boundary-cue-wrapped mention against precomputed concept embeddings and returns the top-10 candidates; in ranking, those candidates are re-scored individually by a contrastively trained binary classifier, so each candidate receives an independent probability. The named cues are [Ms]/[Me], two learnable tokens inserted at the mention start and end to give the encoder an explicit, language-agnostic span, which matters for nested mentions. The third component, dataset augmentation, reformats MedMentions and MCN to the three target entity types (DISO, CHEM, ANATOMY) to enlarge t

Load-bearing premise

The system can only be as accurate as its retrieval stage: if the correct concept is not among the top-10 candidates returned by the retriever, the ranker has no way to recover it.

What would settle it

Re-run the Russian-track ranker on the development set with the [Ms]/[Me] boundary tokens removed while keeping candidates, training data, and hyperparameters fixed. The paper reports Acc@1 falls from 0.6497 to 0.6095 in this ablation; an independent run that does not show a comparable drop would falsify the boundary-cue mechanism. A second check is to force the gold concept into every candidate list and remove the boundary cues: if final accuracy does not fall below the full-system value, the claimed components are not load-bearing.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On the official BioNNE 2025 leaderboard, the system's third-place finish shows a stock biomedical encoder with these three modifications is competitive in a nested, bilingual setting.
  • The boundary-cue ablation quantifies the contribution: [Ms]/[Me] tokens add about 1.2 percentage points on English and the bilingual track and 6.6 points on Russian, suggesting the cues matter most where morphology is rich.
  • Contrastive ranking outperforms listwise ranking by a wide margin at roughly k times the compute, documenting a direct accuracy-versus-cost trade-off in the ranking stage.
  • Adding MedMentions and MCN to the ranking training set improves final accuracy on all tracks, with the largest gain on English, showing that existing corpora transfer after reformatting.
  • Because the same recipe is tested on BioLinkBERT, SapBERT, and BiomedBERT, the paper claims the pipeline is encoder-agnostic and can be lifted onto other biomedical encoders.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • I infer that the retrieval stage is the real bottleneck: with top-10 recall around 0.75-0.82 on the development sets, improving candidate generation could yield larger gains than further ranker tuning.
  • I infer that the boundary-cue gain on Russian is a signal for other morphologically rich or low-resource languages, where inflected surface forms hide the mention span from the encoder.
  • I infer that separating candidate generation from ranking means the two stages can be upgraded independently, so a stronger retriever could be swapped in without retraining the ranker.
  • I infer that the reported training setup (a single consumer GPU) makes this a practical strong baseline for future multilingual nested entity linking tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents BIBERT-Pipe, a lightweight two-stage pipeline for the BioNNE 2025 multilingual biomedical nested entity linking task (English and Russian). The method keeps the base encoder unchanged and makes three modifications: (i) a retrieval stage using off-the-shelf biomedical encoders followed by a cross-encoder ranking stage; (ii) explicit boundary tokens [Ms]/[Me] around each mention; and (iii) augmentation of the ranking training data with MedMentions and MCN. On the official leaderboard the system ranks third in the bilingual track with Acc@1 = 0.637. Dev-set results are used to compare retrievers and ranker architectures, and an ablation study examines the effect of boundary cues.

Significance. If the reported results are reliable, the paper shows that a simple, encoder-agnostic pipeline can be competitive in a multilingual nested EL shared task. The official third-place leaderboard result is externally validated, and the authors release code. The retrieval/ranking decomposition and the use of boundary tokens are sensible and potentially reusable. However, the load-bearing evidence for the boundary-cue contribution is weakened by the fact that the ablation is performed on the held-out evaluation set, and the paper does not report variance or multiple-seed results. The small English and bilingual gains (1.24% and 1.20%) could be within run-to-run noise, so the central claim of 'effectiveness of boundary cues' needs a more rigorous evaluation protocol before it can be fully credited.

major comments (4)
  1. [Section 5 and Section 6.1, Table 5] The boundary-cue ablation is run on the evaluation set, which Section 5 states was 'kept blind for final ranking.' This is a post-hoc test-set comparison, not a blind evaluation. The claimed gains (RU +6.60%, EN +1.24%, BI +1.20%) are presented without confidence intervals, multiple seeds, or a corresponding dev-set ablation. Since boundary cues are one of the three central contributions, the paper should either provide a pre-registered dev-set ablation with variance estimates or clearly frame the test-set numbers as exploratory and not as confirmatory evidence.
  2. [Section 5.3, Table 4] The final submissions include multiple variants with different epoch counts and different training sets (e.g., epoch=1 vs epoch=5; with/without dev in training). No pre-specified selection rule is given. This suggests that evaluation-set feedback may have influenced both model selection and the reported final accuracy. The authors should clarify the selection protocol: which decisions were made on dev, which on eval, and whether the reported Acc@1 values are the best-of-many eval-set trials. Without this, the final leaderboard number, while official, is difficult to interpret as an unbiased estimate of system performance.
  3. [Section 4.2 and Section 6.1] The paper claims boundary cues provide robustness to nested and overlapping mentions, but no experiment isolates nested mentions (e.g., by nesting depth, overlap type, or mention length). The aggregate Acc@1 could improve for reasons unrelated to nesting, such as better span boundary detection in general. The authors should provide a nested-specific breakdown on the dev set to support the claimed connection between [Ms]/[Me] tokens and nested-entity handling.
  4. [Section 5.2, Tables 2 and 3] The ranking experiments are reported as single runs without error bars or multiple seeds. Some differences are small (e.g., Table 3: 0.6083 vs 0.6319; Table 5: 0.0078 and 0.0075 gains) and may not be statistically reliable. I recommend reporting mean and standard deviation over at least 3-5 seeds for the key comparisons, especially the boundary-cue ablation and the data-augmentation comparison.
minor comments (6)
  1. [Abstract] Typo: 'Code are publicly available' should be 'Code is publicly available'.
  2. [Section 4.2] The sentence 'It is efficient to train the rank model with the retrieval model' is unclear. It probably means the rank model shares the encoder with the retrieval model, but this should be stated precisely.
  3. [Section 5] The paper says 'We set the retrieval numbers to k=10' but Table 2 includes experiments with k=5. Please clarify the default value and when k=5 is used.
  4. [Table 5] The Gain column mixes absolute differences and relative percentages. For example, 0.0402 is an absolute gain of 4.02 percentage points, while 6.60% is the relative improvement. Please label which is which to avoid ambiguity.
  5. [Tables 2 and 3] The phrase 'post training epoch' is nonstandard; use 'number of training epochs' or 'epochs.' Also, the 'Details' column in Table 3 is incomplete for some rows (e.g., no 'dev' marker for the last two rows), which makes the exact training setup ambiguous.
  6. [Section 5.3 and Table 4] It would be helpful to report the retrieval Acc@10 on the evaluation set alongside the final ranker Acc@1, since retrieval ceiling is a key constraint and the dev/test statistics may differ.

Circularity Check

0 steps flagged

No significant circularity: leaderboard and held-out dev results provide external evidence; the eval-set ablation is a validity concern, not a circular derivation.

full rationale

The paper's central claim—third place on the BioNNE-L 2025 leaderboard and the effectiveness of its three modifications—is supported by external, held-out evidence: the leaderboard Acc@1 values are produced by the organizers on the hidden evaluation set, and the retrieval and ranking comparisons in Tables 1–3 are measured on development data not used for training. The two-stage pipeline does not define its outputs in terms of its inputs; retrieval and ranking are distinct functions, and Acc@1/Acc@k are standard external metrics. No parameter is fitted to the target leaderboard quantity and then reported as a prediction. There are no load-bearing self-citations: the cited work by Sakhovskiy et al. is the task overview/organizers, not the present authors, and no uniqueness theorem or ansatz is imported from the authors' prior work. The one notable methodological issue is that Section 6.1's boundary-cue ablation is run on the evaluation set despite Section 5 saying it is 'kept blind for final ranking'; this is a validity concern (test-set reuse, no confidence intervals) that could overstate confidence in that ablation, but it is not a circular derivation because the ablation compares two model variants directly rather than folding the target result back into the model's construction.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 1 invented entities

The central claim (third place on BioNNE 2025) rests on standard pretrained encoders, a retrieval candidate ceiling, and dataset compatibility assumptions. The free parameters are modest (k, learning rate, epochs) and are selected via development data. The only introduced entities are the two boundary tokens, which are internal model components rather than independently testable scientific entities.

free parameters (4)
  • Number of retrieval candidates k = 5 or 10
    Set to 10 for ranking; some LTR experiments use 5. Chosen by the authors, not derived.
  • Learning rate = 7e-6 or 1e-5
    Selected from two values based on development performance (Section 5.2).
  • Training epochs = 1, 2, or 5
    Varies by track and experiment; chosen on dev (Tables 3, 4).
  • [Ms]/[Me] token embeddings = Not reported
    Newly added learnable embeddings for boundary tokens, fine-tuned during ranking (Section 4.2).
axioms (3)
  • domain assumption Pre-trained biomedical encoders (SapBERT, BioLinkBERT, BiomedBERT) provide useful representations for entity linking without further pretraining.
    The retrieval and ranking stages rely on these encoders as feature extractors; no analysis is provided of their coverage or bias.
  • domain assumption The NEREL-BIO, MedMentions, and MCN datasets can be reformatted to a common schema with consistent UMLS concept IDs and entity types.
    Section 4.3 describes only that they keep DISO, CHEM, ANATOMY types; no check of concept ID overlap or annotation compatibility is reported.
  • domain assumption The official dev set is representative of the hidden test set for model selection.
    All hyperparameter choices are made on the dev set (Section 5), assumed to transfer to the evaluation set.
invented entities (1)
  • [Ms] and [Me] boundary tokens no independent evidence
    purpose: Explicitly mark the start and end of a mention span in the input sequence for both retrieval and ranking.
    The tokens are new vocabulary items added to the tokenizer and their embeddings are learned during fine-tuning. Their benefit is measured only within this system's ablation; there is no external falsifiable prediction attached to them.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of BIBERT-Pipe on Biomedical Nested Named Entity Linking at BioASQ 2025." pith.science (2026). https://pith.science/paper/FRWKUZF6

@misc{pith2026250909725,
  author       = {Pith},
  title        = {Pith review of: BIBERT-Pipe on Biomedical Nested Named Entity Linking at BioASQ 2025},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FRWKUZF6}},
  note         = {Machine review of arXiv:2509.09725}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Entity linking (EL) for biomedical text is typically benchmarked on English-only corpora with flat mentions, leaving the more realistic scenario of nested and multilingual mentions largely unexplored. We present our system for the BioNNE 2025 Multilingual Biomedical Nested Named Entity Linking shared task (English & Russian), closing this gap with a lightweight pipeline that keeps the original EL model intact and modifies only three task-aligned components: Two-stage retrieval-ranking. We leverage the same base encoder model in both stages: the retrieval stage uses the original pre-trained model, while the ranking stage applies domain-specific fine-tuning. Boundary cues. In the ranking stage, we wrap each mention with learnable [Ms] / [Me] tags, providing the encoder with an explicit, language-agnostic span before robustness to overlap and nesting. Dataset augmentation. We also automatically expand the ranking training corpus with three complementary data sources, enhancing coverage without extra manual annotation. On the BioNNE 2025 leaderboard, our two stage system, bilingual bert (BIBERT-Pipe), ranks third in the multilingual track, demonstrating the effectiveness and competitiveness of these minimal yet principled modifications. Code are publicly available at https://github.com/Kaggle-Competitions-Code/BioNNE-L.

Figures

Figures reproduced from arXiv: 2509.09725 by Chunyu Li, Siqi Liu, Xindi Zheng.

Figure 1
Figure 1. Figure 1: Example of nested named entities in NEREL-BIO. The English phrase “isolated bronchus resection for central cancer” is annotated with overlapping spans: the outer span (magenta) is a diagnostic procedure (medproc); inside it, the token bronchus is an anatomical structure (yellow, anatomy), while resection is again a procedure (magenta, medproc). A separate right-hand branch shows the phrase central cancer, … view at source ↗
Figure 2
Figure 2. Figure 2: Pipeline of our rank stage. (a) Listwise – the 𝑘 candidates returned by the retrieval stage are concatenated after the [SEP] token ([ST0]...[ST𝑘−1] ), so a single forward pass of the encoder produces 𝑘 logits, one for each candidate. (b) Contrastive Learning – each candidate is paired with the mention context in an independent input sequence; the encoder is applied 𝑘 times and outputs a binary logit for ev… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages

  1. [1]

    French, B

    E. French, B. T. McInnes, An overview of biomedical entity linking throughout the years, Journal of biomedical informatics 137 (2023) 104252

    work page 2023

  2. [2]

    Garda, U

    S. Garda, U. Leser, Belhd: Improving biomedical entity linking with homonym disambiguation, Bioinformatics 39 (2023) btad698. doi:10.1093/bioinformatics/btad698

    work page doi:10.1093/bioinformatics/btad698 2023

  3. [3]

    Z. A. Guven, A. Lamurias, Multilingual bi-encoder models for biomedical entity linking, Expert Systems 40 (2023) e13388

    work page 2023

  4. [4]

    F. Liu, I. Vulić, A. Korhonen, N. Collier, Learning domain-specialised representations for cross- lingual biomedical entity linking, arXiv preprint arXiv:2105.14398 (2021)

    Pith/arXiv arXiv 2021

  5. [5]

    Yasunaga, J

    M. Yasunaga, J. Leskovec, P. Liang, Linkbert: Pretraining language models with document links, in: Association for Computational Linguistics (ACL), 2022

    work page 2022

  6. [6]

    F. Liu, E. Shareghi, Z. Meng, M. Basaldella, N. Collier, Self-alignment pretraining for biomedical entity representations, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 4228–

    work page 2021

  7. [7]

    Sakhovskiy, N

    A. Sakhovskiy, N. Loukachevitch, E. Tutubalina, Overview of the BioASQ BioNNE-L Task on Biomedical Nested Entity Linking in CLEF 2025, in: G. Faggioli, N. Ferro, P. Rosso, D. Spina (Eds.), CLEF 2025 Working Notes, 2025

    work page 2025

  8. [8]

    Nentidis, G

    A. Nentidis, G. Katsimpras, A. Krithara, M. Krallinger, M. Rodríguez-Ortega, E. Rodriguez-López, N. Loukachevitch, A. Sakhovskiy, E. Tutubalina, D. Dimitriadis, G. Tsoumakas, G. Giannakoulas, A. Bekiaridou, A. Samaras, G. M. Di Nunzio, N. Ferro, S. Marchesin, M. Martinelli, G. Silvello, G. Paliouras, Overview of BioASQ 2025: The thirteenth BioASQ challeng...

    work page 2025

  9. [9]

    Loukachevitch, S

    N. Loukachevitch, S. Manandhar, E. Baral, I. Rozhkov, P. Braslavski, V. Ivanov, T. Batura, E. Tu- tubalina, NEREL-BIO: A Dataset of Biomedical Abstracts Annotated with Nested Named Entities, Bioinformatics (2023). doi:10.1093/bioinformatics/btad161, btad161

    work page doi:10.1093/bioinformatics/btad161 2023

  10. [10]

    T. Zhu, Y. Qin, Q. Chen, X. Mu, C. Yu, Y. Xiang, Controllable contrastive generation for multilingual biomedical entity linking, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 5742–5753

    work page 2023

  11. [11]

    Sakhovskiy, N

    A. Sakhovskiy, N. Semenova, A. Kadurin, E. Tutubalina, Graph-enriched biomedical entity repre- sentation transformer, in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2023, pp. 109–120

    work page 2023

  12. [12]

    Sakhovskiy, N

    A. Sakhovskiy, N. Semenova, A. Kadurin, E. Tutubalina, Biomedical entity representation with graph-augmented multi-objective transformer, in: Findings of the Association for Computational Linguistics: NAACL 2024, 2024, pp. 4626–4643

    work page 2024

  13. [13]

    Jahan, M

    I. Jahan, M. T. R. Laskar, C. Peng, J. Huang, Evaluation of ChatGPT on biomedical tasks: A zero-shot comparison with fine-tuned generative transformers, in: D. Demner-Fushman, S. Ananiadou, K. Cohen (Eds.), Proceedings of the 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, Association for Computational Linguistics, Toronto...

    work page doi:10.18653/v1/2023.bionlp-1 2023

  14. [14]

    Q. Chen, Y. Hu, X. Peng, Q. Xie, Q. Jin, A. Gilson, M. B. Singer, X. Ai, P.-T. Lai, Z. Wang, V. K. Keloth, K. Raja, J. Huang, H. He, F. Lin, J. Du, R. Zhang, W. J. Zheng, R. A. Adelman, Z. Lu, H. Xu, Bench- marking large language models for biomedical natural language processing applications and rec- ommendations, Nature Communications 16 (2025) 3280. doi...

    doi:10.1038/s41467-025-56989-2 2025

  15. [15]

    Y. Ding, Q. Zeng, T. Weninger, ChatEL: Entity linking with chatbots, in: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Torino, Italia, 2024, pp. 3086–3097. URL: https://aclanthology.org/2024.lrec-main.275/

    work page 2024

  16. [16]

    Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, H. Poon, Domain-specific language model pretraining for biomedical natural language processing, 2020. arXiv:arXiv:2007.15779

    Pith/arXiv arXiv 2020

  17. [17]

    Mohan, D

    S. Mohan, D. Li, Medmentions: A large biomedical corpus annotated with umls concepts, 2019. URL: https://arxiv.org/abs/1902.09476.arXiv:1902.09476

    Pith/arXiv arXiv 2019

  18. [18]

    Loukachevitch, A

    N. Loukachevitch, A. Sakhovskiy, E. Tutubalina, Biomedical concept normalization over nested entities with partial UMLS terminology in Russian, in: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Torino, Italia, 2024, pp. 2383–2389. URL: https://aclan...

    work page 2024

  19. [4238]

    URL: https://aclanthology.org/2021.naacl-main.334/

    work page 2021

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.