{"id":"3432eb50-6e73-48c5-afb3-60a8919e762a","arxiv_id":"2509.09725","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"A two-stage entity linking pipeline with boundary-cue tokens and data augmentation achieves third place on the BioNNE 2025 multilingual nested entity linking task.","lead":"This paper presents BIBERT-Pipe, a biomedical entity linking system for English and Russian that ranked third in the multilingual track of the BioNNE 2025 shared task. It shows that a lightweight combination of two-stage retrieval-ranking, boundary-cue tokens, and data augmentation can handle nested, cross-lingual mentions without changing the underlying encoder.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Boundary-cue ablation is run on the held-out evaluation set, so the claimed effectiveness of the modifications rests on post-hoc test-set evidence rather than a blind evaluation.","rationale":"The reader identified the retrieval-stage ceiling as the weakest assumption, which is a legitimate upper-bound observation but not a flaw in the paper's argument. The more load-bearing concern is that the only direct ablation for one of the paper's three claimed modifications is performed on the evaluation set, despite the paper explicitly saying that set is kept blind. This undermines the internal validity of the claim that the modifications are effective. The leaderboard result itself is credible and externally provided, but the paper's contribution is framed around 'minimal yet principled modifications,' and the evidence for boundary cues is post-hoc and statistically uncontrolled. A dev-set ablation would settle whether the reported gains are genuine or an artifact of test-set selection. Therefore, I recommend conditional acceptance pending that verification, rather than unconditional acceptance.","tokens_in":7820,"tokens_out":6596,"duration_ms":73948,"concrete_test":"Re-run the Table 5 ablation on the development set using 5-fold cross-validation with the same base model (SapBERT-XLMR-large), same learning rates (7e-6/1e-5), and same training epochs, toggling only the presence of [Ms]/[Me] tokens, and report mean ± std for EN, RU, and BI. If the EN/BI gains fall within noise or the RU gain drops below ~2pp, the boundary-cue effectiveness claim is not supported; if the gains persist across folds, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5 states that the evaluation set is 'kept blind for final ranking,' yet the only controlled ablation for boundary cues (Table 5) is performed on that evaluation set. The reported gains (RU +6.60pp, EN +1.24pp, BI +1.20pp) are presented without confidence intervals, multiple seeds, or any indication that they are stable across runs. Moreover, the final submissions in Table 4 include multiple variants with different epoch counts and training sets, suggesting test-set feedback may have influenced model selection. Since boundary cues are one of the three central modifications claimed to demonstrate 'the effectiveness and competitiveness of these minimal yet principled modifications,' the load-bearing evidence for this claim is an uncontrolled comparison on the same data used to report the final result. If the gains do not reproduce on a held-out development split, the paper's internal evidence for the contribution of boundary cues collapses, leaving only the leaderboard rank, which lacks contextualization (e.g., number of participants, score margins).","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents BIBERT-Pipe, a lightweight two-stage pipeline for the BioNNE 2025 multilingual biomedical nested entity linking task (English and Russian). The method keeps the base encoder unchanged and makes three modifications: (i) a retrieval stage using off-the-shelf biomedical encoders followed by a cross-encoder ranking stage; (ii) explicit boundary tokens [Ms]/[Me] around each mention; and (iii) augmentation of the ranking training data with MedMentions and MCN. On the official leaderboard the system ranks third in the bilingual track with Acc@1 = 0.637. Dev-set results are used to compare retrievers and ranker architectures, and an ablation study examines the effect of boundary cues.","tokens_in":8116,"tokens_out":3080,"duration_ms":34837,"significance":"If the reported results are reliable, the paper shows that a simple, encoder-agnostic pipeline can be competitive in a multilingual nested EL shared task. The official third-place leaderboard result is externally validated, and the authors release code. The retrieval/ranking decomposition and the use of boundary tokens are sensible and potentially reusable. However, the load-bearing evidence for the boundary-cue contribution is weakened by the fact that the ablation is performed on the held-out evaluation set, and the paper does not report variance or multiple-seed results. The small English and bilingual gains (1.24% and 1.20%) could be within run-to-run noise, so the central claim of 'effectiveness of boundary cues' needs a more rigorous evaluation protocol before it can be fully credited.","major_comments":[{"comment":"The boundary-cue ablation is run on the evaluation set, which Section 5 states was 'kept blind for final ranking.' This is a post-hoc test-set comparison, not a blind evaluation. The claimed gains (RU +6.60%, EN +1.24%, BI +1.20%) are presented without confidence intervals, multiple seeds, or a corresponding dev-set ablation. Since boundary cues are one of the three central contributions, the paper should either provide a pre-registered dev-set ablation with variance estimates or clearly frame the test-set numbers as exploratory and not as confirmatory evidence.","section":"Section 5 and Section 6.1, Table 5"},{"comment":"The final submissions include multiple variants with different epoch counts and different training sets (e.g., epoch=1 vs epoch=5; with/without dev in training). No pre-specified selection rule is given. This suggests that evaluation-set feedback may have influenced both model selection and the reported final accuracy. The authors should clarify the selection protocol: which decisions were made on dev, which on eval, and whether the reported Acc@1 values are the best-of-many eval-set trials. Without this, the final leaderboard number, while official, is difficult to interpret as an unbiased estimate of system performance.","section":"Section 5.3, Table 4"},{"comment":"The paper claims boundary cues provide robustness to nested and overlapping mentions, but no experiment isolates nested mentions (e.g., by nesting depth, overlap type, or mention length). The aggregate Acc@1 could improve for reasons unrelated to nesting, such as better span boundary detection in general. The authors should provide a nested-specific breakdown on the dev set to support the claimed connection between [Ms]/[Me] tokens and nested-entity handling.","section":"Section 4.2 and Section 6.1"},{"comment":"The ranking experiments are reported as single runs without error bars or multiple seeds. Some differences are small (e.g., Table 3: 0.6083 vs 0.6319; Table 5: 0.0078 and 0.0075 gains) and may not be statistically reliable. I recommend reporting mean and standard deviation over at least 3-5 seeds for the key comparisons, especially the boundary-cue ablation and the data-augmentation comparison.","section":"Section 5.2, Tables 2 and 3"}],"minor_comments":[{"comment":"Typo: 'Code are publicly available' should be 'Code is publicly available'.","section":"Abstract"},{"comment":"The sentence 'It is efficient to train the rank model with the retrieval model' is unclear. It probably means the rank model shares the encoder with the retrieval model, but this should be stated precisely.","section":"Section 4.2"},{"comment":"The paper says 'We set the retrieval numbers to k=10' but Table 2 includes experiments with k=5. Please clarify the default value and when k=5 is used.","section":"Section 5"},{"comment":"The Gain column mixes absolute differences and relative percentages. For example, 0.0402 is an absolute gain of 4.02 percentage points, while 6.60% is the relative improvement. Please label which is which to avoid ambiguity.","section":"Table 5"},{"comment":"The phrase 'post training epoch' is nonstandard; use 'number of training epochs' or 'epochs.' Also, the 'Details' column in Table 3 is incomplete for some rows (e.g., no 'dev' marker for the last two rows), which makes the exact training setup ambiguous.","section":"Tables 2 and 3"},{"comment":"It would be helpful to report the retrieval Acc@10 on the evaluation set alongside the final ranker Acc@1, since retrieval ceiling is a key constraint and the dev/test statistics may differ.","section":"Section 5.3 and Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central result (official third place) is externally grounded. However, the main methodological weakness is the use of the evaluation set for the boundary-cue ablation and the lack of a clear model-selection protocol across the multiple final submissions. These issues are fixable with additional dev-set experiments and variance reporting. I would also encourage the authors to clarify how many teams participated and the margins, so the third-place claim can be contextualized."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a legitimate shared-task system paper with a real third-place result, but the boundary-cue ablation is run on the test set, so the abstract overstates what that ablation proves.\n\nWhat's actually new here is the empirical combination: a two-stage retrieval-ranking pipeline that keeps the base encoder frozen, adds learnable [Ms]/[Me] boundary tokens, and augments training data with MedMentions and MCN. The paper is clearly written, the dev-set comparisons across retrievers and rankers are useful, and the public code is a plus. The official leaderboard rank gives an external validity check that many system papers lack.\n\nThe stress-test note is on target. Section 5 says the evaluation set is 'kept blind for final ranking,' but Table 5 reports the boundary-cue ablation on that same evaluation set. There are no error bars, no multiple seeds, and no note that this is post-hoc. The final-submission table also varies epoch counts and includes dev sets in training, which suggests some test-set feedback. That's a real methodological blemish.\n\nThe blemish is contained, though. The leaderboard result itself is blind and external. The ablation only supports one of three claimed modifications, and the main point—that a lightweight pipeline can be competitive on multilingual nested EL—doesn't depend on that ablation. The gains in Russian are large, so they deserve a proper dev-set check, but the failure to do that doesn't sink the paper.\n\nI'd score soundness around 5.5, in line with the reader. The paper deserves a serious referee. Ask for a dev-set ablation or a clear statement that the test-set numbers are exploratory, and it's acceptable as a working-notes contribution. The audience is people building biomedical EL systems or tracking shared-task results; I'd bring it to a reading group focused on reproducibility rather than a methods discussion. I wouldn't cite it in my own next paper, but I'd recommend it for review.","headline":"A legitimate shared-task system paper with a real third-place result, but the boundary-cue ablation runs on the test set, so the evidence for one of the three central modifications is weaker than the abstract suggests.","tokens_in":8515,"tokens_out":3701,"would_cite":false,"duration_ms":42537,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BIBERT-Pipe shows that a standard biomedical encoder, wrapped with two-stage retrieval-ranking, learnable boundary cues, and dataset augmentation, can reach third place in multilingual nested biomedical entity linking.","keywords":["biomedical entity linking","nested named entities","multilingual entity linking","dense retrieval","contrastive learning","boundary cues","dataset augmentation","BioNNE shared task"],"falsifier":"Re-run the Russian-track ranker on the development set with the [Ms]/[Me] boundary tokens removed while keeping candidates, training data, and hyperparameters fixed. The paper reports Acc@1 falls from 0.6497 to 0.6095 in this ablation; an independent run that does not show a comparable drop would falsify the boundary-cue mechanism. A second check is to force the gold concept into every candidate list and remove the boundary cues: if final accuracy does not fall below the full-system value, the claimed components are not load-bearing.","tokens_in":7789,"feed_emoji":"🧬","tokens_out":14107,"duration_ms":117121,"temperature":0.7,"pith_summary":"The paper's claim is that competitive multilingual, nested biomedical entity linking does not require a new architecture or a generative model. The authors build BIBERT-Pipe as a two-stage retrieval-ranking system that reuses the same pretrained biomedical encoder for both stages, marks each mention with learnable start and end tokens ([Ms]/[Me]), and expands the ranking training data with existing English and Russian corpora. On the BioNNE 2025 shared task, this system ranks third in the bilingual (English/Russian) track, with best top-1 accuracy of 0.6497 on Russian, 0.6370 on English, and 0.6342 on the bilingual track. The paper argues that the three modifications carry the improvement: boundary cues alone add 6.6 percentage points on Russian in the ablation, contrastive per-candidate ranking beats listwise ranking, and data augmentation consistently helps. A reader should care because nested and non-English mentions are common in real biomedical text and existing English-only, flat-mention systems handle them poorly.","feed_headline":"Minimal pipeline takes third in multilingual biomedical linking","feed_subtitle":"Keeps a standard encoder untouched, still third on nested English-Russian linking","key_machinery":"The load-bearing mechanism is the two-stage candidate pipeline. In retrieval, a pretrained biomedical encoder scores a boundary-cue-wrapped mention against precomputed concept embeddings and returns the top-10 candidates; in ranking, those candidates are re-scored individually by a contrastively trained binary classifier, so each candidate receives an independent probability. The named cues are [Ms]/[Me], two learnable tokens inserted at the mention start and end to give the encoder an explicit, language-agnostic span, which matters for nested mentions. The third component, dataset augmentation, reformats MedMentions and MCN to the three target entity types (DISO, CHEM, ANATOMY) to enlarge t","core_discovery":"On its own terms, the paper's central claim is that the hard parts of nested and multilingual entity linking can be handled by task-aligned modifications around an unchanged encoder, rather than by a purpose-built model. Retrieval first narrows the knowledge base to ten candidate concepts using cosine similarity from a pretrained biomedical encoder; the ranking stage then re-scores each candidate independently with a contrastively fine-tuned encoder. Two details carry the argument: wrapping the mention span in learned [Ms]/[Me] tokens tells the encoder exactly where the entity starts and ends, which the ablation shows matters most for Russian, and re-ranking candidates as independent binary","pith_inferences":["I infer that the retrieval stage is the real bottleneck: with top-10 recall around 0.75-0.82 on the development sets, improving candidate generation could yield larger gains than further ranker tuning.","I infer that the boundary-cue gain on Russian is a signal for other morphologically rich or low-resource languages, where inflected surface forms hide the mention span from the encoder.","I infer that separating candidate generation from ranking means the two stages can be upgraded independently, so a stronger retriever could be swapped in without retraining the ranker.","I infer that the reported training setup (a single consumer GPU) makes this a practical strong baseline for future multilingual nested entity linking tasks."],"forward_implications":["On the official BioNNE 2025 leaderboard, the system's third-place finish shows a stock biomedical encoder with these three modifications is competitive in a nested, bilingual setting.","The boundary-cue ablation quantifies the contribution: [Ms]/[Me] tokens add about 1.2 percentage points on English and the bilingual track and 6.6 points on Russian, suggesting the cues matter most where morphology is rich.","Contrastive ranking outperforms listwise ranking by a wide margin at roughly k times the compute, documenting a direct accuracy-versus-cost trade-off in the ranking stage.","Adding MedMentions and MCN to the ranking training set improves final accuracy on all tracks, with the largest gain on English, showing that existing corpora transfer after reformatting.","Because the same recipe is tested on BioLinkBERT, SapBERT, and BiomedBERT, the paper claims the pipeline is encoder-agnostic and can be lifted onto other biomedical encoders."],"supporting_citations":[{"why":"Supplies SapBERT, the self-aligned pretrained encoder whose PubMedBERT and XLMR variants serve as the retrieval-ranking backbone in the best runs.","marker":"[6]"},{"why":"Supplies LinkBERT/BioLinkBERT, one of the off-the-shelf encoder families tested at retrieval to demonstrate encoder-agnostic behavior.","marker":"[5]"},{"why":"Defines the BioNNE-L shared task, its UMLS-based evaluation, and the leaderboard on which the third-place result is measured.","marker":"[7]"},{"why":"Supplies the MCN nested-entity corpus in Russian that is reformatted and added to the ranking training set as augmentation.","marker":"[9]"},{"why":"Supplies BiomedBERT, the encoder used in the English final run that reaches 0.6370 top-1 accuracy with MedMentions.","marker":"[16]"},{"why":"Supplies MedMentions, the English UMLS-annotated corpus reformatted to the three target entity types and used to expand ranking training.","marker":"[17]"},{"why":"Supplies the NEREL-BIO/MCN base data for the Russian and bilingual tracks on which the system is trained and evaluated.","marker":"[18]"}],"fun_headline_variants":["Unchanged encoder still nails third in nested biomedical EL","Three tweaks, one encoder: third in multilingual linking","BIBERT-Pipe: minimal fix lands third in BioASQ 2025","Nested and multilingual? A light pipe takes third","Ranking tweaks beat purpose-built models in BioNNE"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The system can only be as accurate as its retrieval stage: if the correct concept is not among the top-10 candidates returned by the retriever, the ranker has no way to recover it.","fun_headline_variants_meta":{"raw":{"variants":["Unchanged encoder still nails third in nested biomedical EL","Three tweaks, one encoder: third in multilingual linking","BIBERT-Pipe: minimal fix lands third in BioASQ 2025","Nested and multilingual? A light pipe takes third","Ranking tweaks beat purpose-built models in BioNNE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1197,"prompt_tokens":740,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":484,"tokens_out":457,"duration_ms":4719,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:38:55.013637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Russian-track ranker on the development set with the [Ms]/[Me] boundary tokens removed while keeping candidates, training data, and hyperparameters fixed. The paper reports Acc@1 falls from 0.6497 to 0.6095 in this ablation; an independent run that does not show a comparable drop would falsify the boundary-cue mechanism. A second check is to force the gold concept into every candidate list and remove the boundary cues: if final accuracy does not fall below the full-system value, the claimed components are not load-bearing.","supporting_citations":[],"review_version":1}