{"id":"84f8850e-49c6-49cd-a711-aeb69c89310c","arxiv_id":"2505.11582","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On a private dataset of 1,472 structured medical documents, lexical vector search (BM25) achieved slightly better classification accuracy than off-the-shelf semantic embeddings while embedding and searching much faster.","lead":"This paper tests seven ways to classify medical documents by vector search: three lexical methods and four neural embedding methods. On its private medical dataset, the simple lexical method BM25 matched or beat the neural models in accuracy while running much faster.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The lexical-vs-semantic conclusion may be an artifact of the aggressive preprocessing in Sec. III-D, which strips numbers, dates, and stopwords and stems text before feeding off-the-shelf semantic models.","rationale":"The reader's weakest assumption is exactly the preprocessing fairness issue, and I agree that it is the most load-bearing concern. The paper's headline comparison depends on treating all seven methods equally, but Section III-D applies a lexical-retrieval preprocessing regime (stemming, stopword removal, deletion of numbers and dates) that is known to be non-neutral for transformer-based embeddings. Since no significance tests are reported for the central Fig. 4 comparison and no data or code are released, the conditional verdict is appropriate. I would not move the verdict to REJECT: the experiments are clearly described, repeated over 10 seeds, and BM25's strong performance is plausible; the issue is that the causal framing ('lexical is more accurate than off-the-shelf semantic') is not yet supported relative to preprocessing. The proposed concrete test would resolve the confound directly and could be run without releasing the sensitive data, since the authors have access to the corpus and a MacBook.","tokens_in":10662,"tokens_out":6475,"duration_ms":76484,"concrete_test":"Re-run the 75:25 and 50:50 classification experiments for MiniLM and mxbai using (a) raw transcribed text and (b) lowercase-only text, keeping BM25 on the full Sec. III-D preprocessing. Use the same 10 random seeds and report mean and standard deviation of accuracy. If either raw or lowercase-only semantic model matches or exceeds BM25's preprocessed accuracy by more than one standard deviation, then the headline claim is an artifact of the shared preprocessing pipeline rather than a property of lexical versus semantic search.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the comparison between BM25 and the semantic models (MiniLM, mxbai) is a fair test of the methods themselves. Section III-D applies one pipeline to all methods: lowercase, remove all punctuation, numbers, dates, parentheses, and unicode, then tokenize, stem, and remove stopwords. This pipeline is designed for lexical retrieval and is not how off-the-shelf transformer embeddings are typically used. MiniLM and mxbai tokenizers were trained on raw or lightly cleaned text; stemmed forms such as 'prescrib' can be broken into unnatural subwords, and removing numbers and dates removes medically discriminative content (dosages, appointment dates, reference numbers) from documents whose classes F and G are described as very rigidly structured. Meanwhile, stemming and stopword removal are standard, beneficial steps for BM25 and TF-IDF because they collapse surface variants and expose term matches. So the reported result is better described as 'BM25 with this preprocessing pipeline beat MiniLM/mxbai with the same pipeline,' not necessarily 'lexical search beat off-the-shelf semantic search.' The abstract frames the finding as a property of method families, but the preprocessing step is a confound. The paper is honestly reported and repeated seeds help, but no significance tests or error bars appear in Fig. 4, and no code or data are released, so the magnitude of the BM25-vs-MiniLM gap cannot currently be separated from seed variance, let alone from preprocessing bias.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript compares seven embedding methods (TF, TF-IDF, BM25+, word2vec, med2vec, MiniLLM/MiniLM, and mxbai) for classifying 1,472 rigidly structured medical documents into seven classes using k-nearest-neighbour vector search. The main empirical claims are that lexical vector search, especially BM25, achieves slightly higher predictive accuracy than off-the-shelf semantic vector search while requiring less execution time, that an unbalanced class distribution biases accuracy, and that larger stored sets improve accuracy. The evaluation uses held-out splits with 10 shuffled replicates, per-class precision/recall/F1, and transparent variation of the data-split ratio and the number of neighbours.","tokens_in":10985,"tokens_out":6070,"duration_ms":62176,"significance":"If the central comparison were fully supported, the paper would be a useful empirical counterpoint to the default use of neural embeddings: it demonstrates on a real, rigidly structured medical corpus that a simple lexical method with corpus-specific preprocessing can match or beat off-the-shelf sentence transformers while being much faster. The held-out evaluation, repeated seeds, explicit default hyperparameters, and per-class figures are strengths. However, the headline claim is currently threatened by the use of a single aggressive preprocessing pipeline tailored to lexical matching and applied unchanged to transformer models, and by the absence of variance reporting or significance tests for the small accuracy differences. The contribution is a single-domain observational benchmark with no released code or data, so its generalizability and reproducibility are limited.","major_comments":[{"comment":"The single preprocessing pipeline is a confound for the headline lexical-vs-semantic comparison. Lowercasing, removing all punctuation, numbers, dates, parentheses, and unicode, then stemming and removing stopwords, is standard for BM25 and TF-IDF, but it is not how off-the-shelf transformer embedding models such as MiniLM and mxbai are normally used. Stemmed forms are unnatural inputs to subword tokenizers, and removing numbers and dates can strip discriminative content such as dosages, reference numbers, and appointment dates from the rigidly structured classes F and G. The abstract therefore overstates the finding: the experiment demonstrates that BM25 under this preprocessing pipeline beats MiniLM/mxbai under the same pipeline, not that lexical search in general beats off-the-shelf semantic search. Please add a condition in which the semantic models are run on raw or lightly cleaned text, or otherwise justify that the preprocessing does not disproportionately handicap the neural models.","section":"Sec. III-D, Table I, Fig. 4"},{"comment":"The claim that BM25 is 'slightly' more accurate than MiniLM is based on mean accuracy over 10 seeds, but Fig. 4 shows no standard deviations, confidence intervals, or significance tests for those means. The differences across data splits appear small, and with only 10 seeds the gap could plausibly be within seed variance. Please report the per-seed distributions (or confidence intervals) and a paired significance test across the 10 seeds for the BM25-vs-MiniLM comparison at each split, or weaken the claim accordingly.","section":"Fig. 4, Sec. V-A"},{"comment":"Because the default class E contains 803 of 1,472 documents (about 55%), the overall accuracy values in Fig. 4 are strongly influenced by how well class E is predicted. The paper discusses this class-imbalance bias in Sec. V-B but never reports the trivial majority-class baseline (always predicting E) or a macro-averaged accuracy. Without that baseline, the reader cannot determine how much of the reported accuracy reflects genuine discrimination rather than the class prior. Please add the majority-class baseline and macro-averaged metrics to Fig. 4 and the associated discussion.","section":"Sec. IV, Table I, Fig. 4"},{"comment":"The model is referred to inconsistently as 'MiniLLM' in the text and reference [19], and as 'minilm' in the figures and Sec. II-E2. Reference [19] describes a knowledge-distillation method for autoregressive language models, not an off-the-shelf sentence-embedding model, while MiniLM is a different architecture. Please specify the exact model and checkpoint used (for example, the specific ollama model or Hugging Face identifier), the parameter variant, and any quantization, and correct the terminology and reference. The comparison depends on which semantic model is actually being evaluated.","section":"Sec. II-E2, Sec. III-E, Figs. 2-5"}],"minor_comments":[{"comment":"The BM25+ formula has unbalanced parentheses in the denominator; please rewrite it so that the term-frequency normalisation is unambiguous.","section":"Eq. (5)"},{"comment":"The caption says the timing process was repeated 25 times, while the methodology in Sec. III-G says experiments were repeated 10 times with different random seeds; please clarify which number is correct.","section":"Fig. 3 caption"},{"comment":"The text says BM25+ with a delta term was used, but Sec. III-E says the 'default BM25 algorithm' from bm25s was used; please reconcile whether the experiments used BM25+ or plain BM25.","section":"Sec. II-D vs Sec. III-E"},{"comment":"There is a typo in 'Stored:T est Data Split'; it should read 'Stored:Test Data Split'.","section":"Fig. 4 caption"},{"comment":"The phrase 'under the initial assumption that more is better' conflicts with the later result that accuracy decreases as k increases; rephrase to describe the choice neutrally.","section":"Sec. III-G"},{"comment":"The colour descriptions for the lexical and semantic methods overlap ('brown' appears in both groups); please make the colour references unambiguous or add patterns to the figures for accessibility.","section":"Sec. V-A"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a modest, honestly reported benchmark that mostly replicates a known result, and its headline comparison is muddied by one methodological choice. The stress-test note is right. The single aggressive preprocessing pipeline in Sec. III-D—lowercase, strip all numbers/dates/punctuation, stem, remove stopwords—is a natural fit for BM25/TF-IDF and an unnatural one for MiniLM/mxbai, whose tokenizers expect closer-to-raw text. So the paper does not establish \"lexical beats semantic on medical documents\"; it establishes \"BM25 with this pipeline beat MiniLM with this pipeline on this private dataset.\" That is still a useful engineering data point, but the abstract and conclusion state it as a property of method families.\n\nWhat is genuinely useful: a clean seven-method comparison on a real (though private) medical-document classification task, with repeated seeds, a held-out split, per-class precision/recall/F1, and timing and memory numbers. The discussion of class imbalance is sensible, and the authors cite [21] as prior similar work, so they are not overclaiming novelty.\n\nSoft spots, in order. First, no majority-class baseline. Class E is 803 of 1472 documents, about 55%; Fig. 4 accuracies of 64–78% are above majority but we do not know by how much, or what a majority classifier scores on the same splits. Second, no variance or significance in Fig. 4; with seven curves averaged over 10 seeds, small gaps like BM25 vs MiniLM cannot be separated from seed noise. Fig. 2 has standard deviations, but the headline figure does not. Third, the preprocessing confound already noted: classes F and G are described as very rigidly structured, and the pipeline removes numbers and dates—exactly where dosage, appointment, and reference information would live. Fourth, no code or data. The data cannot be shared for privacy, but the pipeline could be released, and the paper would be stronger for it. Minor: the model called \"MiniLLM\" in the methods is elsewhere referred to as \"minilm\"; naming should be cleaned up.\n\nVerdict: worth a serious referee, but only with major revision. The comparison needs the majority baseline, error bars, and a preprocessing-robustness check—for example, run the semantic models on raw text and on the stemmed text. If the BM25-vs-MiniLM gap survives that, the paper becomes a solid practical result. As it stands, it is a conditional data point.","headline":"Honest but confounded benchmark: the lexical-vs-semantic conclusion holds for the paper's preprocessing pipeline, not as a general claim about the methods.","tokens_in":11429,"tokens_out":2418,"would_cite":false,"duration_ms":25314,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For rigidly structured medical documents, a bespoke lexical BM25 vector search matches or beats off-the-shelf semantic neural embeddings in accuracy, and does it far faster.","keywords":["medical document classification","automatic indexing","vector search","lexical search","semantic search","BM25","k-nearest neighbour","text embeddings"],"falsifier":"Run the same 75:25 k-nearest-neighbour classification on an equivalent set of rigidly structured medical documents, changing only the preprocessing from the paper's aggressive pipeline to lowercasing alone; if MiniLM or mxbai then exceeds BM25 in accuracy, the paper's central conclusion is an artifact of the shared preprocessing rather than a property of the methods.","tokens_in":10488,"feed_emoji":"📄","tokens_out":10811,"duration_ms":103699,"temperature":0.7,"pith_summary":"This paper asks whether the neural embeddings that dominate modern vector search are the right tool for every classification task. Its test case is sorting 1,472 rigidly structured medical documents into 7 classes by nearest-neighbour search, comparing three lexical term-weighting methods (TF, TF-IDF, BM25) with four off-the-shelf semantic models (word2vec, a medical word2vec variant, MiniLM, and mxbai). The central finding is that a bespoke lexical BM25 model, with a vocabulary built from the training set, classified these documents slightly more accurately than the best semantic model (MiniLM) and embedded and searched much faster. The paper also finds that the heavily majority class (E) biases accuracy in its favour, and that more stored/training data improves every method. If true, this is a concrete case where a simple, fast, interpretable lexical method beats popular neural semantic search on a real task.","feed_headline":"Lexical BM25 beats neural search on medical documents","feed_subtitle":"A bespoke term-matching model was more accurate and much faster than semantic embeddings on structured medical records.","key_machinery":"The machinery is a controlled comparison of seven embedding methods inside one vector-search classifier. Lexical methods produce sparse vectors from term statistics, such as TF counts, TF-IDF weights, and BM25+ scores with saturation and document-length normalisation, while semantic methods produce dense vectors from neural models (word2vec, med2vec, MiniLM, mxbai). The same pipeline feeds every method: transcribed text is lowercased, stripped of punctuation, numbers, dates, parentheses, and unicode, stemmed, and filtered for stopwords; a vocabulary is built from the training set; embeddings are stored in a vector database; and each test document's class is the majority of its 5 nearest stored embeddings under Euclidean distance. The identity that carries the comparison is therefore the embedding itself: if the best sparse term-statistic embedding beats the best dense neural embedding on this data, the win is attributed to the representation rather than to any model-specific training, since the downstream classifier is identical.","core_discovery":"On the authors' own terms, the discovery is that the choice between lexical and semantic representation can be reversed by task structure. Using the same preprocessing, the same k-nearest-neighbour classifier (k of 5, Euclidean distance), and the same 75:25 data split, BM25 achieved the highest average predictive accuracy of the seven methods, edging out MiniLM; word2vec, med2vec, and mxbai trailed, and TF and TF-IDF were at the bottom. The detail that carried the argument was speed: BM25's embedding and search times were near negligible, while the neural models, especially mxbai, took substantially longer. The paper interprets this as evidence that off-the-shelf semantic vector search is not always the best solution and that lexical methods deserve to be contenders, particularly when execution time or hardware constraints matter. It also reports that med2vec, trained on medical text, did not perform particularly well, suggesting document structure mattered more than medical vocabulary in this dataset.","pith_inferences":["Editorial inference: the aggressive preprocessing pipeline, especially stemming and deleting numbers and dates, likely harms dense neural embeddings disproportionately; rerunning with method-appropriate cleaning could shrink or reverse the accuracy gap.","Editorial inference: because classes F and G, the most rigidly structured classes, were nearly perfectly classified by every method, this corpus may be an easy case for lexical methods, and free-form clinical notes would probably shift the balance toward semantic models.","Editorial inference: the memory comparison overstates the practical cost of lexical embeddings, since sparse vectors can be stored by non-zero entries alone; a sparse-storage implementation could remove the main resource advantage the paper grants to semantic methods.","Editorial inference: a direct test of the class-imbalance claim would be to re-run all methods on a balanced subsample or with class weights, and the paper's own results predict that the majority class's advantage would shrink and the ranking could shift."],"forward_implications":["Traditional lexical search should be included as a baseline in any document-classification benchmark; on rigidly structured corpora it can beat state-of-the-art neural embeddings.","For tasks where document template and layout matter more than vocabulary, expensive domain-specific language models may add little, and structure-aware simple models may be the better investment.","Class imbalance distorts k-nearest-neighbour accuracy even when embeddings are good, so per-class precision and recall reporting and balancing are necessary for meaningful comparisons.","When execution time is a constraint, such as high-throughput or on-premise processing of sensitive medical records, BM25's speed advantage makes it a practical choice over semantic search.","Increasing the amount of stored or training data raises accuracy across methods, so data collection may improve results more than adopting a more sophisticated embedding model."],"supporting_citations":[{"why":"Provides the BM25 scoring formula that generates the winning lexical embeddings.","marker":"[30]"},{"why":"Introduces the BM25+ lower-bound term-frequency normalisation variant used in the paper.","marker":"[28]"},{"why":"Supplies the fast sparse-scoring BM25 implementation used to produce BM25 embeddings and time measurements.","marker":"[36]"},{"why":"Defines the MiniLM model, the strongest semantic competitor in the comparison.","marker":"[19]"},{"why":"Documents the mxbai embedding model used as the state-of-the-art semantic baseline.","marker":"[20]"},{"why":"Introduces the word2vec approach used for the generic neural baseline.","marker":"[17]"},{"why":"Describes the med2vec model trained on medical text, the domain-specific neural baseline.","marker":"[18]"},{"why":"States the received view that semantic embeddings outperform lexical ones, the premise this comparison tests.","marker":"[23]"}],"fun_headline_variants":["BM25 edges out neural embeddings on medical document classification","Lexical search faster and more accurate than neural for medical docs","Task structure shifts advantage to lexical over semantic search","BM25 beats MiniLM on accuracy and speed for medical docs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one aggressive preprocessing pipeline, removing punctuation, numbers, dates, parentheses, and unicode, then stemming and removing stopwords, treats lexical and semantic methods fairly, even though the neural models are normally used with raw or lightly cleaned text.","fun_headline_variants_meta":{"raw":{"variants":["BM25 edges out neural embeddings on medical document classification","Lexical search faster and more accurate than neural for medical docs","Task structure shifts advantage to lexical over semantic search","BM25 beats MiniLM on accuracy and speed for medical docs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2660,"prompt_tokens":864,"completion_tokens":1796,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":1730}},"tokens_in":480,"tokens_out":1796,"duration_ms":11676,"temperature":1.0,"reasoning_tokens":1730,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:52:10.279566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 75:25 k-nearest-neighbour classification on an equivalent set of rigidly structured medical documents, changing only the preprocessing from the paper's aggressive pipeline to lowercasing alone; if MiniLM or mxbai then exceeds BM25 in accuracy, the paper's central conclusion is an artifact of the shared preprocessing rather than a property of the methods.","supporting_citations":[{"cited_title":"Okapi at trec-3,","cited_arxiv_id":null,"evidence_quote":"Provides the BM25 scoring formula that generates the winning lexical embeddings."},{"cited_title":"Lower-bounding term frequency normalization,","cited_arxiv_id":null,"evidence_quote":"Introduces the BM25+ lower-bound term-frequency normalisation variant used in the paper."},{"cited_title":"Launching into clinical space with medspacy: a new clinical text processing toolkit in python,","cited_arxiv_id":null,"evidence_quote":"Describes the med2vec model trained on medical text, the domain-specific neural baseline."},{"cited_title":"Improving lexical embeddings with semantic knowledge,","cited_arxiv_id":null,"evidence_quote":"States the received view that semantic embeddings outperform lexical ones, the premise this comparison tests."}],"review_version":1}