{"id":"40a8939c-e8cd-45fc-a5e6-b26130fe397f","arxiv_id":"2506.13467","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NeuroEmbed combines ontology-based metadata normalization and synonym expansion with contrastive fine-tuning of PubMedBERT, improving retrieval of neurodegenerative disease cohorts on a synthetic question-answering benchmark.","lead":"This paper introduces NeuroEmbed, a pipeline that cleans and expands metadata for neurodegenerative disease cohorts using biomedical ontologies, then fine-tunes a text-embedding model to answer natural-language cohort search queries. A generalist reader might care because it offers a possible way to make thousands of public omics studies findable through plain-language searches instead of manual filtering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported retrieval gains may be inflated by lexical leakage: test synonyms are held out of NLQ construction but never removed from the cohort metadata being searched, so test queries can match by exact term overlap. The MPR formula and its reported direction are also inconsistent.","rationale":"The reader's weakest assumption was that synthetic NLQ-cohort pairs are a valid proxy for real researcher queries; my concern is a sharper version of the same evaluation problem. The paper has real strengths: a reproducible pipeline, public code and catalogue, detailed normalization counts, and internally coherent loss curves. But the evaluation protocol is the load-bearing element for the headline claim, and it has a concrete, avoidable leak: test vocabulary is held out from NLQ construction but not from the indexed cohort texts. This is not a claim about author intent; it is a standard train/test separation issue. If the proposed holdout test shows little change, conditional acceptance can proceed; if it shows a large drop, the quantitative headline must be revised and external validation is required. I therefore keep the reader's CONDITIONAL verdict with no adjustment, because the concern does not by itself prove failure—it identifies exactly what evidence is missing.","tokens_in":15292,"tokens_out":6970,"duration_ms":78838,"concrete_test":"Re-run the evaluation with strict synonym holdout: after the 80/20 vocabulary split, strip every test-synonym string from all cohort metadata records before building the embedding index and running retrieval, while keeping the 1,886 test NLQs unchanged. Recompute mean Retrieval Precision and MPR. If precision drops substantially toward the 0.277 baseline, the reported gains are largely lexical leakage; if it stays near 0.866, the generalization claim survives. In the same run, report both rank/total and 1 - rank/(N+1) so the MPR direction is unambiguous.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that fine-tuning PubMedBERT on NeuroEmbed's QA set improves cohort retrieval—requires the 1,886 test queries to measure semantic generalization. That condition is not secured. In Step 1, each cohort's metadata is expanded with all ontology synonyms (Table 1 final counts: Ti 326, Po 105, As 51, Ph 292). In Step 2, the 774 augmented values are split 80/20 into training and test synonyms, and test NLQs are built exclusively from test synonyms. The paper never states that test synonyms are removed from the cohort metadata that is embedded and ranked. Therefore a test query containing, for example, a test-split tissue synonym can contain that synonym verbatim in the target cohort's augmented metadata, and the model can succeed by lexical term overlap rather than by generalizing to unseen vocabulary. MNRL training reinforces exactly this kind of matching. The reported mean Retrieval Precision of 0.866 and MPR of 0.896 are thus compatible with a system that has memorized the augmented vocabulary, not with one that understands novel researcher phrasings. A second, independent inconsistency affects the headline numbers: the printed MPR equation (rank divided by total) is lower-is-better, but the results treat increasing MPR from 0.355 to 0.896 as improvement. Either the formula or the reported direction is wrong, so the quantitative claim is not self-consistent as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents NeuroEmbed, a four-stage pipeline for building semantically searchable catalogues of neurodegenerative-disease omics cohorts. Cohorts are retrieved from GEO, their metadata are normalized and augmented with ontology-derived synonyms, a synthetic natural-language QA dataset is generated by combining augmented metadata terms, and PubMedBERT is fine-tuned with a contrastive loss to improve retrieval. The authors report that fine-tuning raises mean Retrieval Precision from 0.277 to 0.866 and mean Percentile Rank from 0.355 to 0.896 on 1,886 held-out synthetic queries, and they release the catalogue and code. The main claimed contribution is that ontology augmentation plus fine-tuning yields a model that can retrieve relevant cohorts from natural-language queries.","tokens_in":15571,"tokens_out":5092,"duration_ms":52596,"significance":"If the quantitative claims are valid, NeuroEmbed would be a useful practical contribution: it provides a large curated and ontology-augmented catalogue of ND cohorts (2,801 cohorts, 150,924 samples), an open-source pipeline, and a method that could transfer to other biomedical domains. The work also demonstrates substantial normalization effort (e.g., 1,700 heterogeneous tissue labels reduced to 326 canonical terms). However, the central empirical claim is currently not secured. The evaluation may be contaminated by lexical leakage because test-split synonyms remain in the cohort metadata used for retrieval, and the reported MPR definition is internally inconsistent with the direction of the reported improvement. These issues must be resolved before the retrieval gains can be accepted as evidence of semantic generalization. The strengths are the released resources and the reproducible, well-documented pipeline; the weakness is the validity of the headline evaluation.","major_comments":[{"comment":"The held-out test split does not prevent lexical leakage into the retrieval index. The paper states that the 774 augmented metadata values are split 80/20 into training and test synonyms and that test NLQs are built exclusively from test synonyms, but it never states that test synonyms are removed from the cohort metadata that is embedded and ranked. Since Step 1 expands each cohort's metadata with all ontology synonyms, a test query containing a test-split synonym (e.g., a tissue synonym) can match that synonym verbatim in the target cohort's indexed metadata. Under these conditions, the reported Retrieval Precision of 0.866 and MPR of 0.896 can be achieved by exact-term matching rather than by generalizing to unseen vocabulary, and the claim that the model generalizes across 'novel synonym instances' is not supported. Please either demonstrate that test synonyms were excluded from the indexed cohort metadata, or redesign the evaluation so that the test set is free of this overlap (e.g., remove test synonyms from all cohort documents before retrieval, or evaluate on independently collected real researcher queries).","section":"Step 2: QA dataset generation; Step 4: embedding's evaluation"},{"comment":"The MPR definition as printed is inconsistent with the reported results. The equation MPR = (1/N) Σ (Rank_of_Cohort_i / Total_number_of_Cohorts) is a lower-is-better metric if rank 1 denotes the highest similarity; yet the paper reports an increase from 0.355 to 0.896 as an improvement and Figure 2C treats MPR values near 1.0 as desirable. As written, the equation would imply that the fine-tuned model performs worse than the base model (0.896 > 0.355). This is a load-bearing inconsistency in the headline quantitative claim. Please correct the formula (e.g., MPR = 1 − (1/N) Σ Rank/Total, or define rank as 0 = best) or correct the description of the reported direction.","section":"Step 4: embedding's evaluation, MPR equation"},{"comment":"The evaluation uses only synthetic NLQs generated from the same templates, synonym vocabulary, and gold-cohort mapping as the training data. While the held-out synonym split avoids the simplest memorization, the test set does not contain real researcher queries or independently judged relevance. The paper's claim that NeuroEmbed enhances 'cohort discovery for research' therefore rests on the untested assumption that the synthetic query distribution approximates real usage. I recommend adding a small human-annotated evaluation, or at least a qualitative assessment using real queries typed by researchers into the deployed Gradio interface, to support the claimed real-world utility.","section":"Step 2 and Discussion"},{"comment":"The formula for 'Retrieval Precision' is stated as Number_of_Relevant_Cohorts_Retrieved / Total_number_of_relevant_cohorts, which is the standard definition of recall, not precision. Moreover, no retrieval cutoff is specified, so it is unclear how a cohort counts as 'retrieved' in a ranked list. This ambiguity makes the reported numerical values (0.277 and 0.866) difficult to interpret and reproduce. Please clarify the metric: specify whether a fixed top-k cutoff is used, and use either standard precision or recall terminology, or define a combined measure explicitly.","section":"Step 4: embedding's evaluation, Retrieval Precision definition"}],"minor_comments":[{"comment":"The text contains numerous typos and grammatical errors, including 'Medicins' for 'Medicines', 'LWB' for 'Lewy Body Disease' (standard abbreviation is LBD), 'Gradios's' for 'Gradio's', and 'e.g, ontologies' missing a period. A thorough language edit is needed.","section":"Throughout"},{"comment":"The MNRL/InfoNCE loss equation is referenced as 'Equation 1' but the equation does not appear in the provided text. Please include the explicit formula, as it is central to the training procedure.","section":"Step 3: model fine-tuning"},{"comment":"Algorithm 1 is described in the text but the actual algorithm box/listing is missing from the manuscript. Please include the algorithm pseudocode so that the data-generation procedure is fully reproducible.","section":"Methodology, Step 2"},{"comment":"Table 1 is difficult to parse because the column layout mixes raw counts with percentages without clear grouping. For example, the Ti row lists 560 matched values, of which 326 are unique standardized terms, but the relationship between these numbers and the 'Final Count' column is not immediately clear. Please reformat the table with separate columns for matched values, unique canonical terms, and synonym counts.","section":"Results, Table 1"},{"comment":"Several references are cited with numbers that do not correspond to the reference list (e.g., [25] is used for both 'thefuzz' and 'NeuML/pubmedbert-base-embeddings'). The reference list also contains duplicated entries (e.g., Le-Khac et al. appears twice) and incomplete citations. A careful revision of references is required.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The referee report follows the reader's report and stress-test note. The most serious concern is the potential lexical leakage in the evaluation: if the test-split synonyms appear in the indexed cohort metadata, the reported retrieval gains could be largely a byproduct of exact term matching, which would invalidate the authors' claim of semantic generalization. The MPR inconsistency is also a clear correctness issue. Both are fixable with a re-run of the evaluation and a corrected metric definition, so I recommend major revision rather than rejection. Given the public availability of the code and data, an independent check of the leaked-vocabulary scenario should be straightforward."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The evaluation doesn't support the headline numbers. Test synonyms are held out of query construction but never removed from the indexed metadata, so the model can match by exact lexical overlap; and the printed MPR equation is lower-is-better while the results treat higher as better. These two issues compromise the central claim. That said, there is real value in the resource and the pipeline.\n\nThe paper ships a public catalogue, normalizes 1,770 tissue annotations into 326 UBERON-aligned terms, expands metadata 2.7–20x, and builds a QA training set from ontology synonyms. That is reproducible, useful work. The idea of augmenting metadata with synonyms and fine-tuning a biomedical embedder with contrastive loss is not new—they cite prior hybrid methods—but the concrete application to ND cohort discovery and the released catalogue are a genuine addition.\n\nNow the soft spots, in proportion. The leakage is the big one. Step 1 expands each cohort's metadata with all ontology synonyms; Step 2 splits the synonym vocabulary into train/test, but nothing removes test synonyms from the corpus being ranked. So a test query containing a test-split synonym can match that same string in the target cohort. The model can succeed by lexical matching, not by generalizing to unseen phrasings. The held-out split does not secure semantic generalization. The authors say the split tests \"novel synonym instances\" and \"unseen query formulations,\" but only the template is unseen, and that is trivial. So the 0.866 versus 0.277 comparison is not evidence of semantic improvement. Second, the MPR formula: rank/total, so smaller is better; they report an increase from 0.355 to 0.896 as an improvement. Either the formula is wrong or the direction is wrong—as written, it is self-contradictory. Third, there are no error bars, and the test set consists of synthetic queries generated from the same templates used for training. That is a lesser issue but still worth noting. Finally, the abstract claims sample-level indexing of 150,924 samples, but the Discussion says they are \"currently working\" to index individual samples. That overclaim should be fixed.\n\nWho is this for? Bioinformaticians building semantic search over GEO. The resource is useful, and the methodological pitfalls are instructive. But the quantitative claims need major revision before anyone should rely on them. A serious referee should engage, but with a clear expectation that the evaluation be redone—either by removing test synonyms from the corpus or by using an external gold standard of real researcher queries.","headline":"The pipeline and public catalogue are real contributions, but the evaluation's head, line numbers are undermined by lexical leakage and an internally inconsistent MPR formula.","tokens_in":16081,"tokens_out":2650,"would_cite":false,"duration_ms":27823,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that ontology-augmented metadata combined with contrastive fine-tuning turns heterogeneous omics cohort descriptions into a semantically searchable catalogue for neurodegeneration research.","keywords":["neurodegenerative diseases","omics cohort discovery","metadata normalization","biomedical ontologies","question-answering dataset generation","embedding fine-tuning","semantic search","PubMedBERT"],"falsifier":"Collect a set of queries written by researchers who have not seen the paper's templates, have domain experts mark the relevant cohorts in the same archive, and compare the fine-tuned embedder's precision and percentile rank on those real queries against the reported 0.866 and 0.896; if the scores fall back toward the base model's 0.277 and 0.355, the synthetic-query proxy is the reason.","tokens_in":15108,"feed_emoji":"🧬","tokens_out":7498,"duration_ms":68115,"temperature":0.7,"pith_summary":"The paper argues that the main barrier to finding omics cohorts for neurodegeneration research is not scarce data but messy, unstandardized metadata, much of it stored as free text. NeuroEmbed addresses this by normalizing and expanding metadata terms with biomedical ontologies, generating a large synthetic question-answering dataset from the augmented terms, and fine-tuning a biomedical language model to match natural-language queries to cohort descriptions. On 1,886 held-out test queries built from synonyms the model never saw during training, mean retrieval precision rises from 0.277 to 0.866 and mean percentile rank from 0.355 to 0.896. If these synthetic queries reflect how researchers actually search, the same pipeline could make omics cohorts query-ready for any disease area that has a reference ontology.","feed_headline":"Fine-tuned embeddings lift omics cohort search to 0.866","feed_subtitle":"NeuroEmbed adds ontology synonyms and contrastive fine-tuning; four-term queries retrieve with perfect precision.","key_machinery":"The central device is a four-dimensional metadata model (Population, Assay, Phenotype, Tissue) whose values are normalized and expanded through biomedical ontologies, followed by automated construction of a question-answering dataset. The dataset generator takes the augmented vocabulary, splits it into disjoint training and test synonym sets, forms random combinations of one to four terms, keeps only combinations that match at least one cohort, and converts each query-cohort pair into natural language using predefined templates. Fine-tuning the embedder with a contrastive loss then teaches it to pull the correct query-cohort pair together in embedding space while pushing unrelated cohorts in the batch apart, so free-text queries align with structured cohort descriptions.","core_discovery":"NeuroEmbed establishes that unstructured cohort metadata can be converted into a semantically searchable catalogue through three coupled steps: ontology-based normalization and synonym expansion of four metadata dimensions (Population, Assay, Phenotype, Tissue), automated generation of natural-language queries by randomly combining those augmented terms, and fine-tuning of a PubMedBERT-based embedder with a contrastive ranking loss. Applied to 2,801 cohorts and 150,924 samples from a major public omics archive, the method normalizes more than 1,700 heterogeneous tissue labels into 326 standardized concepts and expands phenotype terms roughly twenty-fold. After fine-tuning, the model attains a mean retrieval precision of 0.866 and a mean percentile rank of 0.896 on 1,886 held-out queries, compared with 0.277 and 0.355 for the untuned base embedder, and queries combining all four metadata dimensions reach perfect precision.","pith_inferences":["If the synthetic-query proxy holds, the same recipe should transfer to other repositories and disease domains wherever a reference ontology exists, making it a general cure for heterogeneous cohort metadata rather than a neurodegeneration-specific fix.","Reported precision measures exact-match retrieval against one gold cohort per query, whereas real researchers often accept several relevant cohorts; deployment precision could differ until evaluated with graded relevance judgments.","The authors state they are extending the approach to sample-level indexing and multi-omics similarity; if cohort-level gains persist at sample granularity, compound queries such as 'two mouse samples with Parkinsonian phenotypes and the most similar multi-omics fingerprints' become practically achievable."],"forward_implications":["A semantically indexed catalogue of 2,801 neurodegeneration cohorts and 150,924 samples becomes queryable by population, assay, phenotype, and tissue without manual filtering.","Ontology normalization collapses thousands of heterogeneous tissue labels into standardized concepts, making cross-cohort comparison and automated bioinformatic pipelines feasible.","Fine-tuning raises mean retrieval precision from 0.277 to 0.866 and mean percentile rank from 0.355 to 0.896 on held-out natural-language queries.","Queries that combine all four metadata dimensions are retrieved with perfect precision, indicating that the model captures multi-condition constraints well.","Residual retrieval failures concentrate in two-term queries with near-synonymous population terms, pointing to ontological disambiguation as the next improvement rather than a new architecture."],"supporting_citations":[{"why":"Supplies the public omics archive from which the 2,801 neurodegeneration cohorts and their raw metadata were collected.","marker":"[1]"},{"why":"Supplies the medical subject terms and synonyms used to query the archive and filter disease-specific cohorts.","marker":"[15]"},{"why":"Supplies ontology terms used to normalize and expand assay and phenotype metadata values.","marker":"[17]"},{"why":"Supplies the anatomy ontology that maps heterogeneous tissue labels into standardized concepts.","marker":"[18]"},{"why":"Supplies the taxonomy used to normalize and expand population descriptors.","marker":"[19,20]"},{"why":"Supplies a fallback terminology source for synonym expansion when primary ontology matches fail.","marker":"[21,22]"},{"why":"Provides the pretrained biomedical embedder that is fine-tuned; its baseline retrieval scores are the comparison point for the reported gains.","marker":"[34]"}],"fun_headline_variants":["NeuroEmbed lifts cohort retrieval precision to 0.866","Ontology-augmented embeddings boost omics cohort search to 0.866","Fine-tuned embeddings + ontology synonyms: 0.866 precision for cohort queries","From 0.277 to 0.866: NeuroEmbed sharpens cohort discovery","Four-term cohort queries reach perfect precision with NeuroEmbed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic natural-language queries created by randomly combining ontology-augmented metadata terms are a valid proxy for how real researchers search for cohorts, so the retrieval precision measured against these self-generated gold answers reflects genuine cohort discovery utility.","fun_headline_variants_meta":{"raw":{"variants":["NeuroEmbed lifts cohort retrieval precision to 0.866","Ontology-augmented embeddings boost omics cohort search to 0.866","Fine-tuned embeddings + ontology synonyms: 0.866 precision for cohort queries","From 0.277 to 0.866: NeuroEmbed sharpens cohort discovery","Four-term cohort queries reach perfect precision with NeuroEmbed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001002,"raw_usage":{"total_tokens":4288,"prompt_tokens":1043,"completion_tokens":3245,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":3149}},"tokens_in":659,"tokens_out":3245,"duration_ms":25114,"temperature":1.0,"reasoning_tokens":3149,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:00:11.479057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a set of queries written by researchers who have not seen the paper's templates, have domain experts mark the relevant cohorts in the same archive, and compare the fine-tuned embedder's precision and percentile rank on those real queries against the reported 0.866 and 0.896; if the scores fall back toward the base model's 0.277 and 0.355, the synthetic-query proxy is the reason.","supporting_citations":[],"review_version":2}