{"id":"ab0b73bf-6a83-412a-8c25-a37e73d51b9f","arxiv_id":"2608.01734","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using an LLM to select biologically similar neighbor drugs and averaging their measured effects improves zero-shot prediction of molecular perturbation responses, particularly for unseen cell lines.","lead":"A new method, LLM-Guided Retrieval, uses a large language model to rank similar drugs and averages their measured gene-expression effects to predict an untested drug's response. On the Tahoe-100M atlas it beats several baselines, especially when predicting responses in cell lines never seen during training.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unseen cell-line gains conflated: LGR has test-time access to the held-out cell line's own measured drug responses via P_c, while baselines are cell-agnostic; this advantage is independent of retrieval quality.","rationale":"The reader's weakest assumption concerns leakage from the LLM's pretraining corpus. While that is a legitimate concern, it is not the most load-bearing issue. A more fundamental problem is that LGR's access to the target cell line's own measured responses (via P_c) gives it an advantage that is independent of retrieval quality. In the unseen-cell-line regime, LGR's aggregation uses deltas from drugs profiled in the same held-out cell line. Any baseline that averages over a random subset of those deltas would already capture cell-line-specific baseline expression and response tendencies, which drug-mean or global-mean baselines cannot. The paper's comparison therefore does not isolate the contribution of LLM-guided selection; it conflates retrieval quality with test-time access to cell-specific data. This is not merely a subtle statistical point: it directly undermines the central claim that retrieval quality is a key driver of zero-shot prediction, because the observed gains could be reproduced by a non-LLM, non-selective method that uses the same candidate pool. The paper should add baselines that use the same test-time candidate pool (e.g., random neighbors, pool mean, or chemistry kNN with cell-specific aggregation) before drawing that conclusion. The unseen-drug regime does not have this confound because the candidate pool contains training drugs, and LGR's improvements there are more limited but still present. Therefore the overall verdict remains CONDITIONAL: the method shows promise, but the strongest evidence requires re-evaluation with fair baselines. My disagreement with the reader is on which assumption is weakest: I identify the test-time access asymmetry, not LLM memorization, as the most load-bearing.","tokens_in":10576,"tokens_out":6009,"duration_ms":63840,"concrete_test":"Implement a 'random-neighbor cell-specific' baseline: for every test query (c,d) in the unseen-cell-line split, sample k drugs uniformly without replacement from P_c\\{d}, average their observed deltas, and report the same metrics as Figure 2. Also implement a 'cell-line mean' baseline equal to the mean of all deltas in P_c\\{d}. If LGR does not clearly beat these, then its gains are due to test-time candidate access, not retrieval quality. Additionally, run chemistry kNN restricted to P_c and aggregate cell-specific deltas (rather than drug-mean profiles) to isolate the retrieval mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the unseen-cell-line regime (Section 4.1), LGR retrieves neighbors from the candidate pool P_c, which by Section 3 (Eq. 2) and Appendix B contains all drugs with measured responses in the held-out cell line. The prediction is the mean of these cell-line-specific deltas (Eq. 4). Thus even a selector that ignores biology and picks random drugs from P_c would produce a cell-line-specific prediction, whereas the baselines compared in Figure 2 are cell-agnostic: drug mean, chemistry kNN, and PCA+RR are trained on other cell lines, and cell mean falls back to the global mean for unseen cell lines. LGR therefore has privileged access to the target cell line's own response distribution—an advantage unrelated to LLM-guided retrieval. The reported >0.15 Pearson improvement over drug mean and roughly doubled R^2 may largely reflect this test-time information. The paper's claim that retrieval quality is a key driver is supported only if LGR outperforms a baseline with identical test-time access (e.g., random or chemically blind retrieval from P_c). No such baseline is reported. This confound is distinct from the acknowledged LLM-pretraining leakage (Section 4.4.1) and is not discussed.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLM-Guided Retrieval (LGR), a retrieve-and-aggregate method for predicting transcriptomic perturbation responses. For an unobserved (cell line, drug) pair, LGR uses a pretrained LLM to rank candidate drugs from a cell-line-specific candidate pool P_c, then predicts the response as the uniform mean of the observed expression deltas of the top-k retrieved candidates. The method is evaluated on the Tahoe-100M single-cell perturbation atlas under unseen-drug, unseen-cell-line, and open-world regimes, against drug mean, cell mean, PCA+RR, chemistry kNN, and ChemCPA baselines. The paper reports consistent improvements, with the largest gains in the unseen-cell-line regime, and attributes these gains to retrieval quality rather than predictor complexity.","tokens_in":10875,"tokens_out":4878,"duration_ms":53965,"significance":"If the empirical claims hold, the paper makes a useful contribution: it provides a transparent, parameter-free retrieve-and-aggregate baseline that can outperform more complex supervised predictors, and it demonstrates a concrete way to use LLMs as constrained selectors rather than as numerical predictors. The reproducibility measures — cached neighbor lists, deterministic parsing, fixed prompting, and a deliberately simple aggregator — are strengths. However, the central unseen-cell-line claim is currently undermined by a test-time information asymmetry between LGR and the baselines, and by the unaddressed possibility of LLM pretraining leakage. The paper's own Section 4.4.1 acknowledges the latter but treats it as a feature rather than as a confound. Because the main conclusions rest on the unseen-cell-line comparison, these issues must be resolved before the claims can be accepted.","major_comments":[{"comment":"The unseen-cell-line evaluation gives LGR test-time access to the held-out cell line's own measured deltas, while the baselines are cell-agnostic. By Eq. (2) and Appendix B, the candidate pool P_c contains all drugs with measured responses in the target cell line, including the held-out cell line. LGR's prediction (Eq. 4) is the mean of those cell-line-specific deltas. In Fig. 2, drug mean, chemistry kNN, and PCA+RR are trained on other cell lines, and cell mean falls back to the global mean for unseen cell lines. Thus even a random selector over P_c would produce a cell-line-specific prediction and would likely outperform cell-agnostic baselines. The reported >0.15 Pearson improvement may reflect this privileged access to the target cell line's response distribution, not retrieval quality. The paper must add ablations with identical test-time access — e.g., random retrieval from P_c and","section":"§4.1, Fig. 2, Eq. (2)-(4)"},{"comment":"The closed-world evaluation restricts the candidate pool to drugs observed during training, but this restriction does not apply to the LLM's internal knowledge. The LLM is a pretrained model whose training corpus may contain published transcriptional responses for exactly the held-out cell lines and drugs. Section 4.4.1 explicitly attributes the unseen-cell-line gain to 'literature transfer' from the LLM's pretraining. This is a leakage concern: the reported gains may reflect memorization of test answers rather than a generalizable retrieval mechanism. The paper needs a control that blocks or quantifies this leakage, for example by comparing LGR against a non-LLM selector using only mechanism tags or other cell-line-agnostic features, or by evaluating on cell lines/drugs that postdate the LLM's training cutoff. Merely acknowledging the transfer does not establish that the closed-world co","section":"§4.1, §4.4.1"},{"comment":"No error bars, confidence intervals, or significance tests are reported anywhere in the experimental section. The abstract and conclusions use language such as 'consistently improves' and 'strongest gains', but every figure shows only point estimates over what appears to be a single split. Since the differences between LGR and cell-mean/drug-mean are sometimes small on magnitude metrics (e.g., MAE in Fig. 3), the paper should provide variance estimates over bootstrap resamples of cell lines/drugs or over multiple data splits. Without this, it is impossible to assess whether the reported improvements are reliable.","section":"Fig. 2-4"}],"minor_comments":[{"comment":"The description of the closed-world regime says the candidate pool 'available to all methods is restricted to drugs observed during training', but Appendix B defines P_c as all non-control molecules measured in cell line c, with no intersection with training drugs. For the unseen-cell-line task these are different sets. Please clarify exactly which candidate pool is used for each regime and for each baseline.","section":"§4.1"},{"comment":"The paper says it records k_used for each query to 'characterize selector coverage and its impact on variance', but no k_used statistics are reported. Since low k_used is mentioned as a failure mode in §4.5, please report the distribution of k_used in each regime.","section":"§3.1"},{"comment":"ChemCPA is absent from Fig. 2 because it cannot be evaluated on unseen cell lines. This is stated in the baseline section, but it would be helpful to repeat this in the figure caption so the reader does not wonder about the missing bar.","section":"§4.2"},{"comment":"'The performance gain can be attributed to zero-shot learning via literature transfer' appears to be a fragment; the sentence reads as if an explanation was intended. Also, 'attributed' should be 'attributed' (typo in the provided text).","section":"§4.4.1"},{"comment":"The numeric labels under the bars are ambiguous (e.g., '0.39 0.23 0.37 0.20 0.59' in Fig. 2a appears to list bar values, but the alignment is unclear). Please format the figure labels so each value is directly above its bar.","section":"Fig. 2-4"},{"comment":"The LLM is described only as 'GPT-5–class model'. For reproducibility, please specify the exact model name, version, and query date, as LLM behavior can vary across versions.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim about unseen-cell-line generalization is currently confounded by a test-time information asymmetry that is easy to fix with additional baselines. The LLM pretraining leakage is also acknowledged but not controlled. If the authors add the requested ablations and the results still show LGR outperforming equivalent-access baselines, the paper would be a solid contribution. The lack of error bars is also a concern for a journal submission. I would not reject at this stage, but the revision must address the confounds directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a clean idea and a significant evaluation flaw. The unseen-cell-line result, which is the paper's strongest claim, likely reflects LGR's access to measured response data from the held-out cell line, not the LLM's retrieval judgment. I'd want a random-retrieval control before believing the headline.\n\nWhat's genuinely new: framing small-molecule perturbation prediction as retrieve-and-aggregate, adapting the LangPert selector-aggregator to a cell-line-restricted candidate pool. The method is transparent: fixed mean aggregator, no fitted parameters, LLM only ranks candidates, and the authors cache all LLM outputs. The evaluation uses Tahoe-100M, covers three regimes, and they report sign accuracy, which is often ignored. The sign-accuracy gains are consistent and might be the most robust part of the paper.\n\nThe problem: In the unseen-cell-line regime, the candidate pool P_c (Eq. 2) contains every drug with measured response in that held-out cell line. LGR's prediction is the mean of top-k neighbors drawn from P_c. Even a random selection from P_c would yield a cell-line-specific prediction, while all baselines (drug mean, chemistry kNN, PCA+RR, cell mean with global fallback) are cell-agnostic. So LGR has a test-time information advantage entirely unrelated to retrieval quality. The paper's closed-world wording is misleading: the candidate pool is not restricted to training drugs. They never compare against a baseline that also uses P_c (e.g., random or chemical-similarity retrieval inside P_c). Without that control, the reported >0.15 Pearson gain and doubled R^2 are unexplained.\n\nThe LLM pretraining leakage (Section 4.4.1) is a related but separate concern. The paper even attributes the gain to literature transfer, which suggests they are aware of memorization, but they don't see the direct P_c effect. This is the load-bearing flaw.\n\nSecondary issues: no error bars or significance tests; the cell-line subset is underspecified; and the mechanism-tag mapping is a bit hand-wavy. Those are minor by comparison.\n\nWho's this for? The perturbation-prediction community, especially people working on zero-shot drug response. The direction is good, and if the P_c confound is resolved, this could be a solid contribution.\n\nRecommendation: send it to peer review, but with a clear instruction to require a random/chem-blind retrieval baseline from P_c and a discussion of the information asymmetry. It deserves a serious referee, but the current evidence doesn't support the central claim.","headline":"The unseen-cell-line gains likely come from LGR's test-time access to the held-out cell line's own response data, not from LLM-guided retrieval.","tokens_in":11356,"tokens_out":5215,"would_cite":false,"duration_ms":53793,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Retrieval quality, not model complexity, drives drug response prediction","keywords":["molecular perturbation prediction","transcriptomic response","retrieve-and-aggregate","LLM-guided retrieval","zero-shot generalization","cell-line-specific retrieval","sign accuracy","drug response prediction"],"falsifier":"Run LGR on held-out cell lines and drugs using an LLM whose training data provably excludes those specific perturbations (or retrain an open-weights model with those records withheld); if the correlation and sign-accuracy gains over mean baselines disappear, the effect is literature memorization, not retrieval. Alternatively, replace the LLM selector with a similarity score derived from an independent, non-overlapping perturbation atlas; if performance drops to chemistry-kNN levels, the LLM prior is the active ingredient.","tokens_in":10496,"feed_emoji":"🧬","tokens_out":4725,"duration_ms":46921,"temperature":0.7,"texified_at":"2026-08-05T21:58:23.151043+00:00","pith_summary":"This paper argues that the main bottleneck in predicting how a drug perturbs gene expression in an unmeasured cell line is not the complexity of the prediction model but the quality of the retrieved drug neighbors. It proposes LLM-Guided Retrieval (LGR), which uses a large language model to rank, from a candidate pool of drugs already profiled in the target cell line, the compounds most likely to share the query drug's transcriptional program; a fixed mean of their observed effects forms the prediction. On the Tahoe-100M single-cell perturbation atlas, LGR consistently outperforms drug-mean, chemistry-based k-nearest-neighbor, and ChemCPA baselines, with the largest gains when generalizing to cell lines never seen in training. If correct, this reframes zero-shot perturbation prediction as a retrieval problem and suggests LLMs can serve as constrained biological priors rather than as end-to-end predictors.","texify_model":"deepseek-v4-flash","texify_usage":{"total_tokens":2771,"prompt_tokens":737,"completion_tokens":2034,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":737,"completion_tokens_details":{"reasoning_tokens":1395}},"feed_headline":"LLM-chosen drug neighbors top complex baselines in cell-line tests","feed_subtitle":"A fixed mean of LLM-ranked analogs predicts unseen cell-line responses with higher correlation and lower error.","key_machinery":"The central object is the cell-line-restricted candidate pool P_c together with an LLM acting as a selector over it. For a query (cell line c, drug d), the LLM ranks the top-k candidates from $P_c \\setminus \\{d\\}$ using mechanistic and pathway reasoning; outputs are parsed, validated against the pool, and the uniform mean of the retrieved drugs' observed control-subtracted expression deltas is the prediction. The mean aggregator is deliberately fixed and non-parametric, so any performance difference is attributed to neighborhood construction. This isolates retrieval quality as the variable of interest.","core_discovery":"The paper's central claim is that a retrieve-and-aggregate estimator—selecting a small neighborhood of biologically related compounds and averaging their measured transcriptomic deltas—can match or beat learned end-to-end predictors for zero-shot molecular perturbation response, provided the neighborhood is chosen well. The selection is done by an LLM prompted to reason over mechanisms, pathways, and cell-line context, but constrained to a closed candidate pool measured in the target cell line. Across unseen-drug, unseen-cell-line, and open-world regimes on Tahoe-100M, LGR improves correlation, error, and especially sign accuracy of gene regulation over chemistry-based kNN and supervised bas","pith_inferences":["If retrieval quality is the bottleneck, investing in richer or better-profiled candidate pools should improve prediction more than upgrading the aggregator—a direct test of the paper's thesis.","The reported unseen-cell-line gains may partly reflect literature memorization by the pretrained LLM; a truly closed evaluation would require an LLM trained without the held-out lines.","The sign-accuracy improvements suggest LGR could be used as a gene-set prioritization tool for pathway enrichment even when effect-size calibration is poor.","A hybrid that falls back to chemistry-based retrieval when LLM coverage is low may mitigate the reported instability without losing the biological prior."],"forward_implications":["Simple, transparent predictors paired with good retrieval can match or beat complex learned models in zero-shot settings.","Directional (sign) accuracy of gene regulation improves more than magnitude metrics, which matters for pathway and mechanism interpretation.","LLMs can act as constrained selectors that inject a biological prior without generating high-dimensional numerical outputs.","Cell-line-specific candidate pooling makes aggregation well-defined and enables clean closed-world evaluation.","Retrieval-based frameworks offer an interpretable alternative to end-to-end supervised perturbation models."],"supporting_citations":[{"why":"Supplies the Tahoe-100M single-cell perturbation atlas used for all evaluations.","marker":"Zhang et al. 2025"},{"why":"Closest prior selector–aggregator approach for genetic perturbations, adapted here to small molecules.","marker":"Märtens et al. 2025"},{"why":"Shows simple baselines remain competitive, motivating the retrieval-focused framing.","marker":"Ahlmann-Eltze et al. 2025"},{"why":"ChemCPA is a learned baseline that LGR must beat or match.","marker":"Hetzel et al. 2022"},{"why":"Provides OOD benchmark design with leakage controls that shapes the evaluation splits.","marker":"Szałata et al. 2024"},{"why":"CPA is a compositional latent-variable baseline representing the learned-predictor approach compared against.","marker":"Lotfollahi et al. 2023"}],"fun_headline_variants":["LLM-picked drug neighbors beat learned predictors","LLM-guided analog retrieval improves perturbation prediction","Retrieval quality trumps model complexity in zero-shot drug response","Mean of LLM-ranked analogs boosts unseen cell-line accuracy","LLM selection of neighbors outperforms chemistry-based kNN"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The held-out cell lines and drugs are treated as unobserved by all methods, but the LLM's pretraining corpus may already contain published responses for exactly those cell lines and drugs, so the reported unseen-cell-line gains could reflect memorized literature rather than a generalizable retrieval mechanism.","fun_headline_variants_meta":{"raw":{"variants":["LLM-picked drug neighbors beat learned predictors","LLM-guided analog retrieval improves perturbation prediction","Retrieval quality trumps model complexity in zero-shot drug response","Mean of LLM-ranked analogs boosts unseen cell-line accuracy","LLM selection of neighbors outperforms chemistry-based kNN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1167,"prompt_tokens":754,"completion_tokens":413,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":335}},"tokens_in":498,"tokens_out":413,"duration_ms":5233,"temperature":1.0,"reasoning_tokens":335,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T21:55:36.814694+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LGR on held-out cell lines and drugs using an LLM whose training data provably excludes those specific perturbations (or retrain an open-weights model with those records withheld); if the correlation and sign-accuracy gains over mean baselines disappear, the effect is literature memorization, not retrieval. Alternatively, replace the LLM selector with a similarity score derived from an independent, non-overlapping perturbation atlas; if performance drops to chemistry-kNN levels, the LLM prior is the active ingredient.","supporting_citations":[{"cited_title":"and Thompson, John D","cited_arxiv_id":null,"evidence_quote":"Supplies the Tahoe-100M single-cell perturbation atlas used for all evaluations."},{"cited_title":"Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines","cited_arxiv_id":null,"evidence_quote":"Shows simple baselines remain competitive, motivating the retrieval-focused framing."},{"cited_title":"Predicting cellular responses to complex perturbations in high‐throughput screens","cited_arxiv_id":null,"evidence_quote":"CPA is a compositional latent-variable baseline representing the learned-predictor approach compared against."}],"review_version":1}