{"id":"e1a4c88e-4b1d-4af2-b640-f51b8777f2dd","arxiv_id":"2607.18144","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"General-purpose LLMs can satisfy simple local 3D constraints such as anchor fragments and pharmacophores, but their unoptimized poses dock much worse than those from specialized diffusion models.","lead":"This paper introduces 3D-Fit, a benchmark that asks general-purpose language models to generate drug-like molecules in 3D under spatial constraints from protein pockets, anchor fragments, pharmacophore points, and mandatory protein-ligand interactions. Across two structure-based drug-design datasets, frontier LLMs can follow simple local constraints but produce poorer raw docking poses than specialized diffusion models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Condition-success metric counts bare atom placement, not fragment/feature identity, so the paper's positive 'LLMs can satisfy spatial constraints' claim is inflated.","rationale":"The reader's weakest assumption correctly identified the placement-only condition-success metric as the main threat to the paper's positive claim. I agree: the negative result (LLMs lag diffusion models on docking quality and PoseBusters validity) is robust to this concern because it is based on UniDock scores and pose plausibility, not on the placement metric. The positive claim, however, is only as strong as the condition-success definition, and that definition is deliberately decoupled from chemical connectivity. Because the authors transparently state this choice in §5.3 and §7, the issue is not hidden or internally inconsistent, but it does limit what the benchmark can conclude. The paper also provides useful independent support: a released code/data link, a format ablation, and a clear description of the evaluation pipeline. These do not resolve the metric-validity concern. The reader's CONDITIONAL verdict is appropriate; my analysis does not move it, hence UNCHANGED. I chose agreement 'agree' because the reader's weakest assumption is exactly the same load-bearing concern, and my concrete test directly tests that assumption.","tokens_in":23211,"tokens_out":4623,"duration_ms":57005,"concrete_test":"Recompute anchor-fragment and pharmacophore condition success on the released generated molecules using a substructure-based checker: parse each generated molecule with RDKit, perform a 2D substructure match against the exact anchor fragment (and a pharmacophore-feature match requiring the relevant functional group, not just any atom of the right element), then check whether matched atoms fall within 0.5 Å of the reference coordinates. Also report raw vs UniDock-optimized SR separately. If the placement-only SR (e.g., ~99% after optimization in Tables 2–4) drops materially to, say, below 50%, the claim that LLMs satisfy spatial constraints should be softened to 'LLMs can place atoms at specified coordinates,' and the positive half of the central claim would need revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: LLMs meaningfully follow explicit spatial constraints, and they lag diffusion models on physical plausibility and binding quality. The second part is robust: raw UniDock scores are consistently poor across all LLMs and improve only after local optimization, and the optimized scores remain behind MolSnapper/PocketXMol/PDMD. The load-bearing weakness is the first part, because it rests on the condition-success metric defined in §5.3. For anchor fragments and pharmacophore points, success is scored purely by the presence of an atom of the right element within 0.5 Å of the specified coordinate; 2D connectivity and the chemical identity of the fragment/feature are explicitly not checked. A model can therefore 'satisfy' an anchor fragment by copying the coordinate list from the prompt into the output, or by scattering unrelated atoms near the specified points, without ever forming the requested substructure. The paper's own §6 observation that LLMs perform best when conditions can be 'directly copied, mirrored, or locally reconstructed' suggests this is not a hypothetical failure mode. The conclusion's phrasing that LLMs 'follow spatial constraints' and 'perform well on Anchor Fragment and Pharmacophore Point conditioning' therefore overstates what the metric demonstrates: it measures coordinate placement, not chemical constraint satisfaction. This is especially problematic for the multi-condition setting (§5.4, Table 4), where the headline 'handle multiple spatial constraints simultaneously' rests on the same placement-only definition. The absence of statistical significance testing, flagged in §7, compounds the issue: the finer-grained comparisons among individual LLMs and between raw and optimized SR are not supported by error bars or tests.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 3D-Fit, a benchmark for evaluating general-purpose LLMs on 3D molecule generation conditioned on protein pockets together with additional spatial constraints: anchor fragments, pharmacophore points, and mandatory protein–ligand interactions. It proposes token-efficient textual representations of these conditions and a Simplified SDF output format, then evaluates a range of proprietary and open-weight LLMs against specialized diffusion baselines on CrossDocked2020 and PLINDER. The headline findings are that LLMs can parse the spatial instructions and frequently place atoms near the requested coordinates, but their raw docking scores are poor and, even after local UniDock optimization, remain behind the best diffusion models; LLM outputs also show weaker internal and external physical validity. The paper concludes that LLMs are increasingly capable of multi-condition generation but do not yet match diffusion models in physical plausibility and binding quality.","tokens_in":23503,"tokens_out":6313,"duration_ms":54359,"significance":"If the results hold, 3D-Fit provides a valuable, standardized benchmark for LLM-based 3D molecular design that goes beyond pocket-only generation to multi-constraint settings. The study's strengths include the breadth of models (11 LLMs and 8 diffusion baselines), use of two established test sets, an ablation of output formats, a code/data link, and explicit acknowledgment of limitations. The conclusion that LLMs lag diffusion models in docking quality and physical plausibility is robust: raw UniDock scores for all LLMs are consistently poor and improve only after local optimization, and optimized scores remain behind MolSnapper, PocketXMol, and PDMD. However, the positive claim that LLMs 'satisfy' anchor fragment and pharmacophore constraints is weakened by the placement-only success metric, which does not verify chemical identity or connectivity. With a stricter metric, the qualitative conclusions about LLM spatial reasoning would need to be reassessed.","major_comments":[{"comment":"The condition-success metric for Anchor Fragments and Pharmacophore Points is purely 3D atom placement within 0.5 Å; 2D connectivity and feature identity are explicitly not checked. This conflicts with §3's definition of anchor fragments as 'chemically significant ligand substructures' and pharmacophores as 'key functional features.' Consequently the high SRs in Tables 2–4 can be obtained by copying the coordinate list from the prompt and emitting isolated atoms, without forming the requested substructure. §6's observation that LLMs do best when conditions can be 'directly copied, mirrored, or locally reconstructed' indicates this is not a hypothetical concern. The Section 8 claim that LLMs 'satisfy explicit local 3D constraints' is therefore inflated. Please re-evaluate with substructure matching for anchors and functional-group checks for pharmacophores, and report whether the qualitat","section":"§5.3, §3"},{"comment":"The paper states as a limitation that no statistical significance testing was performed. For a comparative benchmark across many models and conditions, this matters: the claims 'adding more spatial conditions often improves pocket-related metrics' (§6) and 'still remain behind the best pocket-specific diffusion models' (§6) rest on differences in median scores and SRs without any error bars or hypothesis tests. Given the test sets have 948 and 505 complexes, paired bootstrap confidence intervals or a Wilcoxon signed-rank test on per-complex UniDock scores are feasible and should be reported for at least the main comparisons (Tables 1 and 4). Without this, small differences (e.g., among GPT-5.5 and Opus 4.8 in Table 2) cannot be distinguished from noise.","section":"§7"},{"comment":"The experimental protocol does not specify the number of generated molecules per test complex or how the reported percentages are aggregated across models and targets. The success rate is defined 'on all outputs' but the paper does not state whether one sample per complex or multiple samples were generated, nor how many outputs were produced in total. This information is essential for interpreting the binary metrics, particularly the many 0% and low SRs. Please add the sampling count, total output count per model/dataset, and a description of how multiple samples per complex are aggregated (e.g., per-complex success then averaging, or pooled).","section":"§5.1/§5.4"}],"minor_comments":[{"comment":"Typo: 'enviornment' should be 'environment'.","section":"§7"},{"comment":"In the pocket description example, 'Residue 109, V AL:' should be 'Residue 109, VAL:'.","section":"Appendix A"},{"comment":"The prompt says 'close the tags and generate a new molecule from scratch in a new set of mol tags,' but the format uses <sdf> tags; make the terminology consistent.","section":"Appendix A prompt"},{"comment":"Reference [5] has 'URLhttps://' missing a space; should be 'URL https://'.","section":"Reference [5]"},{"comment":"DeepSeek v3.2 is listed as an open-weight LLM but does not appear in any results table; clarify whether it was excluded and why.","section":"§5.1"},{"comment":"The capitalization of 'UniDock' is inconsistent (e.g., 'Unidock' in headers). Please standardize.","section":"Tables 1–4"}],"recommendation":"major_revision","confidential_remarks":"The paper's self-citations to BindGPT and nach0-pc are used as format baselines and are not problematic. The GitHub repository link is provided; as a reviewer I could not verify its contents at the time of review. The central weakness is the placement-only condition-success metric, which is directly tied to the paper's most positive claim; this should be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the headline: this is a useful benchmark with a robust main result and a flawed secondary metric that inflates one of its positive claims. If you work on structure-based drug design or LLMs for 3D generation, it's worth a read.\n\nWhat's new: 3D-Fit is the first benchmark I know that pushes general-purpose LLMs with pocket + anchor fragment + pharmacophore + mandatory interaction conditions, all textual, with the Simplified SDF output format. The design is clearly described, the datasets are standard, and the authors report raw and optimized UniDock scores, PoseBusters checks, and parsing rates. The main negative result—LLMs lag diffusion models on docking and physical plausibility—is consistent across every table and robust to the choice of dataset. The raw LLM poses are poor; local optimization helps but doesn't close the gap. That conclusion seems solid.\n\nWhat the paper does well: the token-efficient textual representations, the fixed sampling protocol, and the honest limitations section. They explicitly flag the lack of statistical significance testing in §7. The random-fragment and single-pharmacophore choices are stated as limitations, not hidden.\n\nWhere it's soft: the condition-success metric for anchor fragments and pharmacophore points counts an atom of the right element within 0.5 Å of the specified coordinate, without checking 2D connectivity or feature identity. So a model can 'satisfy' an anchor fragment by placing scattered atoms near the coordinates rather than forming the actual fragment. The paper's own observation that models do best when conditions can be directly copied, mirrored, or locally reconstructed makes this concern concrete. This inflates the claim that LLMs 'follow spatial constraints' and 'handle multiple spatial constraints simultaneously.' The 0.5 Å tolerance appearing in both prompt and metric is a design choice, not circularity, but the metric needs a connectivity-aware variant before the positive claims can be taken at face value.\n\nAlso, the model inventory is sloppy: §5.1 lists DeepSeek v3.2, which never appears in the results, while Opus 4.8 and Grok 4.3 show up in tables without being declared. That's an easy fix but it matters for reproducibility. And without error bars or significance tests, the finer per-model comparisons are suggestive, not definitive.\n\nWho it's for: anyone comparing general-purpose LLMs to specialized diffusion models for 3D molecular generation. It deserves a serious referee; I'd send it out. I'd probably cite it for the benchmark setup and the robust negative result, with a caveat on the condition-success definition.","headline":"Useful benchmark with a robust negative result; the positive claim that LLMs can satisfy spatial constraints is inflated by a placement-only success metric.","tokens_in":24064,"tokens_out":2247,"would_cite":true,"duration_ms":22204,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"General-purpose LLMs can follow explicit 3D spatial instructions to design ligand molecules, but their binding poses remain physically less plausible than those from specialized diffusion models.","keywords":["structure-based drug design","3D molecule generation","large language models","spatial constraints","benchmark","pharmacophore","anchor fragments","Simplified SDF"],"falsifier":"A reader could falsify the central capability claim by testing a strong LLM on anchor-fragment conditions where the requested fragment is a rare or synthetically unusual substructure, then checking whether the generated molecule's 2D graph contains the exact fragment rather than just scattered atoms near the coordinates; if placement-success rates collapse under 2D connectivity verification, the reported spatial instruction-following would be largely placement mimicry.","tokens_in":1239,"feed_emoji":"🧬","tokens_out":2701,"duration_ms":53886,"temperature":0.7,"pith_summary":"This paper asks whether general-purpose large language models can reason about 3D geometry well enough to generate small molecules that satisfy realistic spatial constraints in protein binding. To answer it, the authors build 3D-Fit, a benchmark that conditions generation on a protein pocket plus up to three additional spatial requirements: anchor fragments, pharmacophore points, and mandatory pocket-ligand interactions. The central finding is that frontier LLMs show an emerging ability to follow such multi-condition spatial instructions, and adding more conditions tends to improve their pocket placement. However, their raw poses contain severe steric clashes and internal geometry problems, and even after local optimization their docking scores and physical validity trail state-of-the-art diffusion models. A fair reader would take away that off-the-shelf LLMs are not yet competitive structure-based drug designers, but they form a flexible substrate for instruction-following 3D generation.","feed_headline":"LLMs follow 3D binding rules but lag diffusion models","feed_subtitle":"LLMs can satisfy 3D binding constraints in text, but raw poses still need optimization.","key_machinery":"The load-bearing machinery is 3D-Fit, a benchmark built on token-efficient textual condition representations——pockets written as sequential residue atom coordinates, anchor fragments as element-plus-sphere constraints, pharmacophore points as typed coordinates——paired with a Simplified SDF output format that removes redundant header fields and uses explicit atom indices and bonds. The mechanism that carries the argument is the benchmark's ability to convert diverse 3D conditions into plain-text instructions an LLM can be prompted with, together with an evaluation protocol that cleanly separates condition satisfaction (atomic placement) from physical validity (PoseBusters filters and UniDock","core_discovery":"On the paper's own terms, the discovery is a comparative capability profile: LLMs can parse compact textual descriptions of spatially explicit molecular generation conditions and produce syntactically valid Simplified SDF output that places atoms within 0.5 Å of requested anchor and pharmacophore coordinates at meaningful success rates, and they can juggle pocket, fragment, pharmacophore, and interaction conditions simultaneously. Yet their generated conformations are physically rough: raw UniDock scores are poor, PoseBusters intra- and inter-molecular pass rates are low, and only local pose optimization brings docking scores into the moderate range. Diffusion baselines that support the same","pith_inferences":["A natural extension the paper leaves implicit: because anchor and pharmacophore success only checks 3D placement of atoms of the right element, the reported 'condition satisfied' rates likely overstate chemical reconstruction fidelity; a stricter check on 2D connectivity of the fragment would probably lower the numbers.","The success of textual condition-following suggests that stronger reasoning or domain-specific fine-tuning of LLMs could narrow the gap with diffusion models, especially by training on simplified SDF and explicit spatial reasoning tasks.","One could extend 3D-Fit to deliberately conflict conditions, such as anchor and pharmacophore constraints that cannot be satisfied simultaneously, to quantify how LLMs prioritize validity over condition satisfaction; the prompt already instructs prioritization but the paper does not measure it.","The same benchmarking strategy could be adapted to other 3D design domains, such as materials or protein design, wherever constraints are naturally expressed as point-placement requirements."],"forward_implications":["If the paper is right, general-purpose LLMs can serve as drop-in generators for multi-condition 3D molecular design prompts without retraining, as long as downstream local pose optimization is applied.","Adding more explicit seed-ligand conditions (anchor fragments, pharmacophore points, interactions) tends to improve LLM pocket placement, so richer condition sets may be a practical path to better LLM-generated poses.","A simplified, indexed SDF-like text output outperforms SMILES+XYZ for this task, indicating that output format design is a first-order factor in LLM 3D molecular generation.","Current LLM raw poses are not reliable enough for direct structure-based drug design; they require substantial repositioning, whereas diffusion models' poses are stable under optimization.","The benchmark enables scaling to heterogeneous condition combinations that no single diffusion model currently supports natively."],"fun_headline_variants":["LLMs handle 3D constraints but produce rough poses","LLMs satisfy 3D rules yet trail diffusion in geometry","LLMs reason about 3D space but fall short on physics","LLMs meet spatial constraints but lose on docking quality","LLM molecules follow 3D hints but need post-processing"],"cache_read_input_tokens":25344,"weakest_assumption_plain":"The paper's success metric for anchor fragments and pharmacophore points counts a condition as satisfied if atoms of the right element land within 0.5 Å of the prescribed coordinates, without checking whether the requested chemical fragment's bond network is actually present.","fun_headline_variants_meta":{"raw":{"variants":["LLMs handle 3D constraints but produce rough poses","LLMs satisfy 3D rules yet trail diffusion in geometry","LLMs reason about 3D space but fall short on physics","LLMs meet spatial constraints but lose on docking quality","LLM molecules follow 3D hints but need post-processing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1500,"prompt_tokens":719,"completion_tokens":781,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":697}},"tokens_in":463,"tokens_out":781,"duration_ms":8161,"temperature":1.0,"reasoning_tokens":697,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T15:50:15.289858+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could falsify the central capability claim by testing a strong LLM on anchor-fragment conditions where the requested fragment is a rare or synthetically unusual substructure, then checking whether the generated molecule's 2D graph contains the exact fragment rather than just scattered atoms near the coordinates; if placement-success rates collapse under 2D connectivity verification, the reported spatial instruction-following would be largely placement mimicry.","supporting_citations":[],"review_version":1}