{"id":"9e23cdaf-db11-4395-9cf3-c49d52c8f670","arxiv_id":"2607.26397","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"For LLM enzyme classification, external homology evidence is what drives accuracy; reasoning wrappers add almost nothing on aggregate, giving opposite gains and losses on adversarial versus multi-functional enzyme splits.","lead":"The paper builds a no-training diagnostic benchmark showing that general LLMs predict enzyme EC numbers almost at zero closed-book, but jump to roughly 0.7 with retrieved homolog evidence, matching simple neighbor voting on average while differing sharply on adversarial and multi-functional splits. It separates formatting failures from knowledge gaps and argues reasoning mainly arbitrates conflicting evidence.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BLAST+HMMER neighbor-vote baseline underperforms its own best channel; the 19-vs-19 cancellation is an artifact of that fusion, and a competitive retrieval reference could erase the per-split arbitration claim.","rationale":"The reader identified the fairness of the BLAST+HMMER neighbor-vote reference as the weakest assumption; I agree and sharpen it with a concrete internal inconsistency: the fused reference is worse than its best component on two of four splits, which a well-designed baseline should never be. This is load-bearing because Finding 3's headline numbers (0.723 vs 0.728 aggregate tie; +0.127 price vs -0.090 promiscuous) are defined entirely relative to this reference. The exact 19-vs-19 cancellation is not robust to even a trivial alternative (HMMER-only), so the central 'arbitration' claim is vulnerable to baseline choice. The authors are transparent about small samples and lack of paired tests, which further compounds the issue, but the reference-construction problem is more fundamental. I still find the paper valuable as a diagnostic protocol and the qualitative results plausible; the concern does not warrant rejection, but it does require the proposed recomputation before the per-split cancellation is treated as settled. Hence the reader's CONDITIONAL verdict stands unchanged.","tokens_in":35444,"tokens_out":7071,"duration_ms":70782,"concrete_test":"Recompute Table 2 with a score-based best-hit baseline: for each query, predict the EC number of the highest-scoring hit among the union of all cached channels (BLAST, HMMER, ESM-kNN, Foldseek, active-site), breaking ties by bit score, and score with the same any-label rule. If the full-set L4 rises above 0.728, or the price gain relative to this baseline drops below 0.05 or the promiscuous loss below 0.02, the claimed arbitration effect and cancellation are artifacts of the underperforming BLAST+HMMER fusion. Also report HMMER-only as a simple reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Finding 3's central comparison depends on the 'plain neighbor voting' baseline defined as the top hits of BLAST and HMMER merged in an unspecified way. Table 6(b) shows this fused baseline is not a well-behaved retrieval rule: on promiscuous it scores 0.938, worse than both BLAST-only (0.957) and HMMER-only (0.957); on price it scores 0.372, below HMMER-only (0.378); on the full set it scores 0.723, below HMMER-only (0.725). A sensible fusion should be at least as good as its best component; the degradation implies the combination makes poor decisions when the two channels' top hits disagree (e.g., requiring agreement, arbitrary tie-breaking, or dropping a correct label). The paper's elegant '19 items converted vs 19 items given back' is therefore an artifact of this specific underperforming fusion: using HMMER-only as the reference changes the price gain to 0.121 (~18 items) and the promiscuous loss to 0.051 (~11 items), eliminating the exact cancellation. While the qualitative direction (gain on adversarial, loss on multi-label) may survive, the precise aggregate tie and the symmetric cancellation—the key evidence for 'arbitration rather than knowledge'—are not robust to the choice of a competitive retrieval reference. The paper never specifies how the two channels are merged, and no code is released, so the baseline is not reproducible as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"EC-Reason-Bench is a training-free diagnostic protocol for explaining why general LLMs collapse on full EC-number prediction. The paper decomposes the failure into four levers (output structure, external knowledge, reasoning structure, robustness), implemented as one baseline (B0) and four inference-time methods (M1–M4) evaluated with five reasoning LLMs over 1,349 CARE test items in four splits. The headline findings are: (1) open-book access to a strictly inductive offline evidence cache raises L4 accuracy from ≤0.16 to 0.70–0.76; (2) whether closed-book cascade/CoT help or hurt depends on a model's abstention tendency; (3) the best LLM settings tie a 'plain neighbor voting' retrieval baseline in aggregate, but this tie masks a large gain on the adversarial 'price' split and an equally large loss on the multi-label 'promiscuous' split; and (4) open-book accuracy rises monotonically with sequence homology to the training database.","tokens_in":35778,"tokens_out":9715,"duration_ms":94315,"significance":"The paper is methodologically strong in several important respects: strictly inductive retrieval (training-only database, anonymized UniProt IDs), fully offline evidence cache, valid-EC rate guaranteed by construction for cascade methods, bootstrap confidence intervals, and an unusually candid appendix that discloses several limitations. The per-split arbitration story — LLM reasoning as an arbiter of conflicting evidence rather than a source of knowledge — is original and, if made robust, would be a valuable caution against relying on single-number leaderboards. The benchmark itself, with its four levers and matched controls, is a useful community asset. These strengths are real and should be credited. However, the quantitative form of the central claim depends on an underspecified and apparently underperforming retrieval baseline, and the main-text wording overstates the statistical support that the paper's own appendix qualifies. The qualitative direction of the arbitration finding may survive, but the exact 19-versus-19 cancellation and the aggregate 'tie' are not yet established as stated.","major_comments":[{"comment":"The 'plain neighbor voting' reference is underspecified and is not a competitive retrieval baseline. The 'BLAST plus HMMER' fusion scores below HMMER-only on the full set (0.723 vs 0.725), below HMMER-only on price (0.372 vs 0.378), and below both component channels on promiscuous (0.938 vs 0.957). No rule is given for how the top hits of the two channels are merged. Because the headline 19-versus-19 cancellation is computed against this reference, it is not robust to the choice of a reasonable retrieval baseline. Recomputing against HMMER-only changes the M2 deltas to +0.121 on price and −0.109 on promiscuous, and the M4-open deltas to +0.054 and −0.051, removing the exact symmetry. The paper must specify the fusion rule, justify it, and report the comparison against each channel separately and against a well-defined fusion of the two channels.","section":"§6 Finding 3; Tables 6(b) and 14"},{"comment":"The main text says the best LLM setting is 'indistinguishable' from neighbor voting and that the three settings 'tie overall,' but the appendix correctly states that overlapping bootstrap intervals do not establish equivalence, no equivalence margin was prespecified, and no paired between-model tests were run. The per-split contrasts of +0.127 and −0.090 rest on only 148 and 209 samples (about 19 items each), and the appendix says these contrasts are not covered by the reported intervals. The abstract and conclusion nonetheless state the cancellation without these caveats. Please either provide paired bootstrap intervals for the per-split deltas and a prespecified equivalence analysis, or rephrase the claims as directional and sign-consistent rather than as a tie and an exact cancellation.","section":"§6 Finding 3; §7 Conclusion; Appendix 'Statistical Significance'"},{"comment":"The open-book condition supplies, by design, the EC numbers of homologous training neighbors in the evidence cache. The statement that 'external knowledge is decisive' is therefore close to a consequence of the protocol: the model is handed the answer in the retrieved label. The authors acknowledge this ('the retrieved evidence carries the EC numbers of homologous enzymes, so it puts the answer in context'), but the abstract and conclusion still present it as an empirical discovery. The 'law of homology availability' is similarly entangled, because the homology bins are computed from the same BLAST cache that provides the evidence. I recommend reframing Finding 1 as a sanity check of evidence use, or adding a control condition in which retrieved evidence contains sequences and homology scores but the EC labels are withheld, forcing the model to reason from similarity rather than read off","section":"§3 and §6 Finding 1; §7 Conclusion"}],"minor_comments":[{"comment":"Figure 2(C) lists 'pI' among the derived features, but Appendix B ('What a closed-book prompt contains') explicitly states that the descriptor block carries no isoelectric point. Please reconcile the figure with the text.","section":"Fig. 2(C) vs Appendix B"},{"comment":"The appendix discloses that M3 is the only method that receives the intrinsic-motif scan, so the M3−B0 contrast is not prompt-matched. Running B0 with the same motif list appended would cleanly isolate reasoning structure; as it stands, the direction of the confound is acknowledged but the clean control is missing.","section":"Appendix B, M3 subsection"},{"comment":"The annotation 'memorization' in the DeepSeek point label is speculative ('we suspect'). Please soften this to a hypothesis or add supporting evidence.","section":"Fig. 5"},{"comment":"All runs use a single seed, and the only repeated sampling is the k=3 self-consistency chains. This is disclosed in the appendix but should be stated in the main experimental section so that readers are not misled by the bootstrap intervals, which are sample-level only.","section":"§6, experimental setup"},{"comment":"The LLaPA reference is listed as 'OpenReview; ICLR 2025 submission, withdrawn.' If the work was withdrawn, please cite the available version or explain why it is cited; if it was rejected/withdrawn, the citation format should be checked.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I think this paper deserves another round. The empirical protocol and the benchmark construction are strong, and the qualitative arbitration finding is plausible and interesting. However, the central quantitative claim must be re-anchored to a well-defined and competitive retrieval baseline, and the main-text statistical claims need to be brought in line with the appendix's own caveats. I would not reject, because these issues are fixable within the manuscript's scope and the benchmark itself has value beyond the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read if you care about LLM evaluation for protein function. The paper builds a four-lever diagnostic for why LLMs collapse on EC numbers, evaluates five reasoning models, and shows that once you open the book, accuracy is almost entirely a function of homology availability; model choice matters mainly on the adversarial and multi-label splits. The aggregate leaderboard tie between a reasoning LLM and a neighbor-vote script is a nice piece of evidence for diagnostic reporting.\n\nWhat is actually new: the four-lever decomposition (output structure, external knowledge, reasoning structure, robustness), the strictly inductive offline evidence cache, the homology-availability curve, and the split-contingent findings—format repair helps or hurts depending on abstention tendency, and the LLM's net contribution is a gain on the 'price' split and a loss on promiscuous ones. The evaluation discipline is genuinely good: anonymized IDs, no test-to-test retrieval, valid-EC guarantee, bootstrap intervals, and unusually explicit limitations. The paper doesn't oversell; it repeatedly states what it is not claiming.\n\nThe soft spots are real but not fatal. The neighbor-vote baseline is the weakest point. 'Top BLAST and HMMER hits' is never specified as a fusion rule, and Table 6 shows the fusion is worse than HMMER alone on the full set and on both problematic splits. A sane fusion should be at least as good as its best component. That means the '19 items converted vs 19 items given back' symmetry is an artifact of that under-performing baseline. Using HMMER-only as the reference changes the magnitudes—roughly 18 gained on price versus 23 lost for M2, or 18 versus 11 for M4—so the exact cancellation disappears. The qualitative opposite-sign story survives, and the paper has a stronger argument on price: M2 beats the oracle best-channel bound (0.499 vs 0.426), which cannot be explained by channel selection. So the arbitration claim is not wrong, but the quantitative framing needs rework.\n\nTwo other things. The open-book condition is defined as feeding the model the EC numbers of homologous training neighbors, so 'knowledge is decisive' and the homology law are partly consequences of the protocol. The paper acknowledges this, but it does mean the contribution is a measurement protocol, not a discovery about LLMs. Second, one seed, no code or prompts released, and the per-split contrasts rest on 148 and 209 samples. The authors flag this too, but it limits how much weight the headline numbers can carry.\n\nWho's this for: people designing or critiquing LLM evaluation benchmarks, and anyone working on retrieval-augmented LLMs for structured prediction. I'd send it to peer review with a request to fix the baseline (specify the fusion, add HMMER-only and best-of references) and to release the prompt templates. With those changes, the core diagnostic contribution stands.","headline":"A carefully built diagnostic benchmark with an unusually honest limitations section; the headline cancellation narrative is partly an artifact of an under-specified retrieval baseline, but the qualitative arbitration claim and the homology-availability law hold up well enough to warrant serious refereeing.","tokens_in":36300,"tokens_out":4463,"would_cite":false,"duration_ms":43695,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that once retrieved homolog evidence is available, a general LLM's enzyme classification accuracy becomes indistinguishable from a script that votes the EC numbers of the nearest neighbors, and that the apparent tie h","keywords":["enzyme classification","EC number prediction","large language models","retrieval-augmented reasoning","hierarchical classification","diagnostic benchmark","homology availability","multi-label evaluation"],"falsifier":"Run an LLM-free voting baseline that merges hits from all five evidence channels (or a learned top-1 channel selector) on the adversarial price split. If this baseline reaches or passes the 0.499 L4 accuracy achieved by the LLM agentic setting on that split, while keeping promiscuous accuracy near the 0.938 neighbor-vote level, then the 'arbiter, not knowledge' claim falls: the LLM's apparent gain would be recoverable by better retrieval without any reasoning. Alternatively, a paired bootstrap over per-sample differences between M2 and the neighbor vote on the 148 price items would show whethe","tokens_in":35323,"feed_emoji":"🧬","tokens_out":5614,"duration_ms":57896,"temperature":0.7,"pith_summary":"This paper tries to establish why general-purpose language models score almost zero when asked for a full four-level Enzyme Commission number, and how much of that failure can be reversed without retraining. It builds a diagnostic benchmark that separates four levers—output structure, external knowledge, reasoning structure, and reasoning robustness—and measures each with a training-free inference-time method against a shared zero-shot baseline. The central finding is that knowledge, not reasoning, is what the models lack: closed-book fourth-level accuracy is at best 0.16 for these models, while opening an offline cache of retrieved homologs lifts it to 0.70–0.76 and erases most differences between models and methods. Once evidence is available, the best LLM setting matches a script that simply votes the EC numbers of the nearest retrieved neighbors; that aggregate tie is a cancellation of a large gain on an adversarial split against a large loss on multi-label enzymes. The paper concludes that LLM reasoning over evidence acts as an arbiter of conflicting neighbors rather than a source of knowledge, and that no single-number leaderboard can expose this.","feed_headline":"Open-book LLM enzyme scores tie a plain neighbor-voting script","feed_subtitle":"The aggregate tie hides a 19-case gain on adversarial enzymes and a 19-case loss on multi-functional ones.","key_machinery":"The carrying mechanism is the four-lever diagnostic protocol, not a new model. A zero-shot free-generation baseline (B0) reproduces the reported near-zero failure; M1 walks the EC hierarchy level by level with valid candidates and an abstain option; M2 runs a budgeted agentic loop that decides which offline evidence channel to query and when to stop; M3 forces reasoning along the four biochemical axes of the EC number; M4 wraps any base reasoner with self-consistency and prefix voting. All methods share the same offline evidence cache, the same hierarchical scorer, and the same four test splits, and the LLM's contribution is isolated by comparing against a no-model neighbor-vote reference bu","core_discovery":"The paper's core claim is that an LLM's apparent enzyme-classification ability is, in the open-book regime, almost entirely a function of which homologs can be retrieved, not of what the model knows or how it reasons. In aggregate the best of nine settings (0.728 full-set L4 accuracy, five-model mean) is indistinguishable from neighbor voting (0.723). But splitting the data shows the tie is a cancellation: the model converts about 19 more adversarial price items than voting (0.499 vs 0.372), even exceeding an oracle that always picks the best single channel, and loses about 19 promiscuous multi-label items (0.848 vs 0.938) by committing to one branch too early when evidence multiplies. The n","pith_inferences":["I infer the same cancellation pattern may appear in other retrieval-augmented classification tasks: aggregate parity with a retrieval baseline can hide opposite-signed gains on hard subsets and losses on ambiguous or multi-label subsets, so diagnostic splits should become standard practice.","The paper's zero-net-contribution claim is anchored to one specific voting rule (top hits from the two sequence-homology channels, no merging). If a stronger LLM-free baseline—using structure or representation channels, or optimally merging hits—narrows or reverses the price-split gain, the 'arbiter, not knowledge' conclusion would need to be softened to 'arbiter relative to that reference.'","The 19-converted / 19-lost magnitudes rest on 148 and 209 samples and a single seed; the paper itself notes the intervals do not cover those per-split contrasts. A reader should treat the direction and cross-model sign consistency as the evidence, not the exact counts."],"forward_implications":["If the paper is right, a single aggregate accuracy number for enzyme classification is not merely coarse but actively misleading; systems should be compared separately on homology-graded, adversarial, and multi-label splits.","General LLMs with no weight updates can be lifted from near-zero closed-book accuracy to specialist-level accuracy on evidence-rich and adversarial enzyme splits, provided retrieval is offline and inductive.","Closed-book format repair is not universally helpful: hierarchical cascades and chain-of-thought help only models that are willing to commit, and hurt abstention-prone models by converting silence into errors.","On homology-graded splits the choice of method matters little (about 0.05 spread), while on misleading or plural evidence splits it matters about three times more, so retrieval strategy should be chosen for the expected evidence regime.","The concrete practical handle for low-homology cases is a conflict-aware retriever or a lightweight router over evidence channels, not a stronger reasoner."],"fun_headline_variants":["Open-book LLM enzyme tie with voting hides adversarial gain vs multi-loss","Enzyme LLM scores match neighbor voting; split shows reasoning arbitrates","LLM enzyme ability is homolog retrieval, not reasoning power","Aggregate tie in enzyme benchmark masks 19-case adversarial win and 19-case loss","Open-book enzyme LLM equals voting; cancellation obscures role of reasoning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that voting the EC numbers of the top sequence-homology neighbors, scored with the same any-label rule, is the right reference for what a non-reasoning retrieval baseline achieves; if a different or better-tuned voting rule—using additional evidence channels or smarter merging—erases the LLM's adversarial-split edge, the headline cancellation becomes an artifact of the chosen baseline rather than a property of LLM reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Open-book LLM enzyme tie with voting hides adversarial gain vs multi-loss","Enzyme LLM scores match neighbor voting; split shows reasoning arbitrates","LLM enzyme ability is homolog retrieval, not reasoning power","Aggregate tie in enzyme benchmark masks 19-case adversarial win and 19-case loss","Open-book enzyme LLM equals voting; cancellation obscures role of reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1497,"prompt_tokens":854,"completion_tokens":643,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":547}},"tokens_in":598,"tokens_out":643,"duration_ms":7934,"temperature":1.0,"reasoning_tokens":547,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:46:02.776730+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an LLM-free voting baseline that merges hits from all five evidence channels (or a learned top-1 channel selector) on the adversarial price split. If this baseline reaches or passes the 0.499 L4 accuracy achieved by the LLM agentic setting on that split, while keeping promiscuous accuracy near the 0.938 neighbor-vote level, then the 'arbiter, not knowledge' claim falls: the LLM's apparent gain would be recoverable by better retrieval without any reasoning. Alternatively, a paired bootstrap over per-sample differences between M2 and the neighbor vote on the 148 price items would show whethe","supporting_citations":[],"review_version":1}