{"id":"1c5516e3-3bf6-4a0e-9472-69d19883f2b1","arxiv_id":"2411.13688","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A systematic comparison shows ECFP fingerprints still beat GNNs at standard QSAR prediction, while GIN features and a new frequency-based fingerprint method (Sort & Slice) improve activity-cliff and property prediction.","lead":"This PhD thesis compares graph neural networks, extended-connectivity fingerprints, and physicochemical descriptors for predicting molecular activity and activity cliffs, and introduces a simple frequency-based alternative to fingerprint hashing called Sort & Slice. It finds that classical fingerprints still win at ordinary property prediction while GNNs can be more useful for activity-cliff classification.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sort & Slice's entropy-optimality theorem assumes training-set substructure frequency tracks target informativeness; rare but decisive substructures are discarded by construction, so the 'robustly outperforms hashing' claim is not established for tasks where rare substructures carry the signal.","rationale":"The reader's weakest assumption identifies exactly the same point: the Sort & Slice optimality proof assumes substructure frequency tracks informativeness about the target and that training frequencies generalize to the test distribution. My stress-test confirms this is the most load-bearing condition. The empirical demonstration in the thesis is extensive, but it covers only a limited set of datasets and settings; those datasets may be ones where common substructures happen to correlate with the target. The synthetic benchmark I propose would directly test whether the frequency-informativeness link is necessary or whether hashing's preservation of rare substructures gives an advantage when rare substructures are decisive. Because the proof is not fully visible in the provided text, the conditional verdict is appropriate: the claim is plausible but not yet established. I do not see a reason to reject or to change the reader's verdict; rather, the concern strengthens the conditions already attached to acceptance. No ad hominem is intended: the issue is purely the argument's dependence on an unverified statistical premise.","tokens_in":49893,"tokens_out":3281,"duration_ms":34466,"concrete_test":"Construct a synthetic benchmark where the target is a deterministic function of one rare substructure present in about 1% of the training set, plus independent noise, while many frequent substructures are irrelevant. Run the paper's exact Sort & Slice versus hashing protocol (same downstream RF/MLP regressors, same cross-validation splits, same hyperparameter tuning) with the same ECFP radii and fingerprint lengths. If hashing attains lower MAE than Sort & Slice, the frequency-informativeness assumption is falsified; if Sort & Slice still wins, the concern is mitigated. As a complementary analytical check, re-derive the theorem and verify whether the stated assumptions actually imply that the top-k most frequent substructures maximize entropy reduction, rather than merely maximizing coverage of the molecular distribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central new claim is that Sort & Slice, which keeps the most frequent circular substructures from the training set, 'robustly leads to higher predictive performance than hash-based folding' and is entropy-optimal under 'reasonable theoretical assumptions' (abstract; Section 5.2.2.2). The proof must therefore rely on a link between substructure frequency and informativeness about the target. That link is the load-bearing assumption. In QSAR data, rare substructures can be highly decisive: a single substituent present in a small fraction of compounds can change activity by orders of magnitude (as in the activity-cliff examples in Chapter 3). If the target depends on such a rare substructure, Sort & Slice will discard it when truncating to the most frequent substructures, while hashing preserves it subject to collision noise. Frequency is a property of the molecular distribution, not of the label; without a demonstrated monotonic relationship between frequency and mutual information with the target, the entropy-optimality theorem does not transfer to real property-prediction tasks. A second aspect of the same concern is stability: features are selected from the training fold, and if the substructure-frequency ranking is not stable across cross-validation folds, the selection itself becomes a source of overfitting. The published empirical chapters are careful, but Chapter 5 appears to be unpublished and no code or data are provided to check the robustness claim on distributions where rare substructures dominate. This concern is load-bearing because both the theoretical optimality and the empirical robustness claim collapse if the frequency-informativeness link fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript is a PhD thesis that investigates molecular featurisation methods for two tasks: quantitative structure-activity relationship (QSAR) prediction and activity-cliff (AC) prediction. Chapter 2 reviews physicochemical-descriptor vectors (PDVs), extended-connectivity fingerprints (ECFPs), and message-passing graph neural networks (GNNs), with an emphasis on graph isomorphism networks (GINs). Chapter 3 reports a computational study comparing nine combinations of featurisation and regression models on three pharmacological targets, and it introduces a pair-based data-splitting scheme for evaluating AC prediction. Chapter 4 proposes a twin neural network for AC and potency-direction (PD) classification, with proofs of built-in order-invariance and order-equivariance properties, and evaluates four versions of the model on a SARS-CoV-2 main protease data set. Chapter 5 introduces 'substructure pooling' as a general operation for vectorising structural fingerprints, proposes 'Sort & Slice' as an alternative to hashing that keeps the most frequent substructures, claims an entropy-optimality theorem under stated assumptions, and reports computational experiments suggesting that Sort & Slice outperforms hashing and other selection schemes for molecular property prediction. Chapter 6 outlines two future research directions.","tokens_in":50357,"tokens_out":6121,"duration_ms":92613,"significance":"If the Sort & Slice claim is correct, the manuscript identifies a simple, interpretable change to ECFP vectorisation that could replace hash-based folding in many molecular property prediction pipelines. The comparison against external baselines (hashing, standard QSAR models) rather than against the proposed models themselves is a methodological strength, and the symmetry proofs for the twin neural network (Propositions 4.1-4.3) are clean and machine-checkable in principle. The computational studies in the published chapters are careful in their use of cross-validation and hyperparameter optimisation. However, the central novelty of Chapter 5 is not yet supported to the standard claimed: the theoretical result relies on an unvalidated frequency-informativeness link, and the empirical robustness claim is not accompanied by code, full data, or significance tests. The featurisation comparisons in Chapter 3 also rest on a small number of targets and trials.","major_comments":[{"comment":"The entropy-optimality proof for Sort & Slice is conditional on the assumption that the frequency of a circular substructure in the training set tracks its informativeness about the target property. The manuscript states this only as 'reasonable theoretical assumptions' and does not provide a formal, testable condition under which the proof holds. The method truncates to the most frequent substructures, so it discards rare substructures by construction; the activity-cliff example in Figure 3.1 shows that a rare substituent change can alter pKi by almost three orders of magnitude. Thus, the claim that Sort & Slice 'robustly leads to higher predictive performance than hashing' is not established for distributions in which rare substructures carry the predictive signal. The theoretical result collapses to a frequency filter if this assumption fails. Please either state and prove a label-dependent guarantee or explicitly restrict the scope of the optimality and robustness claims.","section":"Chapter 5, Section 5.2.2.2; abstract"},{"comment":"The central empirical claim that Sort & Slice outperforms hashing 'across a large number of settings' is not supported by the information provided in the manuscript. No code or data are made available, the number of independent repetitions or seeds is not reported, and no statistical significance tests or confidence intervals are given for the comparisons in Figures 5.2-5.7. Table 5.2 lists hyperparameter ranges but not the per-model variability or the exact data-split protocol. For a claim of robust superiority, please provide the full experimental protocol, effect sizes with uncertainty, and a reproducibility package so that the comparison can be independently checked.","section":"Chapter 5, Section 5.3 and Figures 5.2-5.7"},{"comment":"The featurisation ranking (ECFPs best for QSAR, GINs best for AC classification) is based on only three pharmacological targets and six trials per model (2-fold cross-validation with three seeds), with no hypothesis tests. The error bars showing twice the standard deviation in Figures 3.7-3.12 indicate substantial variability relative to the reported differences, particularly for factor Xa in Figure 3.8. Statements such as 'ECFPs consistently deliver the best performance' and 'robust evidence' are stronger than the evidence supports. Please add significance tests (for example, paired bootstrap tests over the seeds) or additional datasets before making general claims about the relative merits of the featurisations.","section":"Chapter 3, Section 3.4 and Figures 3.7-3.12"},{"comment":"The conclusion that the twin architecture 'outperforms standard QSAR models at AC-prediction in a variety of scenarios' is based on a single data set (SARS-CoV-2 main protease). In addition, the evaluation compares the twin model only to QSAR baselines, not to the existing tailored AC-prediction methods cited in Section 3.2 (for example, references [51], [104], and [111]), even though the chapter criticises those methods for lacking appropriate baselines. Please either add at least one additional target and include existing AC-prediction baselines, or explicitly limit the conclusion to the data set and baselines actually studied.","section":"Chapter 4, Section 4.3.2"}],"minor_comments":[{"comment":"There are several typos: 'activites' in Section 4.1, 'compunds' in Section 4.2.3, 'preferrable' in Section 2.3, 'alond' in Section 2.2, and 'T able' in the caption of Table 2.1. A proofreading pass is needed.","section":"Throughout"},{"comment":"The manuscript states that Chapter 3 is 'the first study that investigates the capabilities of QSAR models to classify between ACs and non-ACs,' but Section 3.2 cites related work that indirectly evaluates QSAR models on cliffy compounds. Please clarify the precise novelty claim.","section":"Section 1.1"},{"comment":"The text mentions 'one other work [48]' that investigates a technique similar to Sort & Slice, but the full reference is not visible in the bibliography excerpt. Please ensure the citation is complete.","section":"Section 1.1"},{"comment":"The notation `Mdouble` is introduced without a formal definition. Please define it explicitly, as the subsequent argument about containing one orientation of each MMP depends on it.","section":"Section 4.2.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a thesis rather than a focused journal article, and the most novel part (Chapter 5) is unpublished. For a journal submission, I would recommend extracting the Sort & Slice material, making the theoretical assumptions explicit and testable, and strengthening the empirical evaluation with code/data release and significance testing. The published chapters are competent but their claims should be scaled to the evidence. The self-citation pattern is not problematic in this context, but the novelty claim in Chapter 3 should be checked against the cited prior literature."},"author_rebuttal":null,"desk_editor":null,"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This thesis argues that replacing hash-based folding with frequency-ranked substructure selection improves ECFPs and is entropy-optimal under stated assumptions.","keywords":["extended-connectivity fingerprints","substructure pooling","Sort & Slice","hashing","molecular property prediction","activity-cliff prediction","graph neural networks","QSAR"],"falsifier":"Construct a property-prediction benchmark whose target is determined almost entirely by a deliberately rare substructure present in under 1% of training molecules, then compare Sort & Slice with hashed ECFPs on that benchmark: if hashing wins, the frequency-informativeness premise fails, since Sort & Slice is designed to drop rare substructures.","tokens_in":49712,"feed_emoji":"🧪","tokens_out":7683,"duration_ms":77239,"temperature":0.7,"pith_summary":"At its core this thesis asks whether trainable graph neural networks actually beat classical molecular featurisations, and it answers with a qualified no for ordinary QSAR prediction while pointing to where each method wins. The central methodological contribution is Sort & Slice, a change to how extended-connectivity fingerprints (ECFPs) are built: instead of hashing every detected circular substructure into a fixed bit vector, sort the substructures by how often they occur in the training set and keep only the most frequent ones. The thesis proves that, under assumptions it states, this selection is entropy-optimal, and its experiments show it consistently outperforms hashing across data sets, regressors, splits, and ECFP hyperparameters. If true, a practically free modification improves both accuracy and interpretability of one of the most widely used molecular representations.","feed_headline":"Frequent-substructure fingerprints beat hashed ECFPs","feed_subtitle":"Keeping the most common chemical substructures instead of folding them into bits cuts collisions and lifts prediction accuracy.","key_machinery":"Substructure pooling is the general operation that maps a multiset of enumerated circular substructures, the unordered output of an ECFP-style enumeration, to a real-valued vector; hashing is one instance, and Sort & Slice is another. Sort & Slice ranks the substructures by training-set frequency and truncates to the most frequent ones. This frequency ranking is also the load-bearing mechanism of the proof: with the stated assumption that substructure frequency tracks informativeness, the top-frequency slice is shown to be entropy-optimal, meaning no other fixed-size selection of substructures can carry more Shannon information about the target. The thesis also develops a pair-based data-splitting scheme and a twin neural network whose max-pooling and odd-MLP design hard-code order-invariance for activity-cliff labels and order-equivariance for potency-direction labels.","core_discovery":"On the paper's own terms, the key discovery is that the hashing step in standard ECFP vectorisation is not a neutral technical detail: it is a lossy, collision-prone form of substructure pooling, and it can be replaced by a simpler frequency-ranked procedure that works better. Sort & Slice first sorts the enumerated circular substructures of a molecule according to their frequency in the training set and then keeps the most frequent substructures in the final vector, so each retained dimension corresponds to one concrete chemical substructure with no bit collisions. The accompanying mathematical argument shows that under the stated frequency-informativeness assumption, keeping the most frequent substructures is exactly the choice that maximises information about the target in an entropic sense. Empirically, the thesis reports that this simple pooling technique robustly beats hashing and two supervised substructure-selection schemes for molecular property prediction, with the advantage growing as the expected number of hash collisions increases. For the field's broader question, the thesis finds ECFPs still deliver the best QSAR predictions, while GIN-based features are competitive or better for activity-cliff classification.","pith_inferences":["The entropy-optimality result implies a testable design rule: choose the fingerprint dimension as the number of top-frequency substructures that cover most of the training-set mass, which could let practitioners set ECFP length without grid search.","Because the frequency ranking is task-agnostic, a natural extension is to make the selection differentiable, for instance with self-attention weights, so the model learns which frequency bands matter rather than assuming frequency equals informativeness.","The sharp activity-cliff sensitivity drop in the both-activities-unknown setting suggests that QSAR errors concentrate on the most informative pairs; a training loss that explicitly rewards sensitivity on predicted cliffs might improve models more than adding featurisation capacity.","The twin network's hard-coded symmetry properties transfer to other pairwise chemistry problems, such as predicting reaction outcomes or matched-molecular-pair transformations, where label reversal rules are known a priori."],"forward_implications":["Molecular-property pipelines can replace hashed ECFPs with Sort & Slice at no extra model cost and obtain higher predictive performance, with each retained bit tied to one explicit substructure.","The gap between Sort & Slice and hashing should widen when fingerprints are shorter or radii larger, because those settings raise the expected number of hash collisions.","For QSAR prediction, classical ECFP features remain at least as strong as the tested GNN features, so claims that message-passing GNNs categorically supersede fingerprints need to be conditioned on task and data.","For activity-cliff classification, GIN-based features and a twin network architecture can outperform repurposed QSAR baselines, giving practical baselines for future AC-prediction studies.","When the activity of one compound in a matched pair is known, QSAR models detect cliffs much better than when both activities are unknown, so cliff-prediction performance should be reported separately for these two settings."],"supporting_citations":[{"why":"Defines the ECFP algorithm and its hash-based folding of circular substructures, which Sort & Slice is proposed to replace.","marker":"[16]"},{"why":"The paper identifies this as the one other work investigating a technique similar to Sort & Slice, framing the novelty claim.","marker":"[48]"},{"why":"Provides the software implementation used to generate ECFPs and descriptors across all computational experiments.","marker":"[70]"}],"fun_headline_variants":["Drop hashing: keep top substructures for better predictions","Frequency-ranked substructures beat hashed ECFPs","Less hashing, more accuracy: Sort & Slice fingerprints","Substructure frequency beats hashing in molecular prediction","Hashed fingerprints get beaten by frequency-based selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof depends on training-set substructure frequency being a faithful proxy for how informative a substructure is about the target, and on those frequencies persisting at test time; if a rare substructure carries the signal, Sort & Slice discards exactly the feature the model needs.","fun_headline_variants_meta":{"raw":{"variants":["Drop hashing: keep top substructures for better predictions","Frequency-ranked substructures beat hashed ECFPs","Less hashing, more accuracy: Sort & Slice fingerprints","Substructure frequency beats hashing in molecular prediction","Hashed fingerprints get beaten by frequency-based selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1373,"prompt_tokens":1106,"completion_tokens":267,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":722,"completion_tokens_details":{"reasoning_tokens":190}},"tokens_in":722,"tokens_out":267,"duration_ms":3195,"temperature":1.0,"reasoning_tokens":190,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:59:44.715386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a property-prediction benchmark whose target is determined almost entirely by a deliberately rare substructure present in under 1% of training molecules, then compare Sort & Slice with hashed ECFPs on that benchmark: if hashing wins, the frequency-informativeness premise fails, since Sort & Slice is designed to drop rare substructures.","supporting_citations":[],"review_version":1}