{"id":"66fdbb36-d3a9-488c-92b2-7a5f6306bb13","arxiv_id":"1908.02065","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper introduces two edge-feature-aware graph pooling layers and reports improved MoleculeNet benchmark results on three of four datasets and state-of-the-art results on HIV.","lead":"This paper adds two graph-pooling layers that keep bond features when a molecule is coarse-grained, and tests them on four drug-discovery benchmarks. The models report better numbers than previously published ones on three of the four datasets, and the pooling layers speed up training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Highlighted pooling gains may be a selection artifact: the best pooling ratio per dataset is chosen after seeing test results, with no validation-based selection rule or multiple-comparison correction; HIV's pooling gain is within sampling noise.","rationale":"The reader's weakest assumption, comparability of published baselines, is valid, but I find a more directly load-bearing issue: the internal comparison that attributes gains to pooling is itself subject to selection bias. Section 3 describes hyperband tuning for stack number and channel dimensions while holding the pooling ratio fixed, then Table 1 lists results for ten pooling ratios per dataset without stating that the ratio was chosen on validation. Because the paper's contribution is the pooling layer, the claim that pooling improves three of four tasks must survive a validation-based selection rule. The HIV result illustrates the problem: the highlighted improvement over no-pooling is within sampling noise. This does not overturn the paper, since the methodological idea is novel and the no-pooling versus pooling comparison is informative if the selection protocol is clarified and code is released. The reader already gave a conditional verdict, and this concern reinforces that conditionality rather than moving the verdict, so I recommend UNCHANGED.","tokens_in":10972,"tokens_out":5968,"duration_ms":67275,"concrete_test":"Run one prespecified protocol: before any further experiments, commit to using only the validation split to choose pooling type, keep ratio, and architecture from the candidate set in Table 4, then evaluate the selected configuration once on the held-out test split, repeating over several scaffold validation/test folds for BBBP and HIV and random folds for ESOL and Lipophilicity. If pooling does not beat no-pooling under this selection-on-validation rule, the pooling-benefit claim is unsupported; separately re-run at least one published baseline, such as Weave or MPNN from DeepChem, under identical 80/10/10 scaffold splits to settle the comparability question.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the two new edge-feature pooling layers are the cause of benchmark gains is not protected by a prespecified model-selection rule. Section 3 says hyperband tunes the number of stacks and channel dimensions while the pooling keep ratio is fixed, and Table 1 then reports results for ten pooling configurations per dataset. The text highlights whichever ratio works best (e.g., SimplePooling (0.8) for BBBP, CoarseGrainPooling (0.9) for Lipophilicity), but nothing states that the pooling ratio was chosen on the validation set. With only 5 runs per configuration, selecting the best of many configurations per dataset inflates apparent improvements. For HIV, the highlighted gain is not statistically robust: SimplePooling (0.6) AUC 0.840±0.019 versus NoPool 0.825±0.008 gives a two-sample t≈1.6 (p≈0.14), so the claimed pooling benefit on that dataset can be explained by chance. This selection issue is especially load-bearing because the internal no-pooling versus pooling comparison is the only controlled evidence that the pooling layers themselves matter. The comparison to literature baselines has the additional uncontrolled element noted by the reader: published numbers may use different splits, tuning budgets, or preprocessing, and no code is provided to re-run baselines under the authors' 80/10/10 scaffold-split protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes sparse hierarchical graph pooling layers for graphs with edge features, building on the Graph U-Net idea of selecting top-scoring nodes and adding edge-feature reconstruction when dropped nodes are removed. Two pooling variants are introduced: SimplePooling, which sums edge features along connecting paths, and CoarseGrainPooling, which learns effective edge features with neural networks. The architecture uses a dual-message graph-convolutional layer that updates both node and edge features. Experiments are reported on four MoleculeNet datasets (ESOL, Lipophilicity, BBBP, HIV), comparing pooled and unpooled versions of the proposed model and against literature numbers. The paper claims significant outperformance on three datasets, state-of-the-art on the fourth, and training speedups; a supplementary section applies the model to a photovoltaic materials dataset (CEP-2017).","tokens_in":11303,"tokens_out":2656,"duration_ms":27676,"significance":"If the stated results are taken at face value, the contribution is practically relevant: it extends sparse hierarchical pooling to edge-featured molecular graphs, reports speedups from pooling, and introduces a plausible coarse-graining mechanism for chemical bonds. The authors include a no-pooling baseline within their own architecture, which is a useful internal control, and the supplementary material provides additional results on a new dataset. However, the significance of the headline claims is not yet established because the internal pooling comparison is confounded by architecture changes and the literature comparison is uncontrolled. The core ideas are promising and the conceptual motivation is sound, but the empirical evidence as presented does not support the strongest conclusions without revision.","major_comments":[{"comment":"The pooling-versus-no-pooling comparison is confounded by differing architectures. For each dataset, the NoPooling row uses one specific stack count and channel configuration, while each pooling ratio uses different node/edge channel dimensions and sometimes a different number of stacks (e.g., ESOL NoPooling uses [128,128] with two stacks, while SimplePooling (0.7) uses [512,512,512] with three stacks; BBBP NoPooling uses [128,128] while SimplePooling (0.9) uses [128,128,128]). Consequently, differences in performance between pooling and no pooling may reflect changes in model capacity or depth rather than the pooling mechanism itself. To support the causal claim that pooling improves or preserves performance, the authors should run a controlled comparison that fixes the number of stacks and channel dimensions and varies only the pooling layer and keep ratio.","section":"Section 3, Table 4"},{"comment":"The pooling keep ratio appears to have been selected after seeing test results, not on the validation set. The text states that hyperband tunes the number of stacks and channel dimensions while the keep ratio is fixed, but it does not state that the best-performing ratio for each dataset was chosen on the validation split. Table 1 reports ten pooling configurations per dataset, and the narrative highlights the best one per dataset (e.g., SimplePooling (0.8) for BBBP, SimplePooling (0.6) for HIV). Without a pre-specified model-selection rule or a multiple-comparison correction, the apparent pooling gains are inflated. This is not merely a statistical nicety: for HIV, SimplePooling (0.6) gives ROC-AUC 0.840±0.019 versus NoPool 0.825±0.008, and a two-sample t-test gives t≈1.6 (p≈0.14), so the reported pooling benefit on HIV is within sampling noise. The authors should either select the ratio on validation data, apply an appropriate correction, or report the selection procedure explicitly.","section":"Section 3, Table 1"},{"comment":"The comparison to literature benchmarks is uncontrolled. Baseline numbers from Wu et al. and other cited papers were not re-run under the authors' protocol (80/10/10 split, scaffold split for classification, same hyperparameter tuning method), and the paper only states that classification splits follow DeepChem's scaffold split. If the literature baselines used different train/validation/test fractions, different splits, or different tuning budgets, the reported outperformance could be an evaluation artifact. The claim of \"significantly outperform[ing] previous benchmarks\" is therefore not supported by the evidence as presented. I recommend either re-running key baselines (e.g., GC, Weave, MPNN, or the strongest cited model per dataset) under the authors' exact protocol, or substantially softening the claims and explicitly labeling the comparison as indicative rather than controlled.","section":"Section 3, Table 1"}],"minor_comments":[{"comment":"The word \"significantly\" is used repeatedly without any statistical test or definition; please either define the significance criterion or replace the word with descriptive language such as \"numerically better.\"","section":"Abstract and Section 3"},{"comment":"There is a typo: \"drop the the ones linked to the dropped nodes\" should read \"drop the ones linked to the dropped nodes.\"","section":"Section 2.2, Figure 1 caption"},{"comment":"The speed-up table reports a single number per ratio without error bars or details on how many runs were measured; adding variance and the hardware/software context would help interpretability.","section":"Section 3, Table 2"},{"comment":"Some entries in Table 3 have anomalously large standard deviations (e.g., RMSE 0.179±0.097 for ratio 0.7), which suggests possible outliers or implementation instability; please comment on these entries or explain the source of the variance.","section":"Appendix A.1, Table 3"},{"comment":"No code or model configuration files are provided, which prevents reproducibility; releasing the implementation and exact hyperparameter search details would strengthen the paper significantly.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style contribution with a useful idea, but the central empirical claims are not yet protected. The most serious issue is that the internal pooling comparison varies architecture jointly with pooling, and the reported pooling gains appear to include test-set selection of the keep ratio. The HIV result is not statistically significant. All of these are fixable with additional controlled experiments and a clearer model-selection narrative, so I do not recommend rejection, but the claims need substantial revision before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: the two pooling layers for edge-featured graphs are a genuine extension of sparse pooling, and the paper deserves a serious referee. But the headline benchmark claims are not fully supported by the evidence as presented. The pooling ratio is chosen per dataset from ten configurations with no stated validation-based selection rule, and on HIV the best pooling gain is within sampling noise.\n\nWhat is new: Gao and Ji's Graph U-Net and Cangea et al.'s sparse pooling assume graphs without edge features. This paper extends sparse pooling to edge-featured graphs, defining a simple sum-based pooling layer and a neural-network coarse-graining layer that builds effective edge features during coarsening. That is a real, useful contribution, and the paper explains the operations clearly. The dual-message convolution is a modest variation of existing message passing, but it is not the main selling point. The speedup numbers in Table 2 are a practical plus.\n\nThe best evidence is the internal comparison between no pooling and pooling using the same architecture and tuning protocol. On BBBP and Lipophilicity, pooling improves the mean metrics in a direction consistent across several ratios. That suggests pooling can help, though the size of the benefit is less certain.\n\nSoft spots: the evaluation table reports ten pooling configurations per dataset (five ratios times two layer types) and highlights whichever performs best, without saying the ratio was selected on the validation set. With five runs per configuration, that is an uncontrolled multiple comparison. On HIV, for example, SimplePooling (0.6) gives AUC 0.840±0.019 vs 0.825±0.008 for NoPool; a two-sample t-test gives p≈0.14, so that gain is plausibly chance. The same selection issue inflates the BBBP and Lipophilicity gains, although the effect sizes are larger there.\n\nThe comparisons to literature baselines are also uncontrolled: those numbers come from different papers with different splits, tuning budgets, and preprocessing. The paper says it follows Wu et al. for scaffold splits but does not re-run any baseline under its own protocol. Calling results \"significantly better\" than published numbers is not supportable without re-running the baselines. No code is provided, which makes verification harder.\n\nWho this is for: anyone working on graph pooling or molecular property prediction with edge features. The method is clearly presented and the authors are honest about dataset-dependent behavior, but the evaluation needs a prespecified pooling-ratio selection rule and ideally a re-run of baselines. I would send this to referees rather than desk-reject, with expectations of major revisions.","headline":"Genuinely new edge-feature pooling layers, but the benchmark claims rest on unguarded model selection and uncontrolled baseline comparisons.","tokens_in":11776,"tokens_out":4215,"would_cite":true,"duration_ms":43179,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces edge-aware sparse pooling layers for molecular graphs and reports benchmark improvements on three of four drug-discovery datasets, with best-in-class results on the fourth.","keywords":["graph neural networks","pooling","molecular graphs","edge features","hierarchical representation learning","drug discovery","MoleculeNet","sparse graphs"],"falsifier":"Rerun every baseline model from Table 1 under the exact protocol used here (same scaffold splits, same 80/10/10 fractions, same hyperband search budget, same number of repeats) and re-measure the test metrics; if the reported margins shrink to within one standard deviation, the outperformance claim collapses. As a narrower check, train the no-pooling and pooled variants with identical hyperparameters rather than retuning per variant to see whether the pooling layer, and not a better hyperparameter draw, is the cause of the gain.","tokens_in":10827,"feed_emoji":"🧪","tokens_out":10927,"duration_ms":106579,"temperature":0.7,"pith_summary":"Earlier sparse pooling methods for graph neural networks coarsen a graph by dropping low-scoring nodes, but they assume edges carry no features; that assumption breaks for molecules, where bond type matters. This paper closes the gap with two pooling layers that create new 'effective' edges—with features recomputed from the contracted paths—when a graph is coarsened. The resulting models, built on a dual-message convolution that updates node and edge features separately, are evaluated on four standard molecular benchmarks: two regression tasks (ESOL, lipophilicity) and two classification tasks (BBBP, HIV). The authors report lower errors and higher AUCs than previously published numbers on three of the four datasets and state-of-the-art performance on the fourth, with pooling improving three of four tasks, leaving the fourth stable, and speeding up training by up to 70%.","feed_headline":"New graph pooling beats prior benchmarks on 3 of 4 molecular tasks","feed_subtitle":"Hierarchical edge-feature pooling lifts BBBP, HIV and lipophilicity scores, stabilises ESOL, and speeds training up to 70%.","key_machinery":"The machinery is the top-k sparse pooling layer extended to edge-featured graphs. Each node is scored by a learned projection vector $p^{(k)}$; the $K$ highest-scoring nodes are kept, and the kept nodes are rewired by contracting dropped nodes into effective edges, so the graph stays sparse and the selection stays differentiable through a gating $a_i^{(k)} = \\tilde a_i^{(k)} \\tanh(y_i^{(k)})$. The paper's contribution is in how the new effective edge features are computed: SimplePooling sums the edge-feature vectors along all contracted paths, while CoarseGrainPooling passes the dropped node's features, the adjoining edge features, and the kept edges' features through fully connected networks. This preserves bond-type information at every level of the hierarchy, and the accompanying dual-message convolution ensures edge features are updated before each pooling step.","core_discovery":"The central claim is that edge-featured molecular graphs can be hierarchically coarsened without discarding bond information, and that doing so improves both accuracy and speed. The paper defines two pooling layers for this purpose: SimplePooling, which sums the edge-feature vectors along every path contracted when a node is dropped (merging overlaps), and CoarseGrainPooling, which uses two fully connected networks to compute effective edge features—one for new edges that pass through dropped nodes and one for surviving edges. Combined with a dual-message graph-convolutional layer that updates node features from neighbour node features plus edge features, and updates edge features from the two incident node features, the models outperform previously reported numbers on ESOL, lipophilicity, and BBBP, and reach the best ROC-AUC the paper lists on HIV (0.840 ± 0.019 with simple pooling at keep ratio 0.6). Pooling improves performance on BBBP, HIV, and lipophilicity, leaves ESOL stable, and cuts HIV training time by up to 70%.","pith_inferences":["Editorial inference: the keep ratio is fixed per task and only the other hyperparameters are tuned, so treating the keep ratio itself as tunable could change which of the two pooling layers wins on a given dataset.","Editorial inference: the two edge-aggregation schemes are two points on a spectrum; a natural testable extension is attention-based or message-passing aggregation over contracted paths, which the paper does not explore.","Editorial inference: if the speedup is driven by graph size, the 70% figure on small drug-like molecules likely undersells the benefit for larger graphs; the supplementary CEP-2017 results, while not central, are consistent with pooling compressing molecules with little accuracy loss."],"forward_implications":["Bond information survives coarse-graining: even after several pooling steps, effective edges encode the bonds that were contracted, so predictions remain sensitive to chemistry at all scales.","On the paper's protocol, the best results are an RMSE of 0.410 ± 0.023 on ESOL, an RMSE of 0.517 ± 0.005 on lipophilicity, a ROC-AUC of 0.869 ± 0.010 on BBBP, and a ROC-AUC of 0.840 ± 0.019 on HIV; these are the values the authors use to claim outperformance and state-of-the-art status.","Pooling also pays in compute: simple pooling on HIV shortens elapsed training time by 16% at keep ratio 0.9 up to 70% at keep ratio 0.5, while memory requirements drop because the graph shrinks.","The authors expect the benefit to grow for larger graphs, such as protein structures, where a sequence of pooling steps can produce a deeper hierarchy than small drug-like molecules allow."],"supporting_citations":[{"why":"Supplies the four benchmark datasets, the 80/10/10 scaffold-split protocol, and most of the prior baseline numbers in Table 1.","marker":"[33]"},{"why":"Introduced the sparse top-k node-dropping pooling mechanism that the two new edge-feature pooling layers extend.","marker":"[14]"},{"why":"Defines the message-passing framework generalised by the dual-message graph-convolutional layer with node and edge updates.","marker":"[15]"},{"why":"Provides the additional baseline values marked 'a' in Table 1 and a comparison point for molecular property prediction with edge information.","marker":"[11]"},{"why":"Describes the hyperband algorithm used to tune stack depth and channel dimensions, which the reported comparisons depend on.","marker":"[21]"}],"fun_headline_variants":["Edge-aware graph pooling outperforms on 3 of 4 molecular benchmarks","Hierarchical pooling with edge features boosts three molecular benchmarks","New graph pooling handles bonds, wins on 3 of 4 tasks, speeds training","Pooling that respects bonds: state-of-the-art HIV, better on 3 tasks","Edge-feature pooling: better scores on 3/4 benchmarks and 70% faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison assumes the previously published baseline numbers in Table 1 were produced under the same protocol—same 80/10/10 scaffold split, same evaluation repeats, and comparable hyperparameter tuning; if the baselines were tuned less or evaluated differently, the reported margins could be artifacts of the comparison rather than properties of the new pooling layers.","fun_headline_variants_meta":{"raw":{"variants":["Edge-aware graph pooling outperforms on 3 of 4 molecular benchmarks","Hierarchical pooling with edge features boosts three molecular benchmarks","New graph pooling handles bonds, wins on 3 of 4 tasks, speeds training","Pooling that respects bonds: state-of-the-art HIV, better on 3 tasks","Edge-feature pooling: better scores on 3/4 benchmarks and 70% faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000888,"raw_usage":{"total_tokens":3804,"prompt_tokens":887,"completion_tokens":2917,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2815}},"tokens_in":503,"tokens_out":2917,"duration_ms":19128,"temperature":1.0,"reasoning_tokens":2815,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:54:31.553082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun every baseline model from Table 1 under the exact protocol used here (same scaffold splits, same 80/10/10 fractions, same hyperband search budget, same number of repeats) and re-measure the test metrics; if the reported margins shrink to within one standard deviation, the outperformance claim collapses. As a narrower check, train the no-pooling and pooled variants with identical hyperparameters rather than retuning per variant to see whether the pooling layer, and not a better hyperparameter draw, is the cause of the gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduced the sparse top-k node-dropping pooling mechanism that the two new edge-feature pooling layers extend."},{"cited_title":"Feinberg, Debnil Sur, Zhenqin Wu, Brooke E","cited_arxiv_id":null,"evidence_quote":"Provides the additional baseline values marked 'a' in Table 1 and a comparison point for molecular property prediction with edge information."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the hyperband algorithm used to tune stack depth and channel dimensions, which the reported comparisons depend on."}],"review_version":1}