{"id":"40b2f1dc-4f7e-448d-8399-26921a190834","arxiv_id":"2412.09030","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A graph transformer that builds a hierarchical graph of atoms and rings, using cross-attention on ring connections, improves power conversion efficiency prediction on the CEPDB dataset by 22.77% relative MAE.","lead":"RingFormer adds ring-level structure to graph transformers and reports lower prediction errors for organic solar cell properties across five datasets. On the large CEPDB set it beats the nearest baseline by about 23% relative error, though the gains are smaller and less certain on experimental datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline improvement may depend on undertuned baselines: all deep baselines share RingFormer's fixed depth/width, so the 22.77% CEPDB margin needs a per-model hyperparameter check.","rationale":"RingFormer's headline claim is a 22.77% relative improvement on CEPDB. For this to be a genuine result, the baselines must be competitive. The Appendix states that all deep baselines are trained with the same layer count, hidden dimension, and head count as RingFormer, and only 'other hyperparameters' are taken from original papers. This is a classic setting in which a new model can win by giving baselines suboptimal capacity. The paper does not report per-baseline validation tuning curves or a hyperparameter search budget. On CEPDB, the difference between RingFormer (0.189) and GraphViT (0.244) is large, but without tuned baselines we cannot distinguish a real architectural advantage from a capacity mismatch. This concern is testable and does not require reimplementation of RingFormer. The ring-extraction ambiguity identified by the reader is real but secondary: the released code pins down one implementation, and the empirical claim is about that implementation. In contrast, baseline fairness directly affects the central comparison. I therefore recommend keeping the verdict CONDITIONAL: the paper should be accepted only if the authors can show that per-model tuned baselines do not erase the margin. This matches the reader's overall conditional verdict, hence UNCHANGED.","tokens_in":17780,"tokens_out":5413,"duration_ms":49377,"concrete_test":"On CEPDB, run a hyperparameter search for each deep baseline (especially GraphViT, GPS, GINE, and O-GNN) over layer count in {2,4,8,12}, hidden dim in {128,256,512}, learning rate in {1e-4,5e-4,1e-3}, and dropout in {0,0.1,0.2}, using the same scaffold split and 5 random seeds. Report the best validation MAE for each baseline and the corresponding test MAE. If the best-tuned GraphViT or GPS test MAE comes within 5% of RingFormer's 0.189 (i.e., approximately 0.198 or lower), the claimed 22.77% improvement is not robust; if all tuned baselines remain above 0.22, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RingFormer's central claim is that it 'consistently outperforms existing methods' with a 22.77% relative improvement on CEPDB (test MAE 0.189 vs 0.244, Table 2). The experimental protocol in the Appendix ('Implementation Details') states that for deep baselines 'we fix the number of layers as 8, hidden dimension as 512, and the number of attention heads as 4, which is the same as RingFormer.' This imposes RingFormer's architecture capacity on all baselines without per-model hyperparameter optimization. For models like GINE, AttentiveFP, TopKPool, SAGPool, Graphormer, and GPS, depth and width strongly affect performance; the chosen values may be suboptimal for those architectures. GraphViT, the closest competitor on CEPDB, is implemented with 4 GNN + 4 transformer layers (also fixed, though different from RingFormer). No evidence is provided that each baseline's hyperparameters were tuned on the validation split. Consequently, the large CEPDB margin could partly reflect capacity mismatch rather than RingFormer's ring-level modeling. This is the load-bearing condition: if the closest baselines are tuned to their own optima, the 22.77% improvement may shrink or disappear. The reader noted overlapping confidence intervals on small datasets; the baseline-tuning issue is more directly tied to the headline CEPDB number.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RingFormer, a graph-transformer model for predicting organic solar cell (OSC) properties. It constructs a hierarchical graph with an atom-level graph, a ring-level graph of 'smallest rings' and their connections, and a bipartite inter-level graph connecting atoms to rings. Each RingFormer layer performs atom-level message passing, ring-level cross-attention with a virtual node, inter-level message passing, and a fusion MLP. The model is evaluated on five OSC datasets under scaffold splitting, with PCE as the main target, and is reported to outperform 11 baselines, including a 22.77% relative improvement on the large CEPDB dataset. Ablations and additional analyses of ring-system complexity are provided.","tokens_in":18100,"tokens_out":5413,"duration_ms":47683,"significance":"If the reported gains are robust, RingFormer is a meaningful advance for OSC property prediction: it is among the first graph transformers to explicitly model ring systems and their interconnections, it is evaluated on a very large DFT-computed dataset (CEPDB), and the authors release code. The multi-task results on CEPDB and the layer/backbone robustness checks are useful. The significance is tempered, however, by the fixed hyperparameters imposed on all deep baselines and by unspecified ring-extraction details, so the magnitude of the headline improvement is not yet fully established.","major_comments":[{"comment":"The protocol fixes the number of layers to 8, hidden dimension to 512, and attention heads to 4 for all deep baselines except GraphViT, rather than tuning each model on the validation split. This is load-bearing for the headline CEPDB result (Table 2), since a 22.77% relative improvement over an undertuned GraphViT may reflect capacity mismatch rather than the ring-level representation. Please report per-model hyperparameter selection on the validation set, or compare against the best numbers reported in the original papers, and detail the search ranges.","section":"Appendix: Implementation Details"},{"comment":"The 'smallest ring' definition ('no proper subset of its nodes can form a smaller ring') does not specify a unique set of rings for fused or bridged polycycles, and the paper does not name the algorithm used (e.g., RDKit's SSSR or GetSymmSSSR). Because the ring-level graph and all subsequent modules depend on this set, the representation is not uniquely determined by the molecular graph as presented. Specify the exact ring-perception routine and discuss how non-uniqueness is handled.","section":"Ring-level Graph Construction"},{"comment":"These tables report single MAE values, although the experimental setup section states that five seeds are run and mean ± std are reported. Several ablation comparisons are close (e.g., Table 5, Cross-attention vs. without virtual node on CEPDB is 0.1886 vs 0.1860; Table 4, G\\GI vs G on HOPV is 1.497 vs 1.477). Without variance or significance tests, the claims that all graph levels and the cross-attention module are necessary are not supported. Please report standard deviations and, where helpful, paired tests across seeds.","section":"Model Analysis, Tables 4, 5, 7, 8"},{"comment":"The claim that RingFormer 'consistently outperforms existing methods' is stronger than Table 2 supports: on PFD, RingFormer's MAE (1.776) is slightly worse than MACCS (1.770), and on HOPV its mean is tied with GraphViT within one standard deviation (1.477±0.021 vs 1.479±0.061). Please qualify the claim, e.g., 'best or second-best on all datasets, and best on the large-scale CEPDB dataset.'","section":"Overall Performance / Abstract"}],"minor_comments":[{"comment":"There are typos: 'firsts constructs' should be 'first constructs', and 'formulaiton' in the problem formulation section should be 'formulation'.","section":"The RingFormer Method"},{"comment":"Table 7 uses dataset names 'PolymerFA', 'nNFA', 'pNFA', while the rest of the paper uses 'PFD', 'NFA', 'PD'; please use consistent naming throughout.","section":"Table 7"},{"comment":"One panel of Figure 4 is labeled 'PFA', which should be 'PFD' to match the dataset name used elsewhere.","section":"Figure 4"},{"comment":"The introduction calls RingFormer 'the first graph transformer framework capturing ring systems'; since O-GNN also captures rings but is not a transformer, consider rephrasing to 'the first graph transformer that explicitly models ring systems and ring-ring connections'.","section":"Introduction"},{"comment":"The paper says 'we curate 5 OSC molecule datasets' for datasets that already exist in the literature; consider saying 'we compile and preprocess' to avoid implying the data were newly collected.","section":"Datasets and Evaluation Metrics"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the baseline-tuning protocol; if the authors cannot show per-model tuning, the CEPDB margin may shrink. I would ask for this before acceptance. The paper is otherwise within scope for an applied machine-learning venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: RingFormer is a clean, well-motivated graph transformer for OSC property prediction, and the CEPDB result is worth taking seriously. But the headline 22.77% margin is not yet established because the deep baselines were fixed to RingFormer's depth and width instead of tuned per model, and the smaller-dataset wins are often within noise.\n\nWhat's new: the paper builds an explicit three-level graph—atoms, rings, and atom-ring membership—and encodes the ring level with cross-attention where edge attributes enter keys and values, plus a virtual node for global mixing. That combination is not in O-GNN or motif-based baselines. The evaluation is real work: five datasets, 2.3M-molecule CEPDB, scaffold splits, repeated seeds with std on the main tables, and code is released. On CEPDB, 0.189 vs 0.244 is a big gap and the std is small. The multi-task CEPDB results are also consistently better, which lends some support.\n\nSoft spots, in order. First, the baseline tuning. The appendix says all deep baselines except GraphViT get 8 layers, hidden 512, 4 heads—same as RingFormer. For GINE, AttentiveFP, GPS, and Graphormer, that fixed capacity may be fine or far off; no validation tuning is reported. The CEPDB margin could partly be capacity mismatch. The stress-test note landed; this is the load-bearing issue. Second, the 'consistently outperforms' language is too strong. On HOPV, RingFormer and GraphViT overlap within std; on PFD, RingFormer is runner-up. The paper acknowledges this only in a parenthetical, but the abstract claims more. Third, the ablation tables have no error bars, so the component contributions in Tables 4, 5, and 7 are hard to judge. Fourth, 'smallest rings' is not tied to a specific algorithm; for fused ring systems the set of smallest rings can be non-unique, and RDKit's SSSR is the obvious choice. That is a reproducibility gap, fixable by naming the implementation. The label curation for PFD/NFA/PD—keeping max PCE for duplicate molecules—is defensible but should be flagged as a modeling choice.\n\nBottom line: the architecture is sensible and the large-scale result is promising, but the central performance claim needs a per-model hyperparameter check before I would trust the 22.77% number. This deserves refereeing, with revision rather than desk rejection.","headline":"Sensible architecture and a promising CEPDB result, but the headline 22.77% margin is not yet trustworthy because the deep baselines were tuned to RingFormer's capacity rather than their own.","tokens_in":18572,"tokens_out":2616,"would_cite":true,"duration_ms":26832,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RingFormer, a graph transformer that represents organic solar cell molecules at both atom and ring levels, predicts power conversion efficiency and related properties more accurately than existing fingerprint, GNN, and transformer…","keywords":["graph transformer","organic solar cells","molecular property prediction","ring systems","hierarchical graph","power conversion efficiency","cross-attention","message passing"],"falsifier":"Re-run the CEPDB PCE benchmark with the ring-level graph built from two different smallest-ring extraction algorithms (or from an exhaustive enumeration of all simple rings) on the same train/validation/test split; if the test MAE moves away from 0.189 or the ranking against baselines changes, the central claim depends on an arbitrary preprocessing choice rather than on the ring system itself.","tokens_in":17602,"feed_emoji":"☀️","tokens_out":7934,"duration_ms":70022,"temperature":0.7,"pith_summary":"RingFormer is a graph transformer for predicting properties of organic solar cell (OSC) molecules, with power conversion efficiency (PCE) as the main target. The paper's central claim is that a molecule should be read at two structural levels at once—its atoms and the rings that form its conjugated core—and that models which ignore the ring system are leaving out the feature that most determines OSC performance. On the large CEPDB dataset, RingFormer reports a test MAE of 0.189 for PCE, a 22.8% relative improvement over the best competing model at 0.244, and it also wins on three of four smaller experimental datasets. If this holds, cheap computational screening of candidate OSC molecules becomes more reliable, since the model captures exactly the kind of fused and non-fused ring architectures that synthetic chemists vary.","feed_headline":"Ring-aware model cuts organic-solar-cell prediction error by 22.8%","feed_subtitle":"It reads molecules as atom-plus-ring graphs and beats 11 baselines on five OSC datasets.","key_machinery":"The central object is the hierarchical OSC graph $\\mathcal{G} = \\{\\mathcal{G}_A, \\mathcal{G}_R, \\mathcal{G}_I\\}$. A molecule's smallest rings—closed loops with no proper subset forming a smaller ring—become ring nodes; rings are connected when they share atoms or are linked by a non-aromatic chain, and the edge attribute records how they connect. A bipartite inter-level graph joins each ring to its constituent atoms. The mechanism that does the main work is the ring-level cross-attention: for each ring node, the query is the node itself, the keys and values are formed by concatenating a neighboring ring's representation with the connection-type edge attribute, and attention is restricted to immediate ring neighbours plus a virtual molecule node that carries global information. This keeps edge semantics in the output representation and avoids quadratic all-pairs attention, which is what makes the model both more expressive for ring systems and scalable to the 2.3-million-molecule CEPDB database.","core_discovery":"The paper claims that OSC property prediction is substantially improved by encoding a molecule as a hierarchical graph with three levels: the atom-level bonding graph, a ring-level graph whose nodes are the molecule's smallest rings and whose edges encode fusions or linking chains between rings, and an inter-level bipartite graph that ties each ring to its constituent atoms. RingFormer layers then combine local message passing on the atom graph, a ring-level cross-attention that folds edge attributes into keys and values and uses a virtual molecule node for global communication, and inter-level message passing, fusing hierarchies before pooling atom and ring representations separately into the final descriptor. The claim is that this design captures ring systems—especially interconnections between rings—that fingerprints, ordinary GNNs, and generic graph transformers miss, and the reported experiments support the claim with consistent wins across five datasets and a 22.77% relative improvement on CEPDB.","pith_inferences":["The ring-level representation is likely transferable to other chemistry domains, such as predicting properties of drug-like molecules with fused aromatic cores, but the paper does not test this.","Because the paper does not specify the smallest-ring extraction algorithm, a reproducibility check would be to rebuild the ring-level graphs with two different ring-set algorithms on fused-ring molecules; if the reported 22.8% gain depends on the algorithm, the representation is not uniquely defined by the molecule.","The largest gain appears on the DFT-computed CEPDB database; experimental datasets are smaller and noisier, so the method's practical advantage for screening real synthesized molecules could be smaller than the headline number.","The combination of hierarchical representation with a linear-cost virtual-node attention suggests a natural test in active learning: use the model's uncertainty to prioritise which candidate OSC molecules to synthesize."],"forward_implications":["If the reported gains are real, the same hierarchical atom-plus-ring encoding should improve prediction of other electronic properties of conjugated molecules, and the multi-task results on CEPDB for HOMO, LUMO, band gap, $V_{oc}$, and $J_{sc}$ already point that way.","Molecules with larger fused ring systems should benefit the most; the paper shows that RingFormer's relative improvement over baselines grows with the number of rings.","Because the ring-level cross-attention only looks at ring neighbours plus a virtual node, it should scale to very large molecular libraries where full graph transformers become too slow.","The architecture is not tied to one GNN backbone; the paper reports that swapping GINE for GraphSAGE or GatedGCN keeps RingFormer ahead of the best deep competitor on four of five datasets.","The two-level pooling scheme gives a principled replacement for fingerprint-plus-random-forest pipelines in OSC screening, since it works on raw molecular graphs and still beats ECFP on the experimental datasets."],"supporting_citations":[{"why":"Defines the OSC property prediction task and reports that conventional GNNs underperform fingerprint methods, the gap RingFormer aims to close.","marker":"Eibeck et al. 2021"},{"why":"Introduces ring priors into GNNs (O-GNN) but does not model ring interconnections, the closest predecessor RingFormer extends.","marker":"Zhu et al. 2023"},{"why":"Supplies the theoretical motivation that GNNs cannot reliably count substructures such as rings.","marker":"Chen et al. 2020"},{"why":"Provides the depth-versus-width analysis cited for GNNs' weakness on high-order structure.","marker":"Loukas 2019"},{"why":"GPS is the state-of-the-art graph transformer baseline that also combines message passing with global attention.","marker":"Rampášek et al. 2022"},{"why":"Localized neighborhood attention with a virtual node, the basis for RingFormer's efficient ring-level cross-attention.","marker":"Shirzad et al. 2023"},{"why":"Source of the CEPDB dataset of about 2.3 million DFT-computed OSC molecules used for the headline result.","marker":"Hachmann et al. 2011"},{"why":"Provides the NFA and PD experimental datasets and the analysis of machine learning for OSC materials.","marker":"Miyake and Saeki 2021"},{"why":"Supplies the GINE backbone used for atom-level and inter-level message passing.","marker":"Hu et al. 2019"},{"why":"Graphormer baseline and the degree-based position encoding adopted for ring nodes.","marker":"Ying et al. 2021"}],"fun_headline_variants":["RingFormer boosts organic solar cell property prediction by 22.8%","Ring-aware transformer beats baselines in organic solar cell property prediction","Hierarchical ring graph transformer cuts OSC prediction error by 22.8%","Organic solar cell property prediction improved via ring-enhanced transformer","RingFormer improves OSC property prediction by 22.8%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a molecule has a well-defined set of 'smallest rings' that can be extracted consistently; for fused ring systems the set of smallest rings can be algorithm-dependent, and if the extraction is not unique, the entire hierarchical graph is not uniquely determined by the molecule.","fun_headline_variants_meta":{"raw":{"variants":["RingFormer boosts organic solar cell property prediction by 22.8%","Ring-aware transformer beats baselines in organic solar cell property prediction","Hierarchical ring graph transformer cuts OSC prediction error by 22.8%","Organic solar cell property prediction improved via ring-enhanced transformer","RingFormer improves OSC property prediction by 22.8%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3005,"prompt_tokens":921,"completion_tokens":2084,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":1995}},"tokens_in":537,"tokens_out":2084,"duration_ms":16176,"temperature":1.0,"reasoning_tokens":1995,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:51.255484+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the CEPDB PCE benchmark with the ring-level graph built from two different smallest-ring extraction algorithms (or from an exhaustive enumeration of all simple rings) on the same train/validation/test split; if the test MAE moves away from 0.189 or the ranking against baselines changes, the central claim depends on an arbitrary preprocessing choice rather than on the ring system itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the OSC property prediction task and reports that conventional GNNs underperform fingerprint methods, the gap RingFormer aims to close."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces ring priors into GNNs (O-GNN) but does not model ring interconnections, the closest predecessor RingFormer extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical motivation that GNNs cannot reliably count substructures such as rings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the depth-versus-width analysis cited for GNNs' weakness on high-order structure."},{"cited_title":"J.; and Sinop, A","cited_arxiv_id":null,"evidence_quote":"Localized neighborhood attention with a virtual node, the basis for RingFormer's efficient ring-level cross-attention."},{"cited_title":"S.; Gold-Parker, A.; Vogt, L.; Brockway, A","cited_arxiv_id":null,"evidence_quote":"Source of the CEPDB dataset of about 2.3 million DFT-computed OSC molecules used for the headline result."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the NFA and PD experimental datasets and the analysis of machine learning for OSC materials."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Graphormer baseline and the degree-based position encoding adopted for ring nodes."}],"review_version":1}