{"id":"141a88d6-7a01-4c92-986c-f04a1ba80813","arxiv_id":"2504.19538","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Removing the two deepest interaction blocks from the JMP-L molecular foundation model reduces size by 32% and speeds inference by 1.3x while keeping downstream accuracy nearly unchanged.","lead":"This paper shows that pruning the two deepest interaction blocks of JMP-L, a large molecular property prediction model, cuts parameters by 32% and speeds up inference by 1.3x with little accuracy loss. The result suggests large molecular foundation models are over-parameterized and can be compressed for cheaper fine-tuning and screening.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed wall-clock fine-tuning budget confounds the claim that the 4-block JMP-L is comparably accurate: smaller models receive more optimization steps, so the 6-block baseline may simply be undertrained.","rationale":"The reader identified the fixed GPU budget as one of several concerns, and the GradCAM ordering reliability as the weakest assumption. I agree the GradCAM issue is real but secondary: Appendix C's layer-wise ablation shows only a local ordering mismatch (f4 vs f3 on rMD17), whereas the final two blocks still appear among the least important, so it does not directly undermine the specific 6-to-4-block pruning decision. The more load-bearing threat to the central claim is the fixed wall-clock fine-tuning budget, because it affects every downstream comparison used to support 'minimal performance drop' and 'over-parameterized.' A per-step or per-epoch fairness argument would not fix it either: the conclusion is about model capacity, so both models need to be trained to convergence or at least to matched optimization progress, with error bars. The code being public and the from-scratch comparison in Appendix B are genuine supporting evidence, but they do not settle the convergence confound. The paper is a useful empirical study, and the conditional verdict is appropriate; my concern reinforces that same conditionality rather than changing it.","tokens_in":15594,"tokens_out":3202,"duration_ms":36709,"concrete_test":"Fine-tune the 6-block teacher and the 4-block pruned model on the five downstream tasks with multiple random seeds using a convergence-based budget (e.g., train until validation MAE plateaus, or for 4–8 GPU-days, whichever comes first), then compare best test MAE with standard errors. If the 6-block model's best MAE is statistically indistinguishable from or worse than the 4-block model's, the over-parameterization claim survives; if it improves by more than the standard error, the headline 'minimal performance drop' is an artifact of the fixed 1-GPU-day budget rather than evidence of removable capacity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that removing two interaction blocks yields a minimal performance drop—is established in Section 4.3 by fine-tuning every model for a fixed wall-clock budget of 1 GPU-day on a V100 (2 GPU-days for QM9) and comparing test MAE. This protocol is biased toward smaller models: the 4-block model has fewer FLOPs per step, so in the same wall-clock window it performs many more optimizer steps than the 6-block model. Equality under this budget therefore demonstrates compute-efficiency per unit of wall-clock time, not that the extra capacity of the 6-block model is unnecessary. The paper's own convergence analysis (Section 4.4, Figure 4) shows the 6-block model retaining a noticeable advantage on MD22 at the 1-GPU-day cutoff and does not establish that it has plateaued; with additional budget it could pull ahead on other tasks as well. Because no error bars or multiple seeds are reported, the observed 'minimal drop' on 5 of the 40 JMP tasks cannot distinguish the pruning hypothesis from the undertrained-baseline hypothesis. The abstract's 'JMP-L is over-parameterized' conclusion thus rests on a comparison that is confounded with training budget.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the layer efficiency of JMP-L, the 160M-parameter GemNet-OC-based foundation model for molecular property prediction, and proposes block reduction as a compression strategy. The authors compute GradCAM relevance scores for the concatenated per-block features on the JMP pre-training distribution, find that the later interaction blocks receive the lowest scores, and construct reduced models by removing the last interaction blocks and truncating the first FinalMLP layer accordingly (with either a 'sliced' weight initialization or a randomly re-initialized alternative). Optionally, a brief knowledge-distillation phase is run before fine-tuning, transferring output, node, and edge features from the teacher. The headline result is that removing two of six interaction blocks reduces parameters by 32% (160M to 108M) and improves inference throughput by 1.3x while incurring only a minimal accuracy drop when pruned and full models are fine-tuned under the same wall-clock budget (1 GPU-day on a V100; 2 GPU-days for QM9) and evaluated on five targets: Aspirin (rMD17), U0 (QM9), solvated amino acids (SPICE), Ac-Ala3-NHMe (MD22), and band gap (QMOF). Additional experiments cover pre-training-time MAE on the four JMP upstream datasets (with and without KD), convergence curves within the fixed budget, inference benchmarks, a layer-wise ablation on rMD17 (Appendix C), a comparison with training from scratch, and an extension to EquiformerV2 in the appendix.","tokens_in":15785,"tokens_out":11553,"duration_ms":113789,"significance":"If the main claim holds, the paper delivers a practically useful result: a 108M-parameter JMP variant that is 32% smaller and 1.3x faster at inference with accuracy comparable to the full model, which is valuable for high-throughput screening and resource-constrained settings. The work is one of the first layer-redundancy analyses of a large atomistic foundation model, and its conclusion is consistent with the broader observation that later transformer/GNN blocks give diminishing returns. The manuscript is transparent in several important ways: code is released; the JMP-L baseline is reproduced rather than taken from the original paper on faith; the GradCAM-based block ordering is checked against a layer-wise ablation; the EquiformerV2 study honestly reports only partial transfer; and the mixed KD results are acknowledged in Section 4.3. These strengths make the empirical core credible. The significance is nevertheless conditional: the over-parameterization conclusion rests on a fixed-wall-clock comparison that does not separate model capacity from optimization budget, on single-run results without error bars, and on an evaluation of only five of the 40 JMP tasks.","major_comments":[{"comment":"The claim that removing two interaction blocks yields a 'minimal performance drop' is confounded by the fixed wall-clock fine-tuning budget. The authors fine-tune every configuration for 1 GPU-day on a V100 (2 GPU-days for QM9); since the 4- and 5-block models have fewer FLOPs per step, they receive proportionally more optimizer steps within that window. Equality of test MAE under this protocol therefore demonstrates efficiency per unit of wall-clock time, not that the removed blocks are superfluous at convergence. This is not a pedantic distinction: Section 4.4 (Figure 4) shows the 6-block model retaining a clear advantage on MD22 at the 1-GPU-day cutoff, and Section 2.1 states that JMP-L fine-tuning requires over 275 GPU hours to converge, so the baseline is likely far from plateau. The manuscript discloses the fixed-budget constraint ('constrained by a fixed computational budget') but does not discuss how the constraint interacts with the over-parameterization conclusion. The abstract's conclusion that 'JMP-L is over-parameterized' requires a comparison in which the 6-block model has either converged or demonstrably ceased to improve; the current protocol cannot distinguish the pruning hypothesis from the undertrained-baseline hypothesis. I ask the authors to add training curves extending past the budget cutoff (or converged fine-tuning runs) for the 6-block and 4-block models, or to reframe the central claim as fixed-budget efficiency.","section":"§4.3, Figures 3-4; Abstract"},{"comment":"All reported results are single runs with no error bars, multiple seeds, or significance tests. The differences that carry the paper's conclusion are small (e.g., in Figure 3, the 4-block and 5-block BR curves are within a few meV/Angstrom of the 6-block baseline on rMD17 and SPICE), and the KD comparisons change sign across block counts on rMD17. Without variance estimates, 'minimal performance drop' and 'comparable accuracy' are not quantitatively supported, and the reader cannot tell whether the 4-block-versus-6-block gap is real or within run-to-run noise. Please report mean and standard deviation over at least three seeds for the headline fine-tuning comparisons, and preferably for the pre-training MAE table as well.","section":"Figures 3-4; Tables 1-2"},{"comment":"The downstream evaluation covers five targets (Aspirin from rMD17, U0 from QM9, solvated amino acids from SPICE, Ac-Ala3-NHMe from MD22, and band gap from QMOF) out of the 40 tasks used in [38], yet the abstract and introduction generalize to 'many tasks' and to JMP-L as such. The five targets are predominantly molecular force/energy benchmarks plus one MOF property; no OC20/OC22 adsorption or catalysis tasks are included, so the over-parameterization conclusion is not tested where JMP's pre-training capacity may matter most. Either expand the task coverage, or explicitly restrict the over-parameterization and 'comparable performance' claims to the evaluated regime.","section":"§4.3; Abstract; §1"}],"minor_comments":[{"comment":"Appendix C shows an inversion in the GradCAM ordering: removing f4 hurts more than removing f3, contrary to the importance scores in Figure 2. Since the headline result removes the last two blocks (f6 and f7), this discrepancy does not invalidate the main claim, but the main text presents the scores as the justification for the pruning order without stating that the ablation validates only the global trend; a sentence acknowledging the f3/f4 caveat should appear in Section 3.2.","section":"§3.2, Appendix C"},{"comment":"The introduction's statement that 'pruning and distillation improve JMP-L's efficiency while preserving comparable performance for most tasks' and the abstract's mention of a 'pruned and distilled variant' are stronger than the results in Section 4.3: KD hurts the 4- and 5-block models on rMD17 and helps mainly at the largest reductions. The conclusion's phrasing ('knowledge distillation can help mitigate performance degradation in certain tasks') is accurate and should be used consistently in the abstract and introduction.","section":"§1, Abstract, §4.3"},{"comment":"The notation in the GradCAM description is imprecise: the symbol ∇CAM is never defined, and r = ReLU(f ∘ ∇CAM) should specify that the gradient is taken of L0 with respect to the concatenated feature tensor; also, the decomposition of r into per-block partitions should state the dimensions involved, since the interaction blocks and the embedding block do not share identical output shapes.","section":"§3.2"},{"comment":"Figure 2's caption says 'This table illustrates...' although it is a figure, and Table 2 does not state the measurement conditions for throughput and GFlops (batch size, input graph sizes, warm-up, and whether the batch size is held constant across block counts); please specify these details.","section":"Figure 2 caption; Table 2"},{"comment":"The text says the 4-block model 'outperforms the original JMP-L on the QM9 target,' but it is ambiguous whether 'original JMP-L' refers to the published converged numbers from [38], which were obtained under a different fine-tuning protocol, or to the authors' own 1-GPU-day reproduction; the comparison is only valid if the protocols are matched, so this should be clarified.","section":"§4.3, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a competent compression study with a clear practical payoff, and the authors are transparent in the appendix (layer-wise ablation, from-scratch comparison, honest reporting of limited EquiformerV2 transfer). The main risk is that the headline over-parameterization claim rests on a fixed-wall-clock comparison whose confound is real; the revision should either add converged or saturated baselines plus repeated seeds, or explicitly reframe the contribution as fixed-budget efficiency. The novelty is incremental relative to layer-pruning and KD literature in other domains, but the molecular-foundation-model context gives it reasonable fit and value for the readership. One factual claim worth checking editorially: the statement in Section 2.2.3 that no prior work has combined pruning and distillation for DFT/MD molecular property prediction is strong and should be verified against the broader ML-potential literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper ships a real, reproducible compression study of JMP-L, and the appendix work is more careful than the abstract. But the headline claim—'JMP-L is over-parameterized'—is not established by the experiments, because the fixed wall-clock fine-tuning budget confounds capacity with optimization.\n\nWhat's genuinely useful: nobody had run a GradCAM-style importance analysis on GemNet-OC interaction blocks and then pruned JMP-L along those lines. The 4-block variant (108M params, 1.3x throughput) is a practical artifact, and the code is public. The layer-wise ablation in Appendix C is the right check, and it mostly supports the ordering—later blocks are less important—even though f3/f4 invert. The from-scratch comparison in Appendix B is also a good control: it shows pruning a pre-trained model beats training the smaller model from scratch, which is a real point in favor of the pipeline.\n\nThe soft spot is exactly what the stress-test flags. Section 4.3 fine-tunes every model for 1 GPU-day (2 for QM9). Since the 4-block model is cheaper per step, it gets more optimizer steps in that window. So 'comparable MAE at equal wall-clock' means 'better compute efficiency', not 'the extra capacity of the 6-block model is unnecessary.' That distinction matters because the paper's own convergence plot (Figure 4) shows the 6-block model still ahead on MD22 at the cutoff, with no evidence it has plateaued. With more steps it could pull ahead on other tasks too. No error bars or multiple seeds make it worse: the 'minimal drop' on five of forty tasks cannot separate pruning from undertraining.\n\nThe KD part is honestly mixed, and the paper says so. That's fine; it doesn't sink anything.\n\nBottom line: reframed as 'within a fixed fine-tuning budget, a 108M-parameter JMP-L gives the better accuracy/compute trade-off,' the result is solid and worth publishing. The abstract overreaches. I'd send it to peer review, because the artifact and appendix are useful and the confound is fixable with a matched-step comparison or loss curves. A careful referee should ask for that.","headline":"Useful compression study with a real confound: fixed wall-clock budget makes the 'over-parameterized' conclusion unproven.","tokens_in":16363,"tokens_out":2743,"would_cite":false,"duration_ms":26126,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Removing two interaction blocks shrinks JMP-L by 32% with minimal accuracy loss.","keywords":["molecular property prediction","foundation models","model pruning","block reduction","knowledge distillation","GradCAM","graph neural networks","GemNet-OC"],"falsifier":"Run the paper's layer-wise ablation on rMD17 and at least one more dataset, removing each interaction block individually; if removing a block with higher GradCAM relevance consistently causes no larger a performance drop than removing a lower-ranked block, the ranking that selects the blocks to prune is not doing the work.","tokens_in":15371,"feed_emoji":"⚛️","tokens_out":4563,"duration_ms":41743,"temperature":0.7,"pith_summary":"This paper argues that JMP-L, a 160-million-parameter foundation model for molecular property prediction, is over-parameterized for most downstream tasks. It claims that the later interaction blocks of the model contribute least to prediction quality, so they can be removed. Pruning the two deepest interaction blocks yields a 108-million-parameter model with a 1.3x inference speedup and only a minimal drop in accuracy. If correct, practitioners can fine-tune and deploy a substantially cheaper model without retraining from scratch.","feed_headline":"Removing two blocks cuts JMP-L size 32%, with little accuracy loss","feed_subtitle":"A 108M-parameter variant runs 1.3x faster and matches the 160M model on most molecular tasks.","key_machinery":"The load-bearing mechanism is a GradCAM-style relevance score: features from each interaction block are weighted by their gradient with respect to the loss, passed through a ReLU, and averaged to give an importance rank per block. The paper uses this ranking to prune blocks sequentially from the deepest end, then adjusts the first FinalMLP layer either by slicing its weights or by random initialization. A short knowledge-distillation phase aligns the pruned model's output and intermediate node and edge representations with the teacher model.","core_discovery":"The central claim is that structured block reduction of a pre-trained GemNet-OC backbone preserves most of JMP-L's accuracy while materially improving efficiency. Using GradCAM relevance scores, the authors rank the six interaction blocks and find diminishing returns in deeper blocks. Removing the last two blocks cuts parameters from 160M to 108M and raises inference throughput from 19.1 to 25.6 samples/s on QMOF, while the pruned model matches the full model on rMD17, SPICE, and QMOF and even outperforms it on QM9. Pre-training distillation with node-to-node and edge-to-edge losses can reduce the remaining gap, though it helps only in some settings.","pith_inferences":["The paper's own layer-wise ablation suggests the GradCAM ordering is imperfect, so a simpler rule such as 'always remove the deepest blocks' might work as well without computing relevance scores.","Because the 4-block model beats the full model on QM9, block reduction may act as a regularizer; this could be tested by comparing fine-tuning variance across multiple random seeds.","The diminishing-returns pattern mirrors observations in large language models, raising the question of whether block reduction is a general property of deep pre-trained networks rather than specific to GemNet-OC.","A testable extension is to apply the same pipeline to other molecular backbones with publicly available pre-trained checkpoints, predicting that the benefit scales with the teacher's parameter count."],"forward_implications":["A 4-interaction-block JMP-L (108M parameters) can replace the 160M model on many tasks with negligible accuracy change.","Fine-tuning the smaller model converges as fast as or faster than the full model on tasks like QM9 within a fixed 1 GPU-day budget.","Resource-constrained workflows and high-throughput screening can use a 1.3x faster inference model with 32% fewer parameters.","Block reduction transfers to another architecture, EquiformerV2, but the gains shrink when the teacher model is already compact at 31M parameters.","Pre-training distillation is not uniformly beneficial downstream; it helps most for heavily pruned models and can hurt when only one or two blocks are removed."],"supporting_citations":[{"why":"Provides the JMP-L pre-trained checkpoint and the pre-train-then-finetune protocol this paper compresses.","marker":"[38]"},{"why":"Defines the GemNet-OC architecture whose interaction blocks are being pruned.","marker":"[18]"},{"why":"Supplies the GradCAM method used to compute block relevance scores.","marker":"[36]"},{"why":"Supplies the node-to-node and edge-to-edge distillation losses used in the knowledge-distillation phase.","marker":"[16]"},{"why":"Provides the OC20 pre-training dataset used in the block-reduction and distillation evaluation.","marker":"[8]"},{"why":"Provides the OC22 pre-training dataset used in the same evaluation.","marker":"[44]"},{"why":"Provides the rMD17 target used for downstream fine-tuning and the layer-wise ablation.","marker":"[10]"},{"why":"Provides the QM9 target where the 4-block model matches or beats the full model.","marker":"[29]"}],"fun_headline_variants":["Cutting two blocks shrinks JMP-L 32%, speeds it 1.3x","JMP-L pruned: 32% smaller, 1.3x faster, similar accuracy","Two blocks less: JMP-L drops 32% size, gains 1.3x throughput","Pruning JMP-L: remove two blocks, retain accuracy, boost speed","JMP-L over-parameterized: 32% smaller, 1.3x faster, same results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gradient-based importance scores (GradCAM) correctly rank which interaction blocks are least useful, so removing the lowest-scoring blocks preserves accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Cutting two blocks shrinks JMP-L 32%, speeds it 1.3x","JMP-L pruned: 32% smaller, 1.3x faster, similar accuracy","Two blocks less: JMP-L drops 32% size, gains 1.3x throughput","Pruning JMP-L: remove two blocks, retain accuracy, boost speed","JMP-L over-parameterized: 32% smaller, 1.3x faster, same results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1455,"prompt_tokens":923,"completion_tokens":532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":410}},"tokens_in":539,"tokens_out":532,"duration_ms":3987,"temperature":1.0,"reasoning_tokens":410,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:49:07.554550+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's layer-wise ablation on rMD17 and at least one more dataset, removing each interaction block individually; if removing a block with higher GradCAM relevance consistently causes no larger a performance drop than removing a lower-ranked block, the ranking that selects the blocks to prune is not doing the work.","supporting_citations":[{"cited_title":"Accelerating molecular graph neu- ral networks via knowledge distillation","cited_arxiv_id":null,"evidence_quote":"Supplies the node-to-node and edge-to-edge distillation losses used in the knowledge-distillation phase."},{"cited_title":"Open catalyst 2020 (oc20) dataset and community challenges","cited_arxiv_id":null,"evidence_quote":"Provides the OC20 pre-training dataset used in the block-reduction and distillation evaluation."},{"cited_title":"The open catalyst 2022 (oc22) dataset and challenges for oxide electrocatalysts","cited_arxiv_id":null,"evidence_quote":"Provides the OC22 pre-training dataset used in the same evaluation."},{"cited_title":"Machine learning of accurate energy-conserving molecular force fields","cited_arxiv_id":null,"evidence_quote":"Provides the rMD17 target used for downstream fine-tuning and the layer-wise ablation."},{"cited_title":"Quantum chemistry structures and properties of 134 kilo molecules","cited_arxiv_id":null,"evidence_quote":"Provides the QM9 target where the 4-block model matches or beats the full model."}],"review_version":1}