{"id":"58a382ed-adae-4525-bf10-f8522e6a2633","arxiv_id":"2606.09377","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Adapts TP and FSDP to bound-propagation verification, with FSDP delivering bitwise-identical bounds and 80-90% baseline memory reduction while TP trades some tightness for ~2x peak-memory savings.","lead":"This paper adapts tensor parallelism and fully sharded data parallelism from training to neural network verification tools like α-CROWN to cut GPU memory use. A smart generalist might read it because verifying safety properties on bigger models matters for reliable AI systems.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"FSDP AllGather claim of bitwise-identical bounds rests on unstated assumption that sharded weight access + gather produces exactly the same floating-point sequence as monolithic execution","rationale":"The reader's weakest_assumption already isolates the same soundness-preservation issue for FSDP. The abstract-only limitation noted by the reader remains the dominant source of uncertainty; no additional internal inconsistency is visible from the provided text.","tokens_in":1846,"tokens_out":403,"duration_ms":10678,"concrete_test":"Instrument the FSDP implementation to dump the exact sequence of weight matrices seen by the bound-propagation kernels on both the sharded and single-GPU runs (e.g., via a deterministic hash of each matrix before every matmul); if any hash differs or if the final bound tensors differ in even one bit on the MNIST-FC or ResNet-large instances, the bitwise-identity claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central FSDP claim (bitwise-identical bounds, 80-90% baseline / 34-39% peak memory reduction, and complete unsat on CIFAR-100 ResNet-large) requires that the per-layer AllGather followed by standard bound propagation yields identical intermediate tensors to the single-GPU case. This is only true if (a) the gather is performed before every matrix multiplication that uses the sharded weights, (b) no re-association or fused kernels alter FP order, and (c) the bound-propagation code path (including BoundConv and beta-CROWN) is literally unchanged once the full weight matrix is materialized. The abstract gives no equation or pseudocode showing the exact insertion point of the AllGather relative to the IBP/CROWN forward passes, nor any mention of how alpha tensors or relaxation coefficients are handled when weights are temporarily sharded. Any deviation in these details would silently violate the bitwise-identity guarantee even if the final output happens to match on the reported benchmarks.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper adapts Tensor Parallelism (TP) and Fully Sharded Data Parallelism (FSDP) from training to the auto_LiRPA / α,β-CROWN verification framework. TP shards both weights and A-matrices for ≈2× peak-memory reduction at P=2, with soundness confirmed on VNN-COMP 2022 MNIST-FC benchmarks but with bound tightness degrading due to forced IBP substitutions in sharded zones. FSDP shards only weights via per-layer AllGather, claims bitwise-identical bounds to the single-GPU baseline, reports 80–90% baseline and 34–39% peak memory reductions on wide MLPs, integrates with β-CROWN + Branch-and-Bound and BoundConv, and obtains a complete unsat result on CIFAR-100 ResNet-large (VNN-COMP 2024). The work identifies per-neuron alpha tensors as the remaining memory bottleneck.","tokens_in":2082,"tokens_out":619,"duration_ms":17740,"significance":"If the bitwise-identity claim for FSDP holds without numerical drift, the result is significant because it directly addresses the GPU-memory barrier that currently limits bound-propagation verification to modest-sized networks. The complete verification result on a 2024 ResNet benchmark and the clean integration with existing BoundConv and BaB machinery are concrete engineering advances. The identification of alpha tensors rather than weights as the next bottleneck supplies a clear direction for follow-on work. The experimental grounding on standard VNN-COMP suites strengthens the practical utility.","major_comments":[{"comment":"Abstract (FSDP paragraph): the bitwise-identity guarantee for bounds produced after per-layer AllGather rests on the unstated assumptions that (a) the gather occurs immediately before every matrix multiplication that consumes the sharded weights, (b) no kernel fusion or re-association changes floating-point evaluation order, and (c) alpha tensors and relaxation coefficients are materialized and used identically to the monolithic case. The manuscript must supply pseudocode or a precise description of the AllGather insertion point relative to the IBP/CROWN forward passes and confirm that the BoundConv and β-CROWN code paths are literally unchanged once the full weight matrix is assembled.","section":"Abstract / FSDP implementation"},{"comment":"TP description: the claim that soundness is preserved under forced IBP substitution inside sharded zones is load-bearing for the overall contribution, yet the manuscript provides no quantitative characterization of how the substitution affects final bound values or the number of verified properties on the VNN-COMP 2022 suite when the number of sharded zones increases beyond P=2.","section":"TP section"}],"minor_comments":[{"comment":"The abstract refers to “wide MLPs” for the memory-reduction numbers but does not list the exact layer widths or depths used; a table or explicit architecture description would improve reproducibility.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. The two major comments identify areas where additional detail and data would strengthen the manuscript. We address each point below and will revise accordingly.","responses":[{"response":"We agree that the bitwise-identity claim requires explicit justification of the listed assumptions. In the revised manuscript we will insert pseudocode that shows the per-layer AllGather occurring immediately before each matrix multiplication that consumes the reconstructed weight matrix, for both the IBP and CROWN forward passes. We will also state that the BoundConv and β-CROWN code paths remain literally unchanged once the full weight matrix has been assembled by the AllGather, and that alpha tensors and relaxation coefficients are materialized and used exactly as in the monolithic baseline. Our current implementation performs no kernel fusion or re-association that would alter floating-point order.","revision_made":"yes","referee_comment":"[Abstract / FSDP implementation] Abstract (FSDP paragraph): the bitwise-identity guarantee for bounds produced after per-layer AllGather rests on the unstated assumptions that (a) the gather occurs immediately before every matrix multiplication that consumes the sharded weights, (b) no kernel fusion or re-association changes floating-point evaluation order, and (c) alpha tensors and relaxation coefficients are materialized and used identically to the monolithic case. The manuscript must supply pseudocode or a precise description of the AllGather insertion point relative to the IBP/CROWN forward passes and confirm that the BoundConv and β-CROWN code paths are literally unchanged once the full weight matrix is assembled."},{"response":"We acknowledge that a quantitative characterization for P>2 would strengthen the soundness claim. While the manuscript already notes degradation of bound tightness with increasing sharded zones and confirms soundness at P=2, the revised version will add experimental results for P=4 on the VNN-COMP 2022 MNIST-FC benchmarks, reporting the effect on final bound values and the number of verified properties.","revision_made":"yes","referee_comment":"[TP section] TP description: the claim that soundness is preserved under forced IBP substitution inside sharded zones is load-bearing for the overall contribution, yet the manuscript provides no quantitative characterization of how the substitution affects final bound values or the number of verified properties on the VNN-COMP 2022 suite when the number of sharded zones increases beyond P=2."}],"tokens_in":1641,"tokens_out":511,"duration_ms":17212,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core result is that FSDP can be added to the auto_LiRPA / α,β-CROWN stack with a per-layer AllGather on weights, delivering 80-90% baseline memory reduction and 34-39% peak reduction on wide MLPs while producing bitwise-identical bounds to the single-GPU case. They also show it works with BoundConv and with complete verification via β-CROWN plus branch-and-bound, including a full unsat result on a CIFAR-100 ResNet-large from the 2024 VNN-COMP benchmark. TP gives roughly 2x peak memory savings at two GPUs but requires IBP substitution inside sharded zones, which degrades bound tightness as the number of shards grows.\n\nThe new part is simply the application itself: these training sharding patterns had not been tried inside bound-propagation verifiers before, and the paper supplies concrete memory numbers plus the soundness checks on VNN-COMP 2022 MNIST-FC models. The FSDP path integrates without changing the core propagation logic once the full weight matrix is reassembled.\n\nThe main soft spot is that the bitwise-identity claim for FSDP rests on the AllGather happening before every relevant matrix multiply and on no change to floating-point order or to how alpha tensors and relaxation coefficients are handled. The abstract does not give the exact insertion point or any stability measurements across sharding, so the guarantee is plausible but not yet fully documented. TP's forced IBP substitution is a known compromise rather than a flaw in the sharding idea. They correctly flag that alpha tensors, not weights, are now the dominant memory user in α-CROWN+BaB mode.\n\nThis is useful engineering for the certified-robustness group that routinely hits single-GPU memory walls. The work is straightforward, cites the right baselines, and reports results on standard benchmarks, so it deserves a serious referee.","headline":"FSDP ports cleanly into α-CROWN for exact bounds and big memory cuts on weights; TP works but loosens bounds and the alpha tensors stay the real limit.","tokens_in":2611,"tokens_out":462,"would_cite":false,"duration_ms":12096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Fully sharded data parallelism on weight matrices produces bitwise-identical verification bounds while cutting baseline memory 80-90%.","keywords":["neural network verification","bound propagation","tensor parallelism","fully sharded data parallelism","FSDP","alpha-CROWN","memory reduction","complete verification"],"falsifier":"Run the identical verification query once on a single GPU and once under FSDP on multiple GPUs; any bound that differs beyond floating-point error falsifies the claim of bitwise identity.","tokens_in":2729,"feed_emoji":"💾","tokens_out":656,"duration_ms":11667,"temperature":0.7,"pith_summary":"The paper adapts tensor parallelism and fully sharded data parallelism from training to bound-propagation verification frameworks such as auto_LiRPA and alpha-CROWN. Tensor parallelism shards both weights and A-matrices across GPUs for roughly 2x peak-memory reduction, though it requires forced IBP substitutions inside sharded zones that loosen bounds. Fully sharded data parallelism shards only weights via per-layer AllGather, keeps every bound bitwise identical to the single-GPU run, and integrates directly with beta-CROWN branch-and-bound and convolutional layers. This enables a complete unsat result on a CIFAR-100 ResNet-large instance from VNN-COMP 2024. The experiments identify per-neuron alpha tensors, not weights, as the remaining memory limit.","feed_headline":"FSDP gives identical verification bounds at 80-90% lower memory","feed_subtitle":"Sharding only weights via per-layer AllGather keeps bounds bitwise the same while scaling to larger models like ResNet on CIFAR-100.","key_machinery":"The per-layer AllGather operation in Fully Sharded Data Parallelism that shards only weight matrices while leaving relaxation coefficients on each device.","core_discovery":"FSDP shards only weight matrices with a per-layer AllGather, producing bounds that are bitwise identical to the single-GPU baseline: baseline memory drops by 80-90%, peak memory by 34-39% on wide MLPs. A complete unsat result is obtained for CIFAR-100 ResNet-large under FSDP.","pith_inferences":["Sharding the alpha tensors themselves would be the next logical step to remove the new bottleneck.","The same FSDP pattern could be applied to other bound-propagation libraries without altering their core relaxation logic.","Tensor parallelism tightness loss might be mitigated by keeping more intermediate bounds on every device rather than forcing IBP."],"forward_implications":["FSDP integrates directly with complete verification using beta-CROWN plus branch-and-bound.","BoundConv convolutional layers remain compatible under the FSDP sharding scheme.","Tensor parallelism at two GPUs yields approximately 2x peak-memory reduction on MNIST-FC benchmarks.","Per-neuron alpha tensors become the dominant memory consumer once weights are sharded."],"fun_headline_variants":["FSDP matches single-GPU bounds at 80-90% lower memory","Weight sharding with AllGather preserves exact verification bounds","FSDP cuts memory 80-90% while matching baseline verification bounds","Identical bounds for CIFAR-100 ResNet via FSDP"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The sharding and AllGather steps preserve exact numerical values and overall soundness of bound propagation without introducing drift.","fun_headline_variants_meta":{"raw":{"variants":["FSDP matches single-GPU bounds at 80-90% lower memory","Weight sharding with AllGather preserves exact verification bounds","FSDP cuts memory 80-90% while matching baseline verification bounds","Identical bounds for CIFAR-100 ResNet via FSDP"]},"model":"grok-4.3","cost_usd":0.007289,"raw_usage":{"total_tokens":3395,"prompt_tokens":744,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":72887000,"prompt_tokens_details":{"text_tokens":744,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2578,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":744,"tokens_out":73,"duration_ms":13586,"temperature":1.0,"reasoning_tokens":2578,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T17:18:35.299879+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run the identical verification query once on a single GPU and once under FSDP on multiple GPUs; any bound that differs beyond floating-point error falsifies the claim of bitwise identity.","supporting_citations":[],"review_version":1}