{"id":"848898c8-e9af-451e-9c22-9517b3e9cff8","arxiv_id":"2505.12906","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A CMA-ES plus attention-pruning training algorithm for block optical neural networks prunes 60-80% of parameters with under 5% accuracy loss and shows improved noise robustness.","lead":"Researchers combine an evolutionary optimizer (CMA-ES) with attention-based block pruning to train optical neural networks built from Mach-Zehnder interferometers. The method reportedly removes 60 to 80 percent of trainable parameters with small accuracy loss and stays robust under simulated phase-shifter noise.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Robustness advantage over CMA-ES is not supported: on MNIST absolute noisy accuracies are essentially tied (67.569% vs ~67.4%) with no error bars; the smaller 'degradation' reflects a lower pruned baseline, not stronger noise tolerance.","rationale":"The reader's weakest_assumption was the Gaussian noise model, which is a valid external-validity concern. My primary concern is more internal: the headline robustness ordering for MNIST is an artifact of comparing percentage degradations from different baselines, and the residual 0.14-point absolute gap is far too small to support a claim of stronger robustness without error bars. This does not invalidate the entire paper: the pruning results are plausible and partially validated in hardware, the Fashion-MNIST absolute improvement is real, and the attention-based pruning idea is a reasonable engineering contribution. However, the central 'strongest robustness' claim is overstated for MNIST. The reader's CONDITIONAL verdict remains appropriate; the condition should explicitly require a multi-seed comparison with absolute noisy accuracies reported. I therefore keep the verdict unchanged rather than moving it.","tokens_in":15758,"tokens_out":9928,"duration_ms":103610,"concrete_test":"Extract per-seed results behind Fig. 4(d): run CAP and CMA-ES on MNIST with sigma_P=0.5 for at least 10 independent seeds using the same total generation budget, and report the mean and standard deviation of avg10acc/test accuracy. Also report the absolute noisy accuracy for both methods, not only the degradation from their respective baselines. If CAP's mean absolute accuracy is not above CMA-ES by more than the pooled standard error, or if the baseline-adjusted degradation difference is not significant, the headline robustness ordering over CMA-ES should be withdrawn or downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's headline robustness ordering rests on the performance-degradation metric: CAP 22.327% vs CMA-ES 25.757% on MNIST. However, these degradations are measured from different noiseless baselines. CAP's pruned baseline is 89.896% (Fig. 4(a), S=40%), while CMA-ES's baseline is 93.185% (Fig. 2(a)). Therefore the absolute noisy avg10acc at sigma_P=0.5 is CAP 67.569% versus CMA-ES approximately 67.428% (93.185 - 25.757), a gap of only 0.14 percentage points. The paper reports no error bars or repeated-seed statistics for CAP or CMA-ES; the only multi-trial band shown is for BAT (Fig. 4(d), sand-purple area). A 0.14-point separation is well within the run-to-run variation expected for evolutionary training, so the claimed advantage over CMA-ES on MNIST is not statistically established. On Fashion-MNIST the absolute advantage is larger (56.519% vs 52.36%), which is more meaningful, but the MNIST claim as summarized is not supported by the reported data. Additionally, the entire robustness comparison is performed under the Section 3.2 assumption of additive Gaussian phase-shifter noise; the hardware experiment does not inject or independently measure this noise distribution, so even the larger Fashion-MNIST advantage is not validated on-chip.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CAP, an on-chip training algorithm for large-scale MZI-based block optical neural networks (BONNs) that combines CMA-ES with attention-based block pruning. Learning attention coefficients for each matrix block, the method prunes low-attention blocks and then retrains the remaining phase shifters with CMA-ES. The authors report that CAP can prune 60% and 80% of parameters on MNIST and Fashion-MNIST with test-accuracy degradation of 3.289% and 4.693%, respectively. Under simulated dynamic Gaussian phase-shifter noise with standard deviation 0.5, they report that CAP exhibits smaller accuracy degradation (22.327% MNIST, 24.019% Fashion-MNIST) than their previously reported BAT algorithm (43.963%, 41.074%) and CMA-ES (25.757%, 32.871%). They also experimentally train a 4x4 MZI PIC on a simplified MNIST task, achieving 88.5% accuracy with 60% pruning, close to the noiseless simulation value of 92.1%, and they demonstrate that omitting external phase shifters reduces parameter count without major accuracy loss.","tokens_in":16097,"tokens_out":3846,"duration_ms":38050,"significance":"If the central robustness ordering holds, CAP would be a meaningful practical contribution to on-chip training of optical neural networks, combining evolutionary optimization with attention-based structural pruning in a setting where gradient signals are noisy and chip area is constrained. The paper includes a genuine hardware experiment, which supports the feasibility of online evolutionary training on a fabricated PIC. The population-based approach is a reasonable alternative to gradient-based methods for noisy hardware. However, the statistical support for the headline robustness claim is weak, and the noise model is assumed rather than measured, so the significance of the claimed advantage over CMA-ES on MNIST in particular is not established. The pruning results are directionally interesting but are also selected on test-set accuracy without a validation split or repeated trials.","major_comments":[{"comment":"The headline robustness advantage of CAP over CMA-ES on MNIST is not supported by the reported data. The performance-degradation metric in the abstract is computed relative to different noiseless baselines: CAP's pruned baseline is 89.896% (Fig. 4(a), S=40%) while CMA-ES's unpruned baseline is 93.185% (Fig. 2(a)). At sigma_P=0.5, CAP's absolute avg10acc is 67.569% and CMA-ES's absolute avg10acc is 93.185% - 25.757% = 67.428%, a gap of only 0.14 percentage points. No error bars or repeated-seed statistics are reported for CAP or CMA-ES; the only multi-trial band shown is for BAT (Fig. 4(d)). A 0.14-point separation is well within run-to-run variation for evolutionary training, so the claim that CAP is more robust than CMA-ES on MNIST is not statistically established. On Fashion-MNIST the absolute gap is larger (56.519% vs 52.36%), but even there no error bars are given. The paper should either report absolute noisy accuracies with uncertainty or restrict the robustness claim to Fashion-MNIST with appropriate error characterization.","section":"Section 3.2, Fig. 4(d), Abstract"},{"comment":"The entire robustness comparison relies on the assumption introduced in Section 3.2 that dynamic phase-shifter error is an additive Gaussian random variable with mean zero and standard deviation sigma_P, and that static error is fully captured by the extinction ratio of the directional coupler. The hardware experiment in Section 3.3 does not measure this noise distribution on the fabricated chip, nor does it inject calibrated dynamic noise to validate the simulation model. The authors attribute the small simulation-experiment gap for the BONN without external phase shifters to external phase-shifter noise, but no noise characterization is provided. Consequently, the transferability of the reported robustness ordering to real hardware is not demonstrated, and the claim that CAP is 'strongest robustness' under real device conditions remains an extrapolation from an unvalidated model.","section":"Section 3.2, noise model; Section 3.3, experiment"},{"comment":"The hyperparameters S (attention-training ratio), R (pruning ratio), and population size h are all selected using avg10acc, which is the test-set accuracy, and no validation split is used. For example, S=40% is chosen because it gives the highest avg10acc on the MNIST test set (Fig. 4(a)), and the reported pruning degradations (3.289% and 4.693%) are the results at the selected R and S. This creates an optimistic bias in the headline numbers, since the same test set is used for both model selection and evaluation. In addition, no repeated trials are reported for CAP or CMA-ES, so the reported gaps (e.g., the 4.159 percentage-point advantage on Fashion-MNIST) cannot be distinguished from noise. The authors should use a validation set for hyperparameter selection, report means and standard deviations over multiple independent runs, and state the number of seeds.","section":"Sections 3.1 and 3.2, hyperparameter selection"}],"minor_comments":[{"comment":"There are numerous grammatical errors and typos, e.g., 'have increasingly drawn attentions', 'the CAP algorithm show excellent potential', and inconsistent spacing in equations. A thorough language edit is needed.","section":"Abstract and throughout"},{"comment":"In the Introduction, the statement that MNIST requires around one thousand ports and CIFAR-10 requires thousands or tens of thousands of ports is cited to references [7] and [8], which are papers about coin betting optimization and mixing Adam/SGD, respectively. These citations appear incorrect and should be replaced with appropriate sources.","section":"References [7] and [8]"},{"comment":"The cross-entropy fitness formula appears garbled in the typeset equation; the softmax normalization and logarithm are not clearly expressed. Please rewrite the equation and define all symbols explicitly.","section":"Equation (7)"},{"comment":"The figures show solid lines for CAP and CMA-ES without error bars, while the sand-colored bands are only for BAT. Adding shading or error bars for all methods would make the comparison more transparent. Also, the x-axis labels for Fig. 4(d) and Fig. 5(d) should clearly state whether the curves correspond to training with noise or testing an already-trained network under noise, since the text discusses both settings.","section":"Figures 4 and 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim of superior robustness on MNIST rests on a comparison of relative degradation from different baselines, which is not statistically meaningful without error bars or absolute accuracy. The authors may be able to fix this by re-analyzing their data and softening the claim, but the current presentation overstates the result. The mismatch of references [7] and [8] with the cited claims about port requirements is concerning and should be corrected. The experimental demonstration is valuable and should be retained, but the noise model needs validation or explicit qualification. The paper fits the journal's scope, so I prefer a major revision over rejection, provided the authors can provide repeated-run statistics and a more careful robustness analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhiwei Yang et al. propose CAP, a CMA-ES training recipe with attention-based block pruning for MZI-based block optical neural networks. The pruning result is the strongest part: they show you can drop 60-80% of the weight blocks and lose only a few points on MNIST/Fashion-MNIST, beating random pruning, and they demonstrate a 4×4 MZI chip trained online. That is a legitimate engineering contribution.\n\nThe headline robustness claim, however, is not supported by the reported numbers. The abstract says CAP's performance degradation under phase-shifter noise (std 0.5) is 22.327% versus CMA-ES 25.757% on MNIST. But these degradations are measured from different baselines: CAP's pruned noiseless average is 89.896%, CMA-ES's unpruned is 93.185%. The absolute noisy accuracies are 67.569% for CAP and about 67.4% for CMA-ES—statistically indistinguishable, and there are no error bars for either method. The same logic applies to the BAT comparison. On Fashion-MNIST the absolute gap is larger (56.519 vs 52.36), so there may be a real effect there, but it still rests on the same assumed Gaussian noise model and no repeated-seed statistics.\n\nOther soft spots: hyperparameters (block size, population, S, R) are selected using test-set accuracy; the simulation noise model is introduced in Sec. 3.2 as an assumption, not measured on the fabricated chip; no code or data is released. The hardware experiment uses 100 simplified MNIST images with a pre-trained conv front end, so it is a proof of concept rather than a full validation.\n\nNone of this kills the paper. The pruning algorithm itself is new and useful, and the comparison to random pruning is the right baseline. But the robustness ordering in the abstract and conclusion should be rewritten around absolute noisy accuracies, ideally with multiple seeds and matched pruned/retrained baselines.","headline":"The pruning contribution is real and worth peer review, but the robustness advantage over CMA-ES is not supported by the reported metrics.","tokens_in":16596,"tokens_out":2970,"would_cite":false,"duration_ms":29579,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-15T20:23:36.918003+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}