{"id":"697bb548-37ef-4807-b002-38dfbe103a4c","arxiv_id":"2505.03303","paper_version":5,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"EfficientNetV2 is reported as the most accurate and MobileNetV3 as the most balanced of five lightweight models, but the abstract and body disagree on the experiments, and missing code and hyperparameters prevent reproduction.","lead":"This preprint benchmarks five lightweight image classifiers on CIFAR-10, CIFAR-100, and Tiny ImageNet, reporting that EfficientNetV2 is most accurate while MobileNetV3 offers the best efficiency balance. The usable finding for practitioners is limited by unresolved mismatches between the abstract and the body and by missing training details.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline rankings are not traceable to the body: the abstract reports 97.57/86.98/78.73 and EfficientNet-B0, while the body reports 96.53/90.82/76.87 with no B0, and the pretrained-vs-scratch section mixes EfficientNetV2 pretrained with MobileNetV3 scratch.","rationale":"The reader's verdict is REJECT with high confidence, and my check reinforces that rejection rather than overturning it. The single most load-bearing problem is that the advertised central claim is not connected to the data in the body: abstract and body disagree on the number of models, the metrics, and the headline accuracies, and the one section devoted to a key secondary claim, pretraining advantage, compares different architectures across its table, figures, and discussion. This is an internal inconsistency, not a disagreement with consensus, and it is directly testable by tracing numbers to rows or rerunning the missing scratch configuration. The reader's weakest assumption about tuning artifacts is a further, independent concern: with no seeds, no per-model hyperparameters, and single runs, even a harmonized version would not establish that EfficientNetV2-S is genuinely the most accurate architecture. I mark agreement as 'partial' because my primary attack is on traceability of the numbers and the mismatched pretraining comparison, while the reader's stated weakest assumption emphasizes tuning artifacts; both point in the same direction. No code, seeds, or error bars are provided anywhere in the manuscript, so the narrower defensible claim would need to be rephrased as an observation about these particular runs rather than about architecture quality. The verdict stands as REJECT.","tokens_in":11780,"tokens_out":9146,"duration_ms":84974,"concrete_test":"Decisive check: locate the training logs or checkpoints for Table 4's scratch column and rerun EfficientNetV2-S from random initialization on CIFAR-10 under the Section 3.4 protocol with a fixed seed and the same hyperparameters used for the pretrained run. If the rerun's accuracy is not 92.51%, the transfer-learning claim is invalid because Section 4.5 compared pretrained EfficientNetV2 to scratch MobileNetV3. In parallel, search the manuscript for the abstract's values 97.57, 86.98, 78.73, and 'EfficientNet-B0'; if no table row contains them, the headline numbers have no supporting evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the reported accuracy and efficiency numbers to be the output of the described benchmark. That condition fails at the first check. The abstract advertises seven CNNs, top-5 accuracy, macro F1, FP32 parameter storage, and MACs, plus an EfficientNet-B0 comparison; none of these appear in the body. Tables 1-3 cover five models and report accuracy, F1, inference time, FLOPs, and model size, and the EfficientNetV2-S accuracies are 96.53/90.82/76.87, not the abstract's 97.57/86.98/78.73. The transfer-learning section is internally contradictory: Table 4 is headed 'Pretrained vs. Scratch-Trained MobileNetV3', but Figures 9-10 and Section 4.5 describe EfficientNetV2, and Section 4.5 compares pretrained EfficientNetV2's CIFAR-10 accuracy of 96.53% with 92.51%, a value that comes from Table 4's MobileNetV3 scratch column. There is therefore no matched pretrained-versus-scratch comparison for EfficientNetV2. Even if those contradictions were fixed, the body's ranking rests on single runs with unreported per-model learning rates, batch sizes, optimizers, schedulers, augmentation schedules, and seeds, so the ordering cannot be attributed to architecture quality rather than tuning artifacts.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to benchmark five lightweight CNNs (MobileNetV3-Small, ResNet18, SqueezeNet, EfficientNetV2-S, and ShuffleNetV2) on CIFAR-10, CIFAR-100, and Tiny ImageNet, reporting accuracy, F1 score, inference time, FLOPs, and model size. It also reports a pretrained-versus-scratch comparison, concluding that transfer learning helps, that EfficientNetV2-S is most accurate, that MobileNetV3-Small offers the best accuracy-efficiency balance, and that SqueezeNet is fastest and most compact. The abstract of the arXiv posting additionally claims seven models, top-5 accuracy, FP32 parameter storage, MACs, and a comparison with EfficientNet-B0, reporting EfficientNetV2-S accuracies of 97.57%, 86.98%, and 78.73%; the body reports EfficientNetV2 accuracies of 96.53%, 90.82%, and 76.87% and never mentions EfficientNet-B0. The manuscript's experimental protocol is underspecified, and the transfer-learning section is internally contradictory.","tokens_in":11970,"tokens_out":4346,"duration_ms":37463,"significance":"If the results were reliable, the paper would provide a useful practical reference for selecting lightweight architectures under memory and compute constraints, especially because it evaluates multiple datasets and several efficiency metrics in a single protocol. The multi-dataset scope and the explicit attention to FLOPs and model size are commendable features. However, the central claims are not traceable to the reported experiments: the abstract and body describe different numbers and different model sets, the pretrained-versus-scratch comparison is mislabeled, and the training protocol omits per-model hyperparameters and seeds. Without those details, the architecture rankings and the transfer-learning advantage cannot be verified, so the paper's current contribution is primarily a set of unconfirmed empirical observations rather than a reproducible benchmark.","major_comments":[{"comment":"The abstract reports that EfficientNetV2-S reaches 97.57%, 86.98%, and 78.73% top-1 accuracy on CIFAR-10, CIFAR-100, and Tiny ImageNet, and that EfficientNet-B0 remains within 0.85 percentage points while using only about 21% of the parameters and 14% of the MACs; the body's Tables 1–3 report EfficientNetV2 accuracies of 96.53%, 90.82%, and 76.87% and never mention EfficientNet-B0. The abstract also advertises seven CNNs, top-5 accuracy, macro F1, FP32 parameter storage, and MACs, none of which appear in the body. Because the paper's central ranking claim rests on numbers that cannot be traced to the reported experiments, the headline results are not verifiable from the manuscript.","section":"Abstract vs. Sections 4.1–4.3"},{"comment":"The transfer-learning comparison is internally contradictory. Table 4 is headed 'Pretrained vs. Scratch-Trained MobileNetV3' and lists MobileNetV3 scratch accuracies, but Figures 9–10 are captioned for EfficientNetV2, and Section 4.5 states that pretrained EfficientNetV2 achieved 96.53% versus 92.51% on CIFAR-10; 92.51% is the MobileNetV3 scratch value from Table 4. There is therefore no matched pretrained-versus-scratch comparison for EfficientNetV2, and the claim in Section 4.5 that pretraining provides an 'immense value of transfer learning' is not supported by the data as presented. The post hoc selection in Section 3.2 of MobileNetV3 for scratch training because of its pretrained performance also makes the later 'best balance' conclusion partly a re-description of that choice.","section":"Sections 4.4–4.5, Figures 9–10"},{"comment":"The training protocol is underspecified in a way that affects the central ranking claims. Section 3.4 states that learning rate, batch size, optimizer, scheduler, and early stopping were tuned, but no final values are reported for any model or dataset, no random seeds are given, and all tables appear to report single runs. Under these conditions, the observed ordering (for example, EfficientNetV2 over MobileNetV3 by 1.04 points on CIFAR-10) cannot be attributed to architecture quality rather than to per-model tuning choices. The same problem affects the augmentation claims in Sections 2.3 and 3.3.1, which assert that SqueezeNet and ShuffleNetV2 degraded under additional augmentations but present no experimental results for augmentation policies.","section":"Section 3.4, Tables 1–3"},{"comment":"The efficiency metrics are reported at a precision that undermines the trade-off analysis. FLOPs are listed as 0.00 GFLOPs for SqueezeNet on CIFAR-10 and CIFAR-100 and for ShuffleNetV2 on CIFAR-10 and CIFAR-100, which makes it impossible to distinguish models that the text claims differ in FLOPs; model size is given only in MB, with no reported parameter counts or distinction between pretrained and scratch checkpoint sizes. Since the paper's practical recommendations depend on comparing parameter and FLOP efficiency, the tables need more precise values and a clear statement of how the efficiency metrics were computed.","section":"Tables 1–3"}],"minor_comments":[{"comment":"The manuscript is internally inconsistent about the number of models evaluated: the title and introductory abstract say five, while the arXiv abstract says seven; please harmonize these descriptions.","section":"Title, abstract, and Section 1"},{"comment":"The model name is spelled inconsistently as both 'MobileNetV3' and 'MobilenetV3' (for example, the Section 4.4 heading and parts of the conclusion); please use a single convention throughout.","section":"Section 4.4 heading, Section 5"},{"comment":"The caption misspells 'Comparison' as 'Comaprison'; please correct the typo.","section":"Figure 6 caption"},{"comment":"References [10] and [23] are duplicate entries for the same ShuffleNetV2 paper, and reference [12] (Radosavovic et al., 'RegNet: Designing Network Design Spaces') is cited in Section 2.2 as arguing that training from scratch can yield competitive results, which does not appear to match the cited paper's content; please correct these citations.","section":"References [10], [12], [23]"},{"comment":"Section 3.3.1 lists AutoAugment, CutMix, and MixUp as applied augmentations, while Section 6 describes these as future work; please clarify which augmentations were actually used in the reported experiments.","section":"Section 3.3.1 and Section 6"}],"recommendation":"reject","confidential_remarks":"To the editor: the abstract and the body describe different experiments, with different model sets, different accuracy numbers, and no EfficientNet-B0 in the body despite its headline role in the abstract. This is not a presentation-only issue; it makes the central claims unverifiable. I also note that the augmentation assertions in Sections 2.3 and 3.3.1 appear without any supporting experiments, and the transfer-learning section mixes MobileNetV3 and EfficientNetV2 results. These issues cannot be fixed with local edits; the experimental record and the narrative would both need substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this one is not usable as-is. The abstract claims seven models, top-5 accuracy, macro F1, FP32 storage, MACs, and a specific EfficientNet-B0 comparison; the body reports five models, accuracy, F1, inference time, FLOPs, and model size, with no EfficientNet-B0 anywhere. The EfficientNetV2-S numbers don't match either: abstract says 97.57/86.98/78.73, body tables say 96.53/90.82/76.87. That's not a minor copyedit—it means the central claim is not supported by the reported experiment.\n\nWhat's legitimately there: the idea of a controlled comparison under a shared fine-tuning protocol is the right kind of question for practitioners choosing among these architectures. The paper also correctly notes that some architectures respond better to augmentation than others, though that point is asserted, not demonstrated. The tables, such as they are, show a plausible ranking for five models on three datasets: EfficientNetV2 highest accuracy, MobileNetV3 good balance, SqueezeNet fast but weak. If those tables were the whole paper, it would be a modest but acceptable empirical note.\n\nThe soft spots are load-bearing, not cosmetic. First, the pretrained-vs-scratch section is internally contradictory: Table 4 is headed MobileNetV3 but Figures 9–10 and Section 4.5 describe EfficientNetV2, and the 92.51% scratch number that is compared against pretrained EfficientNetV2's 96.53% actually comes from the MobileNetV3 scratch column. There is no matched EfficientNetV2 scratch run at all. Second, all results are single runs with no reported per-model learning rates, batch sizes, optimizers, schedulers, augmentation policies, or seeds. The rankings could easily be tuning artifacts, and the paper provides no way to check. Third, the augmentation claims (e.g., \"SqueezeNet and ShuffleNetV2 experienced degradation\") appear in the text without any ablation or table supporting them. No code is provided, so nothing is reproducible.\n\nWho is this for? A reader who wants a quick, informal glance at five public architectures on three standard datasets might skim the tables, but they cannot trust the headline numbers. This is not a paper I would send to a referee in its current state; it needs a rewrite that harmonizes abstract and body, fixes the transfer-learning section, reports full training configurations, adds error bars or multiple seeds, and releases code. Then it could become a modest but honest benchmark note.\n\nFor this version: reject. If the author is willing to do that revision, it would be worth a second look.","headline":"The abstract and the body report different experiments: the paper's leading claims about EfficientNetV2 and EfficientNet-B0 are untraceable to the presented results, and the pretrained-vs-scratch comparison is internally mislabeled, so the paper is not ready for peer review.","tokens_in":12625,"tokens_out":1147,"would_cite":false,"duration_ms":11890,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper benchmarks five lightweight CNNs under one shared protocol and claims EfficientNetV2-S leads in accuracy on all three datasets.","keywords":["lightweight CNN","EfficientNetV2-S","MobileNetV3-Small","CIFAR-10","CIFAR-100","Tiny ImageNet","transfer learning","resource-constrained deployment"],"falsifier":"Re-run the five models on the three datasets with every hyperparameter logged and fixed identically, including learning rate, batch size, optimizer, scheduler, augmentation policy, and early-stopping criterion; if EfficientNetV2-S no longer ranks first on all three, or if a scratch-trained model matches the pretrained one once trained beyond 50 epochs, the paper's rankings and pretraining conclusion are overturned.","tokens_in":11439,"feed_emoji":"📊","tokens_out":10080,"duration_ms":82367,"temperature":0.7,"pith_summary":"The paper sets out to make lightweight-CNN rankings interpretable by testing five architectures — MobileNetV3-Small, ResNet18, SqueezeNet, EfficientNetV2-S, and ShuffleNetV2 — on CIFAR-10, CIFAR-100, and Tiny ImageNet under a single fine-tuning protocol. It reports that EfficientNetV2-S reaches the highest top-1 accuracy on every dataset and that MobileNetV3-Small keeps competitive accuracy with a much smaller footprint, making it the recommended balance for edge deployments. A matched pretrained-versus-scratch comparison on MobileNetV3-Small shows transfer learning pays more as dataset difficulty grows. The practical stake is a like-for-like selection reference for deploying image classifiers on memory-constrained hardware.","feed_headline":"EfficientNetV2-S tops five lightweight CNNs in controlled benchmark","feed_subtitle":"MobileNetV3-Small nearly matches it at a fraction of the size, giving edge-device deployers a clear trade-off.","key_machinery":"The mechanism that carries the argument is the shared evaluation protocol: all models are fine-tuned for 50 epochs under the same preprocessing and augmentation stack (ImageNet normalization, random horizontal flips, random crops with padding, AutoAugment, CutMix, and MixUp), and compared on top-1 accuracy, macro F1, top-5 accuracy, parameter count, FP32 storage, and multiply-accumulate operations. The protocol is what turns the results from scattered literature numbers into an architecture ranking; the pretrained-versus-scratch arm isolates the effect of initialization by holding the rest of the pipeline fixed.","core_discovery":"The paper claims that architecture rankings among lightweight CNNs are only trustworthy when models are compared under a shared training recipe, and that doing so for five established models yields a clear ordering. EfficientNetV2-S tops all three datasets in the body's tables (96.53% on CIFAR-10, 90.82% on CIFAR-100, 76.87% on Tiny ImageNet), with MobileNetV3-Small close behind (95.49%, 89.62%, 72.54%) at roughly a tenth of the storage. The abstract states an even stronger headline — 97.57%, 86.98%, 78.73% — and an EfficientNet-B0 comparison within 0.85 points at about 21% of the parameters and 14% of the MACs on Tiny ImageNet, though B0 is absent from the body's tables. On the pretraining question, the paper reports that ImageNet-pretrained MobileNetV3-Small beats its randomly initialized counterpart by 3.23 points on CIFAR-10, 4.61 on CIFAR-100, and 7.90 on Tiny ImageNet, concluding that transfer learning matters most for harder, higher-class datasets.","pith_inferences":["The abstract's EfficientNet-B0 claim is not backed by the body's experiments; treating it as a confirmed result would require adding B0 to the same protocol. This is an editorial caution, not a paper claim.","The pretraining gap may be inflated by the fixed 50-epoch budget, which likely under-trains randomly initialized models; a test would be to train scratch models to convergence and see how much of the 7.90-point Tiny ImageNet gap remains.","The FLOPs and parameter counts in the paper are theoretical; on actual edge hardware, memory bandwidth and quantization could reorder models, so measuring on-device latency and energy would be a natural extension.","Extending the protocol to newer NAS-designed models such as FBNet or RegNet, and to int8-quantized versions of these five, would show whether the rankings survive deployment constraints."],"forward_implications":["On CIFAR-10 and CIFAR-100, the accuracy spread among EfficientNetV2-S, MobileNetV3-Small, and ShuffleNetV2 is small, so storage and speed can decide the choice without much accuracy cost.","For Tiny ImageNet-scale tasks, EfficientNetV2-S is the stronger accuracy pick, while SqueezeNet collapses to 20.5% and should be avoided when class count is high.","For ultra-low-resource deployments, MobileNetV3-Small is the paper's recommended default, since it keeps within about one to four points of the leader while using far fewer parameters and FLOPs.","Pretrained weights are worth more on harder datasets: the gap grows from 3.23 points on CIFAR-10 to 7.90 on Tiny ImageNet, so scratch training is most defensible for simple tasks.","Model size alone is not a proxy for accuracy: SqueezeNet is smallest and fastest but trails badly, while ResNet18 is fast but heavy."],"supporting_citations":[{"why":"Supplies the EfficientNetV2 architecture and pretrained checkpoint that the paper reports as the top-accuracy model.","marker":"[8]"},{"why":"Defines MobileNetV3, the architecture the paper identifies as the best accuracy-efficiency balance and uses for the pretrained-versus-scratch comparison.","marker":"[6]"},{"why":"Defines ShuffleNetV2, one of the compared baselines, and the memory-access-cost design principle the paper invokes.","marker":"[10]"},{"why":"Defines SqueezeNet, the ultra-lightweight baseline that anchors the low-size, low-accuracy end of the comparison.","marker":"[7]"},{"why":"Defines ResNet18, the residual baseline whose fast inference and larger storage the paper contrasts with the mobile-oriented models.","marker":"[9]"},{"why":"Supplies the CIFAR-10 and CIFAR-100 datasets used for the benchmark.","marker":"[13]"},{"why":"Supplies the transfer-learning evidence that pretrained models transfer better, which the pretraining arm is testing.","marker":"[11]"},{"why":"Supplies CutMix, one of the augmentation methods in the shared protocol.","marker":"[16]"},{"why":"Supplies MixUp, another augmentation method in the shared protocol.","marker":"[14]"}],"fun_headline_variants":["Controlled benchmark ranks lightweight CNNs: EfficientNetV2-S leads","EfficientNetV2-S tops, MobileNetV3-Small shines for ultra-low-resource","Matched training recipe changes lightweight CNN rankings","Pretraining boosts accuracy most on harder datasets for CNNs","Lightweight CNN trade-off: EfficientNetV2-S vs MobileNetV3-Small"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings depend on the assumption that the untracked per-model training details — final learning rates, batch sizes, optimizers, schedulers, and early-stopping choices — do not determine which model wins, so the reported 50-epoch protocol represents each architecture fairly.","fun_headline_variants_meta":{"raw":{"variants":["Controlled benchmark ranks lightweight CNNs: EfficientNetV2-S leads","EfficientNetV2-S tops, MobileNetV3-Small shines for ultra-low-resource","Matched training recipe changes lightweight CNN rankings","Pretraining boosts accuracy most on harder datasets for CNNs","Lightweight CNN trade-off: EfficientNetV2-S vs MobileNetV3-Small"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000449,"raw_usage":{"total_tokens":2344,"prompt_tokens":1107,"completion_tokens":1237,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":723,"completion_tokens_details":{"reasoning_tokens":1140}},"tokens_in":723,"tokens_out":1237,"duration_ms":10779,"temperature":1.0,"reasoning_tokens":1140,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:55:13.827721+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the five models on the three datasets with every hyperparameter logged and fixed identically, including learning rate, batch size, optimizer, scheduler, augmentation policy, and early-stopping criterion; if EfficientNetV2-S no longer ranks first on all three, or if a scratch-trained model matches the pretrained one once trained beyond 50 epochs, the paper's rankings and pretraining conclusion are overturned.","supporting_citations":[{"cited_title":"Efficientnetv2: Smaller models and faster training,","cited_arxiv_id":null,"evidence_quote":"Supplies the EfficientNetV2 architecture and pretrained checkpoint that the paper reports as the top-accuracy model."},{"cited_title":"Searching for mobilenetv3,","cited_arxiv_id":null,"evidence_quote":"Defines MobileNetV3, the architecture the paper identifies as the best accuracy-efficiency balance and uses for the pretrained-versus-scratch comparison."},{"cited_title":"Shufflenet v2: Practical guidelines for efficient cnn architecture design,","cited_arxiv_id":null,"evidence_quote":"Defines ShuffleNetV2, one of the compared baselines, and the memory-access-cost design principle the paper invokes."},{"cited_title":"Deep residual learning for image recognition,","cited_arxiv_id":null,"evidence_quote":"Defines ResNet18, the residual baseline whose fast inference and larger storage the paper contrasts with the mobile-oriented models."},{"cited_title":"Do better imagenet models transfer better?","cited_arxiv_id":null,"evidence_quote":"Supplies the transfer-learning evidence that pretrained models transfer better, which the pretraining arm is testing."},{"cited_title":"Cutmix: Regularization strategy to train strong classifiers with localizable features,","cited_arxiv_id":null,"evidence_quote":"Supplies CutMix, one of the augmentation methods in the shared protocol."},{"cited_title":"Mixup: Beyond empirical risk minimization,","cited_arxiv_id":null,"evidence_quote":"Supplies MixUp, another augmentation method in the shared protocol."}],"review_version":1}