{"id":"240f6efb-3b09-4b9b-b5ff-d97bdc948b1f","arxiv_id":"2505.02239","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Across three platforms, lattice-based PQC schemes (ML-KEM, ML-DSA, Falcon) offer the best execution-time and artifact-size trade-offs for consumer electronics, while code-based and hash-based schemes face practical size or speed barriers.","lead":"This paper benchmarks post-quantum cryptography (PQC) algorithms on a Mac, an Ubuntu desktop, and a Raspberry Pi, measuring execution times and key, ciphertext, and signature sizes. It concludes that lattice-based schemes such as ML-KEM and ML-DSA give the best speed-to-size balance for most consumer electronics, while Classic McEliece and SPHINCS+ carry heavy size or speed costs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Raspberry Pi 4 is not a valid proxy for MCU-class devices; Table IX's low-power recommendations (e.g., Falcon-512) are unsupported by measured data and the paper's own memory-measurement gap.","rationale":"The reader identified the Raspberry Pi 4 proxy as the weakest assumption, and my review agrees. The paper's own text confirms the gap: Section IV.E admits that RAM footprint could not be measured reliably, and Section VII item 1 lists microcontroller benchmarking as future work. These self-acknowledged limitations directly weaken the deployment recommendations in Table IX for 'low-power / highly constrained' devices, which is part of the paper's central practical contribution. The concern is an external validity gap rather than an internal inconsistency: the benchmark methodology is reasonable, and the qualitative conclusion about ML-KEM and ML-DSA on desktop and Pi-class hardware is consistent with prior literature. The paper also deserves credit for transparently listing the missing memory measurement and the MCU extrapolation. Since the concern is addressable and does not invalidate the core results for the platforms actually tested, keeping the CONDITIONAL verdict is appropriate. No additional concern about algorithm naming (liboqs 0.8.0 predates final FIPS 203/204) or library comparability appears to change the verdict materially.","tokens_in":29930,"tokens_out":7050,"duration_ms":80374,"concrete_test":"Run the same liboqs-based benchmarks for ML-KEM-512, ML-DSA-44, and Falcon-512 on a Cortex-M4 board (e.g., STM32F4, 128 KB SRAM) using a standard embedded PQC suite (e.g., pqm4/pqbench), measuring cycles and peak stack/RAM usage. If Falcon-512 key generation or signing exceeds available SRAM, or if the Pi-4-based ranking among these algorithms changes (e.g., ML-DSA-44 becomes relatively slower or faster), then Table IX's 'low-power / highly constrained' row and the 'most consumer device classes' takeaway require revision. Also compute the ratio of MCU-to-Pi4 times; if it exceeds roughly 10x for any recommended algorithm, the Pi-4 proxy is not representative of that device class.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ML-KEM/ML-DSA are suitable for many CE applications, and the actionable takeaway to prioritize them across most consumer device classes (including 'low-power / highly constrained' devices in Table IX), rests on benchmarks where the only constrained platform is a Raspberry Pi 4 (Cortex-A72, 4 GB RAM, Linux). Cortex-M-class MCUs in wearables and smart locks have SRAM measured in tens-to-hundreds of KB, no cache or MMU, and no NEON SIMD; their per-operation time and RAM footprint are not derivable from Pi 4 results. The paper's own limitations confirm this: Section IV.E states that reliable RAM measurement was not achieved, so 'memory footprint indicators' are only key/signature sizes, and Section VII item 1 defers microcontroller benchmarking to future work. In particular, Table IX recommends Falcon-512 as primary for MCUs, but Falcon's reference implementation has substantial stack/RAM requirements that may exceed typical MCU SRAM; no measurement in this paper supports that recommendation. Consequently, the deployment guidance for the most resource-constrained device classes overreaches the data, even though the qualitative conclusion for desktops and moderately capable embedded platforms is well supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper benchmarks a broad set of post-quantum KEMs and signature schemes—ML-KEM, ML-DSA, Falcon, SPHINCS+, Classic McEliece, BIKE, HQC, FrodoKEM, MAYO, and CROSS/RSDP—against classical RSA, ECDH/ECDSA, and Ed25519 on three platforms: an Apple M4 Mac mini, an Intel i7-11700 Ubuntu desktop, and a Raspberry Pi 4. It reports execution times, communication/storage artifact sizes, message-size scaling behavior, and derives deployment recommendations in Table IX for several consumer-electronics device classes. The central claim is that NIST-standardized lattice schemes, particularly ML-KEM and ML-DSA, offer the best practical balance of computational efficiency and moderate communication/storage overhead for many consumer electronics applications, while Classic McEliece and SPHINCS+ have size/bandwidth drawbacks.","tokens_in":30183,"tokens_out":5844,"duration_ms":74909,"significance":"If taken at face value, the paper is a useful broad benchmark study for practitioners: it covers both KEMs and signatures, includes multiple NIST security levels, reports per-operation means and standard deviations, uses warm-up and 1000 iterations for most measurements, and makes the benchmark source code available. The qualitative finding that lattice schemes dominate on the time/size trade-off is consistent with the reported data and with the wider PQC literature. The most novel contribution, however, is the device-class-specific deployment guidance, and that guidance is only partially supported: the only 'constrained' platform is a Raspberry Pi 4, and the paper explicitly does not measure RAM footprint. The central claim for desktops and moderately capable embedded Linux devices is well supported, but the recommendations for the most resource-constrained class overreach the evidence.","major_comments":[{"comment":"The Raspberry Pi 4 is described as a proxy for 'resource-constrained CE devices' in Section IV-A, and Table IX extends recommendations to 'Low-power / highly constrained (wearables, smart locks, BLE gadgets, MCUs)'. A Cortex-A72 at 1.5 GHz with 4 GB LPDDR4 RAM running Linux is not representative of Cortex-M-class MCUs, which typically have tens to hundreds of kilobytes of SRAM, no MMU, no NEON, and no operating-system-level caching. The paper presents no measurements on such hardware, and Section VII item 1 explicitly lists microcontroller benchmarking as future work. Consequently, the recommendation of Falcon-512 and ML-KEM-512 for the most constrained device class is unsupported by the empirical results, even though those choices are plausible for the Pi-like 'moderately constrained' class.","section":"Section IV-A and Table IX"},{"comment":"The abstract and Section I list memory footprint as an evaluated metric, but Section IV-E states that reliable cross-platform RAM measurement was not achieved and that only 'memory footprint indicators' are used, i.e., public-key/ciphertext/signature sizes. For low-memory devices, stack and heap consumption during operations are decisive; this is especially relevant to Falcon-512, whose reference implementation has substantial RAM requirements that may exceed typical MCU SRAM. The deployment recommendations for low-power devices therefore rest on incomplete evidence. Either add stack/heap measurements on a relevant constrained platform or explicitly narrow the memory-related claims and Table IX accordingly.","section":"Section IV-E and Section VII item 7"},{"comment":"The aggregate slowdown figures in Table II, such as '45-50x slower on the Raspberry Pi', average over very heterogeneous algorithms (e.g., FrodoKEM slowdowns exceeding 100x, ML-KEM slowdowns near 50x, McEliece decapsulation near 19x). While the per-algorithm tables in the appendix mitigate the risk, the main text uses these averages as headline numbers without noting that they are dominated by the slowest families. The authors should either report family-specific aggregates in Section V-A or explicitly caution that these averages are not representative of any single algorithm.","section":"Section V-A and Tables II, XII, XIII"}],"minor_comments":[{"comment":"The consolidation logic maps historical Kyber/Dilithium entries to ML-KEM/ML-DSA names and keeps only the 'official standard' entry. Since the benchmark uses liboqs 0.8.0, the authors should clarify whether those official entries are final FIPS 203/204 implementations or aliases for round-3 Kyber/Dilithium parameter sets, because this affects the exactness of the claim that NIST-standardized algorithm performance was measured.","section":"Section IV-C"},{"comment":"The text cites the 'SUPERCOP project [17]', but reference [17] is an ePrint report on analyzing reference post-quantum software, not the SUPERCOP benchmarking suite. The citation should be corrected or replaced.","section":"Section II, reference [17]"},{"comment":"For very fast operations such as ML-KEM key generation (around 0.01 ms on macOS), Python wrapper overhead may be non-negligible relative to the measured operation. The paper would benefit from reporting the measured overhead of an empty Python call and providing medians or confidence intervals for these sub-millisecond operations.","section":"Section IV-B and Table X"},{"comment":"The observation that message-size scaling affects lattice verification more than SPHINCS+ verification is presented as an algorithm-family property, but much of the increase is attributable to the cost of hashing the larger message, which affects all schemes. Clarifying this would prevent readers from over-interpreting the lattice-specific scaling behavior.","section":"Section V-G and Figure 10"},{"comment":"Table VI contains over thirty columns and is very difficult to read. Splitting it into separate KEM and signature tables, or transposing it so that operations are rows and algorithms are columns, would substantially improve usability.","section":"Table VI"}],"recommendation":"major_revision","confidential_remarks":"The paper's core benchmark data are valuable and the main qualitative claim about lattice-based schemes is credible. The revision should focus on aligning the deployment recommendations with the platforms actually measured; the current Table IX overreaches for the MCU/wearable class. The GitHub repository was not independently verified during review, so the reproducibility statements are taken at face value."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a confirmatory benchmark with a usable dataset, and one genuine overreach: it recommends algorithms for MCU-class wearables and smart locks from measurements taken on a Raspberry Pi 4, which is a 4GB Linux device with a Cortex-A72. For the platforms they actually measured, the paper is solid and the qualitative conclusions hold. The authors tested a wide suite—ML-KEM, ML-DSA, Falcon, SPHINCS+, BIKE, HQC, Classic McEliece, FrodoKEM, MAYO, RSDP, plus RSA/ECC baselines—on three platforms using liboqs 0.8.0 and OpenSSL 3.3.1. The timing methodology (1000 iterations, warm-up, mean and standard deviation) is reasonable, and they shipped the benchmarking code publicly. The consolidation of historical Kyber/Dilithium names into the NIST standard names is handled carefully, and the message-size scaling analysis is genuinely useful for firmware-update and data-logging scenarios. If you work on applied crypto for CE, this is a useful reference point.\n\nThe soft spots are real but, for the most part, proportionate. The paper states in Section IV.E that reliable RAM measurement was not achieved, so 'memory footprint indicators' are really just key and signature sizes. That should be in the abstract, not buried in methodology. More importantly, Table IX recommends Falcon-512 and ML-KEM-512 for 'low-power / highly constrained' devices (wearables, smart locks, MCUs), but the only constrained platform measured is a Raspberry Pi 4. The paper itself lists microcontroller benchmarking as future work in Section VII, so the recommendation for the most resource-constrained classes rests on an unsupported analogy. That is a load-bearing flaw for the paper's stated purpose, but it is correctable by re-framing the deployment table or adding true MCU measurements. The Python wrapper timing is a minor concern: since all algorithms go through the same wrapper, relative ordering is probably trustworthy, but absolute milliseconds should be treated as indicative, not definitive.\n\nThe central argument—lattice-based NIST standards are the practical default for most consumer devices, while McEliece's keys and SPHINCS+'s signatures create real integration friction—is well supported by the measured data and consistent with prior literature. The new contribution is the cross-platform dataset and the concrete recommendations, not a new research direction. The citation pattern is fair: they cite the relevant benchmark studies and standards. This is a paper for engineers and product managers more than for cryptography researchers, and it deserves a serious referee. I would ask for a revision that distinguishes measured device classes from inferred ones and expands the memory-footprint caveat.","headline":"A useful confirmatory benchmark on desktop and Linux-capable ARM, but the recommendation table overreaches when it applies Pi 4 results to MCU-class devices.","tokens_in":30704,"tokens_out":2667,"would_cite":false,"duration_ms":30854,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A cross-platform benchmark of post-quantum algorithms finds that the standardized lattice schemes ML-KEM and ML-DSA offer the best speed-versus-size balance for consumer electronics, with Falcon-512 as the compact option for low-power…","keywords":["post-quantum cryptography","consumer electronics","cryptographic performance benchmarking","key encapsulation mechanisms","digital signatures","ML-KEM","ML-DSA","resource-constrained devices"],"falsifier":"Measure ML-KEM-512, ML-DSA-44, and Falcon-512 on a Cortex-M-class microcontroller with realistic RAM and energy budgets: if key generation or signing exceeds the latency or memory available on such a chip, or if verification of a 1 kB update takes longer than the device's duty cycle allows, the paper's recommendations for the most constrained device class would not transfer.","tokens_in":29785,"feed_emoji":"🔐","tokens_out":7558,"duration_ms":91723,"temperature":0.7,"pith_summary":"This paper tries to establish which post-quantum cryptographic algorithms are actually viable for consumer electronics, where processing power, memory, and bandwidth are limited. It reports cross-platform measurements of key encapsulation mechanisms and digital signatures on a high-end Apple M4 system, an Intel x86 desktop, and a Raspberry Pi 4, and compares the results with RSA and ECC baselines. The central finding is that standardized lattice schemes—ML-KEM for key exchange and ML-DSA for signatures—offer the best balance of speed and moderate key/signature sizes across these platforms, making them the most practical near-term choices for most consumer devices. The paper then turns that data into concrete per-device-class recommendations, such as Falcon-512 for low-power wearables and ML-KEM-768/1024 for phones and laptops.","feed_headline":"Benchmark: lattice PQC schemes best fit consumer devices","feed_subtitle":"Desktops and Raspberry Pi tests show ML-KEM and ML-DSA balance speed and size for phones, hubs, and wearables.","key_machinery":"The carrier of the argument is the cross-platform benchmark itself: a Python harness wrapping a portable C implementation of the PQC algorithms, with classical RSA and ECC baselines supplied by the platform's OpenSSL-backed cryptography stack, each operation repeated roughly 1000 times after warm-up and reported as mean timings. On top of the raw timings, the paper builds a resource trade-off analysis that plots total operation time against communication/storage size (public key plus ciphertext or signature) on the Raspberry Pi, which is what turns isolated timings into a deployment argument. A naming-consolidation step maps the historical Kyber and Dilithium parameter sets onto the official ML-KEM and ML-DSA standard names so that the graphs compare final standardized specifications rather than submission variants.","core_discovery":"On the paper's own terms, the discovery is that lattice-based standards are the sweet spot for consumer electronics: ML-KEM key generation, encapsulation, and decapsulation complete in under 0.02 ms on the reference macOS platform, and even on the Raspberry Pi the schemes slow down by roughly 30–50× but remain usable, while their public keys and ciphertexts stay in the 0.8–1.6 kB range. ML-DSA similarly offers signing around 0.3 ms and verification around 0.2 ms at security level 3 on the desktop class, with signatures between 2.4 and 4.6 kB. By contrast, Classic McEliece public keys run to 1 MB or more, and SPHINCS+ signatures run to tens of kilobytes, with signing hundreds of milliseconds or slower. From these measurements the paper concludes that manufacturers should prioritize ML-KEM and ML-DSA for most product classes, treat Falcon as the compact-signature option, and reserve SPHINCS+ for firmware verification where signing happens offline.","pith_inferences":["A natural testable extension is to run the same suite on ARM Cortex-M-class microcontrollers; if ML-KEM-512 or Falcon-512 exceeds the memory or energy budget there, the recommendations for the most constrained device class would need scaling down to lighter or hardware-accelerated variants.","The paper's slowdown ratios suggest that energy consumption on constrained devices will track execution time roughly linearly, but direct battery measurements are the only way to confirm which algorithms actually extend or shorten device life.","Because lattice verification times grow with message size in the data, designs that frequently verify large payloads on-device may want to pre-hash messages or use hash-based signatures even when lattice signatures look faster on small messages.","The cross-platform ratio data also hints that architecture choice matters as much as algorithm choice: on the x86 platform, FrodoKEM and NTRU-Prime fared better relative to the ARM-based macOS, so a manufacturer targeting one CPU family could pick differently than one shipping both."],"forward_implications":["Manufacturers of smart-home hubs, gateways, and similar mid-range devices can adopt ML-KEM-512/768 with ML-DSA-44/65 without redesigning their networks around large keys or slow handshakes.","Low-power wearables and microcontroller-class devices should start with ML-KEM-512 and Falcon-512, which carry the smallest latency and size overhead among the robust post-quantum options, with hybrid classical ECDH easing backward compatibility.","SPHINCS+ is best reserved for secure-boot and firmware-verification paths, where the slow signing is done once offline and the large signature is outweighed by fast verification and hash-based security assurance.","Classic McEliece and FrodoKEM present integration hurdles for typical consumer gear because of megabyte-scale public keys or large ciphertexts, respectively, absent dedicated hardware support.","Moving from security level 1 to level 3 or 5 costs a moderate time penalty for lattice schemes but a larger size penalty, so long-lived devices should weigh security margin against bandwidth and storage."],"supporting_citations":[{"why":"Supplies the portable C implementations of the post-quantum algorithms used for all measurements in the benchmark suite.","marker":"[52]"},{"why":"Defines the ML-DSA digital signature standard that the paper benchmarks under its official name.","marker":"[9]"},{"why":"Defines the SPHINCS+/SLH-DSA stateless hash-based signature standard whose variants are measured.","marker":"[10]"},{"why":"Documents the selection status of the standardized and candidate algorithms that the benchmark suite draws from.","marker":"[8]"},{"why":"Provides the earlier x86-focused TLS benchmarking approach that this study extends to constrained ARM platforms.","marker":"[11]"},{"why":"Gives the TLS 1.3 handshake performance baseline for connected-device comparisons that this work updates with post-quantum algorithms.","marker":"[12]"},{"why":"Prior IoT-focused KEM comparison on ESP32 and Raspberry Pi whose algorithm coverage this work broadens to signatures and the full standard suite.","marker":"[15]"},{"why":"Benchmarks post-quantum KEMs on Cortex-M4 microcontrollers, highlighting the very low-end device class that this paper's Raspberry Pi proxy is meant to approximate but does not directly test.","marker":"[19]"}],"fun_headline_variants":["ML-KEM and ML-DSA top PQC speed-size tests on consumer gear","Lattice PQC schemes win on speed and size for phones and Pi","Kyber and Dilithium are the PQC sweet spot for consumer devices","PQC on Raspberry Pi: lattice schemes beat McEliece and SPHINCS+","For wearables and hubs: ML-KEM, ML-DSA offer best balance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Raspberry Pi 4 is a fair proxy for resource-constrained consumer electronics, so slowdowns measured on it can be extrapolated to wearables and microcontrollers; the paper itself lists true microcontroller benchmarking as future work.","fun_headline_variants_meta":{"raw":{"variants":["ML-KEM and ML-DSA top PQC speed-size tests on consumer gear","Lattice PQC schemes win on speed and size for phones and Pi","Kyber and Dilithium are the PQC sweet spot for consumer devices","PQC on Raspberry Pi: lattice schemes beat McEliece and SPHINCS+","For wearables and hubs: ML-KEM, ML-DSA offer best balance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000903,"raw_usage":{"total_tokens":3911,"prompt_tokens":997,"completion_tokens":2914,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":2809}},"tokens_in":613,"tokens_out":2914,"duration_ms":22003,"temperature":1.0,"reasoning_tokens":2809,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:56:40.213111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure ML-KEM-512, ML-DSA-44, and Falcon-512 on a Cortex-M-class microcontroller with realistic RAM and energy budgets: if key generation or signing exceeds the latency or memory available on such a chip, or if verification of a 1 kB update takes longer than the device's duty cycle allows, the paper's recommendations for the most constrained device class would not transfer.","supporting_citations":[{"cited_title":"Post-quantum Key Exchange for the Internet and the Open Quantum Safe Project,","cited_arxiv_id":null,"evidence_quote":"Supplies the portable C implementations of the post-quantum algorithms used for all measurements in the benchmark suite."},{"cited_title":"Module-lattice- based digital signature standard,","cited_arxiv_id":null,"evidence_quote":"Defines the ML-DSA digital signature standard that the paper benchmarks under its official name."},{"cited_title":"Stateless hash-based digital signature standard,","cited_arxiv_id":null,"evidence_quote":"Defines the SPHINCS+/SLH-DSA stateless hash-based signature standard whose variants are measured."},{"cited_title":"Status Report on the Third Round of the NIST Post-Quantum Cryptography Standardization Process,","cited_arxiv_id":null,"evidence_quote":"Documents the selection status of the standardized and candidate algorithms that the benchmark suite draws from."},{"cited_title":"Benchmarking Post-quantum Cryptography in TLS,","cited_arxiv_id":null,"evidence_quote":"Provides the earlier x86-focused TLS benchmarking approach that this study extends to constrained ARM platforms."},{"cited_title":"Post-quantum authentication in TLS 1.3: a performance study,","cited_arxiv_id":null,"evidence_quote":"Gives the TLS 1.3 handshake performance baseline for connected-device comparisons that this work updates with post-quantum algorithms."},{"cited_title":"A Comparative study of post-quantum cryptosystems for Internet-of-Things applications,","cited_arxiv_id":null,"evidence_quote":"Prior IoT-focused KEM comparison on ESP32 and Raspberry Pi whose algorithm coverage this work broadens to signatures and the full standard suite."}],"review_version":1}