{"id":"73569810-80c3-4c21-8c43-4fe3ddbd7bbe","arxiv_id":"1908.01924","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Edge AIBench defines an end-to-end benchmark suite with four edge AI scenarios spanning client devices, edge nodes, and cloud servers.","lead":"This paper introduces Edge AIBench, a benchmark suite that models four edge AI scenarios across client, edge, and cloud layers. It is a roadmap and preliminary framework, not yet a validated measurement study.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The four scenarios are not shown to exercise distinct data-distribution or workload-collaboration patterns, so the end-to-end claim and the 'comprehensive' label are unsubstantiated.","rationale":"The reader identified representativeness of the four scenarios as the weakest assumption. My stress-test finds an even more fundamental gap: the paper does not define or implement any data-distribution or workload-collaboration mechanism beyond a static train-on-cloud/infer-on-edge split. Even if the scenarios were chosen representatively, the benchmark artifact is absent and the end-to-end collaboration claim is not operationalized. This strengthens the reader's CONDITIONAL verdict rather than changing it: the framework needs concrete executable workloads with explicit collaboration patterns before the comprehensive end-to-end claim can be accepted. I partially agree with the reader because representativeness is a valid concern, but the lack of measurable workload collaboration is more load-bearing; without it, there is no benchmark to be comprehensive.","tokens_in":5368,"tokens_out":3939,"duration_ms":43642,"concrete_test":"Inspect the public Edge AIBench repository and testbed, then enumerate the executable component benchmarks and their data flows. Specifically, check whether all six component benchmarks are implemented as reproducible workloads, and for each, analyze the cross-layer interaction graph. If fewer than six workloads exist, or if every description reduces to 'client sends data to edge; edge sends inference results or preprocessed data to cloud; cloud trains,' with no additional mechanism such as model partitioning, federated averaging, or split inference, then the end-to-end workload-collaboration claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Edge AIBench is a comprehensive end-to-end edge AI benchmark that captures data distribution and workload collaboration across client, edge, and cloud. This claim requires the benchmark to actually realize such collaboration patterns. The scenario descriptions in Section 3 and Table 2, however, assign only coarse roles: the client generates data, the edge performs inference/preprocessing, and the cloud trains. No workload specifications, data distribution parameters, collaboration mechanisms, or metrics are defined. All six component benchmarks reduce to the same pipeline (client data → edge inference → cloud training), with the only variation being optional compression/decompression in the surveillance scenario. Thus, even if the four scenarios are representative applications, the described benchmark does not instantiate the end-to-end workload collaboration that is the paper's stated focus. The federated learning testbed in Section 3.5 is separate and currently only being implemented for the ICU scenario, so it does not support the claimed focus in the other scenarios. The 'comprehensive' label is also asserted without selection criteria (Section 3: 'These four AI scenarios can present the complexity...'), but this is secondary to the absence of any operational definition of end-to-end collaboration. The paper presents a roadmap, not a benchmark, so the central claim cannot be confirmed from the manuscript alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that edge-computing AI benchmarking must take an end-to-end view spanning client-side devices, the edge computing layer, and cloud servers, and introduces Edge AIBench, a BenchCouncil-coordinated open-source suite. The suite is said to model four application scenarios (ICU patient monitoring, surveillance camera, smart home, and autonomous vehicle) with six component benchmarks, and the paper also describes an edge-computing testbed with a federated learning framework. The manuscript is entirely descriptive: it names the datasets and models, assigns coarse roles to the three layers in Table 2, and points to public URLs, but it reports no workload specifications, metric definitions, experimental measurements, or validation results.","tokens_in":5559,"tokens_out":2864,"duration_ms":28592,"significance":"If the described suite were actually implemented and validated, the end-to-end perspective and the coordinated multi-layer benchmarking effort would be a useful contribution to the edge-AI benchmarking community. The paper's strengths are that it builds on established public datasets and models (MIMIC-III, LibriSpeech, LFW) and makes the suite available as part of the open-source AIBench project. However, as submitted, the manuscript is a proposal or roadmap rather than a completed benchmark study: the central claims of 'comprehensive' and 'end-to-end' are not supported by any operational definition, baseline, or experiment, so the significance currently rests on intention rather than demonstrated result.","major_comments":[{"comment":"The central claim that Edge AIBench realizes end-to-end data distribution and workload collaboration is not operationalized. Every component benchmark in Table 2 reduces to the same coarse tri-partition: the client generates data, the edge performs inference (optionally preceded or followed by compression in the surveillance scenario), and the cloud trains. No specification defines how data are distributed across layers, how workload collaboration is coordinated, what data rates, latencies, bandwidth constraints, or privacy mechanisms are exercised, or what metrics are collected. Without such specifications, a reader cannot determine whether the suite measures anything beyond a conventional cloud-training-plus-edge-inference pipeline, and the paper's stated focus on data distribution and workload collaboration remains unsubstantiated.","section":"Section 3, Tables 1-2"},{"comment":"The manuscript contains no experimental evidence that the suite runs, is reproducible, or produces useful measurements. For a benchmark paper, at minimum a description of the harness, a baseline configuration, and preliminary runs with reported metrics are needed; the conclusion merely restates the intended design. The claims that Edge AIBench 'models four typical application scenarios' and provides 'an end-to-end application benchmarking framework' cannot be confirmed from the manuscript alone.","section":"Section 4"},{"comment":"The federated learning testbed is described as separate from the four scenarios and as currently under implementation for the ICU scenario only ('At present, we are implementing the ICU scenario on the testbed'). Therefore the privacy and security contribution does not yet cover the other three scenarios, and the testbed does not currently substantiate the end-to-end collaboration claim for the suite as a whole.","section":"Section 3.5"},{"comment":"The sentence 'These four AI scenarios can present the complexity of edge computing AI scenarios from different perspectives' is asserted without selection criteria or a coverage argument. Since 'comprehensive' appears in the title and abstract, this is a load-bearing qualifier: the authors should either justify the scenario selection systematically or weaken the claim to 'four representative scenarios' rather than a comprehensive set.","section":"Section 3, first paragraph"}],"minor_comments":[{"comment":"The word 'e ort' appears to be missing the letter 'f' in the abstract text; this is likely a LaTeX/encoding error and should be corrected.","section":"Abstract"},{"comment":"The statement 'Unfortunately, the previous work ignores this most important point' is an overgeneralization; the cited prior benchmarks focus on particular layers or platforms, which is a design choice rather than a wholesale ignorance of the end-to-end perspective.","section":"Section 2"},{"comment":"In the EEMBC ML-Mark row, 'Not Clear' is repeated four times and should either be explained in a footnote or replaced with a more informative characterization of what the benchmark actually covers.","section":"Table 1"},{"comment":"The testbed URL is given as http://www.benchcouncil.org/testbed.html in Section 3.5 but as http://www.benchcouncil.org/testbed/index.php in the introduction; the URLs should be reconciled.","section":"Section 3.5"},{"comment":"The phrase 'The uniqueness of the autonomous vehicle scenario is that the high demand for validity' is grammatically awkward and should be rewritten, e.g., 'what is distinctive about the autonomous vehicle scenario is its high demand for validity.'","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"This is a short descriptive paper about an ongoing benchmarking effort. The topic is timely and the BenchCouncil context is relevant, but the manuscript currently reads as a position statement rather than a benchmark contribution. The major revisions I request are substantial because the central claims require either implementation evidence or a significant weakening of the claims. If the venue is a workshop or symposium that explicitly accepts work-in-progress reports, this may be acceptable; otherwise, the authors should add at least a minimal validation of one scenario end to end before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nRead this as a proposal/roadmap, not a benchmark. The gap is real: MLPerf inference, EEMBC MLMark, EdgeBench, and AI Benchmark each cover only part of the three-layer edge stack, and none ties client, edge, and cloud into one end-to-end workload. Edge AIBench is the first thing I've seen that explicitly tries to do that, with four plausible scenarios (ICU monitoring, surveillance, smart home, autonomous vehicle) built on established datasets and models. That alone is worth a look.\n\nWhat it does well: the scenario choices are sensible, the use of MIMIC-III, LibriSpeech, LFW, DeepSpeech2, FaceNet and the rest is sound, and the paper is honest that the testbed is still being implemented. The related-work table is coarse but not misleading.\n\nNow the soft spots. The central claim — that Edge AIBench is comprehensive and captures data distribution and workload collaboration across layers — is unsubstantiated. There are no workload specifications, no metric definitions, no baseline, no experiment. Table 2 is telling: every component reduces to the same pipeline (client generates data, edge infers/preprocesses, cloud trains), with the only real variation being compression/decompression in the surveillance scenario. That doesn't instantiate the 'collaboration' the abstract promises; it just places the same functions on different boxes. The four scenarios are asserted to be representative ('can present the complexity...') without any selection criteria, so 'comprehensive' is a label, not a result. The federated learning testbed is a separate effort, currently only being implemented for the ICU scenario, so it can't support the general claim either.\n\nNone of this is a sign of dishonesty. It's just an early-stage paper with big claims. The criticism of prior work is fair in direction, but it would carry more weight if Edge AIBench itself had even one measured run.\n\nWho is this for? People working on benchmarking methodology for edge AI, and reviewers who want to see what a benchmark proposal should include. It's not citable as a validated benchmark. I'd send it to peer review — a serious referee can force the authors to either add artifacts and measurements or pull the 'comprehensive' language. Without that, accept only as a position statement.","headline":"A well-intentioned roadmap for an end-to-end edge AI benchmark, but the 'comprehensive' claim and the criticism of prior work outrun the actual artifacts — no measurements, no workload specs, no metrics.","tokens_in":6115,"tokens_out":1989,"would_cite":false,"duration_ms":19832,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark suite treats edge AI as a three-layer pipeline.","keywords":["edge computing","AI benchmarks","end-to-end benchmarking","federated learning","testbed","ICU patient monitoring","smart home","autonomous vehicle"],"falsifier":"A survey or field study of deployed edge AI systems would settle the comprehensive claim: if a substantial class of systems (for example, augmented-reality glasses or industrial control loops) has a data-flow and training structure that does not fit any of the four scenarios' three-layer patterns, then the suite covers only part of edge AI. A smaller, direct check is to run a fifth scenario, chosen independently, through the Edge AIBench framework and see whether its data collection, edge inference, and cloud training steps are already expressible.","tokens_in":5138,"feed_emoji":"⚡","tokens_out":7740,"duration_ms":76764,"temperature":0.7,"pith_summary":"This paper sets out to establish that edge AI benchmarking must be end to end: a benchmark that measures only the edge layer, or only the client, misses the data distribution and workload collaboration that define edge computing. To support that claim, it presents Edge AIBench, a suite of four application scenarios—ICU patient monitoring, surveillance cameras, smart homes, and autonomous vehicles—each with a fixed split of work between client devices, the edge layer, and cloud servers. The suite pairs these scenarios with a federated-learning testbed meant to preserve privacy and security while training on realistic workloads. If the claim holds, Edge AIBench would be the first coordinated benchmark of the full three-layer edge AI pipeline, giving system builders a common way to measure latency, bandwidth, and deployment tradeoffs.","feed_headline":"Edge AIBench puts all three layers of edge AI into one benchmark","feed_subtitle":"Four realistic application scenarios and a federated-learning testbed make the full pipeline measurable.","key_machinery":"The central object is the three-layer end-to-end application framework: client-side devices generate data, the edge computing layer performs pre-processing and inference, and the cloud server trains or refines models. Edge AIBench's machinery is the mapping of each scenario's workload collaboration pattern onto that framework—including where data is compressed, where alarms or decisions are issued, and where training runs—plus a federated-learning testbed, that is, a scheme in which model parameters rather than raw data are shared for collaborative training while preserving privacy.","core_discovery":"On the paper's own terms, the central discovery is that previous edge AI benchmarking efforts miss the property that actually defines edge computing: the end-to-end pipeline across three layers. Edge AIBench operationalizes that view by modelling four typical scenarios and six component benchmarks—heart failure and endpoint prediction, person re-identification, speech and face recognition, and road-sign recognition—and by specifying which layer collects, compresses, infers, trains, and decompresses data in each scenario. The routine split follows a common pattern: high-latency or privacy-sensitive inference happens at the edge, full model training happens in the cloud, and raw data originates on client-side devices. The paper further claims that the four scenarios collectively represent the complexity of edge AI, and that the accompanying federated-learning testbed addresses the privacy and security concerns that otherwise prevent realistic shared benchmarks.","pith_inferences":["The paper leaves implicit that its three-layer design is also an architectural recommendation: moving inference to the edge and training to the cloud is presented as the natural collaboration pattern, but other valid splits are possible and would need separate benchmark scenarios.","A natural extension is to test whether the four workload-collaboration patterns are closed under the scenarios that actually appear in production edge AI, such as augmented reality or industrial control.","A quantitative extension would be to instrument each scenario with bandwidth and privacy cost metrics, so that the claimed tradeoffs between cloud and edge placement become measurable rather than descriptive."],"forward_implications":["System builders can compare edge AI deployments on the full data path, not just on single-device inference speed.","Latency, bandwidth, and privacy tradeoffs become explicit axes of evaluation, because the benchmark fixes where data is moved and processed.","The four scenario definitions give a common vocabulary for describing where a new edge AI application places its work.","The federated-learning testbed gives researchers a shared starting point for validating privacy-preserving training on realistic workloads."],"supporting_citations":[{"why":"Supplies the edge inference benchmark whose single-layer scope the paper contrasts with its end-to-end view.","marker":"[4]"},{"why":"Provides an embedded ML benchmark that is closed and in beta, justifying the open-suite design.","marker":"[6]"},{"why":"Shows a smartphone-only AI benchmark that covers only client-side devices, motivating the three-layer coverage.","marker":"[7]"},{"why":"Compares two edge platforms but lacks an end-to-end application framework, a gap Edge AIBench fills.","marker":"[11]"},{"why":"Supplies the critical-care database that grounds both ICU patient monitor benchmarks.","marker":"[12]"},{"why":"Supplies the speech recognition model used in the smart home scenario.","marker":"[15]"},{"why":"Supplies the face recognition model used in the smart home scenario.","marker":"[17]"},{"why":"Supplies the federated learning concept that the privacy-preserving testbed is built on.","marker":"[19]"}],"fun_headline_variants":["Edge AIBench: end-to-end benchmarking across all three layers","One benchmark, three layers: Edge AIBench covers the full pipeline","Edge AIBench fixes the missing end-to-end view in edge AI","Four edge AI scenarios, one testbed: Edge AIBench goes end-to-end","Benchmarking edge AI the right way: from device to cloud"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the four chosen scenarios—ICU monitoring, surveillance, smart home, and autonomous driving—cover enough of the real edge AI design space that a suite built on them can honestly be called comprehensive; if a common scenario needs a different pattern of data distribution and workload collaboration, that label would not survive.","fun_headline_variants_meta":{"raw":{"variants":["Edge AIBench: end-to-end benchmarking across all three layers","One benchmark, three layers: Edge AIBench covers the full pipeline","Edge AIBench fixes the missing end-to-end view in edge AI","Four edge AI scenarios, one testbed: Edge AIBench goes end-to-end","Benchmarking edge AI the right way: from device to cloud"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1548,"prompt_tokens":863,"completion_tokens":685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":592}},"tokens_in":479,"tokens_out":685,"duration_ms":6377,"temperature":1.0,"reasoning_tokens":592,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:59:02.257621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A survey or field study of deployed edge AI systems would settle the comprehensive claim: if a substantial class of systems (for example, augmented-reality glasses or industrial control loops) has a data-flow and training structure that does not fit any of the four scenarios' three-layer patterns, then the suite covers only part of edge AI. A smaller, direct check is to run a fifth scenario, chosen independently, through the Edge AIBench framework and see whether its data collection, edge inference, and cloud training steps are already expressible.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the edge inference benchmark whose single-layer scope the paper contrasts with its end-to-end view."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides an embedded ML benchmark that is closed and in beta, justifying the open-suite design."},{"cited_title":"In Proceedings of the European Conference on Computer Vision (ECCV) 2018 (p p","cited_arxiv_id":null,"evidence_quote":"Shows a smartphone-only AI benchmark that covers only client-side devices, motivating the three-layer coverage."},{"cited_title":"In 2018 IEEE/ACM International Conference on Utilit y and Cloud Comput- ing Companion (UCC Companion) 2018 Dec 17 (pp","cited_arxiv_id":null,"evidence_quote":"Compares two edge platforms but lacks an end-to-end application framework, a gap Edge AIBench fills."},{"cited_title":"Scientific data 3 (2016): 160035","cited_arxiv_id":null,"evidence_quote":"Supplies the critical-care database that grounds both ICU patient monitor benchmarks."},{"cited_title":"Deep speech 2: End-to- end speech recog- nition in english and mandarin","cited_arxiv_id":null,"evidence_quote":"Supplies the speech recognition model used in the smart home scenario."},{"cited_title":"Facenet: A unified em bedding for face recog- nition and clustering","cited_arxiv_id":null,"evidence_quote":"Supplies the face recognition model used in the smart home scenario."},{"cited_title":"ACM Transactions on Intelligent Systems and Technology (TIST)","cited_arxiv_id":null,"evidence_quote":"Supplies the federated learning concept that the privacy-preserving testbed is built on."}],"review_version":1}