pith. sign in

arxiv: 1907.10890 · v1 · pith:U5IAWOJSnew · submitted 2019-07-25 · 💻 cs.DC

DeFog: Fog Computing Benchmarks

Jonathan McChesney , Nan Wang , Ashish Tanwer , Eyal de Lara , Blesson Varghese This is my paper

Pith reviewed 2026-05-24 16:19 UTC · model grok-4.3

classification 💻 cs.DC
keywords fog computingbenchmarking suiteedge computingperformance metricscloud-edge deploymentapplication latencyworkload evaluation
0
0 comments X

The pith

DeFog supplies the first standard benchmark suite for comparing application performance across cloud-only, edge-only and cloud-edge fog deployments.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents DeFog to remove the need for custom benchmarking setups when testing fog computing applications. It supplies six applications that are well suited to edge resources and a fixed methodology for running them on different platform combinations. Experiments collect measurements of communication and computation latencies, plus the effects of load and multiple users. The results let developers see which service placements across cloud and edge resources actually reduce overall response times. The suite is released publicly so others can repeat and extend the measurements on their own hardware.

Core claim

DeFog is a fog benchmarking suite that applies a uniform methodology to run six edge-suited applications on cloud-only, edge-only and hybrid platforms, while recording a catalogue of metrics that includes communication latency, computation latency, and the influence of stress and concurrent users on those latencies.

What carries the argument

DeFog benchmarking suite of six applications together with a repeatable measurement protocol that records latency and resource metrics under varied cloud-edge service placements.

If this is right

  • Developers can directly measure whether splitting an application between cloud and edge resources reduces end-to-end latency compared with cloud-only execution.
  • The collected metrics quantify how concurrent users and background stress change observed latencies on each platform combination.
  • A public catalogue of results makes it possible to rank different hardware and network configurations for a given workload.
  • Repeated runs on new target platforms reveal whether adding edge nodes improves, degrades or leaves unchanged the performance of each benchmark.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the suite gains adoption it could become a reference point for reporting fog performance results, much as SPEC or TPC benchmarks function for servers.
  • Future versions would need to add workloads whose communication patterns or data volumes differ from the initial six to test broader applicability.
  • The metric catalogue could later be used to drive automated placement algorithms that choose cloud versus edge locations without manual trial runs.

Load-bearing premise

The six chosen applications are representative of the workloads that typically benefit from fog deployments.

What would settle it

Obtaining markedly different latency and placement rankings when the same DeFog protocol is applied to a fresh collection of applications not among the original six.

Figures

Figures reproduced from arXiv: 1907.10890 by Ashish Tanwer, Blesson Varghese, Eyal de Lara, Jonathan McChesney, Nan Wang.

Figure 1
Figure 1. Figure 1: DeFog deployment modes, namely cloud-only, edge￾only and cloud-edge (Fog). who want to demonstrate the benefit of using the Fog via bench￾marks, (ii) Internet Service Providers (ISPs) who want to deploy micro data centres at the edge of the network and want to tabulate performance of Fog applications for their customers, (iii) system software administrators who want to investigate the impact on Fog applica… view at source ↗
Figure 3
Figure 3. Figure 3: The DeFog edge-only deployment mode [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. Figure 2: shows the two tier cloud-only deployment mode compris￾ing the cloud resources and user devices. In DeFog, the application is built and deployed on the cloud resource using Docker containers. The application is then executed in the container and the benchmark metrics are generated. The method adopted is based on a container￾based cloud benchmarking approach previously reported [28]. The user device uses the… view at source ↗
Figure 5
Figure 5. Figure 5: These single board computers have resources comparable [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Latencies of applications for different deployment modes. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Latencies of RealFD for different combination of [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Communication latency of benchmark applica [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Latencies of the RealFD benchmark when the [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Impact of edge resource stress on response latency [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Impact of concurrent users on latency of benchmark applications. [PITH_FULL_IMAGE:figures/full_fig_p011_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Impact of concurrent users on average bytes transferred. [PITH_FULL_IMAGE:figures/full_fig_p011_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Impact of concurrent users on real time factor for [PITH_FULL_IMAGE:figures/full_fig_p011_14.png] view at source ↗
read the original abstract

Fog computing envisions that deploying services of an application across resources in the cloud and those located at the edge of the network may improve the overall performance of the application when compared to running the application on the cloud. However, there are currently no benchmarks that can directly compare the performance of the application across the cloud-only, edge-only and cloud-edge deployment platform to obtain any insight on performance improvement. This paper proposes DeFog, a first Fog benchmarking suite to: (i) alleviate the burden of Fog benchmarking by using a standard methodology, and (ii) facilitate the understanding of the target platform by collecting a catalogue of relevant metrics for a set of benchmarks. The current portfolio of DeFog benchmarks comprises six relevant applications conducive to using the edge. Experimental studies are carried out on multiple target platforms to demonstrate the use of DeFog for collecting metrics related to application latencies (communication and computation), for understanding the impact of stress and concurrent users on application latencies, and for understanding the performance of deploying different combination of services of an application across the cloud and edge. DeFog is available for public download (https://github.com/qub-blesson/DeFog).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces DeFog, the first benchmarking suite for fog computing. It consists of six applications and aims to (i) alleviate benchmarking burden via a standard methodology for comparing cloud-only, edge-only, and hybrid deployments and (ii) facilitate platform understanding by collecting a catalogue of metrics (primarily latencies under varying stress and concurrency). Experimental studies on multiple target platforms are described to illustrate these uses, and the suite is released publicly on GitHub.

Significance. If the six applications prove representative of fog workloads, DeFog could standardize evaluation practices in edge-cloud systems and reduce ad-hoc benchmarking effort. The public GitHub release supports reproducibility and adoption. The work addresses a genuine gap, as no prior direct comparison benchmarks for the three deployment modes are cited.

major comments (2)
  1. [Abstract] Abstract: The central claim that the collected metrics 'facilitate the understanding of the target platform' rests on the premise that the six applications are 'relevant' and 'conducive to using the edge.' No selection criteria, coverage argument, or mapping to workload dimensions (latency sensitivity, data volume, concurrency) is provided. This is load-bearing for both stated contributions.
  2. [Abstract] Abstract (experimental studies paragraph): The manuscript states that studies were performed to demonstrate metric collection, stress impact, and deployment combinations, yet the abstract supplies no concrete results, tables of measured latencies, or analysis showing actionable insights. Without such evidence the demonstration remains descriptive.
minor comments (1)
  1. The GitHub link is given but no commit hash or version tag is supplied, which would aid reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address the two major comments on the abstract point by point below, proposing targeted revisions where appropriate.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that the collected metrics 'facilitate the understanding of the target platform' rests on the premise that the six applications are 'relevant' and 'conducive to using the edge.' No selection criteria, coverage argument, or mapping to workload dimensions (latency sensitivity, data volume, concurrency) is provided. This is load-bearing for both stated contributions.

    Authors: We agree the abstract would be strengthened by briefly indicating the basis for application selection. The manuscript body (Section 3) motivates each of the six applications by their latency sensitivity, data locality needs, and suitability for edge offloading, drawing from representative fog use cases. We will revise the abstract to include a short clause on selection criteria and a high-level mapping to the mentioned workload dimensions. revision: yes

  2. Referee: [Abstract] Abstract (experimental studies paragraph): The manuscript states that studies were performed to demonstrate metric collection, stress impact, and deployment combinations, yet the abstract supplies no concrete results, tables of measured latencies, or analysis showing actionable insights. Without such evidence the demonstration remains descriptive.

    Authors: We accept that the current abstract remains at a descriptive level regarding the experimental studies. The full paper reports quantitative latency results across platforms, stress levels, and deployment modes. We will revise the abstract to incorporate one or two key illustrative findings (e.g., latency differences observed between deployment modes) to convey the nature of the actionable insights obtained. revision: yes

Circularity Check

0 steps flagged

No circularity: proposal of benchmark suite with no derivation chain

full rationale

The paper introduces DeFog as a new benchmarking suite and standard methodology for fog computing, selecting six applications as a portfolio without any equations, fitted parameters, predictions, or self-citational reductions. No load-bearing step reduces a claimed result to its own inputs by construction; the work is self-contained as a tool proposal rather than a derived claim.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The paper introduces a new benchmarking tool and methodology. It does not introduce fitted parameters, new physical entities, or non-standard mathematical axioms beyond the domain assumption that the chosen applications are representative.

axioms (1)
  • domain assumption The six applications are relevant and conducive to using the edge
    Stated directly in the abstract as the basis for the benchmark portfolio.

pith-pipeline@v0.9.0 · 5737 in / 1207 out tokens · 23743 ms · 2026-05-24T16:19:18.131085+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages · 1 internal anchor

  1. [1]

    D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi, P. O. Frederickson, T. A. Lasinski, R. S. Schreiber, H. D. Simon, V. Venkatakrishnan, and S. K. Weeratunga. 1991. The NAS Parallel Benchmarks - Summary and Preliminary Results. In Proceedings of the ACM/IEEE Conference on Supercomputing. 158–165

    work page 1991

  2. [2]

    Bermbach, J

    D. Bermbach, J. Kuhlenkamp, A. Dey, A. Ramachandran, A. Fekete, and S. Tai

    work page

  3. [3]

    In Proceedings of the 15th International Conference on Service-Oriented Computing

    BenchFoundry: A Benchmarking Framework for Cloud Storage Services. In Proceedings of the 15th International Conference on Service-Oriented Computing

    work page

  4. [4]

    Bonomi, R

    F. Bonomi, R. Milito, J. Zhu, and S. Addepalli. 2012. Fog Computing and Its Role in the Internet of Things. In Proceedings of the First Edition of the MCC Workshop on Mobile Cloud Computing . 13–16

    work page 2012

  5. [5]

    Z. Chen, W. Hu, J. Wang, S. Zhao, B. Amosand G. Wu, K. Ha, K. Elgazzar, P. Pillai, R. Klatzky, D. Siewiorek, and M. Satyanarayanan. 2017. An Empirical Study of Latency in an Emerging Class of Edge Computing Applications for Wearable Cognitive Assistance. In Proceedings of the 2nd ACM/IEEE Symposium on Edge Computing. Article 14, 14:1–14:14 pages

    work page 2017

  6. [6]

    B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears. 2010. Bench- marking Cloud Serving Systems with YCSB. In Proceedings of the 1st ACM Sym- posium on Cloud Computing . 143–154

    work page 2010

  7. [7]

    A. Das, S. Patterson, and M. P. Wittie. 2018. EdgeBench: Benchmarking Edge Com- puting Platforms. In Proceedings of the 4th International Workshop on Serverless Computing

    work page 2018

  8. [8]

    J. J. Dongarra, P. Luszczek, and A. Petitet. 2003. The LINPACK Benchmark: Past, Present and Future. Concurrency and Computation: Practice and Experience 15, 9 (2003), 803–820

    work page 2003

  9. [9]

    Gupta, A

    H. Gupta, A. V. Dastjerdi, S. K. Ghosh, and R. Buyya. 2017. iFogSim: A Toolkit for Modeling and Simulation of Resource Management Techniques in the Internet of Things, Edge and Fog Computing Environments.Software: Practice and Experience 47, 9 (2017), 1275–1296

    work page 2017

  10. [10]

    Hasenburg, M

    J. Hasenburg, M. Grambow, E. Grunewald, S. Huk, and D. Bermbach. 2019. Mock- Fog: Emulating Fog Computing Infrastructure in the Cloud. In IEEE International Conference on Fog Computing . 192–201

    work page 2019

  11. [11]

    C. H. Hong and B. Varghese. 2019. Resource Management in Fog/Edge Computing: A Survey on Architectures, Infrastructure, and Algorithms. Comput. Surveys (2019)

    work page 2019

  12. [12]

    Huggins-Daines, M

    D. Huggins-Daines, M. Kumar, A. Chan, A. W. Black, M. Ravishankar, and A. I. Rudnicky. 2006. Pocketsphinx: A Free, Real-Time Continuous Speech Recognition System for Hand-Held Devices. InProceedings of the IEEE International Conference on Acoustics Speech and Signal Processing

    work page 2006

  13. [13]

    Z. Jia, L. Wang, J. Zhan, L. Zhang, and C. Luo. 2013. Characterizing Data Anal- ysis Workloads in Data Centers. In IEEE International Symposium on Workload Characterization. 66–76

    work page 2013

  14. [14]

    Kasture and D

    H. Kasture and D. Sanchez. 2016. Tailbench: A Benchmark Suite and Evaluation Methodology for Latency-critical Applications. In IEEE International Symposium on Workload Characterization

    work page 2016

  15. [15]

    C. P. Kruger and G. P. Hancke. 2014. Benchmarking Internet of Things Devices. In 12th IEEE International Conference on Industrial Informatics (INDIN) . 611–616

    work page 2014

  16. [16]

    Z. Li, L. O’Brien, H. Zhang, and R. Cai. 2012. On a Catalogue of Metrics for Eval- uating Commercial Cloud Services. In ACM/IEEE 13th International Conference on Grid Computing. 164–173

    work page 2012

  17. [17]

    M. M. Lopes, W. A. Higashino, M. A. M. Capretz, and L. F. Bittencourt. 2017. MyiFogSim: A Simulator for Virtual Machine Migration in Fog Computing. In Companion Proceedings of the 10th International Conference on Utility and Cloud Computing. 47–52

    work page 2017

  18. [18]

    C. Luo, J. Zhan, Z. Jia, L. Wang, G. Lu, L. Zhang, C.-Z. Xu, and N. Sun. 2012. CloudRank-D: Benchmarking and Ranking Cloud Computing Systems for Data Processing Applications. Frontiers of Computer Science 6, 4 (2012), 347–362

    work page 2012

  19. [19]

    N. Z. Naqvi, T. Vansteenkiste-Muylle, and Y. Berbers. 2015. Benchmarking Leading-edge Mobile Devices for Data-intensive Distributed Mobile Cloud Ap- plications. In IEEE Symposium on Computers and Communication . 50–57

    work page 2015

  20. [20]

    Olguin, J

    M. Olguin, J. Wang, M. Satyanarayanan, and J. Gross. 2018. Scaling on the Edge – A Benchmarking Suite for Human-in-the-Loop Applications. In Proceedings of the IEEE/ACM Symposium on Edge Computing

    work page 2018

  21. [21]

    Redmon, S

    J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In IEEE Conference on Computer Vision and Pattern Recognition. 779–788

    work page 2016

  22. [22]

    YOLOv3: An Incremental Improvement

    J. Redmon and A. Farhadi. 2018. YOLOv3: An Incremental Improvement. CoRR abs/1804.02767 (2018). arXiv:1804.02767 http://arxiv.org/abs/1804.02767

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  23. [23]

    Satyanarayanan

    M. Satyanarayanan. 2017. The Emergence of Edge Computing. Computer 50, 1 (2017), 30–39

    work page 2017

  24. [24]

    Schuss, C

    M. Schuss, C. A. Boano, and K. Romer. 2018. Moving Beyond Competitions: Extending D-Cube to Seamlessly Benchmark Low-Power Wireless Systems. In IEEE Workshop on Benchmarking Cyber-Physical Networks and Systems . 30–35

    work page 2018

  25. [25]

    W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu. 2016. Edge Computing: Vision and Challenges. IEEE Internet of Things Journal 3, 5 (2016), 637–646

    work page 2016

  26. [26]

    Sonmez, A

    C. Sonmez, A. Ozgovde, and C. Ersoy. 2017. EdgeCloudSim: An Environment for Performance Evaluation of Edge Computing Systems. In International Conference on Fog and Mobile Edge Computing . 39–44

    work page 2017

  27. [27]

    Varghese, O

    B. Varghese, O. Akgun, I. Miguel, L. Thai, and A. Barker. 2019. Cloud Benchmark- ing for Maximising Performance of Scientific Applications. IEEE Transactions on Cloud Computing 7, 1 (2019), 170–182

    work page 2019

  28. [28]

    Varghese and R

    B. Varghese and R. Buyya. 2018. Next Generation Cloud Computing: New Trends and Research Directions. Future Generation Computer Systems 79, 3 (2018), 849– 861

    work page 2018

  29. [29]

    Varghese, L

    B. Varghese, L. T. Subba, L. Thai, and A. Barker. 2016. Container-Based Cloud Virtual Machine Benchmarking. In IEEE International Conference on Cloud Engi- neering. 192–201

    work page 2016

  30. [30]

    Varghese, L

    B. Varghese, L. T. Subba, L. Thai, and A. Barker. 2016. DocLite: A Docker- Based Lightweight Cloud Benchmarking Tool. In IEEE/ACM 16th International Symposium on Cluster, Cloud and Grid Computing . 213–222

    work page 2016

  31. [31]

    Varghese, N

    B. Varghese, N. Wang, S. Barbhuiya, P. Kilpatrick, and D. S. Nikolopoulos. 2016. Challenges and Opportunities in Edge Computing. In Proceedings of the IEEE International Conference on Smart Cloud . 20–26

    work page 2016

  32. [32]

    N. Wang, B. Varghese, M. Matthaiou, and D. S. Nikolopoulos. 2017. ENORM: A Framework for Edge Node Resource Management. IEEE Transactions on Services Computing (2017). https://doi.org/10.1109/TSC.2017.2753775

    work page doi:10.1109/tsc.2017.2753775 2017

  33. [33]

    Y. Wang, S. Liu, X. Wu, and W. Shi. 2018. CAVBench: A Benchmark Suite for Connected and Autonomous Vehicles. In Proceedings of the 3rd ACM/IEEE Symposium on Edge Computing . 12

    work page 2018