Using a small number of devices to experimentally estimate the packet delivery ratio on a lorawan network with a large number of end devices

Maxim S. Ilderyakov; Nikita V. Stepanov

REVIEW 4 major objections 3 minor 7 references

Using a small number of devices to experimentally estimate the packet delivery ratio on a lorawan network with a large number of end devices

T0 review · 4 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small LoRaWAN testbed can estimate the packet delivery ratio of a much larger network by scaling packet length and transmission rate so that the product of rate, device count, and packet duration stays constant.

desk verdict The paper's 4:1000 scaling claim is unsupported because the derivation is internally inconsistent and the only validation uses the authors' own model; the software utility and SF-mix observation don't rescue it. read the letter →

arxiv 1908.06434 v1 pith:QANGHP4Y submitted 2019-08-18 cs.NI

Maxim S. Ilderyakov , Nikita V. Stepanov This is my paper · ORCID

classification cs.NI

keywords LoRaWANpacketdeliveryratioLPWANnetworkscalingcollisionprobabilityspreadingfactorexperimentaltestbedsoftwarepackage

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper puts forward a way to measure the packet delivery ratio of a LoRaWAN network without deploying thousands of end devices: run a small experiment in which each device sends longer, more frequent messages. The argument is that the probability of a successful delivery depends on the product of packet duration, transmission rate, and device count, so keeping $\lambda N t$ fixed should reproduce the collision load of the large network. The authors state this as roughly four experimental messages standing in for a thousand real messages, and they build a software package to run such experiments and compute delivery statistics. They demonstrate the approach with 36 devices configured to simulate 8,835 real devices, using the testbed to assess how mixing spreading factors SF7 and SF8 changes delivery probability.

What carries the argument

The load-bearing object is the collision-load invariant $\lambda N t$: the product of per-device message rate $\lambda$, number of devices $N$, and packet duration $t$, which appears in the exponential formulas (1) and (2) for the probability of successful delivery. The paper uses this product to translate a small, dense experimental configuration into an equivalent large, sparse real configuration by setting $t_e N_e \lambda_e \approx t_r N_r \lambda_r$; that equivalence is what produces the 4:1000 message ratio and the 36-device stand-in for 8,835 real devices. The software complex is the supporting mechanism that turns this translation into a repeatable experiment, controlling device turn-on/off, polling the LoRaWAN server, and computing delivery statistics.

What would settle it

Measure the packet delivery ratio for two parameter sets with the same value of $\lambda N t$—for example, 36 devices sending long packets every 7 seconds versus 8,835 devices sending short packets every 600 seconds—in the same environment. If the two measured ratios differ by more than statistical noise, the product-invariance assumption on which the paper's scaling rests is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that the packet delivery ratio of a LoRaWAN network with thousands of end devices can be estimated from a small testbed, because what sets the probability of a lost packet is the total overlap load, measured by the product $\lambda N t$. Using the standard collision model, the probability that a message is not overlapped is $e^{-\lambda N t}$ (or $e^{-\lambda N \cdot 2t}$ when the whole message must be covered), so the system's behavior is governed by the product rather than by the individual factors. The paper therefore matches the experimental product $t_e N_e \lambda_e$ to the real product $t_r N_r \lambda_r$ and reports the resulting ratio as about four experimental messages for every thousand real messages. On this basis, 36 physical devices were configured to stand in for 8,835 real devices, with different mixtures of spreading factors SF7 and SF8 (LoRa's data-rate settings). The measured delivery ratios reproduce the qualitative conclusion of earlier analytic work that mixing the two spreading factors improves delivery probability, while showing that the optimal mixture shifts when realistic conditions replace idealized assumptions.

Load-bearing premise

The method rests on assuming that the packet delivery ratio depends only on the product $\lambda N t$, so a small group of devices sending long, frequent messages behaves exactly like a large group sending short, rare messages; this equivalence is asserted and never validated against a real large-scale deployment.

Editorial extensions

If this is right

LoRaWAN scalability experiments can be run with tens of physical devices instead of thousands, which cuts the cost and time of performance measurement.
The same scaling rule is claimed to apply to other LPWAN technologies, not only LoRaWAN, as the paper states explicitly.
Network operators can estimate delivery quality from the aggregate load $\lambda N t$ rather than from raw device counts alone.
Mixing spreading factors SF7 and SF8 is observed to improve delivery probability in a real testbed, though the optimal mixture differs from the idealized analytical prediction.
The software package automates the experiment workflow, so the delivery ratio estimates are reproducible from device tables and server logs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension would be to use the same invariance to compare retransmission policies or spreading-factor plans without fielding a full deployment, since only the aggregate collision load would need to be matched.
Because the exponential formula ignores LoRa's capture effect, where a stronger packet can survive an overlap, the 4:1000 ratio may need correction when devices transmit at different powers or distances; this could be tested by repeating the scaled experiment with capture-aware metrics.
The per-device timestamp data that the software collects could support measuring latency and per-spreading-factor delivery, not just the aggregate packet delivery ratio.
If the invariance fails when checked against a full-scale deployment, the failure would most likely show up as an independent dependence on packet length, device density, or duty-cycle occupancy, which would indicate which LoRaWAN-specific effect the scaling rule needs to incorporate.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Punchline: the central claim that four devices can emulate a thousand-node LoRaWAN network is not established. The scaling argument in Section II is internally inconsistent, and the experimental comparison targets the authors' own earlier model [7]. The software tool is useful and the SF7/SF8 mix observation is plausible, but the paper as a scientific report is not ready.

What is actually new: the idea to exploit the λNt invariant from ALOHA collision analysis is not new, but applying it to shrink a LoRaWAN testbed is a practical engineering step. The observation that the optimal SF7/SF8 mix differs from [7] is a concrete experimental result, and the software-hardware complex described in Sections III–VII is a real utility for running a multi-device LoRaWAN experiment and collecting PDR data. That contribution is modest but real.

Where it falls down: the derivation of the 4:1000 ratio never closes. Equations (1) and (2) disagree by a factor of 2, and neither accounts for LoRaWAN capture, duty-cycle limits, or SF orthogonality. The numerical bridge is also wrong as written: with N_p = 10000 and λ_p = 1/600, t_p N_p λ_p is about 0.7 if t_p is in seconds, but the paper writes 687 without a decimal point. Worse, the matching experimental expression "Te. Ne. λe." uses the transmission interval Te = 7 s, so the left side collapses to Ne = 41 and cannot equal a dimensionless collision load. The claimed ratio 4:1000 is asserted, not derived. Table I appears to swap the "real system" and "experiment" rows: the row labelled "SF7 in real system" contains 36, 31, 22, ... while the "SF7 in experiment" row contains 8835, 7608, ...; if taken literally, the experiment used thousands of devices, contradicting the premise. No raw data, repeated trials, or error bars are shown, and the validation uses the authors' own model [7] as ground truth, which is circular to the extent that the testbed was configured to match the same λNt product.

These are load-bearing flaws. The quantitative bridge from testbed to real network fails, and without it the PDR curves have no defined reference configuration. The SF-mix result might survive proper testing, but it is not rigorously established here.

The paper is for an engineer who wants a cheap LoRaWAN testbed and is willing to treat the scaling rule as a hypothesis rather than a proven method. The software description and the SF-mix footnote have some value, but the paper as a whole does not justify publication. I would desk reject it, with a suggestion to fix the derivation, release data and software, and validate against a full-scale simulation or real deployment.

Recommendation: not for peer review in current form.

Referee Report

4 major / 3 minor

Summary. The paper proposes a methodology to experimentally estimate the packet delivery ratio (PDR) of a large LoRaWAN network using only a small number of physical devices. The idea is to increase transmission rate and packet duration while keeping the product λNt (arrival rate, device count, packet duration) close to that of the large network, and the authors claim a ratio of 4 experimental devices to 1000 real devices. The paper describes a software-hardware complex that manages device power-on/off, queries the LoRaWAN server, and computes delivery statistics. An experiment with 36 devices is presented as simulating 8,835 real devices, and the measured PDR is compared with upper and lower bounds taken from a prior study [7]. The conclusion asserts that the program complex is effective because the experimental results match theoretical calculations.

Significance. If the scaling hypothesis were valid, the approach would be practically valuable: it would allow inexpensive testbed experiments to predict PDR for large LoRaWAN deployments, a question of direct importance for LPWAN scalability. The paper also describes a concrete software tool for running such experiments. However, the central methodological claim is not established. The derivation in Section II relies on an unvalidated assumption that PDR depends only on λNt, and the numerical example is internally inconsistent. No comparison against a real large-scale network or a validated simulator is provided, so the claimed equivalence lacks empirical support. The paper therefore does not currently meet the standard for a scientific contribution to networking measurement methodology.

major comments (4)

[Section II, Eqs. (1)-(2)] The central scaling claim is not derived. The paper states that from formulas (1) and (2) 'we can see that packet delivery ratio is affected by product e^{-λNt}', but Eq. (1) contains 2t and Eq. (2) contains t, with no explanation for the difference. Neither formula includes LoRaWAN-specific mechanisms such as capture effect, duty-cycle limitations, spreading-factor orthogonality, or server scheduling. Thus the invariant λNt at best controls offered load in a pure ALOHA model; it does not follow that the PDR of a real LoRaWAN network is preserved. The paper provides no validation against a full-scale deployment or a validated network simulator, so the load-bearing premise is an unverified assumption.
[Section II, numerical values] The numbers do not close as stated. With Np=10000 and λp=1/600, the equation tр·Nр·λр = 687 implies tр ≈ 41.2 seconds, which is not a realistic 'short packet' duration in LoRaWAN (typical airtimes are tens of milliseconds). On the experimental side, Te=7 s, Ne=41, and λe=1/7 give Te·Ne·λe = 41, which does not equal 687. The claimed approximate equality Te·Ne·λe ≈ tр·Nр·λр is therefore not satisfied by the paper's own parameters, and the ratio 'four messages in experimental system for thousand messages in real system' is asserted without a derivation. The scaling argument is numerically inconsistent and cannot serve as the basis for the experiment.
[Table I, Section X] Table I appears to have its rows interchanged. The row labelled 'Number of devices with SF7 in real system' contains the values 36, 31, 22, 14, 7, which must be the experimental device counts because only 36 devices were used in the experiment. Conversely, the row labelled 'Number of devices with SF7 in experiment' contains 8835, 7608, 5399, 3436, 1718, which are clearly the real-system counts taken from [7]. This swap makes the claimed correspondence between 36 experimental devices and 8835 real devices unverifiable, and it undermines the quantitative interpretation of Fig. 1.
[Section X, Fig. 1] The experimental validation is incomplete. The results are compared only with the theoretical bounds of [7], a paper co-authored by one of the present authors that itself relies on idealized assumptions. No measurement from a genuine large-scale LoRaWAN deployment is reported, so the claim that a small testbed reproduces the PDR of a large network is never tested against ground truth. Additionally, the experiment appears to be performed only once; no error bars, confidence intervals, or repeated trials are shown, so the agreement with the bounds in Fig. 1 cannot be assessed quantitatively.

minor comments (3)

[Abstract and Introduction] The abstract contains grammatical errors, e.g., 'In this paper, the proposed methodology for conducting experiments...' is an incomplete sentence, and 'benefits before other similarity technologies' should be rephrased. The paper would benefit from careful language editing.
[Section II] The phrase 'the melodic allowing using to replace 4 devices in experiment on 1000 devices in real system' should read 'the method allowing to replace 4 devices in experiment with 1000 devices in real system'. The notation for packet duration is inconsistently written as tр, te, and tmes; a single consistent symbol should be used.
[References] Reference [3] contains a stray 'Available at: https://www.sigfox.com/en' link that is unrelated to the LoRa airtime calculator, and the reference formatting is inconsistent (e.g., Cyrillic titles in [5] and [6]). The software package described in the paper is not made available, which limits reproducibility.

Circularity Check

2 steps flagged · score 6.0 of 10

Small-to-large equivalence is built from the same λNt product that equations (1)-(2) use, and validation relies on a same-author prior paper.

self definitional [Section II, 'Rationale for use approach to the organization of the experiment', Eqs. (1)-(2) and Te·Ne·λe ≈ tp·Np·λp statement]
"From formulas (1) and (2) we can see that packet delivery ratio is affects on production e-λNt . In experiment we increase lenth of message and reduce period. ... Te. Ne. λe. ≈ tр. Nр. λр., where te. – duration of long packet. So, we imitation a large number of messages overlaps in real system, then we have next ratio: four messages in experimental system for thousand messages in real system."

The claimed 4:1000 scaling is obtained by making the experimental product λe·Ne·te equal to the real-system product λp·Np·tp. But Eqs. (1)-(2) define the success probability as exponential in exactly that product, so equal products force equal theoretical PDR by construction. The later measurement of PDR on the 36-device testbed is therefore not an independent test of the scaling law; it is the same λNt value substituted into the same ALOHA formula. The equivalence 'small testbed = large network' is assumed in the parameter choice, not verified against any real large deployment.
self citation load bearing [Section X, 'Experiment's results', and Conclusion, comparison with reference [7]]
"Result of experiment shows that idea from work[7] can up the probability of success . The optimal combination of devices with SF7 and SF8 with fewer of them is changed from optimal value in work[7]. ... Effectively of this program follows from matches the program results with theoretic calculations."

Reference [7] is a prior paper coauthored by the present author N. Stepanov. The experiment is configured to emulate the 8835-device scenario described in [7], and the measured PDR is judged against the upper and lower bounds taken from [7]. The Conclusion then states 'Effectively of this program follows from matches the program results with theoretic calculations.' Thus the validation loop closes on a same-author simulation: [7] defines the target, the testbed is scaled to match the target by the assumed invariant, and agreement with [7] is offered as evidence of effectiveness. No independent large-scale LoRaWAN measurement is involved.

full rationale

The paper contains no machine-checked or otherwise independent validation of its core scaling hypothesis. Section II defines the experimental equivalence by equating the product that already appears in the ALOHA formulas, so the mapping from 36 devices to 8835 devices is an imposed identity rather than a derived or empirically tested relation. The comparison target is a same-author prior paper ([7]) rather than a real large network, making the agreement self-referential. Some independent content exists: the measured optimal SF7/SF8 mix differs from [7]'s prediction, and the measured PDR curve is real testbed data, so the paper is not wholly a restatement of its inputs. However, the central claim—that a small testbed can estimate PDR for a large LoRaWAN deployment—rests on the unvalidated λNt invariant and on self-citation, so the circularity score is substantial. Table I's swapped labels and the inconsistent numerical match (Te·Ne·λe vs. tp·Np·λp) are additional correctness defects but are not themselves circularity steps.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several assumed quantities: the experimental timing parameters, the real-system size, and the packet duration scaling factor. The collision formulas are invoked without proof, and the key scaling equivalence is assumed rather than demonstrated. No new physical entities are introduced.

free parameters (4)

T_e, experimental transmission interval = 7 s
Chosen from device firmware limitation [6] and used to set the experimental traffic intensity; the central equivalence depends on this value.
N_e, number of experimental devices = 41 (Section II), 36 (Section X)
Chosen so that the product T_e N_e λ_e approximates t_p N_p λ_p; the value changes between sections without explanation.
N_p, number of real devices = 10,000 (Section II), 8,835 (Section X)
Assumed real-system size; the two values are inconsistent within the paper and directly set the scaling ratio.
Packet duration scaling factor = not specified
The paper says message length is increased but never gives the actual airtimes for short and long packets, so the claimed product match cannot be checked.

assumptions (5)

standard math Formulas (1) and (2): successful delivery probability under Poisson message overlaps is e^{-λNt} or e^{-λN2t}.
Invoked in Section II as the basis for scaling the experiment; no derivation or citation is provided in the text.
domain assumption All devices use one spreading factor, and in the absence of overlaps messages are successfully delivered.
Stated at the start of Section II; this ignores capture effects, interference from other SFs, and losses unrelated to collision.
ad hoc to paper PDR depends only on the product λNt, so scaling N, λ, and t while preserving the product preserves PDR.
The entire method rests on this equivalence, which is asserted from formulas (1) and (2) and never validated against a full-scale deployment.
domain assumption Turning devices off during the experiment according to the priority algorithm does not bias the measured PDR.
Described in Section VII; no analysis is given for how device shutdown affects the remaining traffic or the final ratio.
domain assumption The real-system parameters N_p=10,000 and T_p=10 min are representative.
Taken from equipment references [5] and [6]; no empirical justification for these values as a realistic LoRaWAN deployment.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Using a small number of devices to experimentally estimate the packet delivery ratio on a lorawan network with a large number of end devices." pith.science (2026). https://pith.science/paper/QANGHP4Y

@misc{pith2026190806434,
  author       = {Pith},
  title        = {Pith review of: Using a small number of devices to experimentally estimate the packet delivery ratio on a lorawan network with a large number of end devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QANGHP4Y}},
  note         = {Machine review of arXiv:1908.06434}
}

read the original abstract

In this paper, the proposed methodology for conducting experiments on a small number of devices for evaluating systems with a large number of devices. A software package for assessing the performance of LPWAN (Long Range Wide Area Networks, LoRaWAN) networks is presented. An example of the operation of this software system as applied to the LoRaWAN network for estimating the probability of message delivery from a number of devices to a base station when implementing this technique is shown. This technique can be used to evaluate the performance of other LPWAN networks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 7 canonical work pages

[7]

Tiurlikova A., Stepanov N., Mikhaylov K. Method of Assigning Spreading Factor to Improve the Scalability of the LoRaWan Wide Area Network //2018 10th International Congress on Ultra Modern Telecommunications and Control Systems and Workshop s (ICUMT). – IEEE, 2018. – С. 1-4

work page 2018
[1]

LPWAN, LoRa, LoRaWAN and the Internet of Things

Ram, Prashant (2018 -08-07). "LPWAN, LoRa, LoRaWAN and the Internet of Things". Coinmonks. Retrieved 2019-02-09

work page 2018
[2]

What is LoRa and how does it work?

"What is LoRa and how does it work?". enterpriseiotinsights.com. Retrieved 2019-02-09

work page 2019
[3]

URL: https://www.loratools.nl/#/airtime Available at: https://www.sigfox.com/en (date of reference : 20.05.2018)

LoRa калькулятор. URL: https://www.loratools.nl/#/airtime Available at: https://www.sigfox.com/en (date of reference : 20.05.2018)

work page 2018
[4]

Limitations

"Limitations". The Things Network. 2019-02-18. Retrieved 2019-02-18

work page 2019
[5]

URL: http://www.auroramobile.ru/product_855.html (date of reference : 30.03.2019)

Оборудование LoRa. URL: http://www.auroramobile.ru/product_855.html (date of reference : 30.03.2019)

work page 2019
[6]

URL: http://iotvega.com/product/si11 (date of reference : 30.03.2019)

Оборудование LoRa. URL: http://iotvega.com/product/si11 (date of reference : 30.03.2019)

work page 2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [7]

Tiurlikova A., Stepanov N., Mikhaylov K. Method of Assigning Spreading Factor to Improve the Scalability of the LoRaWan Wide Area Network //2018 10th International Congress on Ultra Modern Telecommunications and Control Systems and Workshop s (ICUMT). – IEEE, 2018. – С. 1-4

work page 2018

[2] [1]

LPWAN, LoRa, LoRaWAN and the Internet of Things

Ram, Prashant (2018 -08-07). "LPWAN, LoRa, LoRaWAN and the Internet of Things". Coinmonks. Retrieved 2019-02-09

work page 2018

[3] [2]

What is LoRa and how does it work?

"What is LoRa and how does it work?". enterpriseiotinsights.com. Retrieved 2019-02-09

work page 2019

[4] [3]

URL: https://www.loratools.nl/#/airtime Available at: https://www.sigfox.com/en (date of reference : 20.05.2018)

LoRa калькулятор. URL: https://www.loratools.nl/#/airtime Available at: https://www.sigfox.com/en (date of reference : 20.05.2018)

work page 2018

[5] [4]

Limitations

"Limitations". The Things Network. 2019-02-18. Retrieved 2019-02-18

work page 2019

[6] [5]

URL: http://www.auroramobile.ru/product_855.html (date of reference : 30.03.2019)

Оборудование LoRa. URL: http://www.auroramobile.ru/product_855.html (date of reference : 30.03.2019)

work page 2019

[7] [6]

URL: http://iotvega.com/product/si11 (date of reference : 30.03.2019)

Оборудование LoRa. URL: http://iotvega.com/product/si11 (date of reference : 30.03.2019)

work page 2019