{"id":"a51ef9a9-fc48-46d0-baba-64a09e98e62a","arxiv_id":"1908.06434","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A small-device experimental setup is claimed to emulate a much larger LoRaWAN network by matching the product of device count, transmission rate, and packet duration.","lead":"This paper describes an experimental method for estimating LoRaWAN packet delivery ratios using only a few dozen devices instead of thousands, by shortening the time between transmissions and lengthening packets. It also reports a software tool that runs the experiment and processes server logs, which makes large-scale network testing cheaper if the method holds.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proposed scaling law is not actually established: Eqs. (1)-(2) are dimensionally inconsistent, the 4:1000 ratio is never derived, and Table I swaps experiment/real device counts, so the 36-device-to-8,835-device correspondence is unverifiable.","rationale":"The reader's weakest_assumption is the unvalidated scaling hypothesis. I agree that the hypothesis is unvalidated, but I would sharpen the critique: the paper does not even present a self-consistent statement of the scaling law to validate. The dimensional mismatch in tр Nр λр = 687, the conflation of Te with te, the unexplained 4:1000 ratio, and the swapped rows of Table I mean that a good-faith reader cannot tell what experiment was configured to emulate what real deployment. This is not a rejection based on disagreement with an established model; capture effects and LoRaWAN-specific constraints are real, but secondary to the fact that the paper's own arithmetic is not reproducible. The strongest claim identified in the reader's verdict — that 36 devices emulate 8835 devices — is directly tied to Table I, and that table is internally inconsistent. A corrected derivation might still fail because LoRaWAN PDR depends on capture, duty-cycle limits, and path loss, not only on λNt; but the immediate, decisive problem is that the claimed mapping is not established even at the level of algebra. Consequently, the experiment's output cannot be interpreted as an estimate of any specified real deployment, and the rejection stands.","tokens_in":3897,"tokens_out":5221,"duration_ms":53197,"concrete_test":"Reproduce the scaling chain from Section II with explicit units and packet airtimes: compute t_p and t_e for a specific LoRaWAN payload and spreading factor, verify whether t_p N_p λ_p = 687 (or identify the missing factor), then solve N_e λ_e t_e = N_p λ_p t_p for N_e with λ_e=1/7, N_p=10000, λ_p=1/600. If the solution is not 41 and does not map 36 devices to 8835 under the same relation, the central correspondence is arithmetically false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the testbed to reproduce the collision statistics of a much larger LoRaWAN deployment via the invariant λNt. Section II is the only derivation, and it does not support the claim. Eq. (1) gives Pr{T=2t}=e^{-λN2t}, which is the pure-ALOHA success probability under full overlap; Eq. (2) drops the factor 2 without explanation. Neither equation incorporates LoRaWAN capture, duty-cycle limits, or spreading-factor orthogonality, so at best the invariant controls offered load in an ALOHA model, not the packet delivery ratio of an actual LoRaWAN network. More importantly, the numbers do not close: the paper states tр Nр λр = 687 with Nр=10000 and λр=1/600, but tр is never given; for a typical SF7 packet of a few tens of milliseconds, Nрλр tр is about 0.7, so a factor of 1000 is missing or the units are inconsistent. The matching experimental statement 'Te. Ne. λe. ≈ tр. Nр. λр.' compares the device interval Te=7 s on the left with packet duration tр on the right, mixing two different quantities. The substitution ratio 'four messages in experimental system for thousand messages in real system' is asserted without derivation. Finally, Table I is internally swapped: the row labelled 'Number of devices with SF7 in real system' contains the values 36, 31, 22, 14, 7, while the row labelled 'Number of devices with SF7 in experiment' contains 8835, 7608, ...; if the labels are corrected, the stated 36-device/8835-device scaling is not obtained from the 41-device/10000-device relation given in Section II. Since this mapping is the sole quantitative bridge from testbed to real system, the experimental PDR curves have no well-defined reference configuration.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a methodology to experimentally estimate the packet delivery ratio (PDR) of a large LoRaWAN network using only a small number of physical devices. The idea is to increase transmission rate and packet duration while keeping the product λNt (arrival rate, device count, packet duration) close to that of the large network, and the authors claim a ratio of 4 experimental devices to 1000 real devices. The paper describes a software-hardware complex that manages device power-on/off, queries the LoRaWAN server, and computes delivery statistics. An experiment with 36 devices is presented as simulating 8,835 real devices, and the measured PDR is compared with upper and lower bounds taken from a prior study [7]. The conclusion asserts that the program complex is effective because the experimental results match theoretical calculations.","tokens_in":4311,"tokens_out":3552,"duration_ms":35560,"significance":"If the scaling hypothesis were valid, the approach would be practically valuable: it would allow inexpensive testbed experiments to predict PDR for large LoRaWAN deployments, a question of direct importance for LPWAN scalability. The paper also describes a concrete software tool for running such experiments. However, the central methodological claim is not established. The derivation in Section II relies on an unvalidated assumption that PDR depends only on λNt, and the numerical example is internally inconsistent. No comparison against a real large-scale network or a validated simulator is provided, so the claimed equivalence lacks empirical support. The paper therefore does not currently meet the standard for a scientific contribution to networking measurement methodology.","major_comments":[{"comment":"The central scaling claim is not derived. The paper states that from formulas (1) and (2) 'we can see that packet delivery ratio is affected by product e^{-λNt}', but Eq. (1) contains 2t and Eq. (2) contains t, with no explanation for the difference. Neither formula includes LoRaWAN-specific mechanisms such as capture effect, duty-cycle limitations, spreading-factor orthogonality, or server scheduling. Thus the invariant λNt at best controls offered load in a pure ALOHA model; it does not follow that the PDR of a real LoRaWAN network is preserved. The paper provides no validation against a full-scale deployment or a validated network simulator, so the load-bearing premise is an unverified assumption.","section":"Section II, Eqs. (1)-(2)"},{"comment":"The numbers do not close as stated. With Np=10000 and λp=1/600, the equation tр·Nр·λр = 687 implies tр ≈ 41.2 seconds, which is not a realistic 'short packet' duration in LoRaWAN (typical airtimes are tens of milliseconds). On the experimental side, Te=7 s, Ne=41, and λe=1/7 give Te·Ne·λe = 41, which does not equal 687. The claimed approximate equality Te·Ne·λe ≈ tр·Nр·λр is therefore not satisfied by the paper's own parameters, and the ratio 'four messages in experimental system for thousand messages in real system' is asserted without a derivation. The scaling argument is numerically inconsistent and cannot serve as the basis for the experiment.","section":"Section II, numerical values"},{"comment":"Table I appears to have its rows interchanged. The row labelled 'Number of devices with SF7 in real system' contains the values 36, 31, 22, 14, 7, which must be the experimental device counts because only 36 devices were used in the experiment. Conversely, the row labelled 'Number of devices with SF7 in experiment' contains 8835, 7608, 5399, 3436, 1718, which are clearly the real-system counts taken from [7]. This swap makes the claimed correspondence between 36 experimental devices and 8835 real devices unverifiable, and it undermines the quantitative interpretation of Fig. 1.","section":"Table I, Section X"},{"comment":"The experimental validation is incomplete. The results are compared only with the theoretical bounds of [7], a paper co-authored by one of the present authors that itself relies on idealized assumptions. No measurement from a genuine large-scale LoRaWAN deployment is reported, so the claim that a small testbed reproduces the PDR of a large network is never tested against ground truth. Additionally, the experiment appears to be performed only once; no error bars, confidence intervals, or repeated trials are shown, so the agreement with the bounds in Fig. 1 cannot be assessed quantitatively.","section":"Section X, Fig. 1"}],"minor_comments":[{"comment":"The abstract contains grammatical errors, e.g., 'In this paper, the proposed methodology for conducting experiments...' is an incomplete sentence, and 'benefits before other similarity technologies' should be rephrased. The paper would benefit from careful language editing.","section":"Abstract and Introduction"},{"comment":"The phrase 'the melodic allowing using to replace 4 devices in experiment on 1000 devices in real system' should read 'the method allowing to replace 4 devices in experiment with 1000 devices in real system'. The notation for packet duration is inconsistently written as tр, te, and tmes; a single consistent symbol should be used.","section":"Section II"},{"comment":"Reference [3] contains a stray 'Available at: https://www.sigfox.com/en' link that is unrelated to the LoRa airtime calculator, and the reference formatting is inconsistent (e.g., Cyrillic titles in [5] and [6]). The software package described in the paper is not made available, which limits reproducibility.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript is not suitable for publication in its current form. The central scaling claim is neither derived correctly nor validated empirically, and the numerical inconsistencies suggest the relation between the experimental and real-system parameters was not carefully checked. A revision would require a fundamentally different validation strategy, such as comparing the small-testbed results with a real large-scale deployment or a well-established simulator, and would need to correct the internal inconsistencies in Section II and Table I. These are not local issues but concern the core of the claimed contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the central claim that four devices can emulate a thousand-node LoRaWAN network is not established. The scaling argument in Section II is internally inconsistent, and the experimental comparison targets the authors' own earlier model [7]. The software tool is useful and the SF7/SF8 mix observation is plausible, but the paper as a scientific report is not ready.\n\nWhat is actually new: the idea to exploit the λNt invariant from ALOHA collision analysis is not new, but applying it to shrink a LoRaWAN testbed is a practical engineering step. The observation that the optimal SF7/SF8 mix differs from [7] is a concrete experimental result, and the software-hardware complex described in Sections III–VII is a real utility for running a multi-device LoRaWAN experiment and collecting PDR data. That contribution is modest but real.\n\nWhere it falls down: the derivation of the 4:1000 ratio never closes. Equations (1) and (2) disagree by a factor of 2, and neither accounts for LoRaWAN capture, duty-cycle limits, or SF orthogonality. The numerical bridge is also wrong as written: with N_p = 10000 and λ_p = 1/600, t_p N_p λ_p is about 0.7 if t_p is in seconds, but the paper writes 687 without a decimal point. Worse, the matching experimental expression \"Te. Ne. λe.\" uses the transmission interval Te = 7 s, so the left side collapses to Ne = 41 and cannot equal a dimensionless collision load. The claimed ratio 4:1000 is asserted, not derived. Table I appears to swap the \"real system\" and \"experiment\" rows: the row labelled \"SF7 in real system\" contains 36, 31, 22, ... while the \"SF7 in experiment\" row contains 8835, 7608, ...; if taken literally, the experiment used thousands of devices, contradicting the premise. No raw data, repeated trials, or error bars are shown, and the validation uses the authors' own model [7] as ground truth, which is circular to the extent that the testbed was configured to match the same λNt product.\n\nThese are load-bearing flaws. The quantitative bridge from testbed to real network fails, and without it the PDR curves have no defined reference configuration. The SF-mix result might survive proper testing, but it is not rigorously established here.\n\nThe paper is for an engineer who wants a cheap LoRaWAN testbed and is willing to treat the scaling rule as a hypothesis rather than a proven method. The software description and the SF-mix footnote have some value, but the paper as a whole does not justify publication. I would desk reject it, with a suggestion to fix the derivation, release data and software, and validate against a full-scale simulation or real deployment.\n\nRecommendation: not for peer review in current form.","headline":"The paper's 4:1000 scaling claim is unsupported because the derivation is internally inconsistent and the only validation uses the authors' own model; the software utility and SF-mix observation don't rescue it.","tokens_in":4862,"tokens_out":4000,"would_cite":false,"duration_ms":39512,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small LoRaWAN testbed can estimate the packet delivery ratio of a much larger network by scaling packet length and transmission rate so that the product of rate, device count, and packet duration stays constant.","keywords":["LoRaWAN","packet delivery ratio","LPWAN","network scaling","collision probability","spreading factor","experimental testbed","software package"],"falsifier":"Measure the packet delivery ratio for two parameter sets with the same value of $\\lambda N t$—for example, 36 devices sending long packets every 7 seconds versus 8,835 devices sending short packets every 600 seconds—in the same environment. If the two measured ratios differ by more than statistical noise, the product-invariance assumption on which the paper's scaling rests is false.","tokens_in":3678,"feed_emoji":"📡","tokens_out":14146,"duration_ms":124322,"temperature":0.7,"pith_summary":"The paper puts forward a way to measure the packet delivery ratio of a LoRaWAN network without deploying thousands of end devices: run a small experiment in which each device sends longer, more frequent messages. The argument is that the probability of a successful delivery depends on the product of packet duration, transmission rate, and device count, so keeping $\\lambda N t$ fixed should reproduce the collision load of the large network. The authors state this as roughly four experimental messages standing in for a thousand real messages, and they build a software package to run such experiments and compute delivery statistics. They demonstrate the approach with 36 devices configured to simulate 8,835 real devices, using the testbed to assess how mixing spreading factors SF7 and SF8 changes delivery probability.","feed_headline":"Four test messages can stand in for a thousand in LoRaWAN","feed_subtitle":"A 36-device experiment stands in for 8,835 nodes by matching collision load.","key_machinery":"The load-bearing object is the collision-load invariant $\\lambda N t$: the product of per-device message rate $\\lambda$, number of devices $N$, and packet duration $t$, which appears in the exponential formulas (1) and (2) for the probability of successful delivery. The paper uses this product to translate a small, dense experimental configuration into an equivalent large, sparse real configuration by setting $t_e N_e \\lambda_e \\approx t_r N_r \\lambda_r$; that equivalence is what produces the 4:1000 message ratio and the 36-device stand-in for 8,835 real devices. The software complex is the supporting mechanism that turns this translation into a repeatable experiment, controlling device turn-on/off, polling the LoRaWAN server, and computing delivery statistics.","core_discovery":"The paper's central claim is that the packet delivery ratio of a LoRaWAN network with thousands of end devices can be estimated from a small testbed, because what sets the probability of a lost packet is the total overlap load, measured by the product $\\lambda N t$. Using the standard collision model, the probability that a message is not overlapped is $e^{-\\lambda N t}$ (or $e^{-\\lambda N \\cdot 2t}$ when the whole message must be covered), so the system's behavior is governed by the product rather than by the individual factors. The paper therefore matches the experimental product $t_e N_e \\lambda_e$ to the real product $t_r N_r \\lambda_r$ and reports the resulting ratio as about four experimental messages for every thousand real messages. On this basis, 36 physical devices were configured to stand in for 8,835 real devices, with different mixtures of spreading factors SF7 and SF8 (LoRa's data-rate settings). The measured delivery ratios reproduce the qualitative conclusion of earlier analytic work that mixing the two spreading factors improves delivery probability, while showing that the optimal mixture shifts when realistic conditions replace idealized assumptions.","pith_inferences":["A natural extension would be to use the same invariance to compare retransmission policies or spreading-factor plans without fielding a full deployment, since only the aggregate collision load would need to be matched.","Because the exponential formula ignores LoRa's capture effect, where a stronger packet can survive an overlap, the 4:1000 ratio may need correction when devices transmit at different powers or distances; this could be tested by repeating the scaled experiment with capture-aware metrics.","The per-device timestamp data that the software collects could support measuring latency and per-spreading-factor delivery, not just the aggregate packet delivery ratio.","If the invariance fails when checked against a full-scale deployment, the failure would most likely show up as an independent dependence on packet length, device density, or duty-cycle occupancy, which would indicate which LoRaWAN-specific effect the scaling rule needs to incorporate."],"forward_implications":["LoRaWAN scalability experiments can be run with tens of physical devices instead of thousands, which cuts the cost and time of performance measurement.","The same scaling rule is claimed to apply to other LPWAN technologies, not only LoRaWAN, as the paper states explicitly.","Network operators can estimate delivery quality from the aggregate load $\\lambda N t$ rather than from raw device counts alone.","Mixing spreading factors SF7 and SF8 is observed to improve delivery probability in a real testbed, though the optimal mixture differs from the idealized analytical prediction.","The software package automates the experiment workflow, so the delivery ratio estimates are reproducible from device tables and server logs."],"supporting_citations":[{"why":"It supplies the LoRa airtime calculation used to set packet durations for real and experimental configurations.","marker":"[3]"},{"why":"It documents the real-system base station hardware and parameters from which $t_r$, $N_r$, and $\\lambda_r$ are taken.","marker":"[5]"},{"why":"It sets the experimental device's fixed transmission interval of 7 seconds, which fixes $\\lambda_e$.","marker":"[6]"},{"why":"It supplies the SF7/SF8 assignment scenario and the analytic bounds that the experiment reproduces and compares against.","marker":"[7]"}],"fun_headline_variants":["Small testbed, big network: How 36 LoRaWAN nodes predict 8,835","Matching collision load: 36 LoRaWAN devices mimic 8,835","Four test messages stand in for a thousand in LoRaWAN","Tiny LoRaWAN testbed estimates big network delivery","36 LoRaWAN nodes model 8,835 by matching load"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on assuming that the packet delivery ratio depends only on the product $\\lambda N t$, so a small group of devices sending long, frequent messages behaves exactly like a large group sending short, rare messages; this equivalence is asserted and never validated against a real large-scale deployment.","fun_headline_variants_meta":{"raw":{"variants":["Small testbed, big network: How 36 LoRaWAN nodes predict 8,835","Matching collision load: 36 LoRaWAN devices mimic 8,835","Four test messages stand in for a thousand in LoRaWAN","Tiny LoRaWAN testbed estimates big network delivery","36 LoRaWAN nodes model 8,835 by matching load"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000422,"raw_usage":{"total_tokens":2129,"prompt_tokens":870,"completion_tokens":1259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":1157}},"tokens_in":486,"tokens_out":1259,"duration_ms":9715,"temperature":1.0,"reasoning_tokens":1157,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:45:14.634731+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the packet delivery ratio for two parameter sets with the same value of $\\lambda N t$—for example, 36 devices sending long packets every 7 seconds versus 8,835 devices sending short packets every 600 seconds—in the same environment. If the two measured ratios differ by more than statistical noise, the product-invariance assumption on which the paper's scaling rests is false.","supporting_citations":[{"cited_title":"URL: https://www.loratools.nl/#/airtime Available at: https://www.sigfox.com/en (date of reference : 20.05.2018)","cited_arxiv_id":null,"evidence_quote":"It supplies the LoRa airtime calculation used to set packet durations for real and experimental configurations."},{"cited_title":"URL: http://www.auroramobile.ru/product_855.html (date of reference : 30.03.2019)","cited_arxiv_id":null,"evidence_quote":"It documents the real-system base station hardware and parameters from which $t_r$, $N_r$, and $\\lambda_r$ are taken."},{"cited_title":"URL: http://iotvega.com/product/si11 (date of reference : 30.03.2019)","cited_arxiv_id":null,"evidence_quote":"It sets the experimental device's fixed transmission interval of 7 seconds, which fixes $\\lambda_e$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the SF7/SF8 assignment scenario and the analytic bounds that the experiment reproduces and compares against."}],"review_version":1}