{"id":"70091679-cbaa-41bd-bf4a-3d7e3dc6fd91","arxiv_id":"1908.05085","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"On the public Antwerp LoRaWAN fingerprinting dataset, a tuned multilayer perceptron outperformed k-nearest neighbors and Extra Trees, reaching 358 m mean and 204 m median test error.","lead":"This paper compares three machine learning methods for locating low-power IoT devices from LoRaWAN signal strengths and finds that a neural network achieves the lowest error, with a mean of 358 meters and a median of 204 meters on a public city dataset. It also publishes the train/validation/test split and code so other groups can repeat the comparison.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MLP-vs-kNN comparison is confounded: Extra Trees and MLP receive the LoRa spreading factor as an extra input feature, while kNN does not, so the reported margin may reflect features rather than method.","rationale":"The reader correctly flags the single random split as a threat to the reliability of the ranking, but the more load-bearing problem is that the comparison is not method-versus-method. Section V explicitly gives Extra Trees and MLP an additional input feature, the LoRa spreading factor, that kNN does not receive. The paper's own justification, that trees and neural networks can naturally process non-commensurable features, does not withstand scrutiny: kNN can operate on normalized or one-hot encoded features and is routinely used with mixed data. Because the spreading factor is tied to LoRa modulation and range, it is a plausible source of localization information, so the reported test-set result of 358m for MLP versus 394m for kNN cannot be interpreted as evidence that the MLP is the better algorithm. This is a correctable flaw rather than a reason to discard the empirical work: the fixed split, released code, and honest reporting are real contributions. The verdict therefore remains CONDITIONAL, but the condition should explicitly include rerunning kNN with the same input features as the other two methods. If the rerun confirms the MLP advantage, then the single-split issue becomes the main remaining limitation; if it does not, the headline claim must be revised.","tokens_in":8205,"tokens_out":4917,"duration_ms":50759,"concrete_test":"Rerun the kNN method on the same train/validation/test split with the LoRa spreading factor added as an input feature, either as a normalized continuous variable or one-hot encoded, retuning k and the distance metric on the validation set. If kNN+SF's test mean error closes the 36-meter gap to MLP or falls within split-to-split variability, then the reported MLP advantage is attributable to the extra feature rather than to the method itself. Also report Extra Trees without the spreading factor as a control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the MLP is the most accurate method is undermined by an input-feature asymmetry in Section V. The paper states that for kNN only the RSSI values from the 68 basestations are used as input features, while \"the two other methods (Extra Trees and MLP), apart from the RSSI values, also utilize as an additional feature the reported LoRa spreading factor.\" The LoRa spreading factor is not a constant; it varies per message and directly affects signal range, so it is potentially informative about location. Giving this feature only to Extra Trees and MLP means the comparison bundles algorithm choice with feature availability. The 36-meter mean-error gap between MLP and kNN on the test set could therefore be caused by the extra feature rather than by the neural network. The stated rationale that kNN cannot use non-commensurable features is not persuasive: kNN can be applied to normalized or one-hot encoded features, or with Gower-type distances. Consequently, the paper's headline comparison does not isolate the method. The reader's single-split concern is valid but secondary: repeating splits would not resolve this confound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three fingerprinting localization methods (k-nearest neighbours, Extra Trees, and a multilayer perceptron) on a public LoRaWAN RSSI dataset collected in Antwerp. The authors apply four RSSI preprocessing schemes, tune hyperparameters on a validation split, and report localization errors on a held-out test split. They find that the MLP achieves the lowest mean error (358 m) and median error (204 m) on the test set, and they release the code, data splits, and preprocessing details to support reproducibility.","tokens_in":8460,"tokens_out":4990,"duration_ms":44332,"significance":"Given the scarcity of public LPWAN fingerprinting benchmarks, a careful, reproducible comparison is potentially valuable. The authors correctly use a held-out test set and perform hyperparameter selection on validation, and the decision to publish the exact split is a concrete step toward comparability. The paper also demonstrates that on this particular sparse dataset, preprocessing choices have little effect, which is an interesting empirical observation. However, the headline claim that MLP is the most accurate method is currently supported only by point estimates from a single split and rests on a comparison in which kNN receives a strictly smaller feature set, so the significance is conditional on fixing these issues.","major_comments":[{"comment":"The comparison is confounded by an input-feature asymmetry. Section V states that kNN uses only the RSSI values from the 68 basestations as features, while Extra Trees and MLP also receive the LoRa spreading factor as an additional feature. The spreading factor is not a constant; it varies per message and is known to affect signal propagation, so it may contain location-relevant information. As a result, the reported test-set gap between MLP (358 m) and kNN (394 m) cannot be attributed solely to the choice of method. The claim that kNN 'cannot naturally process' a non-commensurable feature is not convincing, since kNN can be applied with normalized, standardized, or one-hot encoded features, or with a Gower-type distance. I recommend either giving the same features to all three methods or running an ablation that removes the spreading factor from Extra Trees and MLP; the current Table III does not isolate the algorithmic contribution.","section":"Section V (Experimental Setup) and Table III"},{"comment":"All conclusions are based on a single random 70/15/15 split of the dataset into training, validation, and test (Section III). Table III reports point estimates without confidence intervals, bootstrap results, or repeated runs. The test-set gap between MLP and Extra Trees is only 22 m in mean (358 vs. 380) and 57 m in median (204 vs. 261), which could easily be within sampling noise for a dataset with 18530 test points and inherent variability. The authors should repeat the split multiple times (or use bootstrap sampling) and report the distribution of test errors, or at least a paired or stratified test, before claiming that MLP is the most accurate method. In addition, the '10% improvement' over Aernouts et al. [2] is computed across different test splits and should be clearly flagged as an informal comparison rather than a head-to-head result.","section":"Section III, Section VI, Table III"}],"minor_comments":[{"comment":"Equation (2) appears to be missing a division operator between the numerator and denominator; please check the typesetting of the powed representation.","section":"Section IV, Equation (2)"},{"comment":"The text reporting the α tuning gives a kNN test mean of 398 m and median of 277 m for α=60, and for β=1.1 a test mean of 394 m and median of 272 m; Table III reports kNN test mean 394 m and median 273 m. These numbers are not fully consistent and should be reconciled.","section":"Section VI-A and Table III"},{"comment":"For a reproducibility-focused paper, the code and data should be available through a persistent archive (e.g., Zenodo) at the time of publication; the current text promises a camera-ready upload and gives a Dropbox link, which is not stable enough for the stated reproducibility goal.","section":"Sections V and VII"},{"comment":"The manuscript contains several typographical issues, including 'Extra Tress' in the Introduction, 'prepossessing' for 'preprocessing', 'comonly' for 'commonly', and 'coresponds' for 'corresponds'; a careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable empirical contribution to a reproducibility-oriented venue, but the main result's validity depends on resolving the feature-asymmetry issue; the problem is fixable within the manuscript's scope. I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a clean, reproducible benchmark on the public Antwerp LoRaWAN dataset, and the authors deserve credit for releasing their split and being transparent about hyperparameters. But the main result is not method-vs-method. In Section V they state that kNN uses only the 68 RSSI features while Extra Trees and MLP also get the LoRa spreading factor. That is an input-feature asymmetry, and the spreading factor is not constant; it affects signal range and is plausibly correlated with location. So the 36-meter mean gap between MLP and kNN could come from the extra feature, not from the neural network. The MLP-vs-ExtraTrees gap (22 meters) avoids that confound because both get the feature, but with a single random split and no significance testing that margin is well within sampling noise. The reader's report already flagged the single-split issue; the stress-test note correctly adds that repeating splits would not fix the feature confound.\n\nWhat is genuinely new: first follow-up evaluation of the Aernouts dataset with methods beyond kNN, plus a published 70/15/15 split. The tuning is sensible and validation-based; Table III directly backs the reported numbers. The observation that preprocessing schemes matter little on this sparse dataset is a useful, data-driven point.\n\nSoft spots, in order: (1) the feature asymmetry described above; (2) single split, no confidence intervals or repeated trials, so rankings are fragile; (3) code/data only promised to be uploaded to Zenodo, so 'reproducible' is not yet fully true; (4) the 10% improvement over the original kNN benchmark compares against a different test split, which the paper honestly acknowledges.\n\nCitation pattern is fine and relevant: Sigfox precedents [6,7], Torres-Sospedra preprocessing [8], Geurts for Extra Trees. No invented entities.\n\nWho this is for: anyone building on the Antwerp LoRaWAN fingerprinting dataset, or working on outdoor IoT localization. It is a useful baseline paper, not a breakthrough. It deserves a serious referee, but the referee should request equal features across methods, repeated splits, and an archived code/data deposit before acceptance.","headline":"Honest, reproducible benchmark on the Antwerp LoRaWAN dataset, but the headline MLP-vs-kNN win is confounded by giving only the MLP and Extra Trees the LoRa spreading factor as an extra input feature.","tokens_in":9002,"tokens_out":2291,"would_cite":false,"duration_ms":22469,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multilayer perceptron is the most accurate of three tested fingerprinting methods on a public LoRaWAN dataset, achieving a 358-meter mean localization error.","keywords":["LoRaWAN","RSSI fingerprinting","outdoor localization","multilayer perceptron","k-nearest neighbours","Extra Trees","reproducibility","LPWAN"],"falsifier":"Re-run the same three methods with their reported hyperparameters on 10 different random 70/15/15 splits of the Antwerp dataset and compute the distribution of test-set mean errors; if the MLP's mean error does not stay clearly below Extra Trees and kNN across most splits, the claimed ordering is not established.","tokens_in":7993,"feed_emoji":"📡","tokens_out":2606,"duration_ms":27531,"temperature":0.7,"pith_summary":"This paper asks which machine-learning method best estimates the location of low-power LoRaWAN devices from RSSI fingerprints, using a publicly available urban dataset. It compares k-nearest neighbours, Extra Trees, and a multilayer perceptron under a shared train/validation/test split. The paper's central claim is that the MLP is the most accurate, with a mean test error of 358 meters and a median of 204 meters, roughly 36 meters better than the kNN baseline. It also argues that the choice of RSSI preprocessing makes little difference for this sparse dataset, and it publishes code and the exact split so other researchers can reproduce and extend the comparison.","feed_headline":"Neural net wins LoRaWAN location test: 358 m mean error","feed_subtitle":"Reproducible benchmark on the public Antwerp LoRaWAN dataset beats kNN and Extra Trees; code and data split released.","key_machinery":"The central object is the public LoRaWAN RSSI fingerprint dataset collected in urban Antwerp, together with a fixed 70/15/15 random split into training, validation, and test sets. The argument is carried by the comparison protocol: each method gets its hyperparameters tuned on the validation set, and the test set provides a shared, reproducible accuracy estimate. For the MLP, the decisive components are a 7-layer architecture with 1024-1024-1024-256-128-128-2 neurons, dropout at rate 0.15, batch normalization, ReLU activations, the Adam optimizer, and early stopping, all chosen after first verifying that the architecture can overfit the training set.","core_discovery":"On the Antwerp LoRaWAN fingerprinting dataset, a 7-layer feed-forward neural network with dropout, batch normalization, early stopping, and the spreading factor as an extra input feature yields the lowest localization error among the three methods tested. The paper reports a mean error of 358 meters and a median error of 204 meters on the test set, compared with 380 and 261 meters for Extra Trees and 394 and 273 meters for kNN. The authors also observe that the four RSSI preprocessing schemes they evaluate (positive, normalized, exponential, and powed) produce nearly identical accuracy, which they attribute to the dataset's sparsity: most fingerprints contain RSSI values from only three gateways out of 68.","pith_inferences":["Because most fingerprints contain only three gateway readings, treating the set of receiving gateways as categorical features, rather than just RSSI magnitudes, could yield further gains; the paper's observation about preprocessing insensitivity hints at this but does not test it.","The reported superiority of the MLP rests on a single split; a natural extension is to repeat the comparison across multiple random splits or geographic subregions to see whether the ranking is stable.","The method's performance on the rural LoRaWAN dataset released by the same data authors remains untested, and rural sparsity patterns may change which method wins.","The near-equivalence of preprocessing schemes on this dataset suggests that benchmark comparisons on LPWAN data should report gateway-coverage statistics, since those may matter more than RSSI scaling."],"forward_implications":["LoRaWAN RSSI fingerprinting can reach mean localization errors around 350 meters in a dense urban setting, which is useful for applications where GPS is too power-hungry.","Neural networks appear to extract more location information from sparse RSSI fingerprints than kNN or tree ensembles, especially when non-RSSI features like the spreading factor are included.","Preprocessing transformations of RSSI values matter less when most fingerprints are extremely sparse, because which gateways received a signal is the dominant signal.","The released train/validation/test split and code allow future methods to be compared directly against the numbers reported here without re-implementing the pipeline.","The gap between mean and median error indicates that outliers still dominate the mean, so localization accuracy is best described by the full error distribution rather than a single number."],"supporting_citations":[{"why":"Supplies the public LoRaWAN fingerprint dataset used for all experiments and the prior kNN baseline (398 m mean error) that the MLP result is compared against.","marker":"[2]"},{"why":"Proposes the exponential and powed RSSI representations and the default parameter values that this study tunes and evaluates.","marker":"[8]"},{"why":"Defines the Extra Trees ensemble method that is one of the three compared approaches.","marker":"[9]"},{"why":"Provides the dropout regularization technique that the paper finds more effective than L2 regularization for the MLP.","marker":"[10]"},{"why":"Provides batch normalization, used at every layer of the MLP to speed and stabilize training.","marker":"[11]"},{"why":"Provides the Adam optimization algorithm used to train the MLP.","marker":"[13]"}],"fun_headline_variants":["Neural net tops LoRaWAN fingerprinting: 358 m mean error","LoRaWAN localization: neural net beats kNN, Extra Trees","Best LoRaWAN fingerprinting: NN, 358 m mean error","Reproducible LoRaWAN test: neural net wins with 358 m","NN outdoes kNN and Extra Trees in LoRaWAN RSSI test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of methods relies on a single random 70/15/15 split of the dataset, with no repeated splits, confidence intervals, or significance tests, so the observed differences in mean error could be sampling noise.","fun_headline_variants_meta":{"raw":{"variants":["Neural net tops LoRaWAN fingerprinting: 358 m mean error","LoRaWAN localization: neural net beats kNN, Extra Trees","Best LoRaWAN fingerprinting: NN, 358 m mean error","Reproducible LoRaWAN test: neural net wins with 358 m","NN outdoes kNN and Extra Trees in LoRaWAN RSSI test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3500,"prompt_tokens":854,"completion_tokens":2646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":2542}},"tokens_in":470,"tokens_out":2646,"duration_ms":16403,"temperature":1.0,"reasoning_tokens":2542,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:24:04.883099+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same three methods with their reported hyperparameters on 10 different random 70/15/15 splits of the Antwerp dataset and compute the distribution of test-set mean errors; if the MLP's mean error does not stay clearly below Extra Trees and kNN across most splits, the claimed ordering is not established.","supporting_citations":[{"cited_title":"Sigfox and lorawan datasets for fingerprint localization in large urban and rural areas,","cited_arxiv_id":null,"evidence_quote":"Supplies the public LoRaWAN fingerprint dataset used for all experiments and the prior kNN baseline (398 m mean error) that the MLP result is compared against."},{"cited_title":"Comprehensive analysis of distance and similarity measures for wi-fi fingerprinting indoor positioning systems,","cited_arxiv_id":null,"evidence_quote":"Proposes the exponential and powed RSSI representations and the default parameter values that this study tunes and evaluates."},{"cited_title":"Dropout: A simple way to prevent neural networks from overfitting,","cited_arxiv_id":null,"evidence_quote":"Provides the dropout regularization technique that the paper finds more effective than L2 regularization for the MLP."}],"review_version":1}