{"id":"a58edeb7-591e-4a78-9bca-983239851ad3","arxiv_id":"1908.06851","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Tuning preprocessing parameters and kNN hyperparameters on a public Sigfox dataset reduces mean localization error from 344 to 298 meters, with median error 109 meters.","lead":"This paper tunes data preprocessing and kNN settings to improve outdoor Sigfox localization on a public Antwerp dataset, reporting a mean error of 298 meters. It is a reproducibility-focused study for the LPWAN fingerprinting community.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random split likely leaks temporally and spatially correlated messages from the same postal vehicles into train and test, making the reported 298 m mean error optimistically biased; a temporal or distance-based holdout check is needed.","rationale":"The reader's weakest_assumption correctly identifies the random split as the most load-bearing risk. The paper's contribution is an empirical evaluation of preprocessing and hyperparameter tuning on a single public dataset; the headline number is the test error. Since the data are sequential messages from a small number of vehicles, a random message-level split can place near-identical fingerprints from the same route into both training and test partitions. This would bias the error downward, and the fact that the test error (298 m) is lower than the validation error (317 m) for the final configuration is consistent with an easier test set, though not conclusive. I found no internal inconsistency in the preprocessing equations, and the paper's protocol for computing min only on the training set is correct. The release of code and split is genuine support for reproducibility, strengthening the paper's secondary claims. The proposed concrete test directly checks whether the split independence assumption holds; if it fails, the central accuracy claim is overstated, but the relative comparison of preprocessing methods might still be qualitatively valid. Therefore the conditional verdict remains appropriate: conditional on passing the leakage test, the paper's claims are acceptable. My read does not change the reader's verdict, so verdict_should_be is UNCHANGED.","tokens_in":10110,"tokens_out":4312,"duration_ms":45903,"concrete_test":"Using the released split (DOI:10.5281/zenodo.3228744), compute for each test message the great-circle distance to its nearest neighbor in the training set. Recompute mean and median error for the best configuration (Bray-Curtis, k=6, powed, τ=-159, β=2.6) after excluding test messages whose nearest training location is within 100 m (and separately within 200 m). If the mean error increases substantially (e.g., by more than 30% over 298 m), the original estimate is inflated by near-duplicate fingerprints. Additionally, re-split the data temporally per vehicle (training on the first 70% of messages per vehicle, validation/test on the later 15%/15%) and rerun the τ/β/k tuning; if the resulting test error is markedly larger than 298 m, temporal leakage is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dataset (Section III) consists of 14,378 Sigfox messages transmitted by only 20 postal-service vehicles, each with GPS ground truth. The train/validation/test split described in Section V is a random 70/15/15 split of messages, with no information about vehicle identity or timestamps used to enforce independence. Because kNN fingerprinting works by matching RSSI vectors, a test message taken from the same vehicle's route at a nearby location and time will have a nearly identical fingerprint to a training message, yielding an artificially small error. The reported test mean of 298 m and median of 109 m are point estimates from this single split after extensive validation-set tuning; the text reports no repeated splits, no confidence intervals, and no analysis of temporal or spatial leakage. This is the load-bearing concern because the paper's central claim (that the tuned preprocessing achieves 298 m mean error) depends directly on the test set being representative of unseen locations. If the test set contains near-duplicate fingerprints from the same trajectories, the reported accuracy is not a reliable estimate of outdoor localization performance for new data. The paper provides the exact split and code, so the concern is testable without re-collecting data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a reproducible study of RSSI fingerprinting for outdoor localization using a public Sigfox dataset from Antwerp, Belgium. The authors evaluate k-nearest-neighbor localization with several distance metrics and four data preprocessing schemes (positive, normalized, exponential, and powed representations), and they tune preprocessing parameters τ, α, β and the neighbor count k on a validation set. After selecting the best configuration on the validation set, they report a test-set mean error of 298 meters and a median error of 109 meters. The train/validation/test split and the code are made publicly available, and the results are compared with prior work by Janssen et al. and Aernouts et al.","tokens_in":10365,"tokens_out":5707,"duration_ms":53592,"significance":"If the reported accuracy holds, the paper provides a useful benchmark and a careful analysis of preprocessing choices for LPWAN fingerprinting, extending prior work on the same dataset. The strengths include the use of a public dataset, a clear separation of training, validation, and test sets, and the release of code and the exact data split, which facilitates reproducibility and comparability. However, the validity of the headline result depends critically on the test set being truly representative of unseen locations; the random message-level split raises a serious leakage concern that is not addressed in the manuscript. The paper also lacks statistical uncertainty quantification for the reported improvements.","major_comments":[{"comment":"The random 70/15/15 split of the 14,378 messages is performed at the message level without considering that the data were collected by only 20 postal vehicles, each emitting many messages along the same trajectories. A test message from a vehicle route that also appears in the training set will have an almost identical RSSI fingerprint to nearby training messages, so kNN will return an artificially small error. The paper does not report any analysis of temporal or spatial correlation between the train and test partitions, nor does it use vehicle identity to enforce independence. The reported 298 m mean and 109 m median test errors are therefore not yet established as unbiased estimates for unseen locations. The authors should either split by vehicle or by contiguous trajectory segments, or demonstrate (e.g., by computing the minimum time or distance gap between training and test points) that leakage is negligible. Because the split is published, this concern is directly testable.","section":"Section III and Section V"},{"comment":"Hyperparameters and preprocessing parameters are selected on a single validation split, and the reported improvements are not accompanied by any uncertainty estimates. For instance, the best threshold τ = −159 yields a mean validation error of 317 m versus 319 m for τ = −157 (Section VI-B), and the β tuning in Section VI-C changes the validation mean from 319 m to 318 m. These differences are far smaller than the sampling variability one would expect from a single split of this size, and the paper provides no confidence intervals, repeated splits, or significance tests. The claim that tuning these parameters 'further improves' accuracy requires evidence that the improvements are not within the noise of the evaluation protocol.","section":"Section VI-A and VI-B"},{"comment":"The test set is used for informal evaluation at multiple stages of the hyperparameter search: the paper reports test errors for the configuration of Table II (301 m), for τ = −159 (298 m), for α = 19 (318 m), for α = 18 with k = 4 (322 m), and for β = 2.6 (298 m). Although the final model is chosen on the validation set, each of these reported test results can influence the authors' subsequent choices, which is a form of adaptive data analysis that inflates the probability that the final test number is a favorable outlier. The authors should either pre-commit to a single final test evaluation, or clearly state that the test set was used multiple times and treat the reported values as exploratory rather than as a confirmatory estimate.","section":"Section VI-A through VI-C"}],"minor_comments":[{"comment":"The median test error for the final configuration is reported as 109 m in Section VI-B and in the abstract, but as 108 m in Section VI-C.2; please reconcile this discrepancy.","section":"Section VI-B and VI-C"},{"comment":"In the paragraph discussing Figure 7, the text says 'several tuples of values α and k' but the figure and analysis concern the parameters β and k; this should be corrected.","section":"Section VI-C.2"},{"comment":"The sentence 'the following conclusions can be drown' contains a typo; it should be 'drawn'.","section":"Section VI-A"},{"comment":"The citation to the dataset authors is misspelled as 'Aenrouts et al.'; it should be 'Aernouts et al.'.","section":"Section III"},{"comment":"The heading 'Parameters a and b of the Exponential and Eowed Data Representations' contains a typo: 'Eowed' should be 'Powed'.","section":"Section VI-C heading"},{"comment":"The word 'preproseccing' appears in the second paragraph; it should be 'preprocessing'.","section":"Section VII"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid contribution to reproducible LPWAN fingerprinting if the leakage concern can be resolved. The dataset is public and the authors provide code and the exact split, so the required follow-up analyses are feasible. I would encourage the editor to seek a revision that either demonstrates independence of the split or re-runs the evaluation with a vehicle-based or temporal holdout. Also note the inconsistency in the reported median error (108 vs 109 m), which should be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis one is worth a look if you work on LPWAN fingerprinting, but read the error bars carefully.\n\nWhat's new: the paper doesn't propose new methods. It takes the kNN/preprocessing framework from Torres-Sospedra and Janssen, and systematically tunes the preprocessing parameters tau, alpha, and beta that were left at WiFi defaults for the Antwerp Sigfox dataset. That tuning produces a mean test error of 298 m and median 109 m, down from the ~340 m reported with default parameters. The most useful contribution is procedural: a public train/validation/test split plus code, so future work can compare on identical data. The writing is straightforward and the authors are careful about preprocessing leakage, e.g. the min for normalization is computed from the training set only. Credit where due: this is a legitimate extension, not a rehash.\n\nWhere I'd push back: the 298 m is a point estimate from one random split. The dataset has 14,378 messages from 20 postal vehicles, and the split is done at the message level. Vehicles drive the same streets repeatedly; a random split will put messages from the same vehicle's trajectory into both train and test. For kNN, near-duplicate fingerprints in the training set are exactly what make test errors small. The paper reports no temporal or distance-based split, no vehicle-grouped evaluation, no repeated splits, and no confidence intervals. After an extensive validation search, reporting a single test number also gives no sense of selection bias. I don't think this is fatal, because the split is public and the concern is checkable, but it does mean the headline accuracy is likely optimistic and should not be quoted without that caveat.\n\nThe citation pattern is fair; no self-citation inflation, and the authors position their contribution relative to prior work honestly.\n\nBottom line: worth serious refereeing. I would tell the authors to add a vehicle/trajectory-aware split and repeated random splits, and to report the variance. If the 298 m survives that, it's a solid benchmark. If it doesn't, the paper still has value as a reproducible study of preprocessing sensitivity, but the accuracy claim needs to be reframed.","headline":"A careful, reproducible kNN tuning study on public Sigfox data with a genuinely useful public split, but the headline 298 m error comes from a single random split that likely leaks temporally and spatially correlated messages from the same postal vehicles into train and test.","tokens_in":10887,"tokens_out":3074,"would_cite":true,"duration_ms":34444,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Careful preprocessing, not algorithm choice, drives Sigfox localization accuracy.","keywords":["IoT","fingerprinting","Sigfox","localization","positioning","reproducibility","preprocessing","k-nearest neighbours"],"falsifier":"Re-run the reported best configuration ($\\tau = -159$, $\\beta = 2.6$, Bray-Curtis distance, $k = 6$) with a split that keeps all messages from each vehicle trajectory in one partition; if the mean test error rises substantially above 298 meters, the random-split result was inflated by correlated samples.","tokens_in":9918,"feed_emoji":"📡","tokens_out":7218,"duration_ms":61748,"temperature":0.7,"pith_summary":"This paper tries to establish that careful preprocessing, not just algorithm choice, is what unlocks accuracy in outdoor Sigfox localization by fingerprinting. Using a publicly available 14,378-message Sigfox dataset collected in Antwerp, it shows that nonlinear transformations of the received signal strength, together with a tuned threshold for out-of-range values and tuned transform parameters, far outperform the linear handling used in earlier work. Within a k-nearest-neighbour fingerprinting method, the tuned configuration reaches a mean test error of 298 meters and a median of 109 meters, with the best validation error at 317 meters. The paper releases its code and its exact train/validation/test split so future systems can be compared on equal terms.","feed_headline":"Tuning preprocessing cuts Sigfox location error to 298 m","feed_subtitle":"Nonlinear RSSI transforms and a tuned threshold beat linear scaling, with median error of 109 m.","key_machinery":"The central object is the parametric fingerprint transform applied to each RSSI vector before nearest-neighbour search. A threshold $\\tau$ replaces every value below it, including the artificial $-200$ placeholder for non-received signals, and the surviving values are mapped by one of four representations: positive shift, normalized, exponential $\\exp(\\mathrm{Positive}/\\alpha)$, or power-law $\\mathrm{Positive}^{\\beta}$. These transforms matter because RSSI is already logarithmic, so linear scaling distorts the distances between fingerprints; tuning $\\tau$, $\\alpha$, and $\\beta$ to the Sigfox signal distribution adapts the geometry of the fingerprint space. The work this machinery does is to turn a fixed preprocessing recipe into a tunable part of model selection, and the paper shows most of the accuracy gain comes from that step.","core_discovery":"Within a k-nearest-neighbour fingerprinting pipeline, the paper's central claim is that the way RSSI values are transformed before distance computation determines localization accuracy more than the choice of distance metric or $k$. On a public Sigfox dataset from Antwerp, representing each message as a vector of per-base-station RSSIs and comparing fingerprints with the Bray-Curtis distance, the authors find that replacing missing receptions with a threshold just below the smallest observed RSSI ($\\tau = -159$) and using a power-law transform with exponent $\\beta = 2.6$ gives a mean test error of 298 meters and a median of 109 meters. This improves on a 344-meter validation error with default transform parameters and on a 552-meter error from linear scaling. The same tuning also flips the preferred representation from exponential to power-law, showing that the preprocessing choices interact.","pith_inferences":["A trajectory-aware split, holding out entire vehicle trips rather than individual messages, would test whether the random split inflates the reported accuracy; the paper does not report this check.","The same threshold-and-exponent tuning recipe should transfer to the companion LoRaWAN and rural Sigfox datasets released with the Antwerp data; if it does, preprocessing tuning is a general LPWAN fingerprinting step rather than a dataset-specific fix.","The gap between mean error (298 m) and median error (109 m) implies a heavy-tailed error distribution; reporting error quantiles or a map of failures would show where the remaining errors concentrate."],"forward_implications":["On this urban Sigfox test set, tuned nonlinear transforms bring mean error down from above 500 meters under linear scaling to 298 meters, so preprocessing deserves the same scrutiny as the learning algorithm.","Setting the out-of-range placeholder just below the smallest received RSSI (here $-159$) outperforms keeping the dataset's artificial $-200$ value.","Default transform parameters imported from indoor Wi-Fi fingerprinting ($\\alpha = 24$, $\\beta = e$) are not transferable; retuning $\\beta$ to $2.6$ alone lowers the validation error.","Because the code and exact train/validation/test split are public, future methods can report test error on the same 2,157-message test set, making cross-paper comparisons meaningful."],"supporting_citations":[{"why":"Supplies the public Sigfox dataset of 14,378 messages and the original 689-meter mean-error baseline, and defines the material for the train/validation/test split.","marker":"[3]"},{"why":"Establishes the prior best configuration on the same dataset (340-meter validation error) and the set of distance metrics and preprocessing choices this paper tunes further.","marker":"[13]"},{"why":"Introduces the four RSSI data representations (positive, normalized, exponential, powed) with default parameters $\\alpha = 24$ and $\\beta = e$ that this paper adapts.","marker":"[14]"}],"fun_headline_variants":["Preprocessing beats k and metric for Sigfox localization","Nonlinear RSSI transform cuts mean error to 298 m","Threshold and power-law tune Sigfox fingerprinting to 298 m","Why RSSI transform matters more than k or distance metric","Out-of-range RSSI fix and beta=2.6 yield 298 m error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that randomly splitting the 14,378 messages into 70/15/15 gives independent training and test samples; if messages from the same vehicle trajectory appear on both sides, the reported 298-meter test error is optimistically biased.","fun_headline_variants_meta":{"raw":{"variants":["Preprocessing beats k and metric for Sigfox localization","Nonlinear RSSI transform cuts mean error to 298 m","Threshold and power-law tune Sigfox fingerprinting to 298 m","Why RSSI transform matters more than k or distance metric","Out-of-range RSSI fix and beta=2.6 yield 298 m error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000938,"raw_usage":{"total_tokens":4015,"prompt_tokens":955,"completion_tokens":3060,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2971}},"tokens_in":571,"tokens_out":3060,"duration_ms":20643,"temperature":1.0,"reasoning_tokens":2971,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:24:51.045934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the reported best configuration ($\\tau = -159$, $\\beta = 2.6$, Bray-Curtis distance, $k = 6$) with a split that keeps all messages from each vehicle trajectory in one partition; if the mean test error rises substantially above 298 meters, the random-split result was inflated by correlated samples.","supporting_citations":[],"review_version":1}