REVIEW 3 major objections 2 minor 4 references
A Novel Evaluation Metric for Unsupervised Learning in AIS-Based Maritime Anomaly Detection: MADQI
T0 review · 3 major / 2 minor · reviewed 2026-06-29 · grok-4.3
Pith's one-line read MADQI is a composite metric that evaluates unsupervised anomaly detection in maritime AIS data without requiring labels.
desk verdict MADQI introduces a four-part composite for scoring unsupervised AIS anomaly detectors without labels, but the physical plausibility and extreme-case components appear to embed fixed maritime thresholds that undermine the unsupervised claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The Maritime Anomaly Detection Quality Index (MADQI), a composite index that integrates ARC, PPS, SDS, and ECE using Haversine distance calculations for spatial analysis and adaptive scaling for combination without labelled data.
What would settle it
A direct comparison of MADQI scores with performance on a held-out labeled AIS dataset to check if higher MADQI predicts better actual anomaly detection accuracy.
Extended reading notes
Core claim
The proposed MADQI evaluation framework integrates four interconnected metrics: Anomaly Rate Consistency (ARC), Physical Plausibility Score (PPS), Score Distribution Separation (SDS), and Extreme Case Evidence (ECE). These metrics are combined through automatic normalisation using multi-chunk evaluation and adaptive scaling techniques, achieving a MADQI score of 80.37% on the AIS dataset for unsupervised anomaly detection.
Load-bearing premise
The four sub-metrics combined through automatic normalisation provide a reliable and unbiased measure of anomaly detection performance without any labelled data.
Editorial extensions
If this is right
- Machine learning models for anomaly detection can be evaluated consistently in the absence of ground truth labels.
- The framework emphasizes physical plausibility and detection of extreme anomalies in vessel behaviors.
- Anomaly rate consistency is maintained across different data chunks.
- Strong performance in identifying abnormal vessel movements related to speed, position, time, and turns.
Reading between the lines
- Applying MADQI to other unsupervised anomaly detection tasks beyond maritime data could provide label-free evaluation.
- Integrating MADQI with existing algorithms might improve reliability assessments in real-time maritime monitoring systems.
- Further validation could involve testing on varied AIS datasets from different regions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces the Maritime Anomaly Detection Quality Index (MADQI), a composite label-free metric for evaluating unsupervised anomaly detectors on AIS data. MADQI integrates four sub-metrics—Anomaly Rate Consistency (ARC), Physical Plausibility Score (PPS), Score Distribution Separation (SDS), and Extreme Case Evidence (ECE)—combined via automatic normalisation, multi-chunk evaluation, and adaptive scaling. The central empirical claim is that the framework achieves a MADQI score of 80.37% (with ARC=1.000 and ECE=0.907) on an AIS dataset, demonstrating reliable evaluation of models such as Isolation Forest for anomalies in speed, position jumps, time gaps, and turn angles.
Significance. If the sub-metrics can be shown to be computed strictly from model outputs without external heuristics, MADQI would address a genuine gap in unsupervised evaluation for maritime anomaly detection. The attempt to combine consistency, physical plausibility, distributional separation, and extreme-case evidence into a single index is conceptually coherent and potentially useful where labels are unavailable. No machine-checked proofs or reproducible code are provided, but the multi-component design itself is a constructive contribution if the circularity concerns are resolved.
major comments (3)
- [Abstract] Abstract: The claim that MADQI supplies an unbiased evaluation 'without requiring labelled data' is load-bearing for the central contribution, yet PPS and ECE are defined using Haversine distance on spatial/behavioural thresholds (speed, position jumps, turn angles) that match the very anomaly types the detector is intended to discover. No derivation is supplied showing these thresholds are obtained solely from the model's score distribution rather than pre-specified maritime rules; this renders the 80.37% score a hybrid measure whose interpretation as purely unsupervised quality cannot be assessed from the presented material.
- [Abstract] Abstract (experimental results paragraph): The reported MADQI value of 80.37% is given without the explicit combination formula, the adaptive scaling parameters, the number of chunks, or the individual sub-metric values beyond ARC and ECE. In the absence of these details, the numerical result cannot be reproduced or validated, directly undermining the effectiveness claim.
- [Abstract] Abstract: No baselines, alternative unsupervised evaluation methods, or comparison against supervised metrics on any labelled subset are reported, nor are error bars or sensitivity analyses provided for the 80.37% figure. This omission leaves the superiority and stability of MADQI unestablished relative to simpler or existing approaches.
minor comments (2)
- [Abstract] Abstract: Typo 'prosed MADQI' should read 'proposed MADQI'.
- [Abstract] Abstract: Dataset size, exact unsupervised algorithm configuration, and preprocessing steps are not stated, hindering assessment of the experimental setup.
Simulated Author's Rebuttal
We thank the referee for the constructive comments on our manuscript. We respond point by point to the major comments, indicating planned revisions where appropriate to improve clarity and reproducibility.
read point-by-point responses
-
Referee: [Abstract] Abstract: The claim that MADQI supplies an unbiased evaluation 'without requiring labelled data' is load-bearing for the central contribution, yet PPS and ECE are defined using Haversine distance on spatial/behavioural thresholds (speed, position jumps, turn angles) that match the very anomaly types the detector is intended to discover. No derivation is supplied showing these thresholds are obtained solely from the model's score distribution rather than pre-specified maritime rules; this renders the 80.37% score a hybrid measure whose interpretation as purely unsupervised quality cannot be assessed from the presented material.
Authors: The Haversine distance provides a standard, label-free geometric computation from raw AIS latitude/longitude coordinates. Thresholds for speed, jumps, and angles incorporate established maritime domain knowledge to evaluate physical plausibility, which is essential for meaningful interpretation in this domain; purely score-derived thresholds without domain context would not assess plausibility. We acknowledge the abstract does not explicitly derive the thresholds or clarify this hybrid aspect. In revision we will add a dedicated paragraph in the methods section deriving the thresholds from a combination of statistical analysis of the score distribution and standard AIS maritime rules, and update the abstract to state that the metric remains label-free while using domain knowledge for physical checks. revision: yes
-
Referee: [Abstract] Abstract (experimental results paragraph): The reported MADQI value of 80.37% is given without the explicit combination formula, the adaptive scaling parameters, the number of chunks, or the individual sub-metric values beyond ARC and ECE. In the absence of these details, the numerical result cannot be reproduced or validated, directly undermining the effectiveness claim.
Authors: We agree that the abstract omits critical details needed for reproducibility. The full manuscript defines the combination as a normalised weighted sum with adaptive scaling based on per-chunk variance and uses five chunks. To resolve this, we will revise the abstract to include the explicit combination formula, the adaptive scaling parameters, the number of chunks (five), and the complete set of sub-metric values (ARC=1.000, PPS, SDS, ECE=0.907). revision: yes
-
Referee: [Abstract] Abstract: No baselines, alternative unsupervised evaluation methods, or comparison against supervised metrics on any labelled subset are reported, nor are error bars or sensitivity analyses provided for the 80.37% figure. This omission leaves the superiority and stability of MADQI unestablished relative to simpler or existing approaches.
Authors: No standard unsupervised evaluation baselines exist specifically for AIS maritime anomaly detection, which motivated the new metric; the manuscript therefore emphasises the design of MADQI rather than comparative experiments. The dataset used contains no labels, precluding supervised metric comparisons. We will add a discussion subsection explaining the absence of direct baselines and include a sensitivity analysis (varying chunk count and scaling factors) with error bars derived from multi-chunk results to demonstrate stability of the 80.37% score. revision: partial
Circularity Check
No significant circularity; MADQI is an explicitly author-defined composite metric with no reduction of claims to inputs by construction.
full rationale
The paper defines MADQI directly as the composite of four author-specified sub-metrics (ARC, PPS, SDS, ECE) combined via automatic normalisation, multi-chunk evaluation and adaptive scaling; the reported 80.37% is simply the value obtained by applying this definition to the AIS data. No derivation chain, prediction, or first-principles result is asserted that reduces to the inputs by construction. No self-citation load-bearing step, uniqueness theorem, or ansatz smuggling appears in the provided text. The framework is presented as a novel proposal rather than an independent derivation, so the evaluation remains self-contained by design. Concerns about domain heuristics in PPS/ECE are correctness issues, not circularity.
Assumptions & free parameters
free parameters (1)
- adaptive scaling factors and normalisation parameters
assumptions (2)
- standard math Haversine formula provides accurate great-circle distances for vessel positions
- ad hoc to paper The four metrics ARC, PPS, SDS, ECE capture distinct and relevant aspects of anomaly detection quality
invented entities (1)
-
MADQI
Cite this review
Pith. "Pith review of A Novel Evaluation Metric for Unsupervised Learning in AIS-Based Maritime Anomaly Detection: MADQI." pith.science (2026). https://pith.science/paper/CKJEPIW5
@misc{pith2026260530388,
author = {Pith},
title = {Pith review of: A Novel Evaluation Metric for Unsupervised Learning in AIS-Based Maritime Anomaly Detection: MADQI},
year = {2026},
howpublished = {\url{https://pith.science/paper/CKJEPIW5}},
note = {Machine review of arXiv:2605.30388}
}
read the original abstract
This paper introduces a new systematic framework for detecting anomalies in maritime Automatic Identification System (AIS) datasets. These anomalies include abnormal vessel behaviours related to speed, position jumps, time gaps, and turn angles. Although unsupervised learning algorithms such as Isolation Forest are widely used for detecting anomalous vessel movements, they often lack systematic and meaningful evaluation measures. To address this limitation, we propose a novel quality metric called Maritime Anomaly Detection Quality Index (MADQI). The prosed MADQI is a composite index designed to evaluate the anomaly detection performance of machine learning models without requiring labelled data. The proposed framework uses Haversine distance calculations to analyse AIS datasets and identify anomalies based on their spatial and behavioural characteristics. The proposed MADQI evaluation framework integrates four interconnected metrics: Anomaly Rate Consistency (ARC), Physical Plausibility Score (PPS), Score Distribution Separation (SDS), and Extreme Case Evidence (ECE). These metrics are combined through automatic normalisation using multi-chunk evaluation and adaptive scaling techniques. Experimental results on the AIS dataset show that the proposed framework achieved a MADQI score of 80.37%, demonstrating its effectiveness for unsupervised anomaly detection. In particular, the algorithm performed strongly in identifying abnormal vessel behaviour. Among the individual MADQI components, ECE and ARC achieved scores of 0.907 and 1.000, respectively, indicating excellent capability in detecting extreme anomalies and maintaining anomaly rate consistency. Overall, these results are encouraging and demonstrate that the proposed framework provides a reliable and meaningful approach for evaluating unsupervised anomaly detection in maritime AIS data.
Figures
Reference graph
Works this paper leans on
-
[1]
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. Goodfellow, I., Bengio, Y ., & Courville, A. (2016). Deep Learning. MIT Press. https://www.deeplearningbook.org/ Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. https://link.springer...
-
[2]
https://coast.noaa.gov/htdata/CMSP/AISDataHandler/2022/index.html Pallotta, G., Vespe, M., & Bryan, K. (2013). Vessel Pattern Knowledge Discovery from AIS Data. Entropy, 15(6), 2218–2245. https://doi.org/10.3390/e15062218 Riveiro, M., Pallotta, G., & Vespe, M. (2018). Maritime anomaly detection: A review. Wiley Interdisciplinary Reviews: Data Mining and K...
-
[3]
https://doi.org/10.1162/089976601750264965 24 Sinnott, R. W. (1984). Virtues of the Haversine. Sky and Telescope, 68,
-
[4]
https://www.scirp.org/reference/referencespapers?referenceid=2500014 Tu, E., Zhang, G., Rachmawati, L., Rajabally, E., & Huang, G.-B. (2018). Exploiting AIS data for intelligent maritime navigation. IEEE T-ITS. https://doi.org/10.1109/TITS.2017.2724551 Zhang, H., Li, W., Shi, G., Desrosiers, R. & Wang, X. (2024). A ship trajectory clustering algorithm bas...
Reviewed June 29, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.