REVIEW 4 major objections 6 minor 31 references
A Data-Driven Novelty Score for Diverse In-Vehicle Data Recording
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A novelty filter can shrink driving datasets while improving classifier accuracy.
desk verdict Plausible streaming novelty filter, but the main generalization claim rides on a single stream order; deserves a careful review, not a desk reject. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is the dynamic Mean Shift algorithm, a streaming extension of the static mean-shift novelty test. It maintains an empirical mean $\mu_n(t)$ and Ledoit-Wolf-shrunk covariance $\Sigma_n(t)$ of previously accepted object embeddings, scores each new patch by the unnormalized Hotelling $\tilde{T}^2$ statistic $(\mu^* - \mu_n)^\top \Sigma^{-1} (\mu^* - \mu_n)$, and updates both statistics online via recurrences (3) and (4) when a frame is accepted as novel. This gives constant-time scoring independent of the number of seen samples, which is what makes 32 FPS real-time operation possible.
What would settle it
Run the same dynamic novelty filter on many random shuffles of one traffic-sign dataset, train the identical classifier on each selected subset, and measure the spread of test accuracies; if the improvement over random sampling disappears or reverses for some orders, the claimed benefit is an artifact of a single stream order.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that novelty-based dataset reduction reverses the usual volume-value trade-off: smaller training sets selected by the dynamic Mean Shift score can beat the full dataset, and the more redundant the original stream, the more aggressively it can be filtered without loss. The mechanism is a dynamic Hotelling T-squared statistic: each object patch is embedded with EfficientNet, and frames whose mean-shift score exceeds threshold T are recorded and folded into the running mean and covariance (updated by closed-form recurrences), while low-scoring frames are discarded. The authors show that across redundancy factors 1x to 16x and five thresholds, the best test accuracy consistently came from novelty-filtered sets around 80 percent of the original size, and that random sampling of equal size is less stable and more prone to overfitting.
Load-bearing premise
The evaluation assumes that a single randomized order of the data stream is representative, even though whether a frame counts as novel depends on what has already been seen.
Editorial extensions
If this is right
- In-vehicle logging can discard most redundant frames while preserving rare objects, cutting storage and downstream training cost without sacrificing classifier accuracy.
- The larger the redundancy in the raw stream, the more aggressive the filter can be: at redundancy factor 8, over 90 percent of frames could be dropped with no loss.
- Random sampling of the same size is a weaker baseline: it occasionally matches but is less consistent and tends to overfit.
- The optimal filtered set size in these experiments was consistently about 80 percent of the original data, suggesting a practical target for logging budgets.
- Because scoring and update cost do not grow with the number of stored samples, the method is deployable on embedded hardware at real-time frame rates.
Reading between the lines
- The order-dependence of the novelty score means results may vary across different stream orders; testing the same pipeline over many shuffles would show how much of the gain is robust.
- The same online covariance mechanism should extend to scene-level novelty by replacing object patches with whole-frame embeddings, though the authors note mixture models may be needed to handle multi-modal normality.
- A natural testable extension is combining the novelty score with a diversity or coverage term, since pure novelty filtering can over-concentrate on the tail of the distribution.
- The fixed threshold $T$ is a free parameter that trades reduction rate against overfitting; an adaptive threshold schedule could make the method parameter-free in practice.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an online, object-level novelty filter for in-vehicle video streams. Each detected object patch is embedded with a pretrained EfficientNet-B4, and a dynamic mean and covariance model of 'normal' patches is maintained; a patch is recorded when its Mahalanobis distance exceeds a threshold T, and the model is updated only with accepted patches. The authors derive recursive mean and covariance updates (Eqs. 3-4), measure runtime on a traffic-sign dataset, and evaluate downstream classifier accuracy after filtering under redundancy factors 1x-16x, comparing novelty filtering with random sampling and with using all data. The main claims are that novelty filtering can reduce dataset size while improving or maintaining test accuracy, that more aggressive filtering becomes beneficial as redundancy increases, that random sampling is less stable, and that the method runs in real time at 32 FPS with constant-time operation.
Significance. If the central claims hold, the method would be practically valuable for data-logging pipelines in autonomous driving, since it is lightweight, memory-bounded, and based on a sound recursive update of the normality model. The controlled redundancy experiment is a useful evaluation design, and the diversity and balance tables (Tables II and III) provide supporting evidence. The algebraic derivations in Appendix A1 and A2 are correct as far as I checked. However, the evaluation as presented does not yet establish the main performance claims: the online selection is order-dependent but only one stream order is used, the best threshold per redundancy factor is selected post hoc from the sweep, Fig. 5 has no error bars or significance tests despite five classifier seeds, and the real-time/constant-time assertion is not backed by a complete algorithmic description. These gaps are fixable with additional experiments and analysis, so the paper is promising but not yet ready in its current form.
major comments (4)
- [V-B.2 / Fig. 5] The evaluation uses a single randomized stream order per redundancy-factor/threshold configuration, but the selection rule is inherently order-dependent because Eqs. (3)-(4) update the Gaussian model only with accepted patches; the same sample is more likely to be accepted early in the stream than late. The five seeds are used only for classifier training, not for stream permutations, so the reported improvements (e.g., RF1 test accuracy improving from 42% to 44%) could be an artifact of one favorable ordering. Please repeat the full filtering pipeline over multiple stream permutations and report per-threshold distributions with error bars and a significance test.
- [V-A / Eq. (4)] The real-time and constant-time claims are not backed by the paper's update scheme. Eq. (2) requires the inverse covariance matrix Sigma^{-1}, but Eq. (4) updates only Sigma; the manuscript does not state how the inverse is maintained or recomputed after each accepted patch. If the inverse is recomputed from scratch, the cost is O(d^3) with d=2560, which contradicts the constant-time claim. In addition, the numbers in Table I for 64 patches at 64x64 sum to about 32.66 ms per frame (14.33 ms novelty score plus 18.33 ms covariance update), which corresponds to about 30.6 FPS, not the claimed 32 FPS. Please specify the inverse-update procedure and report end-to-end worst-case latency.
- [V-B.3 / Fig. 5] The claim that the optimal training set size is 'consistently 80% of the original dataset size' is internally inconsistent: for RF1 the text reports an improvement when 10% of the data is discarded (i.e., 90% retained), while the reductions shown in Fig. 5 span 25-75%. Also, the best threshold per redundancy factor appears to be selected post hoc from the five thresholds (2500, 5000, 10000, 15000, 30000), and Fig. 5 shows no error bars despite five classifier seeds. Please clarify the 80% statement, report results for all thresholds rather than only the best one, and add confidence intervals or per-seed scatter.
- [V-B.2 / Fig. 5] The random-sampling baselines are single draws from the data, so the comparison in Fig. 5 does not quantify sampling randomness on either side of the comparison. Since the novelty-filtered dataset also depends on the random stream order, a fair comparison requires multiple random subsets and multiple stream orders; otherwise the claim that random sampling has higher variance than novelty filtering is not supported by the data shown.
minor comments (6)
- [II] There is a typo in the problem setup: 'video steam' should be 'video stream', and the frame domain is written as f in R^{w x h x c} although the text defines height h and width w; please use R^{h x w x c} consistently.
- [III / Eq. (2)] The notation for the test sample is inconsistent: the text defines mu^*_c = mu^* - mu_n, but Eq. (2) writes the novelty score as N(mu^*_i, mu_n, Sigma_n); please align the subscripts.
- [V] The dataset is called GTSB, but the cited benchmark is the German Traffic Sign Recognition Benchmark (GTSRB); please use the correct acronym to avoid confusion.
- [V-A / Table I] The caption says the runtime analysis is averaged over 8165 computations, but the training set contains 8164 images; please reconcile this number.
- [V-B.2] The description of redundancy factors is vague: 'duplicating the dataset accordingly' should state whether duplication is exact replication of the whole set, whether it is with or without replacement, and how the 8164-image stream is ordered relative to the duplicates.
- [Fig. 5] The figure does not state whether the plotted markers are means over the five seeds or individual runs, and the marker symbols are difficult to distinguish in grayscale; please add a clear statement and use more distinguishable markers or error bars.
Circularity Check
No circularity: the novelty score is an independent filter and the performance claim is evaluated by an external downstream classifier.
full rationale
The claimed chain is: embed object patches with a frozen pretrained EfficientNet; maintain an online mean and covariance; compute a Mahalanobis-distance novelty score (Eq. 2); threshold to record or discard; then train a separate traffic-sign classifier on the selected data and measure held-out GTSB test accuracy. The novelty score is not optimized against the downstream classifier, so the central performance result is external to the method's own definition. Eqs. (3)-(4) are algebraic identities for updating mean and covariance, proved in the Appendix from the definitions, and they do not assume the target finding. The mildest concern is that the 'optimal training set size' is identified from the same threshold sweep whose best test accuracy is then reported; that is post-hoc model selection rather than a predictive derivation, and it is a statistical/correctness issue, not a circular one. No fitted parameter is renamed as a prediction, and no equation reduces to its own input. The self-citations ([4], [5], [18], [28]) are contextual and non-load-bearing, while the stream-order sensitivity raised by the reader is a robustness limitation, not a circular step. Hence no significant circularity is present.
Assumptions & free parameters
free parameters (4)
- Novelty threshold T =
2500, 5000, 10000, 15000, 30000 (swept)
- Patch size =
64x64 pixels
- Initial Gaussian noise statistics =
unspecified
- EfficientNet-B4 intermediate layer =
after fifth block, feature dimension 2560
assumptions (5)
- domain assumption Incoming frames come with object patches already detected, and each patch is cropped and resized before embedding.
- domain assumption Pretrained EfficientNet features are a sufficient representation for defining normal versus novel content.
- ad hoc to paper A single randomized stream order is representative of the data stream.
- ad hoc to paper The inverse covariance required by Eq. (2) can be maintained efficiently at runtime.
- domain assumption The empirical covariance remains well-conditioned after online updates.
Cite this review
Pith. "Pith review of A Data-Driven Novelty Score for Diverse In-Vehicle Data Recording." pith.science (2026). https://pith.science/paper/IF4BBJGO
@misc{pith2026250704529,
author = {Pith},
title = {Pith review of: A Data-Driven Novelty Score for Diverse In-Vehicle Data Recording},
year = {2026},
howpublished = {\url{https://pith.science/paper/IF4BBJGO}},
note = {Machine review of arXiv:2507.04529}
}
read the original abstract
High-quality datasets are essential for training robust perception systems in autonomous driving. However, real-world data collection is often biased toward common scenes and objects, leaving novel cases underrepresented. This imbalance hinders model generalization and compromises safety. The core issue is the curse of rarity. Over time, novel events occur infrequently, and standard logging methods fail to capture them effectively. As a result, large volumes of redundant data are stored, while critical novel cases are diluted, leading to biased datasets. This work presents a real-time data selection method focused on object-level novelty detection to build more balanced and diverse datasets. The method assigns a data-driven novelty score to image frames using a novel dynamic Mean Shift algorithm. It models normal content based on mean and covariance statistics to identify frames with novel objects, discarding those with redundant elements. The main findings show that reducing the training dataset size with this method can improve model performance, whereas higher redundancy tends to degrade it. Moreover, as data redundancy increases, more aggressive filtering becomes both possible and beneficial. While random sampling can offer some gains, it often leads to overfitting and unpredictability in outcomes. The proposed method supports real-time deployment with 32 frames per second and is constant over time. By continuously updating the definition of normal content, it enables efficient detection of novelties in a continuous data stream.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
CoCar NextGen: a Multi-Purpose Platform for Connected Autonomous Driving Research
M. Heinrich, M. Zipfl, M. Uecker, S. Ochs, M. Gontscharow, T. Fleck, J. Doll, P. Schörner, C. Hubschneider, M. R. Zofka, A. Viehl, and J. M. Zöllner, “Cocar nextgen: a multi-purpose platform for connected autonomous driving research,” 2024. [Online]. Available: https://arxiv.org/abs/2404.17550
work page Pith review arXiv 2024
-
[2]
Curse of rarity for autonomous vehicles,
H. X. Liu and S. Feng, “Curse of rarity for autonomous vehicles,” Nature Communications, vol. 15, no. 1, p. 4808, Jun 2024. [Online]. Available: https://doi.org/10.1038/s41467-024-49194-0
-
[3]
Iso/pas 8800:2024 road vehicles — safety and artificial intel- ligence,
ISO, “Iso/pas 8800:2024 road vehicles — safety and artificial intel- ligence,” International Organization for Standardization, Geneva, CH, Standard, Dec. 2024
work page 2024
-
[4]
J. Langner, R. Pohl, J. Ransiek, P. Elspas, and E. Sax, “Statistical consideration of the representativeness of open road tests for the validation of automated driving systems,” 10 2023, pp. 1–8
work page 2023
-
[5]
Towards a data engineering process in data-driven systems engineering,
P. Petersen, H. Stage, J. Langner, L. Ries, P. Rigoll, C. Philipp Hohl, and E. Sax, “Towards a data engineering process in data-driven systems engineering,” in 2022 IEEE International Symposium on Systems Engineering (ISSE) , 2022, pp. 1–8
work page 2022
-
[6]
Systematization of corner cases for visual perception in automated driving,
J. Breitenstein, J.-A. Termöhlen, D. Lipinski, and T. Fingscheidt, “Systematization of corner cases for visual perception in automated driving,” in 2020 IEEE Intelligent V ehicles Symposium (IV) , 2020, pp. 1257–1264
work page 2020
-
[7]
The fishyscapes benchmark: Measuring blind spots in semantic segmentation,
H. Blum, P.-E. Sarlin, J. Nieto, R. Siegwart, and C. Cadena, “The fishyscapes benchmark: Measuring blind spots in semantic segmentation,” International Journal of Computer Vision , vol. 129, no. 11, pp. 3119–3135, Nov 2021. [Online]. Available: https://doi.org/10.1007/s11263-021-01511-6
-
[8]
Segmentmeifyoucan: A benchmark for anomaly segmentation,
R. Chan, K. Lis, S. Uhlemeyer, H. Blum, S. Honari, R. Siegwart, P. Fua, M. Salzmann, and M. Rottmann, “Segmentmeifyoucan: A benchmark for anomaly segmentation,” in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , J. Vanschoren and S. Yeung, Eds., vol. 1, 2021. [Online]. Available: https://datasets-benchmarks-proc...
work page 2021
Show all 31 references
-
[9]
Scaling out-of-distribution detection for real-world settings,
D. Hendrycks, S. Basart, M. Mazeika, A. Zou, M. Mostajabi, J. Steinhardt, and D. X. Song, “Scaling out-of-distribution detection for real-world settings,” in International Conference on Machine Learning, 2022. [Online]. Available: https://api.semanticscholar.org/ CorpusID:227407829
2022
-
[10]
Anovox: A benchmark for multimodal anomaly detection in autonomous driving,
D. Bogdoll, I. Hamdard, L. N. Rößler, F. Geisler, M. Bayram, F. Wang, J. Imhof, M. de Campos, A. Tabarov, Y . Yang, H. Gottschalk, and J. M. Zöllner, “Anovox: A benchmark for multimodal anomaly detection in autonomous driving,” CoRR, vol. abs/2405.07865, 2024. [Online]. Availa...
-
[11]
Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding,
A. Kendall, V . Badrinarayanan, and R. Cipolla, “Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding,” CoRR, vol. abs/1511.02680, 2015. [Online]. Available: http://arxiv.org/abs/1511.02680
2015 arXiv
-
[12]
Densely connected normalizing flows,
M. Grci ´c, I. Grubiši ´c, and S. Šegvi ´c, “Densely connected normalizing flows,” 2021. [Online]. Available: https://arxiv.org/abs/2106.04627
2021 arXiv
-
[13]
Predictive uncertainty estimation via prior networks,
A. Malinin and M. Gales, “Predictive uncertainty estimation via prior networks,” 2018. [Online]. Available: https://arxiv.org/abs/1802.10501
2018 arXiv
-
[14]
Fast and scalable outlier detection with approximate nearest neighbor ensembles,
E. Schubert, A. Zimek, and H.-P. Kriegel, “Fast and scalable outlier detection with approximate nearest neighbor ensembles,” in Database Systems for Advanced Applications , M. Renz, C. Shahabi, X. Zhou, and M. A. Cheema, Eds. Cham: Springer International Publishing, 2015, pp. 19–36
2015
-
[15]
Algorithms for mining distance-based outliers in large datasets,
E. M. Knorr and R. T. Ng, “Algorithms for mining distance-based outliers in large datasets,” in Proceedings of the 24rd International Conference on V ery Large Data Bases, ser. VLDB ’98. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1998, p. 392–403
1998
-
[16]
Deep Transfer Learning for Multiple Class Novelty Detection ,
P. Perera and V . M. Patel, “ Deep Transfer Learning for Multiple Class Novelty Detection ,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2019, pp. 11 536– 11 544. [Online]. Available: https:...
2019
-
[17]
Unsupervised anomaly detection with generative adver- sarial networks to guide marker discovery,
T. Schlegl, P. Seeböck, S. M. Waldstein, U. Schmidt-Erfurth, and G. Langs, “Unsupervised anomaly detection with generative adver- sarial networks to guide marker discovery,” in Information Processing in Medical Imaging , M. Niethammer, M. Styner, S. Aylward, H. Zhu, I. Oguz, P...
2017
-
[18]
Isolation forest for anomaly detection in raw vehicle sensor data,
J. Hofmockel. and E. Sax., “Isolation forest for anomaly detection in raw vehicle sensor data,” in Proceedings of the 4th International Conference on V ehicle Technology and Intelligent Transport Systems - VEHITS , 2018
2018
-
[19]
f-anogan: Fast unsupervised anomaly detection with generative adversarial networks,
T. Schlegl, P. Seeböck, S. M. Waldstein, G. Langs, and U. Schmidt- Erfurth, “f-anogan: Fast unsupervised anomaly detection with generative adversarial networks,” Medical Image Analysis , vol. 54, pp. 30–44, 2019. [Online]. Available: https://www.sciencedirect.com/ science/arti...
2019
-
[20]
CSI: novelty detection via contrastive learning on distributionally shifted instances,
J. Tack, S. Mo, J. Jeong, and J. Shin, “CSI: novelty detection via contrastive learning on distributionally shifted instances,” CoRR, vol. abs/2007.08176, 2020. [Online]. Available: https: //arxiv.org/abs/2007.08176
2007 arXiv
-
[21]
Submodularity in data subset selection and active learning,
K. Wei, R. Iyer, and J. Bilmes, “Submodularity in data subset selection and active learning,” in Proceedings of the 32nd International Con- ference on International Conference on Machine Learning - V olume 37, ser. ICML’15. JMLR.org, 2015, p. 1954–1963
2015
-
[22]
Learning from less data: Diversified subset selection and active learning in image classification tasks,
V . Kaushal, A. Sahoo, K. Doctor, N. R. Uppalapati, S. Shetty, P. Singh, R. K. Iyer, and G. Ramakrishnan, “Learning from less data: Diversified subset selection and active learning in image classification tasks,” ArXiv, vol. abs/1805.11191, 2018. [Online]. Available: https://a...
2018 arXiv
-
[23]
Semantic redundancies in image-classification datasets: The 10% you don’t need,
V . Birodkar, H. Mobahi, and S. Bengio, “Semantic redundancies in image-classification datasets: The 10% you don’t need,” ArXiv, vol. abs/1901.11409, 2019. [Online]. Available: https://api.semanticscholar. org/CorpusID:59523736
1901 arXiv
-
[24]
Making stochastic neural networks from deterministic ones,
K. Lee, J. Kim, S. Chong, and J. Shin, “Making stochastic neural networks from deterministic ones,” 2017. [Online]. Available: https://openreview.net/forum?id=B1akgy9xx
2017
-
[25]
Deepcore: A comprehensive library for coreset selection in deep learning,
C. Guo, B. Zhao, and Y . Bai, “Deepcore: A comprehensive library for coreset selection in deep learning,” in International Conference on Database and Expert Systems Applications , 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:248239610
2022
-
[26]
Coresets for data- efficient training of machine learning models,
B. Mirzasoleiman, J. Bilmes, and J. Leskovec, “Coresets for data- efficient training of machine learning models,” in Proceedings of the 37th International Conference on Machine Learning , ser. ICML’20. JMLR.org, 2020
2020
-
[27]
Grad-match: Gradient matching based data subset selection for efficient deep model training,
K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, A. De, and R. Iyer, “Grad-match: Gradient matching based data subset selection for efficient deep model training,” 2021. [Online]. Available: https://arxiv.org/abs/2103.00123
2021 arXiv
-
[28]
Behavior forests: Real-time discovery of dynamic behavior for data selection,
P. Reis, P. Rigoll, and E. Sax, “Behavior forests: Real-time discovery of dynamic behavior for data selection,” 2024. [Online]. Available: https://arxiv.org/abs/2407.02008
2024 arXiv
-
[29]
Fast and efficient image novelty detection based on mean- shifts,
M. Hermann, G. Umlauf, B. Goldlücke, and M. O. Franz, “Fast and efficient image novelty detection based on mean- shifts,” Sensors, vol. 22, no. 19, 2022. [Online]. Available: https://www.mdpi.com/1424-8220/22/19/7674
2022
-
[30]
Honey, i shrunk the sample covariance matrix,
O. Ledoit and M. Wolf, “Honey, i shrunk the sample covariance matrix,” [2004]
2004
-
[31]
The German Traffic Sign Recognition Benchmark: A multi-class classification com- petition,
J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel, “The German Traffic Sign Recognition Benchmark: A multi-class classification com- petition,” in IEEE International Joint Conference on Neural Networks , 2011, pp. 1453–1460
2011
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.