REVIEW 4 major objections 5 minor 15 references
Good Enough to Learn: LLM-based Anomaly Detection in ECU Logs without Reliable Labels
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A decoder-only LLM detects ECU timing anomalies with 0.81 recall and precision despite noisy labels.
desk verdict A promising proof-of-concept on entropy regularization for inconsistent anomaly labels, but the evaluation does not yet support the generalization claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the entropy regularizer, a term added to the standard cross-entropy next-token prediction loss, weighted by a scalar alpha, that maximizes the model's predictive entropy on tokens labeled anomalous while leaving ordinary tokens under standard loss. Its purpose is to make the model uncertain specifically at known anomaly positions, with the paper's stated intuition that similar tokens will then behave similarly and the uncertainty will transfer to unlabeled anomalies. The detection side uses two derived metrics, top-k and perplexity; perplexity is chosen as the main metric because it degrades more clearly on perturbed sequences. The supporting architecture is a decoder-only Transformer pre-trained from scratch on byte-pair-encoding tokenized UDP message fields, with a sliding window that splits each window into prompt and prediction parts.
What would settle it
Hold out every labeled anomaly for one specific message type (PDU) during fine-tuning and then run detection on delayed or missing messages of that type. If region-level recall on the held-out PDU is no better than the no-signal baseline, the entropy regularizer is memorizing labeled anomalies rather than transferring uncertainty to unseen ones.
Extended reading notes
Core claim
The central claim is that a decoder-only LLM can learn to detect anomalies in UDP ECU logs when the training labels are inconsistent, by being made uncertain where it is told an anomaly occurred. The paper treats the unlabeled negative class under the open world assumption: absence of a label does not mean the event is normal. Fine-tuning therefore uses the next-token prediction loss plus a weighted entropy regularizer that acts only on tokens with an anomaly mask of 1, which the authors show as Equation 3. At inference the model's perplexity per token is compared to a threshold, with a filter that keeps only consecutive outliers, and this flags delayed or missing messages. The authors report region-level recall 0.81 and precision 0.81 for the entropy regularizer, versus 0.35 recall and 0.63 precision for a contrastive regularizer they tested, and they note that these numbers may understate real performance because the ground-truth labels themselves are inconsistent.
Load-bearing premise
The load-bearing premise is that increasing the model's uncertainty on tokens that the noisy labeler marked as anomalous will make the model uncertain about similar but unlabeled anomalies; the paper asserts this transfer by intuition rather than demonstrating it.
Editorial extensions
If this is right
- A detector for timing anomalies in ECU logs can be produced from a small number of imperfect labels, reducing the dependence on hand-written rules.
- The pre-trained model on UDP communication can be reused and fine-tuned for other ECU communication use cases, since it captures the general protocol before anomaly specialization.
- The entropy regularizer changes only the final probability distribution, so it can be combined with other fine-tuning objectives and applied to encoder-only or encoder-decoder LLMs as well.
- The detection pipeline is offline and threshold-based, so it can be deployed on fresh traces without running the rule-based annotator.
Reading between the lines
- A direct way to test whether the regularizer generalizes would be to hold out all labeled anomalies for one PDU during fine-tuning; the paper does not report such a per-PDU held-out experiment.
- The same uncertainty-raising objective could be applied to other protocol logs, such as CAN or industrial control traffic, where labels come from unreliable heuristics.
- A practical improvement might be to set the perplexity threshold per PDU or per activity state, since different message types have different cycle-time variation.
- Because the labels are inconsistent, part of what the evaluation counts as false positives may in fact be unlabeled true anomalies; the paper itself makes this point.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-stage decoder-only LLM pipeline for detecting cycle-time anomalies (delayed or missing messages) in UDP-based ECU communication logs. In the first stage, a randomly initialized Qwen2 model is pre-trained from scratch on next-token prediction over tokenized ECU log lines. In the second stage, the model is fine-tuned with LoRA using a combination of next-token prediction and an entropy regularizer that increases the model's uncertainty on tokens marked as anomalous by an existing rule-based labeling system. Anomaly detection is then performed by thresholding the perplexity of predicted tokens, optionally followed by a filter that removes isolated spikes. The paper reports line-level and region-level recall and precision for the entropy-regularized fine-tuning objective in Table IV, compares several model sizes and tokenizers on pre-training perplexity, and includes qualitative visualizations of detections.
Significance. If the central claim were established, the paper would make a useful contribution to automotive log analysis: a decoder-only LLM that learns communication patterns from raw UDP logs, a handling strategy for inconsistent rule-based labels, and a pre-trained ECU-specific language model that could be adapted to other tasks. The authors also provide useful engineering comparisons across model sizes, tokenizers, and prompt lengths, and they train models from scratch on a domain-specific tokenization, which is non-trivial. However, the current evidence does not isolate the proposed method's generalization behavior: the evaluation is conducted against the same rule-based label source that generated the training masks, and the entropy regularizer directly raises perplexity on known anomaly tokens, so high recall on labeled anomalies is partly by construction. The paper's stated contribution of learning from a minimal set of examples and improving detection accuracy in complex environments therefore remains unverified. With additional held-out evaluation, baselines, and ablations, the underlying idea is worth pursuing, but the present manuscript needs substantial revision.
major comments (4)
- [Section IV-D, Table IV] Another major concern is that Equation (2) does not define the entropy of the model's predictive distribution. The expression sums the negative log-likelihood of the observed tokens weighted by the anomaly mask, i.e., -Σ log pθ(x_i | x_<i) · â_i, which is a weighted negative log-likelihood over anomalous positions, not the entropy -Σ_v p(v | x_<i) log p(v | x_<i). Consequently, the objective in Equation (3) is better described as a regularizer that directly increases the model's surprisal on known anomalous tokens, not as an entropy maximizer. This distinction matters because the detection metric used at inference is perplexity: the regularizer directly optimizes the metric on the labeled anomalies, which reinforces the need for a genuine transfer experiment rather than relying on the asserted intuition that 'similar tokens will behave in the same way' (Section III-D).
- [Section III-D, Eq. (3)] The claim that entropy regularization on known anomalies transfers to previously unlabeled anomalies is not demonstrated. The only anomaly-detection experiment with independently known perturbations, the manual log-line displacement in Section IV-E and Figure 6, is performed on the base pre-trained model before the entropy fine-tuning step, so it cannot validate the regularizer's proposed generalization effect. I recommend an ablation that compares the fine-tuned model with and without the entropy regularizer on the same held-out perturbed data, and an experiment where the anomaly masks used for fine-tuning cover only a subset of anomaly types while evaluation is performed on the omitted types. Without such evidence, the abstract's statement that the system 'is able to learn from a minimal set of examples' and generalizes to other anomalies is unsupported.
- [Section IV-C, IV-D] The paper does not state clearly which hyperparameter choices were made on a validation set versus the test set. The text reports that 'we found out experimentally' that top-k detections result in lower recall and that a filter width of 3 and a filter threshold of 1.5 were used; the prompt length, LoRA rank, learning rate, and α are also presented without a separate validation procedure. If these values were selected by inspecting the test traces, the reported recall and precision are optimistic. The same applies to the choice of perplexity as the main detection metric after observing Figure 6. I ask the authors to specify the model-selection protocol and to report results with error bars over multiple test traces or repeated runs.
- [Section IV-A] The pre-training data is described as including 'known and unidentified anomalies' (Section IV-A, first paragraph). Since the pre-trained model is later fine-tuned for anomaly detection, contamination of the pre-training corpus with anomalies that may overlap with the fine-tuning labels could confound the results. The paper should quantify how many anomalies appear in pre-training, or re-run the main evaluation with a pre-training set filtered to contain only normal communication, in order to ensure that the reported detection performance is not partly due to the base model already having seen anomalous patterns.
minor comments (5)
- [Section I] The phrase 'Open Word Assumption (OWA)' should be 'Open World Assumption'; the typo appears in both the introduction and the abstract.
- [Section III-B, Eq. (1) and (2)] The symbol V is used for the vocabulary size in Equation (1) and then apparently for the sequence length in Equation (2); please use distinct notation, such as L or T for the sequence length, to avoid confusion.
- [Table IV] Table IV reports only point estimates for recall and precision without the number of test examples, number of traces, or any measure of variability; adding these would make the comparison interpretable.
- [Section IV-E] The manual perturbation experiment is described as a validation of the detection pipeline, but it uses only the base model; the caption and text should explicitly state that this does not evaluate the fine-tuned model and therefore does not test the entropy-regularized objective.
- [Abstract and Section I] The contribution is described as 'a decoder-only anomaly detection architecture,' but the architecture is the existing Qwen2 model; the novelty is the application and training scheme, not the architecture itself. Please rephrase to avoid overclaiming.
Circularity Check
Headline recall/precision in Table IV reduce by construction: the fine-tuning objective maximizes the exact metric (perplexity) used for detection on the same labeled anomalies that serve as ground truth.
-
fitted input called prediction
[Section III-D (Eqs. 2-3); Section IV-D, Table IV caption]
"Equation P P L(S, ·|θ) = eH(S,·|θ) tells us that maximizing the entropy has also the effect of maximizing the perplexity as perplexity is the exponential of entropy, a frequently used metric for evaluating large language models and one of the metrics used in this work to detect anomalies. ... For the final solution we used the entropy regularizer which resulted in the highest recall, that measures how many of the labeled anomalies did we correctly identify. ... Note that the final results captured in Table IV might be higher in reality as they reflect training on inconsistent anomaly labels."
At inference, anomaly detection is a perplexity threshold on tokens. The fine-tuning loss L = CE - alpha*H(S,A|theta) directly maximizes H on tokens marked anomalous by mask A, and the paper states that maximizing H also maximizes perplexity. Table IV then reports recall/precision 'with respect to GT anomalies as marked by the traditional approach,' and the text concedes the results 'reflect training on inconsistent anomaly labels.' Thus the reported recall on labeled anomalies measures how well the model complied with the training objective on the same label source, not whether it detects previously unlabeled anomalies.
full rationale
The paper is not built on a self-citation chain; its references are standard external works and no uniqueness theorem is imported from the authors. The circularity is narrow but real: the entropy-regularizer objective is written directly on the anomaly mask whose labels later serve as ground truth for the headline recall/precision numbers, and the detection metric (perplexity) is the exponential of the quantity the objective maximizes. This makes Table IV's recall on labeled anomalies a training-objective echo rather than a test of generalization to 'similar' unlabeled anomalies. The paper's own admission that false positives 'might be actually correct' and that results 'reflect training on inconsistent anomaly labels' reinforces that the reported numbers are fitted, not predictive. The manual perturbation experiment provides some independent signal, but it explicitly uses the phase-1 base model, so it cannot validate the entropy regularizer's proposed transfer. Overall, the central detection numbers reduce by construction to the fine-tuning objective on the same labels, giving partial circularity; the pre-training and architecture choices themselves remain independent contributions.
Assumptions & free parameters
free parameters (6)
- alpha (entropy regularization weight) =
0.5
- Perplexity filter threshold =
1.5
- Filter width =
3 consecutive lines
- Prompt length W =
4096 tokens
- LoRA rank and learning rate =
rank 64, lr 5e-5
- BPE vocabulary size =
408 tokens
assumptions (5)
- domain assumption Open World Assumption for unlabeled logs
- domain assumption ECU log 'language' is learnable via next-token prediction
- ad hoc to paper Entropy maximization on known anomalies transfers to similar tokens
- domain assumption Rule-based GT labels are reliable enough for evaluation
- domain assumption Pre-training on data with anomalies still yields a normal-behavior model
Cite this review
Pith. "Pith review of Good Enough to Learn: LLM-based Anomaly Detection in ECU Logs without Reliable Labels." pith.science (2026). https://pith.science/paper/HVHZ6IQK
@misc{pith2026250701077,
author = {Pith},
title = {Pith review of: Good Enough to Learn: LLM-based Anomaly Detection in ECU Logs without Reliable Labels},
year = {2026},
howpublished = {\url{https://pith.science/paper/HVHZ6IQK}},
note = {Machine review of arXiv:2507.01077}
}
read the original abstract
Anomaly detection often relies on supervised or clustering approaches, with limited success in specialized domains like automotive communication systems where scalable solutions are essential. We propose a novel decoder-only Large Language Model (LLM) to detect anomalies in Electronic Control Unit (ECU) communication logs. Our approach addresses two key challenges: the lack of LLMs tailored for ECU communication and the complexity of inconsistent ground truth data. By learning from UDP communication logs, we formulate anomaly detection simply as identifying deviations in time from normal behavior. We introduce an entropy regularization technique that increases model's uncertainty in known anomalies while maintaining consistency in similar scenarios. Our solution offers three novelties: a decoder-only anomaly detection architecture, a way to handle inconsistent labeling, and an adaptable LLM for different ECU communication use cases. By leveraging the generative capabilities of decoder-only models, we present a new technique that addresses the high cost and error-prone nature of manual labeling through a more scalable system that is able to learn from a minimal set of examples, while improving detection accuracy in complex communication environments.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Large language models for forecasting and anomaly detection: A systematic literature review,
J. Su, C. Jiang, X. Jin, Y . Qiao, T. Xiao, H. Ma, R. Wei, Z. Jing, J. Xu, and J. Lin, “Large language models for forecasting and anomaly detection: A systematic literature review,” 2024. [Online]. Available: https://arxiv.org/abs/2402.10350
arXiv 2024
-
[2]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1810.04805
arXiv 2019
-
[3]
Logbert: Log anomaly detection via bert,
H. Guo, S. Yuan, and X. Wu, “Logbert: Log anomaly detection via bert,” 2021. [Online]. Available: https://arxiv.org/abs/2103.04475
arXiv 2021
-
[4]
Lanobert: System log anomaly detection based on bert masked language model,
Y . Lee, J. Kim, and P. Kang, “Lanobert: System log anomaly detection based on bert masked language model,” 2023. [Online]. Available: https://arxiv.org/abs/2111.09564
arXiv 2023
-
[5]
Can- bert do it? controller area network intrusion detection system based on bert language model,
N. Alkhatib, M. Mushtaq, H. Ghauch, and J.-L. Danger, “Can- bert do it? controller area network intrusion detection system based on bert language model,” 2022. [Online]. Available: https://arxiv.org/abs/2210.09439
arXiv 2022
-
[6]
Weakly Supervised Anomaly Detection via Knowledge-Data Alignment
H. Zhao, C. Zi, Y . Liu, C. Zhang, Y . Zhou, and J. Li, “Weakly supervised anomaly detection via knowledge-data alignment,” 2024. [Online]. Available: https://arxiv.org/abs/2402.03785
work page Pith review arXiv 2024
-
[7]
Few-shot Anomaly Detection in Text with Deviation Learning
A. S. Das, A. Ajay, S. Saha, and M. Bhuyan, “Few-shot anomaly detection in text with deviation learning,” 2023. [Online]. Available: https://arxiv.org/abs/2308.11780
work page Pith review arXiv 2023
-
[8]
Neural machine translation of rare words with subword units,
R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” 2016. [Online]. Available: https://arxiv.org/abs/1508.07909
arXiv 2016
Show all 15 references
-
[9]
Language models are unsupervised multitask learners,
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019. [On- line]. Available: https://api.semanticscholar.org/CorpusID:160025533
2019
-
[10]
Qwen2 technical report,
A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. ...
2024 arXiv
-
[12]
Lora: Low-rank adaptation of large language models,
E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,”
-
[13]
Mamba: Linear-time sequence modeling with selective state spaces,
A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” 2024. [Online]. Available: https://arxiv.org/abs/2312.00752
2024 arXiv
-
[14]
Training compute-optimal large language models,
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifr...
2022 arXiv
-
[2020]
Available: https://arxiv.org/abs/2004.10964
[Online]. Available: https://arxiv.org/abs/2004.10964
2004 arXiv
-
[2021]
Available: https://arxiv.org/abs/2106.09685
[Online]. Available: https://arxiv.org/abs/2106.09685
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.