{"id":"f8c97861-f39e-46e9-bc3b-1fb1b2434287","arxiv_id":"2507.01077","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A decoder-only LLM pre-trained on ECU log text and fine-tuned with an entropy regularizer detects cycle-time anomalies with 0.81 region recall despite noisy labels.","lead":"This paper trains a small decoder-only language model from scratch on car-internal UDP communication logs and then fine-tunes it to flag delayed or missing messages. It aims to make anomaly detection work even when the training labels come from an unreliable rule-based system.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Entropy-regularizer generalization is untested: Table IV's ground truth is the same inconsistent rule-based labels, so memorizing known anomaly tokens could also yield 0.81 region recall; held-out anomaly and manual-perturbation tests are needed.","rationale":"I read the paper as a proof-of-concept that a decoder-only LLM can use a small set of inconsistently labeled anomalies to detect cycle-time anomalies. For that claim to hold, the entropy regularizer must confer uncertainty on unlabeled anomalies that resemble known ones; otherwise the model only memorizes the labels, and the reported metrics could be high simply because the test anomalies overlap with training anomalies. The reader's weakest assumption identifies exactly this transfer problem, and my reading of the evidence does not change the verdict: Table IV is evaluated against the same rule-based labels the paper calls unreliable, and the false positives are speculatively reinterpreted as true anomalies. The region-level metric is more lenient than line-level (0.81 versus 0.6), further weakening the headline. The Figure 6 manual perturbation is run on the base model, before entropy fine-tuning, so it does not test the regularizer's proposed transfer. A leave-one-out test over anomaly classes is the minimal experiment that would separate transfer from memorization; this is the concrete check I propose. There is also a formalization concern: Equation 2 calls a masked negative log-likelihood 'entropy' and Equation 3 uses it as a term to be maximized, so the exact optimized objective is ambiguous without code; this reinforces the need for a controlled comparison with alpha = 0 and with a held-out anomaly class. None of this is an attack on the authors; it is a request for evidence that the claimed generalization is real. The paper is a plausible proof-of-concept, but the central claim is not yet empirically supported, so a conditional verdict remains appropriate.","tokens_in":8080,"tokens_out":8727,"duration_ms":111253,"concrete_test":"Fine-tune Qwen2-0.4B with the Section III-D objective on a training set from which a complete class of labeled anomalies is withheld (e.g., all anomalies for one PDU name, or all restart-scenario regions), then evaluate region-level recall and precision on those held-out anomaly regions using the same rule-based ground truth. Compare against a control fine-tuned only with next-token prediction (alpha = 0) on the same data. If held-out recall is near the base-model rate or not substantially above the alpha = 0 control, the entropy regularizer is memorizing known anomaly tokens rather than transferring to similar tokens, and the Table IV result would not support the generalization claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Section III-D's assertion that maximizing entropy on tokens from known anomalies makes 'similar tokens behave in the same way,' so the model becomes uncertain on previously unlabeled anomalies and generalizes from a minimal set of examples. This transfer is the load-bearing premise, and the paper provides no experiment that isolates it. Table IV (Section IV-D) reports recall and precision computed against the same rule-based labels that Sections I and IV-D describe as inconsistent; the text even allows that false positives 'might be actually correct' (Section IV-D). Under that evaluation, a model that has simply memorized the labeled anomaly tokens by raising their surprisal during fine-tuning would also show high recall on the labeled test anomalies. The only anomaly-detection experiment with independently known perturbations (Figure 6, Section IV-E) is run on the base model before entropy fine-tuning, so it cannot validate the regularizer's proposed transfer. Additionally, region-level recall is lenient: a single hit anywhere in a long consecutive-anomaly region counts, and line-level recall is only 0.6 versus 0.81 for regions. The reported numbers therefore do not distinguish genuine generalization to unlabeled anomalies from memorization of unreliable labels, and the abstract's 'minimal set of examples' and 'improving detection accuracy' claims are unsupported by the current evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage decoder-only LLM pipeline for detecting cycle-time anomalies (delayed or missing messages) in UDP-based ECU communication logs. In the first stage, a randomly initialized Qwen2 model is pre-trained from scratch on next-token prediction over tokenized ECU log lines. In the second stage, the model is fine-tuned with LoRA using a combination of next-token prediction and an entropy regularizer that increases the model's uncertainty on tokens marked as anomalous by an existing rule-based labeling system. Anomaly detection is then performed by thresholding the perplexity of predicted tokens, optionally followed by a filter that removes isolated spikes. The paper reports line-level and region-level recall and precision for the entropy-regularized fine-tuning objective in Table IV, compares several model sizes and tokenizers on pre-training perplexity, and includes qualitative visualizations of detections.","tokens_in":8319,"tokens_out":4088,"duration_ms":53675,"significance":"If the central claim were established, the paper would make a useful contribution to automotive log analysis: a decoder-only LLM that learns communication patterns from raw UDP logs, a handling strategy for inconsistent rule-based labels, and a pre-trained ECU-specific language model that could be adapted to other tasks. The authors also provide useful engineering comparisons across model sizes, tokenizers, and prompt lengths, and they train models from scratch on a domain-specific tokenization, which is non-trivial. However, the current evidence does not isolate the proposed method's generalization behavior: the evaluation is conducted against the same rule-based label source that generated the training masks, and the entropy regularizer directly raises perplexity on known anomaly tokens, so high recall on labeled anomalies is partly by construction. The paper's stated contribution of learning from a minimal set of examples and improving detection accuracy in complex environments therefore remains unverified. With additional held-out evaluation, baselines, and ablations, the underlying idea is worth pursuing, but the present manuscript needs substantial revision.","major_comments":[{"comment":"Another major concern is that Equation (2) does not define the entropy of the model's predictive distribution. The expression sums the negative log-likelihood of the observed tokens weighted by the anomaly mask, i.e., -Σ log pθ(x_i | x_<i) · â_i, which is a weighted negative log-likelihood over anomalous positions, not the entropy -Σ_v p(v | x_<i) log p(v | x_<i). Consequently, the objective in Equation (3) is better described as a regularizer that directly increases the model's surprisal on known anomalous tokens, not as an entropy maximizer. This distinction matters because the detection metric used at inference is perplexity: the regularizer directly optimizes the metric on the labeled anomalies, which reinforces the need for a genuine transfer experiment rather than relying on the asserted intuition that 'similar tokens will behave in the same way' (Section III-D).","section":"Section IV-D, Table IV"},{"comment":"The claim that entropy regularization on known anomalies transfers to previously unlabeled anomalies is not demonstrated. The only anomaly-detection experiment with independently known perturbations, the manual log-line displacement in Section IV-E and Figure 6, is performed on the base pre-trained model before the entropy fine-tuning step, so it cannot validate the regularizer's proposed generalization effect. I recommend an ablation that compares the fine-tuned model with and without the entropy regularizer on the same held-out perturbed data, and an experiment where the anomaly masks used for fine-tuning cover only a subset of anomaly types while evaluation is performed on the omitted types. Without such evidence, the abstract's statement that the system 'is able to learn from a minimal set of examples' and generalizes to other anomalies is unsupported.","section":"Section III-D, Eq. (3)"},{"comment":"The paper does not state clearly which hyperparameter choices were made on a validation set versus the test set. The text reports that 'we found out experimentally' that top-k detections result in lower recall and that a filter width of 3 and a filter threshold of 1.5 were used; the prompt length, LoRA rank, learning rate, and α are also presented without a separate validation procedure. If these values were selected by inspecting the test traces, the reported recall and precision are optimistic. The same applies to the choice of perplexity as the main detection metric after observing Figure 6. I ask the authors to specify the model-selection protocol and to report results with error bars over multiple test traces or repeated runs.","section":"Section IV-C, IV-D"},{"comment":"The pre-training data is described as including 'known and unidentified anomalies' (Section IV-A, first paragraph). Since the pre-trained model is later fine-tuned for anomaly detection, contamination of the pre-training corpus with anomalies that may overlap with the fine-tuning labels could confound the results. The paper should quantify how many anomalies appear in pre-training, or re-run the main evaluation with a pre-training set filtered to contain only normal communication, in order to ensure that the reported detection performance is not partly due to the base model already having seen anomalous patterns.","section":"Section IV-A"}],"minor_comments":[{"comment":"The phrase 'Open Word Assumption (OWA)' should be 'Open World Assumption'; the typo appears in both the introduction and the abstract.","section":"Section I"},{"comment":"The symbol V is used for the vocabulary size in Equation (1) and then apparently for the sequence length in Equation (2); please use distinct notation, such as L or T for the sequence length, to avoid confusion.","section":"Section III-B, Eq. (1) and (2)"},{"comment":"Table IV reports only point estimates for recall and precision without the number of test examples, number of traces, or any measure of variability; adding these would make the comparison interpretable.","section":"Table IV"},{"comment":"The manual perturbation experiment is described as a validation of the detection pipeline, but it uses only the base model; the caption and text should explicitly state that this does not evaluate the fine-tuned model and therefore does not test the entropy-regularized objective.","section":"Section IV-E"},{"comment":"The contribution is described as 'a decoder-only anomaly detection architecture,' but the architecture is the existing Qwen2 model; the novelty is the application and training scheme, not the architecture itself. Please rephrase to avoid overclaiming.","section":"Abstract and Section I"}],"recommendation":"major_revision","confidential_remarks":"The central idea is potentially interesting, but the evaluation design does not yet support the generalization claim. I would be willing to reconsider after the authors provide a non-circular evaluation (independent held-out anomalies or post-fine-tuning perturbations), a baseline comparison, and a clearly separated hyperparameter-selection protocol. No code or data availability is mentioned, which also limits reproducibility of the quantitative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible proof-of-concept with a genuinely interesting idea—entropy regularization for training on inconsistently labeled anomaly data—but the evaluation as written does not support the generalization claim. The stress-test note is right. The fine-tuning objective raises entropy, hence perplexity, on tokens from known anomalies, and Table IV measures recall against the same rule-based labels that produced the training masks. A model that simply memorized those labeled anomalies by raising their surprisal would also look good by this metric. Region-level recall is also lenient: line-level recall is 0.6, not 0.81. The manual perturbation experiment in Figure 6 is run on the base model before fine-tuning, so it cannot validate the regularizer's transfer. And Section IV-D itself says false positives 'might be actually correct' because the labels are inconsistent, which makes the reported precision uninterpretable. There is also no baseline—no comparison to the rule-based detector, to a threshold on delta-time, or even to the pre-trained model without fine-tuning. So the abstract's claims about 'minimal set of examples' and 'improving detection accuracy' are not yet earned.\n\nWhat is genuinely new: the specific combination of a from-scratch decoder-only LLM on UDP ECU logs, a custom BPE tokenizer, and an entropy regularizer that pushes the model toward uncertainty on known anomalies. The pre-training comparisons across models, tokenizers, and prompt lengths are real measurements, not filler. The authors are also candid about their labels being inconsistent, even if they then use those same labels as ground truth.\n\nWhere it is soft: the central transfer mechanism is asserted by intuition ('similar tokens will behave in the same way') without a held-out experiment isolating it. Hyperparameters (perplexity threshold, filter width, alpha) were selected on the test set. Pre-training data reportedly contains anomalies, which weakens the normal-behavior baseline. No error bars, no code or data, and the novelty discussion ignores that decoder-only LLMs for log anomaly detection already exist (they cite LogBERT, LanoBERT, CAN-BERT, but those are encoder-based).\n\nAll that said, the idea is worth a serious look. This is a solid proof-of-concept for an industrial niche, not a settled method. With a baseline, a held-out anomaly set with independently verified labels, and sensitivity analysis, the claim could hold. As it stands, I would not cite it as evidence of generalization, but I would point a student to it as a useful start and a fair description of the problem.","headline":"A promising proof-of-concept on entropy regularization for inconsistent anomaly labels, but the evaluation does not yet support the generalization claim.","tokens_in":8879,"tokens_out":1480,"would_cite":false,"duration_ms":17420,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A decoder-only LLM detects ECU timing anomalies with 0.81 recall and precision despite noisy labels.","keywords":["anomaly detection","ECU logs","decoder-only LLM","entropy regularization","next-token prediction","inconsistent labels","perplexity threshold","automotive communication"],"falsifier":"Hold out every labeled anomaly for one specific message type (PDU) during fine-tuning and then run detection on delayed or missing messages of that type. If region-level recall on the held-out PDU is no better than the no-signal baseline, the entropy regularizer is memorizing labeled anomalies rather than transferring uncertainty to unseen ones.","tokens_in":7834,"feed_emoji":"🚗","tokens_out":8571,"duration_ms":90511,"temperature":0.7,"pith_summary":"The paper's aim is to make anomaly detection in automotive ECU communication logs work without a reliable ground-truth label set. The authors train a decoder-only large language model from scratch on UDP message traces, then fine-tune it by combining next-token prediction with an entropy regularizer that deliberately increases the model's uncertainty on tokens marked as anomalous. Because perplexity is the exponential of entropy, this raises the perplexity at those positions, and the model flags any token whose perplexity exceeds a threshold as a cycle-time anomaly, meaning a delayed or missing message. On real vehicle logs labeled by an inconsistent rule-based system, the entropy-regularized model reaches 0.81 region-level recall and 0.81 precision, which the paper presents as evidence that a model can learn from a minimal set of imperfect labels. If the claim holds, it offers a more scalable alternative to hand-written communication rules for regression testing of vehicle software.","feed_headline":"LLM catches missing car-ECU messages at 0.81 recall and precision","feed_subtitle":"A decoder-only model trained on UDP logs flags delayed or missing messages even when training labels are inconsistent.","key_machinery":"The central mechanism is the entropy regularizer, a term added to the standard cross-entropy next-token prediction loss, weighted by a scalar alpha, that maximizes the model's predictive entropy on tokens labeled anomalous while leaving ordinary tokens under standard loss. Its purpose is to make the model uncertain specifically at known anomaly positions, with the paper's stated intuition that similar tokens will then behave similarly and the uncertainty will transfer to unlabeled anomalies. The detection side uses two derived metrics, top-k and perplexity; perplexity is chosen as the main metric because it degrades more clearly on perturbed sequences. The supporting architecture is a decoder-only Transformer pre-trained from scratch on byte-pair-encoding tokenized UDP message fields, with a sliding window that splits each window into prompt and prediction parts.","core_discovery":"The central claim is that a decoder-only LLM can learn to detect anomalies in UDP ECU logs when the training labels are inconsistent, by being made uncertain where it is told an anomaly occurred. The paper treats the unlabeled negative class under the open world assumption: absence of a label does not mean the event is normal. Fine-tuning therefore uses the next-token prediction loss plus a weighted entropy regularizer that acts only on tokens with an anomaly mask of 1, which the authors show as Equation 3. At inference the model's perplexity per token is compared to a threshold, with a filter that keeps only consecutive outliers, and this flags delayed or missing messages. The authors report region-level recall 0.81 and precision 0.81 for the entropy regularizer, versus 0.35 recall and 0.63 precision for a contrastive regularizer they tested, and they note that these numbers may understate real performance because the ground-truth labels themselves are inconsistent.","pith_inferences":["A direct way to test whether the regularizer generalizes would be to hold out all labeled anomalies for one PDU during fine-tuning; the paper does not report such a per-PDU held-out experiment.","The same uncertainty-raising objective could be applied to other protocol logs, such as CAN or industrial control traffic, where labels come from unreliable heuristics.","A practical improvement might be to set the perplexity threshold per PDU or per activity state, since different message types have different cycle-time variation.","Because the labels are inconsistent, part of what the evaluation counts as false positives may in fact be unlabeled true anomalies; the paper itself makes this point."],"forward_implications":["A detector for timing anomalies in ECU logs can be produced from a small number of imperfect labels, reducing the dependence on hand-written rules.","The pre-trained model on UDP communication can be reused and fine-tuned for other ECU communication use cases, since it captures the general protocol before anomaly specialization.","The entropy regularizer changes only the final probability distribution, so it can be combined with other fine-tuning objectives and applied to encoder-only or encoder-decoder LLMs as well.","The detection pipeline is offline and threshold-based, so it can be deployed on fresh traces without running the rule-based annotator."],"supporting_citations":[{"why":"Surveys LLM anomaly-detection approaches and motivates the choice of a decoder-only model over encoder-only ones.","marker":"[1]"},{"why":"Shows the closest prior application of an LLM to automotive traffic, specifically CAN log intrusion detection, which this work extends to UDP ECU logs.","marker":"[5]"},{"why":"Supplies the byte-pair encoding algorithm that the custom tokenizer is based on.","marker":"[8]"},{"why":"Gives the open-source decoder-only Transformer architecture used as the base for the main experiments.","marker":"[10]"},{"why":"Supplies the LoRA low-rank adaptation method used in the anomaly-detection fine-tuning phase.","marker":"[12]"},{"why":"Provides the alternative state-space model compared against in pre-training, supporting the architecture choice.","marker":"[13]"},{"why":"Provides the Chinchilla scaling rule used to set the number of pre-training tokens for each model size.","marker":"[14]"}],"fun_headline_variants":["Entropy trick lets LLM spot ECU anomalies without clean labels","Car-ECU anomaly detection: LLM tolerates messy labels, hits 0.81","No reliable labels? This LLM still detects ECU log anomalies","Decoder-only LLM handles inconsistent ECU labels for anomaly detection","LLM finds car-ECU anomalies at 0.81 recall and precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that increasing the model's uncertainty on tokens that the noisy labeler marked as anomalous will make the model uncertain about similar but unlabeled anomalies; the paper asserts this transfer by intuition rather than demonstrating it.","fun_headline_variants_meta":{"raw":{"variants":["Entropy trick lets LLM spot ECU anomalies without clean labels","Car-ECU anomaly detection: LLM tolerates messy labels, hits 0.81","No reliable labels? This LLM still detects ECU log anomalies","Decoder-only LLM handles inconsistent ECU labels for anomaly detection","LLM finds car-ECU anomalies at 0.81 recall and precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000994,"raw_usage":{"total_tokens":4201,"prompt_tokens":928,"completion_tokens":3273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":3177}},"tokens_in":544,"tokens_out":3273,"duration_ms":25422,"temperature":1.0,"reasoning_tokens":3177,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:06:54.017382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out every labeled anomaly for one specific message type (PDU) during fine-tuning and then run detection on delayed or missing messages of that type. If region-level recall on the held-out PDU is no better than the no-signal baseline, the entropy regularizer is memorizing labeled anomalies rather than transferring uncertainty to unseen ones.","supporting_citations":[],"review_version":1}