{"id":"676be56d-8d42-427e-bbae-a51ab1b0fa14","arxiv_id":"2608.10710","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A confidence-weighted ensemble of three LLMs improves Top-1 fault-classification accuracy to 0.917 in automotive HiL validation, exceeding the best single LLM.","lead":"This paper tests whether combining three open-weight large language models improves fault classification in automotive hardware-in-the-loop test recordings. The best three-model ensemble reaches 91.7% Top-1 accuracy, slightly beating the strongest single model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Top-3 ensemble is selected on the same test scenarios whose gain is then reported; the 0.917-vs-0.903 improvement may reflect selection bias, not a genuine multi-LLM advantage.","rationale":"The paper has real strengths: the pipeline is clearly specified, the leakage-safe temporal split between few-shot examples (15-25 s), embargo (25-30 s), and test (30-45 s) is a thoughtful protection against prompt-window overlap, and the explanation examples in Table 4 are concrete. My objection is not that ensemble LLMs cannot help, nor that the numbers are impossible; it is that the reported comparison is internally vulnerable to selection on the test set. The gain over the best single model is small and concentrated in a single scenario, which makes the risk concrete rather than hypothetical. The reader's final conditionality is appropriate, but the load-bearing weakness is not the healthy-reference assumption (an explicit scope condition) so much as the absence of any distinction between model-selection data and evaluation data. A leave-one-scenario-out reanalysis would settle whether the claimed advantage survives unbiased selection.","tokens_in":11692,"tokens_out":9632,"duration_ms":86915,"concrete_test":"Leave-one-scenario-out re-evaluation: for each of the three settings (Gas-Hwy, Gas-Urb, EV-Hwy), select the Top-3 LLMs using only the other two settings' windows, then run the confidence-weighted ensemble on the held-out setting. Average the three held-out folds and compare with the best single model on the same folds, reporting bootstrap 95% confidence intervals and per-window counts. If the held-out ensemble Top-1/F1/MCC does not exceed the best member's, the reported 0.917/0.913/0.902 gain is consistent with test-set selection and the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (abstract, Section 5.2) is that confidence-weighted Top-3 voting reaches 0.917 Top-1 accuracy, 0.913 macro F1, and 0.902 MCC, beating every single model. As reported, the evaluation cannot support that claim because model selection and final evaluation share the same scenario-averaged windows: Section 5.2 says the three models were chosen because they achieved the strongest single-model results, and Table 1 is computed on the same evaluation windows that produce Table 3. No separate validation set, embargo-based model selection, or pre-registered model set is described. With seven candidate models and a gain of only +0.014 Top-1 and +0.015 MCC over the best member, post hoc selection on the test set can easily manufacture the difference. The same post hoc selection applies to the choice of confidence-weighted voting as the best fusion strategy and to the Top-5 comparison. Table 2 further shows that two of the three scenarios are perfect for every fusion method, so the entire ensemble gain rests on the Gas-Hwy scenario (0.750), yet no window counts, confidence intervals, or repeated runs are reported to bound the variance. The healthy-reference requirement is a real scope condition, but it is explicitly assumed in Section 3.1 and does not threaten the internal validity of the reported comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an explainable multi-LLM ensemble framework for sensor-level fault classification in automotive hardware-in-the-loop (HiL) validation. The pipeline converts windowed multivariate signals into compact statistical evidence by comparing faulty recordings with time-aligned healthy references, builds structured few-shot prompts, and queries several open-weight LLMs. The individual predictions and confidences are fused with three strategies (confidence-weighted voting, Borda count, reciprocal rank fusion). The authors evaluate seven LLMs on three driving scenarios from two HiL systems (gasoline-engine highway, gasoline-engine urban, electric-vehicle highway) with ten single-fault classes. They report that a Top-3 ensemble of Mistral Small 24B, Qwen2.5 32B, and Phi-4 14B with confidence-weighted voting achieves scenario-averaged Top-1 accuracy 0.917, macro F1 0.913, and MCC 0.902, outperforming every single model, with the best calibration among the tested ensemble strategies. A Top-5 ensemble does not improve over the Top-3 configuration.","tokens_in":11906,"tokens_out":3497,"duration_ms":34071,"significance":"If the reported results are reliable, the paper would provide a useful demonstration that coordinated multi-LLM reasoning with compact, reference-based evidence can outperform single LLMs on structured diagnostic classification while generating engineer-readable explanations. The study is one of the first to apply LLM ensembles to sensor-level fault diagnosis in automotive HiL data, and it contributes a concrete prompt-engineering recipe and a leakage-aware train/embargo/test split. The runtime table and the explicit comparison of fusion strategies are also valuable. However, the significance is tempered by the evaluation design: the ensemble members and the final fusion strategy are selected using the same evaluation scenarios on which the improvement is reported, and the statistical evidence is thin (three scenarios, no confidence intervals, no repeated runs). The framework's dependence on a healthy reference run is a real scope limitation, though the paper states this assumption explicitly.","major_comments":[{"comment":"The Top-3 model selection and the choice of confidence-weighted voting are made after inspecting the same scenario-averaged results that are then reported as the final ensemble gain. Section 5.2 states that the three models were 'selected' because they achieved the strongest single-model results, and the fusion strategy is chosen using the same Table 2/3 numbers. No held-out model-selection set, embargo-based selection, or pre-registered model set is described. With seven candidate models and a reported improvement of only +0.014 Top-1 accuracy and +0.015 MCC over the best single model, post hoc selection on the test set can plausibly manufacture the observed difference. The authors should either perform model selection on a separate validation split (e.g., the 25–30 s embargo region) or re-frame the single-model comparison and ensemble gain as exploratory and report selection-corrected estimates.","section":"Section 5.2, Table 1 vs. Table 3"},{"comment":"The scenario-averaged ensemble gain rests almost entirely on the Gas-Hwy scenario, because Gas-Urb and EV-Hwy achieve perfect 1.000 scores for every fusion method. Yet the paper reports no window counts, no confidence intervals, and no repeated runs to bound the variance of the 0.917 Top-1 estimate. LLM outputs are stochastic, and without temperature/seed details or multiple inference runs, the +0.014 improvement over Mistral 24B may be within run-to-run noise. The authors should report per-window sample sizes, bootstrap confidence intervals, and per-scenario error bars, and should explicitly acknowledge that the effective evidence for the ensemble advantage comes from one difficult scenario.","section":"Section 5.2, Tables 2 and 3"},{"comment":"The entire framework depends on the availability of a time-aligned healthy reference run for each faulty recording under the same driving profile. The paper states this as an assumption and uses it to compute deviation evidence, but it provides no evidence about how often such references exist in production HiL campaigns or how the method behaves when a reference is missing or misaligned. Since this is a load-bearing scope condition, the authors should state it as an explicit limitation and, ideally, provide a sensitivity analysis with perturbed or missing references.","section":"Section 3.1 and Section 4.3"}],"minor_comments":[{"comment":"The sentence 'Although Borda rank aggregation and reciprocal rank fusion reach a highered voting achieves 0.917 Top-1 accuracy, 0.913 macro F1, and Top-2 accuracy of 0.972' contains a text corruption and is not readable; it should be rewritten.","section":"Section 5.2, paragraph after Table 3"},{"comment":"The table footnote says 'The dagger marks the final selected ensemble strategy', but no dagger appears in the table; either add the dagger to the CW vote row or change the footnote.","section":"Table 3, footnote"},{"comment":"Several rows lack spaces between method names and values (e.g., 'Borda1.000', 'RRF1.000'), making the table harder to read.","section":"Table 2, formatting"},{"comment":"In the 'True / predicted' row, the format 'predictedBAT_HV_Gain_3' and 'predictedAccPedal_Gain_5' is missing a separating space; this should be fixed.","section":"Table 4, formatting"},{"comment":"The paper emphasizes that confidence-weighted voting provides the best calibration among ensemble strategies, but the CW Brier score of 0.153 is notably worse than the single-model Mistral 24B Brier of 0.102 (Table 1). Since the introduction motivates ensembles as improving 'confidence reliability', the authors should explicitly compare ensemble calibration with the single-model baseline and temper any implication that the ensemble improves calibration overall.","section":"Section 5.2, calibration comparison"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is interesting and the engineering is careful in some respects (leakage-safe split, compact evidence, explanation output). The main barrier is the test-set-selection issue: the ensemble members and fusion strategy are chosen on the same scenarios used for the headline numbers, and the gain is small. This is fixable in revision by adding a validation-based selection procedure or by reducing the strength of the claims. The journal should also weigh whether the three-scenario evaluation with two perfect scenarios provides sufficient evidence for the general claim; asking for CIs and per-scenario counts is essential. The healthy-reference dependence is a real scope limitation but honestly stated; it should be made prominent in a limitations section."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the claimed Top-3 ensemble advantage over the best single model is real in their numbers, but the evaluation as reported cannot support it, because the three ensemble members were selected from the same scenarios used to report the final gain. The +0.014 Top-1 and +0.015 MCC improvement over Mistral Small is exactly the kind of gap that post hoc model selection on the test set can manufacture.\n\nWhat is actually new: as far as the cited literature goes, this is the first coordinated multi-LLM ensemble for sensor-level fault classification in HiL validation. The design has several good properties: instead of prompting raw time-series, they compute compact deviation statistics against a time-aligned healthy reference; the few-shot examples come from a separate time interval with an embargo region; and the structured JSON output with ranked classes and explanations is a sensible way to get not just labels but a justification an engineer can check. The explanation examples in Table 4 look like the model really is keying on the root-cause signal first, which is the right behavior for this use case.\n\nThe soft spots are real. The main one is the load-bearing evaluation flaw: Section 5.2 says the Top-3 set was chosen because those models had the strongest single-model results on the same scenario-averaged windows that produce Table 3. There is no held-out model selection or pre-registered model set. Second, the ensemble gain is small and unevenly distributed: in Table 2, two of the three scenarios are already perfect for every fusion method, so the entire ensemble value rests on the Gas-Hwy scenario with Top-1=0.750. Third, all numbers are point estimates—no confidence intervals, no per-window counts, no repeated runs, no non-LLM baseline. Fourth, no data, prompts, or code are released. The healthy-reference requirement is a genuine scope condition, but it is explicitly assumed from Section 3.1 and does not threaten the internal validity of the comparison the paper actually runs.\n\nThe citation pattern looks fine; the related work is relevant and recent, and the authors do not oversell prior work. There is a minor text corruption in Section 5.2 (\"reaches a highered voting achieves...\") that should be cleaned up.\n\nWho this is for: researchers working on LLM-based diagnostics and anyone building evaluation pipelines for time-series classification. It deserves peer review, but the revision needs to fix the selection problem, add uncertainty quantification, include a classical baseline, and release artifacts. I would send it to referees rather than desk-reject it.","headline":"Plausible and useful in design, but the ensemble gain is not yet supported: the Top-3 set is selected on the same test scenarios whose improvement is reported.","tokens_in":12455,"tokens_out":1915,"would_cite":false,"duration_ms":17233,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-LLM ensemble with confidence-weighted voting beats every single model at classifying injected faults in automotive hardware-in-the-loop data.","keywords":["large language models","ensemble learning","fault classification","hardware-in-the-loop validation","automotive software","explainable diagnosis","confidence-weighted voting","time-series classification"],"falsifier":"Run the same pipeline on fault recordings that lack a paired healthy reference, or on a held-out HiL dataset with unseen driving profiles and fault classes; if Top-1 accuracy falls far below the reported 0.917 (or to chance level without the healthy baseline), the central empirical claim would be contradicted. A simpler check is to replace the healthy reference with a randomly chosen run of a different profile; the paper's premise predicts a large accuracy drop.","tokens_in":11455,"feed_emoji":"🚗","tokens_out":5796,"duration_ms":59398,"temperature":0.7,"pith_summary":"The paper claims that a coordinated ensemble of three mid-sized open large language models can classify injected single faults in automotive hardware-in-the-loop (HiL) test recordings better than any individual model, reaching a scenario-averaged Top-1 accuracy of 0.917, macro F1 of 0.913, and Matthews correlation coefficient of 0.902, while also producing the most reliable confidence scores among tested fusion strategies. The authors argue this matters because current practice relies on manual review and rule-based anomaly detection that cannot name the fault type, while single-LLM diagnostics are prompt-sensitive, overconfident, and unstable. The proposed framework turns each faulty window into a compact statistical comparison against a time-aligned healthy reference, sends that evidence to three complementary LLMs, and fuses their structured outputs with confidence-weighted voting. A larger Top-5 ensemble does not improve results, which the paper reads as evidence that model complementarity matters more than ensemble size. If the claim holds, multi-LLM reasoning could give test engineers a reliable, calibrated, and explainable first pass over large volumes of HiL data.","feed_headline":"Three-LLM ensemble hits 91.7% fault classification in car HiL tests","feed_subtitle":"Confidence-weighted voting over three open LLMs beats each model alone and stays well calibrated.","key_machinery":"The load-bearing object is the compact evidence representation: instead of feeding raw time series, each 5-second faulty window is compared with the time-aligned healthy window and summarized per signal by statistical descriptors (mean difference, mean absolute deviation, maximum deviation, zero and saturation fractions, and a standardized deviation score). The prompt separates direct root-cause evidence from propagated support evidence, and the query is framed as a fault-only classification task with leakage-safe few-shot examples from a disjoint time region. The second mechanism is ensemble fusion: the same prompt is sent to three heterogeneous LLMs that each return a structured JSON with predicted class, confidence, ranked alternatives, and explanation, and the outputs are combined by confidence-weighted voting (with Borda count and reciprocal rank fusion as comparisons). This design is what lets the ensemble exploit model complementarity while keeping the input small enough for practical prompting and the output parseable for automated review.","core_discovery":"The central discovery is that, on the evaluated gasoline-engine and electric-vehicle HiL fault-injection datasets across three driving settings and ten single-fault classes, the Top-3 ensemble of Mistral Small 24B, Qwen2.5 32B, and Phi-4 14B with confidence-weighted voting outperforms every single model on the main diagnostic metrics: 0.917 Top-1 accuracy, 0.913 macro F1, 0.902 MCC, and the lowest expected calibration error among fusion strategies. Borda rank aggregation and reciprocal rank fusion match or slightly beat the Top-2 accuracy but have much worse Brier scores and ECE, so the paper selects confidence-weighted voting as the final strategy. Adding Qwen2.5 14B and DeepSeek R1 Distill Qwen 32B to make a Top-5 ensemble slightly degrades accuracy and calibration, indicating diminishing returns from scale alone. The authors also show that the models' explanations refer to the expected root-cause signal before propagated evidence, which is the behavior a test engineer would want.","pith_inferences":["A direct testable extension is to ablate the healthy reference: feed the same prompts without the time-aligned baseline and measure the accuracy drop; the paper's premise predicts a large collapse, which would confirm how dependent the result is on that design choice.","The compact-evidence prompting style could transfer to other XiL domains (e.g., brake-by-wire, ADAS sensor validation) where a healthy reference run is available, though the 0.917 figure is specific to these two platforms and ten fault classes and should not be extrapolated.","The complementarity-over-size result suggests ensemble selection could be done by measuring pairwise disagreement among candidate LLMs on a small validation set, rather than by adding models blindly; the paper does not propose such a selection criterion.","The explanation outputs might be used to build a feedback loop that catches misclassifications: if a top-ranked class's explanation cites only support signals and never the expected root signal, an engineer could flag that prompt for inspection; this is a use the paper gestures at but does not formalize."],"forward_implications":["HiL validation teams could replace a large fraction of manual fault-label review with an LLM ensemble that names the fault, ranks alternatives, and points to the deciding signals, cutting validation turnaround time.","The 0.917 accuracy and 0.902 MCC are scenario-averaged across highway gasoline, urban gasoline, and highway EV settings, so the method is not tied to one propulsion architecture.","Because the ensemble returns ranked classes and explanations, it can function as a triage tool that flags the most confident diagnoses for automated handling and the less confident ones for engineer review, consistent with ISO 26262 traceability expectations.","Model size alone is not the driver: the quantized Llama 3.1 70B underperforms smaller models, and the Top-5 ensemble underperforms the Top-3, so the practical guidance is to select a small complementary set of models rather than the largest available.","The confidence scores of the Top-3 ensemble are calibrated well enough that downstream systems could threshold on them: the Brier score of 0.153 and ECE of 0.058 are the best among fusion strategies."],"supporting_citations":[{"why":"Supplies the virtual HiL testing framework with fault injection used to generate the recordings.","marker":"[2]"},{"why":"Documents single-LLM weaknesses (poor temporal reasoning, run-to-run instability) that the ensemble is designed to overcome.","marker":"[9]"},{"why":"Shows domain-adapted compact LLMs can outperform larger ones in HIL testing, supporting the use of mid-sized open models.","marker":"[11]"},{"why":"Provides the ensemble-learning principle that complementary base learners improve robustness, which the paper transfers to LLMs.","marker":"[15]"},{"why":"Presents a prior HIL-based ensemble method on similar fault-injection data, a baseline the paper situates itself against.","marker":"[1]"}],"fun_headline_variants":["Three-LLM ensemble hits 91.7% in car HiL fault tests","LLM trio tops single models in auto fault diagnosis","Confidence-weighted LLMs beat singles in HiL accuracy","Small LLM ensemble beats larger ones in HiL validation","Ensemble of three LLMs improves car fault classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every faulty recording must have a time-aligned healthy recording of the same driving profile, because the framework only looks at deviations from that healthy run; without such a reference the pipeline cannot be applied at all.","fun_headline_variants_meta":{"raw":{"variants":["Three-LLM ensemble hits 91.7% in car HiL fault tests","LLM trio tops single models in auto fault diagnosis","Confidence-weighted LLMs beat singles in HiL accuracy","Small LLM ensemble beats larger ones in HiL validation","Ensemble of three LLMs improves car fault classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1673,"prompt_tokens":1064,"completion_tokens":609,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":524}},"tokens_in":680,"tokens_out":609,"duration_ms":59490,"temperature":1.0,"reasoning_tokens":524,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:50:53.009282+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on fault recordings that lack a paired healthy reference, or on a held-out HiL dataset with unseen driving profiles and fault classes; if Top-1 accuracy falls far below the reported 0.917 (or to chance level without the healthy baseline), the central empirical claim would be contradicted. A simpler check is to replace the healthy reference with a randomly chosen run of a different profile; the paper's premise predicts a large accuracy drop.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the virtual HiL testing framework with fault injection used to generate the recordings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents single-LLM weaknesses (poor temporal reasoning, run-to-run instability) that the ensemble is designed to overcome."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ensemble-learning principle that complementary base learners improve robustness, which the paper transfers to LLMs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents a prior HIL-based ensemble method on similar fault-injection data, a baseline the paper situates itself against."}],"review_version":1}