{"id":"557028cd-798a-4930-8a0e-0baa5c501ed5","arxiv_id":"2507.00742","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A benchmark of 29 LLMs and four prompting strategies for classifying faulty computer components from user reports, finding that small models like gemma-2-2b match larger ones.","lead":"This paper tested 29 AI language models to see which one is best at guessing which computer part is broken from a user's written complaint. It found that some small models perform nearly as well as huge ones, which could let diagnostics run directly on laptops or phones.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out split and undisclosed provenance of few-shot examples leave the small-model F1 gains potentially inflated; the central deployment recommendation is not yet established.","rationale":"The reader identified the possibility that the few-shot examples were drawn from the evaluation set. I agree that this is a real risk and that it is unaddressed in the paper. However, I would frame the concern more broadly: the paper never describes any held-out split, so even the zero-shot and CoT results are measured on the same 853 reports used to design and illustrate the prompts, with no evidence of out-of-sample generalization. The few-shot provenance question is one concrete manifestation of this wider issue. The paper has genuine strengths: a large number of inferences, multiple model families and sizes, bootstrap error bars, and a Pareto-frontier analysis. These support a comparative benchmark under a fixed protocol, which is why I do not recommend rejection. But the central claim about deploying small models on end-user devices requires that the reported F1 scores transfer to new reports. The proposed check—verifying whether the Figure 2 examples overlap the evaluation set and rerunning with a disjoint training fold—would settle whether the concern lands. Because the issue is concrete and addressable, and the reader already conditioned the verdict on similar grounds, I recommend keeping the verdict unchanged.","tokens_in":10663,"tokens_out":8373,"duration_ms":96759,"concrete_test":"Obtain the FACTO dataset and the exact prompt templates used. Check whether the three user queries in Figure 2 (or near-duplicates) appear among the 853 evaluation reports. If they do, rerun Table 2 with (a) those reports removed from the evaluation set and (b) few-shot examples drawn from a held-out training fold disjoint from the test reports. The central claim survives only if the F1 of gemma-2-2b and llama-3.2-1b, and the few-shot gains of qwen2.5-1.5b and yi-6b, change by less than about 0.02; otherwise the small-model recommendation needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recommendation—that llama-3.2-1b, gemma-2-2b, and mistral-small-24b offer the best size/performance trade-off—rests on the F1 scores in Table 2 and the Pareto frontier in Figure 5. Those scores are load-bearing only if they measure generalization to unseen user reports. The paper never describes a train/validation/test split for the 853 FACTO reports (Sections 4.1 and 4.2), and the three few-shot examples shown in Figure 2 (Section 3.2) are not stated to be disjoint from the evaluation set. If those examples were drawn from the same 853 reports, the FS and CoT+FS prompts expose the model to ground-truth labels for test items, contaminating those results. The risk is not limited to the three displayed examples: without a held-out split, prompt design itself can be implicitly tuned to the evaluation distribution. This matters most for the small-model claim because the largest strategy-dependent gains occur for small models (e.g., qwen2.5-1.5b: 0.000 in ZS vs 0.673 in FS; yi-6b: 0.272 in ZS vs 0.702 in FS). If those gains are partly an artifact of seeing in-context labels from the test distribution, the abstract's claim that 1B-2B models are competitive with larger models is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks 29 LLMs (27 open-source models from 1B to 72B parameters, plus GPT-4o and GPT-4o-mini) on an eight-class hardware fault diagnosis task using the FACTO dataset of 853 user reports. Four prompting strategies are compared: Zero-Shot, Few-Shot, Chain-of-Thought, and CoT combined with Few-Shot. Performance is measured with F1-score, and a Pareto frontier analysis of model size versus F1 is used to recommend llama-3.2-1b-instruct, gemma-2-2b-it, and mistral-small-24b-instruct as the best size/performance trade-offs. The paper claims an F1-score of up to 0.76.","tokens_in":10908,"tokens_out":4700,"duration_ms":47899,"significance":"This is a large-scale empirical benchmark: 98,948 inferences over 29 models, with bootstrap error bars and a clear Pareto analysis. If the evaluation is methodologically sound, the finding that 1B-2B models can reach F1 around 0.72 close to a 24B model is practically relevant for on-device hardware fault triage. The paper explicitly targets deployment constraints such as VRAM and NPU compatibility, which makes the study directly useful to practitioners. However, the lack of a described held-out split and the undisclosed provenance of the few-shot examples place the central deployment recommendation at risk.","major_comments":[{"comment":"The evaluation uses all 853 FACTO reports as inference examples, and the three few-shot examples (Memory, Audio, Storage) shown in Figure 2 are not stated to be disjoint from this set. Since no train/validation/test split is described, it is impossible to rule out that the few-shot demonstrations came from the same 853 reports that are scored. This contaminates the FS and CoT+FS results for every model. The impact is largest for small models: Table 2 shows qwen2.5-1.5b jumping from 0.000 (ZS) to 0.673 (FS) and yi-6b from 0.272 (ZS) to 0.702 (FS). Please state the provenance of the few-shot examples and, ideally, re-run the evaluation on a held-out split or with demonstrations drawn from an independent source.","section":"Section 4.1 and Section 3.2, Figure 2"},{"comment":"The paper does not report decoding parameters (temperature, top-p, max tokens, seed) for llama.cpp or for the proprietary GPT API calls. F1 differences between models and strategies could be an artifact of decoding choices rather than model capability. Please provide these settings for all models and both proprietary APIs, including model version identifiers, to make the benchmark reproducible and the comparisons fair.","section":"Section 4.2, Evaluation Setup"},{"comment":"The paper defines F1 for binary classification but applies it to an eight-class problem without specifying whether the reported F1 is macro-averaged, micro-averaged, or weighted. Because the dataset is described as class-imbalanced, this choice materially affects the reported numbers and the ranking of models. Please state the averaging method and, ideally, report per-class F1 or a confusion matrix in an appendix.","section":"Section 3.3 and Table 2"},{"comment":"The Pareto frontier in Figure 5 does not state which F1 value is plotted for each model: the best F1 across the four prompting strategies, or the F1 from a single fixed strategy. Since the text identifies llama-3.2-1b and gemma-2-2b as frontier models, the figure must be explicit about this choice. If the best-over-strategies value is used, report the strategy that achieves each frontier point; otherwise, the central recommendation cannot be verified from the reported results.","section":"Figure 5 and Section 4.3"}],"minor_comments":[{"comment":"The conclusion states that 'the observed trends suggest generalizability to other failure types with similar diagnostic patterns,' but no cross-domain or out-of-distribution experiment is presented. This overclaims beyond the evidence; please soften or remove the generalization statement.","section":"Section 5"},{"comment":"The x-axis in Figure 4 lists values 1, 8, 15, 30, 70 without indicating that the scale is custom or logarithmic. Please clarify the axis scale to avoid misleading visual spacing.","section":"Figure 4"},{"comment":"The model name 'stablelm-zephyr-1 6b' appears malformed; please correct it to 'stablelm-zephyr-1.6b' or as appropriate.","section":"Table 1"},{"comment":"The Nadaraya-Watson non-parametric regressions are said to be estimated, but the bandwidth selection rule is not stated. Please report the bandwidth or the method used to choose it.","section":"Section 4.3"},{"comment":"Please state whether the FACTO dataset is publicly available and, if so, provide a URL or access instructions, as the paper currently only cites a workshop paper.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the few-shot contamination issue: without a held-out split or provenance for the few-shot examples, the abstract's claim about 1B-2B models being competitive with larger models is not yet established. The authors should be asked to provide the experimental artifacts (prompts, data split, code) during revision. The paper's topic is well within the scope of cs.LG and the empirical scale is a strength, but the methodological gaps must be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a sizable, well-structured empirical benchmark of 29 LLMs on a practical hardware fault classification task, with bootstrap error bars and a clear Pareto analysis. The central recommendation—that small models like gemma-2-2b and llama-3.2-1b are competitive enough for on-device triage—is plausible but not fully nailed down, because the evaluation never describes a held-out split and the few-shot example provenance is undisclosed.\n\nWhat's genuinely new: the scale (nearly 100k inferences across 29 models and four prompting strategies), the public FACTO dataset (even if it comes from the same group), and the explicit size/performance trade-off analysis. The paper makes a useful practical point: for small models, few-shot prompting is often the difference between near-zero and usable F1 (e.g., qwen2.5-1.5b jumps from 0.000 to 0.673). That is a real finding, and the bootstrap error bars give the reader a rough sense of uncertainty.\n\nThe soft spots are real but mostly addressable. The biggest is the absence of any description of how the 853 reports were used: is this the entire dataset, and are the three few-shot examples in Figure 2 included in the evaluation set? If they are, the contamination is tiny in raw terms (3/853) but it signals that the prompts may have been designed with knowledge of the test distribution. The paper should state explicitly that few-shot examples were chosen from a separate set, or at least quantify the overlap. Second, the evaluation uses only a single dataset, so the generalizability claims in Section 5 are speculative. Third, there is no comparison to the prior smaller-model work by Silva et al. (2025) beyond a mention, which would have strengthened the practical framing. Finally, no code or prompts are released, which matters for reproducibility.\n\nI don't think the stress-test concern sinks the paper. The headline small-model claim rests on gemma-2-2b's 0.727 F1 in the zero-shot and few-shot conditions, and that number is consistent across strategies. The few-shot-specific gains for the smallest models could be inflated if the examples come from the evaluation set, but the magnitude of the overall finding (small models can do this task) would probably survive. Still, the authors should be asked to clarify the split and the example provenance before the paper is accepted.\n\nWho is this for: anyone in applied ML for hardware support, or benchmarking small LLMs for on-device classification. It deserves a serious referee, but the referee should push for the missing methodological details. I'd like to see the revised version after those clarifications.","headline":"A large, practically useful LLM benchmark for hardware fault triage whose central small-model recommendation is plausible but needs explicit train/test separation and few-shot provenance disclosure.","tokens_in":11465,"tokens_out":2795,"would_cite":true,"duration_ms":30645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Open-source LLMs can identify faulty PC hardware from user text at F1 up to 0.76, with a 2B-parameter model nearly matching a 24B-parameter model.","keywords":["hardware fault diagnosis","large language models","prompt engineering","few-shot prompting","chain-of-thought","text classification","FACTO dataset","on-device inference"],"falsifier":"Re-run the Few-Shot and CoT+FS evaluations with the three example reports (Memory, Audio, Storage) removed from the FACTO test set, or replaced by held-out reports; if qwen2.5-1.5b no longer jumps from 0.000 to 0.673 and stablelm-zephyr-3b no longer jumps from 0.474 to 0.708, the few-shot advantage for small models is an artifact of example overlap.","tokens_in":10488,"feed_emoji":"🖥️","tokens_out":8348,"duration_ms":82314,"temperature":0.7,"pith_summary":"The paper tries to establish that open-source LLMs, including some as small as 1–2 billion parameters, can classify free-text user complaints into the faulty hardware component (network, video card, memory, storage, audio, battery, motherboard, CPU/fan/heatsink) well enough to be useful in automated support pipelines. It runs 29 models and four prompting strategies over the 853-report FACTO dataset, logging 98,948 inferences and over 51 million input tokens. The headline result is a flattened size–performance curve: F1 scores top out near 0.76, and gemma-2-2b-it reaches 0.727 with only 2B parameters, close to the 0.761 of mistral-small-24b-instruct. A sympathetic reader would care because the finding suggests manufacturers can run diagnosis on laptops or phones with NPUs without sending customer reports to cloud APIs. The paper also argues that chain-of-thought prompting helps larger models most, while few-shot examples are nearly essential for small models to produce parseable output.","feed_headline":"A 2B-parameter LLM nearly matches a 24B model at PC fault diagnosis","feed_subtitle":"With few-shot prompting, the 2B model scores 0.727 F1 versus 0.761 for the 24B model.","key_machinery":"The mechanism that carries the argument has three parts: a fixed prompting protocol, a labeled dataset, and a Pareto-frontier evaluation. The prompting protocol begins with a 'Common Prompt' that defines the eight-component taxonomy and demands a single Python dictionary as output, optionally augmented by a 'Chain of Thoughts' frame and a 'Few-Shot' frame containing three worked examples (Memory, Audio, Storage). The FACTO dataset supplies the 853 labeled user reports every model is scored on, and the evaluation parses the dictionary outputs into component labels, computing F1 with bootstrap error bars and then tracing the Pareto frontier in the F1-versus-parameter-count plane. The dictionary-output constraint is doing critical work: models that identify the right component but wrap it in prose are scored as failures, which is why few-shot examples rescue small models.","core_discovery":"On the paper's own terms, the discovery is that mapping a report like 'my connection keeps timing out' to one of eight hardware components is solvable by instruction-tuned open-source LLMs at a practical level, and the size–performance frontier is surprisingly flat: the 2B-parameter gemma-2-2b-it achieves an F1 of 0.7269, while the 24B mistral-small-24b-instruct-2501 achieves 0.7608 and the 1B llama-3.2-1b-instruct reaches 0.659 under CoT+FS. The authors identify these three as Pareto-optimal, meaning no other tested model is both smaller and more accurate. They further find that prompt design shifts results more for small models than for large ones: qwen2.5-1.5b-instruct goes from 0.000 F1 under zero-shot to 0.673 under few-shot because zero-shot outputs fail to conform to the required dictionary format, while gpt-4o gains most from chain-of-thought, reaching 0.767.","pith_inferences":["If the three worked examples shown in the Few-Shot and CoT+FS prompts were drawn from the same 853-report FACTO set used for evaluation—the paper does not state where they came from—the few-shot results are partially contaminated by test-set labels, and the reported small-model gains (e.g., qwen2.5-1.5b jumping from 0.000 to 0.673) could be inflated.","Because the evaluation requires a dictionary output, the F1 scores conflate diagnosis quality with formatting compliance; a more forgiving parser that extracts the component name from prose would likely raise the zero-shot scores of small models and could change the model ranking.","The few-shot examples use the same eight label names as the taxonomy, so a model could pass by label-matching rather than by reasoning; rephrasing the examples or using previously unseen symptom phrasings would test whether the gains generalize.","If the observed plateau holds on other hardware failure types, a small specialized model distilled from mistral-small-24b or qwen2.5-32b could plausibly approach the 0.76 ceiling on a phone NPU, making fully local diagnostics standard in support applications."],"forward_implications":["Hardware manufacturers can deploy local, on-device LLM triage for user reports without cloud APIs: a 2B model at about 4.2 GB VRAM reaches F1 0.727, and a 1B model at 2.8 GB reaches 0.659.","For large models, chain-of-thought is the strongest single strategy (gpt-4o 0.767, mistral-small-24b 0.761), while for models under about 7B parameters, few-shot examples are the difference between a usable classifier and one that fails to format output at all.","The performance plateau near 30B parameters means most of the diagnostic value is available at far lower compute; scaling beyond that buys little on this eight-class task.","The Pareto frontier gives practitioners a concrete shortlist—llama-3.2-1b-instruct, gemma-2-2b-it, and mistral-small-24b-instruct—for choosing a model based on available memory and target accuracy.","These results position the high-scoring models as teacher candidates for knowledge distillation into compact NPU-friendly student models for real-time on-device diagnostics."],"supporting_citations":[{"why":"Supplies the FACTO dataset of 853 labeled user reports that all models and prompting strategies are scored against.","marker":"Silva et al. 2024"},{"why":"Defines the task of classifying user reports into faulty computer components and provides the smaller-model baselines this work extends.","marker":"Silva et al. 2025"},{"why":"Provides the bootstrap resampling method used to attach standard deviations to every F1 score.","marker":"Efron and Tibshirani 1986"},{"why":"Supplies the Pareto-frontier visualization concept used to identify models balancing F1 against parameter count.","marker":"Lotov and Miettinen 2008"},{"why":"Provides the non-parametric regression method used to draw the size–performance trends in the F1 plots.","marker":"Nadaraya 1964"}],"fun_headline_variants":["2B LLM nearly matches 24B at PC fault diagnosis","Small LLMs rival big ones for hardware fault finding","LLM size matters less in diagnosing PC faults","2B gemma-2 close to 24B mistral in fault F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the three worked examples shown in the Few-Shot and CoT+FS prompts were not taken from the same 853-report FACTO set used for scoring; if they were, the few-shot results are partially contaminated by test-set labels and the reported small-model gains are inflated.","fun_headline_variants_meta":{"raw":{"variants":["2B LLM nearly matches 24B at PC fault diagnosis","Small LLMs rival big ones for hardware fault finding","LLM size matters less in diagnosing PC faults","2B gemma-2 close to 24B mistral in fault F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1610,"prompt_tokens":994,"completion_tokens":616,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":544}},"tokens_in":610,"tokens_out":616,"duration_ms":7281,"temperature":1.0,"reasoning_tokens":544,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:07:05.995337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Few-Shot and CoT+FS evaluations with the three example reports (Memory, Audio, Storage) removed from the FACTO test set, or replaced by held-out reports; if qwen2.5-1.5b no longer jumps from 0.000 to 0.673 and stablelm-zephyr-3b no longer jumps from 0.474 to 0.708, the few-shot advantage for small models is an artifact of example overlap.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FACTO dataset of 853 labeled user reports that all models and prompting strategies are scored against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the task of classifying user reports into faulty computer components and provides the smaller-model baselines this work extends."},{"cited_title":"and Tibshirani, R","cited_arxiv_id":null,"evidence_quote":"Provides the bootstrap resampling method used to attach standard deviations to every F1 score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Pareto-frontier visualization concept used to identify models balancing F1 against parameter count."}],"review_version":1}