RL fine-tuning of LLMs improves clean-benchmark accuracy while degrading accuracy under three injected-distractor evaluation scenarios, though one of the three scenarios contradicts the headline claim.
Evaluating LLMs and Prompting Strategies for Automated Hardware Diagnosis from Textual User-Reports
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Computer manufacturers offer platforms for users to describe device faults using textual reports such as "My screen is flickering". Identifying the faulty component from the report is essential for automating tests and improving user experience. However, such reports are often ambiguous and lack detail, making this task challenging. Large Language Models (LLMs) have shown promise in addressing such issues. This study evaluates 27 open-source models (1B-72B parameters) and 2 proprietary LLMs using four prompting strategies: Zero-Shot, Few-Shot, Chain-of-Thought (CoT), and CoT+Few-Shot (CoT+FS). We conducted 98,948 inferences, processing over 51 million input tokens and generating 13 million output tokens. We achieve f1-score up to 0.76. Results show that three models offer the best balance between size and performance: mistral-small-24b-instruct and two smaller models, llama-3.2-1b-instruct and gemma-2-2b-it, that offer competitive performance with lower VRAM usage, enabling efficient inference on end-user devices as modern laptops or smartphones with NPUs.
fields
cs.AI 1years
2025 1verdicts
REJECT 1representative citing papers
citing papers explorer
-
Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning
RL fine-tuning of LLMs improves clean-benchmark accuracy while degrading accuracy under three injected-distractor evaluation scenarios, though one of the three scenarios contradicts the headline claim.