HomeBench is a new smart home benchmark that exposes near-zero success rates for top LLMs on invalid multi-device instructions.
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
5 Pith papers cite this work, alongside 5 external citations. Polarity classification is still indexing.
5
Pith papers citing it
5
external citations · external index
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
unclear 1representative citing papers
Extending training-dynamics data maps to multi-label, labeling, and parsing tasks shows LLM-generated data has distinct learnability patterns across model families and responds differently to difficulty-based data selection than human data.
Introduces a three-dimension three-scope variance-based monitoring and FMEA-adapted triage methodology for structural defects in partially integrated agentic systems, validated on synthetic data showing scope-specific failure signatures and effective automation of 97% of findings.
For meaning-matched paraphrases, higher estimated sentence frequency improves LLM prompting and fine-tuning across math, translation, commonsense, and tool-calling tasks.
A metric-oriented survey that classifies intrinsic quality and trustworthiness metrics for LLM-generated data across six modalities and documents systematic evaluation gaps in the current literature.
citing papers explorer
-
HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices
HomeBench is a new smart home benchmark that exposes near-zero success rates for top LLMs on invalid multi-device instructions.
-
Synthetic Data Characterization via Training Dynamics
Extending training-dynamics data maps to multi-label, labeling, and parsing tasks shows LLM-generated data has distinct learnability patterns across model families and responds differently to difficulty-based data selection than human data.
-
Monitoring Agentic Systems Before They're Reliable
Introduces a three-dimension three-scope variance-based monitoring and FMEA-adapted triage methodology for structural defects in partially integrated agentic systems, validated on synthetic data showing scope-specific failure signatures and effective automation of 97% of findings.
-
Adam's Law: Textual Frequency Law on Large Language Models
For meaning-matched paraphrases, higher estimated sentence frequency improves LLM prompting and fine-tuning across math, translation, commonsense, and tool-calling tasks.
-
A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
A metric-oriented survey that classifies intrinsic quality and trustworthiness metrics for LLM-generated data across six modalities and documents systematic evaluation gaps in the current literature.