Pith. sign in

REVIEW 4 major objections 6 minor 45 references

HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HomeBench is the first smart-home dataset pairing valid and invalid instructions with single- and multi-device control, and on it GPT-4o scores 0% on invalid multi-device commands.

desk verdict HomeBench is the first benchmark for invalid and multi-device smart-home instructions, with a real 0% zero-shot finding, but the abstract overstates how much ICL and fine-tuning fail. read the letter →

arxiv 2505.19628 v2 pith:DE3DCIJJ submitted 2025-05-26 cs.CL

Silin Li , Yuhang Guo , Jiashu Yao , Zeming Liu , Haifeng Wang This is my paper
classification cs.CL
keywords smarthomebenchmarklargelanguagemodelsinvalidinstructionsmulti-deviceinstructionexecutionvirtualenvironmentin-contextlearningretrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HomeBench is a benchmark built to test whether large language models can act as smart-home assistants when user commands are not simple, valid, single-device requests. It contributes the first smart-home dataset that combines valid and invalid instructions with single- and multi-device operations, totaling more than 170,000 instructions across 100 virtual homes with at least 47 devices each. The paper evaluates 13 LLMs and reports that models perform much worse on invalid and multi-device commands than on straightforward ones, with GPT-4o scoring 0.0% success on fully invalid multi-device instructions. The intended message is that current assistants, even with in-context learning, retrieval augmentation, and fine-tuning, are not safe enough for real deployment in this setting.

What carries the argument

The load-bearing component is the virtual home environment and its five-way instruction taxonomy. Each scenario is a Python-defined house with rooms, device classes, callable methods, and deliberately planted non-existent devices and methods; every user instruction is generated by GPT-3.5 paraphrasing a hand-coded API command (or an intentionally wrong one), and the model's output is matched against the gold API call sequence with success rate and F1 as metrics. The invalid-instruction machinery matters most: by embedding non-existent devices and methods into the environment, such as an air conditioner with no heating system, the benchmark forces the model either to emit error_input or to hallucinate a call, which is exactly where the 0% result comes from.

What would settle it

Collect a corpus of real smart-home instructions, including invalid and multi-device ones, from deployed assistants or user logs; run the same API-call evaluation protocol on the same LLMs. If real-request success rates diverge substantially from HomeBench's numbers, for example because users rarely issue fully invalid multi-device commands or because real devices accept commands HomeBench marks invalid, then the benchmark's difficulty does not transfer. Alternatively, have human raters annotate a random sample of HomeBench instructions for executability and compare their labels with the automatically generated gold labels; any systematic mismatch would invalidate the 0% reading.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that existing LLM-based smart-home systems have been evaluated almost exclusively on valid single-device instructions, and that this misses two routine failure modes of real users: commands that cannot be executed (because the device or function does not exist) and commands that span multiple devices. HomeBench is presented as the first dataset covering both dimensions, with five instruction categories: valid single, invalid single, valid multi, invalid multi, and mixed multi. The empirical finding is that 13 LLMs, including GPT-4o and reasoning models, collapse on invalid multi-device instructions: GPT-4o achieves a 0.0% success rate, and even the best model with few-shot prompting reaches only roughly 60 to 66 percent success on that category. The paper concludes that simple ICL and fine-tuning close some of the gap, but the benchmark remains far from practical use.

Load-bearing premise

The whole result rests on the assumption that GPT-3.5's paraphrases of hand-coded API commands, placed in hand-built virtual houses, capture the difficulty of real users' smart-home language and real device capabilities; quality control samples only 10% of instructions per scenario and is not shown to be human-validated against a corpus of real requests.

Editorial extensions

If this is right

  • If correct, current LLM assistants should not be trusted to reject invalid multi-device commands without additional safeguards, because a wrong action such as unlocking a door is a safety risk.
  • The performance trend VS > VM > IS > MM > IM means difficulty is roughly predictable by device count and invalidity, so benchmarks and products should report these categories separately rather than one aggregate score.
  • Simple few-shot ICL helps but is not sufficient: even o3-mini with ICL reaches only 38.49% success on mixed multi-device instructions, so routine deployment needs stronger mechanisms.
  • Fine-tuning on the training split pushes Qwen2.5-7B past GPT-4o-ICL overall, and out-of-distribution results on two unseen devices stay consistent, suggesting the dataset supports transferable instruction-following training.
  • Room-level retrieval-augmented generation is counterproductive in this setting because embedding similarity often retrieves the wrong rooms, so context pruning must be validated before use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 0% figure probably says more about how the task is posed, with long context of every device state, exact API syntax, and an error_input convention, than about real-world assistant safety; a deployed system with intent confirmation or a device registry lookup would likely do better.
  • The same valid/invalid and single/multi taxonomy could transfer to other tool-use and API-calling domains, making HomeBench a stress test for long-context reasoning and hallucination beyond smart homes.
  • A useful extension would be to compare GPT-3.5-generated instructions against a human-written test set of the same API commands, isolating whether the paraphrases add difficulty or noise beyond the underlying command structure.
  • The mutual-exclusion rules in the virtual rooms mean some invalid instructions are physically impossible, while others are merely unsupported by a device model; a deployed assistant may need to treat these two cases differently, and HomeBench does not separate them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. HomeBench introduces a benchmark of over 170k smart-home instructions in five categories (valid/invalid crossed with single/multi-device, plus mixed multi-device), synthesized from API-level commands in 100 virtual homes. Thirteen LLMs are evaluated under zero-shot prompting, four-shot in-context learning, retrieval-augmented generation, and LoRA fine-tuning, with success rate and F1 as metrics. The headline finding is that zero-shot models, including GPT-4o, achieve 0% success on invalid multi-device instructions, while ICL and fine-tuning produce substantial but incomplete improvements.

Significance. If validated, HomeBench would fill a genuine gap: existing smart-home benchmarks concentrate on valid single-device commands, whereas the rejection of invalid instructions and the coordination of multiple devices are safety-relevant and underexplored. The main zero-shot result is consistently observed across models and is a useful stress test. The released code/data, the OOD evaluation, the error analysis, and the systematic comparison of ICL, RAG, and fine-tuning are concrete strengths. However, the abstract's claim that the failure persists 'even with the help of' ICL, RAG, and fine-tuning is contradicted by the paper's own numbers, and the synthetic instruction pipeline's external validity is underdocumented. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection.

major comments (4)
  1. [Abstract; §4.3, Table 3; §5.4, Figure 5] The abstract states that GPT-4o achieves a 0.0% success rate on invalid multi-device instructions and that state-of-the-art LLMs 'still cannot perform well in this situation even with the help of in-context learning, retrieval-augmented generation, and fine-tuning.' This overstates the persistence of the failure. Table 3 shows GPT-4o-ICL reaching 61.86% SUCC on IM, and §5.4 reports that the fine-tuned Qwen2.5-7B exceeds 80% SUCC on IM and most other tasks. The zero-shot 0.0% result is defensible, but the claim that the failure remains under all three interventions is not supported. Please revise the abstract and conclusion to distinguish the zero-shot result from the substantial but incomplete gains obtained with ICL and fine-tuning.
  2. [§3.3; Appendix A.1.1] The external validity of the synthetic instruction pipeline is the load-bearing assumption for the headline finding, yet it is not established. The quality-control step samples only 10% of generated instructions per scenario, does not state whether the evaluator is human or an LLM, does not define the scoring scale behind 'score > 4,' and reports no inter-annotator agreement. Appendix A.1.1 claims that ChatGPT-3.5 instructions are 'comparable to human-written instructions' based on 200 samples, but the comparison data, metrics, and statistical results are not shown. Without a human-language validation corpus or a documented human evaluation of the synthesized instructions, it remains possible that the 0.0% IM result and the cross-model ranking reflect properties of the GPT-3.5 synthesis pipeline rather than of deployed smart-home assistants.
  3. [§4.1; §5.1; Appendix A.2.1, Table 12] The few-shot configuration is ambiguous and affects the main ICL results. The main text says the authors 'chose 4-shot' after observing limited improvement beyond 3-shot, and §5.1 describes a fixed order of adding demonstration types (VS, IS, VM, MM, IM). However, Table 12 shows five example blocks (VS, IS, VM, MM, and IM), and §5.1 reports that adding the IM shot raises IM F1 from 6.27% to 12.67%. If the ICL rows in Table 3 use four demonstrations without an IM example, the effect of ICL on IM tasks is understated; if they use five, the '4-shot' statement is incorrect. Please specify the exact number and type composition of demonstrations used in Table 3 and provide the corresponding prompt.
  4. [§4.2; Table 11; Table 3] The evaluation of invalid instructions needs clarification. Table 11 represents an IM gold output as three separate 'error_input' tokens, and Equation (2) defines recall with 'operation_gold_num' as the denominator. The paper does not state how precision and recall are computed when the entire gold output is error_input, nor how partial credit is assigned when a model emits a mixture of error_input and API calls. This matters because Table 3 reports IM F1 as high as 55.19 for GPT-4o despite a SUCC of 0.00; readers cannot interpret this discrepancy without the exact scoring rule.
minor comments (6)
  1. [Table 1] The symbols in Table 1 do not render correctly ('" %"', '!', '%'), making the comparison with existing datasets hard to read; use standard check/cross marks and ensure the PDF compiles.
  2. [§4.1, first paragraph] The sentence 'Inspired by Fei et al.; Huang et al.' lacks years and proper citation formatting; the same applies to the 'Shi et al.; Rivkin et al.' mentions in §4.2.
  3. [§4.3; Appendix A.2.5] The main results are reported as point estimates without confidence intervals or significance tests; for the IM task with only 97 test samples, stating the raw counts underlying 0.00% would improve interpretability.
  4. [§5.4; Appendix A.2.4] The OOD evaluation is described as introducing 'two previously unseen devices,' but Figure 14 shows counts for five categories (including 'beds' and 'pet feeders'); please clarify which devices are unseen and report the OOD numbers numerically rather than only graphically.
  5. [Appendix A.2.2] The RAG retrieval threshold ('similarity score exceeds 0.5') appears arbitrary; a sensitivity analysis over the threshold would help, since the RAG results are explained by retrieval context errors.
  6. [Throughout] There are several typos and OCR artifacts, including 'VacuumRobotrDevice' (Table 7), 'Inferface latecy' (Table 15), 'V o' (References), and garbled text in Table 1; a thorough proofreading pass is needed.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: HomeBench's gold labels are defined by the virtual environment before any evaluated model is run, and the headline 0.0% results are empirical outcomes rather than constructions.

full rationale

HomeBench's evaluation is not circular: the gold API sequences (or error_input) are generated from the virtual home's device/method inventory (§3.3, Fig. 2) before any evaluated model is run, and the evaluated models receive the same device state/method context (Table 8). Thus the headline results (e.g., GPT-4o 0.0% SUCC on IM in Table 3) are empirical outcomes of a fixed benchmark, not quantities fitted from the models. The self-citations—Wang et al. 2024d for GPT-3.5-based synthesis, and Wang et al. 2024b / Liu et al. 2020 for fluency checks—are methodological pointers and are not load-bearing: the selection of ChatGPT-3.5 is argued in Appendix A.1.1 from the paper's own 200-sample comparison, and the quality-check criteria are stated in §3.3. No uniqueness theorem or prior result by the authors is invoked to forbid alternative benchmark designs. The main defensible concern is external validity (GPT-3.5 paraphrases versus real user language, with the A.1.1 human-comparison data not shown), but that is a correctness or representativeness risk, not a circularity: the benchmark's labels are not derived from the claims it tests. Minor self-citations exist, so a low non-zero score is appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

HomeBench's central claims rest on construction choices (device pools, generation prompt, QC thresholds) and proxy assumptions (synthetic instructions stand in for real ones). None of these are fitted to external data, so they are enumerated as free parameters and domain assumptions. No new physical or conceptual entities are introduced.

free parameters (4)
  • Quality control sampling rate = 10%
    Section 3.3: only 10% of generated instructions per scenario are checked; the entire scenario is accepted if the sample scores above 4. This threshold shapes the benchmark's validity.
  • QC acceptance threshold = score > 4
    Section 3.3: scenario data is considered valid only if sampled instructions score above 4; the scoring procedure is not fully specified.
  • RAG retrieval similarity threshold = 0.5
    Appendix A.2.2: chunks whose cosine similarity exceeds 0.5 are returned directly, affecting the RAG experiment conclusions.
  • Few-shot demonstration count = 4
    Section 4.1: 4-shot chosen because improvement beyond 3-shot is limited in their analysis; this choice affects all ICL results.
assumptions (4)
  • domain assumption GPT-3.5-generated paraphrases are a faithful proxy for real user instructions
    Section 3.3 and A.1.1 justify selecting GPT-3.5 based on quality comparable to human-written instructions on 200 samples, but the proxy assumption itself is untested beyond that.
  • domain assumption The virtual environment API surface (15 device types and their methods in Table 7) captures the essential difficulty of real smart home control
    Section 3.3 simplifies device methods by hand and adds random methods to create diversity; no evidence that this matches real platforms.
  • domain assumption Sampled quality control (10% per scenario) is representative of the whole dataset
    Section 3.3: a scenario is accepted in full if the sampled 10% has an average score above 4, leaving 90% of instructions unchecked.
  • domain assumption API-call correctness is the right measure of assistant performance
    Section 4.2 defines success as generating API calls matching the gold specification, which ignores dialogue, safety, and user satisfaction aspects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices." pith.science (2026). https://pith.science/paper/DE3DCIJJ

@misc{pith2026250519628,
  author       = {Pith},
  title        = {Pith review of: HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DE3DCIJJ}},
  note         = {Machine review of arXiv:2505.19628}
}
read the original abstract

Large language models (LLMs) have the potential to revolutionize smart home assistants by enhancing their ability to accurately understand user needs and respond appropriately, which is extremely beneficial for building a smarter home environment. While recent studies have explored integrating LLMs into smart home systems, they primarily focus on handling straightforward, valid single-device operation instructions. However, real-world scenarios are far more complex and often involve users issuing invalid instructions or controlling multiple devices simultaneously. These have two main challenges: LLMs must accurately identify and rectify errors in user instructions and execute multiple user instructions perfectly. To address these challenges and advance the development of LLM-based smart home assistants, we introduce HomeBench, the first smart home dataset with valid and invalid instructions across single and multiple devices in this paper. We have experimental results on 13 distinct LLMs; e.g., GPT-4o achieves only a 0.0% success rate in the scenario of invalid multi-device instructions, revealing that the existing state-of-the-art LLMs still cannot perform well in this situation even with the help of in-context learning, retrieval-augmented generation, and fine-tuning. Our code and dataset are publicly available at https://github.com/BITHLP/HomeBench.

Figures

Figures reproduced from arXiv: 2505.19628 by the authors.

Figure 1
Figure 1. An example of valid single-device instruction, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The whole process of collecting dataset HomeBench, from device construction, room setting, in￾structions generation, and user instructions synthesis to quality control. instructions. Sasha implements a decision-making process where each step (e.g., device selection or routine verification) is supported by LLMs. The SAGE system addresses the limitations of LLMs in lacking specific knowledge about users and house￾hold… view at source ↗
Figure 3
Figure 3. The performance of Qwen2.57BInstruct model in adding different types of data samples (the order of adding is: VS, IS, VM, MM, IM) [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: The performance gap between Qwen-ICL and [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Performance of Qwen2.5-7B-Instruct in test [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: ALL room types in HomeBench. A.1 Data Collection [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Smart home virtual room configuration: ensuring logical device selection. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 9
Figure 9. Figure 9: Performance of Gemma2-9B-Instruct on test [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: The LoRA configuration used for fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: The performance of Llama3-8B-Instruct model in adding different types of data samples (the order of adding is: VS, IS, VM, MM, IM) [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: The performance of Gemma2-9B-Instruct model in adding different types of data samples (the order of adding is: VS, IS, VM, MM, IM). 0 1 2 3 4 5 0 20 40 60 80 100 Shot Numbers SUCC ALL VS IS VM MM IM 0 1 2 3 4 5 0 20 40 60 80 100 Shot Numbers F1 ALL VS IS VM MM IM [PI…
Figure 13
Figure 13. Figure 13: The performance of Mistral-7B-v0.3 model [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: OOD data distribution. potential issues affecting the model’s performance across different data types. A.2.4 OOD We have created two new smart devices: beds and pet feeders to construct ood data. The data distribu￾tion is shown in the [PITH_FULL_IMAGE:figures/full_fi…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 9 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    2020

  5. [5]

    Yuxiang Cai, Qiao Liu, Yanglei Gan, Run Lin, Changlin Li, Xueyi Liu, Da Luo, and JiayeYang JiayeYang. 2024. https://doi.org/10.18653/v1/2024.acl-long.349 D i F i N et: Boundary-aware semantic differentiation and filtration network for nested named entity recognition . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisti...

    work page doi:10.18653/v1/2024.acl-long.349 2024

  6. [6]

    Liying Cheng, Lidong Bing, Ruidan He, Qian Yu, Yan Zhang, and Luo Si. 2022. https://doi.org/10.18653/v1/2022.acl-long.162 IAM : A comprehensive and large-scale dataset for integrated argument mining tasks . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2277--2287, Dublin, Ireland....

    doi:10.18653/v1/2022.acl-long.162 2022

  7. [7]

    Gabriele Civitarese, Michele Fiori, Priyankar Choudhary, and Claudio Bettini. 2024. https://arxiv.org/abs/2407.01238 Large language models are zero-shot recognizers for activities of daily living . Preprint, arXiv:2407.01238

    arXiv 2024

  8. [8]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

Show all 45 references
  1. [9]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, , and et al

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2024 arXiv

  2. [10]

    Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, Jidong Ge, and Vincent Ng. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.452 L aw B ench: Benchmarking legal knowledge of large language models . In ...

    2024 doi

  3. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  4. [12]

    Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, and Kang Liu. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.748 DA -code: Agent data science code generation benchmark for large language models . In Proceedi...

    2024 doi

  5. [13]

    Shadi Iskander, Sofia Tolmach, Ori Shapira, Nachshon Cohen, and Zohar Karnin. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.285 Quality matters: Evaluating synthetic data for tool-using LLM s . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language ...

    2024 doi

  6. [14]

    John Jaihar, Neehal Lingayat, Patel Sapan Vijaybhai, Gautam Venkatesh, and K. P. Upla. 2020. https://doi.org/10.1109/INCET49848.2020.9154007 Smart home automation using machine learning algorithms . In 2020 International Conference for Emerging Technology (INCET), pages 1--4

    2020

  7. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  8. [16]

    Chung‐Wha (Chloe) Ki, Erin Cho, and Jung-Eun Lee. 2020. https://api.semanticscholar.org/CorpusID:219065737 Can an intelligent personal assistant (ipa) be your friend? para-friendship development mechanism between ipas and their users . Comput. Hum. Behav., 111:106412

    2020

  9. [17]

    Evan King, Haoxiang Yu, Sangsu Lee, and Christine Julien. 2024. https://doi.org/10.1145/3643505 Sasha: Creative goal-oriented reasoning in smart homes with large language models . Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 8(1)

    2024 doi

  10. [18]

    Sunjun Kweon, Yeonsu Kwon, Seonhee Cho, Yohan Jo, and Edward Choi. 2023. https://doi.org/10.18653/v1/2023.findings-acl.526 Open- W iki T able : Dataset for open domain question answering with complex reasoning over table . In Findings of the Association for Computational Lingu...

    2023 doi

  11. [19]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f78...

    2020

  12. [20]

    Qiwei Li, Zuchao Li, Ping Wang, Haojun Ai, and Hai Zhao. 2024. https://doi.org/10.18653/v1/2024.acl-long.162 Hypergraph based understanding for document semantic entity recognition . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vo...

    2024 doi

  13. [21]

    Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020. https://doi.org/10.18653/v1/2020.acl-main.98 Towards conversational recommendation over multi-type dialogs . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguis...

    2020 doi

  14. [22]

    like having a really bad pa

    Ewa Luger and Abigail Sellen. 2016. https://doi.org/10.1145/2858036.2858288 "like having a really bad pa": The gulf between user expectation and experience of conversational agents . In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16, page...

    2016

  15. [23]

    OpenAI, :, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 contrib...

    2025 arXiv

  16. [24]

    Praveen Kumaar, U

    Paul Jasmin Rani, Jason Bakthakumar, B. Praveen Kumaar, U. Praveen Kumaar, and Santhosh Kumar. 2017. https://doi.org/10.1109/ICONSTEM.2017.8261311 Voice controlled home automation system using natural language processing (nlp) and internet of things (iot) . In 2017 Third Inter...

    2017

  17. [25]

    Dmitriy Rivkin, Francois Hogan, Amal Feriani, Abhisek Konar, Adam Sigal, Xue Liu, and Gregory Dudek. 2024. https://doi.org/10.1109/JIOT.2024.3471904 Aiot smart home via autonomous llm agents . IEEE Internet of Things Journal, pages 1--1

    2024

  18. [26]

    Yingtian Shi, Xiaoyi Liu, Chun Yu, Tianao Yang, Cheng Gao, Chen Liang, and Yuanchun Shi. 2024 a . https://arxiv.org/abs/2408.12687 Bridging the gap between natural user expression with complex automation programming in smart homes . Preprint, arXiv:2408.12687

    2024 arXiv

  19. [27]

    Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.397 Generate-then-ground in retrieval-augmented generation for multi-hop question answering . In Proceedings of the 62nd Annual Meet...

    2024 doi

  20. [28]

    John Sweller. 2011. https://doi.org/10.1016/B978-0-12-387691-1.00002-8 Chapter two - cognitive load theory . volume 55 of Psychology of Learning and Motivation, pages 37--76. Academic Press

    2011 doi

  21. [29]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  22. [30]

    Qwen Team. 2025. https://qwenlm.github.io/blog/qwq-32b/ Qwq-32b: Embracing the power of reinforcement learning

    2025

  23. [31]

    Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, and Hao Wang. 2024 a . https://arxiv.org/abs/2406.11230 Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models . Preprint, ...

    2024 arXiv

  24. [32]

    Pan, and Kam-Fai Wong

    Hongru Wang, Rui Wang, Boyang Xue, Heming Xia, Jingtao Cao, Zeming Liu, Jeff Z. Pan, and Kam-Fai Wong. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.856 A pp B ench: Planning of multiple API s from various APP s for complex user instruction . In Proceedings of the 2024 ...

    2024 doi

  25. [33]

    Huiming Wang, Liying Cheng, Wenxuan Zhang, De Wen Soh, and Lidong Bing. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.421 Order-agnostic data augmentation for few-shot named entity recognition . In Proceedings of the 62nd Annual Meeting of the Association for Computationa...

    2024 doi

  26. [34]

    Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, and Yunhong Wang. 2024 d . https://arxiv.org/abs/2410.12896 A survey on data synthesis and augmentation for large language models . Preprint, arXiv:2410.12896

    2024 arXiv

  27. [35]

    Zheng Wang, Shu Teo, Jieer Ouyang, Yongjun Xu, and Wei Shi. 2024 e . https://doi.org/10.18653/v1/2024.acl-long.108 M - RAG : Reinforcing large language model performance through retrieval-augmented generation with multiple partitions . In Proceedings of the 62nd Annual Meeting...

    2024 doi

  28. [36]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  29. [37]

    Ziqi Yin, Mingxin Zhang, and Daisuke Kawahara. 2024. https://arxiv.org/abs/2410.14252 Harmony: A home agent for responsive management and action optimization with a locally deployed large language model . Preprint, arXiv:2410.14252

    2024 arXiv

  30. [38]

    Haoxiang Yu, Jie Hua, and Christine Julien. 2021. https://doi.org/10.1145/3485730.3494115 Analysis of ifttt recipes to study how humans use internet-of-things (iot) devices . In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, SenSys ’21, page 537–541. ACM

    2021

  31. [39]

    Xin Zeng, Xiaoyu Wang, Tengxiang Zhang, Chun Yu, Shengdong Zhao, and Yiqiang Chen. 2024. https://doi.org/10.1145/3698145 Gesturegpt: Toward zero-shot free-form hand gesture understanding with large language model agents . Proc. ACM Hum.-Comput. Interact., 8(ISS)

    2024 doi

  32. [40]

    Bowen Zhang, Kehua Chang, and Chunping Li. 2024 a . Simple techniques for enhancing sentence embeddings in generative language models. In Advanced Intelligent Computing Technology and Applications, pages 52--64, Singapore. Springer Nature Singapore

    2024

  33. [41]

    Wenxiao Zhang, Xiangrui Kong, Thomas Braunl, and Jin B. Hong. 2024 b . https://arxiv.org/abs/2409.01630 Safeembodai: a safety framework for mobile robots in embodied ai systems . Preprint, arXiv:2409.01630

    2024 arXiv

  34. [42]

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024 c . https://doi.org/10.18653/v1/2024.acl-long.814 B ench: Extending long context evaluation beyond 100 K tokens . In Proceedings of the 6...

    2024 doi

  35. [43]

    Zhuo Zhang, Xiangjing Hu, Jingyuan Zhang, Yating Zhang, Hui Wang, Lizhen Qu, and Zenglin Xu. 2023. https://doi.org/10.18653/v1/2023.acl-long.193 FEDLEGAL : The first real-world federated learning benchmark for legal NLP . In Proceedings of the 61st Annual Meeting of the Associ...

    2023 doi

  36. [44]

    Xiaoling Zhou, Wei Ye, Yidong Wang, Chaoya Jiang, Zhemg Lee, Rui Xie, and Shikun Zhang. 2024. https://doi.org/10.18653/v1/2024.acl-long.155 Enhancing in-context learning via implicit demonstration augmentation . In Proceedings of the 62nd Annual Meeting of the Association for ...

    2024 doi

  37. [45]

    Zihao Zhu, Bingzhe Wu, Zhengyou Zhang, Lei Han, Qingshan Liu, and Baoyuan Wu. 2024. https://arxiv.org/abs/2408.04449 Earbench: Towards evaluating physical risk awareness for task planning of foundation model-based embodied ai agents . Preprint, arXiv:2408.04449

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.