{"id":"ca0ecb43-5b15-423b-814a-ce5f7480edeb","arxiv_id":"2501.16899","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuned, quantized LLMs achieve about 93% accuracy on an internal RoboCup@Home planning benchmark and run locally on Jetson hardware.","lead":"The authors fine-tune small, quantized language models on a new 27,000-example household robotics planning dataset, reporting about 93% accuracy for RoboCup@Home style tasks. The value is in showing a local, self-aware planning stack can run on edge hardware, though the evaluation relies on the authors' own benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 92.98% accuracy claim is not verifiable: Section IV-A reports no train/test split, and Section III-A uses the same dataset for training and evaluation, so the result may reflect memorization rather than generalization.","rationale":"The reader's weakest assumption correctly identifies the evaluation protocol as the most load-bearing point: the accuracy figure is the linchpin of the paper's claims, but the paper never documents a train/test split. Section III-A explicitly describes the dataset as both the training data and the benchmark, strongly suggesting that the reported accuracy may be computed on training examples. This is not merely a missing detail; it directly determines whether the central claim—that fine-tuned 4-bit LLMs achieve high-accuracy planning—holds. The memory issue (8GB claim vs. RDMM-8B's 8.5GB VRAM requirement) is also relevant, but it is disclosed in Section IV-B and primarily affects the framing of the abstract. The dataset-split concern is more damaging because it threatens the validity of the primary experimental result. The reader's CONDITIONAL verdict is appropriate: the paper can be salvaged if the authors provide a clear train/test split and re-report accuracy on held-out data. No change to the reader's verdict is needed.","tokens_in":7917,"tokens_out":5485,"duration_ms":46866,"concrete_test":"Download the released repository (github.com/shadynasrat/RDMM) and inspect the dataset for an explicit train/test split or evaluation script. If none exists, re-run the Section IV-A evaluation on a stratified 20% held-out subset of the 27,514 examples (held out from fine-tuning) and compare the RDMM-8B accuracy to the reported 92.98%. A large drop (e.g., to near the base model's 44%) would confirm that the original figure was an artifact of training-set evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the 92.98% accuracy figure (Section IV-A) being a measure of generalization, not memorization. The paper never specifies a train/test split. Section III-A states the 27,514-example dataset 'serves as a benchmark for evaluating our models but also plays a crucial role in training the robot,' which implies the same data may be used for both. The evaluation section only says baselines and GPT models were conditioned with 20-shot examples from the dataset; it does not report how many held-out prompts were used, how accuracy was scored, or any variance. If RDMM-8B was evaluated on fine-tuning examples, the reported average of 92.98% (vs. base model's 44.34%) could reflect the model learning to reproduce training outputs, not the ability to plan in new RoboCup@Home scenarios. This is the weakest load-bearing assumption because every downstream claim—on-device viability, speed, competition performance—presupposes that the accuracy number is meaningful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RDMM, a framework that fine-tunes small quantized LLMs (Llama3-8B, Mistral-7B, Qwen2-0.5B) on a 27,514-example RoboCup@Home planning dataset augmented with robot self-knowledge, and integrates VLM, YOLO, Whisper, and TTS components for a home-service robot. The authors report that RDMM-8B achieves 92.98% average planning accuracy versus 44.34% for its base model, provide inference-speed and memory measurements across several Jetson devices, and describe an anecdotal deployment at the RoboCup@Home competition. The contributions include the framework, the dataset, the benchmark, and the model weights, which are stated to be publicly available on GitHub.","tokens_in":8285,"tokens_out":4288,"duration_ms":38593,"significance":"If the accuracy figure is a genuine measure of held-out generalization, the paper would make a useful practical contribution: it demonstrates that a fine-tuned 4-bit LLM can generate domain-specific household plans on edge hardware, and the released dataset and models would support reproducibility. The comparison across three model sizes and multiple Jetson platforms provides concrete engineering data that is often missing in LLM-robotics papers. However, the central quantitative claim is currently under-supported by the reported evaluation protocol, and the memory claims in the abstract and conclusion are not consistent with the measurements in Section IV-B.","major_comments":[{"comment":"The headline accuracy of 92.98% is not established as a measure of generalization because the manuscript never specifies a train/test split and states in Section III-A that the 27,514-example dataset 'serves as a benchmark for evaluating our models but also plays a crucial role in training the robot.' Without reporting how many held-out prompts were used, how outputs were scored, and the per-task variance, the improvement over the base model (44.34%) could reflect reproduction of training examples rather than planning ability in new scenarios. This is the central evidence for every downstream claim, so the evaluation protocol must be made explicit.","section":"III-A / IV-A"},{"comment":"The abstract and conclusion claim operation 'on hardware with as little as 8GB of memory,' but Section IV-B2 reports that the complete framework (RDMM plus VLM, Whisper, TTS, and YOLO) required 30GB and that the 32GB Xavier AGX was the smallest device able to run it. Section IV-B1 also states that RDMM-8B itself needs 8.5GB VRAM and 1.1GB RAM, so on the 8GB Xavier NX only RDMM-0.5B fits. The 8GB claim should be scoped to the RDMM-0.5B model alone, or the framework-level memory measurements should be reconciled with the abstract.","section":"Abstract / IV-B"},{"comment":"The real-world evaluation is anecdotal. It lists activities performed at RoboCup@Home but reports no number of trials, task success rates, failure cases, or comparison with offline accuracy. As written, it cannot support the claimed contribution of 'Real-world evaluation of our system at the RoboCup@Home competition, demonstrating its ability to handle complex robotic tasks within a household environment.'","section":"IV-D"}],"minor_comments":[{"comment":"The phrase 'an 93% accuracy' should be corrected to 'a 93% accuracy.'","section":"Abstract"},{"comment":"The text-to-speech model is called 'Seliro-TTS' in Section III-C4 and 'Serlio-TTS' in Section IV-B2; the spelling should be unified and a reference or model card citation provided.","section":"III-C4 / IV-B2"},{"comment":"The definition of doubleDeq as dequant(dequant(c1,c2), W_4bit) does not match the standard QLoRA double-quantization description, in which the first quantization constants dequantize the second constants and then the 4-bit weights are dequantized; the notation should be corrected.","section":"III-B, Eq. (2)"},{"comment":"The term 'Progressive Fine-Tuning with Layer-wise Re-calibration' is introduced to explain the inference-speed difference, but this method is not described or cited anywhere else in the paper; as written it is unexplained.","section":"IV-C"},{"comment":"The statement that baseline and GPT models were 'conditioned with 20-shots examples from the dataset' should specify whether the 20 examples were randomly sampled or fixed, and whether those examples were excluded from any reported training data.","section":"IV-A"},{"comment":"The RDMM row in Table I is visually confusing because the checkmarks and model names appear in separate cells without clear alignment with the other rows; reformatting would improve readability.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is the evaluation protocol: the 92.98% accuracy claim cannot be checked as written because no train/test split or scoring procedure is reported. This is fixable within the scope of the paper if the authors have retained a held-out test set and can report per-task counts and variances. The framework-level memory contradiction with the abstract should also be resolved before the paper is considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the reusable artifact here is the dataset, not the accuracy number. The paper ships a 27k-example RoboCup@Home planning dataset, 1.3k annotated images, and fine-tuned 4-bit LLMs with self-awareness prompts. That is a useful engineering package for the household-robotics crowd. But the headline 92.98% accuracy is not interpretable as a generalization result because the paper never documents a train/test split. Section III-A says the dataset 'serves as a benchmark for evaluating our models but also plays a crucial role in training the robot,' and Section IV-A gives no held-out set, error bars, or scoring protocol. Under those conditions, the improvement from 44.34% (base) to 92.98% (fine-tuned) could simply be memorization of training examples. That is the load-bearing weakness, and the stress-test note lands.\n\nWhat is actually good: they release the data and models, so the community can re-run and check. The on-device compatibility table across Jetson devices is concrete and useful, and the speed numbers give a real sense of deployment constraints. Fine-tuning with QLoRA on three model sizes is a sensible recipe. The self-awareness angle (system messages with personal memory) is a small twist, not a scientific leap, but it is integrated cleanly into the dataset design.\n\nOther soft spots, in proportion: the abstract's '8GB of memory' applies to the RDMM model alone; their own Section IV-B2 says the full framework needs 30GB, so the framing is misleading. Section IV-C introduces an unexplained term, 'Progressive Fine-Tuning with Layer-wise Re-calibration', that never appears in the method section. The real-world evaluation is anecdotal, fine as a demo but not evidence. None of these are fatal once the train/test issue is fixed.\n\nBottom line: this is a plausible engineering contribution with a potentially valuable dataset, but the central accuracy claim is currently unverified. A serious referee should see it, with the expectation of heavy revision: add a proper held-out evaluation with variance, clarify what memory figure applies to what, and clean up the phantom method name. If those are fixed, this could be a solid contribution to a robotics or LLM-applications venue. I would not desk-reject it.","headline":"The dataset and fine-tuned models are real, reusable assets, but the 92.98% accuracy claim is uninterpretable without a documented train/test split; worth sending to review, not desk-rejecting.","tokens_in":8681,"tokens_out":1913,"would_cite":true,"duration_ms":18486,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A quantized 8B model plans household robot tasks with 92.98% accuracy.","keywords":["on-device LLM inference","robotic planning","RoboCup@Home","QLoRA fine-tuning","4-bit quantization","self-aware robot","household service robot","planning dataset"],"falsifier":"Run RDMM-8B on a held-out set of 1,000 task descriptions drawn from live RoboCup@Home runs rather than from the authors' dataset, and compare its generated plans to expert human plans. If accuracy falls to the base model's roughly 44% level, the reported 92.98% would reflect dataset-specific fitting rather than transferable planning ability.","tokens_in":7749,"feed_emoji":"🤖","tokens_out":5688,"duration_ms":46168,"temperature":0.7,"pith_summary":"This paper claims that a small, quantized language model fine-tuned on a domain-specific dataset can give a household service robot both task-planning and self-awareness abilities, while running entirely on the robot's edge hardware. The authors build a family of 'RDMM' models from Llama3-8B, Mistral-7B, and Qwen2-0.5B by applying 4-bit quantization and QLoRA fine-tuning to 27,514 manually annotated RoboCup@Home planning examples. On their benchmark, the best model (RDMM-8B) plans with 92.98% accuracy versus 44.34% for the unmodified base model, and even the 0.5B variant outperforms GPT-4o-mini. The smallest model runs on an 8GB-class device, while the full framework, including vision, speech, and detection models, needs about 30GB. The paper also releases the dataset and an open-source framework.","feed_headline":"Fine-tuned LLM hits 93% planning accuracy for robots","feed_subtitle":"Local edge inference lets a 0.5B variant outplan GPT-4o-mini and keeps data on the robot.","key_machinery":"The load-bearing object is the RDMM model: a 4-bit GPTQ-quantized LLM with a QLoRA adapter, fine-tuned on the authors' 27,514-example household-planning dataset. The dataset's system messages give the model action descriptions, usage information, and access to the robot's 'personal memory,' which is what lets the model answer questions about itself rather than only emit plans. Around the model sits a parser that translates generated plans into executable commands and a controller that calls a 4-bit quantized vision-language model, YOLOv10/YOLOv8-pose, Whisper, and a TTS module to carry them out.","core_discovery":"The central claim is that incorporating the robot's own knowledge—identity, role, capabilities, and personal memory—into LLM-based planning is what makes small models competitive for real household robotics. The authors fine-tune quantized open-weight LLMs on a dataset organized into action-oriented tasks, self-awareness-oriented tasks, and combined action-and-memory tasks. The resulting RDMM-8B reaches 92.98% average planning accuracy, up from 44.34% for Llama3-8B; RDMM-7B reaches 87.21%, up from 38.48%; and RDMM-0.5B reaches 54.44%, up from 1.75%, beating GPT-4o-mini's 52.23%. The authors argue this closes the gap with cloud-based systems while operating locally on Jetson devices, and they validate the approach in real RoboCup@Home competition runs with the Lucio robot.","pith_inferences":["Editorial inference: The same fine-tune-and-quantize recipe could transfer to other structured robotics domains, such as warehouse logistics or assistive care, wherever a corpus of scenario-annotated plans can be assembled.","Editorial inference: The self-awareness capability points toward robots that can state their own limits and preferences, which could make human-robot interaction more predictable, though the paper only demonstrates scripted identity responses.","Editorial inference: A natural stress test is to measure accuracy on planning requests that mix multiple skills or require chain-of-thought, since the current benchmark appears to score each task type separately."],"forward_implications":["RDMM-8B's 92.98% planning accuracy is a 48.64-point gain over its base model, suggesting domain-specific fine-tuning can transform a generic small LLM into an expert planner without cloud inference.","The 0.5B variant runs on an 8GB Xavier NX and still outplans GPT-4o-mini, making private, low-latency planning feasible on inexpensive embedded hardware.","Because plans are emitted as structured action strings and parsed by a controller, the same RDMM family can drive different robots by swapping the low-level controllers and perception models.","The full system needs 30GB of memory, so the smallest practical edge deployment is a 32GB platform rather than the 8GB device the abstract highlights.","Releasing the 27,514 planning instances and 1.3k annotated images lets other groups benchmark and extend the approach."],"supporting_citations":[{"why":"Provides Llama3-8B, the base model for RDMM-8B and the 44.34% baseline.","marker":"[17]"},{"why":"Provides Mistral-7B-v0.3, the base model for RDMM-7B.","marker":"[18]"},{"why":"Provides Qwen2-0.5B, the base model for RDMM-0.5B.","marker":"[19]"},{"why":"Supplies the GPTQ method used to compress the models to 4-bit precision for edge deployment.","marker":"[20]"},{"why":"Supplies QLoRA, the efficient fine-tuning method that trains low-rank adapters on the quantized base models.","marker":"[21]"},{"why":"Defines the low-rank adapter mechanism that QLoRA builds upon.","marker":"[22]"},{"why":"Provides the quantized vision-language model that the framework uses for visual grounding and descriptions.","marker":"[23]"}],"fun_headline_variants":["93% robot planning with 8GB on-device LLMs","Fine-tuned 0.5B LLM outplans GPT-4o-mini","Self-aware LLM boosts robot planning to 93% accuracy","Household robots plan better with context-aware tiny LLMs","On-device LLM with self-knowledge hits 93% planning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result assumes the 27,514 manually annotated planning examples are representative, correctly labeled, and split so the test set does not overlap the training set; if the benchmark omits hard cases or leaks training data, the 92.98% accuracy will not generalize to competition conditions.","fun_headline_variants_meta":{"raw":{"variants":["93% robot planning with 8GB on-device LLMs","Fine-tuned 0.5B LLM outplans GPT-4o-mini","Self-aware LLM boosts robot planning to 93% accuracy","Household robots plan better with context-aware tiny LLMs","On-device LLM with self-knowledge hits 93% planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000317,"raw_usage":{"total_tokens":1803,"prompt_tokens":966,"completion_tokens":837,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":742}},"tokens_in":582,"tokens_out":837,"duration_ms":7510,"temperature":1.0,"reasoning_tokens":742,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T05:45:42.454702+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RDMM-8B on a held-out set of 1,000 task descriptions drawn from live RoboCup@Home runs rather than from the authors' dataset, and compare its generated plans to expert human plans. If accuracy falls to the base model's roughly 44% level, the reported 92.98% would reflect dataset-specific fitting rather than transferable planning ability.","supporting_citations":[{"cited_title":"mistral-7b-instruct-0.3v,","cited_arxiv_id":null,"evidence_quote":"Provides Mistral-7B-v0.3, the base model for RDMM-7B."},{"cited_title":"Qlora: Efficient finetuning of quantized llms,","cited_arxiv_id":null,"evidence_quote":"Supplies QLoRA, the efficient fine-tuning method that trains low-rank adapters on the quantized base models."}],"review_version":1}