{"id":"297d1507-e88e-4dda-a37e-e8fb9d8ebc5f","arxiv_id":"2507.22896","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An interactive robot pipeline that clarifies user intent, retrieves past corrected answers, and fine-tunes an MLLM on dialogue history improved accuracy on a 10-bottle medicine recognition task from 28.8% to 71.6%.","lead":"This paper describes a robot system that asks users clarifying questions, stores corrected answers, and reuses that history to answer better both before and after a model update. The goal is to let non-experts improve a vision-language robot's perception through ordinary conversation instead of waiting for retraining.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified user-supplied labels serve as both the training signal and the evaluation ground truth, so the reported accuracy gains may reflect learning to match user beliefs rather than true bottle attributes.","rationale":"The reader's weakest_assumption already identified the core issue: user corrections are treated as correct answers without verification. My analysis agrees and sharpens it: the same unverified user judgments also provide the evaluation labels, creating a potential circularity that threatens even the internal validity of the reported numbers. This is more load-bearing than the generalization concern (testing on the same bottles) because it means the measured improvement could be an artifact of self-confirming labels, not a genuine gain in perception. The other limitations—no error bars, unspecified thresholds, same-bottle testing—are addressable with standard experimental hygiene, but if the ground-truth circularity holds, the central claim cannot be believed regardless of those fixes. The framework is coherent and the components are standard, so this is not a rejection of the approach; it is a demand for an external, verified ground-truth evaluation and an explicit handling of unreliable user corrections. The reader's CONDITIONAL verdict is the right call, and no verdict change is needed; the proposed test would either resolve the concern or escalate it to a rejection of the current evidence.","tokens_in":7963,"tokens_out":4278,"duration_ms":48408,"concrete_test":"Construct a held-out set of the same 10 medicine bottles with labels independently verified against package inserts or a pharmacist, then recompute the round-1/2/3 accuracies against this expert ground truth instead of the participants' self-reported correctness. If the expert-verified accuracies differ materially from 0.288/0.616/0.716, or if injecting one deliberately wrong user correction causes the round-3 model to repeat that error, the central claim of truthful improvement is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that iLearnRobot improves visual recognition by learning from natural dialogues with non-expert users. Section 3.2 (Data Construction) distills user corrections into a 'Correct Answer' and stores it verbatim in the history event database, with no verification, confidence score, or conflict resolution. Section 3.3 then fine-tunes LLaVA-NeXT on (image, question, user-supplied answer). Meanwhile, the evaluation in Section 4.1 asks the same 25 non-expert participants to 'record whether the robot generated the correct answer'; no independent ground-truth label set is used. Thus the training signal and the evaluation metric are both the participants' own judgments. If a participant gives or accepts a wrong label (e.g., confusing Thiamine and Riboflavin in Fig. 12), that error is encoded, retrieved in round 2, and reinforced in round 3, so the reported improvement from 0.288 to 0.716 may reflect the model learning to parrot user beliefs rather than learning true bottle attributes. The framework has no safeguard against a single bad correction propagating through retrieval and fine-tuning, directly contradicting the 'avoid repeating same mistakes' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes iLearnRobot, a framework that integrates interactive learning into an MLLM-based robot. It combines a chain-of-question module to clarify user intent, a dual-modality retrieval module that recalls past interaction events (image-region embeddings and question embeddings) before answering, and periodic LoRA fine-tuning of LLaVA-NeXT on distilled dialogue events. The authors report three rounds of experiments with 25 participants and 10 medicine bottles: accuracy rises from 0.288 in the baseline round to 0.616 with retrieval and to 0.716 after fine-tuning, with corresponding dialogue scores of 3.34, 5.38, and 7.1. An ablation compares freezing versus updating the visual encoder during fine-tuning. The central claim is that the robot can continuously improve its visual perception and recognition by learning from natural dialogues with non-expert users rather than relying on expert-annotated datasets.","tokens_in":8204,"tokens_out":4568,"duration_ms":49808,"significance":"If properly validated, the framework would address a genuine and under-studied problem: adapting a deployed robot's visual perception and recognition to novel domains by exploiting everyday human corrections, while maintaining a good interaction experience through clarification and retrieval. The architecture is simple, the qualitative examples are illustrative, and comparing visual-encoder freezing versus updating is a useful practical insight. However, the current empirical evidence does not support the central claim because the evaluation protocol is circular: the fine-tuned model is assessed on the same medicine bottles used to construct its training data, and the ground truth for both training and evaluation is supplied by the same non-expert participants without independent verification. The paper also omits key implementation details such as retrieval thresholds and lacks statistical validation. These are load-bearing issues that require a re-designed evaluation rather than cosmetic fixes.","major_comments":[{"comment":"The round-3 evaluation is performed on the same 10 medicine bottles whose interaction events were used to fine-tune LLaVA-NeXT, so the reported accuracy of 0.716 is at least partly a measure of training-data memorization rather than generalization to novel scenarios, which is the paper's stated motivation. Please add a held-out evaluation, for example leave-one-bottle-out or a new set of bottles and new participants, and report per-bottle and per-question accuracy for that split.","section":"§4.1, Fig. 12 and §3.3"},{"comment":"The stored 'Correct Answer' is extracted from user interaction without any verification, and the evaluation score is also recorded by the same non-expert participants who hold the bottles. This makes the training signal and the evaluation ground truth the same subjective judgment, so the measured improvement may reflect the model learning to reproduce participant beliefs rather than true bottle attributes. I request independent ground-truth labels, for example labels verified against manufacturer information, and a separate evaluation of whether wrong user labels propagate through retrieval and fine-tuning.","section":"§3.2 Data Construction and §4.1 evaluation protocol"},{"comment":"The retrieval success condition is described as 'If the cosine similarities each meet a certain specific threshold,' but the threshold values are never reported or analyzed. Without these values and a sensitivity analysis, the round-2 result of 0.616 cannot be reproduced or assessed; please report the thresholds, how they were chosen, and how retrieval precision and recall vary with them.","section":"§3.1 Dual-Modality Retrieval"},{"comment":"The accuracy and dialogue-score results are reported as point estimates without error bars, confidence intervals, or significance tests. The round-to-round differences are based on the same 25 participants answering questions about the same 10 bottles, so the dependencies should be accounted for with paired tests or bootstrap resampling. Please include per-question counts and appropriate statistical tests.","section":"§4.1, Table 1"},{"comment":"The claim that the system avoids repeating mistakes is not tested against the failure mode that matters most for the proposed interactive-learning loop: a non-expert user supplies a wrong label. The paper does not describe any confidence scoring, conflict resolution, or evaluation of label quality, and Fig. 12 suggests confusions such as Thiamine versus Riboflavin. Please add a quantitative analysis of user-label accuracy and an explicit treatment of conflicting or incorrect corrections.","section":"§3.2 Data Construction"}],"minor_comments":[{"comment":"The paper states d=576 for the CLIP ViT-B/16 embedding dimension, but the standard CLIP ViT-B/16 image and text embedding dimension is 512; please verify the checkpoint and dimension.","section":"§3.1"},{"comment":"The entry '5.387.1' appears to be a typographical join of 5.38 and 7.1; please separate the values clearly.","section":"Table 1a"},{"comment":"The paper does not state whether code, prompts beyond those shown, or collected data are available, which makes the reported experiments difficult to reproduce.","section":"General"},{"comment":"Several grammatical slips need correction, such as 'similar event' in the Fig. 1 caption and 'both quantitively and qualitatively' in the abstract.","section":"§1 and Fig. 1"},{"comment":"The description of the medicine set says 'Vitamin B1 (Thiamine), Vitamin B6 (Riboflavin), etc.'; the example labels are confusing because Thiamine and Riboflavin are easy to confuse, and the source of the ground-truth bottle attributes is never defined.","section":"§4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems demonstration rather than a controlled user study. For a cs.HC venue, the evaluation needs a proper ground-truth protocol, held-out data, and statistical rigor; without those, the reported numbers cannot be interpreted as evidence of continuous improvement. The core idea is salvageable, but the experimental design must be redone. I also note that no code or data release is mentioned, which compounds the reproducibility concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tim,\n\nQuick read: this is a coherent system paper, not a crackpot one. The integration of chain-of-question clarification, CLIP-based dual-modality retrieval of past corrections, and LoRA fine-tuning of LLaVA-NeXT is genuinely assembled rather than just proposed, and the ablation comparing frozen versus updatable visual encoders is a sensible piece of engineering. The writing is clear, the motivation (learning from natural dialogue with non-experts) is real, and the hardware demonstration with medicine bottles is a nice concrete setting. Credit where due.\n\nThe problem is the evaluation. The stress-test note lands. Section 3.2 says the system distills whatever the user says into a \"Correct Answer\" with no verification, conflict resolution, or confidence score. Section 4.1 then asks the same 25 participants to judge whether the robot's answer is correct. So both the training labels and the metric are the participants' own opinions. If a participant misidentifies Thiamine as Riboflavin, that error is stored, retrieved in round 2, and reinforced by fine-tuning in round 3. The reported jump from 0.288 to 0.716 could reflect the model learning to parrot user beliefs, not learning true bottle attributes.\n\nEven setting that aside, the evaluation is too weak to support the \"continuous improvement\" claim. Round 3 is tested on the same ten bottles used to build the training data, so the fine-tuned model is being measured on its training set. There are no error bars, no significance tests, and no independent ground-truth label set. The retrieval cosine-similarity thresholds are never reported. The variance numbers are interesting (round 2's variance jumps to 8.4, which they attribute to retrieval inconsistency), but without statistical analysis that's just descriptive. This is a small dataset and a single scenario; the generalization claim is unearned.\n\nWho gets value from this? Someone building a similar dialogue-driven robot pipeline might pick up useful design ideas, especially the retrieval-augmented response generation and the visual-encoder-updating ablation. But they should not cite the accuracy numbers as evidence of effectiveness.\n\nRecommendation: I would not accept this as-is, but I would send it to peer review rather than desk-reject. The flaws are addressable—an independent ground-truth set for bottle attributes, held-out bottles for fine-tuning evaluation, error bars, and reported thresholds would make the claims testable. A referee who understands evaluation issues could give the authors a clear path to a respectable revision.\n\nBest,\n[Your name]","headline":"A reasonable integration of known components whose reported gains are not credible because user-supplied labels serve as both training signal and evaluation ground truth on the same ten bottles.","tokens_in":8707,"tokens_out":1860,"would_cite":false,"duration_ms":23486,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A robot can improve its visual recognition after deployment by learning from ordinary conversations with non-expert users, using clarification questions, retrieval of past corrections, and periodic fine-tuning.","keywords":["interactive learning","multimodal large language model","chain of question","dual-modality retrieval","LoRA fine-tuning","human-robot dialogue","visual perception","continuous improvement"],"falsifier":"Have a user deliberately supply a wrong label during a round-1 dialogue; if the round-2 retrieval and round-3 fine-tuned model confidently repeat that wrong label, the loop is encoding user error as ground truth.","tokens_in":7796,"feed_emoji":"🤖","tokens_out":7171,"duration_ms":76608,"temperature":0.7,"pith_summary":"iLearnRobot is a framework for letting a deployed robot get better at seeing and recognizing objects after it has left the lab. The paper's central claim is that a multimodal large language model can learn from natural dialogues with non-expert users: the robot asks clarifying questions, retrieves similar past interaction events to avoid repeating corrected mistakes, and periodically fine-tunes on the accumulated dialogues. In a study with ten similar medicine bottles, average answer accuracy rose from 0.288 in the initial round to 0.616 with retrieval and 0.716 after fine-tuning, with user satisfaction scores moving in the same direction. The point of the work is that robot perception can keep improving in the field without a dedicated team of expert annotators.","feed_headline":"Accuracy climbs from 29% to 72% as robot learns from chats","feed_subtitle":"A robot asks questions, remembers corrections, and fine-tunes itself to recognize novel objects.","key_machinery":"Three mechanisms carry the argument. The chain-of-question module prompts the MLLM to keep asking clarifying questions until it can state the user's intent, distilling the dialogue into a single clear question and a bounding box. The dual-modality retrieval module encodes the cropped subject region with a CLIP image encoder and the distilled question with a CLIP text encoder, then retrieves a past event when both cosine similarities clear fixed thresholds, feeding the stored answer into the response prompt. The model update module fine-tunes LLaVA-NeXT with LoRA (low-rank adaptation) on the accumulated events; the ablation that unfreezes the visual encoder is the part that lets the model distinguish objects with only tiny pixel differences.","core_discovery":"The paper claims that everyday human-robot dialogue can be turned into a self-improvement loop for visual perception. When a user asks about an object, the chain-of-question module first resolves the true intent; the dual-modality retrieval module then checks the history event database for a past interaction matching both the cropped object region and the distilled question, and if found, uses the stored correct answer as a reference. Every interaction is distilled into a question with clear intent, a subject bounding box, and the user-supplied correct answer, stored as embeddings, image, question, and answer. Once enough events accumulate, the robot fine-tunes LLaVA-NeXT with LoRA, and the authors' ablation shows that also updating the visual encoder, rather than freezing it, gives the best accuracy and lowest variance. The measured result is that accuracy on ten novel medicine bottles goes from 0.288 to 0.616 with retrieval, then to 0.716 after fine-tuning.","pith_inferences":["Because user corrections are stored without verification, an implication the paper leaves implicit is that the system's long-run accuracy is capped by the reliability of non-expert labels; a wrong label will be retrieved and then reinforced by fine-tuning.","A natural next experiment would be to test the same pipeline on a broader set of visually similar objects and measure whether the round-3 gain transfers to objects never discussed in any dialogue.","The paper's ablation suggests a testable split: fine-tune with only the visual encoder updated versus only the LLM updated, to attribute the round-3 improvement to perception versus language.","The retrieval threshold is a free parameter; a useful extension would map accuracy and variance against threshold values to find where retrieval starts and stops helping."],"forward_implications":["Retrieval alone, without any weight update, can lift a robot's answer accuracy substantially between model updates, so users get corrected behavior immediately after a mistake is corrected once.","Periodic fine-tuning on distilled dialogues raises the no-retrieval baseline, meaning the robot's intrinsic perception improves and it no longer depends on the database for every answer.","Unfreezing the visual encoder during fine-tuning is what allows the model to separate visually similar novel objects, so future fine-tuning pipelines for novel perception should not assume frozen vision towers.","The same interactive loop can be applied to any object attribute a user can name, such as color, usage, breed, or shape, not just the name of a medicine bottle."],"supporting_citations":[{"why":"Supplies the base multimodal large language model whose reasoning, OCR, and world knowledge drive the robot's responses and dialogue distillation.","marker":"[7]"},{"why":"Provides the CLIP image and text encoders used to embed subject regions and questions for dual-modality retrieval.","marker":"[12]"},{"why":"Supplies LoRA, the parameter-efficient fine-tuning method used for periodic model updates.","marker":"[4]"},{"why":"Inspires the idea of distilling generalizable knowledge from language corrections and retrieving it for future interactions.","marker":"[17]"},{"why":"Provides the original visual-instruction-tuning recipe whose fine-tuning script the model update step modifies.","marker":"[8]"},{"why":"Motivates the interactive-learning approach in which an agent asks users for help to improve grounding.","marker":"[1]"}],"fun_headline_variants":["Chats train robot: object accuracy jumps from 29% to 72%","Robot learns from dialogues, boosts recognition to 72%","Interactive learning lifts robot vision from 29% to 72%","Robot's everyday chats sharpen its object spotting to 72%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole loop treats whatever answer a user gives during conversation as the correct answer, with no check on whether the user is right.","fun_headline_variants_meta":{"raw":{"variants":["Chats train robot: object accuracy jumps from 29% to 72%","Robot learns from dialogues, boosts recognition to 72%","Interactive learning lifts robot vision from 29% to 72%","Robot's everyday chats sharpen its object spotting to 72%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1396,"prompt_tokens":884,"completion_tokens":512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":437}},"tokens_in":500,"tokens_out":512,"duration_ms":6368,"temperature":1.0,"reasoning_tokens":437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:57:03.813078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a user deliberately supply a wrong label during a round-1 dialogue; if the round-2 retrieval and round-3 fine-tuned model confidently repeat that wrong label, the loop is encoding user error as ground truth.","supporting_citations":[{"cited_title":"Advances in neural information processing systems36, 2434–2447 (2024)","cited_arxiv_id":null,"evidence_quote":"Provides the original visual-instruction-tuning recipe whose fine-tuning script the model update step modifies."},{"cited_title":"In: Proceedings of the AAAI conference on artificial intelligence","cited_arxiv_id":null,"evidence_quote":"Motivates the interactive-learning approach in which an agent asks users for help to improve grounding."}],"review_version":1}