{"id":"10fd3755-4175-4643-a028-00a399ce59fe","arxiv_id":"2508.01131","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"COLLAGE adaptively weights demonstration subsets chosen by different feature cues, based on their ability to predict target actions, and uses these weights for importance sampling to boost few-shot imitation learning performance.","lead":"This paper presents COLLAGE, a method that improves few-shot robot learning by combining several similarity signals to pick the most useful training demonstrations from a large dataset. The approach matters because better automatic data selection could let robots learn new tasks from just a few examples, lowering the cost of collecting and curating robot data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full-text manuscript is a different paper (TWISTER), not COLLAGE; the central retrieval claim is unsupported by the submitted text.","rationale":"The reader's verdict was UNVERDICTED, with LOW confidence and a rationale that explicitly noted the full-text mismatch between the COLLAGE abstract and the TWISTER paper. That is exactly the load-bearing concern I identify. However, the reader's stated weakest_assumption is the scientific proxy assumption—that action-prediction accuracy on target demonstrations reliably weights subsets for importance sampling—rather than the mismatch itself. I agree that the proxy assumption would be a central vulnerability if the manuscript were present and checkable, but the more fundamental, prior issue is that no methods or experiments are available in the supplied full text to assess any assumption. Therefore I mark agreement as partial: the reader correctly flagged the mismatch in the rationale, but the formal weakest_assumption field points to a downstream concern that cannot yet be evaluated. My recommendation is to keep the reader's verdict unchanged (UNVERDICTED) because the manuscript as submitted does not permit a correctness assessment. This is not an accusation of misconduct; a production or versioning error could explain the mismatch, and the concrete test of retrieving the correct full text would resolve the issue. If the correct COLLAGE manuscript is found, the next step would be testing the proxy assumption, which remains a substantive scientific risk because the weight signal is computed from a very small number of target demonstrations and could overfit or be biased by uninformative features.","tokens_in":1954,"tokens_out":3320,"duration_ms":41635,"concrete_test":"Download the source archive for arXiv:2508.01131 from arXiv or the project page (https://robin-lab.cs.utexas.edu/COLLAGE) and verify that its full text contains the COLLAGE method: the subset-weighting rule based on action-prediction accuracy, the importance-sampling training objective, and the experiment tables reporting 10 simulation and 6 real-world tasks. If the correct text is present, proceed to a held-out validity check of the weighting proxy by correlating each subset's action-prediction accuracy on target demonstrations with the final policy performance after importance sampling; if the correct text is absent, the manuscript remains unverifiable and the verdict should stay UNVERDICTED.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of COLLAGE cannot be evaluated because the full text accompanying arXiv:2508.01131 is a different paper: 'Towards Bridging Review Sparsity in Recommendation with Textual Edge Graph Representation' (TWISTER), carrying identifier arXiv:2508.01128. The abstract describes an adaptive late-fusion retrieval method for few-shot imitation learning: weights are assigned to single-feature-selected subsets based on how well a policy trained on each subset predicts actions in the target demonstrations, and these weights are then used for importance sampling during policy training. The strongest quantitative claim is a 5.1% improvement in simulation across 10 tasks and a 16.6% improvement in the real world across 6 tasks. None of the sections, equations, experiments, or references needed to assess that method appear in the supplied full text. This is not a subtle mathematical weak point but a missing-evidence condition: the load-bearing assumption—that action-prediction accuracy on a few target demonstrations reliably weights subsets for importance sampling—cannot even be checked against the submitted manuscript. If the wrong text was substituted by a production error, the concern evaporates once the correct manuscript is supplied; as it stands, the strongest claim is unverifiable, not demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The submission carries the title 'COLLAGE: Adaptive Fusion-based Retrieval for Augmented Policy Learning' and an abstract describing a data-retrieval method for few-shot imitation learning. The abstract claims that COLLAGE assigns weights to feature-selected subsets by evaluating policies trained on those subsets against target demonstrations, then uses the weights for importance sampling, outperforming state-of-the-art methods by 5.1% in simulation and 16.6% on real-world DROID retrieval. However, the full text of the submission is a completely different paper, 'Towards Bridging Review Sparsity in Recommendation with Textual Edge Graph Representation' (TWISTER), concerning review imputation in recommender systems. None of the COLLAGE method, experiments, equations, or references appear in the supplied full text.","tokens_in":2157,"tokens_out":3336,"duration_ms":36691,"significance":"If the abstract's claims were backed by the full technical content, COLLAGE could be a meaningful contribution to few-shot imitation learning, particularly the idea of learning subset weights from target demonstrations rather than using fixed similarity heuristics. The reported large real-world gains on DROID would be notable. However, as submitted, the significance cannot be assessed: the central method and experiments are entirely absent from the manuscript, so there is no basis for evaluating novelty, soundness, or empirical validity. The paper therefore has no verifiable contribution in its current form.","major_comments":[{"comment":"The supplied full text is not the paper announced in the abstract and title. The abstract describes COLLAGE, an adaptive late-fusion retrieval method for few-shot imitation learning with importance sampling weights; the full text is TWISTER, a method for imputing missing reviews using textual-edge graphs in recommender systems. No section of the full text describes the COLLAGE method, its weight assignment procedure, importance sampling, or the claimed simulation and DROID experiments. Consequently, the central claim of the paper cannot be checked against the submitted evidence.","section":"Full text (entire manuscript)"},{"comment":"Even taken on its own, the abstract reports quantitative results (5.1% improvement across 10 simulated tasks and 16.6% across 6 real-world tasks) without any description of the experimental protocol, baselines, ablations, or variance. In the absence of the corresponding full text, these numbers are unverifiable and do not constitute support for the stated conclusion.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract points to a project website (https://robin-lab.cs.utexas.edu/COLLAGE) for 'More information,' but no such website or additional material is referenced in the full text.","section":"Abstract"},{"comment":"The full text carries an ACM copyright notice and a GitHub link for TWISTER (arXiv:2508.01128), which is a different submission identifier; this reinforces that the wrong manuscript text has been associated with this record.","section":"Full text, first page"}],"recommendation":"reject","confidential_remarks":"I suspect a production or submission error: the uploaded full text appears to be the body of a different arXiv paper, TWISTER (arXiv:2508.01128). I recommend that the editor verify with the authors whether the correct COLLAGE manuscript was intended. If the correct text is available, it should be submitted as a clean submission; the current record is not reviewable as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the only part of this submission I can review is the abstract, because the full text is a completely different paper (TWISTER, on review imputation in recommender systems). That's not a subtle weakness; it's a missing-evidence condition. If this is a production error, fine—but as it stands, the claims about COLLAGE cannot be checked.\n\nWhat the abstract does well: the problem is real. Single-feature retrieval heuristics for few-shot imitation learning are known to pull in irrelevant demonstrations. The proposed adaptive late fusion—weighting single-feature-selected subsets by how well a policy trained on each subset predicts actions in the target demos, then using those weights for importance sampling—is a sensible and reasonably novel way to combine cues. It's also feature-agnostic, which is a plus. The reported gains (5.1% in simulation across 10 tasks, 16.6% in the real world across 6 tasks) are moderate but not implausible.\n\nThe soft spots are hard to assess because there is no manuscript. The key assumption—that action-prediction accuracy on a handful of target demonstrations is a reliable proxy for a subset's contribution under importance sampling—could overfit to those few demos. That's a genuine risk, but I can't tell whether the authors handle it (e.g., with cross-validation or regularization) because the experimental section is absent. The abstract gives no error bars or ablations, so the numbers are unsupported.\n\nThe idea itself is worth a serious look. If the correct manuscript is supplied, I'd want to see: (1) baselines that include the same subsets but without adaptive weighting, (2) sensitivity of the weights to the number of target demos, and (3) failure cases where the proxy misleads. None of that is in front of us.\n\nMy recommendation: as submitted, this should not go to peer review—the body doesn't match the abstract. Ask the authors for the correct text. If they provide it and it matches the abstract's claims, then it deserves refereeing. From what the abstract alone shows, the work is serious and the topic is relevant to the imitation-learning community.","headline":"The abstract describes a plausible adaptive retrieval method, but the submitted full text is a different paper, so the claims can't be verified.","tokens_in":2670,"tokens_out":2929,"would_cite":false,"duration_ms":30482,"reading_group":"no","serious_thinker":"unclear","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"COLLAGE weights each retrieval cue by how well it predicts the target task, then samples training data accordingly.","keywords":["few-shot imitation learning","data retrieval","importance sampling","late fusion","action-prediction weighting","demonstration selection","robot learning","DROID dataset"],"falsifier":"A concrete check would be to run COLLAGE on tasks where the true useful subsets are known by construction, then compare its learned weights against an oracle that measures each subset's marginal contribution to final policy success; if the weights rank subsets differently from the oracle, or if uniform sampling over the union of subsets matches COLLAGE's performance, the central weighting claim is falsified.","tokens_in":1775,"feed_emoji":"🤖","tokens_out":2052,"duration_ms":29698,"temperature":0.7,"pith_summary":"COLLAGE addresses few-shot imitation learning by retrieving demonstrations from a large dataset using only a few target demonstrations. Instead of trusting any single similarity heuristic, it trains a small policy on each feature-selected subset and measures how well that policy predicts actions in the target demonstrations. The resulting prediction accuracies become weights for importance sampling: data from more useful subsets is sampled more densely, and less useful subsets are down-weighted. The paper claims this adaptive late-fusion approach beats state-of-the-art retrieval and multi-task baselines by 5.1% in simulation across 10 tasks and by 16.6% on real-world DROID retrieval across 6 tasks. A sympathetic reader would care because it turns retrieval from a fixed hand-picked distance metric into a task-adaptive, feature-agnostic data-selection mechanism.","feed_headline":"Weigh each retrieval cue by how well it predicts the task","feed_subtitle":"COLLAGE learns per-task which similarity cue matters, then samples demonstrations accordingly, beating baselines by up to 16.6%.","key_machinery":"The central mechanism is a two-stage adaptive late fusion: first, each feature-based retrieval heuristic produces a subset; then a policy is trained on each subset and its action-prediction accuracy on the few target demonstrations is converted into a sampling weight. These weights drive importance sampling during final policy training, so the data mixture is reshaped by measured task relevance rather than by fixed similarity scores. The adaptive weighting is what carries the argument: it lets the method decide per task which cues matter and how much, and it does so without any learned fusion network or task-specific tuning.","core_discovery":"The paper's central claim is that the relevance of a demonstration subset for a target task is best measured by the action-prediction performance of a policy trained on that subset, evaluated on the target demonstrations themselves. COLLAGE operationalizes this: it takes multiple subsets, each pre-selected by a single feature (appearance, shape, motion, or language similarity), trains a lightweight policy per subset, and computes how well each policy predicts actions in the target demonstrations. These validation scores become weights in an importance-sampling scheme that over-samples from subsets whose policies predict well and under-samples from those that predict poorly. The paper argues that this late fusion of multiple cues captures task-specific relevance that any single feature heuristic misses, and that the recipe is simple, feature-agnostic, and effective on both simulated benchmarks and the large-scale DROID real-world dataset.","pith_inferences":["The weight-assignment step might overfit to the few target demonstrations; a practical safeguard would be to split the target demos into weight-fitting and final-validation sets and check whether weight rankings remain stable.","Because the per-subset policies are trained only to measure predictive skill, the true contribution of a subset to the final policy under importance sampling could differ from its standalone prediction accuracy; a direct comparison would require ablating each weight term.","The idea of using action-prediction as a proxy for data relevance may transfer to other policy-learning regimes, such as offline reinforcement learning or video-conditioned control, where demonstration quality is similarly heterogeneous.","One could derive a theoretically grounded reweighting by linking the validation loss of a subset-trained policy to an upper bound on the final policy's error, making the heuristic into a principled selection rule."],"forward_implications":["If the method is right, retrieval for few-shot imitation can be made cheap and adaptive by reusing the target demonstrations as a validation set, avoiding hand-tuned similarity metrics per task.","The approach naturally extends to any number of retrieval heuristics—visual, semantic, motion, language—and could combine heterogeneous data sources where no single feature dominates.","Importance-sampled training from weighted subsets should reduce the number of detrimental demonstrations that single-feature retrieval pulls in, improving policy learning on noisy real-world datasets like DROID.","Because the weighting is computed from action-prediction loss, the same recipe could be recycled to rank data quality or to diagnose which cues a given task actually depends on.","The reported gains—5.1% in simulation and 16.6% in the real world—suggest the mechanism scales with dataset size and diversity, where single-heuristic retrieval degrades most."],"supporting_citations":[],"fun_headline_variants":["Weight demos by their policy's fit to target actions","Action-prediction scores select relevant demos","COLLAGE: let cue policies vote on sampling","Adaptive fusion: policy scores reweight demos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that how well a policy trained on a subset predicts actions on the few target demonstrations is a reliable measure of how much that subset will help the final policy during importance-sampled training.","fun_headline_variants_meta":{"raw":{"variants":["Weight demos by their policy's fit to target actions","Action-prediction scores select relevant demos","COLLAGE: let cue policies vote on sampling","Adaptive fusion: policy scores reweight demos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000259,"raw_usage":{"total_tokens":1640,"prompt_tokens":1052,"completion_tokens":588,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":668,"completion_tokens_details":{"reasoning_tokens":525}},"tokens_in":668,"tokens_out":588,"duration_ms":8355,"temperature":1.0,"reasoning_tokens":525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:47:54.381628+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to run COLLAGE on tasks where the true useful subsets are known by construction, then compare its learned weights against an oracle that measures each subset's marginal contribution to final policy success; if the weights rank subsets differently from the oracle, or if uniform sampling over the union of subsets matches COLLAGE's performance, the central weighting claim is falsified.","supporting_citations":[],"review_version":1}