{"id":"57d95944-1383-4a8f-81f7-4d5327a45aef","arxiv_id":"2607.29172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Closed-loop fine-tuning through a supervised-data API alone — no weights, gradients, or losses — lifts a closed-weight humanoid VLA to near-perfect success on three contact-rich tasks after two self-improvement cycles.","lead":"This paper shows that a closed-weight robot model can be improved through a managed fine-tuning API that hides the model's internals: real-robot trial feedback is converted into training data, pushing a Gemini humanoid policy to near-perfect success on three manipulation tasks. The result matters because it offers a path to self-improvement for users of proprietary robot models who cannot access weights or gradients.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported near-perfect success rates are measured on the same fixed configurations used to generate training data; the gains may be configuration memorization rather than task mastery.","rationale":"I read the paper in good faith and the CLIFT mechanism is coherent: encoding reward feedback as advantage-conditioned supervised data and pushing it through a managed SFT API is a plausible route for closed-loop improvement, and the open-weight comparison with π0.5 is a useful controlled contrast. The most load-bearing weakness, however, is not the unvalidated reward model—it is the evaluation protocol. The success metric is the actual binary task outcome, so even if R_theta is imperfect, the reported success rates are direct measurements. What those measurements lack is any separation between training and evaluation distributions: the fixed configurations are reused for both. That single flaw undermines the generalization claims ('specialist', 'task mastery', emergent behaviors beyond demonstrations) and weakens the GROD-vs-π0.5 ceiling comparison, since models can differ in memorization capacity. The reward-model concern is real but secondary: it affects transfer to novel task conditions, whereas the evaluation leak affects the validity of the reported numbers themselves. Thus I recommend keeping the verdict CONDITIONAL, contingent on held-out evaluation; if held-out success is comparable, the central claim becomes much stronger.","tokens_in":15482,"tokens_out":4828,"duration_ms":51355,"concrete_test":"Evaluate the final CLIFT-GROD policy (after cycle 2) on a held-out set of configurations—for example, 20 novel object poses per task never used in any demonstration or flywheel rollout—using the same success criteria and deployment settings. Compare held-out success rates with the reported 100%/98%/96%. If the held-out rates are substantially lower (e.g., by more than 10 percentage points), the near-perfect results are at least partly configuration memorization. As a stronger check, re-run one task's CLIFT loop with per-rollout randomized object poses and evaluate on a disjoint held-out pose set.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—CLIFT pushes GROD to ~100% success—is supported only by evaluation on the fixed configuration suite that also supplies the relabeled training data. Sec. 4.1 states 'deployment and evaluation coincide: each cycle deploys the current policy for 100 rollouts per task under a fixed evaluation suite, and these same rollouts... form the training data for the next cycle.' App. E.2 confirms the same configurations are 'reused identically for every model, every variant, and every flywheel cycle,' with each rollout reset to the same layout. Thus every policy is tested on initial scene layouts it has already seen in training (in rollouts of earlier cycles). Improvements from cycle to cycle can be explained by memorizing these specific layouts, not by acquiring a transferable closed-loop skill. The claim that CLIFT yields a 'humanoid specialist' or 'task mastery' therefore goes beyond the evidence. Even if R_theta were perfectly calibrated, this evaluation leak would remain; conversely, the reported binary success rates do not depend on R_theta's calibration, so the reward-model concern is secondary to this one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"CLIFT is a closed-loop fine-tuning procedure for closed-weight robot VLAs exposed only through a managed SFT API. Starting from a demonstration-trained policy π0, it deploys the policy on hardware, scores each rollout with a preference-calibrated dense reward model R_θ, labels action chunks with binary advantage tokens obtained by comparing discounted chunk returns against chunks retrieved from visually similar states, and fine-tunes the base model through the API on demonstrations plus relabeled rollouts. The loop repeats. The paper evaluates on a Unitree G1 humanoid on three tabletop but whole-body tasks: box packing, cup insertion, and bimanual plate handover. Across two flywheel cycles, the dense advantage-conditioned variant raises GROD from 93/70/53% to 100/98/96% success on the three tasks, while the same pipeline applied to open-weight π0.5 reaches only 76/56/30%. An invasive FiLM-conditioned variant of π0.5 does not close the gap. The paper concludes that API-only access can support closed-loop policy improvement and that the pretrained prior, rather than access level, sets the adaptation ceiling.","tokens_in":15646,"tokens_out":7672,"duration_ms":70632,"significance":"If the headline results are reliable, this is a significant empirical contribution: it identifies a practically important access regime (managed SFT API) and demonstrates a non-invasive mechanism for closed-loop improvement that requires no weights, gradients, or likelihoods, with carefully matched controls between models (same demonstrations, reward model, rollout budget, deployment configuration). The retrieval-based advantage labeling is a plausible way to normalize returns by state difficulty without learning a value function. The paper also discloses its limitations. However, the significance is conditional on two load-bearing points: the evaluation must be free of train/eval contamination, and the reward model must be shown to encode genuine task progress. As written, neither point is adequately established.","major_comments":[{"comment":"The evaluation protocol suffers from train/eval contamination. Sec. 4.1 states that 'deployment and evaluation coincide: each cycle deploys the current policy for 100 rollouts per task under a fixed evaluation suite, and these same rollouts both define its success rate and, once relabeled, form the training data for the next cycle.' App. E.2 confirms that the same fixed configurations are 'reused identically for every model, every variant, and every flywheel cycle.' Consequently, the success rates reported for π1 and π2 are computed on initial scene layouts whose rollouts from earlier cycles are in the cumulative training set D_k. The observed improvement can therefore be explained by memorizing these specific layouts rather than by acquiring a transferable closed-loop skill; the claims of 'task mastery' and 'humanoid specialist' go beyond what this protocol demonstrates. Please add held","section":"Sec. 4.1 and App. E.2"},{"comment":"The reward model R_θ is the sole source of the learning signal (advantage tokens in Eq. (5) and the top-30% episode selection), yet the paper provides no validation of R_θ against held-out human preferences or task outcomes. It is trained on roughly 200 rollouts with 100 pairwise preference labels, with VLM candidate sequences selected to match those preferences; no accuracy, correlation, or ablation is reported. If R_θ rewards surface progress or smooth motion rather than true task progression, CLIFT would amplify the wrong behavior and the reported success gains would not transfer. Please report (i) R_θ's agreement with held-out human pairwise preferences and with binary task success, and (ii) an ablation replacing the dense reward with task success/failure or a no-reward/random-label baseline, to demonstrate that the dense reward signal is what drives the improvement.","section":"Sec. 3.1, App. B, Eq. (5)"}],"minor_comments":[{"comment":"Success rates are point estimates over 100 trials with no confidence intervals or significance tests. Report binomial confidence intervals, and clarify how the two independent training runs were used (the text says the run with lowest held-out demonstration loss was selected, which introduces selection bias).","section":"Fig. 5 / Sec. 4.2"},{"comment":"The method relies on several global hyperparameters with no sensitivity analysis: retrieval threshold δ, look-ahead horizon H, discount factor γ, the top-30% percentile, and the classifier-free guidance scale β. Please report sensitivity to at least δ, H, and the percentile for one task, since the method's robustness depends on these choices.","section":"Sec. 3.2 / App. C"},{"comment":"The candidate-selection procedure for choosing one reward sequence per rollout to maximize agreement across the 100 human preference pairs is described only verbally. Please specify the optimization method (e.g., greedy, ILP) and its computational feasibility, as the number of combinations grows exponentially in K and the number of rollouts.","section":"App. B"},{"comment":"The 'emergent behaviors' are presented qualitatively. Quantify them, for example by measuring the frequency of reorientation/retry events in evaluation rollouts, to support the claim that these behaviors are acquired and not cherry-picked.","section":"Fig. 6 / Sec. 4.2"},{"comment":"The detailed benchmarking is referenced to an interactive server on the project website; provide a static archival artifact (e.g., a downloadable dataset or a DOI-linked document) for reproducibility and long-term access.","section":"Sec. 4.2"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the evaluation overlap between training rollouts and the fixed evaluation suite. If the authors cannot add held-out evaluation, the headline claims of near-perfect success and task mastery should be substantially reframed. The reward-model validation is the second key issue; without it, the mechanism's contribution to the reported gains is not established. The paper is otherwise interesting and the controlled cross-model comparison is a strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: CLIFT is a clever, well-specified way to do closed-loop policy improvement for a closed-weight VLA using only a managed SFT API — that's genuinely new and worth knowing. But the headline results are measured on the same fixed configurations that generated the training data, so the 'near-perfect success' and 'humanoid specialist' framing goes beyond what the evidence supports.\n\nWhat's new: the retrieval-based advantage estimator (compare a chunk's discounted return against chunks from visually similar states, Eq. 3–5) is a smart replacement for a value model, and the whole pipeline — score rollouts with a preference-calibrated dense reward, convert to SFT tuples with advantage tokens, condition on positive token at inference — fits entirely inside an API-only access regime. The controlled comparison against pi0.5 with the same rollouts, reward model, and budget is the right experimental design for asking whether the pretrained prior or the access regime sets the ceiling. The emergent behavior examples are suggestive. The paper also discloses its protocol clearly; App. E.2 states outright that configurations are reused identically across every model and cycle.\n\nThe soft spots, in order of severity. First, the evaluation leakage: because deployment and evaluation coincide, every policy after cycle 1 has trained on rollouts from the exact initial configurations it is then tested on. The improvement from SFT baseline to cycle 1 could be pure memorization of those layouts — you don't need the reward model or advantage labels to get that, just more supervised data from the test distribution. The dense-vs-episode comparison suggests the labeling matters, but the absolute success rates cannot be read as task mastery. A held-out set of configurations, even just a few, would have settled this. Second, the reward model R_theta is load-bearing and never validated against held-out human preferences or task outcomes; all advantage labels come from it. That's a real gap, though it's secondary to the evaluation leak because the binary success criteria are external. Third, no confidence intervals, and some prose numbers in Sec. 4.2 (the 84%/88% vs 46%) don't match the figure or the summary. Minor. Code and data are not available, so replication is impossible as it stands.\n\nWho's this for? Anyone working on adapting closed-weight robot foundation models, or on humanoid closed-loop training. It's worth arguing with. It deserves a serious referee — but my recommendation would be major revision, mainly to fix the evaluation protocol or reframe the claims as specialization to a fixed deployment setting, not task mastery.","headline":"Clever API-only closed-loop fine-tuning method, but the evaluation reuses the training configurations, so the near-perfect success rates are likely inflated and the 'specialist' claims outrun the evidence.","tokens_in":16277,"tokens_out":4739,"would_cite":true,"duration_ms":44178,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A closed-weight robot model, accessible only through a managed fine-tuning API, can be driven to near-perfect task success by a closed-loop flywheel that converts deployment-time reward feedback into advantage-conditioned supervised data, w","keywords":["robot foundation models","managed SFT API","closed-loop fine-tuning","humanoid manipulation","advantage conditioning","reward model","deployment-time learning","closed-weight adaptation"],"falsifier":"Train R_theta as described, but before running CLIFT, validate its per-step rewards against a held-out set of human preferences and against ground-truth task outcomes. Then run two flywheel cycles using an oracle reward that is perfectly calibrated to task outcomes: if the oracle-driven loop substantially outperforms the R_theta-driven loop on the same tasks, the reward model is the bottleneck; if the two match, R_theta is adequate. Alternatively, run CLIFT with R_theta deliberately corrupted (e.g., rewarding smooth motion regardless of task progress) and show success rates drop—this would set","tokens_in":15238,"feed_emoji":"🤖","tokens_out":8184,"duration_ms":65484,"temperature":0.7,"pith_summary":"The paper asks whether a robot foundation model that can only be adapted by submitting training data to a managed supervised fine-tuning API—no weights, gradients, or losses visible—can still improve closed-loop on a real humanoid. It introduces CLIFT, a flywheel that deploys the current policy, scores each rollout with a preference-calibrated dense reward model, labels action chunks with a binary advantage token via retrieval against visually similar states, and fine-tunes on those chunks through the API. After two cycles, the method takes the model from 93% to 100% on box packing, 70% to 98% on cup insertion, and 53% to 96% on bimanual plate handover. The same pipeline improves an open-weight VLA, but to a much lower ceiling, suggesting the pretrained prior—not the access regime—sets the adaptation limit. A sympathetic reader would care because it shows closed-loop reinforcement-style improvement is possible inside an API-only regime, turning black-box adaptation interfaces into practical policy-improvement tools.","feed_headline":"Robot hits near-perfect success with no model access","feed_subtitle":"Two flywheel cycles lift an API-only humanoid policy from 53–93% to 96–100% on three contact-rich tasks.","key_machinery":"The flywheel's central mechanism is the advantage-conditioned SFT tuple: an observation, language instruction, action chunk, and a binary token that encodes whether the chunk's return ranks in the top 30% of chunks from visually similar states. The token is produced by a retrieval-based advantage estimator that compares each chunk's discounted return over an 1.8-second look-ahead window against peers found with a frozen vision encoder, so the threshold auto-adapts to state difficulty. The reward model R_theta that feeds these returns is itself a VLM fine-tuned on per-step labels distilled from VLM candidates that best matched 100 human pairwise preferences. This pipeline converts deployment-","core_discovery":"The central claim is that reinforcement feedback can be encoded directly into supervised training data, making closed-loop policy improvement compatible with a managed SFT API. CLIFT works by scoring deployment rollouts with a dense reward model that is calibrated by 100 human pairwise preferences selecting among VLM-generated per-step reward candidates, then distilling those into a fixed reward model. Each action chunk is assigned a binary advantage token by comparing its discounted return against chunks retrieved from visually similar starting states (top-30% threshold). The next SFT dataset includes these relabeled chunks, and conditioning on the positive token at deployment steers the po","pith_inferences":["A testable extension is to validate R_theta against held-out human judgments and task outcomes before running the flywheel; if a corrupted reward model (e.g., rewarding smooth motion regardless of task progress) eliminates the gains, that would confirm the reward model is the load-bearing component.","The retrieval-based advantage estimator makes a strong assumption that cosine similarity of frozen vision embeddings captures state equivalence; a sensitivity study swapping the encoder or using task-specific state features would clarify how robust the credit assignment is.","Because the paper's conclusion that the pretrained prior sets the ceiling rests on a single invasive baseline, a broader comparison across several open-weight models of comparable scale would strengthen or qualify that claim.","A natural extension the paper flags is coupling CLIFT with a control-aware world model to reduce the number of real-robot rollouts per cycle; such a world model would also make the approach more practical for safety-sensitive deployment."],"forward_implications":["If true, managed SFT APIs can serve as general-purpose closed-loop improvement interfaces for robot foundation models, not just one-shot customization.","The result implies that the ceiling of API-adaptation is set by the pretrained prior, so securing API access to the strongest closed-weight models is worthwhile even without internal access.","Chunk-level advantage labeling from failed rollouts converts failure into training signal, which is what closes the gap on the hardest task (bimanual handover 53% to 96% vs. episode-selection ~84%).","The same non-invasive pipeline transfers across models and access regimes, improving an open-weight VLA as well, so it is not specific to one API.","Conditioning on a positive advantage token at inference acts as a weak form of guidance, and the emergent retry and pre-grasp reorientation behaviors suggest closed-loop practice can compose new behavior beyond the demonstration distribution."],"fun_headline_variants":["No weights, no problem: humanoid hits near-perfect skill","API-only humanoid learns to nail tasks without model access","Black-box robot tuning hits 96-100% success via CLIFT","Closed-loop flywheel: humanoid improves without weight access","Two flywheel cycles push API-only robot to near-perfect"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire learning signal flows through the dense reward model R_theta, which is trained on about 200 rollouts whose per-frame labels come from VLM-generated candidates filtered by only 100 human pairwise preferences, and is never validated against held-out human judgments or task outcomes; if those rewards reward surface progress or clean-looking motion rather than true task progression, CLIFT amplifies the wrong behavior and the reported gains would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["No weights, no problem: humanoid hits near-perfect skill","API-only humanoid learns to nail tasks without model access","Black-box robot tuning hits 96-100% success via CLIFT","Closed-loop flywheel: humanoid improves without weight access","Two flywheel cycles push API-only robot to near-perfect"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2733,"prompt_tokens":844,"completion_tokens":1889,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1802}},"tokens_in":588,"tokens_out":1889,"duration_ms":12233,"temperature":1.0,"reasoning_tokens":1802,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T12:18:39.700868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train R_theta as described, but before running CLIFT, validate its per-step rewards against a held-out set of human preferences and against ground-truth task outcomes. Then run two flywheel cycles using an oracle reward that is perfectly calibrated to task outcomes: if the oracle-driven loop substantially outperforms the R_theta-driven loop on the same tasks, the reward model is the bottleneck; if the two match, R_theta is adequate. Alternatively, run CLIFT with R_theta deliberately corrupted (e.g., rewarding smooth motion regardless of task progress) and show success rates drop—this would set","supporting_citations":[],"review_version":1}