{"id":"d56ff240-f7e0-4f7a-84a9-d494ac83a3c0","arxiv_id":"2508.05838","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A PPO agent with SAM and YOLOv5 perception in the AI2-THOR simulator improves object-interaction success, reward, and navigation efficiency over a raw-RGB baseline.","lead":"Researchers combined SAM and YOLOv5 vision models with a PPO reinforcement learning agent inside the AI2-THOR simulator. The enhanced agent gets higher rewards, finds objects more often, and navigates more directly than a baseline using only raw camera images.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Under-specified observation encoding and unreported reward weights leave the perception gain unidentifiable; a fixed-feature ablation is needed.","rationale":"I read the paper as claiming that the specific integration of SAM and YOLOv5 into the PPO observation pipeline is what drives the large measured gains. The numbers in Table I are internally consistent, and I am not alleging fabrication or asking for proof of the sim-to-real domain gap. My concern is that the experimental protocol as written does not rule out two much more mundane explanations: different reward shaping for the two agents, and different observation dimensionality/capacity for the two policies. The reward weights α, β, γ are not reported, and the baseline description does not explicitly say the baseline used the same weight values. If the perception agent's weights were tuned to optimize its behavior, the success-rate and reward differences could be partly an artifact of tuning rather than of the perception models. Likewise, the perception pipeline's output representation is described only as 'encoded using a CNN'; without knowing whether the CNN input is raw RGB plus masks, masks alone, rendered boxes, or a higher-dimensional feature stack, the comparison to a raw-RGB-only baseline is not controlled. A randomly initialized CNN with the same input/output dimensions is the natural control; without it, the paper has not shown that pretrained SAM/YOLO features—rather than simply more/better-shaped input features—are the cause. This does not invalidate the possibility that the claim is true, but it makes the central attribution underdetermined. The reader's weakest assumption about synthetic/natural domain gap is related but not the same; I would place the risk in the missing controls, not in the perceptual generalization of SAM/YOLO, which is broadly known to be reasonable. Thus I keep the CONDITIONAL verdict and flag that the authors must supply these details and the ablation before the claim can be accepted.","tokens_in":6785,"tokens_out":4833,"duration_ms":61521,"concrete_test":"Ask the authors to release (a) the exact α, β, γ values used for both agents, and (b) the exact tensor layout of φ(s_t) (input channels, resolution, whether masks/boxes are rendered or stacked, encoder parameters). Then add one ablation: replace the SAM/YOLO pipeline with a randomly initialized CNN encoder of identical input dimensionality and parameter count, trained from scratch, and re-run all four FloorPlans with the same seeds. If this ablation closes the reported 52.5% success-rate gap, the claim that foundation-model pre-training is responsible fails. If the gap persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central attribution—that SAM/YOLO features cause the improvement—is not identifiable from the reported experiments. Equation (1) defines r_t = α·Δd_t + β·s_t − γ·c_t with α, β, γ 'tuned empirically,' but no values are given, and the baseline section says only that the same algorithm/hyperparameters were used, not that the same reward weights were used. If the reward weights were tuned separately for the perception agent, part or all of the 68.2% reward gap and possibly the success-rate gap could come from reward shaping rather than perception. Second, the observation encoding is underspecified: Section III-B-3 says the combined outputs (B, C, M) 'are encoded using a CNN' to form φ(s_t), but the paper never states the channels/resolution of the CNN input, whether raw RGB is concatenated with masks, whether boxes/classes are drawn into an image, or how many parameters the encoder has. The baseline gets raw RGB; a larger/richer input tensor alone could explain the gains. No ablation separates 'extra CNN capacity or feature channels' from 'SAM/YOLO pre-trained knowledge.' These are not deep theoretical doubts; they are missing controls that leave the central claim underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes combining the vision foundation models YOLOv5 and SAM with a PPO agent in the AI2-THOR kitchen environments. The perception pipeline detects objects, generates segmentation masks, and encodes these outputs with a CNN to form the RL observation; the reward function combines distance-to-target change, success, and collision penalties. The authors report large improvements over a raw-RGB baseline in success rate (73.5±2.1 vs. 48.2±4.5), cumulative reward (136.4±5.6 vs. 81.1±9.3), navigation efficiency (82.1±1.9 vs. 61.7±3.1), and interaction efficiency. The abstract and text attribute these gains to the integrated perception models.","tokens_in":7091,"tokens_out":2785,"duration_ms":32084,"significance":"If the reported gains are genuinely caused by the YOLOv5/SAM perception pipeline and are reproducible, this would be a useful demonstration that frozen vision foundation models can be plugged into an RL agent for object-interaction tasks in simulation. The internal arithmetic of Table I is consistent with the claimed percentages, and the use of five random seeds is a positive sign. However, the experimental design as reported does not identify the source of the improvement: the reward weights are not disclosed, the observation encoding is underspecified, and no ablation separates the effect of pre-trained perception from the effect of a richer input representation or reward tuning. These omissions currently make the central claim underdetermined and limit the paper's contribution.","major_comments":[{"comment":"The reward is r_t = α·Δd_t + β·s_t − γ·c_t with α, β, γ described only as 'tuned empirically'. §IV-B states the baseline uses the same algorithm and hyperparameters, but it does not state whether the baseline uses the same reward weights. If the perception-enhanced agent's weights were tuned separately, the 68.2% cumulative-reward gap—and possibly the success-rate gain—could reflect reward shaping rather than perception. Report the numerical values of α, β, γ, state explicitly that identical reward weights were used for both agents, and ideally include a sensitivity analysis.","section":"§III-C, Eq. (1)"},{"comment":"The observation encoding is underspecified. The combined outputs (B, C, M) are 'encoded using a CNN' to produce φ(s_t), but the paper never states the input tensor's channels/resolution, whether raw RGB is concatenated, how bounding boxes and class labels are rasterized, or the CNN architecture and parameter count. The baseline receives raw RGB, so a larger input tensor or a higher-capacity encoder alone could explain the gains. A fixed-feature ablation—e.g., a randomly initialized encoder fed with the same tensor layout, or masks from a trivial detector—is needed to isolate the contribution of pre-trained SAM/YOLO knowledge.","section":"§III-B-3"},{"comment":"The claims throughout the paper refer to improvements 'across all four environments', but Table I reports only environment-averaged aggregates. No per-floor-plan results are given, and no statistical significance tests are reported. Without per-environment means/standard deviations or a paired test across seeds, the reader cannot tell whether the improvement is consistent or dominated by one layout. Please provide a per-environment table and appropriate significance testing.","section":"§IV-A, Table I"},{"comment":"The paper mentions that training curves were recorded and that detailed logs were collected, but none are shown; no code or data are provided. Combined with the unspecified encoder and reward weights, the reported numbers cannot be independently reproduced or audited. At minimum, show the learning curves for both agents and make the training/evaluation code or detailed logs available, or provide a clear reproducibility statement explaining why this is not possible.","section":"§IV-D, §V"}],"minor_comments":[{"comment":"Typo: 'Ph.D. Candidiate' should be 'Ph.D. Candidate'.","section":"§I"},{"comment":"Interaction Efficiency is reported in Table I but not defined in the abstract or discussed beyond one sentence; clarify whether the mean is over successful episodes only.","section":"§IV-E"},{"comment":"The claim that 'the acceptable frame rate achieved suggests feasibility for real-time applications' is not supported by any reported frame-rate or latency numbers. Either provide measurements or soften the claim.","section":"§VI-C"},{"comment":"Figure 3 is qualitative and does not show quantitative interaction outcomes; the text asserts robustness based on a few example images. Refer to the table statistics instead of generalizing from examples.","section":"§V-B"},{"comment":"The related work section cites general references but does not actually compare the proposed method to prior perception-plus-RL pipelines such as object-goal navigation methods. A brief comparative discussion would strengthen the positioning.","section":"§II"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is on point: the manuscript's central attribution is not identifiable from the reported experiments because reward weights and the observation encoding are unspecified, and no fixed-feature or random-perception ablation is included. I do not see this as a fundamental flaw in the idea, but it is a load-bearing gap that must be addressed before acceptance. If the authors can supply the exact reward weights, the exact tensor specification and CNN architecture, and a same-representation ablation, the paper could become a solid empirical contribution. The absence of code/data and per-environment results also needs attention."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small empirical paper that does one useful thing—it shows that feeding SAM+YOLOv5 outputs into a PPO agent in AI2-THOR can beat a raw-RGB baseline across four kitchen scenes. The numbers are internally consistent, the claims don't overreach beyond the reported experiments, and they are honest that the models are frozen, not fine-tuned. It won't change how we think about embodied RL; the general point that semantic perception helps goal-driven navigation is already established in the literature they cite (Chaplot, Mousavian). What's actually new is just the specific instantiation with SAM and YOLOv5, and that has modest value.\n\nThe soft spots are real but not fatal. The stress-test note is on target: the observation encoding is under-specified. They say the combined outputs (B, C, M) are encoded using a CNN, but never state whether the masks and bounding boxes are rendered into an image, whether raw RGB is concatenated, what resolution or channel count the CNN takes, or how many parameters it has. With a raw-RGB baseline, a larger or richer input tensor alone can explain part of the gain. There is no ablation to separate 'more feature channels' from 'pre-trained SAM/YOLO knowledge.'\n\nThe reward weights are also unreported. They say alpha, beta, gamma were 'tuned empirically,' with no values, and the baseline is described as using the same algorithm and hyperparameters but not explicitly the same reward weights. If the perception agent was tuned separately on the shaping weights, part of the reward gap—and possibly the success-rate gap—could come from better reward shaping rather than better perception. This is a standard missing-control problem, not a sign of fabrication. No code, no data, and no external artifacts are provided, so independent verification is not possible.\n\nOn the writer's side: the thinking is coherent and the presentation is honest enough. The limitations section acknowledges generalization is unexplored. The citation pattern is normal; they appropriately cite the prior semantic-exploration work.\n\nVerdict for a journal: this is a conditional-accept or reject-with-encouragement case. The central claim is plausible but not identifiable from the reported experiments. It deserves a serious referee only if the venue is willing to ask for revisions that include the missing details: exact reward weights, a precise description of the input encoding, and ideally a fixed-feature control or an ablation with a non-semantic encoder of comparable capacity. If the authors provide those, this becomes a solid incremental result.\n\nMy recommendation: send it to peer review with instructions to require those specifics before acceptance. It's not a desk reject—the work is coherent and the empirical claim is concrete—but it's not publishable in its current form.","headline":"Plausible integration result in simulation, but missing specification of the perception encoding and reward weights leaves the core attribution underdetermined; still worth a referee's time as a reproducibility-focused artifact.","tokens_in":7504,"tokens_out":1358,"would_cite":false,"duration_ms":12974,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding YOLOv5 and SAM perception to a PPO agent lifts object-interaction success by 52.5% in AI2-THOR kitchens.","keywords":["reinforcement learning","vision foundation models","Segment Anything Model","YOLOv5","object interaction","AI2-THOR","proximal policy optimization","perception integration"],"falsifier":"Run the perception-enhanced agent with SAM masks replaced by random rectangles of the same size and location as YOLO boxes while keeping everything else fixed; if success rate and reward stay at reported levels, segmentation detail is not driving the gain. Alternatively, measure YOLOv5's object-detection mAP directly on AI2-THOR frames; if it is low, the proposed mechanism cannot explain the improvement.","tokens_in":6726,"feed_emoji":"🤖","tokens_out":6512,"duration_ms":63915,"temperature":0.7,"pith_summary":"This paper tries to establish that a reinforcement-learning agent can become substantially better at object interaction when its observations pass through two frozen vision models—YOLOv5 for detection and SAM for segmentation—rather than raw pixels. In AI2-THOR kitchen scenes, the authors report a 52.5% higher success rate, 68.2% higher cumulative reward, 33.1% better navigation efficiency, and fewer interaction attempts compared to a same-policy baseline using raw RGB images. The result matters because it suggests a low-cost route to giving RL agents object-level scene understanding: the large vision models are not fine-tuned, so the gain comes from a fixed preprocessing step the policy learns to exploit. The paper's evidence is limited to four simulated kitchens, and it explicitly leaves generalization to new settings untested.","feed_headline":"Vision models boost RL object-interaction success by 52.5%","feed_subtitle":"In AI2-THOR kitchens, a PPO agent with frozen detection and segmentation beats raw-RGB baseline on reward and navigation too.","key_machinery":"The load-bearing mechanism is the perception-stack encoder. YOLOv5 converts each raw RGB frame into a set of bounding boxes $B=\\{b_i\\}$ and class labels $C=\\{c_i\\}$; SAM takes those boxes as prompts and produces segmentation masks $M=\\{m_i\\}$; a CNN then encodes $(B,C,M)$ into a compact feature representation $\\phi(s_t)$ that serves as the PPO policy's state. The policy is trained with the clipped surrogate objective of PPO. This encoded representation is what separates the enhanced agent from the baseline: the same policy network, reward function, and optimizer, but with object-level semantic and spatial information substituted for raw pixels.","core_discovery":"Across four AI2-THOR kitchen layouts, an agent whose observations are encoded by YOLOv5 (bounding boxes and class labels) and SAM (segmentation masks prompted by those boxes) before a CNN encoder outperforms a PPO agent operating on raw RGB frames. Averaged over five seeded runs, the perception-enhanced agent reaches a success rate of 73.5±2.1% versus 48.2±4.5%, an average cumulative reward of 136.4±5.6 versus 81.1±9.3, navigation efficiency of 82.1±1.9% versus 61.7±3.1%, and interaction efficiency of 1.2±0.1 versus 2.1±0.3 attempts. The authors interpret these numbers as evidence that the frozen perception stack lets the policy disambiguate target objects, plan shorter paths, and position i","pith_inferences":["Inference: The navigation-efficiency jump indicates the perception stack acts partly as a spatial prior, so a natural controlled experiment would ablate masks-only versus boxes-only to identify which modality drives path planning.","Inference: Since the vision models are frozen and not specific to AI2-THOR, the same plug-in could be tried with other RL algorithms or other interactive simulators; that extension is not in the paper.","Inference: The paper's own limitation statement—generalization to new settings is untested—leaves open whether the 52.5% gain survives domain shift; a next test is evaluating on held-out kitchens or real robot images."],"forward_implications":["If the central claim holds, RL practitioners can improve object-interaction performance by prepending frozen detection and segmentation models rather than training perception end-to-end.","The 52.5% success-rate gain and 33.1% navigation-efficiency gain imply that object-level spatial cues carry much of the information PPO needs to find and reach targets in indoor scenes.","Consistency across four distinct kitchen layouts suggests the approach is not tied to one scene arrangement, though unseen layouts are not tested.","Because SAM and YOLOv5 remain frozen, the added cost is inference overhead only; RL training dynamics and hyperparameters are unchanged.","The reported drop in interaction attempts, from 2.1 to 1.2, indicates the agent also learns better when-and-where to act, not just where to go."],"supporting_citations":[{"why":"Supplies the YOLOv5 object detector that outputs bounding boxes and class labels feeding the perception stack.","marker":"[11]"},{"why":"Supplies the SAM segmentation model that turns YOLO boxes into object masks used in the agent's observation.","marker":"[12]"},{"why":"Supplies the AI2-THOR interactive 3D kitchen environments in which the agents are trained and evaluated.","marker":"[13]"},{"why":"Supplies the PPO algorithm and clipped surrogate objective used to train both the enhanced and baseline policies.","marker":"[4]"}],"fun_headline_variants":["Vision models boost RL object success by 52.5%","RL agents see better with SAM and YOLOv5","Frozen vision stack lifts RL agent success 52%","Object interaction up 52.5% with vision-augmented RL","PPO plus SAM and YOLO beats raw pixels in AI2-THOR"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole result depends on YOLOv5 and SAM giving reliable object boxes and masks for AI2-THOR's synthetic kitchen images; if the domain gap makes them unreliable, the reported gains vanish.","fun_headline_variants_meta":{"raw":{"variants":["Vision models boost RL object success by 52.5%","RL agents see better with SAM and YOLOv5","Frozen vision stack lifts RL agent success 52%","Object interaction up 52.5% with vision-augmented RL","PPO plus SAM and YOLO beats raw pixels in AI2-THOR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":2924,"prompt_tokens":720,"completion_tokens":2204,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":2123}},"tokens_in":464,"tokens_out":2204,"duration_ms":14123,"temperature":1.0,"reasoning_tokens":2123,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:06:38.055485+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the perception-enhanced agent with SAM masks replaced by random rectangles of the same size and location as YOLO boxes while keeping everything else fixed; if success rate and reward stay at reported levels, segmentation detail is not driving the gain. Alternatively, measure YOLOv5's object-detection mAP directly on AI2-THOR frames; if it is low, the proposed mechanism cannot explain the improvement.","supporting_citations":[{"cited_title":"YOLOv5 by Ultralytics,","cited_arxiv_id":null,"evidence_quote":"Supplies the YOLOv5 object detector that outputs bounding boxes and class labels feeding the perception stack."}],"review_version":1}