{"id":"9443fffe-cb02-4e10-87f9-f9358090ff7f","arxiv_id":"2507.03984","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A chain-of-thought-based prompt generation method for OOD segmentation in road scenes, claiming consistent gains over prior methods but with evaluation caveats.","lead":"This paper uses GPT-4's chain-of-thought reasoning to generate text prompts that guide an open-vocabulary segmentation model, GroundedSAM, to detect out-of-distribution objects in road scenes. The method reports higher scores than existing OOD segmentation models, but the evaluation depends on per-image threshold tuning and a custom subset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-image oracle threshold tuning in Sec. 4.1 is the load-bearing flaw: it invalidates the SOTA comparison, and without fixed-threshold results the reported CoT gain cannot be separated from threshold overfitting.","rationale":"The reader's weakest assumption—that per-image optimization of GroundingDINO's box and text thresholds is a fair and non-inflating protocol—is exactly the load-bearing weakness of the paper. The central claim, as stated in the abstract and strongest claim, is that CoT-Segmenter consistently outperforms state-of-the-art methods. For that claim to hold, the evaluation must compare methods under equivalent conditions. Section 4.1 shows this condition is violated for the GroundedSAM rows: thresholds are individually optimized on each test image to maximize segmentation accuracy, which uses ground-truth labels during inference. The other methods, which are specialized OOD segmentation systems, are not given this oracle. Therefore the headline comparison in Table 1 conflates two effects: the quality of CoT prompts and the information leaked through per-image threshold tuning. The paper itself acknowledges prompt sensitivity in the conclusion, but it does not provide a fixed-threshold evaluation that would decouple these effects. The lack of released prompts and code further prevents an external check. I agree with the reader's conditional verdict: the idea is plausible and the CoT-vs-object comparison under identical oracle tuning hints that prompts have some value, but until the protocol is cleaned up and artifacts are shared, the central claim is not established. This does not change the reader's verdict, since the reader already identified this flaw and conditioned acceptance on fixing it.","tokens_in":7423,"tokens_out":3523,"duration_ms":42127,"concrete_test":"Re-run Table 1 under a completely fixed-threshold protocol: set GroundingDINO's box_threshold and text_threshold once per method (e.g., using standard values from the GroundedSAM repository or a validation split) and apply those same values to every image in RoadAnomaly and the Challenging subset for both GroundedSAM('object') and CoT-Segmenter, while keeping all other models unchanged. Then recompute mIoU and F1. If CoT-Segmenter's margin over 'object' shrinks to near zero, or if its advantage over S2M/RbA reverses, the claimed superiority is an artifact of per-image threshold optimization rather than CoT reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that CoT prompts make GroundedSAM outperform specialized OOD segmentation models—rests on Table 1. Section 4.1 states: 'For each image, the box threshold and text threshold values in GroundedSAM were individually optimized to maximize segmentation accuracy.' This is an oracle protocol: the test-set ground-truth masks are used to select GroundingDINO's two scalar thresholds per image, directly maximizing the same mIoU/F1 metrics that are then reported. Only the GroundedSAM rows can benefit from this per-image tuning; the compared methods (S2M, RbA, Mask2Anomaly, SynBoost, RPL+CoroCL, PEBAL) have fixed inference-time settings and receive no comparable oracle. Consequently, Table 1's comparison of CoT-Segmenter against these baselines does not establish that the reasoning-derived prompts are the cause of the improvement. The oracle could account for much or all of the margin. Even the prompt-ablation comparisons in Table 2 are affected, because every row is threshold-optimized; the relative ordering under a fixed-threshold protocol is unknown. In addition, the 'Challenging' subset is defined post hoc, and the exact CoT prompts and code are not released, so the result cannot be independently reproduced. Thus the load-bearing assumption is not merely 'threshold sensitivity' but the fairness of the evaluation itself: if per-image oracle tuning is removed, the consistent superiority claim may collapse.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoT-Segmenter, a two-stage framework that uses GPT-4 with Chain-of-Thought prompting to generate textual prompts for GroundedSAM, an open-vocabulary detector and segmenter, and applies it to out-of-distribution (OOD) detection in road scenes. The authors identify three challenging scenarios on the RoadAnomaly dataset, introduce a curated 'Challenging' subset, and report that their method outperforms existing OOD segmentation methods on both the standard and challenging subsets. They also present ablations on prompt types and CoT reasoning stages.","tokens_in":7690,"tokens_out":2378,"duration_ms":28366,"significance":"If the reported gains are real, the paper would demonstrate a useful application of LLM-based visual reasoning to OOD segmentation, and its identification of three failure regimes could be of practical interest. The paper includes a clear motivation, a two-prompt design (V1 state+noun and V2 noun-only), and ablation experiments that attempt to isolate the effect of CoT reasoning. However, the central empirical claim is currently not supported because the evaluation protocol uses per-image threshold optimization on the test set for the proposed method, and the contributions are not accompanied by released prompts or code. The paper's value would be substantially increased by a fixed-threshold evaluation and by providing the exact prompts for reproducibility.","major_comments":[{"comment":"The per-image optimization of the box and text thresholds for GroundedSAM is an oracle protocol: the test ground-truth masks are used to select two thresholds per image that maximize the same mIoU and F1 scores that are then reported. This protocol appears to be applied only to the GroundedSAM rows in Table 1, while the compared methods (S2M, RbA, Mask2Anomaly, SynBoost, RPL+CoroCL, PEBAL) have fixed inference settings. As a result, Table 1 does not establish that the CoT-derived prompts cause the reported improvement, because the per-image oracle tuning alone could account for the margin. The authors should report results with fixed thresholds, for both the proposed method and all baselines, and also report results in which the baselines receive a comparable per-image tuning procedure.","section":"Section 4.1"},{"comment":"The ablation comparisons in Tables 2 and 3 are also performed with per-image threshold optimization for every row. Under that protocol, the relative ordering of prompt types may reflect the optimizer's ability to fit thresholds to the available prompts rather than the semantic quality of the prompts themselves. Notably, the low scores for 'only GPT-4' and '1 step' could be caused by threshold optimization failing for those prompts, not by the absence of CoT reasoning. The authors should rerun these ablations with a single fixed threshold set or with a validation-based threshold selection and report the results.","section":"Tables 2 and 3"},{"comment":"The 'Challenging' subset is defined post hoc after observing performance drops, but the paper does not specify the exact selection criteria, the number of images, the class distribution, or the overlap with the standard RoadAnomaly split. Because the subset is used to support the claim of robustness, the selection protocol must be fully described and, ideally, predefined before evaluation to avoid cherry-picking. The authors should also report the performance of all methods on the full RoadAnomaly test set without threshold optimization, in addition to the subset results.","section":"Section 4.1, Challenging subset"},{"comment":"The CoT prompts themselves are the core method, yet the paper does not provide the exact prompt templates, the GPT-4 version and decoding settings, or the generated prompts for the test images. Without this information, the results cannot be independently reproduced or compared across works. The authors should release the full set of prompts and the code for the reasoning and grounding pipeline.","section":"Section 3.2 and reproducibility"},{"comment":"The paper gives no error bars, variance estimates, or multiple-run statistics for any of the reported numbers. Given that the proposed method depends on stochastic LLM sampling and per-image threshold selection, the reported single-run values, especially the 0.912 vs. 0.803 margin, may not be stable. The authors should report means and standard deviations over multiple runs or, at minimum, over multiple GPT-4 decoding seeds.","section":"Section 4.1 and Table 1"}],"minor_comments":[{"comment":"The phrase 'criti-cal' in the abstract is a line-breaking artifact that should be corrected to 'critical'.","section":"Section 1"},{"comment":"The name 'LLaV A' contains an unintended space and should be 'LLaVA'.","section":"Section 2.2"},{"comment":"The sentence contains 'Section. 4.1' with an extra period; it should read 'Section 4.1'.","section":"Section 4.2"},{"comment":"The notation T Vi ∈ {TV1, TV2} is slightly inconsistent because the subscript i is used without a bound; it would be clearer to define i ∈ {V1, V2}.","section":"Section 3.2"},{"comment":"The table rows 'only GPT-4' and '1 step' lack a clear description of what output is passed to GroundedSAM; the text should clarify whether these rows use the GPT-4 output as the text prompt or as the final segmentation.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially interesting idea, but the evaluation protocol in Section 4.1 is the decisive issue. The per-image oracle threshold tuning is described transparently, which is commendable, but it undermines the comparison with fixed-threshold baselines. The authors should be given the opportunity to resubmit with a fixed-threshold evaluation and a more rigorous protocol for the Challenging subset. I would not recommend rejection solely because the approach uses an LLM-based prompt generator, but the current experimental evidence does not support the abstract's 'consistently outperforms' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The core idea—using GPT-4 chain-of-thought to generate text prompts for GroundedSAM in road OOD detection—is new, and the paper identifies three failure modes (dense overlapping objects, small distant objects, large foreground objects) that are easy to see in practice. The second thing is that the evaluation protocol is not fair enough to support the headline claim. In Section 4.1 they say the box and text thresholds for GroundingDINO were individually optimized per image to maximize segmentation accuracy on the test set. That is an oracle selection using ground truth. Tables 1, 2, and 3 all benefit from this, while the baselines—S2M, RbA, Mask2Anomaly, SynBoost, RPL+CoroCL, PEBAL—run at fixed settings. So the large margins in Table 1 do not establish that the CoT prompts are the cause. They could just reflect per-image threshold overfitting. The stress-test note puts its finger on the right spot.\n\nWhat is actually solid: the within-model ablations in Tables 2 and 3. On the same GroundedSAM pipeline, moving from a generic 'object' prompt to the CoT-derived V1∪V2 prompts raises mIoU from 0.724 to 0.838 on the Challenging subset, and the step-by-step integration of reasoning stages progresses in a believable way. That tells you prompt design matters, even if the SOTA comparison doesn't hold. The dual-prompt strategy is a sensible practical choice, and the paper honestly flags that overly long prompts hurt performance.\n\nSoft spots beyond the oracle threshold: the Challenging subset is defined post hoc, so performance on it is a bit circular. No code, no exact prompts, no error bars. The 'only GPT-4' row in Table 3 drops to 0.380, well below the generic prompt; that looks like a poorly controlled prompt format rather than evidence against unstructured reasoning. And calling this 'zero-shot' while tuning per-image thresholds is misleading.\n\nWho should read it: people working on open-vocabulary segmentation and LLM-guided perception will get a useful idea and a cautionary example. The main claim should not be cited as established until the evaluation is fixed. My recommendation: this deserves a serious referee, but the revision must include fixed-threshold runs, release of the exact prompts and code, and an optional comparison where baselines get the same per-image tuning if the authors want to keep that setup. As it stands, it's a conditional accept at best.","headline":"New idea, but oracle threshold tuning makes the SOTA claim unverifiable.","tokens_in":8223,"tokens_out":3185,"would_cite":false,"duration_ms":32183,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoT-generated prompts lift road anomaly segmentation to 0.91 mIoU","keywords":["out-of-distribution detection","semantic segmentation","chain-of-thought reasoning","road anomaly segmentation","open-vocabulary segmentation","GroundedSAM","GPT-4","prompt-based reasoning"],"falsifier":"Take a fixed set of images from RoadAnomaly and run GroundedSAM twice, once with CoT prompts and once with the generic prompt 'object', keeping the same box and text thresholds for both; if the mIoU gap shrinks to near zero, the paper's claim that CoT reasoning drives the improvement is falsified.","tokens_in":7228,"feed_emoji":"🚗","tokens_out":4797,"duration_ms":47297,"temperature":0.7,"pith_summary":"This paper tries to show that chain-of-thought reasoning by a large language model can make open-vocabulary segmentation detect road anomalies better than specialized out-of-distribution models. It identifies three road-scene situations where current OOD segmentation fails—dense overlapping objects, distant small objects, and large foreground-dominant objects—and proposes using GPT-4 to reason step by step about what is abnormal, turning that reasoning into text prompts. A reader should care because the approach needs no retraining or category-specific data; it only needs a prompt that names the anomaly. On the RoadAnomaly benchmark and a harder subset the paper defines, the prompted pipeline reports higher mIoU and F1 scores than the compared methods.","feed_headline":"CoT prompts lift road anomaly segmentation to 0.91 mIoU","feed_subtitle":"GPT-4 reasoning steers open-vocabulary segmentation for dense, distant, and large foreground hazards.","key_machinery":"The load-bearing mechanism is the CoT-based prompt generation module: GPT-4 is prompted to analyze the scene, identify deviations along appearance, semantic, and spatial axes, and output a short state+noun prompt (V1) and a noun-only prompt (V2). These prompts are then resolved into boxes by GroundingDINO and into masks by SAM, with the final prediction being the pixel union of the two prompt-driven masks. The three-step reasoning and the dual-prompt union are what the paper credits for the improvement; ablations show performance rising from 0.088 mIoU with one step to 0.838 with all three steps.","core_discovery":"The central claim is that structured visual reasoning, expressed as language, is enough to steer an open-vocabulary segmenter to out-of-distribution objects. The framework breaks reasoning into three steps, S1 scene analysis, S2 deviation detection, S3 final candidate selection, and produces two complementary prompts, a state+noun phrase and a plain noun. Feeding both into GroundedSAM (GroundingDINO for boxes, SAM for masks) and taking the union of the masks yields the final OOD prediction. In the paper's experiments this reaches 0.912 mIoU and 0.941 F1 on RoadAnomaly, and 0.838 mIoU and 0.892 F1 on the newly defined challenging subset, compared with 0.803 and 0.862 for the same model using the generic prompt 'object'.","pith_inferences":["The paper does not compare against using per-image threshold optimization for the baseline prompt 'object' with the same tuning budget, so the marginal gain specifically attributable to CoT reasoning over generic open-vocabulary grounding is not isolated; a fixed-threshold study would be the direct test.","The same prompt-generation idea should transfer to other text-conditioned segmenters and other anomaly domains, since the method only needs a vision-language model and a grounded detector; that is an extrapolation, not a claim in the paper.","The paper's own conclusion notes that overly long prompts hurt performance, suggesting a future system could iteratively refine prompts using the segmenter's feedback instead of one-shot generation."],"forward_implications":["OOD segmentation can be improved at inference time by asking a language model to name the anomaly, without retraining the segmenter.","The three challenging scenarios define a test subset that future road-anomaly methods should report separately, since average scores hide large drops on these cases.","Prompt phrasing matters quantitatively: describing the anomaly as an OOD object or with contextual state beats the generic prompt 'object' by a large margin.","The union of a descriptive prompt and a plain noun prompt outperforms either prompt alone, so prompt diversity is an easy robustness lever."],"supporting_citations":[{"why":"GPT-4 supplies the chain-of-thought reasoning that generates the prompts.","marker":"[1]"},{"why":"Prior evidence that long prompts degrade GroundingDINO motivates the concise dual-prompt design.","marker":"[9]"},{"why":"SAM converts the detected boxes into the pixel-level masks that the final union is computed from.","marker":"[10]"},{"why":"RoadAnomaly is the benchmark and the source of the newly defined challenging subset.","marker":"[12]"},{"why":"GroundingDINO retrieves bounding boxes from the CoT text prompts.","marker":"[14]"},{"why":"GroundedSAM is the overall open-vocabulary segmentation backbone the framework steers.","marker":"[21]"},{"why":"Chain-of-thought prompting is the reasoning technique the framework applies to vision.","marker":"[25]"},{"why":"S2M is the state-of-the-art baseline the paper analyzes for failure scenarios and outperforms.","marker":"[26]"}],"fun_headline_variants":["GPT-4 chain-of-thought steers OOD road detection","CoT reasoning with GPT-4 boosts road anomaly segmentation","Stepwise reasoning sharpens open-vocab detection of road hazards","Language-guided steps boost OOD segmentation in dense traffic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that using different per-image threshold values for the detector is fair, and that the baselines were not given the same per-image tuning; if thresholds rather than the CoT prompts explain the gains, the main claim does not follow.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 chain-of-thought steers OOD road detection","CoT reasoning with GPT-4 boosts road anomaly segmentation","Stepwise reasoning sharpens open-vocab detection of road hazards","Language-guided steps boost OOD segmentation in dense traffic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000328,"raw_usage":{"total_tokens":1834,"prompt_tokens":952,"completion_tokens":882,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":812}},"tokens_in":568,"tokens_out":882,"duration_ms":9335,"temperature":1.0,"reasoning_tokens":812,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:57:16.476546+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed set of images from RoadAnomaly and run GroundedSAM twice, once with CoT prompts and once with the generic prompt 'object', keeping the same box and text thresholds for both; if the mIoU gap shrinks to near zero, the paper's claim that CoT reasoning drives the improvement is falsified.","supporting_citations":[{"cited_title":"Kirillov, E","cited_arxiv_id":null,"evidence_quote":"SAM converts the detected boxes into the pixel-level masks that the final union is computed from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RoadAnomaly is the benchmark and the source of the newly defined challenging subset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GroundingDINO retrieves bounding boxes from the CoT text prompts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting is the reasoning technique the framework applies to vision."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"S2M is the state-of-the-art baseline the paper analyzes for failure scenarios and outperforms."}],"review_version":1}