{"id":"f7b0bc82-7f8f-4378-9a52-da5bb6e996b3","arxiv_id":"2505.18291","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"InstructPart is a new benchmark of 2,400 household images with part masks and task instructions, and current vision-language models score poorly on it while a fine-tuned LISA/DINOv2 baseline more than doubles accuracy.","lead":"The paper builds InstructPart, a hand-labeled benchmark of 2,400 household images with part masks and task-oriented instructions, and shows that current vision-language models often fail at part-level reasoning. It also presents a fine-tuned LISA/DINOv2 baseline that more than doubles segmentation accuracy on the benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Twofold-improvement claim rests on an unspecified train/test split; with 2,400 images and 1,800 fine-tuning samples, overlap could inflate Table 4 gains from memorization rather than generalization.","rationale":"The reader's weakest assumption, an unspecified train/test split, is exactly the load-bearing issue. The paper's main quantitative evidence for dataset quality is the two-fold improvement after fine-tuning on 1,800 samples; without a disjoint evaluation set, that improvement is not evidence of generalization. I examined alternative concerns (claim of being the first dataset, inconsistency in class counts between Section 3.2 and Appendix A, and the baseline's architectural change) but none is as directly tied to the central empirical claim as the evaluation protocol. The first-dataset claim is secondary and could be checked against the authors' own 2024 workshop paper, and the class-count inconsistencies do not change the direction of the result. The missing split also undermines Table 2 comparability. I would therefore keep the paper's conditional status: the benchmark may be valuable, but the quantitative claims should not be accepted until the split is disclosed and the reported numbers are reproduced on a held-out set.","tokens_in":20610,"tokens_out":5792,"duration_ms":46390,"concrete_test":"Request the official train/test split from the authors (or inspect the released dataset) and recompute Table 4 on the test subset that is strictly disjoint from the 1,800 fine-tuning images. If the tuned models' improvement over pre-trained shrinks materially (e.g., PISA-Tuned gIoU drops from 76.19 toward the 43.46 PISA-Pretrained value), the twofold claim is substantially explained by training/evaluation overlap. Also verify that the same split is used for all methods in Table 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 (Table 4) reports that fine-tuned LISA and PISA are trained with 1,800 InstructPart samples, but the paper never states the evaluation split. The dataset contains 2,400 images. If evaluation is on the full 2,400 (or any set overlapping the 1,800), then 75% of the evaluated images were seen during fine-tuning, and the reported jump from LISA-Pretrained 32.11 to LISA-Tuned 71.26 gIoU could reflect memorization. If evaluation is on the remaining 600, that split must be disclosed and held constant across all methods in Tables 2 and 4. Appendix D's smooth log-convergence curve as training grows from 200 to 1,800 samples is also consistent with training-set evaluation; a genuine held-out curve at these small sizes would typically be noisier. The central quantitative claim of the paper, that fine-tuning on InstructPart more than doubles performance, is therefore not yet verifiable. A second, related problem is that the paper never defines the evaluation set for Table 2 either, so it is unclear whether zero-shot methods and fine-tuned methods are compared on the same images. This is the single largest threat to the paper's headline result; the dataset itself may be useful, but the benchmark evaluation is compromised until the split is specified and shown disjoint from training.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces InstructPart, a dataset of 2,400 real-world images with hand-labeled part segmentation masks, task instructions, affordances, and actions, plus two evaluation tasks: Task Reasoning Part Segmentation (TRPS) and Oracle Referring Part Segmentation (ORPS). The authors benchmark a wide range of open-vocabulary, referring-expression, and reasoning segmentation models as well as GPT-4V-based pipelines, reporting that current VLMs perform poorly on task-oriented part grounding. They also propose PISA, a LISA-based baseline with a frozen DINOv2 encoder, and report that fine-tuning on 1,800 InstructPart samples more than doubles TRPS performance (Table 4: PISA-Tuned gIoU 76.19 vs. LISA-Pretrained 32.11). An external zero-shot grasping case study (Appendix I) provides additional evidence of generalization to unseen objects and tasks.","tokens_in":20898,"tokens_out":4025,"duration_ms":33215,"significance":"If the findings hold, InstructPart fills a clear gap: existing part datasets lack task-oriented instructions, and current VLM benchmarks largely ignore part-level grounding. The paper's zero-shot evaluation is broad—it covers OVS, RES, and RS model families plus two GPT-4V variants—and it convincingly demonstrates that state-of-the-art VLMs struggle with fine-grained task reasoning. The external grasping case study is a valuable out-of-distribution check. The dataset could become a standard evaluation resource for part-centric instruction following. However, the headline quantitative claim about fine-tuning depends on an unverified evaluation protocol, and the dataset statistics are internally inconsistent, so the significance is conditional until these issues are fixed.","major_comments":[{"comment":"The fine-tuning experiments never specify the evaluation split. The dataset contains 2,400 images and the fine-tuned models are trained on 1,800 samples; if any of those training images appear in the evaluation set, the large gains in Table 4 (e.g., LISA-Tuned gIoU 71.26 vs. LISA-Pretrained 32.11) could partly reflect memorization rather than generalization. The authors must report the exact train/test split, verify disjointness, and report Table 4 metrics on a held-out set. They must also clarify whether the zero-shot results in Table 2 are computed on the same evaluation set, so that Tables 2 and 4 are comparable.","section":"§4.4, Table 4, Appendix D"},{"comment":"The dataset statistics are internally inconsistent. Section 1 states 48 object classes and 44 part classes; Appendix A text repeats 48 objects and 44 parts forming 98 object-part pairs, but Figure 6's caption reports 53 object classes, 46 part classes, and 108 object-part pairs, and the plot shows more than 98 bars. These numbers must be reconciled and the figure/caption corrected; as written, the dataset composition is not verifiable.","section":"Appendix A, Figure 6, and Section 1"},{"comment":"The fine-tuning results are reported as single runs without error bars or seed variation. Given the small training set (200–1,800 samples) and the magnitude of the claimed improvement, the authors should report the mean and standard deviation over at least three runs, or otherwise justify that the differences in Table 4 are not due to training variance. This is load-bearing because the central claim is the twofold improvement from fine-tuning.","section":"§4.4"}],"minor_comments":[{"comment":"The phrase 'in it itself' should be 'in itself'.","section":"§2.1"},{"comment":"'pertaining' should be 'pre-training' in the description of LISA fine-tuning.","section":"§4.2"},{"comment":"The explanation of P@50:95 is unclear; 'a higher least IoU' is awkward and should be rewritten.","section":"§4.1"},{"comment":"The '11/158' and '–' symbols in Table 1 need a legend or inline explanation for readers unfamiliar with the listed datasets.","section":"Table 1"},{"comment":"The sentence 'We formulate the ORPS task in two formats:' is repeated and should be merged with the preceding sentence.","section":"§3.1"},{"comment":"The statement that LERF-TOGO's 6% higher accuracy is 'just 3 images' assumes a specific denominator; clarify the number of test tasks used for that comparison.","section":"Appendix I"}],"recommendation":"major_revision","confidential_remarks":"The zero-shot evaluation and the external grasping case study are the strongest parts of the paper. The dataset could be a useful community resource. However, the current manuscript does not disclose the evaluation protocol for the fine-tuning experiments, which directly undercuts the headline twofold-improvement claim. The inconsistencies in class counts are also concerning for a benchmark paper. I recommend major revision, with the split disclosure and metric reporting as mandatory conditions. The scope is appropriate for a CV venue if the authors can substantiate the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Marco,\n\nThe InstructPart benchmark is a genuinely useful resource for fine-grained part grounding, and the zero-shot evaluation across a dozen models is the kind of legwork the field needs. But the central quantitative claim—that fine-tuning on the dataset more than doubles performance—is not verifiable in the current version. The paper never states whether the 1,800 fine-tuning samples are disjoint from the images used for evaluation in Table 4, and with 2,400 images total, an overlap would let the model memorize the answer distribution. That's a load-bearing omission, not a style quibble. The stress-test concern is on point.\n\nWhat the paper does well: the task design is sensible. TRPS forces reasoning from an instruction that omits the part name, and ORPS isolates pure grounding. The dataset covers 48 object classes and 44 part classes with hand-labeled masks, instruction variations, and affordance/action labels—the annotation effort is real. The authors evaluate a wide range of methods (open-vocab, referring, reasoning, GPT-4V variants), which gives a fair picture of current model weaknesses. The external grasping case study is a good idea and provides some independent signal that the fine-tuned model generalizes to new objects.\n\nSoft spots beyond the split: the paper claims to introduce the 'first dataset that bridges task-oriented interactions with part segmentation,' yet the authors' own 2024 AAAI workshop paper with the same title and same dataset is cited but never acknowledged as prior work. That needs to be reconciled. Also, Appendix A's Figure 6 lists 53 object classes and 46 part classes while the main text says 48 and 44; minor but sloppy. The fine-tuning results are single-run with no error bars. And the evaluation set for Table 2 isn't defined, so it's unclear whether the zero-shot and fine-tuned numbers refer to the same images.\n\nWho this is for: anyone working on part-level segmentation, affordance grounding, or language-guided robot manipulation. The dataset itself could be a useful benchmark even if the fine-tuning experiments need redoing. This deserves a serious referee: the resource is valuable, and the experiments can be fixed with a clearly specified held-out split and an honest positioning statement relative to the workshop paper. I'd engage with it, but condition on those changes.\n\nRecommendation: send it to review, and ask the authors to disclose the train/test split and make the code/data available.","headline":"Useful benchmark, but the headline fine-tuning gain rests on an undisclosed train/test split and a 'first dataset' claim that conflicts with the authors' own 2024 workshop paper.","tokens_in":17,"tokens_out":4713,"would_cite":false,"duration_ms":62082,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a hand-labeled benchmark of 2,400 household images pairing task instructions with part segmentation masks, and shows that fine-tuning a simple baseline on it more than doubles the part segmentation accuracy of…","keywords":["task-oriented part segmentation","instruction reasoning","vision-language models","referring expression segmentation","part affordances","benchmark dataset","fine-tuning","household objects"],"falsifier":"Inspect the released InstructPart files for image-level overlap between the 1,800 training samples and the test set; if any overlap exists, refit PISA on a documented disjoint split and check whether gIoU stays near 76.19 or falls back toward the pretrained level.","tokens_in":20472,"feed_emoji":"🧩","tokens_out":5705,"duration_ms":43679,"temperature":0.7,"pith_summary":"The paper sets out to show that state-of-the-art vision-language models cannot reliably find the part of an object that a task instruction refers to, such as the handle to pull or the lid to open, even when they can name or segment the whole object. To test this, it introduces InstructPart, a hand-labeled real-world benchmark of 2,400 household images with 44 part classes, 30 affordances, 37 actions, and 9,600 task instructions in both human-written and GPT-4-rewritten form. Across open-vocabulary, referring-expression, and reasoning segmentation models, the paper finds large failures on both of its tasks, especially when the part name is hidden inside an instruction. It then fine-tunes a simple baseline, PISA, on 1,800 of its samples and reports that task-reasoning accuracy more than doubles: gIoU rises from 32.11 to 76.19 and P@50 from 30.00 to 87.00 relative to the pretrained LISA baseline. If the benchmark holds up, it offers a way to measure and improve instruction-following part grounding for robots and assistive systems.","feed_headline":"2,400 images more than double VLM part-segmentation accuracy","feed_subtitle":"A hand-labeled benchmark shows VLMs miss task-relevant parts; a simple fine-tune fixes much of the gap.","key_machinery":"The load-bearing artifact is the InstructPart dataset itself, with each item carrying an image, a task instruction in both human and GPT-4 rewritten versions, an object name, a part name, a hand-labeled segmentation mask, and labels for affordance and action. The benchmark defines two tasks that bracket the reasoning: TRPS asks for the mask from an instruction that names no part, and ORPS asks for the mask from an explicit part-of-object query, with or without an affordance phrase. The baseline PISA is the second piece of machinery: starting from the LISA reasoning-segmentation model, it swaps in a frozen DINOv2 image encoder whose multi-level features are linearly fused and fed to a SAM-derived decoder. This architecture lets the authors attribute most of the reported gain to the new training data rather than to a novel decoder.","core_discovery":"The central claim is that task-oriented part segmentation is a distinct, largely unsolved capability: a model must infer which physical component of an object supports a described action and produce a pixel mask of just that component, not the whole object. The paper introduces two tasks to isolate this capability: Task Reasoning Part Segmentation (TRPS), where the instruction deliberately avoids part names, and Oracle Referring Part Segmentation (ORPS), where the object and part are named explicitly. On both, existing models perform poorly, and the ORPS-to-TRPS gap shows that part-level visual grounding is easier for current models than inferring the part from a task description. The paper's baseline, the Part Identification and Segmentation Assistant (PISA), combines LISA with a frozen DINOv2 feature extractor and a SAM-style decoder; fine-tuned on InstructPart, it reaches gIoU 76.19 and P@50 87.00 on TRPS with human instructions, roughly doubling the pretrained LISA's 32.11 and 30.00. The authors also claim InstructPart is the first dataset that pairs task-oriented instructions with part segmentation masks for common household interactions.","pith_inferences":["The paper leaves the affordance and action labels unused in training; a natural extension would be to condition the model on the affordance itself, which could make part grounding directly useful for manipulation planning.","The reported logarithmic performance curve with 200, 600, 1,200, and 1,800 training samples suggests that even a small, carefully annotated instruction-part set may unlock part grounding in a pretrained VLM, but this depends on a clean train/test split.","If the train/test split is not disjoint, the twofold gain should be read as an upper bound; re-evaluating on a documented disjoint split is the test that would settle the benchmark's value."],"forward_implications":["Current models that segment whole objects well still miss task-relevant parts: on TRPS with human instructions, the best off-the-shelf reasoning model reaches only 32.11 gIoU and 30.00 P@50.","Because ORPS scores are consistently higher than TRPS scores, the main shortcoming is reasoning from task language to a part, not part recognition alone.","Fine-tuning LISA and PISA on 1,800 InstructPart samples pushes both to high accuracy, with LISA-Tuned at 71.26 gIoU and PISA-Tuned at 76.19 gIoU, so the dataset is a usable training signal and not just an evaluation set.","Adding affordance words to oracle queries, such as 'handle of the cup that can be held', does not improve average performance, indicating that current models do not yet exploit affordance semantics.","GPT-4-rewritten instructions give better TRPS results than the original human instructions, showing that instruction phrasing noticeably changes part-grounding performance."],"supporting_citations":[{"why":"Supplies the base reasoning-segmentation model LISA that PISA starts from, along with the gIoU and cIoU evaluation protocol.","marker":"(Lai et al., 2024)"},{"why":"Provides the frozen DINOv2 backbone that PISA uses to extract part-level correspondence features.","marker":"(Oquab et al., 2024)"},{"why":"Motivates fusing multi-level DINOv2 features with linear layers for affordance and part correspondence.","marker":"(Li et al., 2024a)"},{"why":"Supplies the SAM image decoder used to turn fused features into segmentation masks.","marker":"(Kirillov et al., 2023)"},{"why":"Provides the LLaVA-1.5 multimodal LLM inside LISA that interprets instructions in PISA.","marker":"(Liu et al., 2024a)"},{"why":"PACO is one of the part datasets used to pretrain LISA and a comparison point for part-level coverage.","marker":"(Ramanathan et al., 2023)"},{"why":"PartImageNet is a part segmentation pretraining set that the paper contrasts with its household-task coverage.","marker":"(He et al., 2022)"},{"why":"AGD20K supplied part of the real-world images used to build InstructPart.","marker":"(Luo et al., 2022)"},{"why":"Set-of-Mark prompting defines the SoM-based GPT-4V baseline used to test fine-grained part referring.","marker":"(Yang et al., 2023a)"},{"why":"Grounding-DINO drives the grid-based GPT-4V baseline and the Grounded-SAM comparison.","marker":"(Liu et al., 2024b)"}],"fun_headline_variants":["Benchmark shows VLMs miss task-relevant parts; fine-tune doubles accuracy","VLMs struggle to segment parts by task; simple fine-tune doubles their accuracy","New dataset doubles part-segmentation accuracy over pretrained VLMs","Task-oriented part segmentation: VLMs lag, but fine-tuning closes the gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything the paper reports about the dataset's training value rests on the assumption that the 1,800 fine-tuning samples are disjoint from the evaluation samples, but no train/test split is described in Section 4.4.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark shows VLMs miss task-relevant parts; fine-tune doubles accuracy","VLMs struggle to segment parts by task; simple fine-tune doubles their accuracy","New dataset doubles part-segmentation accuracy over pretrained VLMs","Task-oriented part segmentation: VLMs lag, but fine-tuning closes the gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00087,"raw_usage":{"total_tokens":3787,"prompt_tokens":984,"completion_tokens":2803,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":2720}},"tokens_in":600,"tokens_out":2803,"duration_ms":17450,"temperature":1.0,"reasoning_tokens":2720,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:33:11.919555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released InstructPart files for image-level overlap between the 1,800 training samples and the test set; if any overlap exists, refit PISA on a documented disjoint split and check whether gIoU stays near 76.19 or falls back toward the pretrained level.","supporting_citations":[],"review_version":1}