InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning

Ce Zhang; Deva Ramanan; Katia Sycara; Simon Stepputtis; Yaqi Xie; Zhiqiu Lin; Zifu Wan; Zihan Wang

REVIEW 3 major objections 6 minor 1 cited by

InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces a hand-labeled benchmark of 2,400 household images pairing task instructions with part segmentation masks, and shows that fine-tuning a simple baseline on it more than doubles the part segmentation accuracy of…

desk verdict Useful benchmark, but the headline fine-tuning gain rests on an undisclosed train/test split and a 'first dataset' claim that conflicts with the authors' own 2024 workshop paper. read the letter →

arxiv 2505.18291 v1 pith:MOJO63DB submitted 2025-05-23 cs.CV cs.CLcs.RO

Zifu Wan , Yaqi Xie , Ce Zhang , Zhiqiu Lin , Zihan Wang , Simon Stepputtis , Deva Ramanan , Katia Sycara This is my paper · ORCID

classification cs.CVcs.CLcs.RO

keywords task-orientedpartsegmentationinstructionreasoningvision-languagemodelsreferringexpressionaffordancesbenchmarkdatasetfine-tuninghouseholdobjects

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that state-of-the-art vision-language models cannot reliably find the part of an object that a task instruction refers to, such as the handle to pull or the lid to open, even when they can name or segment the whole object. To test this, it introduces InstructPart, a hand-labeled real-world benchmark of 2,400 household images with 44 part classes, 30 affordances, 37 actions, and 9,600 task instructions in both human-written and GPT-4-rewritten form. Across open-vocabulary, referring-expression, and reasoning segmentation models, the paper finds large failures on both of its tasks, especially when the part name is hidden inside an instruction. It then fine-tunes a simple baseline, PISA, on 1,800 of its samples and reports that task-reasoning accuracy more than doubles: gIoU rises from 32.11 to 76.19 and P@50 from 30.00 to 87.00 relative to the pretrained LISA baseline. If the benchmark holds up, it offers a way to measure and improve instruction-following part grounding for robots and assistive systems.

What carries the argument

The load-bearing artifact is the InstructPart dataset itself, with each item carrying an image, a task instruction in both human and GPT-4 rewritten versions, an object name, a part name, a hand-labeled segmentation mask, and labels for affordance and action. The benchmark defines two tasks that bracket the reasoning: TRPS asks for the mask from an instruction that names no part, and ORPS asks for the mask from an explicit part-of-object query, with or without an affordance phrase. The baseline PISA is the second piece of machinery: starting from the LISA reasoning-segmentation model, it swaps in a frozen DINOv2 image encoder whose multi-level features are linearly fused and fed to a SAM-derived decoder. This architecture lets the authors attribute most of the reported gain to the new training data rather than to a novel decoder.

What would settle it

Inspect the released InstructPart files for image-level overlap between the 1,800 training samples and the test set; if any overlap exists, refit PISA on a documented disjoint split and check whether gIoU stays near 76.19 or falls back toward the pretrained level.

Watch

Extended reading notes

Core claim

The central claim is that task-oriented part segmentation is a distinct, largely unsolved capability: a model must infer which physical component of an object supports a described action and produce a pixel mask of just that component, not the whole object. The paper introduces two tasks to isolate this capability: Task Reasoning Part Segmentation (TRPS), where the instruction deliberately avoids part names, and Oracle Referring Part Segmentation (ORPS), where the object and part are named explicitly. On both, existing models perform poorly, and the ORPS-to-TRPS gap shows that part-level visual grounding is easier for current models than inferring the part from a task description. The paper's baseline, the Part Identification and Segmentation Assistant (PISA), combines LISA with a frozen DINOv2 feature extractor and a SAM-style decoder; fine-tuned on InstructPart, it reaches gIoU 76.19 and P@50 87.00 on TRPS with human instructions, roughly doubling the pretrained LISA's 32.11 and 30.00. The authors also claim InstructPart is the first dataset that pairs task-oriented instructions with part segmentation masks for common household interactions.

Load-bearing premise

Everything the paper reports about the dataset's training value rests on the assumption that the 1,800 fine-tuning samples are disjoint from the evaluation samples, but no train/test split is described in Section 4.4.

Editorial extensions

If this is right

Current models that segment whole objects well still miss task-relevant parts: on TRPS with human instructions, the best off-the-shelf reasoning model reaches only 32.11 gIoU and 30.00 P@50.
Because ORPS scores are consistently higher than TRPS scores, the main shortcoming is reasoning from task language to a part, not part recognition alone.
Fine-tuning LISA and PISA on 1,800 InstructPart samples pushes both to high accuracy, with LISA-Tuned at 71.26 gIoU and PISA-Tuned at 76.19 gIoU, so the dataset is a usable training signal and not just an evaluation set.
Adding affordance words to oracle queries, such as 'handle of the cup that can be held', does not improve average performance, indicating that current models do not yet exploit affordance semantics.
GPT-4-rewritten instructions give better TRPS results than the original human instructions, showing that instruction phrasing noticeably changes part-grounding performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves the affordance and action labels unused in training; a natural extension would be to condition the model on the affordance itself, which could make part grounding directly useful for manipulation planning.
The reported logarithmic performance curve with 200, 600, 1,200, and 1,800 training samples suggests that even a small, carefully annotated instruction-part set may unlock part grounding in a pretrained VLM, but this depends on a clean train/test split.
If the train/test split is not disjoint, the twofold gain should be read as an upper bound; re-evaluating on a documented disjoint split is the test that would settle the benchmark's value.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Marco,

The InstructPart benchmark is a genuinely useful resource for fine-grained part grounding, and the zero-shot evaluation across a dozen models is the kind of legwork the field needs. But the central quantitative claim—that fine-tuning on the dataset more than doubles performance—is not verifiable in the current version. The paper never states whether the 1,800 fine-tuning samples are disjoint from the images used for evaluation in Table 4, and with 2,400 images total, an overlap would let the model memorize the answer distribution. That's a load-bearing omission, not a style quibble. The stress-test concern is on point.

What the paper does well: the task design is sensible. TRPS forces reasoning from an instruction that omits the part name, and ORPS isolates pure grounding. The dataset covers 48 object classes and 44 part classes with hand-labeled masks, instruction variations, and affordance/action labels—the annotation effort is real. The authors evaluate a wide range of methods (open-vocab, referring, reasoning, GPT-4V variants), which gives a fair picture of current model weaknesses. The external grasping case study is a good idea and provides some independent signal that the fine-tuned model generalizes to new objects.

Soft spots beyond the split: the paper claims to introduce the 'first dataset that bridges task-oriented interactions with part segmentation,' yet the authors' own 2024 AAAI workshop paper with the same title and same dataset is cited but never acknowledged as prior work. That needs to be reconciled. Also, Appendix A's Figure 6 lists 53 object classes and 46 part classes while the main text says 48 and 44; minor but sloppy. The fine-tuning results are single-run with no error bars. And the evaluation set for Table 2 isn't defined, so it's unclear whether the zero-shot and fine-tuned numbers refer to the same images.

Who this is for: anyone working on part-level segmentation, affordance grounding, or language-guided robot manipulation. The dataset itself could be a useful benchmark even if the fine-tuning experiments need redoing. This deserves a serious referee: the resource is valuable, and the experiments can be fixed with a clearly specified held-out split and an honest positioning statement relative to the workshop paper. I'd engage with it, but condition on those changes.

Recommendation: send it to review, and ask the authors to disclose the train/test split and make the code/data available.

Referee Report

3 major / 6 minor

Summary. The paper introduces InstructPart, a dataset of 2,400 real-world images with hand-labeled part segmentation masks, task instructions, affordances, and actions, plus two evaluation tasks: Task Reasoning Part Segmentation (TRPS) and Oracle Referring Part Segmentation (ORPS). The authors benchmark a wide range of open-vocabulary, referring-expression, and reasoning segmentation models as well as GPT-4V-based pipelines, reporting that current VLMs perform poorly on task-oriented part grounding. They also propose PISA, a LISA-based baseline with a frozen DINOv2 encoder, and report that fine-tuning on 1,800 InstructPart samples more than doubles TRPS performance (Table 4: PISA-Tuned gIoU 76.19 vs. LISA-Pretrained 32.11). An external zero-shot grasping case study (Appendix I) provides additional evidence of generalization to unseen objects and tasks.

Significance. If the findings hold, InstructPart fills a clear gap: existing part datasets lack task-oriented instructions, and current VLM benchmarks largely ignore part-level grounding. The paper's zero-shot evaluation is broad—it covers OVS, RES, and RS model families plus two GPT-4V variants—and it convincingly demonstrates that state-of-the-art VLMs struggle with fine-grained task reasoning. The external grasping case study is a valuable out-of-distribution check. The dataset could become a standard evaluation resource for part-centric instruction following. However, the headline quantitative claim about fine-tuning depends on an unverified evaluation protocol, and the dataset statistics are internally inconsistent, so the significance is conditional until these issues are fixed.

major comments (3)

[§4.4, Table 4, Appendix D] The fine-tuning experiments never specify the evaluation split. The dataset contains 2,400 images and the fine-tuned models are trained on 1,800 samples; if any of those training images appear in the evaluation set, the large gains in Table 4 (e.g., LISA-Tuned gIoU 71.26 vs. LISA-Pretrained 32.11) could partly reflect memorization rather than generalization. The authors must report the exact train/test split, verify disjointness, and report Table 4 metrics on a held-out set. They must also clarify whether the zero-shot results in Table 2 are computed on the same evaluation set, so that Tables 2 and 4 are comparable.
[Appendix A, Figure 6, and Section 1] The dataset statistics are internally inconsistent. Section 1 states 48 object classes and 44 part classes; Appendix A text repeats 48 objects and 44 parts forming 98 object-part pairs, but Figure 6's caption reports 53 object classes, 46 part classes, and 108 object-part pairs, and the plot shows more than 98 bars. These numbers must be reconciled and the figure/caption corrected; as written, the dataset composition is not verifiable.
[§4.4] The fine-tuning results are reported as single runs without error bars or seed variation. Given the small training set (200–1,800 samples) and the magnitude of the claimed improvement, the authors should report the mean and standard deviation over at least three runs, or otherwise justify that the differences in Table 4 are not due to training variance. This is load-bearing because the central claim is the twofold improvement from fine-tuning.

minor comments (6)

[§2.1] The phrase 'in it itself' should be 'in itself'.
[§4.2] 'pertaining' should be 'pre-training' in the description of LISA fine-tuning.
[§4.1] The explanation of P@50:95 is unclear; 'a higher least IoU' is awkward and should be rewritten.
[Table 1] The '11/158' and '–' symbols in Table 1 need a legend or inline explanation for readers unfamiliar with the listed datasets.
[§3.1] The sentence 'We formulate the ORPS task in two formats:' is repeated and should be merged with the preceding sentence.
[Appendix I] The statement that LERF-TOGO's 6% higher accuracy is 'just 3 images' assumes a specific denominator; clarify the number of test tasks used for that comparison.

Circularity Check

0 steps flagged · score 2.0 of 10

No formal circularity is demonstrable from the text; the main risk is an undisclosed train/test split behind the Table 4 twofold-improvement claim, which is a verifiability flaw rather than a shown reduction.

full rationale

The paper's derivation chain is largely self-contained: InstructPart is a new hand-labeled benchmark (Section 3.2); the evaluations of existing VLMs in Tables 2 and 3 use external off-the-shelf models; and the PISA baseline is an architectural combination of LISA and DINOv2, with no load-bearing reliance on the authors' own prior theorems. The fine-tuning experiment in Section 4.4 is the only place where the headline quantitative claim ('twofold improvement') could become circular: Table 4 states 'Fine-tuned models are trained with 1,800 samples in InstructPart,' while the dataset is described as comprising 2,400 images, and the paper never specifies the evaluation split. If the Table 4 evaluation set overlaps with the 1,800 training samples, part of the reported gain from LISA-Pretrained 32.11 to LISA-Tuned 71.26 would reflect memorization rather than generalization. However, the paper does not explicitly say that the full 2,400 images were used for evaluation; Section 5 only notes that 'one can also use the entire 2,400 images to evaluate current advanced VLMs,' which is a benchmarking suggestion rather than a description of the actual protocol. The zero-shot grasping case study in Appendix I, evaluated on the external ShapeGrasp tabletop setup, provides independent evidence that a PISA model fine-tuned on InstructPart can identify parts of novel objects with 80% part-selection accuracy, which lowers the circularity burden. Because no explicit equivalence or fitted-parameter-renamed-as-prediction can be quoted from the paper, no formal circular step is identified; the undisclosed split is a missing-support issue that should be corrected before the twofold-improvement claim can be fully verified.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

This is a dataset and empirical benchmark paper, so there are no fitted physical parameters or invented theoretical entities. The main load-bearing assumptions are about annotation quality, split hygiene, and the validity of the instruction design.

assumptions (4)

domain assumption Hand-labeled segmentation masks are accurate ground truth
The benchmark's validity depends on annotation quality; the paper does not report inter-annotator agreement or quality checks beyond 'human verification' (Section 3.2).
domain assumption The train and test splits for fine-tuning are disjoint
Section 4.4 reports fine-tuning on 1,800 samples but never specifies the split, so the reader must assume no leakage for the reported gains.
domain assumption GPT-4-rewritten instructions preserve human intent and add diversity
Section 3.2 says GPT-4 was used for grammatical precision and diversity followed by human verification; this is an unproved assumption about instruction quality.
domain assumption Implicit task descriptions (without part names) measure reasoning rather than ambiguity
Section 3.2 deliberately avoids part names; the paper assumes this design isolates reasoning ability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning." pith.science (2026). https://pith.science/paper/MOJO63DB

@misc{pith2026250518291,
  author       = {Pith},
  title        = {Pith review of: InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MOJO63DB}},
  note         = {Machine review of arXiv:2505.18291}
}

read the original abstract

Large multimodal foundation models, particularly in the domains of language and vision, have significantly advanced various tasks, including robotics, autonomous driving, information retrieval, and grounding. However, many of these models perceive objects as indivisible, overlooking the components that constitute them. Understanding these components and their associated affordances provides valuable insights into an object's functionality, which is fundamental for performing a wide range of tasks. In this work, we introduce a novel real-world benchmark, InstructPart, comprising hand-labeled part segmentation annotations and task-oriented instructions to evaluate the performance of current models in understanding and executing part-level tasks within everyday contexts. Through our experiments, we demonstrate that task-oriented part segmentation remains a challenging problem, even for state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark, we introduce a simple baseline that achieves a twofold performance improvement through fine-tuning with our dataset. With our dataset and benchmark, we aim to facilitate research on task-oriented part segmentation and enhance the applicability of VLMs across various domains, including robotics, virtual reality, information retrieval, and other related fields. Project website: https://zifuwan.github.io/InstructPart/.

Figures

Figures reproduced from arXiv: 2505.18291 by the authors.

**Figure 2.** Examples from our InstructPart dataset are illustrated as follows: instruction queries are denoted in red [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Qualitative comparison of different VLMs and the fine-tuned models. In these examples, the pre-trained [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Qualitative comparison of different VLMs and the fine-tuned models. In these examples, the pre-trained [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Qualitative comparison of different VLMs and the fine-tuned models. In these examples, the pre-trained [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Object-part pair distribution. We collect 2,400 data pieces in total, containing 48 object classes and 44 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
cs.CV 2025-07 conditional novelty 6.0 of 10

A single-layer, single-query intervention that amplifies attention heads with high text-to-visual entropy reduces hallucination in LVLMs at about 1.07x the inference time of regular decoding.

Reference graph

Works this paper leans on

39 extracted references · 38 canonical work pages · cited by 1 Pith paper

[1]

1009786005_d4a02fd811_o- faucet-handle

work page
[2]

2329134125_8a71be7470_o- kettle-handle

work page
[3]

3088942376_8681bb276f_o- spoon-handle

work page
[4]

In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 20406–20417

Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 20406–20417. Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang, Xiaoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, and Hao Dong. 2024. Manipvqa: Injecting robotic affordance and ph...

work page 2024
[5]

Please output the segmentation mask

Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In International Conference on Ma- chine Learning, pages 19730–19742. PMLR. Samuel Li, Sarthak Bhagat, Joseph Campbell, Yaqi Xie, Woojun Kim, Katia P. Sycara, and Simon Stepputtis. 2024b. Shapegrasp: Zero-shot task-oriented grasp- ing with large lang...

arXiv 2023
[6]

laptop_000445-laptop-keyboard

410044558_6145ff0aaa_o- pot-handle 7. laptop_000445-laptop-keyboard

work page
[7]

381204305_e5e937fccc_h- pitcher-handle

work page
[8]

knife_000691-knife-handle

work page

Show all 39 references

[9]

cup_000294-cup-handle
[10]

knife_000911-knife-handle
[11]

shovel_1-shovel-blade
[12]

suitcase_001098-suitcase- telescopic_handle
[13]

4178009615_ed8921d0d1_k- kettle-spout
[14]

cup_000324-cup-handle
[15]

bottle_002805-bottle-body
[16]

knife_000568-knife-handle
[17]

knife_000953-knife-blade
[18]

34465720_f8f20ee31a_c- scissors-handle
[20]

bench_001273-bench-seat
[21]

fork_002954-fork-handle
[22]

knife_000154-knife-handle
[25]

wine_glass_001774- wine_glass-stem
[26]

2491323916_a05ac3648f_o- knife-handle
[27]

4580224808_1194613deb_o- chair-seat
[28]

4471021242_b9d855f193_k- bucket-handle
[29]

8607578325_25221a7726_h- spoon-handle
[30]

bench_002898-bench-seat
[31]

cup_001798-cup-handle
[32]

cup_002055-cup-handle
[33]

knife_000530-knife-blade
[34]

scissors_001402-scissors- handle
[35]

cup_002062-cup-handle
[36]

2939090254_2f01ebed6d_o- computer_mouse-scroll_wheel
[37]

6217625873_411169d784_o- laptop-keyboard
[38]

cup_001104-cup-handle
[39]

image_path

fork_001529-fork-handle Table G6: Index name for samples in Fig. G13, Fig. G14, and Fig. G15. G More Qualitative Results In Figure 3-5 of the main paper, we only include six qualitative results due to space limitations. In Fig. G13, we present more examples where the fine-tune...

2024
[2016]

In European Conference on Computer Vision, pages 108–124

Segmentation from natural language expres- sions. In European Conference on Computer Vision, pages 108–124. Springer. Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith
[2021]

In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pages 15752–15761

Act the part: Learning interaction strategies for articulated object part discovery. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, pages 15752–15761. Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. 2023. Gapartn...

2023
[2023]

In International Conference on Machine Learning, pages 8469–8488

Palm-e: An embodied multimodal language model. In International Conference on Machine Learning, pages 8469–8488. PMLR. Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. 2020. Graspnet-1billion: A large-scale bench- mark for general object grasping. In Proceedings of the IEE...

2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

1009786005_d4a02fd811_o- faucet-handle

work page

[2] [2]

2329134125_8a71be7470_o- kettle-handle

work page

[3] [3]

3088942376_8681bb276f_o- spoon-handle

work page

[4] [4]

In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 20406–20417

Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 20406–20417. Siyuan Huang, Iaroslav Ponomarenko, Zhengkai Jiang, Xiaoqi Li, Xiaobin Hu, Peng Gao, Hongsheng Li, and Hao Dong. 2024. Manipvqa: Injecting robotic affordance and ph...

work page 2024

[5] [5]

Please output the segmentation mask

Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In International Conference on Ma- chine Learning, pages 19730–19742. PMLR. Samuel Li, Sarthak Bhagat, Joseph Campbell, Yaqi Xie, Woojun Kim, Katia P. Sycara, and Simon Stepputtis. 2024b. Shapegrasp: Zero-shot task-oriented grasp- ing with large lang...

arXiv 2023

[6] [6]

laptop_000445-laptop-keyboard

410044558_6145ff0aaa_o- pot-handle 7. laptop_000445-laptop-keyboard

work page

[7] [7]

381204305_e5e937fccc_h- pitcher-handle

work page

[8] [8]

knife_000691-knife-handle

work page