Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning

Jintai Chen; Peng Zhang; Shihui Zhen; Wenjie Li; Xiaoming Shi; Yankai Jiang; Yichen Li; Yujie Zhang

REVIEW 5 major objections 6 minor 3 cited by

Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning

T0 review · 5 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a medical multimodal model trained in three stages—cold-start supervised fine-tuning, self-reflection fine-tuning, and agentic tool reinforcement learning—learns to invoke image tools (zoom and segmentation) during rea

desk verdict A genuinely useful three-stage tool-reasoning recipe for medical MLLMs, with strong ablations and an honest same-tools comparison, but the headline public-benchmark numbers are undermined by an unreported training/evaluation overlap risk. read the letter →

arxiv 2512.14157 v2 pith:4DLLRZEV submitted 2025-12-16 cs.AI cs.CV

Yankai Jiang , Yujie Zhang , Peng Zhang , Wenjie Li , Yichen Li , Jintai Chen , Xiaoming Shi , Shihui Zhen This is my paper

classification cs.AIcs.CV

keywords medicalmultimodalLLMtool-augmentedreasoningagenticreinforcementlearningreasoning-basedsegmentationvisualquestionansweringself-reflectionfine-tuningfine-grainedgrounding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The authors are trying to establish that the bottleneck in medical multimodal reasoning is not model size but the inability to dynamically re-inspect fine-grained image regions. They propose Ophiuchus, a tool-augmented framework that interleaves textual chain-of-thought with calls to image tools, letting the model decide when to probe, where to look, and how to fold the returned visual evidence back into its reasoning. The key claim is that a three-stage training recipe—cold-start SFT, self-reflection fine-tuning, and agentic tool reinforcement learning—is what converts tool calling into genuine visual cognition, rather than a static pipeline. If correct, this would give medical AI agents a practical way to reduce hallucinations and false positives on tasks that require fine-grained spatial evidence. The paper reports consistent gains across eight benchmarks, including in-domain and out-of-domain VQA, detection, and reasoning-based segmentation.

What carries the argument

The load-bearing object is the interleaved reasoning trajectory, formally written as R = {(r_n, t_n, o_n)}_{n=1..N}, where each step contains natural-language thoughts, a tool invocation, and the resulting observation. Training on this object through the three stages teaches a policy for when to call a tool, where to probe, and whether to revise. The reinforcement learning stage uses fine-grained rewards: format adherence, exact-match (with IoU-thresholded piecewise rewards for segmentation), and a conditional tool-use bonus granted only when the model answers correctly and invokes at least one tool.

What would settle it

Take a held-out set of cases where an independent clinician panel's ground truth disagrees with the teacher labels used in training, and check whether tool invocations change answers in the direction of the clinician ground truth. If removing tool use—or replacing the teacher with a different generator—eliminates the gap, the claimed mechanism is not the one driving performance.

Watch

Extended reading notes

Core claim

Ophiuchus is a 7B-scale multimodal model trained to interleave textual thoughts, tool calls, and tool observations in a single reasoning loop. At each step the model may either answer directly or call one of three image tools—a text-prompted segmenter, a bounding-box-prompted segmenter, and a zoom/crop function—and the returned mask or crop is appended to the context as an observation that feeds the next thought. The three-stage training protocol is the central claim: cold-start supervised fine-tuning teaches basic tool selection; self-reflection fine-tuning on cases where the model corrected its own tool use strengthens revision behavior; and agentic tool reinforcement learning with rewards

Load-bearing premise

Everything rests on the teacher-generated data being clinically sound: the QA pairs and tool-use trajectories are synthesized by proprietary frontier models guided by ground-truth masks and then filtered by a verification model plus human cross-checks (described in Section 3.3 and Appendix A.4), so if those teachers systematically mislabel or hallucinate in ways the filters miss, Ophiuchus inherits the bias and part of the reported gains is an artifact of data construction.

Editorial extensions

If this is right

Pre-segmenting regions and feeding crops to a model is not enough: the paper shows a pipeline with ground-truth crops underperforms Ophiuchus by large margins, so the gains come from weaving observations into reasoning, not merely from zooming in.
The policy is not brittle to a specific segmenter: swapping the default segmentation tool for alternatives changes scores by less than a point on average, implying the trained behavior is tool-agnostic orchestration.
Ophiuchus can surpass the external tools themselves, including an ensemble of both segmenters, so the model's reflection and composition add value beyond the tools' own ceilings.
The training dynamics show an expand-compress pattern—tool calls rise early during exploration and fall during exploitation—and the final policy rarely calls tools on questions the base model already answers correctly, indicating learned efficiency.
Across eight benchmarks, the model reports consistent gains over closed-source and open-source SOTA, with the largest relative improvements on fine-grained in-domain VQA and segmentation subsets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the recipe transfers, 'think with images' could extend to non-medical domains where global context hides small signals; a testable extension is applying the same three-stage recipe to satellite or microscopy imagery and measuring whether tool-augmented reasoning outperforms simply scaling up the base model.
The dependence on proprietary teacher models for data generation means open reproduction hinges on releasing that data; a stress test would regenerate trajectories with a smaller open teacher, or add adversarial perturbations to the verification pipeline, to see whether the gains are robust or partly inherited from teacher bias.
The conditional tool-use reward points to a broader principle: reinforcement learning should reward evidence that changes outcomes, not tool invocation itself. A natural variant would reward information gain—how much the tool observation improves the model's predicted answer—rather than conditional correctness.
The paper's own failure case shows that even a reflective agent can still misdiagnose when all tools fail; clinical deployment would likely need an explicit abstain behavior or mandatory human confirmation, which the current reward design does not incentivize.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, the training recipe is real: cold-start SFT, self-reflection SFT, and GRPO with fine-grained tool rewards form a sensible progression, and the ablations back it up. Second, the abstract's SOTA claims are not yet defensible, because the paper never reports deduplication between the training corpora (BiomedParseData, Malenia) and the public VQA benchmarks. BiomedParseData is a large aggregation of public medical images, so overlap with PathVQA, SLAKE, VQA-RAD, or OmniMedVQA is plausible. If those exact images or near-duplicates appeared in training, the out-of-domain scores reflect memorization, not generalization. The held-out in-house set is only used for segmentation, not for the headline VQA numbers. That is a concrete, falsifiable threat to the central claim.

What the paper does well: the ablation design is the strongest part. Each training stage adds a clear increment; self-reflection SFT beats extra plain SFT epochs; the RL-only variant underuses tools; the conditional tool-use reward matters; and the same-tools/same-data comparison in Table 9 is the right kind of control. The training dynamics (tool-call count expand-then-compress) and the failure-case examples are genuinely informative. The verification pipeline for teacher trajectories is also more careful than most: GPT-o3 trace checking plus human cross-validation does not eliminate teacher bias, but it mitigates it.

Soft spots, in proportion. The leakage issue is the load-bearing one and needs a direct answer: report exact overlap checks, image hashing, and benchmark provenance. Without that, the 20-point gains over closed-source models are not credible. Second, all tables are single-run point estimates; no error bars or significance tests. That is common in this subfield, but it matters more when the claim is 'consistent outperformance.' Third, the abstract says 'detection,' but there is no standard detection benchmark in the paper; the experiments are VQA and reasoning-based segmentation. That is an overclaim, though not central.

Is the central recipe sound? I think yes, tentatively. The gains over the same-tools baselines suggest the training framework itself, not merely tool access, is doing work. But the paper has to close the leakage hole before the SOTA numbers can be taken at face value.

For a reader: if you work on medical MLLMs or agentic tool use, this is worth engaging. For a referee: yes, send it to review, but require a deduplication analysis and exact evaluation-split provenance before acceptance. If the overlap check is clean and artifacts are released, this is a solid systems contribution.

Referee Report

5 major / 6 minor

Summary. The paper proposes Ophiuchus, a 7B medical multimodal large language model that interleaves textual chain-of-thought reasoning with calls to external visual tools (zoom-in, SAM2, BiomedParse). The core contribution is a three-stage training recipe: cold-start supervised fine-tuning on 64k GPT-5/Gemini-generated reasoning traces, self-reflection fine-tuning on trajectories that flip from incorrect to correct across checkpoints, and GRPO-based agentic tool reinforcement learning with fine-grained rewards. The authors report consistent state-of-the-art results across eight medical VQA benchmarks and two segmentation test sets, and include ablations (Table 2), a same-tools/same-data comparison (Table 9), and robustness experiments across segmentation tools (Table 4).

Significance. If the empirical claims survive scrutiny, this is a genuinely useful contribution: it demonstrates a concrete training protocol for turning a general-purpose MLLM into a tool-orchestrating medical agent, with careful ablations isolating the contribution of each training stage and a held-out in-house test set. The paper also makes a serious effort to reduce circularity by evaluating on public benchmarks and by describing a multi-stage verification pipeline for teacher-generated data. The same-tools comparison in Table 9 and the tool-replacement robustness study in Table 4 are strong supporting pieces of evidence. However, the validation currently lacks two critical controls—training/test image-overlap analysis and statistical uncertainty quantification—and one comparison is under-specified. These are required before the headline claim of 'consistently outperforming SOTA' can be accepted.

major comments (5)

[§3.3, §4.2, Table 1] Ophiuchus is trained on BiomedParseData (3.4M triples) and Malenia, then evaluated on PathVQA, SLAKE, VQA-RAD, OmniMedVQA, MMMU(H&M), and MedXpertQA. The paper reports no deduplication between these training corpora and the public evaluation images. Since BiomedParseData aggregates large numbers of public biomedical images, exact or near-duplicate overlap is plausible and would directly inflate the headline gains in Table 1. Please report an overlap analysis (e.g., perceptual hashing or exact-match at the image level) between every training image and each benchmark, and re-report all Table 1 numbers on the non-overlapping subset. If overlap is found, state whether the conclusions survive after excluding those samples.
[Appendix A.1, Table 5] The in-domain Dtest-4k is described as a fixed-proportion split of the curated samples, but it is not stated that the split is performed by image identity. Because multiple QA/segmentation samples can be derived from the same image, the same image may appear in Dcold, Drl, and Dtest. That would bias the in-domain results (Dtest-VQA, Dtest-Seg). Please clarify the splitting unit; if samples were split without image-level grouping, re-split at the image level and report the affected numbers.
[Tables 1–4, 9–14] All reported numbers are single-run point estimates with no standard deviations, confidence intervals, or significance tests. Several head-to-head differences are small (e.g., Table 4 VQA-RAD: 73.6 vs 73.3; Table 11: 59.4 vs 59.0), and the claim of 'consistently outperforming' relies on ranking without noise assessment. Please add variance estimates (at least 3 training seeds or bootstrap over test items) for the main comparisons, and give a significance test for the primary benchmark table.
[Appendix C.4, Table 9] The same-tools/same-data comparison is central to the claim that the training framework, not the tool access or data, drives the gains. However, the description says only that open-source baselines were 'fine-tuned using the complete three-stage training data'; it does not state whether they were trained with the same cold-start SFT + self-reflection + ATRL algorithm and reward scheme, or simply SFT on the same data. If the latter, the comparison does not isolate the training method. Please clarify the exact training protocol for each baseline in Table 9, and if possible train one strong baseline with the identical three-stage recipe.
[§3.3, Appendix A.4] The teacher-based data generation and verification pipeline is the sole source of supervised tool-use trajectories. The manuscript does not report how many samples were removed at each filtering step, nor any audit of residual errors in the retained data (e.g., inter-annotator agreement on the human cross-validation). Since the in-domain test set is generated by the same pipeline, systematic teacher bias could inflate the in-domain numbers. Please report filtering statistics and a human-expert audit of a random sample of the retained trajectories, with agreement rates.

minor comments (6)

[Throughout] Typos and formatting: 'LLaV A-Next-13B' unspaced, 'Boldand' in Table 1 caption, 'formate' in the reward description, and 'we also adjusted' in Appendix C.6. The notation 'Mcold−12' is ambiguous; consider 'Mcold-12'.
[Figure 2] The figure is very dense. Add a legend or annotations to clarify the three training stages and the flow of reward signals.
[§3.3] Report the number of QA pairs discarded by format/answer checks and by the five-dimension verification, to substantiate the 'high-quality' claim.
[Table 3] Specify whether the GT region crop includes a margin and whether the same tool output images were used as observations for Ophiuchus.
[Table 9] Add a column indicating whether each baseline was prompted with tools only or fine-tuned, and with which stages; this makes the comparison easier to interpret.
[Appendix A.1] Clarify whether Dreflect (3k instances) is disjoint from Dcold and whether the early/late checkpoints used for self-reflection sampling came from Dcold or Drl.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the central claim is an empirical training/evaluation result with held-out and external benchmarks; self-citations are not load-bearing.

full rationale

The paper makes an empirical systems claim: a three-stage training recipe (cold-start SFT, self-reflection SFT, ATRL) improves tool-augmented medical VQA, detection, and segmentation. There is no derivation chain in which an equation reduces to its own inputs or a fitted parameter is renamed as a prediction. The training data is generated by GPT-5/Gemini-2.5-pro from ground-truth masks and verified by GPT-o3 plus human cross-validation, but the model is evaluated on public benchmarks (PathVQA, SLAKE, VQA-RAD, OmniMedVQA, MMMU, MedXpertQA) and on a separately human-annotated in-house set. The in-domain Dtest is a held-out split of the curated data, not a subset used for training; this is standard practice rather than circularity. The self-citations present (Chiron-o1 as a baseline, Malenia as a training dataset) are not used to justify the central claim. The skeptical concern about possible image-level overlap between training corpora and public benchmarks is a data-leakage/validity risk, not a circularity of the argument: nothing in the paper defines the outcome in terms of the predictor or derives the result from its own assumptions by construction. Overall, the claimed gains are empirically contingent and independently testable, so no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of external annotations, LLM-generated supervision, tool execution fidelity, and benchmark independence. No new physical entities are introduced. The reward design contributes two hand-set constants whose values are not varied.

free parameters (2)

IoU piecewise reward thresholds and values = IoU thresholds 0.5/0.7/0.8; rewards 0/1/2/3
Appendix B, Eq. 7 sets the segmentation reward by hand. These constants directly shape the RL learning signal and the reported segmentation performance, but no sensitivity analysis over them is provided.
Tool-invocation bonus B_bonus = 2
Appendix B, Eq. 8 grants a fixed bonus of 2 only when the answer is correct and a tool was used. The ablation in Fig. 4 removes or makes the bonus unconditional, but never varies its magnitude, so the chosen value is an unexamined hand-set constant.

assumptions (4)

domain assumption Ground-truth masks, labels, and descriptions in BiomedParseData and Malenia are accurate and clinically correct.
Sections 3.3 and A.1 use these annotations as the basis for QA generation, trajectory construction, and the segmentation reward; if they are wrong, the supervised signal and evaluation labels are compromised.
ad hoc to paper GPT-5 and Gemini-2.5-pro-generated QA pairs and tool-use trajectories are clinically correct after filtering; GPT-o3 and human cross-validation remove all material errors.
Section 3.3 and A.4: the cold-start SFT dataset consists of such generated traces. The paper does not quantify residual teacher-error rates, and the verification prompts still rely on another LLM's judgment.
domain assumption Tool execution in the environment (SAM2, BiomedParse, zoom-in) returns the observations the model is trained to reason over, and these outputs are reliable enough to ground reasoning.
Section 3.1 and Appendix B: observations are inserted verbatim and used as reasoning context. A wrong mask or crop can mislead the model; the paper shows only one qualitative failure case (Fig. 11) and no systematic recovery statistics.
domain assumption Public evaluation benchmarks are sufficiently independent of the training distribution to measure generalization.
Sections 4.1-4.2: the SOTA claim assumes no benchmark leakage. The authors note OmniMedVQA overlap only for Med-R1, not for their own training data, and the in-domain Dtest is admittedly split from the same curation pipeline as the training set.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning." pith.science (2026). https://pith.science/paper/4DLLRZEV

@misc{pith2026251214157,
  author       = {Pith},
  title        = {Pith review of: Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DLLRZEV}},
  note         = {Machine review of arXiv:2512.14157}
}

read the original abstract

Recent medical MLLMs have made significant progress in generating step-by-step textual reasoning chains. However, they still struggle with complex clinical tasks that necessitate dynamic and iterative focusing on fine-grained visual regions. To close this gap, we introduce Ophiuchus, a versatile, tool-augmented framework that equips an MLLM to (i) decide when fine-grained visual evidence is needed, (ii) determine where to probe and ground within the medical image, and (iii) seamlessly weave the relevant sub-image content back into an interleaved, multimodal chain of thought for precise segmentation and diagnosis. Ophiuchus moves beyond mere tool-calling by tightly fusing the MLLM's inherent grounding and reasoning capabilities with external tools, enabling more accurate and trustworthy decisions. The core of our method is a three-stage training strategy: cold-start SFT for basic tool selection; self-reflection fine-tuning to strengthen decision revision; and agentic tool reinforcement learning to elicit sophisticated, expert-like diagnostic behaviors. Extensive experiments show that Ophiuchus consistently outperforms both closed-source and open-source SOTA methods across diverse medical benchmarks, including VQA, detection, and reasoning-based segmentation. Our project code is available at https://github.com/SII-zyj/Ophiuchus.

Figures

Figures reproduced from arXiv: 2512.14157 by the authors.

**Figure 1.** Overview of the tool-augmented “thinking with images” paradigm. Compared with (a) text-only CoT, which fails to analyze fine-grained, task-critical image regions and thus limits understanding, increases hallucinations, and produces false positives, (b) our tool-augmented, interleaved vision-language reasoning adaptively generates effective tool-invocation strategies, inspects fine-grained regions, and integrates th… view at source ↗

**Figure 2.** Overview of the Ophiuchus. A versatile “think with images” framework that integrates zoom and segmentation tools, trained through cold-start SFT, self-reflection fine-tuning and ATRL. behavior over tool outputs. To this end, we introduce a novel SRS strategy that further strengthens the model’s self-correction capability. Specifically, we sample self-reflective cases from the model’s training dynamics. For challengi… view at source ↗

**Figure 3.** Performance comparison on medical Segmentation benchmarks. Colors indicate datasets (In-domain-Seg / In-house-Seg). We also provide performance of task-specific SOTA BiomedParse for reference. VQA Segmentation [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (17 more)

**Figure 5.** Figure 5: Training Dynamics [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 6.** Figure 6: Distribution of modalities in our proposed dataset. c Breast ancer Colon tumor Brain cancer Pancreatic cancer Liver tumor Kidney tumor Gallbladder tumor [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

**Figure 8.** Figure 8: Visual-Cue Search example. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]

**Figure 9.** Figure 9: Visual Confirmation example. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]

**Figure 10.** Figure 10: Hallucination Mitigation example. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]

**Figure 11.** Figure 11: Failure example (hard case). 31 [PITH_FULL_IMAGE:figures/full_fig_p031_11.png]

**Figure 12.** Figure 12: Self-Reflection example. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_12.png]

**Figure 13.** Figure 13: Comparative case analyses. It can be observed that both general-domain and medicalspecific agents augmented with tools frequently produce incorrect interpretations and answers when analyzing medical images, often accompanied by inaccurate localizations or segmentatio…

**Figure 14.** Figure 14: Cases illustrating how Ophiuchus combines different tools to further enhance segmentation performance. Ophiuchus can compose multiple tools—by using the zoom-in tool in conjunction with SAM 2—to achieve better segmentation results than relying on SAM 2 alone. 33 [PIT…

**Figure 15.** Figure 15: Cases illustrating that even when all segmentation tools fail, Ophiuchus can still answer the question correctly by leveraging all gathered visual cues to revise its own decision. SFT training loss (a) (b) (c) (d) (e) [PITH_FULL_IMAGE:figures/full_fig_p034_15.png]

**Figure 16.** Figure 16: Training curves of Ophiuchus. (a) Training loss during the SFT stage; (b) overall reward curve during RL training; (c) reasoning-format reward curve; (d) strategic tool-use reward curve; (e) IoU-based component of the final-answer reward. 34 [PITH_FULL_IMAGE:figures/…

**Figure 17.** Figure 17: Prompt for VQA construction. Template for generating clinically grounded questionanswer pairs under strict format control. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_17.png]

**Figure 18.** Figure 18: Prompt for VQA Verification. Template for assessing the quality of each generated VQA item. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_18.png]

**Figure 19.** Figure 19 [PITH_FULL_IMAGE:figures/full_fig_p037_19.png]

**Figure 20.** Figure 20: Prompt for Reasoning Trajectory Verification. Template for validating the correctness and reliability of generated reasoning trajectories. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_20.png]

**Figure 21.** Figure 21: System Prompt. Dialogue History <user> {{ Initial User Input }} </user> <think> Round 1 Model Thought </think> {{ Round 1 model output <tool_call> }} <obs> Round 1 Observation </obs> ... ... Question: {} The index of the given image is {current_image_idx} (width: {wid…

**Figure 22.** Figure 22: User Prompt 39 [PITH_FULL_IMAGE:figures/full_fig_p039_22.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents
cs.CV 2026-05 accept novelty 8.0 of 10

DeepTumorVQA is a new stage-wise 3D CT VQA benchmark showing that quantitative measurement is the main failure point for current medical VLMs and that tool augmentation substantially improves later reasoning stages.
MedScribe: Clinically Grounded CT Reporting through Agentic Workflows
cs.CV 2026-05 unverdicted novelty 6.0 of 10

MedScribe reformulates CT radiology reporting as an agentic evidence-acquisition workflow using LLM-invoked diagnostic tools and pathology-aligned retrieval, yielding higher clinical accuracy and consistency than stan...
UniReason-Med: A Shared Grounded Reasoning Interface for 2D-to-3D Transfer in Medical VQA
cs.CV 2026-06 unverdicted novelty 4.0 of 10

UniReason-Med introduces a unified framework for 2D and 3D medical VQA with shared grounded reasoning, trained on a 220K dataset, claiming that joint 2D+3D supervision improves 3D performance over 3D-only training.

Reference graph

Works this paper leans on

16 extracted references · cited by 3 Pith papers

[1]

Create one multiple-choice question that is directly answerable by analyzing the given image and the specified target region
[2]

Provide four or five options (A–D or A–E), using clear and professional medical terminology
[3]

Indicate the correct answer explicitly
[4]

Question

Return the result strictly in the following JSON format: [{ "Question": "xxx", "Options": { "A": "xxx", "B": "xxx", "C": "xxx", "D": "xxx" }, "Answer": "A" }] Notes: • The question must be independent and self-contained; avoid referential terms such as “this case” or “the image above”. Each question should stand alone. • Only use the information provided ...
[5]

Image Resolution: {resolution} (expected: 1024×1024)
[6]

Target RegionBBox: • {bbox} in the format ["bbox": [x1, y1, x2, y2]] (top-left = (x1,y1), bottom-right = (x2,y2)) • Binary Mask: {mask} (1024×1024)
[7]

Mask Descriptions: {mask_descriptions} (textual descriptors of the segmented region)
[8]

Question

Generated QA JSON: {qa_json} • Expected schema: [{ "Question": "string", "Options": { "A": "string", "B": "string", "C": "string", "D": "string", "E": "string (optional)" }, "Answer": "A/B/C/D/E" }] Validation Criteria Evaluate only using the provided inputs. Do not introduce external knowledge or patient context that isn’t present

Show all 16 references

[9]

• Keys Question, Options, Answer exist

Schema & Format • JSON is valid and parseable. • Keys Question, Options, Answer exist. • Options contains 4 or 5 choices labeled A–D (and optionally E) only. • Answer is exactly a single uppercase letter among the provided options (A/B/C/D/E)
[10]

• Professional, precise clinical terminology; no vague or colloquial phrasing

Independence & Wording • The question is self-contained; avoids referential terms. • Professional, precise clinical terminology; no vague or colloquial phrasing
[11]

• High correlation with the target region: content should pertain to structures/findings inside or tightly adjacent to the masked area

Grounding to Target Region • The question is directly answerable from the image with attention to the segmented target region (use the bbox and mask to focus reasoning). • High correlation with the target region: content should pertain to structures/findings inside or tightly ...
[12]

All of the above,

Options Quality • Options are mutually exclusive, non-overlapping, and plausible; one best answer. • No trivial cues (e.g., “All of the above,” “None of the above”), and no duplicate or near-duplicate options
[13]

question not grounded to target region

Answer Correctness • Determine the ground-truth choice by analyzing the image with the bbox/mask focus and the mask descriptions. • Verify the provided Answer matches the correct option. Output (STRICT) Return exactly one of the following: • If JSON/schema/options/answer forma...
[14]

Break down the problem: what is being asked, and what information or measurements might be needed from the image? Plan an approach before using any tool

Initial Analysis: Begin by examining the question and image carefully. Break down the problem: what is being asked, and what information or measurements might be needed from the image? Plan an approach before using any tool
[15]

liver tumor

Iterative Tool Reasoning: Tackle the task step-by-step using the tools: • Choose the appropriate tool for the current sub-task (e.g. use BiomedParse to detect and segment a tumor by specifying the object type, use SAM2 if you have or need a specific bounding box focus for segm...
[16]

• End your reasoning chain with an Answer action

Formulating the Final Answer: Once you have all necessary information (for example, the tumor is segmented and you’ve derived the size or other required insights from the image), formulate the final answer. • End your reasoning chain with an Answer action. Structure the final ...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Create one multiple-choice question that is directly answerable by analyzing the given image and the specified target region

[2] [2]

Provide four or five options (A–D or A–E), using clear and professional medical terminology

[3] [3]

Indicate the correct answer explicitly

[4] [4]

Question

Return the result strictly in the following JSON format: [{ "Question": "xxx", "Options": { "A": "xxx", "B": "xxx", "C": "xxx", "D": "xxx" }, "Answer": "A" }] Notes: • The question must be independent and self-contained; avoid referential terms such as “this case” or “the image above”. Each question should stand alone. • Only use the information provided ...

[5] [5]

Image Resolution: {resolution} (expected: 1024×1024)

[6] [6]

Target RegionBBox: • {bbox} in the format ["bbox": [x1, y1, x2, y2]] (top-left = (x1,y1), bottom-right = (x2,y2)) • Binary Mask: {mask} (1024×1024)

[7] [7]

Mask Descriptions: {mask_descriptions} (textual descriptors of the segmented region)

[8] [8]

Question

Generated QA JSON: {qa_json} • Expected schema: [{ "Question": "string", "Options": { "A": "string", "B": "string", "C": "string", "D": "string", "E": "string (optional)" }, "Answer": "A/B/C/D/E" }] Validation Criteria Evaluate only using the provided inputs. Do not introduce external knowledge or patient context that isn’t present