Pith. sign in

REVIEW 3 major objections 3 minor 15 references

VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VespaSeg claims a compact ground-then-segment pipeline can match larger referring-expression segmentation models while using far less GPU memory.

desk verdict Honest, artifact-backed evaluation of a modular ground-then-segment pipeline; the headline mIoU is real on its own audited protocol but likely optimistic on unseen data, as the authors themselves concede. read the letter →

arxiv 2608.01077 v1 pith:LMQ23CD5 submitted 2026-08-02 cs.RO

Savindu Dilshan Wickramasinghe (University of Moratuwa) This is my paper · ORCID
classification cs.RO
keywords referringexpressionsegmentationvisualgroundingMobileSAMFlorence-2Moondream2LoRAparameter-efficientfine-tuningRefCOCO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VespaSeg tries to show that referring-expression segmentation does not require a large monolithic model: a small vision–language model that predicts a box, followed by MobileSAM turning the box into a mask, can be competitive while using far less memory. On a RefCOCO validation protocol that keeps the first expression for each of 3,811 object records, the adapted Florence-2-base pipeline reaches 73.64 mIoU and 84.60 P@0.5, and on a matched 500-query subset it beats Florence-2-large while running 1.70 times faster and using 1.17 GB less allocated GPU memory. The paper is explicit that these numbers are not comparable to the full 10,834-expression RefCOCO validation split or to testA/testB, and that the checkpoint was selected on validation behavior. The value of the claim is that it maps a concrete, reproducible operating point where compact modular components suffice.

What carries the argument

The load-bearing object is the explicit box interface between two stages: a compact vision–language model (Florence-2-base or Moondream2) grounds the expression $q$ into a box $\hat b$, and MobileSAM, prompted with that box, produces the mask. MobileSAM's image embedding is computed once per image and reused for multiple expressions, which is what makes the reported cached-image throughput possible. Low-rank adaptation (LoRA) on both the grounder's attention and feed-forward projections and on MobileSAM's mask-decoder linear layers carries the accuracy gains; the box interface is what lets the two components be adapted and measured independently.

What would settle it

Run the same adapted Florence-2-base plus MobileSAM pipeline on all 10,834 RefCOCO validation expressions, and on testA/testB, using a checkpoint not selected during development on that validation set. If the resulting mIoU is materially below 73.64 on the full validation split, or if the base-versus-large margin on the matched subset reverses with multiple random seeds, the central accuracy and resource-efficiency claims would be weakened.

Watch

Extended reading notes

Core claim

The central claim is that separating language-conditioned localization from mask prediction is sufficient: an adapted compact grounder outputs a box, and MobileSAM converts that box to a mask. Under the repository-specific RefCOCO protocol (sentence index zero for each of 3,811 validation records), the adapted Florence-2-base pipeline obtains 73.64 mIoU and 84.60 P@0.5; a matched 500-pair comparison gives 73.73 mIoU for base versus 72.82 for large, with the base model 1.70 times faster and 1.17 GB lighter in allocated memory. The paper attributes the large gain over the unadapted grounder to RefCOCO-specific grounding, and shows with ground-truth boxes that MobileSAM mask-decoder LoRA raises mask IoU from 82.22 to 86.61 on a 1,000-record subset. It also shows that reducing Florence-2's output-token budget from 64 to 32 tokens is accuracy-neutral.

Load-bearing premise

The reported accuracy rests on a custom validation protocol that uses only the first expression for each of 3,811 records and a checkpoint chosen while watching that same validation set, so if first expressions are easier or the selection overfit, the 73.64 mIoU overstates performance on the full benchmark.

Editorial extensions

If this is right

  • A modular ground-then-segment architecture can hit a useful accuracy-resource operating point: 73.64 mIoU with 2.20 GB mean allocated memory and 22.8 cached-image queries per second on an RTX 6000 Ada.
  • At this operating point, a larger grounder does not help: Florence-2-large is slightly less accurate, slower, and uses more memory on the matched subset.
  • The output-token budget of the compact grounder can be halved from 64 to 32 without changing accuracy or memory, making the pipeline cheaper to run.
  • Adapting the mask decoder on ground-truth boxes improves segmentation quality by 4.39 mIoU points, independent of grounding error.
  • The pipeline's claimed performance should be re-measured on the complete standard RefCOCO splits and on edge hardware before being treated as deployment-ready.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If first expressions per record are systematically easier than later ones, the same pipeline on all 10,834 validation expressions would likely score below 73.64 mIoU; this is testable with the released code.
  • The cached-image throughput measures interactive multi-query use, not video frames: an end-to-end frame rate would additionally include the MobileSAM image encoder for every new image.
  • Because the box interface decouples the stages, future improvements to compact grounding models should transfer directly to this pipeline without retraining the segmentation stage.
  • Since no held-out test set or multiple seeds were used, part of the reported margin may reflect selection on validation behavior; a multi-seed run would bound the effect.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes VespaSeg, a modular ground-then-segment pipeline for referring expression segmentation: a compact vision-language model (Florence-2-base, Florence-2-large, or Moondream2) predicts a bounding box, and MobileSAM converts the box to a mask. The authors report results under a deliberately scoped, repository-specific RefCOCO validation protocol consisting of the first expression for each of 3,811 validation records. On this protocol, the LoRA-adapted Florence-2-base pipeline achieves 73.64 mIoU and 84.60 P@0.5, with 22.8 cached-image queries per second and 2.20 GB mean allocated GPU memory. A matched 500-query comparison gives 73.73 mIoU for base and 72.82 for large, with base faster and more memory-efficient. Ablations indicate that ground-truth-box MobileSAM adaptation improves mask mIoU from 82.22 to 86.61 and that reducing the Florence-2 output-token budget from 64 to 32 preserves accuracy. The manuscript repeatedly and explicitly cautions that the protocol is not the full RefCOCO validation split, that testA/testB are not evaluated, and that checkpoint selection used validation behavior.

Significance. If the reported numbers are taken as faithful measurements of the described repository protocol, the work has genuine value: it demonstrates that a compact, modular design can be competitive with a larger grounder within that protocol while using substantially less memory and offering higher cached-query throughput. The paper is commendably transparent about the non-standard evaluation: it names the repository-specific protocol, defines cached-image throughput precisely, discloses that the retained checkpoint was selected on validation behavior, and makes code and artifacts available. The resource measurements are potentially useful for practitioners considering modular RES systems. However, the headline accuracy claim is not an unbiased estimate of pipeline performance on the full RefCOCO benchmark or on unseen data, and the absence of uncertainty intervals means the base-versus-large accuracy comparison is not statistically supported. The paper's significance is therefore conditional on strengthening the evaluation and on readers accepting the carefully worded scope limitations.

major comments (3)
  1. [Sections 4.1, 5.1, and 6] The central accuracy claim rests on a number that is likely optimistic because of two coupled protocol choices. Evaluation uses only sentence index zero for each of 3,811 RefCOCO validation records rather than all 10,834 expressions, and testA/testB are not evaluated. More importantly, Section 5.1 states that the retained Florence-2-base checkpoint was selected during development on validation behavior, and Section 6 repeats that checkpoint selection used validation behavior. The 73.64 mIoU is therefore a best-of-validation estimate, not an unbiased estimate of pipeline accuracy under the stated protocol. If first expressions are systematically easier, or if selection exploited noise in the 3,811-pair protocol, the headline number could overstate pipeline accuracy by several points. The authors disclose this, but the consequence is load-bearing: the paper's main quantitative claim cannot currently be distinguished from an artifact of validation-based selection. A held-out evaluation (e.g., testA/testB, or a properly separated development/evaluation split with multiple seeds) is needed before the headline accuracy can be taken as evidence of pipeline capability.
  2. [Tables 1 and 2] No error bars, multiple seeds, or significance tests are reported, so the accuracy differences are not statistically characterized. In Table 2, the base model's 73.73 mIoU versus large's 72.82 mIoU on 500 pairs is a 0.91-point difference that is very likely within run-to-run variation, especially given checkpoint selection on validation data. The throughput and memory comparisons are more credible because those measurements are deterministic, but the claim that 'increasing grounder capacity does not improve this particular operating point' should be rephrased as a statement about the selected checkpoints on this subset, not a general inference. Reporting mean and standard deviation over at least three seeds, or a paired bootstrap interval for the 500-pair comparison, would materially strengthen the resource-versus-accuracy conclusions.
  3. [Section 5.3 and Table 3] The component ablations change more than one variable in ways that are not fully controlled. The 'Moondream2 none' versus 'Moondream2 grounder full FT + MobileSAM LoRA' rows in Table 1 differ in both the grounder and the mask decoder, so the 1-point gain cannot be attributed to the grounder fine-tuning alone. In Table 3, the MobileSAM ablation uses a 1,000-record subset and the Moondream2 ablation uses a 500-record subset, and the two rows measure different metrics, so only within-row comparisons are valid. The authors acknowledge these limitations, but Section 5.1's statement that 'RefCOCO-specific grounding is the principal source of improvement' is stronger than the controlled evidence supports. I would ask the authors to either add a controlled comparison where only the grounder is changed while MobileSAM is held fixed, or soften the attribution to the level supported by the data.
minor comments (3)
  1. [Section 4.1, Eq. (5)] The definition of mIoU and P@0.5 is clear, but the typesetting of the P@0.5 formula appears corrupted ('N−1 ˝ i ⊮'); please fix the mathematical notation so the indicator function and indexing are legible.
  2. [Sections 4.1 and 5.1] The term 'record' is used for both validation records and training records, while 'expression' is used for individual referring expressions. Since the distinction between a record and an expression is critical to understanding the protocol, please define 'record' at first use and consistently refer to 'instance–query pairs' or 'expression pairs' in the accuracy tables.
  3. [Figure 2] The left panel labels the axes as throughput versus mIoU, but the text says the frontier is 'shallow'; please make clear whether the points are connected as a convex frontier or just individual configurations, and add the numerical mIoU labels to the figure so readers do not need to refer back to Table 4.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all reported quantities are measured under a disclosed protocol, and the validation-selection risk is explicitly acknowledged rather than presented as a derived prediction.

full rationale

The paper makes no predictive derivation that reduces to its own inputs. Equations (1) and (2) define the ground-then-segment inference procedure, and Equation (5) defines the evaluation metric; these are standard definitions, not circular claims. The adaptation procedure in Equation (3) is a standard LoRA update, and the reported accuracies, throughputs, and memory figures are measurements on a specifically named 'repository-specific RefCOCO validation protocol' (Section 4.1). The strongest result, 73.64 mIoU, is presented as an observed value on that protocol, and the paper repeatedly and explicitly states that the checkpoint was selected using validation behavior: Section 5.1 says 'the retained checkpoint was selected during development on validation behavior,' and Section 6 repeats that 'Development and checkpoint selection also used validation behavior, so the reported accuracy may be optimistic.' This is a disclosed statistical-optimism risk, not a circularity: the measured number does not serve as the input to its own derivation, and the paper does not claim to predict it from fitted parameters. No self-citations are load-bearing, no uniqueness theorem is imported, and no known result is renamed. The only interpretive concern is that the reported accuracy may be optimistic relative to the full RefCOCO splits, which the paper itself flags as a limitation rather than concealing. Therefore, there is no circular step, and the appropriate score is 0.

Assumptions & free parameters 8 free parameters · 3 assumptions · 0 invented entities

The central claims rest on manually chosen hyperparameters, a custom evaluation subset, and two transfer assumptions about ground-truth box gains and cached-image timing. No new theoretical entities are introduced.

free parameters (8)
  • Grounder LoRA rank r = 8
    Chosen by hand for Florence-2-base and Florence-2-large; controls adapter capacity and is not swept.
  • Grounder LoRA alpha = 16
    Manual LoRA scale used for all Florence-2 grounders.
  • Florence-2 learning rate = 1e-4 (AdamW)
    Manual choice; no learning-rate sweep is reported.
  • Florence-2 training epochs = 3 (base uses epoch-2 state)
    Training length; the retained base checkpoint was selected by validation behavior, a load-bearing choice.
  • MobileSAM LoRA rank and alpha = 8 and 16
    Hand-selected for mask-decoder adaptation; no sweep reported.
  • Moondream2 learning rate = 1e-5
    Manual choice for full fine-tuning.
  • Input image max side = 320 pixels
    Operating point chosen from a small resolution sweep; 256 and 192 pixels were also tested.
  • Output token budget = 32 tokens
    Operating point; the paper shows reducing from 64 to 32 preserves measured accuracy.
assumptions (3)
  • domain assumption The RefCOCO repository protocol, using only the first expression per validation record, adequately represents performance on the full standard RefCOCO validation split.
    Sections 4.1 and 6 state that 3,811 pairs are used instead of 10,834 and that results must not be placed on a standard leaderboard; the accuracy claim depends on this subset being a fair proxy.
  • domain assumption Mask adaptation measured with ground-truth boxes transfers to the full pipeline with predicted boxes.
    Section 5.3 improves MobileSAM mIoU from 82.22 to 86.61 on ground-truth boxes; the end-to-end gain assumes this holds when boxes come from the grounder.
  • domain assumption Cached-image query throughput is the relevant deployment metric for the reported speeds.
    Section 4.2 reuses MobileSAM image embeddings across queries and Section 6 clarifies the number is not video-frame throughput; the resource claim is meaningful only for interactive multi-query use.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation." pith.science (2026). https://pith.science/paper/LMQ23CD5

@misc{pith2026260801077,
  author       = {Pith},
  title        = {Pith review of: VespaSeg: A Resource-Aware Ground-then-Segment Pipeline for Referring Expression Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMQ23CD5}},
  note         = {Machine review of arXiv:2608.01077}
}
read the original abstract

Referring expression segmentation requires language conditioned localization and pixel-accurate masks, but monolithic models can be costly to deploy. We present VespaSeg, a modular pipeline that grounds a text query with a compact vision-language model and converts the predicted box to a mask with MobileSAM. We study Florence-2-base, Florence-2-large, and Moondream2 grounders together with targeted adaptation of the grounding and segmentation stages. Under a repository-specific RefCOCO validation protocol containing the first expression for each of 3,811 referenced-object records, the adapted Florence-2-base pipeline obtains 73.64 mean intersection over union (mIoU) and 84.60 precision at IoU 0.5. On an NVIDIA RTX 6000 Ada GPU it processes 22.8 cached-image queries per second with 2.20 GB mean allocated GPU memory. A matched 500-query comparison gives 73.73 mIoU for Florence-2-base and 72.82 for Florence-2-large, while the base model is 1.70 times faster and uses 1.17 GB less allocated memory. Ablations show that ground-truth-box adaptation raises MobileSAM mIoU from 82.22 to 86.61 and that reducing the Florence-2 output-token budget from 64 to 32 preserves accuracy. These results support compact, modular grounding and segmentation, while also exposing the need for evaluation on the complete standard RefCOCO expression splits and deployment hardware.

Figures

Figures reproduced from arXiv: 2608.01077 by the authors.

Figure 1
Figure 1. VespaSeg separates language-conditioned box [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Artifact-backed resource trade-offs. Left: the five non-compiled configurations in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Selected high-improvement RefCOCO validation examples from a 200-pair diagnostic subset. Each montage shows the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 3 canonical work pages

  1. [1]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. InInternational Conference on Learning Representations. arXiv:2106.09685

    arXiv 2022

  2. [2]

    Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, and Suha Kwak. 2022. Restr: Convolution-Free Referring Image Segmentation Using Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. arXiv:2203.16768

    work page Pith review arXiv 2022

  3. [3]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. InProceedings of the IEEE/CVF International Conference on Computer Vision. 4015–4026. arXiv:2304.02643

    arXiv 2023

  4. [4]

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2024. LISA: Reasoning Segmentation via Large Language Model. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. arXiv:2308.00692

    arXiv 2024

  5. [5]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Com- mon Objects in Context. InEuropean Conference on Computer Vision. 740–755. arXiv:1405.0312

    arXiv 2014

  6. [6]

    Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. InInternational Conference on Learning Representations. arXiv:1711.05101

    arXiv 2019

  7. [7]

    Moondream AI. 2025. Moondream2 Model Card. https://huggingface.co/ vikhyatk/moondream2. Version 2025-06-21; accessed 2026-08-02

    work page 2025

  8. [8]

    Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. 2022. CRIS: CLIP-Driven Referring Image Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. arXiv:2111.15174

    arXiv 2022

Show all 15 references
  1. [9]

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. 2022. TinyViT: Fast Pretraining Distillation for Small Vision Trans- formers. InEuropean Conference on Computer Vision. 68–85. arXiv:2207.10666

    2022 arXiv

  2. [10]

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. 2024. Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4818–...

    2024 arXiv

  3. [11]

    Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip H. S. Torr. 2022. LAVT: Language-Aware Vision Transformer for Referring Image Segmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. arXiv:2112.02244

    2022 arXiv

  4. [12]

    Berg, and Tamara L

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg

  5. [13]

    Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bae, Seungkyu Lee, and Choong Seon Hong. 2023. Faster Segment Anything: Towards Light- weight SAM for Mobile Applications.arXiv preprint arXiv:2306.14289(2023)

    2023 arXiv

  6. [14]

    Yuxuan Zhang, Tianheng Cheng, Rui Hu, Lei Liu, Heng Liu, Longjin Ran, Xi- aoxin Chen, Wenyu Liu, and Xinggang Wang. 2024. EVF-SAM: Early Vision- Language Fusion for Text-Prompted Segment Anything Model.arXiv preprint arXiv:2406.20076(2024). 4 VespaSeg Figure 3: Selected high-i...

    2024 arXiv

  7. [2016]

    InEuropean Conference on Computer Vision

    Modeling Context in Referring Expressions. InEuropean Conference on Computer Vision. 69–85. arXiv:1608.00272

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.