{"id":"4fcaa07f-d2ac-469e-a196-5f0878f5b02a","arxiv_id":"2608.01077","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A modular ground-then-segment pipeline using Florence-2-base and MobileSAM reaches 73.64 mIoU on a repository-specific RefCOCO validation subset at 22.8 cached-image queries per second.","lead":"VespaSeg is a two-stage pipeline that first locates an object described in a sentence and then produces its pixel-accurate mask, using small models that need little GPU memory. It reports strong accuracy and much lower resource use than a larger alternative model, under a self-described non-standard evaluation protocol.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation-selected checkpoint on a first-expression-only subset risks inflating the reported 73.64 mIoU; the paper's own limitations concede this.","rationale":"The reader's weakest_assumption correctly identifies the first-expression-only protocol and validation-based checkpoint selection as the main threat to the central claim. My review agrees: the paper's own Section 6 explicitly concedes that the reported accuracy may be optimistic, and no held-out test result or multiple-seed analysis is provided. This is not a fatal flaw because the paper is scoped to a repository-specific protocol and is unusually transparent about its limitations, but it does mean the 73.64 mIoU figure should be treated as an upper-bound estimate on a custom protocol rather than a competitive RefCOCO result. The reader's CONDITIONAL verdict is appropriate; my concern does not move the verdict. I also considered the cached-image throughput metric, but since the paper precisely defines it as not video-frame throughput and uses it only for an interactive multi-query use case, that is not a load-bearing issue. The central weakness remains the lack of an unbiased held-out evaluation of the headline accuracy.","tokens_in":174,"tokens_out":2726,"duration_ms":34183,"concrete_test":"Run the released evaluation code with the retained adapted Florence-2-base + MobileSAM checkpoint on all 10,834 validation expressions (not only sentence index zero) and on the standard RefCOCO testA and testB splits. If the full-validation mIoU is more than about 2 points below 73.64 or if testB mIoU is materially lower (e.g., below 70), then the reported number is protocol-dependent and the headline should be re-scoped to the repository protocol. Additionally, retrain the grounder LoRA with 5 random seeds, select each seed's checkpoint on a separate held-out validation fold, and report the mean and 95% CI on testA/testB; this would quantify optimizer noise and selection bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim rests on a number that is likely optimistic because of two coupled protocol choices. First, evaluation uses only sentence index zero for each of 3,811 RefCOCO validation records (Sections 4.1, 5.1), not the full 10,834-expression validation split or testA/testB. Second, the Florence-2-base LoRA checkpoint was selected during development on validation behavior (Section 5.1: \"the retained checkpoint was selected during development on validation behavior\"); Section 6 repeats this: \"Development and checkpoint selection also used validation behavior, so the reported accuracy may be optimistic.\" This means the 73.64 mIoU is an estimate of the best-of-validation accuracy, not an unbiased estimate of pipeline performance. The paper does not provide multiple seeds, uncertainty intervals, or significance tests. If first expressions are systematically easier (shorter, less ambiguous, more likely to match the box decoder), and if validation selection exploited noise in the 3,811-pair protocol, the headline number could overstate the pipeline's accuracy on unseen expressions by several points. The matched 500-pair base/large comparison (Table 2) inherits the same selection effects, so the conclusion that base is competitively accurate is also contingent on this protocol. Without a held-out evaluation, the central claim \"73.64 mIoU and 84.60 P@0.5\" cannot be distinguished as either a real modular-pipeline capability or an artifact of fitting the validation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VespaSeg, a modular ground-then-segment pipeline for referring expression segmentation: a compact vision-language model (Florence-2-base, Florence-2-large, or Moondream2) predicts a bounding box, and MobileSAM converts the box to a mask. The authors report results under a deliberately scoped, repository-specific RefCOCO validation protocol consisting of the first expression for each of 3,811 validation records. On this protocol, the LoRA-adapted Florence-2-base pipeline achieves 73.64 mIoU and 84.60 P@0.5, with 22.8 cached-image queries per second and 2.20 GB mean allocated GPU memory. A matched 500-query comparison gives 73.73 mIoU for base and 72.82 for large, with base faster and more memory-efficient. Ablations indicate that ground-truth-box MobileSAM adaptation improves mask mIoU from 82.22 to 86.61 and that reducing the Florence-2 output-token budget from 64 to 32 preserves accuracy. The manuscript repeatedly and explicitly cautions that the protocol is not the full RefCOCO validation split, that testA/testB are not evaluated, and that checkpoint selection used validation behavior.","tokens_in":6664,"tokens_out":2628,"duration_ms":25372,"significance":"If the reported numbers are taken as faithful measurements of the described repository protocol, the work has genuine value: it demonstrates that a compact, modular design can be competitive with a larger grounder within that protocol while using substantially less memory and offering higher cached-query throughput. The paper is commendably transparent about the non-standard evaluation: it names the repository-specific protocol, defines cached-image throughput precisely, discloses that the retained checkpoint was selected on validation behavior, and makes code and artifacts available. The resource measurements are potentially useful for practitioners considering modular RES systems. However, the headline accuracy claim is not an unbiased estimate of pipeline performance on the full RefCOCO benchmark or on unseen data, and the absence of uncertainty intervals means the base-versus-large accuracy comparison is not statistically supported. The paper's significance is therefore conditional on strengthening the evaluation and on readers accepting the carefully worded scope limitations.","major_comments":[{"comment":"The central accuracy claim rests on a number that is likely optimistic because of two coupled protocol choices. Evaluation uses only sentence index zero for each of 3,811 RefCOCO validation records rather than all 10,834 expressions, and testA/testB are not evaluated. More importantly, Section 5.1 states that the retained Florence-2-base checkpoint was selected during development on validation behavior, and Section 6 repeats that checkpoint selection used validation behavior. The 73.64 mIoU is therefore a best-of-validation estimate, not an unbiased estimate of pipeline accuracy under the stated protocol. If first expressions are systematically easier, or if selection exploited noise in the 3,811-pair protocol, the headline number could overstate pipeline accuracy by several points. The authors disclose this, but the consequence is load-bearing: the paper's main quantitative claim cannot currently be distinguished from an artifact of validation-based selection. A held-out evaluation (e.g., testA/testB, or a properly separated development/evaluation split with multiple seeds) is needed before the headline accuracy can be taken as evidence of pipeline capability.","section":"Sections 4.1, 5.1, and 6"},{"comment":"No error bars, multiple seeds, or significance tests are reported, so the accuracy differences are not statistically characterized. In Table 2, the base model's 73.73 mIoU versus large's 72.82 mIoU on 500 pairs is a 0.91-point difference that is very likely within run-to-run variation, especially given checkpoint selection on validation data. The throughput and memory comparisons are more credible because those measurements are deterministic, but the claim that 'increasing grounder capacity does not improve this particular operating point' should be rephrased as a statement about the selected checkpoints on this subset, not a general inference. Reporting mean and standard deviation over at least three seeds, or a paired bootstrap interval for the 500-pair comparison, would materially strengthen the resource-versus-accuracy conclusions.","section":"Tables 1 and 2"},{"comment":"The component ablations change more than one variable in ways that are not fully controlled. The 'Moondream2 none' versus 'Moondream2 grounder full FT + MobileSAM LoRA' rows in Table 1 differ in both the grounder and the mask decoder, so the 1-point gain cannot be attributed to the grounder fine-tuning alone. In Table 3, the MobileSAM ablation uses a 1,000-record subset and the Moondream2 ablation uses a 500-record subset, and the two rows measure different metrics, so only within-row comparisons are valid. The authors acknowledge these limitations, but Section 5.1's statement that 'RefCOCO-specific grounding is the principal source of improvement' is stronger than the controlled evidence supports. I would ask the authors to either add a controlled comparison where only the grounder is changed while MobileSAM is held fixed, or soften the attribution to the level supported by the data.","section":"Section 5.3 and Table 3"}],"minor_comments":[{"comment":"The definition of mIoU and P@0.5 is clear, but the typesetting of the P@0.5 formula appears corrupted ('N−1 ˝ i ⊮'); please fix the mathematical notation so the indicator function and indexing are legible.","section":"Section 4.1, Eq. (5)"},{"comment":"The term 'record' is used for both validation records and training records, while 'expression' is used for individual referring expressions. Since the distinction between a record and an expression is critical to understanding the protocol, please define 'record' at first use and consistently refer to 'instance–query pairs' or 'expression pairs' in the accuracy tables.","section":"Sections 4.1 and 5.1"},{"comment":"The left panel labels the axes as throughput versus mIoU, but the text says the frontier is 'shallow'; please make clear whether the points are connected as a convex frontier or just individual configurations, and add the numerical mIoU labels to the figure so readers do not need to refer back to Table 4.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest and appropriately scoped, but the evaluation is currently too weak to support the headline accuracy claim as stated. The authors explicitly concede the two main problems: the first-expression-only validation protocol and validation-based checkpoint selection. I see no evidence of deliberate circularity; the issue is statistical optimism, not a logically circular derivation. If the authors add a held-out evaluation (testA/testB or a properly split development set with multiple seeds), report uncertainty intervals, and tighten the attribution claims in Section 5.1, the paper could become acceptable. Without those additions, the central quantitative conclusion remains unsupported. I would not reject outright because the resource measurements and the matched base-versus-large comparison are useful and reproducible, provided the authors address the above concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful, honest empirical paper, but the headline 73.64 mIoU is protocol-bound and likely optimistic on unseen expressions. The authors know this and say so.\n\nWhat's new: VespaSeg reuses known components (Florence-2 as grounder, MobileSAM as segmenter, LoRA), so the architecture isn't novel. The value is in the artifact-backed comparison: a matched 500-pair base/large study showing the base is slightly more accurate, 1.7x faster, and 1.17 GB lighter; a clean ground-truth-box ablation isolating MobileSAM LoRA gains (82.22 to 86.61); and a token-budget test showing 32 tokens is free. The resource measurements use cached-image throughput and allocated PyTorch memory, defined carefully. The GitHub artifacts are the source of truth. The paper is refreshingly candid: it labels the 3,811-pair first-expression protocol as repository-specific, runs no testA/testB, and explicitly warns that validation-based checkpoint selection may make accuracy optimistic.\n\nSoft spots, in proportion: the stress-test note is right. Two coupled choices—first-expression-per-record subset and checkpoint selected on that subset during development—mean 73.64 mIoU is a best-of-validation number, not an unbiased estimate. No multiple seeds or significance tests. The matched 500-pair comparison inherits the same selection. But these are disclosed limitations, not hidden ones. The paper's own Section 6 asks for re-evaluation on full standard splits and held-out checkpoints. So the flaw is real but contained: it caps the claim to the audited protocol, and the authors don't overstate.\n\nCitation pattern is fine: relevant LAVT, CRIS, ReSTR, LISA, EVF-SAM, SAM, Florence-2, LoRA all cited. No suspicious self-citation.\n\nWho is this for: people building or evaluating modular RES on constrained hardware, and anyone wanting a careful budget-accuracy trade-off study with reproducible artifacts. It's not a state-of-the-art claim.\n\nRecommendation: engage with it. As a referee, I'd accept with major revision: ask for full standard splits (testA/testB), multiple seeds, and error bars, and clarify how much the first-expression subset differs from full validation. The core contribution—modular pipeline plus resource-aware comparison—is solid enough to deserve referee time.","headline":"Honest, artifact-backed evaluation of a modular ground-then-segment pipeline; the headline mIoU is real on its own audited protocol but likely optimistic on unseen data, as the authors themselves concede.","tokens_in":7186,"tokens_out":1963,"would_cite":true,"duration_ms":17890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VespaSeg claims a compact ground-then-segment pipeline can match larger referring-expression segmentation models while using far less GPU memory.","keywords":["referring expression segmentation","visual grounding","MobileSAM","Florence-2","Moondream2","LoRA","parameter-efficient fine-tuning","RefCOCO"],"falsifier":"Run the same adapted Florence-2-base plus MobileSAM pipeline on all 10,834 RefCOCO validation expressions, and on testA/testB, using a checkpoint not selected during development on that validation set. If the resulting mIoU is materially below 73.64 on the full validation split, or if the base-versus-large margin on the matched subset reverses with multiple random seeds, the central accuracy and resource-efficiency claims would be weakened.","tokens_in":6165,"feed_emoji":"🎯","tokens_out":5689,"duration_ms":41638,"temperature":0.7,"pith_summary":"VespaSeg tries to show that referring-expression segmentation does not require a large monolithic model: a small vision–language model that predicts a box, followed by MobileSAM turning the box into a mask, can be competitive while using far less memory. On a RefCOCO validation protocol that keeps the first expression for each of 3,811 object records, the adapted Florence-2-base pipeline reaches 73.64 mIoU and 84.60 P@0.5, and on a matched 500-query subset it beats Florence-2-large while running 1.70 times faster and using 1.17 GB less allocated GPU memory. The paper is explicit that these numbers are not comparable to the full 10,834-expression RefCOCO validation split or to testA/testB, and that the checkpoint was selected on validation behavior. The value of the claim is that it maps a concrete, reproducible operating point where compact modular components suffice.","feed_headline":"Compact pipeline beats a larger model with 1.17 GB less memory","feed_subtitle":"It reaches 73.64 mIoU on the paper's 3,811-pair RefCOCO protocol at 2.20 GB memory.","key_machinery":"The load-bearing object is the explicit box interface between two stages: a compact vision–language model (Florence-2-base or Moondream2) grounds the expression $q$ into a box $\\hat b$, and MobileSAM, prompted with that box, produces the mask. MobileSAM's image embedding is computed once per image and reused for multiple expressions, which is what makes the reported cached-image throughput possible. Low-rank adaptation (LoRA) on both the grounder's attention and feed-forward projections and on MobileSAM's mask-decoder linear layers carries the accuracy gains; the box interface is what lets the two components be adapted and measured independently.","core_discovery":"The central claim is that separating language-conditioned localization from mask prediction is sufficient: an adapted compact grounder outputs a box, and MobileSAM converts that box to a mask. Under the repository-specific RefCOCO protocol (sentence index zero for each of 3,811 validation records), the adapted Florence-2-base pipeline obtains 73.64 mIoU and 84.60 P@0.5; a matched 500-pair comparison gives 73.73 mIoU for base versus 72.82 for large, with the base model 1.70 times faster and 1.17 GB lighter in allocated memory. The paper attributes the large gain over the unadapted grounder to RefCOCO-specific grounding, and shows with ground-truth boxes that MobileSAM mask-decoder LoRA raises mask IoU from 82.22 to 86.61 on a 1,000-record subset. It also shows that reducing Florence-2's output-token budget from 64 to 32 tokens is accuracy-neutral.","pith_inferences":["If first expressions per record are systematically easier than later ones, the same pipeline on all 10,834 validation expressions would likely score below 73.64 mIoU; this is testable with the released code.","The cached-image throughput measures interactive multi-query use, not video frames: an end-to-end frame rate would additionally include the MobileSAM image encoder for every new image.","Because the box interface decouples the stages, future improvements to compact grounding models should transfer directly to this pipeline without retraining the segmentation stage.","Since no held-out test set or multiple seeds were used, part of the reported margin may reflect selection on validation behavior; a multi-seed run would bound the effect."],"forward_implications":["A modular ground-then-segment architecture can hit a useful accuracy-resource operating point: 73.64 mIoU with 2.20 GB mean allocated memory and 22.8 cached-image queries per second on an RTX 6000 Ada.","At this operating point, a larger grounder does not help: Florence-2-large is slightly less accurate, slower, and uses more memory on the matched subset.","The output-token budget of the compact grounder can be halved from 64 to 32 without changing accuracy or memory, making the pipeline cheaper to run.","Adapting the mask decoder on ground-truth boxes improves segmentation quality by 4.39 mIoU points, independent of grounding error.","The pipeline's claimed performance should be re-measured on the complete standard RefCOCO splits and on edge hardware before being treated as deployment-ready."],"supporting_citations":[{"why":"Supplies MobileSAM, the box-prompted mask predictor that forms the segmentation stage.","marker":"[13]"},{"why":"Supplies the Florence-2 base and large grounders that predict the grounding box.","marker":"[10]"},{"why":"Supplies Moondream2, the alternative compact grounder compared in the accuracy table.","marker":"[7]"},{"why":"Supplies low-rank adaptation (LoRA), the method used to adapt grounder and mask-decoder weights.","marker":"[1]"},{"why":"Define the RefCOCO benchmark and its expressions, grounding the evaluation protocol.","marker":"[5, 12]"},{"why":"Supplies Segment Anything, the underlying architecture MobileSAM compresses.","marker":"[3]"},{"why":"Supplies TinyViT, the lightweight image encoder used by MobileSAM.","marker":"[9]"}],"fun_headline_variants":["Compact pipeline cuts memory by 1.17 GB, still beats larger model","Lean pipeline: 1.17 GB less, 1.7x faster, better mIoU","Compact pipeline: 73.73 mIoU, 1.17 GB less than larger","Two-stage pipeline: 2.20 GB memory, beats larger model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy rests on a custom validation protocol that uses only the first expression for each of 3,811 records and a checkpoint chosen while watching that same validation set, so if first expressions are easier or the selection overfit, the 73.64 mIoU overstates performance on the full benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Compact pipeline cuts memory by 1.17 GB, still beats larger model","Lean pipeline: 1.17 GB less, 1.7x faster, better mIoU","Compact pipeline: 73.73 mIoU, 1.17 GB less than larger","Two-stage pipeline: 2.20 GB memory, beats larger model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001029,"raw_usage":{"total_tokens":4384,"prompt_tokens":1045,"completion_tokens":3339,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":661,"completion_tokens_details":{"reasoning_tokens":3247}},"tokens_in":661,"tokens_out":3339,"duration_ms":23978,"temperature":1.0,"reasoning_tokens":3247,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:12:32.982716+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same adapted Florence-2-base plus MobileSAM pipeline on all 10,834 RefCOCO validation expressions, and on testA/testB, using a checkpoint not selected during development on that validation set. If the resulting mIoU is materially below 73.64 on the full validation split, or if the base-versus-large margin on the matched subset reverses with multiple random seeds, the central accuracy and resource-efficiency claims would be weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Moondream2, the alternative compact grounder compared in the accuracy table."}],"review_version":1}