{"id":"3b612086-59ad-49a9-b579-76dfdb5c94d4","arxiv_id":"2510.09953","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Jointly training the retrieval model and the segmentation model improves cardiac MRI segmentation over training the segmentation model alone.","lead":"This paper introduces J-RAS, a training method that simultaneously tunes a segmentation model and an image-retrieval model so that retrieved image–mask pairs guide segmentation. On two cardiac MRI benchmarks (ACDC and M&Ms) it reports consistent Dice gains of about 2–4 points across several network backbones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of mutual adaptation is not isolated: no frozen-retriever baseline, so the joint-update benefit remains untested.","rationale":"The reader's weakest assumption is the same one I would flag: the causal role of joint optimization is unverified. My read of Section 3.2, Algorithm 1, and Section 6 confirms that no ablation isolates the retrieval update. The paper's own Section 7 limitation — 'Computational constraints prevented backpropagation through the entire retrieval knowledge base, restricting updates to the query side; database-wide updates could potentially reduce mismatches' — reinforces the need for this baseline: the implemented joint update is partial, so its contribution is an empirical question. A fixed retriever baseline is feasible and inexpensive, and would either support the central claim or show that the method is equivalent to retrieval-augmented segmentation with a pre-trained retriever. This concern is load-bearing because the abstract and contributions are framed entirely around mutual adaptation; if the frozen retriever performs the same, the claimed novelty reduces to the fusion/input construction. Other issues — internal numerical inconsistency in Table 2's SegFormer row, lack of code and hyperparameters, and overstatement of evaluated benchmarks — are secondary and do not change the conditional verdict. I agree with the reader; no verdict change is needed.","tokens_in":16875,"tokens_out":4499,"duration_ms":43251,"concrete_test":"On ACDC with SegFormer at Top-k=2, rerun the J-RAS protocol with the retrieval model frozen after independent training: in Algorithm 1, replace 'Update R_theta, S_phi' with 'Update S_phi only', keeping the same fused 7-channel input (Equation 4), same softmax weights, same knowledge-base embeddings, same loss, and same seeds. Compare held-out mean Dice/HD against the reported +J-RAS row in Table 2. If the frozen retriever matches J-RAS within one standard deviation (or within ~0.005 Dice), the mutual-adaptation claim is not supported; if it is materially worse, the joint update is doing real work.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's distinct contribution is the joint optimization loop: the retrieval encoder is updated by the segmentation loss so that retrieved guides become more useful (Section 3.2, Algorithm 1, line 12). The empirical case for this claim is the consistent improvement of J-RAS over each backbone alone (Table 2). But every J-RAS result changes two variables at once: (1) the segmentation model now receives fused retrieved image-mask pairs, and (2) the retrieval encoder has been adapted by the segmentation loss. The paper does not run the control that separates these variables — a frozen-retriever version with the same 7-channel fusion, same softmax weights, same Top-k, and same segmentation model. The ablations in Section 6 compare fusion architectures, dynamic vs fixed k, noisy guides, and cross-dataset retrieval, but they all keep the joint update active, so they do not test whether the update to R_theta is the cause of the gains. Section 7 also admits that backpropagation is restricted to the query branch, making the 'joint' component even less characterized. Without the frozen baseline, the observed gains could be due entirely to the auxiliary ground-truth masks entering the input, with the retrieval adaptation contributing little or nothing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes J-RAS, a retrieval-augmented segmentation method that couples a segmentation model (U-Net, TransUNet, SAM, SegFormer) with a DINOv2/RAD-DINO retrieval encoder. After an independent contrastive fine-tuning stage, the method retrieves the top-k most similar image–mask pairs, fuses them via temperature-scaled softmax weights, and concatenates the query image, fused guide image, and fused mask into a 7-channel input to the segmentation model. The segmentation loss is then backpropagated into both the segmentation model and the retrieval encoder's query branch, with knowledge-base embeddings recomputed each epoch. The paper reports consistent Dice/HD improvements over backbone-only baselines on ACDC and M&Ms, and includes ablations on fusion strategy, dynamic vs. fixed top-k, noisy guides, and cross-dataset retrieval.","tokens_in":17187,"tokens_out":5117,"duration_ms":46534,"significance":"If the mutual-adaptation loop works as stated, J-RAS would be a practically useful, model-agnostic way to inject retrieved anatomical context into medical image segmentation, especially for small datasets. The paper has genuine strengths: it evaluates four diverse backbones, includes a cross-dataset retrieval experiment, tests robustness to noisy guides, and provides qualitative evidence that retrieval quality changes after joint training. However, the central causal claim is not isolated by any experiment: no frozen-retriever baseline is reported under identical fusion and input conditions, and the gradient mechanism through the discrete top-k selection is not specified. The contributions are therefore plausible but currently under-supported.","major_comments":[{"comment":"The central claim—that jointly optimizing the retrieval model R_theta with the segmentation loss improves segmentation—is not isolated. Every J-RAS result in Table 2 changes two factors relative to the backbone-only baseline: the segmentation input now contains a fused retrieved image-mask pair, and R_theta has been updated by the segmentation loss. There is no control that freezes R_theta after the independent contrastive stage while using the identical 7-channel early fusion, same softmax weights (Eq. 3), same top-k=2, and same training budget. The ablations in Section 6 vary fusion, k, noise, and dataset, but they all keep the joint gradient active, so they cannot separate the contribution of retrieval adaptation from the contribution of simply providing auxiliary ground-truth masks. Please add a frozen-retriever baseline; without it, the mutual-adaptation claim is untested.","section":"Section 3.2, Algorithm 1, Table 2"},{"comment":"The gradient path for the joint update is underspecified. Algorithm 1 line 7 calls Retrieve(q, D'), a discrete top-k operation. The text states that the segmentation loss 'back-propagates through the similarity and weighting computations into the query embedding,' but no differentiable surrogate or straight-through estimator for the discrete selection is described. If the gradient flows only through the softmax weights w_i for already-selected indices, the dependence of the selected set on R_theta is ignored; if a relaxation is used, it should be stated explicitly. Since this gradient path is what makes the method 'joint,' the ambiguity is load-bearing for the method's correctness.","section":"Section 3.2, Algorithm 1"},{"comment":"The arXiv abstract claims evaluation on 'four public benchmarks spanning different imaging modalities, including ACDC and M&Ms (MRI), Breast Cancer Ultrasound, and lung and infection CT,' but Section 4.1 and all experiments use only ACDC and M&Ms, both cardiac MRI. The paper body's own abstract says 'two benchmark datasets.' The generalizability claim is therefore overstated; either add the missing datasets or correct the abstract to match the actual experiments.","section":"Abstract vs. Section 4.1"}],"minor_comments":[{"comment":"The class-wise Dice scores (RV 0.8589, MYO 0.8484, LV 0.8484) do not average to the reported mean of 0.8708; the mean of the listed entries is 0.8519. This looks like a typo in one of the class entries and should be corrected, since readers use this table to verify the headline improvement.","section":"Table 2, SegFormer baseline row"},{"comment":"The text says that after J-RAS the model 'correctly identified slice 1 from the same patient as the closest match,' but Section 3.2 explicitly excludes slices from the same patient, and the figure labels show retrieved slices from different patients (e.g., P.113, P.147, P.120). The wording appears to be an error and should be fixed.","section":"Section 5.2, Figure 9"},{"comment":"No statistical significance tests are reported, and it is unclear whether results come from one run or multiple seeds. Given that some improvements (e.g., U-Net Dice 0.8588±0.040 to 0.9034±0.033) are small relative to the reported variance, paired significance tests across the 100 testing patients would strengthen the central claim.","section":"Section 5.1.2"},{"comment":"Line 8 uses w_1 and w_2 without defining them in the algorithm; the definition in Eq. (3) should be referenced or repeated so the pseudocode is self-contained.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper's main empirical claim hinges on a missing ablation—a frozen-retriever control with identical fusion and input. This is fixable within the manuscript's scope, but it is essential. There is also an abstract/body mismatch regarding the number of benchmark datasets that must be resolved. My recommendation is major_revision rather than reject because the method is plausible and the proposed control is feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: worth a serious referee, but the central claim is under-supported. What's new: relative to Zhao et al. [43], where retrieval is frozen, J-RAS backprops the segmentation loss into the query branch of a DINOv2/RAD-DINO encoder and recomputes the gallery each epoch. That is a genuine, simple extension, and the paper is honest that the update is query-side only (Section 7). The empirical picture is mostly clean: four backbones, two public cardiac datasets, consistent Dice/HD improvements; cross-dataset retrieval from ACDC to M&Ms works; noisy-guide ablation shows robustness; qualitative retrieval examples after training are noticeably more anatomically aligned. The method is architecture-agnostic and the experiments cover a reasonable spread.\n\nSoft spots. The main one is causal attribution. Every J-RAS result changes two things at once: the segmentation model gets fused retrieved image-mask pairs, and the retrieval encoder has been adapted by the segmentation loss. There is no frozen-retriever baseline with the same 7-channel fusion, same temperature weights, same top-k, and same backbone. The fusion ablation (early vs cross-attention vs dual-encoder) and dynamic top-k comparisons keep the joint update active, so they do not tell you whether updating the retriever is what drives the gains. It is entirely possible that most of the benefit comes from simply feeding ground-truth masks as auxiliary input, with the retrieval adaptation adding little. The paper needs that control before the mutual-adaptation claim is supported.\n\nSecond, the abstract claims evaluations on four public benchmarks (ACDC, M&Ms, breast ultrasound, lung/CT), but the full text reports experiments on ACDC and M&Ms only. That overstatement should be fixed. Third, Table 2 has at least one suspicious entry: SegFormer baseline reports the same LV Dice as MYO (0.8484) while the mean 0.8708 does not match the average of the listed class values; numbers need checking. Fourth, no statistical significance tests are reported, and no code or hyperparameters are released, which makes it hard to gauge how sensitive the gains are to top-k, temperature, and the adapter.\n\nBottom line: the paper is a solid incremental contribution to retrieval-augmented segmentation, with an honest limitations section, but the load-bearing claim—that joint optimization of the retriever is what improves segmentation—is unverified. For a peer reviewer I'd send it out, with the frozen-retriever ablation as the condition. I would not cite it as evidence for mutual adaptation until that control exists.","headline":"The contribution—making the retriever trainable through the segmentation loss—is real and the gains are consistent, but the paper never runs the frozen-retriever control that would prove the joint update is what helps.","tokens_in":17656,"tokens_out":2351,"would_cite":false,"duration_ms":23340,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Jointly optimizing a retrieval model with a segmentation model improves medical image segmentation by making the retriever learn which image-mask pairs are most useful for delineating anatomy.","keywords":["Image Segmentation","Retrieval","Joint Training","Retrieval-Augmented Segmentation","Feature Fusion","Medical Image Segmentation","Contrastive Learning","Cardiac MRI"],"falsifier":"An ablation study that keeps the retrieval encoder frozen after contrastive pretraining while using the identical top-k fusion and input representation: if the frozen-retrieval version matches or exceeds J-RAS's Dice and HD, the joint update is not the cause of the improvement.","tokens_in":16761,"feed_emoji":"🫀","tokens_out":5726,"duration_ms":46881,"temperature":0.7,"pith_summary":"J-RAS jointly trains a retrieval model and a segmentation model so that retrieved image-mask pairs guide the segmentation of new images. The retrieval network is first contrastively pretrained and then updated through the segmentation loss, learning to fetch cases that are anatomically relevant rather than merely visually similar. The paper shows this mutual adaptation improves Dice scores and reduces contour errors across four backbones (U-Net, TransUNet, SAM, SegFormer) and across cardiac MRI datasets, including a cross-dataset setting where guides come from one dataset and segmentation is done on another. If correct, this offers a path to better segmentation with less reliance on large labeled datasets.","feed_headline":"Joint training with retrieval boosts segmentation Dice","feed_subtitle":"Co-optimizing the retrieval model with the segmentation model makes retrieved guides more useful for anatomy.","key_machinery":"The central mechanism is a joint optimization loop: a radiology-pretrained DINOv2 vision transformer serves as the retrieval encoder, producing 128-dimensional embeddings for query and knowledge-base slices. Top-k retrieval uses cosine similarity; retrieved images and masks are weighted by a temperature-scaled softmax and fused into a composite image-mask pair, concatenated with the query into a 7-channel tensor, and projected via an adapter into the segmentation model. The same Dice-plus-cross-entropy loss then backpropagates through the similarity and weighting computations into the query branch, updating the retrieval encoder. Knowledge-base embeddings are recomputed each epoch.","core_discovery":"The central claim is that segmentation accuracy improves when the retrieval model is not fixed but co-optimized with the segmentation model. In J-RAS, a query image is embedded, top-k similar slices are retrieved from a knowledge base, and their images and ground-truth masks are merged into a composite guide that is concatenated with the query and fed to the segmentation network. The segmentation loss updates both the segmentation weights and the query branch of the retrieval encoder, so the encoder gradually retrieves examples that are more useful for delineating boundaries. On the ACDC test set, SegFormer's mean Dice rises from 0.8708±0.042 to 0.9115±0.031 and Hausdorff distance falls from","pith_inferences":["The paper does not ablate the joint update: a natural test is to freeze the retrieval encoder after contrastive pretraining and run the same top-k fusion. If Dice gains persist, the improvement may come from the extra ground-truth masks rather than mutual adaptation.","Because only the query branch is updated (gallery embeddings are detached per epoch), the retrieval model may develop a query-specific bias; a full end-to-end update of the knowledge base could improve consistency but is currently limited by computation.","The dynamic Top-K strategy underperformed fixed K=1 in their experiments; a learned or confidence-based selection rule may be a promising extension beyond the fixed threshold used here.","The cross-dataset results suggest the method could be extended to multiorgan or multimodal segmentation, where guides from one modality support segmentation in another."],"forward_implications":["Segmentation models can tap into a growing knowledge base of annotated cases, improving boundary delineation without needing more labeled training data.","The retrieval model becomes a task-aware encoder, potentially useful for other downstream tasks like anomaly detection or case retrieval for clinical decision support.","Cross-dataset retrieval shows guidance can transfer across domains, suggesting applicability to low-resource settings where annotated data comes from a different scanner or protocol.","Consistent gains across CNN, hybrid, and transformer backbones imply the method is model-agnostic and can be layered onto existing segmentation pipelines.","Even noisy or corrupted guides outperform the no-guide baseline, indicating resilience to imperfect retrieval."],"fun_headline_variants":["Co-optimized retrieval boosts segmentation accuracy","Segmentation gains from adaptive retrieval guidance","Joint retrieval-segmentation loop refines boundaries","Retrieval model tuned to aid segmentation nets","Mutual adaptation improves medical image segmentation"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The gains rest on the assumption that updating the retrieval model through the segmentation loss is what drives the improvement, rather than simply providing ground-truth masks to the segmentation network — an assumption not isolated by ablation in the paper.","fun_headline_variants_meta":{"raw":{"variants":["Co-optimized retrieval boosts segmentation accuracy","Segmentation gains from adaptive retrieval guidance","Joint retrieval-segmentation loop refines boundaries","Retrieval model tuned to aid segmentation nets","Mutual adaptation improves medical image segmentation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000113,"raw_usage":{"total_tokens":946,"prompt_tokens":837,"completion_tokens":109,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":43}},"tokens_in":581,"tokens_out":109,"duration_ms":1995,"temperature":1.0,"reasoning_tokens":43,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:21:33.384735+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An ablation study that keeps the retrieval encoder frozen after contrastive pretraining while using the identical top-k fusion and input representation: if the frozen-retrieval version matches or exceeds J-RAS's Dice and HD, the joint update is not the cause of the improvement.","supporting_citations":[],"review_version":1}