{"id":"3ccf5c75-89af-48cf-8b07-7771b231ebdc","arxiv_id":"2412.12432","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RS@k loss, similarity mixup, and strong pre-trained initializations together raise deep metric learning benchmarks close to saturation.","lead":"This paper studies three ingredients for deep metric learning: a smooth surrogate of recall@k used as a training loss, a mixup trick that virtually enlarges batches, and initialization from large pre-trained vision models. Combined, they push recall@1 past 90 percent on three standard image retrieval benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Batch-size confound undermines the RS@k-vs-Smooth-AP comparison: SAP uses batch 384 while RS@k uses 4,000, and Fig. 9 shows batch size alone shifts r@1 by several points; without SAP at matching batch, the claimed loss advantage is unproven.","rationale":"I read the paper in good faith. The empirical protocol is more careful than typical DML work, the results are plausible, and I found no internal mathematical error in the surrogate or SiMix derivations. The load-bearing vulnerability is an uncontrolled variable in the main comparison: the RS@k loss is trained with a much larger batch than the Smooth-AP baseline, and the paper only isolates batch size on Cars196. This matters because the first contribution is specifically the loss, and the large-batch implementation is framed as an enabler of that loss. If SAP at batch 4,000 performs as well as RS@k, the headline gains are a batch-size effect and the loss contribution is not established. The reader's weakest assumption identifies exactly this issue, and I agree with that assessment. I do not see a reason to change the CONDITIONAL verdict: the concern is concrete and testable, but it does not yet amount to an observed failure or contradiction of the central claim.","tokens_in":22100,"tokens_out":7036,"duration_ms":64468,"concrete_test":"Retrain Smooth-AP with the exact RS@k training pipeline (multi-stage backprop, AdamW, d=512, ViT-L/14 DINOv2 initialization) at batch size 4,000 on iNaturalist and SOP, using the paper's train/validation split to select learning rate and epochs. Compare r@1 with RS@k at the same batch size. If the gap falls below roughly 0.5 r@1 points, or if SAP wins, the loss-specific advantage claimed in Sections 1 and 4.3 is not supported; if RS@k keeps a clear margin at matched batch, the confound is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the proposed RS@k loss is superior to Smooth-AP (Section 4.3) rests on comparisons where the batch size is not matched. Section 4.2 sets SAP's batch size to 384 (same as the original implementation) while RS@k is trained with 4,000 on iNaturalist and SOP, and with 392 on Cars196. Figure 9, which is only shown on Cars196, demonstrates that increasing batch size alone substantially improves r@1 for SAP and RS@k, and the curves become close at the largest batch. Because the paper reports no SAP-at-4,000 results on iNaturalist or SOP, the headline gaps (e.g., iNaturalist R50-512: 68.7 SAP vs 71.2 RS@k; ViT-B/16 IN-21k: 79.1 vs 83.9) could be explained by batch size rather than by the surrogate loss. This is not a baseline-tuning detail: the first contribution is the loss, and the large-batch machinery is presented as an enabler of that loss. The confound also weakens the statement that RS@k 'consistently surpasses' SAP; Table 5 itself contains Cars196 entries where SAP matches or beats RS@k without SiMix (e.g., CLIP-400M ViT-B/32: SAP 92.6 vs RS@k 91.8). A matching-batch comparison is therefore the single test that would establish whether the loss, rather than the larger batch, is responsible for the reported gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three components for supervised deep metric learning: a differentiable surrogate of recall@k (RS@k), a similarity-mixup virtual batch expansion (SiMix), and the use of large-scale pre-trained initializations. The RS@k loss replaces the Heaviside functions in the definition of recall with two sigmoids having different temperatures, and is trained on very large batches enabled by a two-stage backpropagation trick. SiMix constructs virtual examples by convexly combining embeddings of same-class pairs and computes their similarities algebraically, avoiding explicit mixed embeddings. The paper reports very high recall@1 numbers on iNaturalist, Stanford Online Products, and Cars196 with ViT backbones initialized from CLIP, DiHT, DINOv2, SWAG, and ImageNet-21k, and argues that these benchmarks are nearly solved. The authors also adopt a train/validation split for hyperparameter tuning, which they identify as a fairer protocol than common practice.","tokens_in":22498,"tokens_out":3434,"duration_ms":30743,"significance":"If the central comparison to Smooth-AP were clean, this would be a valuable empirical contribution: the paper ships a systematic study of three factors (loss, batch expansion, initialization) with a careful validation protocol, and the very large batch implementation with the SiMix trick is likely reusable beyond this specific loss. The breadth of backbones and initializations is a useful reference for the community. However, the headline claim that RS@k consistently surpasses Smooth-AP is weakened by a batch-size confound in the main comparisons, and the paper's internal definition of recall@k is inconsistent between the method section and the experiments section. These issues prevent the current version from fully supporting the paper's central claims.","major_comments":[{"comment":"The comparison between RS@k and Smooth-AP (SAP) is confounded by batch size. Section 4.2 states that SAP uses batch size 384 (the original value), while RS@k uses 4,000 on iNaturalist and SOP, and 392 on Cars196. Figure 9 shows on Cars196 that increasing batch size alone improves r@1 for both SAP and RS@k by several points, with the curves becoming close at the largest batch. Since no SAP-at-4,000 results are reported on iNaturalist, SOP, or VehicleID, the claimed advantage of RS@k over SAP (e.g., iNaturalist R50-512: 68.7 vs 71.2; ViT-B/16 IN-21k: 79.1 vs 83.9) cannot be attributed to the loss rather than the larger batch. Moreover, Table 5 itself contains entries where SAP matches or beats RS@k without SiMix (e.g., CLIP-400M ViT-B/32 Cars196 r@1: SAP 92.6 vs RS@k 91.8; IN-21k ViT-L/16 Cars196: SAP 91.3 vs RS@k 90.4), which contradicts the text's statement that RS@k is 'consistently surpassing' SAP. A matching-batch comparison on the other datasets is required to establish the loss's benefit.","section":"Section 4.2/4.3, Table 5, Fig. 9"},{"comment":"The definition of recall@k is inconsistent between the method and experiments. Equation (1) defines recall@k as the fraction of positive examples retrieved within the top k ranked list, while Section 4.1 defines the evaluation metric r@k as 1 if at least one positive image appears in the top k list, averaged across queries. These are different quantities; the former is a per-query recall of all positives, the latter is a binarized hit-at-k. The proposed surrogate loss in Eq. (5) is derived from Eq. (1), so it optimizes a different objective than the metric reported in the experiments. The paper should clarify which definition is used, how the surrogate relates to the reported metric, and whether the discrepancy affects the conclusions.","section":"Section 3.1 Eq. (1) vs Section 4.1"},{"comment":"The SiMix construction treats convex combinations of two same-class embeddings as valid positive virtual examples without re-normalization. Because the original embeddings are L2-normalized, the virtual embeddings in Eq. (8) generally have norm less than 1, so they live on a different norm sphere than the original examples. The paper does not justify this assumption or ablate it against re-normalizing the mixed embeddings (or against using mixed embeddings only in the database, not as queries). Since the efficiency claim of SiMix relies on this design choice, an analysis or comparison of these variants would strengthen the contribution.","section":"Section 3.3"}],"minor_comments":[{"comment":"The text says 'we archive 97.6% Recall@16' in the introduction and 'faciliated' in the conclusions; these appear to be typos for 'achieve' and 'facilitated'.","section":"Abstract/Conclusions"},{"comment":"The x-axis labels '24 25 26 27 28 29' are unclear; if they denote powers of two (16, 32, 64, ...), the axis should be labeled accordingly (e.g., 2^4, 2^5) for readability.","section":"Figure 9"},{"comment":"The method is called 'SiMix' in some places and 'SimMix' in others (e.g., Algorithm 2, Figures 5 and 8); the spelling should be unified.","section":"Section 3.3 and throughout"},{"comment":"The abstract claims the loss is computed on batches 'nearly equivalent to the entire training set', but the maximum reported batch size is 4,000, which is far smaller than the training sets of iNaturalist (325k) or GLDv1 (1M). This phrasing is an overstatement.","section":"Abstract"},{"comment":"The text states that DINOv2 is the best performing on iNaturalist 'by a large margin', but Table 5 shows the gap to CLIP-2B ViT-L/14 is about 7 points at r@1 (90.0 vs 82.8); while substantial, the claim could be more precise about which comparison is meant.","section":"Table 5 and Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim about the RS@k loss surpassing Smooth-AP is currently undercut by the batch-size confound; the authors need to either run SAP with matching large batches or substantially qualify the comparison. The inconsistency in the recall@k definition also needs to be resolved before the results can be interpreted. The SiMix assumption, while plausible, would benefit from a small ablation. The paper is otherwise well-executed and the validation protocol is a genuine strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three things about this paper: the experimental sweep is genuinely useful, the benchmark-saturation claim is real, and the loss-vs-baseline comparison is confounded by batch size. The RS@k loss and SiMix are extensions of the authors' own prior work [24], so the genuinely new content is the systematic initialization study and the demonstration that large pre-trained ViTs plus large-batch surrogate training nearly solve iNaturalist (90.0 r@1), SOP (90.8), and Cars196 (97.2). Credit where due: the validation protocol is much better than typical in DML — they hold out part of the training classes for hyperparameter tuning, and they reimplemented Smooth-AP with the same pipeline and architecture tweaks. That is exactly the kind of controlled baseline work the field needs.\n\nThe soft spot is the headline loss comparison. On iNaturalist and SOP, RS@k trains at batch 4,000 while SAP uses 384. Figure 9 shows batch size alone shifts Cars196 r@1 by several points, and at the largest tested batch the SAP and RS@k curves nearly converge. Without SAP at batch 4,000 on those two datasets, the statement that RS@k 'consistently surpasses' SAP is not established; the gain could be the batch, not the surrogate. This is not fatal to the broader recipe claim — the paper still shows the combination works — but it means the loss-specific advantage is unproven. Tellingly, on Cars196 where batches are nearly matched, RS@k does not consistently beat SAP (e.g., Table 5, CLIP-400M ViT-B/32). Two smaller points: the overlap with prior work [24] should be stated more prominently in the abstract, and there are no variance numbers across seeds. Both are fixable.\n\nWho this is for: anyone working on deep metric learning or listwise retrieval losses. The benchmark numbers and initialization study are worth citing and should serve as strong baselines. A serious referee should engage; the matching-batch SAP comparison is the one experiment that would settle the loss claim.","headline":"The recipe nearly solves DML benchmarks, but the loss-vs-Smooth-AP comparison is confounded by batch size, so the loss-superiority claim is unproven.","tokens_in":23014,"tokens_out":3010,"would_cite":true,"duration_ms":25057,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a training recipe combining a differentiable surrogate of recall@k, a similarity-space mixup that virtually enlarges the batch, and initialization from large-scale pretrained models nearly solves standard deep metric…","keywords":["deep metric learning","recall@k surrogate","similarity mixup","large batch training","model initialization","image retrieval","vision transformer"],"falsifier":"Train the proposed RS@k and Smooth-AP with identical batch sizes on iNaturalist, SOP, and Cars196 using the same pretrained ViT initialization, and compare recall@1. If the RS@k advantage over Smooth-AP largely disappears when batches are equalized, the central claim that the new loss drives the near-solved results is falsified.","tokens_in":21904,"feed_emoji":"🔍","tokens_out":6142,"duration_ms":51709,"temperature":0.7,"pith_summary":"Supervised deep metric learning trains an embedding so that nearest-neighbor retrieval works on classes unseen during training. This paper claims that three choices together let large models nearly solve standard benchmarks: a differentiable surrogate of the retrieval metric recall@k trained on very large batches, a mixup that operates on pairwise similarities to virtually enlarge the batch, and initialization from large-scale pretrained vision models. With those ingredients the authors report recall@1 of 90.0% on iNaturalist, 90.8% on Stanford Online Products, and 97.2% on Cars196, with higher recall@k values approaching ceiling on all three. A reader should care because the recipe is simple and, if accurate, turns these long-standing benchmarks into saturated targets that no longer separate methods.","feed_headline":"Three ingredients nearly solve image retrieval benchmarks","feed_subtitle":"A recall-at-k loss, similarity mixup, and pretrained models reach 90–97% recall@1.","key_machinery":"The machinery is the RS@k loss, a sigmoid-smoothed version of recall@k. In the paper's formulation, the counting of positives inside the top $k$ and the ranking of each positive are both softened by sigmoids, one with temperature $\\tau_2 = 0.01$ on similarity differences and one with $\\tau_1 = 1$ on rank position, so gradients flow to similarity changes that actually improve the shortlist. Two supporting mechanisms make it work: a multistage backpropagation borrowed from [25] that allows a batch of 4,000 images within GPU memory, and SiMix, which generates virtual examples by linearly mixing the scalar similarities of original same-class pairs, effectively enlarging the batch without creating mixed embeddings. The loss is then applied to each original and virtual image as query, using a set of $k$ values such as $\\{1,2,4,8,16\\}$.","core_discovery":"The central claim is that the non-differentiable recall@k metric can be optimized directly through a hand-designed smooth surrogate, provided the loss is evaluated on a batch almost as large as the training set. The paper expresses recall@k by replacing its Heaviside step functions with two sigmoids controlled by separate temperatures, and shows that the resulting RS@k loss tracks the true metric closely. Training with batch sizes of 4,000 is made feasible by a multistage backpropagation that recomputes embeddings one at a time, and the SiMix variant further expands the effective batch by mixing scalar similarities of same-class pairs without ever constructing mixed embeddings. Combined with initialization from large-scale pretrained models, this pipeline gives top results on iNaturalist, Stanford Online Products, Cars196, and VehicleID and improves mean Average Precision on instance-level retrieval over an Average Precision loss baseline. The authors conclude that the tested benchmarks are nearly solved under these settings.","pith_inferences":["Inference: because most experiments run RS@k at batch size 4,000 while Smooth-AP uses 384, the reported advantage may be partly a batch-size effect; the paper only isolates this on Cars196, so a batch-matched comparison on iNaturalist and SOP would settle attribution.","Inference: SiMix's operation purely on scalar similarities could transfer to other listwise or ranking losses, including non-DML retrieval and recommendation, wherever pairwise similarity matrices are available.","Inference: if the benchmarks are genuinely near-solved, future work should shift to harder evaluations such as open-vocabulary or long-tail retrieval, where near-ceiling scores cannot signal progress."],"forward_implications":["Using RS@k instead of Smooth-AP improves recall@1 consistently across ResNet and ViT backbones and embedding sizes in the paper's experiments.","SiMix gives its largest gains when the physical batch is small, as on Cars196, so it is a cheap way to recover large-batch behavior under hardware limits.","Initialization from CLIP, DINOv2, DiHT, or SWAG generally beats ImageNet-21k initialization, and better pretraining transfers to better retrieval after fine-tuning.","At the strongest settings, iNaturalist reaches 97.6% recall@16, SOP reaches 97.7% recall@10, and Cars196 reaches 99.3% recall@8, which the paper takes as nearly solved benchmarks.","The loss is robust to the choice of $k$ set and rank temperature, with $\\tau_1 = 1.0$ the best setting."],"supporting_citations":[{"why":"Smooth-AP, the main baseline and the source of the sigmoid approximation style used for recall@k.","marker":"[49]"},{"why":"Supplies the multistage backpropagation that makes 4,000-image batches fit in GPU memory.","marker":"[25]"},{"why":"The authors' prior work introducing the RS@k surrogate loss with large batches and similarity mixup, which this paper extends.","marker":"[24]"},{"why":"CLIP, a text-supervised pretraining used as model initialization and compared across backbones.","marker":"[20]"},{"why":"DINOv2, a self-supervised pretraining that gives the strongest iNaturalist results.","marker":"[21]"},{"why":"ProxyNCA++, whose train/validation class split is adopted for fair hyperparameter tuning.","marker":"[32]"},{"why":"The metric learning reality check that motivates the fair validation protocol by showing test-set tuning inflates comparisons.","marker":"[39]"},{"why":"ViT architecture and ImageNet-21k supervised pretraining used as one initialization.","marker":"[17]"},{"why":"SWAG, a weakly supervised pretraining used as initialization.","marker":"[18]"},{"why":"DiHT, a text-supervised pretraining with hard-negative sampling used as initialization.","marker":"[19]"}],"fun_headline_variants":["Smooth recall loss, mixup, pretraining: retrieval nearly solved","Three ingredients for near-perfect image retrieval","Large-batch smooth loss, mixup, pretraining: near-solved retrieval","Retrieval nearly solved by three tricks: smooth loss, mixup, pretraining","Three fixes nearly crack image retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the new loss itself is what produces the performance gains; in most experiments, however, the new loss is trained with a much larger batch than the Smooth-AP baseline, so the improvement could come from batch size rather than from the loss.","fun_headline_variants_meta":{"raw":{"variants":["Smooth recall loss, mixup, pretraining: retrieval nearly solved","Three ingredients for near-perfect image retrieval","Large-batch smooth loss, mixup, pretraining: near-solved retrieval","Retrieval nearly solved by three tricks: smooth loss, mixup, pretraining","Three fixes nearly crack image retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001512,"raw_usage":{"total_tokens":6025,"prompt_tokens":874,"completion_tokens":5151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":5064}},"tokens_in":490,"tokens_out":5151,"duration_ms":32809,"temperature":1.0,"reasoning_tokens":5064,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:05:32.367459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the proposed RS@k and Smooth-AP with identical batch sizes on iNaturalist, SOP, and Cars196 using the same pretrained ViT initialization, and compare recall@1. If the RS@k advantage over Smooth-AP largely disappears when batches are equalized, the central claim that the new loss drives the near-solved results is falsified.","supporting_citations":[{"cited_title":"& Zisser- man, A","cited_arxiv_id":null,"evidence_quote":"Smooth-AP, the main baseline and the source of the sigmoid approximation style used for recall@k."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multistage backpropagation that makes 4,000-image batches fit in GPU memory."},{"cited_title":"& Matas, J.Recall@ k surro- gate loss with large batches and similarity mixup (2022)","cited_arxiv_id":null,"evidence_quote":"The authors' prior work introducing the RS@k surrogate loss with large batches and similarity mixup, which this paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CLIP, a text-supervised pretraining used as model initialization and compared across backbones."},{"cited_title":"W., DeVries, T","cited_arxiv_id":null,"evidence_quote":"ProxyNCA++, whose train/validation class split is adopted for fair hyperparameter tuning."},{"cited_title":"& Lim, S.-N","cited_arxiv_id":null,"evidence_quote":"The metric learning reality check that motivates the fair validation protocol by showing test-set tuning inflates comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ViT architecture and ImageNet-21k supervised pretraining used as one initialization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SWAG, a weakly supervised pretraining used as initialization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DiHT, a text-supervised pretraining with hard-negative sampling used as initialization."}],"review_version":1}