{"id":"bc114ec0-870b-46bf-adb4-27497ff31410","arxiv_id":"2505.11216","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Graph shortest-path geodesic distance as a replacement for cosine similarity improves image-text contrastive pre-training by 1 to 3 retrieval points on ALBEF, TCL, and MAFA baselines.","lead":"This paper replaces the usual cosine similarity in image-text contrastive learning with geodesic distance, the shortest path between samples through a graph of neighboring examples. The change gives small, consistent gains on retrieval, VQA, and related benchmarks, but the training details are incomplete.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The geodesic loss as written is not differentiable: Floyd shortest paths and nearest-cluster assignments in Alg. 2 are argmin operations, and no surrogate or stop-gradient scheme is specified; without it, the claimed mechanism cannot train the encoders.","rationale":"The reader's weakest assumption correctly identifies the absent differentiability mechanism as the central vulnerability. The paper's headline claim depends entirely on the ability to optimize a loss whose logits are geodesic distances, but the described computation is discrete and non-differentiable. No code is provided, and the paper contains no description of how gradients are obtained through the Floyd algorithm or the hierarchical cluster assignment. This is not a disagreement with any external consensus; it is an internal gap between the proposed mechanism and the reported results. The experimental tables are extensive and the gains are consistent, which gives some empirical support, but without a trainable geodesic loss the mechanism cannot be verified from the text. The concern is therefore load-bearing and should be resolved by the authors, either by specifying a differentiable surrogate or by clarifying that the geodesic distance is used only as a detached reweighting of a cosine-based loss. Since the reader already conditioned acceptance on this point, the verdict remains CONDITIONAL; no change to the reader's verdict is needed.","tokens_in":20126,"tokens_out":4471,"duration_ms":48153,"concrete_test":"Reimplement the training loop from Alg. 2 in a minimal setting (two small MLP encoders, 256-d features, one batch, queue size 1024) and compute the gradient of the InfoNCE loss with respect to the image and text encoder parameters, using the paper's described geodesic distance (Floyd + nearest-cluster decomposition, angle normalization). If the gradient norm is numerically zero, or nonzero only because an undocumented cosine/similarity term was added, the mechanism as written is untrainable. As a complementary check, rerun the same toy task with the geodesic term detached via stop-gradient; if the encoders still move, some other loss term is responsible for the reported behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in Sec. 3.5, is that cosine distance in the loss is replaced by the proposed geodesic distance. For this to explain the reported gains, gradients from the InfoNCE loss must reach the image and text encoders through the geodesic distance. In the only algorithm given (Alg. 2), geodesic distances are produced by Floyd shortest-path on a discrete hierarchical graph (lines 3-5 and 10), and distances from out-of-graph queries to queue points are decomposed through a nearest bottom-level cluster center (Eq. 3). Both the Floyd argmin and the nearest-center assignment are piecewise-constant functions of the encoder features, so their derivatives are zero almost everywhere. The paper specifies no straight-through estimator, Gumbel-softmax relaxation, detached-distance reweighting, or auxiliary cosine path in Sec. 3.4 or 3.5. If the geodesic distance is treated as a constant in line 12, the InfoNCE logits are constants, the gradient is zero, and the model cannot learn from the geodesic geometry. If one attempts to backpropagate through the discrete graph, gradients vanish except on measure-zero boundaries. Either way, the described algorithm does not implement 'pulling paired samples closer' via geodesic distance. The paper also never reports the gradient computation or a PyTorch-style forward/backward for the geodesic operator; the Numba/CUDA Floyd implementation described in Sec. 3.5 is not differentiable as written. This is not a missing implementation detail: it is the load-bearing step connecting the proposed metric to the reported experimental gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GeoMM, a modification of contrastive vision-language pre-training that replaces cosine distance in the image-text contrastive (ITC) loss with a geodesic distance computed on a hierarchical graph of momentum features. The authors construct a two-level K-Means graph over a 65,536-entry feature queue, compute shortest-path distances between cluster centers with Floyd-Warshall, decompose out-of-graph query distances through the nearest bottom-level cluster center, and update the structure every 100 iterations. Experiments on ALBEF, TCL, and MAFA report consistent gains on image-text retrieval, VQA, NLVR2, SNLI-VE, and RefCOCO+, plus additional results on CLIP, FLIP, MOCOv2, and SwAV. The paper also states two theorems about connected-component counts in the hierarchical graph.","tokens_in":20551,"tokens_out":8185,"duration_ms":74991,"significance":"If the training mechanism were fully specified and the gains reproducible, the paper would make a useful contribution: it introduces a topological, graph-based distance into contrastive multimodal pre-training and demonstrates consistent improvements across multiple backbones and downstream tasks with modest computational overhead. The experimental coverage (zero-shot and fine-tuned retrieval, VQA, NLVR2, SNLI-VE, grounding, generalization to four other methods) is a genuine strength, and the efficiency analysis in Table 6 is useful. However, the significance is currently conditional: the central mechanism, gradient flow through geodesic distances, is not described, and several definitions are imprecise, so the paper does not yet establish why the reported gains should be attributed to the geodesic distance.","major_comments":[{"comment":"The central claim of the paper is that replacing cosine distance with the proposed geodesic distance in the contrastive loss accounts for the reported performance gains. As written, however, the algorithm cannot deliver gradients from that loss to the image and text encoders. The geodesic distances used in line 10 are produced by the Floyd shortest-path computation on a discrete graph (lines 3-5) and by the nearest-bottom-level-cluster-center decomposition of Eq. (3); both are piecewise-constant functions of the encoder features and have zero derivative almost everywhere. The paper does not specify a straight-through estimator, a soft cluster assignment, a detached-distance reweighting scheme, or any other differentiable surrogate, and the Numba/CUDA Floyd implementation described in Sec. 3.5 is not differentiable as written. If the distances are treated as constants when computing the InfoNCE loss in line 12, the logits are constants and the gradient with respect to theta is zero, so the described procedure cannot pull paired representations closer via the geodesic geometry. If instead one backpropagates through the discrete operations, the gradients vanish almost everywhere. In either case, the reported improvements are not explained by the described training mechanism, and the paper cannot be evaluated for correctness until this step is specified and, ideally, ablated (e.g., geodesic with straight-through gradient versus geodesic with detached distances versus cosine baseline).","section":"Sec. 3.4-3.5, Algorithm 2"},{"comment":"Equation (2) is self-referential and cannot be evaluated as printed: the right-hand side of the recursion for d(xi, C^k(xi)) contains d(xi, C^k(xi)) itself, so the distance from a point to its own cluster center is defined in terms of itself. The intended recursion presumably replaces that term with a distance to the next-level center (e.g., d(xi, C^{k+1}(xi))), but as printed the hierarchical geodesic distance computation is ill-defined. Please correct the formula and include a concrete two-layer example that traces the distance computation for one pair of points.","section":"Sec. 3.3, Eq. (2)"},{"comment":"Theorems 1 and 2 bound the number and size of connected components of the hierarchical graph, but the text's conclusion that these bounds 'imply the rationality of computing geodesic distance' does not follow: a bound on component structure says nothing about whether shortest-path distances on this graph are a good proxy for semantic similarity. More concretely, the implementation sets unreachable pairs to infinity, and if the graph has many connected components, a large fraction of candidate negatives may be excluded from the contrastive loss; the effective number of negatives then differs from the cosine baseline and varies during training. Please report the average fraction of reachable pairs in the graph during pre-training and specify exactly how infinite distances are handled in the angle-normalization and InfoNCE steps (e.g., masking). This is needed to confirm that the comparison with the cosine baseline is not confounded by a change in the number of negatives.","section":"Sec. 3.6, Theorems 1-2 and unreachable pairs"},{"comment":"The hyperparameter analysis does not appear to test the pre-training procedure it is meant to justify. The text says that the authors 'use pre-training models without geodesic distance and only fine-tuned using different hyper-parameters for the image-text retrieval,' but ncluster, the number of neighbors, the number of graph layers, and the update period T0 are all hyperparameters of the geodesic module during pre-training. Varying them only at fine-tuning time cannot establish that the proposed pre-training is insensitive to them. Please clarify what was actually varied; if the ablations did not rerun pre-training, they should be rerun for the most consequential parameters (neighbor count and T0), since the headline comparisons in Tables 1-3 use models pre-trained with specific values of these parameters.","section":"Sec. 4.5, Table 7"}],"minor_comments":[{"comment":"The angle-normalization description is ambiguous: truncating the accumulated angle at 4*pi and then normalizing to [0, pi] is not a standard normalization and changes the scale of the distance; please give the exact formula and define how the cosine is applied to the normalized value.","section":"Sec. 3.4"},{"comment":"The notation is inconsistent: Theorem 1 in the main text uses sigma for the number of neighbors, while the proof in the supplement uses kappa, and Eq. (4) has an unbalanced parenthesis; please unify the symbols and typeset the formula correctly.","section":"Sec. 3.6 / Supplementary Sec. 10.1"},{"comment":"The one-epoch fine-tuning protocol for CLIP_FT and FLIP_FT is unconventional and the margins in Table 4 are small; please report the exact protocol, including whether the baselines were fine-tuned for the same number of steps and with the same data order, and report variance across seeds if available.","section":"Sec. 4.4"},{"comment":"The claim that this paper introduces geodesic distance to multimodal learning 'for the first time' should be qualified in light of GraphWalk [75], which proposes a differentiable geodesic distance estimator, and of prior graph-based contrastive methods; a sentence explaining the difference would help.","section":"Sec. 2.2"},{"comment":"The caption reports cosine and geodesic values that appear to mix similarity and distance scales (0.87 vs 0.24, 0.93 vs 0.91); please clarify whether the numbers are distances or similarities and what the axes of the figure represent.","section":"Fig. 1"}],"recommendation":"major_revision","confidential_remarks":"The differentiability gap is severe enough that I would normally consider rejection, but it is plausibly fixable: the authors may have used a straight-through estimator or detached-distance weighting without reporting it. If they can specify the gradient path and provide an ablation separating the geodesic geometry from the extra machinery (e.g., hierarchical clustering, masking of unreachable pairs, queue updates), the contribution could be sound. I would also ask the editor to check whether the 'first to introduce geodesic distance to multimodal learning' claim is appropriate given the related work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on GeoMM. The paper is a decent engineering contribution: it replaces cosine distance with graph geodesic distance in contrastive vision-language pre-training, using a hierarchical graph and incremental updates to keep the Floyd computation feasible on a 65k queue. The experiments are broad and consistent—zero-shot and fine-tuned retrieval on COCO/Flickr, VQA, NLVR2, SNLI-VE, grounding, plus generalization to CLIP, FLIP, MoCo, and SwAV. The gains are small but systematic, and the hyperparameter sensitivity analysis suggests the method is not brittle. I believe the results are worth taking seriously.\n\nWhat's genuinely new is the specific integration: hierarchical clustering plus shortest paths as a distance for contrastive losses. The individual ingredients (ISOMAP, Floyd, k-means) are old. That is fine, but the paper is thin on the conceptual advance.\n\nThe big problem is the training mechanism. The paper says 'we replace the cosine distance in loss function with the proposed geodesic distance,' but Algorithm 2 never says how gradients get back to the encoders. The geodesic distance from a query point xo to any queue point xi is decomposed as d(xo, C(xo)) + d(C(xo), xi), where C(xo) is the nearest bottom-level cluster center. The first term is common to all xi. With a linear distance, that common term cancels in the softmax, and the encoding gets zero gradient. If you use the described cosine of the normalized angle, the common term does not cancel, but the gradient only flows through d(xo, C(xo))—the distance from the query to its nearest cluster center. The geodesic path and the queue points enter only as fixed constants in the denominator. So the model is not learning to pull positive pairs closer according to geodesic structure; it is learning to move queries relative to a set of static cluster centers. That may still be a useful regularizer, but it is not the proposed mechanism, and the paper doesn't acknowledge or analyze this. A referee needs the actual PyTorch-style forward/backward, or code, to see what is really happening.\n\nSecondary issues: the theoretical section is sloppy (undefined F(ξ), a truncated sentence in Theorem 2, a malformed inequality); the hyperparameters for the headline results are tuned on the same COCO retrieval task, with no variance estimates; and there is no code.\n\nThis paper deserves a serious referee. The idea is interesting and the experimental sweep is impressive, but the missing gradient story is load-bearing. I would send it to review and ask for code and a precise account of the loss's Jacobian. Until then, take the reported improvements with a grain of salt.","headline":"Solid experiments but the geodesic loss as described cannot deliver the gradients the authors claim; the paper needs a detailed forward/backward or code before the results can be attributed to the mechanism.","tokens_in":20987,"tokens_out":8128,"would_cite":false,"duration_ms":82074,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing cosine distance with graph geodesic distance improves contrastive vision-language pre-training.","keywords":["geodesic distance","multi-modal learning","contrastive learning","vision-language pre-training","hierarchical graph","shortest path","negative sample mining","momentum feature queue"],"falsifier":"Train the same model with a control in which the geodesic distances are computed exactly as described but treated as fixed constants inside the loss, detached from the learning signal, and compare with the full method; if the detached control reproduces the reported R@1 gains, the improvement is not being driven by the geodesic geometry steering the encoders, and the central claim's mechanism is falsified. A second control that replaces the geodesic distances with distances along a randomly rewired graph with identical edge counts would isolate the topology content of the signal.","tokens_in":19958,"feed_emoji":"🧭","tokens_out":10732,"duration_ms":92424,"temperature":0.7,"pith_summary":"Geodesic distance measures separation along a curved space, and multimodal embeddings often live in such a space; the standard cosine similarity measures only a pairwise angle and can call two sentences similar when they mean different things. The paper's claim is that replacing cosine with geodesic distance, computed as shortest paths over a graph of the current sample pool, gives contrastive vision-language pre-training a more faithful notion of sample relatedness. It constructs a hierarchical graph over the momentum feature queue, connects points to cluster centers and cluster centers to their nearest neighbors, and inserts the shortest-path distance into the InfoNCE loss. Across image-text retrieval, VQA, NLVR2, visual entailment, and visual grounding, the geodesic versions of the base models consistently outperform their cosine counterparts, with fine-tuned COCO text retrieval R@1 rising from 78.0 to 79.3. A sympathetic reader should take the paper as establishing that the global topology of the feature pool is a usable and beneficial metric signal for multimodal contrastive learning.","feed_headline":"Geodesic distance beats cosine in vision-language pre-training","feed_subtitle":"Graph shortest-path distance lifts COCO text-retrieval R@1 from 78.0 to 79.3 and helps VQA and NLVR2.","key_machinery":"The load-bearing object is the hierarchical geodesic distance: a graph-distance oracle over the momentum feature queue, built as a two-level hierarchy in which each bottom-level point attaches to a K-means cluster center and cluster centers connect to their eight nearest neighbors; the geodesic distance between two samples is the trivial distance to the nearest cluster center plus the Floyd shortest path between centers, propagated up the hierarchy until the two points share a cluster. The distance is converted into a loss input by angle normalization: the accumulated angular path is truncated at $4\\pi$, scaled to $[0,\\pi]$, and mapped through cosine. This machinery makes distance depend on the entire sample pool rather than on a pair of points alone, and it does so at a computational cost that fits inside a large-queue contrastive training loop, with the graph fully rebuilt every 100 iterations and incrementally updated in between.","core_discovery":"The central discovery the paper argues for is a metric substitution: in the image-text contrastive loss, replace the cosine similarity between a query feature and the momentum feature queue with a geodesic distance defined on the queue's graph. The graph is built by connecting each point to its nearest neighbors; distances are shortest paths through this graph, so a query is close to a candidate only if the data manifold itself connects them through intermediate samples. This resolves a failure mode of pairwise metrics: two sentences can have near-identical word-level embeddings yet sit on different parts of the semantic manifold, and only a global, path-based distance reveals that. To keep the computation feasible at a queue size of 65,536, the paper proposes a two-layer hierarchical graph, with K-means clusters at the bottom and cluster centers connected in a k-nearest-neighbor graph at the top, using Floyd shortest paths between centers and incremental updates every 100 iterations. The accumulated path angle is truncated at $4\\pi$, normalized to $[0,\\pi]$, and mapped through the cosine function before the InfoNCE loss. The reported results across five downstream tasks support the claim that this substitution, not any architectural change, is what drives the improvement.","pith_inferences":["One step beyond the paper: if geodesic distance helps because it encodes pool topology, the benefit should grow with queue size and sample-pool diversity; a testable extension would sweep queue sizes and measure whether the gap over cosine widens, whereas the paper fixes the queue at 65,536.","Another testable extension: because the graph is rebuilt only every 100 iterations and otherwise updated incrementally, the metric deliberately lags behind the moving encoders, and that slow-moving-target effect could itself be a regularizer independent of the geodesic geometry; a control that recomputes the graph every step or freezes it entirely would separate the two contributions.","The paper's theorems bound the number and scale of connected components per layer, so unreachable sample pairs, whose distance is set to infinity, are expected under the hierarchy; a reader could infer that part of the gains may come from this implicit filtering of easy negatives rather than from the path lengths themselves."],"forward_implications":["On the fine-tuned COCO benchmark, text retrieval R@1 rises from 78.0 for the cosine-based baseline to 79.3 for the geodesic version, and image retrieval rises from 61.2 to 62.5; Flickr30K moves similarly.","The metric swap transfers beyond the main baselines: zero-shot COCO retrieval improves when geodesic distance is added to other large-scale image-text contrastive models, and ImageNet linear classification improves for instance-discrimination self-supervised models.","Because the geodesic distance is global, it should reduce false negatives: samples that look alike in isolation but lie on different semantic paths remain far apart, so contrastive learning can mine harder, more reliable negatives.","The computational overhead is modest: for the base architecture, peak CUDA memory rises from 25.3G to 25.9G and per-100-iteration time from 255s to 268s.","The gains extend to reasoning tasks: VQA test-dev rises from 75.55 to 76.04, NLVR2 test-P from 80.96 to 81.63, and SNLI-VE val from 80.79 to 81.42 relative to the cosine-based baseline."],"supporting_citations":[{"why":"Supplies the baseline architecture, pre-training recipe, and momentum feature queue that the geodesic distance is inserted into.","marker":"[41]"},{"why":"Supplies the triple-contrastive baseline and its large momentum feature queue that the geodesic version extends.","marker":"[97]"},{"why":"Supplies the false-negative-handling baseline whose cosine distance the geodesic version replaces.","marker":"[6]"},{"why":"Establishes the momentum feature queue and oldest-sample removal rule used to maintain the graph's sample pool.","marker":"[28]"},{"why":"Defines the cosine-similarity contrastive objective that the paper replaces with geodesic distance.","marker":"[78]"},{"why":"Provides the geodesic-distance-as-shortest-path idea, from manifold learning, that motivates building a graph over samples.","marker":"[88]"},{"why":"Supplies the Floyd shortest-path algorithm used to compute distances between cluster centers in the hierarchical graph.","marker":"[25]"},{"why":"Supplies the K-means clustering used to construct the hierarchical graph's layers.","marker":"[52]"}],"fun_headline_variants":["Geodesic metric beats cosine in multimodal contrastive learning","Graph shortest-path distance lifts multimodal retrieval","Geodesic metric replaces cosine for better vision-language","Path-based distance improves multimodal contrastive learning","Geodesic graph metric sharpens multimodal similarity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the loss remains trainable: the paper computes the contrastive loss from geodesic distances that are outputs of a shortest-path algorithm on a discrete graph, and it never specifies how those distances change as the model's parameters change, so if they behave as fixed constants during training, the claimed improvement would not follow from the described mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Geodesic metric beats cosine in multimodal contrastive learning","Graph shortest-path distance lifts multimodal retrieval","Geodesic metric replaces cosine for better vision-language","Path-based distance improves multimodal contrastive learning","Geodesic graph metric sharpens multimodal similarity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000564,"raw_usage":{"total_tokens":2689,"prompt_tokens":973,"completion_tokens":1716,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":1645}},"tokens_in":589,"tokens_out":1716,"duration_ms":11846,"temperature":1.0,"reasoning_tokens":1645,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:55:43.060255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model with a control in which the geodesic distances are computed exactly as described but treated as fixed constants inside the loss, detached from the learning signal, and compare with the full method; if the detached control reproduces the reported R@1 gains, the improvement is not being driven by the geodesic geometry steering the encoders, and the central claim's mechanism is falsified. A second control that replaces the geodesic distances with distances along a randomly rewired graph with identical edge counts would isolate the topology content of the signal.","supporting_citations":[{"cited_title":"Vision-language pre- training with triple contrastive learning","cited_arxiv_id":null,"evidence_quote":"Supplies the triple-contrastive baseline and its large momentum feature queue that the geodesic version extends."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Defines the cosine-similarity contrastive objective that the paper replaces with geodesic distance."},{"cited_title":"Tenenbaum, Vin de Silva, and John C","cited_arxiv_id":null,"evidence_quote":"Provides the geodesic-distance-as-shortest-path idea, from manifold learning, that motivates building a graph over samples."},{"cited_title":"Algorithm as 136: A k-means clustering algorithm","cited_arxiv_id":null,"evidence_quote":"Supplies the K-means clustering used to construct the hierarchical graph's layers."}],"review_version":1}