{"id":"93b70f61-fb4f-4f62-83c4-287e398b5b58","arxiv_id":"2602.04401","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A quantile-based threshold transfer method automates operating-point selection for visual place recognition, maximizing recall at a user-specified precision without manual tuning.","lead":"This paper introduces a method that automatically picks the image-matching threshold for visual place recognition using a small labeled calibration set, then transfers that threshold to new environments as a score quantile. The goal is to maximize correct matches while keeping precision at a user-set level such as 100%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantile invariance is asserted, not demonstrated: Eq. (1)-(2) transfer a calibration quantile to deployment, but the method has no mechanism to detect or correct shifts in the match/non-match boundary; the claim that 100% precision transfers is not supported.","rationale":"The paper proposes a practical calibration method for VPR operating-point selection, with a plausible mechanism and extensive experiments. The strongest claim—that quantile transfer enables a user-defined precision requirement (e.g., 100%) to be met on unseen deployment data—depends on quantile invariance: the quantile of the optimal matching threshold in calibration must equal the quantile in deployment. The paper provides no proof of this, only an assertion of stability. The qualitative failure in Figure 5 demonstrates that the method can violate the precision requirement, which means the invariance assumption is not always valid. This is the most load-bearing concern because if it fails, the central contribution collapses, regardless of AUPC improvements. The proposed concrete test directly measures whether deployment quantiles match calibration quantiles, and would settle the issue. The reader's weakest assumption identified the same mechanism, and the CONDITIONAL verdict is appropriate. No verdict change is needed; rather, the paper should be revised to either prove or empirically validate quantile invariance.","tokens_in":12530,"tokens_out":3786,"duration_ms":42416,"concrete_test":"For each deployment chunk, compute the oracle threshold that maximizes recall at 100% precision using ground truth, then compute its empirical quantile q_val in that chunk's score distribution. Compare the distribution of q_val across all chunks with the calibration-derived Q_k values (and their median). If the median absolute deviation between q_val and Q_k exceeds 5 percentile points, quantile invariance is contradicted and the method's precision guarantee is unsupported. Repeat this diagnostic across all seven VPR techniques and all datasets reported in Table I.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is quantile transfer: thresholds learned on calibration subsets are converted to quantiles (Eq. 1) and applied to the deployment score distribution (Eq. 2). The paper's only justification is the statement in Section III-C that 'the relative ordering of similarities tends to be more stable'—an assertion without a theorem or controlled distribution-shift test. If the quantile of the optimal threshold shifts between calibration and deployment (e.g., because the match/non-match score overlap changes, or the prior over place matches changes), the transferred threshold will not meet the user-specified precision, and the method cannot detect this. This is not merely a theoretical worry: Figure 5 (bottom row) shows a failure case where the method underestimates the threshold and retrieves incorrect matches. The paper reports aggregate AUPC but does not provide per-chunk PSR tables, so the frequency of such failures is unclear. The claim that the method lets any VPR system 'operate at 100% precision' in deployment therefore rests entirely on an unvalidated invariance assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for automatic matching-threshold selection in visual place recognition (VPR). Using a small calibration traversal with known correspondences, the method learns thresholds that maximize recall at a user-specified precision (e.g., 100%), converts these thresholds into quantiles of the calibration similarity distribution (Eq. 1), and transfers them to deployment by applying the same quantile to the deployment similarity distribution (Eq. 2). A per-query adaptive calibration sampling step selects the most correlated calibration queries, and the median of the resulting quantiles is used as the deployment threshold. Experiments with seven VPR techniques on three benchmark datasets report improvements in Area Under the Performance Curve (AUPC) over a baseline and an ablation without quantile transfer. The paper claims the method enables reliable high-precision operation without manual threshold tuning.","tokens_in":12822,"tokens_out":5414,"duration_ms":60107,"significance":"If the quantile-transfer assumption holds, the method is a practical, VPR-technique-agnostic tool for maintaining high precision under environmental change, which is relevant for safety-critical V-SLAM and autonomous navigation. The paper's strengths include a broad evaluation across multiple VPR descriptors and datasets, a clean ablation isolating the quantile transfer step, reported runtime figures, and released code. The central claim is, however, contingent on an invariance assumption that is asserted rather than demonstrated, and several evaluation choices raise concerns about whether the reported improvements reflect genuine reliability gains. The paper has clear potential but needs additional validation and clarification before its conclusions can be fully accepted.","major_comments":[{"comment":"The central mechanism, quantile transfer, rests on the assertion that 'the relative ordering of similarities tends to be more stable' across environments. No theorem, formal condition, or controlled distribution-shift test is provided. The method has no way to detect or correct shifts in the match/non-match boundary between calibration and deployment. Since the paper's core claim is that a user-defined precision (e.g., 100%) is satisfied on deployment data, this assumption is load-bearing. Please provide either a theoretical characterization of when quantile transfer is valid, or an empirical analysis comparing calibration-derived quantiles with oracle quantiles on deployment chunks across different shift conditions.","section":"Section III-C, Eqs. (1)-(2)"},{"comment":"The sentence 'The number of chunks sampled from each dataset corresponds to those achieving a precision requirement, which for the evaluation in Section V is set to 100% precision' suggests that evaluation may be performed only on chunks that already satisfy the precision requirement. If chunks are selected based on outcome, the reported PSR and AUPC are biased by construction. Please clarify the chunk selection procedure, state explicitly whether all deployment queries were included, and re-run evaluations on all chunks without any precision-based filtering. If some chunks were excluded, disclose the fraction excluded and report results for the full deployment set.","section":"Section IV-C"},{"comment":"The illustrative example for Nordland Fall-Summer is misleading. The text says 'at 100% precision, the baseline correctly recalls 247 places out of 285 queries with 38 incorrect matches'; this is 247/(247+38) = 86.7% precision, not 100%. The proposed method is said to recall 57 places with no incorrect matches. This compares two different operating points and inflates the apparent advantage. Please compare both methods at the same achieved precision or clearly state that the baseline failed to meet the 100% precision requirement. Report the PSR for both methods so the reader can see how often the requirement is genuinely met.","section":"Section V-B"},{"comment":"The evaluation reports only AUPC, a summary metric that balances recall gap and PSR across precision levels. The paper's central claim is reliable precision control, but no per-chunk PSR values at 100% precision are presented. Without knowing the proportion of deployment chunks where the method actually meets 100% precision, the reader cannot assess the method's core promise. Please include PSR tables for each VPR technique, dataset, and method at the 100% precision requirement, and provide distributions of achieved precision across chunks (e.g., box plots).","section":"Section IV-D and Table I"},{"comment":"The paper honestly presents a failure case where the proposed method underestimates the threshold and retrieves incorrect matches. This is exactly the scenario where quantile invariance fails. The paper should quantify how often such failures occur across all datasets and VPR techniques, and analyze whether they correlate with conditions that violate the invariance assumption (e.g., large appearance change or low-quality descriptors). This would provide a more complete picture of the method's reliability and guide future improvements.","section":"Section V-D, Figure 5 (bottom row)"}],"minor_comments":[{"comment":"The abstract claims the method enables 'operating at 100% precision in approximately twice as many deployment scenarios (median improvement)' and 'up to 29% more correct matches,' but these specific statistics are not defined or reported in the results sections. Please include the corresponding quantitative results.","section":"Abstract"},{"comment":"C*_k is a matrix, not a scalar random variable. Please clarify that Pr denotes the empirical fraction of scores in the matrix below MT_k, e.g., by writing out the sum over entries. Similarly, define F^{-1}_{Sval} with respect to the empirical distribution of all scores in Sval.","section":"Eq. (1)"},{"comment":"The sentence 'we take the median of the quantile values {Qk} obtained above' is ambiguous: is the median taken across the k selected calibration queries per deployment query, or across the deployment set? Please specify the exact aggregation.","section":"Section III-D"},{"comment":"There is a typo: 'serve as calibration serve as calibration data' should be 'serve as calibration data'.","section":"Section IV-C"},{"comment":"Typo: 'user-specificed' should be 'user-specified'.","section":"Section VI"},{"comment":"The calibration-enhanced baseline variant is described only briefly. Please provide pseudocode or a precise description of how the baseline's threshold is transferred to deployment, so the comparison is reproducible.","section":"Section IV-B"},{"comment":"The methodology diagram is dense and the font size in the matrices is very small. Consider enlarging the key elements and using clearer arrows to improve readability.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a practical and relevant problem, and the empirical scope is commendable (7 VPR techniques, 3 datasets, with code release). However, the evaluation as written has a potentially serious selection-bias issue in chunk sampling, and the reported 100% precision example appears to compare different operating points. The quantile-transfer assumption is not validated. These are fixable with additional analysis and clarification. I would be willing to review a revised version. I do not see grounds for rejection, provided the authors can supply per-chunk PSR results and clarify the chunk selection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, workmanlike empirical contribution. The core idea — express calibration-learned matching thresholds as quantiles of the calibration score distribution, then apply the median quantile to deployment via the deployment distribution — is simple and clearly described. The adaptive calibration sampling via Pearson correlation of ranked score vectors is a reasonable add-on. It removes a real manual-tuning bottleneck for VPR operating points, and the AUPC improvements over Schubert et al. are consistent in most cells of Table I (e.g., Nordland Summer–Winter, SFU, Oxford for most descriptors). The paper ships code, which helps.\n\nWhere it gets soft: the central claim that quantile transfer yields 100% precision on deployment is asserted, not shown. Section III-C gives only a sentence about relative ordering being stable. There is no controlled shift test, no per-chunk PSR table, and Figure 5’s own bottom row shows the method admitting false positives — so the invariance assumption is load-bearing and visibly violated in at least one case. Also, the dataset description says chunks were selected as those achieving the precision requirement, which sounds like filtering on the outcome; if that is what was done, the AUPC numbers are optimistic and the comparison is not apples-to-apples with the baseline evaluated on unfiltered chunks. That needs to be clarified.\n\nA couple of smaller points. The abstract and the full text disagree on headline numbers (29% vs 25% recall improvement; datasets five vs three — the abstract says five benchmarks but the experiments cover three). The example in Section V-B compares the baseline at 247 correct/38 incorrect vs the method at 57 correct/0 incorrect; those are different precision operating points, so the recall comparison is not a like-for-like headline. None of these are fatal, but they are the kind of thing a careful referee would want fixed.\n\nThe empirical pattern does hold up: quantile transfer generally beats both baselines on its own metric, and the method is not circular — thresholds come from calibration labels and are applied to unlabeled deployment data. The paper is a legitimate extension of Schubert et al., not a new framework, and the quantile invariance remains a heuristic. Still, the evidence is strong enough that this deserves serious peer review. I would bring it to a reading group and cite it if the chunk-selection point is resolved.","headline":"Useful, honest empirical paper on VPR threshold transfer; the method works in most tested cells, but the headline 100% precision claim leans on an unproven quantile-invariance assumption and a chunk-filtering choice.","tokens_in":13205,"tokens_out":598,"would_cite":true,"duration_ms":9199,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Quantile transfer sets VPR thresholds that hold 100% precision on unseen routes.","keywords":["visual place recognition","operating point selection","matching threshold","quantile transfer","precision-recall trade-off","calibration","autonomous navigation"],"falsifier":"Take a dataset with ground-truth correspondences and a strong planned appearance shift, compute the oracle matching threshold for each deployment chunk, convert it to a quantile within that chunk's score distribution, and measure the spread of those quantiles across chunks. If the oracle threshold quantile varies substantially between chunks (for instance, more than a few percent), the quantile-invariance premise is false and the transferred threshold will fail to hold 100% precision on at least some chunks.","tokens_in":12468,"feed_emoji":"🎯","tokens_out":4816,"duration_ms":48783,"temperature":0.7,"pith_summary":"The paper tries to establish that the hard part of deploying a visual place recognition system — choosing the similarity threshold without ground truth at run time — can be automated with a short labeled calibration traversal. The method learns the threshold that meets a user-specified precision on calibration scores, converts it to a quantile (rank within the calibration distribution), and applies the same quantile to the deployment score distribution. The claim is that this quantile is more stable across environments than the absolute score, so the transferred threshold keeps precision at the requested level while maximizing recall. Across seven VPR techniques and three benchmark datasets, the method reports lower area-under-performance-curve values and smaller recall gaps than heuristic baselines, and roughly twice as many deployment scenarios reach 100% precision. If correct, this would let robots trust high-precision place matching without manual tuning per environment.","feed_headline":"Score quantiles can set safe VPR thresholds on unseen routes","feed_subtitle":"A short labeled calibration run plus rank-based threshold transfer keeps precision at 100% across new conditions.","key_machinery":"The load-bearing object is the quantile of the matching threshold, Q_k = Pr(C*_k <= MT_k), transferred to deployment via F^{-1}_{Sval}(Q_k). This converts a scale-dependent absolute similarity threshold into a relative position in the score distribution, exploiting the paper's claim that the rank ordering of similarities is more stable across environments than the raw scores. Adaptive calibration sampling (selecting the k most correlated calibration queries per deployment query) and the median of the resulting quantiles are the mechanisms that make the transfer per-query rather than global.","core_discovery":"The central claim is that threshold transfer should be done in quantile space, not score space. For each deployment query, the method finds calibration queries whose ranked similarity vectors are most correlated with it, computes the matching threshold that maximizes recall at the required precision on each such subset, converts those thresholds to quantiles of the calibration distributions, and takes their median. That median quantile is then mapped through the deployment distribution's inverse empirical CDF to produce the operating threshold. The paper argues and demonstrates empirically that this procedure tracks the oracle threshold more closely than static thresholds or heuristic distri","pith_inferences":["Editorial inference: The core premise is quantile invariance between calibration and deployment; the paper provides empirical evidence but no theoretical or controlled distribution-shift test, so synthetic experiments with known score-distribution shifts would clarify when the method breaks.","Editorial inference: The method assumes that calibration queries are representative of deployment queries in their ranked similarity structure; under severe appearance change or perceptual aliasing where that assumption fails, the transferred threshold could violate the precision target without any online detection mechanism.","Editorial inference: The abstract reports five benchmark datasets while the full evaluation uses three datasets (counting two Nordland traversals separately); a reader reconciling the claims should treat the dataset count as three environments, five traversals.","Editorial inference: If stable, the quantile-transfer idea could generalize beyond VPR to any retrieval system with an accept/reject threshold and a small labeled calibration set, such as loop closure in radar or lidar place recognition."],"forward_implications":["VPR systems could be deployed at user-specified precision levels using only a short calibration traversal, removing per-environment manual threshold tuning.","Dynamic per-query thresholds would replace fixed global thresholds, potentially improving recall on long traversals where score statistics drift.","The method is descriptor-agnostic, so any improvement in VPR front-end descriptors directly compounds with the threshold selection gains.","While demonstrated on single-frame matching, the same quantile-transfer principle could extend to sequence-based or filtering-based VPR with appropriate calibration sampling."],"fun_headline_variants":["Quantile transfer sets VPR thresholds without manual tuning","Rank-based thresholds keep VPR precision at 100% on new routes","Quantile normalization fixes VPR threshold drift","One short calibration run adapts VPR thresholds to new scenes","Median quantile threshold beats hand-tuned VPR operating points"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method rests on the assumption that the quantile of the optimal threshold in the calibration score distribution equals the quantile of the optimal threshold in the deployment distribution; if the relative position of the match/non-match boundary shifts between environments, the transferred threshold will not meet the precision requirement and the method has no mechanism to detect it.","fun_headline_variants_meta":{"raw":{"variants":["Quantile transfer sets VPR thresholds without manual tuning","Rank-based thresholds keep VPR precision at 100% on new routes","Quantile normalization fixes VPR threshold drift","One short calibration run adapts VPR thresholds to new scenes","Median quantile threshold beats hand-tuned VPR operating points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000627,"raw_usage":{"total_tokens":2723,"prompt_tokens":719,"completion_tokens":2004,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1922}},"tokens_in":463,"tokens_out":2004,"duration_ms":12832,"temperature":1.0,"reasoning_tokens":1922,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:36:43.795243+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset with ground-truth correspondences and a strong planned appearance shift, compute the oracle matching threshold for each deployment chunk, convert it to a quantile within that chunk's score distribution, and measure the spread of those quantiles across chunks. If the oracle threshold quantile varies substantially between chunks (for instance, more than a few percent), the quantile-invariance premise is false and the transferred threshold will fail to hold 100% precision on at least some chunks.","supporting_citations":[],"review_version":1}