{"id":"961c10d8-2d81-4c67-b811-e67a0dbc96ea","arxiv_id":"2606.19898","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A machine-learned router predicts per-query recall for filtered ANN methods and selects the recall-QPS optimal one, outperforming fixed baselines on five unseen datasets.","lead":"The paper benchmarks filtered approximate nearest neighbor methods across datasets and predicates, finding no single winner, then introduces a query-aware router that uses a lightweight ML model to predict recall and pick the best method-parameter pair for each query based on an offline recall-QPS table. Smart readers might care because vector search with filters powers modern AI retrieval systems, and routing could deliver better speed-accuracy trade-offs without heavy overh","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Generalization of the three-feature regression model from six training datasets to five unseen validation datasets is the key unverified assumption.","rationale":"The reader's weakest assumption directly identifies the same empirical risk. Full text confirms the training/validation split and three-feature choice but does not add independent verification of predictor fidelity on unseen data, so the UNVERDICTED status and low confidence remain appropriate.","tokens_in":1691,"tokens_out":312,"duration_ms":13975,"concrete_test":"On the five validation datasets, compute mean absolute error and Spearman rank correlation between the model's predicted recall and the ground-truth recall (measured by running each candidate method) for every query; if average MAE exceeds 0.05 or correlation falls below 0.8, the routing decisions cannot be guaranteed to match the reported trade-off.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The router's SOTA recall-QPS claim rests on the regression model (trained on six datasets, reduced to three features via ablation) accurately predicting per-query recall for each candidate method on five held-out validation datasets. If the three features omit query attributes such as predicate selectivity variance or local vector density that differ across datasets, predicted recalls will be inaccurate and the offline table lookup will select suboptimal (method, parameter) pairs. The abstract states the model is applied to unseen data and yields better balance than baselines, but provides no per-dataset prediction error, correlation, or ablation retention metrics on the validation split to confirm the reduction preserves accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper benchmarks major categorical filtered ANN methods across datasets and predicates, finding no single method dominates and that the best method can vary even within a dataset. It proposes a query-aware router that trains a lightweight regression model (reduced via ablation from 22 to 3 features) on six datasets to predict per-query recall for candidate methods, then uses an offline table of measured recall/QPS values to select the method-parameter pair with the best trade-off. The model is applied to five unseen validation datasets, with the claim that the router achieves SOTA recall-QPS balance versus baselines at negligible latency cost.","tokens_in":1815,"tokens_out":555,"duration_ms":21195,"significance":"If the reported generalization holds, the framework addresses a practical limitation in filtered ANN search for vector databases by enabling dynamic, query-specific method selection without significant overhead. The choice of regression over classification and the feature ablation are constructive steps toward deployable systems; the offline table plus predictor approach is a reasonable engineering pattern when the predictor is shown to be reliable.","major_comments":[{"comment":"Abstract: the SOTA recall-QPS claim on the five validation datasets is load-bearing on the three-feature regression model's accuracy, yet the manuscript reports no per-dataset prediction error (e.g., MAE, R²), correlation, or ablation-retention metrics on the held-out validation split to confirm that the feature reduction preserves predictive power for unseen data.","section":"Abstract"},{"comment":"The experimental results section (and abstract) states consistent gains on five validation sets after training on six others, but provides no error bars, exact dataset statistics, or full experimental protocol (training procedure, hyper-parameters, query sampling), leaving the magnitude and reliability of the gains difficult to assess.","section":"Experimental results"},{"comment":"The router's selection logic relies on an external offline benchmark table; it is unclear from the description whether this table was constructed independently for the validation datasets or whether any form of data leakage exists between the table construction and the validation queries.","section":"Query-aware routing framework"}],"minor_comments":[{"comment":"Notation for the three retained features after ablation should be defined explicitly (e.g., what the final three features represent) rather than left as 'the final three features.'","section":"Ablation study"},{"comment":"The manuscript would benefit from a table or figure showing the distribution of selected methods across the validation queries to illustrate that routing is indeed query-dependent.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive feedback. We address each major comment below. Where the comments identify missing details or clarity issues, we will revise the manuscript to incorporate the requested information and clarifications.","responses":[{"response":"We agree that explicit per-dataset metrics on the validation split would better substantiate the generalization claim. In the revision we will add a table reporting MAE, R², Pearson correlation, and ablation-retention statistics for the three-feature model on each of the five held-out validation datasets individually.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the SOTA recall-QPS claim on the five validation datasets is load-bearing on the three-feature regression model's accuracy, yet the manuscript reports no per-dataset prediction error (e.g., MAE, R²), correlation, or ablation-retention metrics on the held-out validation split to confirm that the feature reduction preserves predictive power for unseen data."},{"response":"The current version indeed omits these details. We will expand the experimental results section to include error bars on all reported recall-QPS figures, exact per-dataset statistics (size, dimensionality, predicate distributions), and a complete protocol subsection specifying training procedure, hyper-parameters, cross-validation folds, and query sampling methodology.","revision_made":"yes","referee_comment":"[Experimental results] The experimental results section (and abstract) states consistent gains on five validation sets after training on six others, but provides no error bars, exact dataset statistics, or full experimental protocol (training procedure, hyper-parameters, query sampling), leaving the magnitude and reliability of the gains difficult to assess."},{"response":"The offline table for each validation dataset was built using a disjoint query workload sampled independently from the queries used to evaluate the router. No queries or embeddings from the validation evaluation set were used to populate the table. We will revise the framework description to state this separation explicitly and add a short paragraph confirming the absence of leakage.","revision_made":"yes","referee_comment":"[Query-aware routing framework] The router's selection logic relies on an external offline benchmark table; it is unclear from the description whether this table was constructed independently for the validation datasets or whether any form of data leakage exists between the table construction and the validation queries."}],"tokens_in":1397,"tokens_out":500,"duration_ms":20666,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is a router that trains a small regression model on query features to predict recall for each filtered ANN method, then picks from an offline table of measured QPS/recall values to get the best trade-off per query. They first show that no method wins consistently across datasets and predicates, which justifies the routing approach.\n\nThe benchmarking across methods and the decision to use regression plus an ablation down to three features are solid steps. Training on six datasets and evaluating on five held-out ones is a reasonable way to check whether the router transfers.\n\nThe soft spot is the lack of evidence that the three features actually predict recall accurately on the validation sets. The abstract claims better balance than baselines but gives no prediction error, correlation, or ablation retention numbers on the held-out data. If those features miss things like predicate selectivity or local density differences, the router could pick suboptimal methods without anyone noticing. No error bars or detailed protocol details make the gains harder to judge.\n\nThis is for engineers and researchers who tune vector search in databases or RAG systems. A reader looking for a concrete way to handle method variability would find the routing concept useful.\n\nIt deserves peer review because the problem is real and the empirical framing is honest enough to get useful feedback, even if more validation on the predictor is needed.","headline":"The routing idea is practical but rests on an unverified assumption that a three-feature regressor generalizes recall predictions across datasets.","tokens_in":2319,"tokens_out":338,"would_cite":false,"duration_ms":27023,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A lightweight model routes each filtered ANN query to the method with the best recall-QPS tradeoff for that query.","keywords":["filtered ANN","approximate nearest neighbor search","query routing","vector database","recall prediction","QPS optimization","machine learning for databases"],"falsifier":"Measure whether the router's chosen methods on a sixth unseen dataset actually deliver higher recall at the same QPS than the single best fixed baseline across the same queries.","tokens_in":2561,"feed_emoji":"🔍","tokens_out":717,"duration_ms":19035,"temperature":0.7,"pith_summary":"No single filtered ANN method performs best across all queries, even within one dataset and predicate type. The authors therefore train a regression model on six datasets to predict recall for each candidate method from three query features, then consult an offline table of measured recall and QPS values to pick the method with the strongest tradeoff. The router is applied without retraining to five unseen validation datasets under three predicates and records better overall recall-QPS balance than any fixed baseline while adding negligible latency. A reader would care because modern vector databases and retrieval-augmented generation systems depend on fast filtered vector search, and query-dependent selection removes the need to commit to one suboptimal method in advance.","feed_headline":"Router picks best filtered ANN method per query","feed_subtitle":"Lightweight regression model predicts recall from three features and selects the strongest recall-QPS tradeoff on unseen datasets.","key_machinery":"Query-aware routing framework that combines a three-feature regression recall predictor with an offline benchmark table mapping each method and parameter setting to its measured recall and QPS.","core_discovery":"No single categorical filtered ANN method dominates across datasets and predicates, and the best method for a given query can change even within a single dataset and predicate. The query-aware router therefore uses a lightweight regression model, narrowed by ablation to three features, to predict each method's recall on the incoming query; it then consults a precomputed table that records the actual recall and QPS of every method-parameter pair and selects the pair offering the best recall-QPS tradeoff. The model is trained on six real-world datasets and evaluated on five held-out validation datasets, where the router matches or exceeds the recall-QPS frontier of all existing filtered ANN ba","pith_inferences":["The routing approach could be applied to other query-time choices in databases, such as selecting between different index structures or join algorithms.","If the three-feature model continues to generalize, production systems could reduce manual parameter tuning for filtered ANN workloads.","The technique mirrors cost-model-driven operator selection in traditional query optimizers, suggesting a broader pattern for adaptive retrieval pipelines."],"forward_implications":["Vector databases can replace a single fixed filtered ANN method with dynamic per-query selection without rebuilding indexes.","The same router design improves the recall-QPS operating point on every one of the five validation datasets tested.","Only three query features are required to keep prediction accurate enough for the router to outperform static baselines.","Adding the router introduces negligible latency overhead while still achieving state-of-the-art balance on unseen data."],"fun_headline_variants":["ML router picks filtered ANN method by predicted recall","Query features guide best recall-QPS filtered ANN choice","No method leads filtered ANN; router adapts per query","Three-feature model routes to top filtered ANN tradeoff","Predict recall then select optimal filtered ANN per query"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The lightweight regression model trained on six datasets can accurately predict recall for each candidate method on queries from five unseen validation datasets using only the final three features.","fun_headline_variants_meta":{"raw":{"variants":["ML router picks filtered ANN method by predicted recall","Query features guide best recall-QPS filtered ANN choice","No method leads filtered ANN; router adapts per query","Three-feature model routes to top filtered ANN tradeoff","Predict recall then select optimal filtered ANN per query"]},"model":"grok-4.3","cost_usd":0.00292,"raw_usage":{"total_tokens":1616,"prompt_tokens":681,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":29199500,"prompt_tokens_details":{"text_tokens":681,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":864,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":681,"tokens_out":71,"duration_ms":4669,"temperature":1.0,"reasoning_tokens":864,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T15:28:11.978919+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Measure whether the router's chosen methods on a sixth unseen dataset actually deliver higher recall at the same QPS than the single best fixed baseline across the same queries.","supporting_citations":[],"review_version":1}