{"id":"5ca9b066-7cbe-4da5-96ea-051c0f1f8109","arxiv_id":"2412.03573","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-generated retrieval queries, especially from an alignment-trained model, improve tool retrieval for seen and unseen APIs over retrieving with the raw user utterance.","lead":"This paper tests using a large language model to turn a user's request into several search queries, then finds matching tool descriptions by embedding similarity. The best trained variant improves tool retrieval over using the request as-is, particularly for tools not seen during training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Out-of-domain claim may be an artifact of selecting alignment iterations and temperatures on the OOD test set; a held-out validation split is needed before the 3.27-point gain can be trusted.","rationale":"The reader's weakest assumption (fixed embedding model sensitivity) is plausible but secondary: even if the embedding space were varied, the paper would still need a fair OOD evaluation. The more decisive weakness is that the distinctive OOD result may be an artifact of hyperparameter and iteration selection on the test set. The manuscript never describes a validation split. Section 4.3 describes temperature calibration without saying which data are used; Appendix A.3 reports T=5 after observing the OOD curve; Section 4.4 states that the best OOD performance occurs at iteration 5. Selecting the operating point from the evaluation curve invalidates the reported 3.27-point OOD gain as an unbiased estimate. The absence of uncertainty quantification and released code further weakens confidence in this small margin. A nested validation split is the minimal test: if the alignment advantage survives, the paper's contribution stands; if it disappears, the 'especially OOD' claim needs substantial qualification. For these reasons, the reader's CONDITIONAL verdict remains appropriate, and the condition should explicitly include held-out validation of all reported operating points. I partially agree with the reader because they note the temperature-selection problem in passing but identify the embedding model as the weakest assumption; I would prioritize the missing validation split.","tokens_in":10099,"tokens_out":4447,"duration_ms":58904,"concrete_test":"Construct a nested holdout before running any alignment or temperature search: split the 373 out-of-domain APIs and 4,451 requests into disjoint validation and test sets (e.g., 50/50, ensuring no overlapping APIs). Use validation only to select temperature, number of alignment iterations T, reward metric, and prompt variant. Then evaluate the chosen models once on the held-out test set. Report Recall@5 with bootstrap 95% confidence intervals across 1,000 resamples. If Alignment's OOD advantage over Utterance falls below the interval or is no longer significant, the 'especially OOD' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that alignment learning gives the best end-to-end retrieval performance, 'especially for the challenging out-of-domain settings' (Section 4.4). The support is Table 1: Alignment Recall@5 is 78.53% vs. 75.26% for the utterance baseline, a 3.27-point OOD gain. This comparison is fragile because the OOD numbers appear to be selected rather than predicted. Appendix A.1 says each method's temperature was calibrated by measuring Recall@5 while varying temperature from 0 to 1.7, and the best temperature (0.1 for alignment) is then used in the reported table; no evaluation split is specified. Appendix A.3 reports T=5 as the best alignment setting, and Section 4.4 explicitly says 'the best out-of-domain performance is reached after 5 iterations.' If these choices were made by inspecting OOD test Recall@5 (or the same 4,451 examples), then the reported 3.27-point advantage is the maximum of a small search over iterations and temperatures on the evaluation set, not an unbiased estimate. With no confidence intervals and no code, the difference could easily be noise or selection bias. The in-domain SFT result is large and less vulnerable, but the distinctive OOD claim rests on a small margin that needs protection from selection on the test set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to improve dense tool retrieval by using an LLM to generate retrieval queries from user utterances, embedding those generated queries, and then performing nearest-neighbor search over an index of API descriptions. Three query-generation approaches are compared: zero-shot prompting, supervised fine-tuning on tool descriptions, and an iterative alignment-learning method that optimizes a retrieval reward via rejection sampling. Experiments on a ToolLLM-derived dataset report in-domain and out-of-domain retrieval metrics. The main empirical claims are that supervised fine-tuning gives large in-domain gains and that alignment learning gives the best out-of-domain Recall@5, while zero-shot prompting is generally worse than using the utterance directly.","tokens_in":10316,"tokens_out":6419,"duration_ms":62680,"significance":"If the out-of-domain alignment result holds, the paper offers a practical and conceptually simple recipe for improving tool retrieval without retraining the retriever: keep the embedding index fixed and train the query generator against a retrieval reward. The paper includes useful ablations (adding the utterance to the query set, rejection-sampling filter settings, reward-metric choice, and intent-versus-description generation) and qualitative examples that illustrate the intended behavior. The in-domain SFT gains are large and consistent. However, the headline out-of-domain advantage currently rests on a small margin (3.27 Recall@5 points) that appears vulnerable to test-set-based model selection and is reported without variance estimates, so the significance is conditional on those issues being resolved.","major_comments":[{"comment":"The abstract's blanket claim that LLM-generated queries improve retrieval for in-domain and out-of-domain settings is not supported by Table 1: the zero-shot method is below the utterance baseline on Recall@5 in both settings (57.86% vs. 63.82% in-domain; 71.76% vs. 75.26% out-of-domain), and SFT's out-of-domain Recall@5 advantage (76.18% vs. 75.26%) is accompanied by lower MMRR and MAP (0.6130 vs. 0.6290 and 0.6893 vs. 0.7031, respectively). The claim should be restricted to the alignment method, or the stated results should be revised.","section":"Abstract; Table 1"},{"comment":"The out-of-domain advantage of alignment appears to be selected on the test set. Appendix A.1 reports temperature calibration by measuring Recall@5 without specifying a held-out split; Appendix A.3 reports that T=5 was chosen as the best alignment setting; and Section 4.4 states that the best out-of-domain performance is reached after 5 iterations, matching Figure 3's out-of-domain peak. If these choices were made using the same 4,451 out-of-domain test examples, the reported 78.53% vs. 75.26% Recall@5 gain is the maximum of a small search over temperatures and iterations, not an unbiased estimate. Please add a proper validation split or otherwise demonstrate that model selection did not use the test labels, and report variance (e.g., confidence intervals or multiple seeds) for the headline comparison.","section":"Section 4.3, Section 4.4, Appendices A.1 and A.3"},{"comment":"Equation (2) is internally inconsistent with its prose explanation. The text says the numerator is the average rank position for perfect retrieval of n items, which is (n+1)/2, but the formula as typeset uses n/2. For n=1, a perfect retrieval at rank 1 would yield 0.5 rather than 1. Because MMRR is used as the alignment reward (Appendix B.3), the definition needs to be corrected or clarified; otherwise the objective being optimized is unclear.","section":"Equation (2)"},{"comment":"The evaluation fixes the embedding model to all-mpnet-base-v2 and never varies it. The central mechanism assumes that LLM-generated query descriptions and developer-written API descriptions are close in this particular embedding space; without at least one alternative embedding model (or an analysis of query-embedding distribution shift), it is unclear whether the measured gains reflect better query wording or an artifact of the fixed encoder. Please add a sensitivity experiment or discuss this limitation explicitly.","section":"Section 4.2 / Retriever Setup"}],"minor_comments":[{"comment":"The phrase 'rdraf t' appears to be a typo for the variable n_draft defined in Algorithm 1; please harmonize the notation.","section":"Appendix A.3"},{"comment":"The sentence 'we found it it be less inclined to hallucination' contains a typo; it should read 'we found it to be less inclined to hallucinate'.","section":"Section 3.1"},{"comment":"The caption reads 'the distribution of at each iteration'; the word 'rewards' is missing.","section":"Figure 4"},{"comment":"The column header 'No Gen. LLM-Gen.' is awkward; please restructure as separate header levels, e.g., 'Baseline' and 'LLM-Gen'.","section":"Table 1"},{"comment":"The sentence 'Delta value are reported compared to the tool description generation approach' should be 'Delta values are reported'.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper's main contribution is the alignment-learning recipe, and the in-domain results are strong. My main concern is the statistical foundation of the out-of-domain claim: test-set-based model selection appears likely from the appendices. I would want to see a validation split, variance estimates, and ideally code before publication. If the authors can supply those, the paper could be a solid contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a genuinely useful idea—use the LLM to rewrite user requests into tool queries, then optimize that rewriting against retrieval reward via rejection sampling—but the headline out-of-domain result, a 3.27-point Recall@5 gain, is not yet credible because the authors picked temperature and alignment iterations by looking at the test set. The in-domain SFT result is much larger and not in doubt.\n\nWhat's new: alignment learning for query generation, where you generate multiple drafts, score them by retrieval reward on training requests, and fine-tune on the best drafts, iterating. That is a clean way to train an LLM to produce retrieval-friendly queries for unseen tools without retraining the retriever. The paper also does real ablation work: adding the original utterance back, changing the reward metric, filter settings, and intent vs. description generation. The qualitative examples show the model decomposing requests sensibly. The SFT gains on in-domain tools are large (63.82% → 87.29% Recall@5), which is consistent and meaningful.\n\nSoft spots. The abstract says LLM query generation improves both in-domain and out-of-domain retrieval, but zero-shot prompting is worse than the utterance baseline on both test sets (e.g., 71.76% vs 75.26% OOD Recall@5). So the claim only holds for trained methods. No variance or significance is reported anywhere. Temperature calibration (Appendix A.1) and the choice of T=5 alignment iterations (Section 4.4) both appear to be based on Recall@5 on the evaluation sets. That means the OOD alignment number is the best of a small search over temperatures and iterations, not an unbiased estimate. You need a held-out validation split before trusting the 3.27-point margin. The fixed embedding space is a secondary concern: no experiment checks whether generated queries are in-distribution for all-mpnet-base-v2, so part of the gain could be an artifact of the embedding model. And there's no comparison to HyDE or other LLM rewrites, which are the natural baselines for this exact idea.\n\nBottom line: the alignment method deserves attention, and the paper is worth refereeing. But the evaluator should insist on a real validation split, confidence intervals, and those baselines. As is, I'd cite the method but not the OOD number.","headline":"The alignment loop is a real idea, but the 3-point out-of-domain win looks like it was chosen on the test set; treat that number as provisional.","tokens_in":10885,"tokens_out":2373,"would_cite":true,"duration_ms":27181,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-written queries lift tool retrieval up to 23 points on known tools.","keywords":["tool retrieval","query generation","alignment learning","dense retrieval","in-context learning","rejection sampling"],"falsifier":"Re-run the out-of-domain comparison with a different frozen embedding model (e.g., a larger sentence transformer or an instruction-tuned encoder). If the alignment method's gain over the utterance baseline shrinks or disappears under a different embedding space, the central claim that query generation is the driver of the improvement is undermined. A second check: replace the dense index with BM25 over the same descriptions; if the generated queries still outperform raw utterances, the benefit is genuinely in the query text.","tokens_in":9872,"feed_emoji":"🛠️","tokens_out":6574,"duration_ms":58878,"temperature":0.7,"pith_summary":"Tool retrieval in LLM applications is usually done by vector-searching a user request against a fixed index of API descriptions. This paper proposes to insert one extra step: have an LLM read the request and write a few short tool descriptions that serve as retrieval queries. Three ways to produce those descriptions are compared, and the best in-domain results come from supervised fine-tuning on API descriptions, while an iterative alignment method that optimizes a retrieval reward gives the best out-of-domain results. The implication is that much of the retrieval quality comes from query formulation, not from the retriever architecture.","feed_headline":"LLM-written queries lift tool retrieval up to 23 points on known tools","feed_subtitle":"A frozen embedding index can gain 3.3 points on unseen tools when query generation is aligned to retrieval reward","key_machinery":"The load-bearing component is the query generator: a LLaMA-13B model prompted to output up to five tool descriptions, each under twenty words, from a user request. These descriptions are embedded with the frozen all-mpnet-base-v2 model, and the resulting ranked lists are interleaved with the ranked list of the original utterance. The most distinctive mechanism is alignment learning: for each training request, the model stochastically generates 24 drafts, each draft is scored by an MMRR-style retrieval reward computed against the fixed index, the top-scoring drafts are kept after filtering, and the model is fine-tuned on those drafts; the loop repeats for five iterations.","core_discovery":"The paper's central claim is that decomposing a user request into a set of LLM-generated tool descriptions, then embedding those descriptions with a fixed sentence encoder and running cosine nearest-neighbor search over API descriptions, yields large retrieval gains over using the raw utterance. On the in-domain test set, supervised fine-tuning improves Recall@5 from 63.82% to 87.29%; on the out-of-domain test set, alignment learning improves Recall@5 from 75.26% to 78.53% while also improving MMRR and MAP. The authors interpret the alignment result as evidence that teaching the LLM to generate queries that maximize retrieval reward transfers better to unseen tools than forcing it to reproduce developer-written API descriptions.","pith_inferences":["If the same gain appears with a different frozen encoder, the method would be a portable pre-processing step; if not, the results are partly an artifact of all-mpnet-base-v2's embedding geometry, which the paper does not test.","The hallucinated API names in generated queries (e.g., PartyPlanningAPI) could be exploited as a data-augmentation signal or as a failure mode to suppress; even though the paper reports no notable regression, the qualitative examples show the model is composing names rather than retrieving from the catalog.","The alignment objective could be re-run with retrieval reward replaced by end-to-end task success (tool call correctness), which would likely change which drafts are kept and may yield different cross-domain behavior."],"forward_implications":["Retrieval pipelines can keep a frozen embedding index and adapt to new tool domains by retraining only the query-generation LLM.","Alignment learning provides a way to improve out-of-domain tool retrieval without new API data, as long as a training set of request-API pairs exists.","Appending the original utterance to the generated query set is a simple additive boost, especially for zero-shot generation and out-of-domain cases.","The temperature of generation matters: zero-shot works best at 1.3, SFT at 0.6, and alignment at 0.1, suggesting the alignment-trained model becomes more deterministic as it specializes."],"supporting_citations":[{"why":"Supplies the ToolLLM dataset of complex multi-tool requests and relevant APIs used for both training and evaluation.","marker":"Qin et al. (2023b)"},{"why":"Provides the Sentence-BERT architecture and the all-mpnet-base-v2 model used as the frozen retriever encoder.","marker":"Reimers and Gurevych (2019)"},{"why":"Supplies the 13B LLaMA model that is prompted and fine-tuned to generate retrieval queries.","marker":"Touvron et al. (2023)"},{"why":"Provides the rejection-sampling procedure that the alignment loop adapts to reward high-retrieval drafts.","marker":"Bai et al. (2022)"}],"fun_headline_variants":["LLM rewrites beat raw text for tool retrieval by 23 points","23-point tool-retrieval gain from LLM-generated queries","Alignment boosts unseen-tool retrieval recall by 3.3 points","LLM query generation lifts retrieval on known and unseen tools","Simpler retrieval with LLM query generation, 23 points better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes the fixed all-mpnet-base-v2 embedding space places LLM-generated tool descriptions in the same semantic neighborhood as developer-written API descriptions, so that cosine nearest-neighbor search over a frozen index is a faithful retrieval function.","fun_headline_variants_meta":{"raw":{"variants":["LLM rewrites beat raw text for tool retrieval by 23 points","23-point tool-retrieval gain from LLM-generated queries","Alignment boosts unseen-tool retrieval recall by 3.3 points","LLM query generation lifts retrieval on known and unseen tools","Simpler retrieval with LLM query generation, 23 points better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000237,"raw_usage":{"total_tokens":1490,"prompt_tokens":910,"completion_tokens":580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":491}},"tokens_in":526,"tokens_out":580,"duration_ms":5783,"temperature":1.0,"reasoning_tokens":491,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:07:29.531791+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the out-of-domain comparison with a different frozen embedding model (e.g., a larger sentence transformer or an instruction-tuned encoder). If the alignment method's gain over the utterance baseline shrinks or disappears under a different embedding space, the central claim that query generation is the driver of the improvement is undermined. A second check: replace the dense index with BM25 over the same descriptions; if the generated queries still outperform raw utterances, the benefit is genuinely in the query text.","supporting_citations":[],"review_version":1}