{"id":"af05af80-9658-42ff-91e2-1550f9817dee","arxiv_id":"2607.22639","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Training an LLM to think through business rules before outputting tool tokens prevents catastrophic forgetting of tool knowledge and enables fast single-beam tool retrieval.","lead":"TRACE is a two-stage training method that makes language models retrieve enterprise APIs by writing a short reasoning note before listing tool IDs, preserving tool knowledge that earlier training wiped out. Tests on more than 8,000 proprietary SAP tools suggest it can run at production speed, but the headline 86% recall is measured with a looser metric than the 27% embedding baseline it is compared with.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rule-citation evidence is confounded by forced training attribution; no causal test shows reasoning traces drive retrieval, undermining the paper's central reasoning mechanism.","rationale":"The reader's weakest assumption flags both the representativeness of the synthetic data/rule catalog and the possibility that traces are post-hoc rationalizations. I focus on the latter because it is more directly load-bearing for the paper's central mechanism claim, and because the programmatic filter's forced rule attribution provides a concrete, checkable confound that the reader did not fully foreground. If reasoning traces are not causal, the paper's narrative that 'reasoning over business rules' preserves knowledge and enables retrieval is unsupported, even if the empirical dissociation remains. However, the reader's verdict is already CONDITIONAL, citing lack of external validation and statistical issues; my concern adds a specific reason to require causal evidence, but does not change the verdict direction. I therefore recommend UNCHANGED: the paper should remain conditional pending release of data and, additionally, a causal trace-intervention study. I partially agree with the reader's weakest_assumption: I agree on the post-hoc rationalization risk, but do not make the rule catalog's representativeness the primary issue.","tokens_in":17858,"tokens_out":10567,"duration_ms":107976,"concrete_test":"Perform a causal intervention on the trained TRACE(c,m,r+R) checkpoint over the 131 Domain-A PRB queries. For each query, first record the model's generated trace and final JSON tool list. Then re-run inference with the trace edited to replace the cited business rule with a different, incorrect rule (or to remove the rule citation entirely), holding all other text fixed. Measure the proportion of queries for which the final output tool set changes. If the tool list is invariant to the trace manipulation in the large majority of cases, the traces are post-hoc rationalizations and the §4.4 citation association is not causal. As a complementary check, retrain r+R with the rule-attribution clause of filter V removed, and test whether rule citation and knowledge preservation persist without forced supervision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that Stage-2 reasoning-augmented retrieval (r+R) preserves tool knowledge by making the model reason over business rules. The key evidence is the §4.4 rule-citation analysis, which the paper says confirms the model 'genuinely reasons about rules rather than memorizing answer mappings.' But this evidence is confounded by the training pipeline. Appendix B defines a programmatic filter V that enforces 'rule attribution: for rule-targeted samples, z explicitly invokes r∗' — i.e., every rule-targeted training example is required to cite the rule. The model is therefore supervised directly to emit rule citations, so it is unsurprising that after training it cites rules on 71% of Domain-A PRB traces, and that traces citing a rule have higher recall (94.6% vs 63.2%). This is a self-fulfilling pattern, not evidence of emergent, genuine reasoning. The LLM judge's 'trace faithfulness' score is also a model-based judgment, not a causal test. Furthermore, the reference traces are generated by an unnamed teacher LLM after the answer set A is already known (§3.3, Appendix B), so the traces may be post-hoc rationalizations that justify the answer rather than produce it. The 'name-token coupling' mechanism (§3.3) could equally explain knowledge preservation: placing virtual tokens inside descriptive text reinforces Stage-1 binding independent of any reasoning structure. If the traces are not causally responsible for the retrieved tool set, the paper's distinct contribution — business-rule-grounded reasoning as a knowledge-preserving curriculum — is not independently evidenced, even though the empirical dissociation (r/n vs. n) may still hold.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TRACE, a two-stage curriculum for parametric tool retrieval in enterprise settings. Stage 1 reuses the ToolSense multi-format memorization SFT to seed tool knowledge; Stage 2 adds a reasoning-augmented retrieval SFT in which the model is trained to emit a business-rule-grounded thinking trace before producing a JSON list of virtual tool tokens. The central claims are that this recipe preserves and even improves parametric tool knowledge relative to non-reasoning retrieval training, that the rule-grounded traces are causally responsible for the knowledge-preservation and retrieval gains, and that single-beam greedy decoding provides production-viable latency. The paper evaluates on a proprietary 8,283-tool catalog and reports about 86% Domain A and 60% Domain B recall under greedy decoding, while replicating the ToolSense dissociation on three token formats and an expert-authored MCQ probe.","tokens_in":1809,"tokens_out":1764,"duration_ms":79065,"significance":"If the causal interpretation were established, the contribution would be significant: it offers a concrete way to avoid catastrophic forgetting in parametric tool retrieval and removes constrained beam search at inference time. The paper has real strengths: a systematic configuration grid across token formats, a held-out expert MCQ benchmark with a computed random baseline, paired-bootstrap confidence intervals, a Pareto-style rule-data sweep, and an end-to-end latency measurement. These elements are exactly the kind of evidence needed for a production-oriented retrieval paper. However, the evidence for the paper's distinctive mechanism (that reasoning traces over business rules cause knowledge preservation and retrieval improvement) is currently confounded by the training-data construction, and the headline retrieval comparison is presented in an apples-to-oranges way. The contribution is therefore promising but not yet established at the level claimed.","major_comments":[{"comment":"The rule-citation analysis is not probative evidence that the model genuinely reasons about rules rather than memorizing answer mappings. Appendix B's programmatic filter V explicitly requires that, for rule-targeted samples, the reference trace z explicitly invokes the source rule r*. The model is therefore supervised to produce rule citations. Observing a 71% rule-citation rate after rule-grounded training, and higher recall on traces that cite rules (94.6% vs 63.2%), is a self-fulfilling pattern. In addition, the teacher LLM generates the trace after the answer set A is already known, so the traces are explanatory rationalizations, not evidence of causal reasoning. The LLM judge's trace-faithfulness axis does not fix this. A causal test is needed, e.g., train a rule-grounded variant without the rule-attribution filter, or intervene on the trace at inference and measure the effect on r","section":"§4.4 and Appendix B"},{"comment":"The reasoning-retrieval comparison is confounded by the name-token coupling design. In Stage 2, the reference trace and the answer set both substitute virtual tokens into the deliberative context; the paper states this substitution is the mechanism by which Stage 2 reinforces Stage 1 knowledge. Non-reasoning retrieval (R=n) does not provide this token-in-context exposure. Thus the contrast between R=n and R=r conflates two variables: the presence of a reasoning trace and the presence of virtual tokens embedded in descriptive text. The preservation gains attributed to reasoning could be wholly or partly due to additional token-binding exposure. A control condition, e.g., a non-reasoning or minimally-reasoning Stage 2 that still places virtual tokens in a descriptive context, is required to isolate the claimed reasoning mechanism.","section":"§3.3 and §4.2"},{"comment":"The headline comparison of about 86% Domain A and 60% Domain B recall against embedding baseline performance of about 27% and 52% is apples-to-oranges. Table 3 reports R@gen, recall over the model's own uncapped answer set, while the embedding baseline in Table 2 is R@10 over a fixed candidate window. The abstract's phrase 'both with single-beam greedy decoding' does not accurately describe the embedding baseline. Although Table 2 provides a fairer constrained-decoding R@10 comparison, the abstract and Section 4.3 should make the decoding protocol explicit and base the headline claim on the same candidate-set definition, or report an embedding baseline under the same generated-set metric.","section":"Abstract, §4.3, Tables 2 and 3"},{"comment":"The core recipe is heavily dependent on the unpublished ToolSense (Anonymous, 2026) work for the Stage 1 memorization recipe, the RRB generation pipeline, the MCQts/QAts probe construction, and the hyperparameter settings. Appendix C states that optimizer, learning-rate schedule, batch size, sequence length, and LoRA parameters follow the ToolSense recipe without providing the details. This is a load-bearing reproducibility problem: a reader cannot instantiate Stage 1, generate the seed data, or construct the probing benchmarks independently. Please either include the full ToolSense details in the appendix, release the synthetic data and evaluation probes, or use a public analog. The same applies to the unnamed teacher LLM used for query and trace generation.","section":"Appendix C and §3.2"},{"comment":"The reported knowledge-preservation improvements are inconsistent. The Abstract says MCQ accuracy gains +3.2 pp and QA probing gains +9 pp over Stage 1, which roughly matches Table 4 (e.g., F=c, M=m: MCQts mean change about +3.25 pp, QAts mean change about +9 pp). The Conclusion instead says improving Stage-1 MCQ and QA probing by +7.6 pp and +4.5 pp respectively. These are not the same numbers and are not explained. The discrepancy must be resolved, and the specific domains and formats used for each aggregate should be stated.","section":"Abstract vs. Conclusion"}],"minor_comments":[{"comment":"Typo: 'apple-to-apple' should be 'apples-to-apples'.","section":"§4.3"},{"comment":"The line '∅: 69.2 n: 33.7 r: 61.5 r+R: 56.4' is hard to read; it should be formatted as a table or with explicit configuration labels (e.g., F=c, M=m).","section":"§4.2"},{"comment":"The programmatic filter's rule-attribution requirement is defined only for rule-targeted samples. Please clarify whether it also applies to RRB samples and how the filter interacts with the LLM judge's trace-faithfulness score.","section":"Appendix B"},{"comment":"The off-vocabulary rate of 'mere 3%' for TRACE(c,m,r+R) is an important decoding-quality claim but is given without a table or confidence interval. Please report it systematically for all reasoning checkpoints.","section":"Appendix D"},{"comment":"The '(app)' row has a missing MCQexpert entry because the corresponding checkpoint has not yet been evaluated. An incomplete row in a results table should be flagged more prominently, and the missing evaluation should be completed before publication.","section":"Table 6"},{"comment":"The system prompts are detailed, but the identity of the teacher LLM (and its version and temperature) is not disclosed. This matters for reproducibility of the synthetic data.","section":"Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's dependence on an anonymous, unpublished ToolSense paper is substantial: the Stage 1 recipe, RRB pipeline, probing benchmarks, and hyperparameters all come from that work. Even setting aside the confounds I raise in the main comments, I cannot fully verify the central comparison without access to that underlying system. If the authors cannot de-anonymize or release sufficient details, the paper's reproducibility will be much harder to judge. I would also encourage the editor to check whether the configuration axes and evaluation instruments overlap so heavily with Anonymous (2026) that the novelty boundary between the two works is clearly delineated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper deserves a serious referee, but the headline causal claim is weaker than the authors suggest. What's actually new: a two-stage fine-tuning recipe for parametric tool retrieval that appears to preserve probe-measured tool knowledge while enabling single-beam greedy decoding. The dissociation is large and consistent: non-reasoning retrieval collapses MCQ/QA probing to near random across three token formats, and reasoning retrieval keeps them near Stage-1 levels. That is a real empirical finding, and the latency numbers (roughly 200x throughput gap) make it practically relevant for enterprise copilots.\n\nThe business-rule grounding experiment is also genuinely informative: adding rule-grounded data lifts Domain A R@gen substantially, and the Pareto cost on held-out MCQ expert is small. The paper is unusually transparent about its synthesis pipeline.\n\nThe soft spots are in the interpretation. The rule-citation analysis (§4.4) is not strong evidence of \"genuine\" reasoning. The programmatic filter V requires rule attribution in every rule-targeted training sample, so the model is supervised to cite rules; that it does so at 71% and gets higher recall when it does is partly a self-fulfilling pattern. The teacher LLM that writes traces also sees the final answer set, so traces could be post-hoc rationalizations. The name-token coupling mechanism could explain knowledge preservation without any real reasoning. None of this kills the empirical result — the dissociation is still there — but it means the \"reasoning curriculum\" mechanism is not independently evidenced.\n\nThere are also reporting issues. The abstract says +3.2 pp MCQ / +9 pp QA over Stage 1; the conclusion says +7.6 / +4.5. The headline comparison mixes R@gen (uncapped, model-chosen set) with embedding R@10 (fixed top-k), though Table 2 does provide a fair constrained-decoded comparison. The rule-data sweep in Figure 3 doubles as hyperparameter selection on the evaluation benchmark. And the heavy dependence on an anonymous, unreleased ToolSense paper makes the recipe hard to reproduce.\n\nIf I were editing, I would send it out. The empirical dissociation and the latency result are worth refereeing even if the causal story needs to be softened and the data/benchmarks (or a comparable public catalog) released.","headline":"A useful empirical dissociation and a real latency win, but the causal rule-reasoning story is overclaimed and the evaluation depends heavily on an unreleased ToolSense paper.","tokens_in":18724,"tokens_out":2002,"would_cite":true,"duration_ms":19769,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TRACE trains an LLM to reason over business rules before emitting tool tokens, preserving parametric tool knowledge while enabling single-beam greedy retrieval at production latency.","keywords":["parametric tool retrieval","virtual tokens","reasoning traces","business rule grounding","knowledge preservation","greedy decoding","enterprise LLM","tool catalog"],"falsifier":"Retrain Stage 2 on the same query-answer pairs but strip rule citations from every reasoning trace, then compare Domain A recall and the held-out expert multiple-choice accuracy. If recall stays near 86% and probing accuracy stays high, the business-rule mechanism is not doing the work the paper attributes to it; if recall drops sharply, rule citation is load-bearing.","tokens_in":17771,"feed_emoji":"🧠","tokens_out":8453,"duration_ms":75349,"temperature":0.7,"pith_summary":"This paper is trying to establish that parametric tool retrieval — training an LLM to emit a virtual token for the right API — does not have to erase the model's knowledge of the tools it retrieves. The proposed curriculum, TRACE, first memorizes an enterprise tool catalog, then trains the model to produce a reasoning trace that cites governing business rules before committing to a JSON list of tool tokens. The paper argues this dissolves the knowledge-retrieval dissociation: MCQ and QA probing stay near or above the Stage-1 ceiling, retrieval recall reaches ~86% on one domain and ~60% on another under single-beam greedy decoding, and latency drops roughly 200x versus constrained beam search. A sympathetic reader would care because it points to a knowledge-safe path for API routing over thousands of enterprise tools.","feed_headline":"Think-before-retrieve training hits 86% recall, no beam search","feed_subtitle":"Two-stage LLM curriculum preserves tool knowledge and cuts retrieval latency by roughly 200x.","key_machinery":"The load-bearing object is the virtual tool token paired with a reasoning-augmented retrieval objective. Each of the 8,283 APIs is assigned a unique token appended to the model vocabulary; Stage 2 supervision replaces tool names inside the thinking trace with these virtual tokens, so that every token is trained inside the deliberative context that justifies it. A second element is the business-rule catalog: 123 expert-authored rules, each pairing a confusible tool set with a disambiguation directive. Rules feed a synthetic query generator that produces explicit, implicit, and exception-style queries; the model is trained to quote the governing rule by name in its trace. At inference, the mod","core_discovery":"On the paper's own terms, the central discovery is that the collapse of parametric tool knowledge under retrieval training is not inherent to parametric retrieval but a consequence of the training objective. Non-reasoning retrieval fine-tuning maps queries to virtual tool tokens and drives MCQ/QA probing accuracy down to random baselines; TRACE replaces that objective with reasoning-augmented retrieval SFT. Stage 1 seeds tool knowledge through multi-format memorization; Stage 2 trains the model to emit a thinking trace — which explicitly applies expert-written business rules — before outputting the JSON token list. The result is that the same model reaches ~86% R@gen on Domain A and ~60% on","pith_inferences":["Editorial: If the reasoning trace is what preserves knowledge, the same two-stage recipe could generalize to other parametric memory tasks — factual QA, code lookup, entity routing — where retrieval-style fine-tuning currently causes forgetting.","Editorial: The rule-coverage explanation for Domain B suggests a testable extension: automatically mine rules from API change logs, deprecation notices, and version histories, then retrain; if recall closes the gap without human curation, the method scales beyond expert-authored rule sets.","Editorial: The paper's n→r results (adding reasoning after non-reasoning retrieval training) imply that once non-reasoning retrieval has decoupled tokens from descriptions, the damage is largely irreversible; an immediate test is to start all retrieval training with reasoning traces and compare final knowledge retention.","Editorial: The reported ~3% off-vocabulary token rate under free-form decoding suggests a lightweight post-hoc validator that only checks whether generated tokens are in the catalog could make the decoder production-safe without reintroducing beam search."],"forward_implications":["Enterprise copilots can route queries across 8,283 tools with a single greedy decode, answering in about 1.9 seconds instead of about 19 seconds for constrained beam search.","Tool knowledge survives retrieval training: MCQ probing gains +3.2 pp and QA probing gains +9 pp over Stage 1, so downstream post-training that relies on parametric tool knowledge is not foreclosed.","Business-rule grounding is required for retrieval-grade recall: adding rule-targeted data lifts Domain A recall from ~56% to ~86%, and traces that cite a rule reach 94.6% recall versus 63.2% for traces that do not.","The recipe transfers across token formats — flat and wrapped-hierarchical both reach ~86% Domain A recall — but bolting reasoning onto a non-reasoning checkpoint cannot fully undo the damage on hierarchical formats.","Domain B's ~60% recall is attributed to incomplete rule coverage rather than a method limit, making denser rule catalogs the concrete path to scale."],"fun_headline_variants":["Think-before-retrieve keeps tool knowledge, hits 86% recall","Reasoning traces stop tool-training knowledge loss","LLM tool retrieval: think first, then get 86% recall","Training LLMs to think preserves tool knowledge, boosts recall","From 27% to 86% recall: thinking before retrieval works"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The synthetic training queries and the 123 expert-authored business rules faithfully represent real production routing constraints, and the teacher-written reasoning traces reflect genuine decision-making rather than post-hoc rationalization.","fun_headline_variants_meta":{"raw":{"variants":["Think-before-retrieve keeps tool knowledge, hits 86% recall","Reasoning traces stop tool-training knowledge loss","LLM tool retrieval: think first, then get 86% recall","Training LLMs to think preserves tool knowledge, boosts recall","From 27% to 86% recall: thinking before retrieval works"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000678,"raw_usage":{"total_tokens":2947,"prompt_tokens":802,"completion_tokens":2145,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":2058}},"tokens_in":546,"tokens_out":2145,"duration_ms":13274,"temperature":1.0,"reasoning_tokens":2058,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T10:22:33.238217+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain Stage 2 on the same query-answer pairs but strip rule citations from every reasoning trace, then compare Domain A recall and the held-out expert multiple-choice accuracy. If recall stays near 86% and probing accuracy stays high, the business-rule mechanism is not doing the work the paper attributes to it; if recall drops sharply, rule citation is load-bearing.","supporting_citations":[],"review_version":1}