{"id":"699df783-1bd6-4d40-adec-0f3130a54b90","arxiv_id":"2608.03468","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ToolLIFT generalizes tool planning to unseen tool sets by lifting trajectories to function-level workflow graphs and separating workflow planning from tool selection.","lead":"ToolLIFT turns histories of tool use into function-level workflow graphs, letting AI agents plan with tools they have never seen before. The paper reports consistent gains over five baselines on two in-distribution and three out-of-distribution tool-planning benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cold-start cluster assignment (Eq. 4) is untested for OOD tools; the FWG transfer claim depends on unseen tools landing in the correct functional cluster, with no quantitative validation of assignment accuracy.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the cold-start transition inheritance of Eq. (4). The central claim is that function-level FWGs transfer experience to unseen tool sets; that transfer is mediated entirely by assigning each OOD tool to a training cluster via nearest-centroid in UMAP space. If that assignment is wrong, the FWG transitions supplied to the planner are noise. The paper gives only qualitative cluster examples (Appendix C.2) for three selected training clusters and does not report any OOD assignment accuracy, cluster purity, or stability analysis. Because the ablation in Table 3 shows FWG guidance contributes 8.38 Acc points on DailyLifeAPIs, the gains attributed to transfer are directly tied to the correctness of this assignment. I would test it directly. I also noticed an anomaly in Table 1: Seal-Tools Acc for ToolLIFT is identical (56.63) for both Qwen and Llama, which is implausible and suggests a typo or evaluation artifact; if real, it affects the 'three OOD benchmarks' claim. But the more fundamental concern remains the untested cold-start mapping. The verdict stays conditional, and our stress test does not move it: the same test would settle whether the concern actually lands.","tokens_in":18491,"tokens_out":7128,"duration_ms":77190,"concrete_test":"Manually label a random sample of N=200 OOD tools (e.g., 50 each from DailyLifeAPIs, Seal-Tools, ToolAlpaca) with their intended abstract function using the paper's own prompt rubric. For each tool, apply the exact pipeline of Section 4.1 (LLM decomposition, BGE-M3 embedding, UMAP transform, nearest-centroid assignment) and compute agreement with the manual labels mapped to the L=30 training clusters. Report assignment accuracy and a confusion matrix; if accuracy is below ~70%, the FWG inherits noise for most unseen tools and OOD gains cannot be attributed to correct cold-start transfer. Also rerun UMAP/K-means with 10 different random seeds and measure the fraction of OOD tools whose cluster assignment changes; high instability would cast direct doubt on Eq. (4).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central transfer mechanism is the cold-start assignment of Eq. (4): an unseen tool t_new is mapped to the nearest training-cluster centroid in the UMAP/K-means space of LLM-generated functional descriptions, and then inherits that cluster's FWG transitions. The OOD gains are attributed to these inherited transitions (ablation in Table 3 shows removing FWG drops DailyLifeAPIs Acc by 8.38 points), yet the paper provides no quantitative evidence that OOD tools are assigned to the intended functional clusters. Section C.2 gives only three hand-picked clusters from the training data; no OOD cluster-purity or assignment-accuracy numbers are reported. If an OOD tool with function 'send email' is embedded far from the training 'Transmission' centroid (e.g., because BGE-M3 embeddings of LLM-written functional descriptions do not generalize across domains), the FWG transitions it inherits are irrelevant or misleading, and the purported 'strong generalization to unseen tool sets' is an artifact of the planner ignoring or overriding bad graph guidance. This assumption is load-bearing because the method's claimed advantage over tool-level graphs rests on function-level transfer. An additional red flag: in Table 1, Seal-Tools Acc for ToolLIFT is identical (56.63) for both Qwen and Llama, which is implausible and suggests a typo or shared evaluation artifact; if real, it would weaken the 'three OOD benchmarks' evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ToolLIFT, a framework for tool-use planning that abstracts tool-specific trajectories into a function-level workflow graph (FWG). Tool schemas are decomposed into functional and domain descriptions by an LLM, embedded with BGE-M3, reduced by UMAP, and clustered by K-means to define a tool-to-function mapping phi (Eq. 4). Historical trajectories are lifted to function sequences, transition counts are aggregated into an FWG, and unseen tools inherit transitions through nearest-centroid cluster assignment. Planning is decoupled into FWG-guided workflow generation and workflow-constrained tool selection, with GRPO training and source-gated, skill-specific rewards for argument-level dataflow. Experiments on two ID and three OOD benchmarks compare against five baselines with two LLM backbones, plus ablations, sensitivity analyses, and case studies.","tokens_in":18909,"tokens_out":5010,"duration_ms":63638,"significance":"The central idea is timely and plausible: function-level abstractions may transfer collaboration structure across disjoint tool sets, and the paper gives a concrete mechanism with clearly stated equations. If validated, the FWG would be a useful contribution to tool planning. The authors also provide useful ablations and qualitative case studies. However, the current evidence does not yet support the abstract's strong claim of 'consistently outperforming' baselines and 'strong generalization to unseen tool sets.' The evaluation is weakened by filtered test sets, single-run results without error bars, test-set hyperparameter selection, and a lack of quantitative validation of the cold-start cluster assignment that underlies the entire transfer mechanism. The manuscript is within scope and the issues are addressable, but they are load-bearing for the main claim.","major_comments":[{"comment":"","section":"Sec. 4.1, Eq. (4); Appendix C.2"},{"comment":"","section":"Sec. B.1, Table 5"},{"comment":"","section":"Sec. B.6, Table 1"},{"comment":"","section":"Sec. C.1, Tables 1 and 3"},{"comment":"","section":"Sec. A.1, Eqs. (7) and (15); Table 1"}],"minor_comments":[{"comment":"","section":"Sec. 4.2, Eq. (7)"},{"comment":"","section":"Sec. 5.3, Fig. 5"},{"comment":"","section":"Sec. B.2, Eq. (23)"},{"comment":"","section":"Appendix C.4"},{"comment":"","section":"Sec. 4.4"}],"recommendation":"major_revision","confidential_remarks":"The core idea is worth publishing if the evaluation gaps are addressed. The most important missing piece is quantitative validation of the cold-start cluster assignment for OOD tools, followed by multi-seed results and clarification of the filtered benchmarks and epsilon_pert selection. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is good. Lifting concrete tool trajectories to a function-level workflow graph directly targets the known generalization bottleneck of tool-level graphs, and I don't know prior work that does exactly this. The decomposition of tool schemas into functional and domain descriptions, clustering on functional embeddings, and building a transition graph at function level is a clean, sensible abstraction. The decoupled workflow planning and tool selection, plus the RL rewards that explicitly supervise argument sources, are reasonable components that hang together well. The empirical sweep is broad: two ID and three OOD benchmarks, two backbones, and ablations that isolate FWG guidance, the planner, the reward, and perturbation. The Table 3 ablation showing a large OOD drop when FWG guidance is removed gives real support to the claim that the abstraction carries transfer value.\n\nThe main soft spot is the load-bearing assumption in Eq. (4). Unseen tools are assigned to the nearest cluster centroid and inherit that cluster's transitions, but the paper never directly validates whether OOD tools land in the intended functional clusters. The qualitative clusters in C.2 are from training tools only. If an OOD tool is placed in the wrong cluster, the inherited transitions are noise, and the claimed generalization is really a symptom of the planner overriding bad graph guidance. This needs direct evidence: assignment accuracy on OOD tools, or at least per-cluster error analysis. The overall OOD gains are indirect evidence, but not enough on their own.\n\nThe evaluation also has several rigor gaps. Results come from a single training run with no error bars (B.6). The test sets are filtered (B.1), and the filtering criteria could plausibly favor the method; at minimum the effect should be quantified. The perturbation probability epsilon_pert is tuned on DailyLifeAPIs and Multimedia, which both appear in the main table, so that is effectively tuning on the test set. And the identical Seal-Tools accuracy of 56.63 for both Qwen and Llama is suspicious; I'd want that checked before relying on it.\n\nThe self-referential Stage 1 reward is less of a problem in my view, because the final evaluation uses ground-truth tool calls. But the planner is trained to match a clustering that is itself unsupervised, so the soundness of the whole pipeline still hinges on cluster quality.\n\nWho should read this: anyone working on LLM agent tool planning, experience reuse, or graph-based planning. It deserves a serious referee; the idea is solid, and the weaknesses are fixable with additional experiments rather than being fatal. I'd send it to review with a strong request for cluster-assignment validation, multi-seed runs, and a resolution of the Seal-Tools number.","headline":"Function-level workflow graphs are a genuinely useful abstraction for tool planning, and the paper's OOD gains are plausible, but the central transfer mechanism is never directly validated and the evaluation has enough single-run/filtered-set issues that the margins should be treated as provisional.","tokens_in":19346,"tokens_out":2415,"would_cite":true,"duration_ms":28887,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that tool-use experience transfers between tool sets when stored as function-level workflow graphs rather than tool-level graphs, and reports consistent gains on out-of-distribution benchmarks.","keywords":["tool planning","LLM agents","function-level workflow graph","out-of-distribution generalization","trajectory lifting","reinforcement learning","source tracing","tool clustering"],"falsifier":"Take an unseen tool and run its schema through the pipeline twice, once with the original functional description and once with a synonym-heavy paraphrase that emphasizes domain details; if the two near-identical tools land in different clusters, the nearest-centroid assignment is not stable. A stronger version: rebuild the FWG with random tool-to-cluster assignments of the same size and re-run the OOD benchmarks; if accuracy stays close to the full model, the function abstraction is not what carries the generalization.","tokens_in":18433,"feed_emoji":"🛠️","tokens_out":9531,"duration_ms":105236,"temperature":0.7,"pith_summary":"Tool-use experience is usually stored as graphs whose nodes are concrete tools, so a plan learned for one tool set is useless for another. ToolLIFT's central claim is that the reusable part of a trajectory is its function-level workflow: clustering tools by what they do rather than what domain they serve lets historical transitions be aggregated into a Function-Level Workflow Graph (FWG). The framework then plans in two steps—first choose the function-level workflow using the FWG as soft guidance, then instantiate each function with a concrete tool—and uses reinforcement learning to make each argument explicitly a direct value or a reference to a prior tool output. On two in-distribution and three out-of-distribution benchmarks, ToolLIFT outperforms existing planners, with the largest margins on tool sets unseen during training (up to 4.90 accuracy points over the strongest baseline). If the claim holds, an agent can be extended to new tools by reading their schemas and inheriting workflow experience from functionally equivalent tools.","feed_headline":"Lifting tool traces into function graphs enables cross-tool planning","feed_subtitle":"Plans workflows by function first, then chooses concrete tools, and beats baselines on three unseen-tool benchmarks.","key_machinery":"The central object is the Function-Level Workflow Graph (FWG): a directed, weighted graph whose nodes are functional clusters of tools and whose edge weights are the probabilities of one function following another in historical trajectories. It is built by trajectory lifting—mapping each concrete tool to a functional cluster via LLM decomposition, embedding, UMAP projection, and K-means—and it is what makes experience portable. The FWG does three jobs: it shares collaboration evidence across all tools in a cluster, it supplies soft transition guidance for global workflow planning, and, through nearest-centroid assignment, it lets an unseen tool inherit a function's transitions (cold-start in","core_discovery":"The paper claims that analogous tasks performed with different concrete tools share a common function-level workflow structure, and that this structure is the right unit for transferring tool-use experience. ToolLIFT operationalizes the claim by decomposing each tool schema into a functional description and a domain description, embedding only the functional part, projecting the embeddings, and clustering tools into functional roles. Historical tool trajectories are then lifted to sequences of functional roles, and adjacent-role pairs are counted into transition probabilities that form the FWG. An unseen tool is assigned to the role of the nearest cluster centroid and inherits that role's tr","pith_inferences":["The FWG could be used as a cheap pre-filter for tool retrieval: function-level transitions can narrow the candidate tool set before an LLM is asked to plan, which would matter for agents with thousands of APIs. This is an extension, not a claim in the paper.","Cluster coherence is a measurable predictor of transferability: one could check whether the same functional roles co-cluster across different trajectory corpora, and use that overlap to forecast OOD gains before running a planner.","The single-source argument assumption is a natural boundary condition; a testable extension is to let one argument combine several prior outputs and extend the source-gated reward accordingly.","A practical on-boarding protocol follows: when adding a tool, verify its nearest-centroid assignment against a human label of its function before trusting the inherited workflow, since the whole transfer mechanism rests on that assignment."],"forward_implications":["Trajectory libraries collected for one tool set can guide planning with a completely disjoint tool set, requiring only schema descriptions for the new tools.","Planning the full function-level workflow before choosing tools prevents locally plausible single-tool choices from breaking the overall plan, as the paper's case study shows.","Explicitly marking each argument as direct or as a reference to a prior output reduces source errors compared with a reward that only exact-matches values.","Rarely used tools benefit most from function-level sharing, because transitions learned from abundant functionally equivalent tools fill in for sparse tool-level evidence.","Because gains appear with two different open LLM backbones, the function-level abstraction itself, not a specific model, is the driver."],"supporting_citations":[{"why":"Supplies BGE-M3 embeddings used to encode the LLM-decomposed functional descriptions of tools.","marker":"Chen et al. 2024"},{"why":"Supplies UMAP projection of functional embeddings before clustering.","marker":"Healy and McInnes 2024"},{"why":"Supplies K-means clustering that defines the functional roles.","marker":"MacQueen 1967"},{"why":"Supplies the silhouette coefficient used to select the number of functional clusters.","marker":"Rousseeuw 1987"},{"why":"Supplies GRPO, the reinforcement-learning optimizer used in both training stages.","marker":"Shao et al. 2024"},{"why":"ToolRL; source of the adapted tool-matching reward and the strongest baseline to beat.","marker":"Qian et al. 2025"},{"why":"TaskBench; provides the training trajectories and the in-distribution plus DailyLifeAPIs evaluation sets.","marker":"Shen et al. 2024"},{"why":"ToolAlpaca; provides an out-of-distribution benchmark with simulated APIs.","marker":"Tang et al. 2023"},{"why":"Seal-Tools; provides a hard multi-tool OOD benchmark including nested calls.","marker":"Wu et al. 2024"},{"why":"GTool; graph-based planning baseline and source of the l-F1 dependency-link metric.","marker":"Chen et al. 2026"}],"fun_headline_variants":["Plan by function, not by tool: ToolLIFT generalizes","Function-first workflow graphs boost tool planning to unseen tools","ToolLIFT: Lift traces to function graphs, plan across any tool set","From tools to roles: A graph that makes planning tool-agnostic","Generalize tool use by abstracting trajectories into function workflows"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"An unseen tool is assigned to whichever functional group it is closest to and inherits that group's workflow experience; if the grouping is wrong, the experience it inherits describes a different job, and the claimed transfer fails.","fun_headline_variants_meta":{"raw":{"variants":["Plan by function, not by tool: ToolLIFT generalizes","Function-first workflow graphs boost tool planning to unseen tools","ToolLIFT: Lift traces to function graphs, plan across any tool set","From tools to roles: A graph that makes planning tool-agnostic","Generalize tool use by abstracting trajectories into function workflows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1610,"prompt_tokens":748,"completion_tokens":862,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":773}},"tokens_in":492,"tokens_out":862,"duration_ms":9188,"temperature":1.0,"reasoning_tokens":773,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T18:30:48.433302+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an unseen tool and run its schema through the pipeline twice, once with the original functional description and once with a synonym-heavy paraphrase that emphasizes domain details; if the two near-identical tools land in different clusters, the nearest-centroid assignment is not stable. A stronger version: rebuild the FWG with random tool-to-cluster assignments of the same size and re-run the OOD benchmarks; if accuracy stays close to the full model, the function abstraction is not what carries the generalization.","supporting_citations":[{"cited_title":"2024 , eprint =","cited_arxiv_id":null,"evidence_quote":"Supplies BGE-M3 embeddings used to encode the LLM-decomposed functional descriptions of tools."}],"review_version":1}