{"id":"97780d9a-f9a3-4a7a-9d42-0518586d4755","arxiv_id":"2412.15660","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A LoRA fine-tuned 7B model trained on AI-synthesized enterprise HR API data beat GPT-4 and GPT-4o on the authors' private benchmark.","lead":"This paper describes a step-by-step pipeline that synthesizes training data, fine-tunes a 7B function-calling model with LoRA, and evaluates it for enterprise API use. The resulting model outperformed GPT-4 and GPT-4o on the authors' private digital-HR test sets, though one test set overlaps with the training data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DHR_test_A is built from the same 207 seed questions used to generate the augmented training set, so the headline 'surpassing GPT-4' may reflect memorization rather than generalization; a fresh held-out test is needed to support the claim.","rationale":"The reader's weakest assumption identifies the exact flaw: the main test set is derived from the training seeds. My reading of Section 4.2.1 and Section 4.2.3 confirms the overlap. This is not a mere stylistic issue; it directly invalidates the evidence for the headline claim. The paper also provides no code, data, or model, and the tables are poorly formatted, but these are secondary. The single most load-bearing issue is the test-set leakage, and the proposed held-out test would resolve it. Therefore, the reader's REJECT verdict stands.","tokens_in":17897,"tokens_out":8464,"duration_ms":58148,"concrete_test":"Create a truly held-out test set: have a human annotator (or a separate generation protocol) write at least 200 new questions for the same 14 workflows, using entity names, departments, and years that never appear in the 207 seed questions or their augmentations. Run DHR_train_3_ft, GPT-4, and GPT-4o on this set with the same AST evaluation and compare tool-selection accuracy. If DHR_train_3_ft's accuracy drops substantially (e.g., >10 points) relative to its DHR_test_A score while GPT-4's score is stable, the original claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DHR_train_3_ft surpasses GPT-4 and GPT-4o in function-calling accuracy rests on two small custom test sets with a known overlap problem. Section 4.2.1 states that the 207 manually annotated seed questions were used to generate 1,035 augmented training instances (5 per seed). Section 4.2.3 then selects exactly those 207 seeds as DHR_test_A. Although the exact seed text is not in the training set, each seed has been paraphrased into five training variants through replacement, rewriting, simplification, and error introduction (Table 1, Appendix B). The test queries therefore share entities, tool-selection patterns, and parameter values with training queries, and the model has been explicitly trained to output the correct function call for those seed intents. The near-perfect scores (e.g., 95.6% tool selection vs. GPT-4's 88.1%) and the dramatic jump from the 28.1% base-model score are consistent with near-duplicate evaluation. DHR_test_B consists of only 135 questions and its construction is not described; if generated with the same GPT-4 templates used for training-data synthesis, it inherits the same distributional bias. No public benchmark is used as a guard. Thus, the evaluation does not establish that the model generalizes to unseen queries, and the superiority claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a training pipeline for enterprise-scenario LLM function-calling, comprising scenario-specific data synthesis and augmentation, LoRA fine-tuning, and AST-based evaluation. Using a digital HR intelligent-agent scenario with 14 workflows, the authors generate 1,260 AI-augmented and 1,035 manually augmented training samples, fine-tune Qwen2.5-Coder-7B-Instruct with LoRA on four 24GB GPUs, and report that the resulting model surpasses GPT-4 and GPT-4o in structural completeness, tool-selection accuracy, and parameter-filling accuracy on their test sets. The paper also presents ablations on base models, training data composition, cutoff length, tool description length, and LoRA adapter merging.","tokens_in":18139,"tokens_out":4609,"duration_ms":34459,"significance":"If the empirical claims were sound, the pipeline would be a valuable practical contribution: it offers a low-resource recipe for adapting small open-weight models to specialized enterprise function-calling scenarios, and the confusion-matrix-based tool-selection analysis is a useful diagnostic. The paper also provides a transparent description of data synthesis and augmentation templates, and the LoRA-merging experiments address the practical issue of data life-cycle management. However, the central claim of surpassing GPT-4 rests on test sets that are not demonstrably independent of the training data, and the evaluation methodology lacks the statistical support needed to distinguish genuine generalization from memorization. The pipeline concept is defensible, but the current evidence does not establish the headline result.","major_comments":[{"comment":"DHR_test_A is exactly the 207 manually annotated seed questions used to generate the 1,035 augmented training instances in DHR_train_2 and DHR_train_3. Since the augmentation strategies (replacement, rewriting, simplification, error introduction) produce training queries that share entities, tool-selection patterns, and parameter values with the seed intents, the reported near-perfect accuracy on DHR_test_A largely reflects the model's exposure to augmented variants of those same intents. The abstract's claim of 'surpassing GPT-4 and GPT-4o in accuracy on the test set' is therefore not supported as evidence of generalization to unseen queries.","section":"Section 4.2.1 and 4.2.3"},{"comment":"DHR_test_B consists of only 135 questions and its construction is not described. The paper states that it was designed to 'prevent evaluation bias caused by potential over-fitting to seed data,' but without a clear account of how these questions were created, whether they are disjoint from all training and augmentation data, and whether they were produced by the same GPT-4 prompt templates used for training-data synthesis, DHR_test_B cannot be trusted as an independent held-out set. The reader cannot verify that the claimed superiority over GPT-4 on DHR_test_B is not also subject to distributional overlap with the training data.","section":"Section 4.2.3"},{"comment":"The column headers in Table 3 are ambiguous: it is not clear which columns correspond to DHR_test_A and which to DHR_test_B, nor which metrics are reported for each set. Additionally, all results are from single runs with no error bars, confidence intervals, or multiple seeds. Given the small test-set sizes (207 and 135), differences such as the 95.6% vs 88.1% tool-selection accuracy cannot be assessed for statistical significance. The paper should report variance across repeated fine-tuning runs and clearly separate the metrics for each test set.","section":"Table 3 and Section 4.3"},{"comment":"The claim of surpassing GPT-4 is evaluated only on the two custom test sets described above, with no public function-calling benchmark (e.g., BFCL, API-Bench) used as an external anchor. Furthermore, all ablation studies in Section 5 (base-model comparison, data composition, cutoff length, description length, LoRA merging) are conducted on DHR_test_A, which is contaminated by overlap with the training seeds. Consequently, the ablations do not provide reliable evidence about what drives generalization to genuinely new queries, and the pipeline's adaptability beyond the specific HR scenario remains unverified.","section":"Sections 4.3 and 5"}],"minor_comments":[{"comment":"The table layout should be restructured so that each test set has its own set of clearly labeled columns (e.g., Structure, Tool, Parameter) rather than merging the headings as at present.","section":"Table 3"},{"comment":"The paper says checkpoints were 'typically selected between the 7th and 10th iterations' but does not specify the selection criterion; this should be stated (e.g., validation-set performance) to ensure the reported results are not cherry-picked.","section":"Section 4.2.2"},{"comment":"The confusion-matrix heatmaps and F1-score bar charts would benefit from explicit axis labels and a legend indicating which rows/columns correspond to each workflow; currently the reader cannot interpret the confusion pattern without the caption.","section":"Figures 3 and 4"},{"comment":"The prompt templates are difficult to read because of irregular spacing, inconsistent capitalization, and stray braces; they should be typeset in a clean monospaced format.","section":"Appendix A and B"},{"comment":"The 'No Description' condition is surprising as a training configuration because the model has no information about the tool's purpose; a brief explanation of why this condition was included and what the authors expected would improve readability.","section":"Section 5.4"},{"comment":"Reference [23] appears to use a Wikipedia-style URL; the citation should be completed with proper bibliographic details.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The evaluation contamination in DHR_test_A is severe enough that the current version should not be accepted as-is. However, the flaw is fixable within the scope of a revision: the authors could construct a genuinely held-out test set (e.g., newly manually annotated queries or a public function-calling benchmark), describe the construction of DHR_test_B in detail, report multiple runs with variance, and rerun their ablations on a clean test set. If the authors cannot provide such evidence, the paper should be rejected, because the central 'surpassing GPT-4' claim is not credible on the current data. The engineering components of the pipeline, including the augmentation strategies and the LoRA-merging experiments, have some merit and could form a useful revised contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a practical, well-documented industrial pipeline for specializing a 7B LLM to enterprise function calling, but the central claim of surpassing GPT-4 is not backed by the evaluation as it stands. The main test set, DHR_test_A, is built from the same 207 manually annotated seed questions that were paraphrased into five augmented training instances each (Section 4.2.1 and 4.2.3). So high scores on that set largely measure how well the model memorized the seed intents, not generalization. The paper's own DHR_test_B was meant to fix this, but it has only 135 questions, its construction is not described, and it may share the same GPT-4 template distribution as the training data. Without a clean held-out test or a public benchmark, the superiority claim is unsupported.\n\nWhat the paper does well: the pipeline itself is clearly presented, with complete prompt templates in the appendix, and the ablations are genuinely useful. The comparisons of base models, training-data composition, cutoff length, tool-description length, and LoRA merging provide concrete engineering guidance for practitioners working under tight compute budgets. The authors also acknowledge the overfitting concern with DHR_test_B, showing they are aware of the issue, but they don't resolve it.\n\nSoft spots beyond the circularity: the tables are sloppy — Table 3's column headers don't match the numeric columns, and there are no error bars or multiple runs. No code, data, or model weights are released, so independent verification is impossible. The method itself is not new: GPT-4 data synthesis, Self-Instruct-style augmentation, LoRA fine-tuning, and BFCL-style AST evaluation are all established. The novelty is the application to digital HR and the detailed ablation study, which is enough for an applied report but not for a strong research claim.\n\nBottom line: practitioners wanting to replicate a lightweight enterprise function-calling pipeline will find value here. As a scientific claim of beating GPT-4, it fails. I would not cite it for that claim. I would send it to peer review only with the expectation of major revisions: add a genuinely fresh held-out test set, describe DHR_test_B properly, release data and code, and tone down the headline. The paper deserves a serious referee because the engineering content is real and the flaw is fixable.","headline":"A useful enterprise fine-tuning write-up whose headline 'beats GPT-4' claim rests on a circular test set; worth reading for the pipeline details, not for the comparison.","tokens_in":18706,"tokens_out":2755,"would_cite":false,"duration_ms":22694,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a 7B model, fine-tuned with LoRA on a few thousand synthetic and human-augmented samples, can surpass GPT-4 and GPT-4o in function-calling accuracy within a specific enterprise scenario.","keywords":["function calling","LLM fine-tuning","LoRA","data synthesis","data augmentation","AI agent","tool selection","AST evaluation"],"falsifier":"Have domain experts who never saw the training data write a fresh set of questions across the 14 workflows, run DHR_train_3_ft and GPT-4 on it, and compare overall accuracy; if the 7B model's score falls below GPT-4's, the paper's superiority claim is falsified.","tokens_in":17670,"feed_emoji":"🛠️","tokens_out":19196,"duration_ms":129777,"temperature":0.7,"pith_summary":"The paper presents a training pipeline for adapting a small open-source LLM to call enterprise APIs accurately in a specific business scenario, and reports that a 7B model trained with this pipeline outperforms GPT-4 and GPT-4o in function-calling accuracy on its test sets. The pipeline combines AI-generated and human-augmented training data, LoRA fine-tuning on four 24GB GPUs in about five hours, and AST-based evaluation that separates structural, tool-selection, and parameter errors. If the result holds, it would let small and medium enterprises build domain-specific AI agents without large compute budgets or sending data to the cloud. The empirical evidence is a digital HR agent with 14 workflows, where the best fine-tuned model reaches 97.6% overall accuracy on one test set and 95.6% on another.","feed_headline":"7B model beats GPT-4 at enterprise function calls","feed_subtitle":"A data-synthesis, LoRA fine-tuning, and AST-evaluation pipeline lifts a small model past GPT-4 and GPT-4o on tool selection.","key_machinery":"The mechanism is a data-augmentation and fine-tuning loop built around structured function-calling instructions. The pipeline first turns tool metadata (name, description, parameter names, types, and necessity) into seed questions using a high-performance LLM, then applies four augmentation strategies—replacement, rewriting, simplification, and error introduction—to expand each seed into multiple variants. All variants of a seed share the same function-calling instruction, which acts as the training label. LoRA adapters keep the fine-tuning parameter-efficient, and an AST parser decomposes model outputs into structural, tool-selection, and parameter errors, allowing each failure type to be measured and attributed. The same evaluation setup also supports confusion-matrix analysis of tool selection and multi-LoRA adapter merging for post-deployment updates.","core_discovery":"The central discovery is that scenario-specific function-calling can be instilled into a 7B instruction-tuned model through a closed loop of data synthesis, augmentation, LoRA fine-tuning, and AST-based evaluation. In the digital HR scenario, starting from the 14 workflow tools and 207 human-annotated seed questions, the pipeline generated 1,260 AI-augmented and 1,035 human-augmented training samples; the model trained on the combined set (DHR_train_3_ft) reached 100% structural completeness, 97.6% tool-selection accuracy, and 100% parameter-filling accuracy on the 207-question DHR_test_A set, with 97.6% overall accuracy on that set and 95.6% on the separate 135-question DHR_test_B set. On the same test sets, GPT-4 scored 79.2% and 88.1% overall, and GPT-4o scored 32.9% and 34.1%, so the fine-tuned 7B model is reported to surpass both.","pith_inferences":["Editorial inference: the reported margin over GPT-4 is likely inflated because DHR_test_A is built from the same 207 seed questions that generated the augmented training data; a genuinely held-out test set could shrink the gap.","Editorial inference: the pipeline's 100% parameter-filling accuracy may not survive contact with wider parameter distributions than the synthetic templates produce; real user logs with unseen names, dates, and free-text fields would be the stress test.","Editorial inference: the four augmentation strategies are presented as a bundle, so their individual contributions are untested; ablating each one separately would show which strategy carries the gain.","Editorial inference: the multi-LoRA merging result points toward a practical continual-learning recipe—train a fresh adapter on each quarterly data cycle and merge adapters to update the deployed agent without full retraining."],"forward_implications":["A 7B model specialized to one business domain can outperform much larger general-purpose models on that domain's function-calling task.","The whole training runs in about five hours on four 24GB GPUs, so the recipe is reproducible for small and medium enterprises that want on-premise agents.","Mixing AI-generated and human-annotated seed data helps more than either alone, so a modest amount of expert annotation can be amplified by synthesis.","Shortening tool descriptions during training improves robustness across long, short, and absent descriptions, which is a practical guideline for API documentation.","Merging LoRA adapters trained on different data splits can beat either adapter alone, suggesting a training-free path for incorporating new data cycles after deployment."],"supporting_citations":[{"why":"Provides the LoRA fine-tuning method that makes the parameter-efficient training step possible.","marker":"[11]"},{"why":"Defines the Qwen2.5-Coder-7B-Instruct base model that the pipeline fine-tunes.","marker":"[12]"},{"why":"Supplies the automated data-synthesis and validation approach for function-calling datasets that the pipeline adapts.","marker":"[21]"},{"why":"Provides the AST evaluation framework and error taxonomy used to compute the reported accuracy scores.","marker":"[31]"},{"why":"Supplies the fine-tuning framework and data-cutoff controls used in training and ablation experiments.","marker":"[54]"},{"why":"Establishes the method for generating function-calling instructions from questions and extracting parameters for labels.","marker":"[17]"},{"why":"Provides the memory-optimization method (ZeRO) that makes fine-tuning a 7B model on four 24GB GPUs feasible.","marker":"[60]"},{"why":"Supplies the multi-LoRA merging methods compared in the adapter-merging ablation.","marker":"[62]"}],"fun_headline_variants":["7B model beats GPT-4 and GPT-4o on enterprise tool selection","LoRA-tuned 7B surpasses GPT-4 in enterprise function calls","Synthetic data + LoRA: 7B model tops GPT-4 in tool selection","7B beats GPT-4 in enterprise API selection accuracy","How a 7B model outscored GPT-4 on enterprise tool calling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the fine-tuned model beats GPT-4 rests on the assumption that the test sets measure generalization rather than memorization: DHR_test_A is exactly the 207 manually annotated questions that seeded the augmented training data, and DHR_test_B's 135 questions are not documented as being independent of the training-generation prompts.","fun_headline_variants_meta":{"raw":{"variants":["7B model beats GPT-4 and GPT-4o on enterprise tool selection","LoRA-tuned 7B surpasses GPT-4 in enterprise function calls","Synthetic data + LoRA: 7B model tops GPT-4 in tool selection","7B beats GPT-4 in enterprise API selection accuracy","How a 7B model outscored GPT-4 on enterprise tool calling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000934,"raw_usage":{"total_tokens":4024,"prompt_tokens":998,"completion_tokens":3026,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":2923}},"tokens_in":614,"tokens_out":3026,"duration_ms":18451,"temperature":1.0,"reasoning_tokens":2923,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:12:24.156466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have domain experts who never saw the training data write a fresh set of questions across the 14 workflows, run DHR_train_3_ft and GPT-4 on it, and compare overall accuracy; if the 7B model's score falls below GPT-4's, the paper's superiority claim is falsified.","supporting_citations":[{"cited_title":"https://gorilla.cs.berkeley","cited_arxiv_id":null,"evidence_quote":"Provides the AST evaluation framework and error taxonomy used to compute the reported accuracy scores."},{"cited_title":"Toolformer: Language models can teach themselves to use tools","cited_arxiv_id":null,"evidence_quote":"Establishes the method for generating function-calling instructions from questions and extracting parameters for labels."},{"cited_title":"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models","cited_arxiv_id":null,"evidence_quote":"Provides the memory-optimization method (ZeRO) that makes fine-tuning a 7B model on four 24GB GPUs feasible."},{"cited_title":"PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-LoRA merging methods compared in the adapter-merging ablation."}],"review_version":1}