{"id":"1105aefb-4cab-4fd2-965b-e83d6f4b90ed","arxiv_id":"2412.00402","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"DroidCall is a GPT-4-turbo-generated dataset that allows small phone models to invoke Android intents from natural language as accurately as, or better than, GPT-4o on a synthetic benchmark.","lead":"This paper introduces DroidCall, a synthetic dataset of 10,000 natural-language instructions paired with Android intent invocations. Fine-tuning small on-device models on the dataset lets them match or beat GPT-4o accuracy on the authors' generated test set.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline comparison rests on unvalidated GPT-4-turbo-generated test labels; the 85% vs 77% gap is only 16 samples, so a small label-error rate can flip the claimed superiority over GPT-4o.","rationale":"The paper has real merits: the data-generation pipeline is reusable, the demo is concrete, and the within-paper comparison against the general xlam-function-calling dataset shows that task-specific synthetic data helps. Those results do not depend on the GPT-4o comparison. What is load-bearing is the abstract's claim that small fine-tuned models approach or surpass GPT-4o for Android intent invocation. For that claim, the 200-sample test set is the only evidence, and it is generated by the same GPT-4-turbo pipeline used to create the training data. The only filters are format validation and ROUGE-based deduplication, so the labels are unverified and may contain the generator's systematic biases. Fine-tuned models are directly trained to mimic that generator's answer style, while GPT-4o is not, so exact-match scoring can systematically favor the fine-tuned models even when both produce valid invocations. The reported gap is 16 samples out of 200, within the range of plausible label noise or model-selection bias from choosing the best checkpoint on the test set. I therefore agree with the reader's conditional verdict: the dataset and code should be released, and the headline comparison should be re-run on a human-validated, independently collected test set before the claim is accepted. If that re-run confirms the ranking, the claim is solid; if not, the paper should be reframed as a dataset-and-training contribution rather than a GPT-4o-surpassing result.","tokens_in":17810,"tokens_out":9992,"duration_ms":98279,"concrete_test":"Construct an independent evaluation set: collect 100-200 natural-language Android task instructions from real users or crowdsourcing, have at least two Android developers independently annotate the correct intent function calls with arguments, and adjudicate disagreements. Then evaluate the fine-tuned Gemma2-2B and Qwen models plus GPT-4o on this set using the same prompt formats and oracle-retriever protocol as in Section 4. If a fine-tuned small model still outperforms GPT-4o by a margin consistent with Table 2, the central claim survives; if not, the Table 2 gap is an artifact of the GPT-4-turbo-generated test distribution or labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that fine-tuned SLMs approach or even surpass GPT-4o (Abstract, Section 5) rests on Table 2, where Gemma2-2B fine-tuned on DroidCall scores 85% Accuracy versus GPT-4o at 77% on the 200-sample test split. That comparison presupposes the test labels are correct ground truth. The test split is produced by the same GPT-4-turbo pipeline as the training data (Section 3.2), and Section 3.2.3 applies only a JSON format filter and a ROUGE-based SimilarityFilter; no human or programmatic check validates that the generated answer is the correct Android intent invocation or that arguments are actually inferable from the query. The generation prompt instructs the model not to fabricate parameters, but nothing enforces this. Because Accuracy requires exact agreement in function name and argument values, a fine-tuned model can memorize the generator's query-to-label conventions while GPT-4o, not fine-tuned on DroidCall, may produce a semantically equivalent call that is marked wrong. The margin is 16 samples out of 200, so even a modest label-error rate or a few ambiguous queries can flip the ranking. Sections 3.3 and 4.1 also select the best checkpoint by testing on this same test set, making the reported numbers optimistic. The fake retriever in Section 4 means the evaluation measures function completion under oracle retrieval, not end-to-end intent invocation. The dataset may still be valuable, but the headline superiority claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DroidCall, a 10k-example synthetic dataset for fine-tuning small language models to invoke Android intents via predefined functions. The generation pipeline follows the self-instruct paradigm with GPT-4-turbo, starting from seed examples sampled from an existing function-calling dataset, and applies JSON format and ROUGE-based deduplication filters. The authors fine-tune several on-device SLMs with LoRA and compare their accuracy and soft accuracy against GPT-4o and GPT-4o-mini on a 200-sample test split, reporting that Gemma2-2B-it reaches 85% accuracy versus GPT-4o's 77%. They also present an end-to-end Android demo with a GTE-based retriever and the mllm inference engine.","tokens_in":18098,"tokens_out":4120,"duration_ms":40779,"significance":"If the reported results hold, DroidCall would be a useful first open dataset for Android intent invocation, with a reusable and customizable data generation pipeline and a practical on-device deployment demonstration. The paper's strengths include open-sourcing the data generation, fine-tuning, and evaluation code, and making the dataset and demo publicly available. The main contribution is the dataset itself rather than any new learning method. However, the central comparative claim that fine-tuned SLMs approach or surpass GPT-4o is currently not established: the evaluation is partly circular, the test-set size is small, checkpoint selection uses the test split, and the evaluation uses an oracle retriever. The dataset may still be valuable for future work, but the headline claim requires additional validation.","major_comments":[{"comment":"The test split of DroidCall is generated by GPT-4-turbo under the same self-instruct pipeline as the training split, and Section 3.2.3 describes only JsonExtractor, FormatFilter, and SimilarityFilter—there is no human or independent programmatic verification that the generated function call and argument values are correct or inferable from the query. Because fine-tuned SLMs are trained to mimic the generator's query-to-label conventions, the 85% accuracy reported in Table 2 is partly circular: the models are scored on the same distribution they were trained to reproduce, while GPT-4o is not fine-tuned on DroidCall. This does not by itself invalidate the dataset, but it does mean the abstract's claim that SLMs 'approach or even surpass' GPT-4o is not yet supported. A human-verified subset or an independent benchmark of real user intents is needed.","section":"Section 3.2.3 and Section 4 (Metrics)"},{"comment":"The paper states in Section 3.3 that the authors 'pick the best checkpoint' after training for 24 epochs, and Section 4.1 describes selecting checkpoints by testing accuracy on the DroidCall test set. This means the reported test accuracies in Table 2 are optimistic, since the test set is used for model selection rather than being reserved for a single final evaluation. The choice of prompt format in Section 4.1 is also made on the same test split. The authors should report validation-set-based selection, or clearly state and justify the use of the test set for checkpoint selection, and ideally re-evaluate the final selected models on a held-out portion.","section":"Section 3.3 and Section 4.1"},{"comment":"The headline difference between Gemma2-2B-it (85%) and GPT-4o (77%) corresponds to 16 samples out of 200, and no confidence intervals, standard errors, or significance tests are reported. Given that Acc requires exact match to the GPT-4-turbo-generated labels, a small number of incorrect or ambiguous test labels could reverse the reported ranking. The authors should provide bootstrap or repeated-evaluation intervals and, if possible, human adjudication of the test labels to establish that the 8-point gap is statistically meaningful.","section":"Table 2 and Section 4 (Metrics)"},{"comment":"The evaluation uses a 'fake retriever that always retrieves the ground-truth functions,' so Acc and Soft Acc measure function-call generation under oracle retrieval, not the full Android intent invocation pipeline. The abstract's phrase 'accurate Android intent invocation' is therefore stronger than what Table 2 demonstrates. The authors should either integrate the GTE-based retriever from Section 3.4 into the reported evaluation, or explicitly qualify the conclusions as applying to the function-calling component only.","section":"Section 4 (Evaluation setup) and Abstract"}],"minor_comments":[{"comment":"There is a typo in the last sentence: 'on Andrid devices' should be 'on Android devices'.","section":"Section 3.4"},{"comment":"Several typos appear in the prompt templates: 'respone' should be 'response', 'infomation' should be 'information', and 'provider' should be 'provide'.","section":"Appendix A"},{"comment":"Zero-shot results in Figure 6(b) use the code prompt format, while Table 2 reports zero-shot and few-shot results using the json prompt format; this inconsistency in evaluation conditions should be stated clearly in the captions or the text so readers can compare the two sets of numbers correctly.","section":"Figure 6 and Table 2"},{"comment":"The semantic-similarity threshold for non-exact parameters is set to 0.75, and the choice of threshold is not further justified or ablated; a brief justification or sensitivity analysis would strengthen the metric definition.","section":"Section 4 (Metrics)"},{"comment":"The text says 'the final outcome indicates that the json format performed slightly better' but then chooses code_short for subsequent experiments based mainly on token count; the sentence should more clearly separate accuracy considerations from efficiency considerations.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a useful dataset and an open-source pipeline, but the evaluation section currently rests on a circular test set and on test-set-based model selection. I believe these issues can be fixed within the manuscript's scope by adding a human-verified or independently sourced evaluation set, reporting error bars, and clarifying the component-level scope of the evaluation. The work is likely acceptable after such revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: DroidCall is a genuinely useful new dataset and the core engineering story holds up, but the \"surpasses GPT-4o\" headline is not yet supported. The 85 vs 77 gap is 16 samples out of 200, with test labels from the same GPT-4-turbo pipeline that made the training data.\n\nWhat is new and good: this is the first dataset specifically for Android intent invocation, 10k training samples over 24 common functions, built with a self-instruct pipeline seeded from xlam-function-calling-60k. The pipeline is reusable and they open-source it. They fine-tune a wide spread of SLMs, show clear gains over zero- and few-shot baselines, and demonstrate that domain-specific data beats a generic function-calling dataset for this task. That is a concrete, citable contribution for the mobile-agent community.\n\nWhere it is soft: the test set is synthetic, generated by GPT-4-turbo under the same prompts as training, filtered only for JSON format and ROUGE redundancy. No human check verifies that the labels are correct or that arguments are actually inferable from the query. The appendix prompt says \"do not fabricate\" but nothing enforces it. Checkpoint selection uses the test set (train 24 epochs, pick the best checkpoint), and no error bars or variance are reported. The fake retriever is honestly disclosed, but it means the metric measures function completion, not end-to-end intent invocation. The soft-accuracy semantic threshold of 0.75 is a free parameter.\n\nNone of this kills the dataset's value. The training data is useful precisely because a strong model generated it, and the fine-tuning improvements are large and consistent. But the abstract's \"approach or even surpass GPT-4o\" should be softened or re-evaluated on a human-labeled benchmark before it is taken at face value.\n\nWho this is for: people building on-device mobile agents or studying synthetic data for function calling. It deserves a serious referee, but the review should ask for external validation of the test set, error bars, and a more cautious headline. I would cite the dataset, not the claim.","headline":"Useful new dataset for Android intent invocation, but the claimed edge over GPT-4o rests on a 200-sample synthetic test set labeled by the same generator as the training data.","tokens_in":18602,"tokens_out":2205,"would_cite":true,"duration_ms":20879,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 10,000-sample synthetic dataset derived from Android's implicit intents lets small on-device language models match or exceed a large cloud model at translating natural-language requests into function calls.","keywords":["Android intent invocation","function calling","small language models","synthetic dataset","on-device agent","LoRA fine-tuning","GPT-4o baseline","mobile computing"],"falsifier":"Collect a set of real user instructions for the same 24 functions from a live user study, label them by hand, and run the DroidCall-fine-tuned models and GPT-4o on that set under the same fake-retriever condition; if the small models no longer match or beat GPT-4o, or if a noticeable fraction of the synthetic test labels are shown to be wrong, the claimed superiority does not transfer.","tokens_in":17608,"feed_emoji":"📱","tokens_out":5730,"duration_ms":44772,"temperature":0.7,"pith_summary":"The paper claims that a 10,000-sample synthetic dataset, DroidCall, can teach small on-device language models to translate natural-language instructions into Android intent invocations (the mechanism that triggers app actions) more accurately than a large cloud model. The authors fine-tune models such as Qwen2.5-3B and Gemma2-2B with the dataset and report that they reach or surpass GPT-4o's accuracy on a 200-sample test set, while using much shorter prompts and running locally. The claim matters because on-device assistance avoids sending user data to the cloud and removes network latency, and because the dataset pipeline requires no hand-written seed data. If correct, this makes private, responsive mobile agents practical with models small enough to fit on a phone.","feed_headline":"Tiny models beat GPT-4o at Android intent calls with 10k samples","feed_subtitle":"Fine-tuned Gemma2-2B hits 85% accuracy versus GPT-4o's 77% on Android intent invocation, running locally.","key_machinery":"The load-bearing mechanism is the pairing of a function-call representation with an automated synthetic-data generation loop. The authors predefine 24 functions with typed arguments and docstrings, sample examples from a general function-calling dataset to seed the generator, then repeatedly prompt a strong LLM to produce new query–call pairs with varied wording, filtering outputs by JSON validity and ROUGE similarity. Fine-tuning uses the model's own chat template with the function descriptions in the system prompt and the call in the assistant turn, which avoids format gaps; a 'code_short' prompt with 645 average tokens is chosen for efficiency. Evaluation uses exact-match accuracy and a soft accuracy that applies semantic similarity for free-text parameters, with a fake retriever that always returns the ground-truth functions to isolate intent-call accuracy from retrieval.","core_discovery":"The central discovery is that Android intent invocation can be reformulated as function calling, and that a compact domain-specific dataset built by an automated pipeline is enough to unlock this capability in small models. The authors define 24 functions wrapping common Android implicit intents (alarms, timers, contacts, messages, maps, camera, settings), generate user queries and matching function-call annotations with the help of a larger language model, filter and deduplicate them, and fine-tune small models with low-rank adaptation. On their 200-example test set, Gemma2-2B fine-tuned on DroidCall reaches 85% exact-match accuracy versus 77% for GPT-4o, and several other sub-4B models land in the low-to-mid 80s, compared to 17.5–67% for the same models zero-shot. The authors also provide an end-to-end Android app where the fine-tuned model runs on-device, demonstrating tasks like adding a calendar event.","pith_inferences":["A natural next test is to swap the fake retriever for the embedding-based retriever used in the demo and measure end-to-end accuracy; the paper's headline numbers would likely drop because retrieval errors add to call errors.","The same pipeline could be pointed at other on-device APIs, such as iOS shortcuts, automotive infotainment, or smart-home actions, by changing only the function definitions.","Because the test set is synthetic and drawn from the same generator as the training set, the reported accuracy probably overestimates performance on out-of-distribution user phrasing; a human-authored test set would give a fairer comparison.","The dataset construction itself is a small-scale instance of a broader pattern: using a large model to generate training data for a narrow, well-specified task, then distilling that skill into a small model."],"forward_implications":["A general-purpose phone assistant could perform common tasks (set alarm, send message, open camera, search maps) with a sub-2B model running entirely on the device.","Developers could regenerate the dataset for their own predefined function sets without writing seed examples, since the pipeline only requires function signatures and docstrings.","Intent invocation would become faster and cheaper than UI-gesture automation, since the model directly calls the function rather than simulating taps and swipes.","Prompt length for intent calling drops from roughly 1,367 tokens for GPT-4o to about 645 tokens for fine-tuned small models, reducing latency and memory.","Because functions shield the model from version-specific intent details, the same trained model could work across Android versions by reimplementing the function bodies.","If the approach transfers, other structured mobile APIs beyond Android intents could be controlled with the same small-model recipe."],"supporting_citations":[{"why":"Supplies the self-instruct paradigm that the data generation loop is built on, enabling GPT-4-turbo to produce new query–call pairs from existing examples.","marker":"Wang et al., 2022"},{"why":"Provides xlam-function-calling-60k, the external dataset sampled for seed data and later used as the general-function-calling baseline in the effectiveness comparison.","marker":"Liu et al., 2024d"},{"why":"Describes LoRA, the low-rank fine-tuning method used to adapt all small models to DroidCall.","marker":"Hu et al., 2022"},{"why":"Defines ROUGE, which the SimilarityFilter uses to discard generated queries that are too similar to existing ones, controlling diversity.","marker":"Lin, 2004"},{"why":"Provides RoBERTa, used in the soft-accuracy metric to judge semantic consistency of free-text parameters such as titles and subjects.","marker":"Liu et al., 2019"},{"why":"Defines the Llama 3.2 family, several of whose small models are evaluated and fine-tuned in the experiments.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["Small models beat GPT-4o on Android intents with DroidCall","Tiny models outperform GPT-4o for Android intent calls","DroidCall dataset: small models outdo GPT-4o on Android","Fine-tuned 2B models surpass GPT-4o on Android intents","On-device small models top GPT-4o in intent invocation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies assume that the GPT-4-turbo-generated labels in the test set are correct ground truth and that those synthetic queries resemble what real users will actually say; if either assumption fails, the gap over GPT-4o may shrink or disappear.","fun_headline_variants_meta":{"raw":{"variants":["Small models beat GPT-4o on Android intents with DroidCall","Tiny models outperform GPT-4o for Android intent calls","DroidCall dataset: small models outdo GPT-4o on Android","Fine-tuned 2B models surpass GPT-4o on Android intents","On-device small models top GPT-4o in intent invocation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1396,"prompt_tokens":900,"completion_tokens":496,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":399}},"tokens_in":516,"tokens_out":496,"duration_ms":4202,"temperature":1.0,"reasoning_tokens":399,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:25:06.438886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a set of real user instructions for the same 24 functions from a live user study, label them by hand, and run the DroidCall-fine-tuned models and GPT-4o on that set under the same fake-retriever condition; if the small models no longer match or beat GPT-4o, or if a noticeable fraction of the synthetic test labels are shown to be wrong, the claimed superiority does not transfer.","supporting_citations":[],"review_version":1}