{"id":"9f45a0ca-3985-4b80-939a-1e569de4bcc5","arxiv_id":"2412.13233","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"A diploma thesis that reviews existing techniques for giving LLMs access to APIs and proposes a largely untested on-device architecture based on semantic matching to predefined macros.","lead":"This thesis proposes a 7-step methodology for connecting large language models to external APIs and sketches an on-device architecture that runs small models locally. A generalist reader might skim it as a broad summary of how AI agents are built today, though the experimental evidence is minimal.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The on-device architecture's core speed/offline benefit depends on robust mapping of arbitrary user text to predefined macros, yet the only validation is a synthetic 100-example classification with no metrics; this load-bearing assumption is untested.","rationale":"The reader's REJECT verdict is well supported, and my stress-test does not move it. The strongest claim is the comprehensive methodology plus the novel on-device architecture. The methodology portion is largely a survey; the substantive contribution is the architecture. That architecture's value rests on mapping user inputs to stored macros correctly and quickly. The paper's own Section 4.2 states that the experiment \"does not constitute a solid proof,\" and the surrounding text treats on-device integration as a \"valid, futuristic and useful concept\" rather than a measured result. The absence of metrics and code is compounded by a distributional bias: ChatGPT generated both the macro semantics and the 100 test inputs, so the test likely reflects template-like phrasing rather than real user variability. Moreover, the described architecture requires parameter extraction and execution of API-call sequences, but the experiment only checks use-case classification; even perfect classification would not guarantee a valid API call if arguments are wrong. These are internal-evidence concerns about correctness risk, not merely disagreements with current consensus. I therefore see no reason to change the reader's verdict; the preprint would need an independently collected human-utterance evaluation, reported metrics, and ideally a working prototype before its central architectural claims could be accepted.","tokens_in":65843,"tokens_out":4187,"duration_ms":44507,"concrete_test":"Reproduce Section 4.2 with an independent corpus: collect at least 20 natural utterances per macro from human users, varying wording, typos, code-switching, and requests combining two macros. Run the semantic-vector/embedding matcher described in Section 3.2.1 with a similarity threshold and a fallback-to-generative-model path, and also run BERT, Llama-3-8B, and Phi-2 classifiers. Report per-macro top-1 accuracy, macro-recall, argument-extraction F1 for slots such as X and Y in ORDER_FROM_NEAR-BY_MARKET, and fallback rate. Then measure end-to-end latency on a 2024 smartphone for matched and unmatched requests, comparing with a cloud-LLM baseline. If human-utterance accuracy declines sharply relative to ChatGPT-generated inputs, or if frequent-task latency is not below the cloud baseline, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2.1 claims that because the agent stores macros/functions in a local custom database, \"the AI-agent only needs to execute a classification task instead of a generative task,\" yielding very fast execution of frequent tasks. The load-bearing assumption is that the semantic-vector/word-embedding matcher reliably maps real user text to the correct macro, and additionally extracts the arguments needed for the API calls. Section 4.2, the only empirical support, does not validate this assumption. It uses 100 user inputs generated by ChatGPT-4, the same model family used to author the macro descriptions, then reports only that BERT, Llama-3-8B, and Phi-2 \"successfully\" classified them; no accuracy, confusion matrix, parameter extraction, or end-to-end API execution is reported. It tests classification by generative/discriminative models rather than the proposed semantic-vector search, and it evaluates only use-case labels, not the series of API calls with arguments that the agent is supposed to execute. Section 4.2 itself concedes the experiment \"does not constitute a solid proof.\" Without a test on independent human utterances, including paraphrases, ambiguity, multilingual input, and unseen macro combinations, the architecture's claimed low-latency and offline benefits are unsupported; if matching fails, the agent either mis-executes or must fall back to generative inference, eroding the speed claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 7-step methodology for augmenting LLMs with APIs, surveys existing frameworks and tools, and introduces an on-device architecture in which a local custom database stores macros/functions that map user requests to pre-defined API-call sequences. The authors argue that, because the agent only needs to classify a user request rather than generate a response, frequent tasks can be executed very fast and offline. The experimental part consists of a piano-sheet demonstration using ChatGPT-4 and a proof-of-concept classification experiment in which BERT, Llama-3-8B, and Phi-2 classify 100 ChatGPT-generated user inputs into use cases from a custom database.","tokens_in":65997,"tokens_out":3631,"duration_ms":39029,"significance":"If the central claims were rigorously established, the on-device macro-database architecture would be a useful contribution to the growing field of tool-augmented LLMs, particularly for low-latency and privacy-preserving personal assistants. The paper also provides a broad, readable synthesis of existing work on API selection, task decomposition, and feedback, and it is honest about several limitations. However, the claimed contributions are not supported by the evidence presented: the only quantitative experiment reports no accuracy values, no comparison baselines, and no end-to-end API execution, and the piano-sheet example is anecdotal. The value of the paper as submitted is mainly expository, not a validated research contribution.","major_comments":[{"comment":"The architecture's central claim is that replacing generation with classification yields very fast execution of frequent tasks, but the proof-of-concept in Section 4.2.2 only reports that classification was \"successful\" without giving accuracy, a confusion matrix, or any comparison between BERT, Llama-3, and Phi-2. More importantly, the experiment evaluates only use-case label classification; it does not evaluate the extraction of arguments such as X and Y in ORDER_FROM_NEAR-BY_MARKET(X, Y), nor the chaining of multiple macros into a series of executable API calls. These are load-bearing for the claimed speed and offline functionality, so the central claim is untested.","section":"Section 3.2.1 and Section 4.2.2"},{"comment":"The 100 user inputs are generated by ChatGPT-4, the same model family used to author the macro descriptions, so the evaluation is biased toward phrasings that match the database entries. There is no test on independent human utterances, paraphrases, ambiguous or multilingual requests, or unseen macro combinations. Section 4.2.2 itself concedes that the experiment \"does not constitute a solid proof\" of the method's effectiveness; given this concession, the on-device architecture's robustness claims in Sections 3.2.4 and 3.2.6 remain unsupported.","section":"Section 4.2.1"},{"comment":"The claimed advantages of reduced latency, offline availability, and scalability are stated without any quantitative evidence. No end-to-end system is implemented, no latency or memory measurements on a real device are reported, and no comparison with a cloud-based LLM baseline is provided. The example database in Figure 3.3 is explicitly labeled as illustrative, so the feasibility of the matching and execution flow is not demonstrated beyond a synthetic classification exercise.","section":"Sections 3.2.3-3.2.6"},{"comment":"The piano-sheet example is anecdotal: it shows ChatGPT-4 outputs and states that the melody is correct and the fingering is reasonable, but it provides no evaluation rubric, no quantitative measure of correctness, no comparison with alternative approaches, and no evidence that the generated sheet is usable by a pianist. The claim that this demonstrates the effectiveness of the proposed methodology is therefore not supported, and the section does not exercise the proposed on-device architecture.","section":"Section 4.1"}],"minor_comments":[{"comment":"The paper contains both a Greek and an English version with duplicated content and some inconsistencies between the two versions; for example, the Greek Chapter 3 title differs from the English Chapter 3 title. The manuscript should be consolidated into a single coherent article.","section":"Manuscript structure"},{"comment":"The loss-function equation has malformed typesetting and unclear notation; the subscripts and the meaning of n and j should be defined precisely.","section":"Section 2.4.2"},{"comment":"Figure 4.9 appears to be an OCR-like rendering of the 100 user inputs with numerous typos and artifacts (e.g., \"SUmmariz:e\", \"wori:.out\"); it should be replaced with a clean, machine-readable table, and the exact model versions and prompts should be provided for reproducibility.","section":"Figure 4.9"},{"comment":"The text says the LLM was connected to the MuseScore API, but no actual API call, endpoint, authentication method, or raw response is shown; the reader cannot verify whether the integration was real or simulated.","section":"Section 4.1.3"},{"comment":"The claim that tools are designed to be resistant to adversarial attacks and jailbreaking is made without citation or supporting evidence and should be either substantiated or removed.","section":"Section 2.5.2"},{"comment":"Several references are cited incompletely or inconsistently (e.g., [18] and [5,9] in Section 2.1), and figure sources are not uniformly credited; the reference list needs to be checked carefully for completeness.","section":"Bibliography"}],"recommendation":"reject","confidential_remarks":"For the editor: the manuscript is essentially a diploma thesis with a large expository component and a small proof-of-concept that is explicitly disclaimed by the authors. The lack of any quantitative evaluation, the absence of an end-to-end implementation, and the synthetic nature of the only experiment make the paper unsuitable for publication as a research article in its current form. The on-device macro-database idea may be worth revisiting if supported by a real implementation with measured latency and accuracy on diverse user inputs, but that is beyond the scope of the present text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a diploma thesis, not a research paper. It has two parts: a 7-step methodology for building API-augmented LLM agents, which is a competent synthesis of ReAct, CoT, API-Bank and the tool-use survey literature; and an on-device architecture that stores user-defined macros and matches requests via embeddings, so the agent supposedly \"only needs to execute a classification task.\" The second part is the only real proposal, and it is not validated.\n\nWhat's good: the survey is broad and well organized for a newcomer; the piano-sheet walkthrough is a nice concrete illustration of few-shot prompting plus calling a real API (MuseScore). The author is honest about the limits, explicitly conceding in Section 4.2 that the experiment \"does not constitute a solid proof.\" Credit where due.\n\nThe soft spots are exactly where the architecture is load-bearing. The speed and offline story depends on reliably mapping arbitrary user text to predefined macros, including extracting arguments for API calls. The proof-of-concept tests only 100 ChatGPT-generated paraphrases, reports only \"successful classification\" with no accuracy, confusion matrix, or parameter extraction, and never executes an API call end-to-end. It also uses BERT/Llama/Phi-2 for classification, which is not the same as the proposed semantic-vector search. Unseen human phrasings, multilingual input, and ambiguity are not tested. If matching fails, the agent mis-executes or falls back to generative inference, and the latency claim erodes. The author acknowledges the weak evidence, but the architecture's central claim remains unsupported.\n\nMinor issues: the manuscript has typos and Greek/English duplication; the \"novel\" on-device architecture reduces to intent classification over a static macro set, a pattern already common in voice assistants and RPA. None of this is fatal for a tutorial, but it is not a research contribution.\n\nBottom line: this is a useful introductory read for students entering the LLM-agent area, not a paper that advances the field. It does not deserve a serious referee. My recommendation is to desk reject; if someone wants a friendly orientation to LLM+API agents, point them here, but don't cite it as evidence.","headline":"A sincere but under-supported diploma thesis: a useful survey of LLM-API techniques, a plausible but unvalidated on-device macro architecture, and a toy proof-of-concept the author himself stops short of defending.","tokens_in":66604,"tokens_out":2693,"would_cite":false,"duration_ms":27812,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This thesis presents a seven-step methodology for giving large language models API access and an on-device architecture that runs recurring tasks as fast classifications.","keywords":["large language models","AI agents","API integration","semantic vector space","word embeddings","on-device architecture","task decomposition","API selection"],"falsifier":"Take genuinely new user requests—including paraphrases, multilingual inputs, and voice-transcribed queries—run them through the local matcher, and count how often the correct macro or API sequence is selected; a substantial drop in accuracy, or frequent user failures while teaching new macros, would refute the paper's central premise.","tokens_in":65543,"feed_emoji":"🔌","tokens_out":10345,"duration_ms":88717,"temperature":0.7,"pith_summary":"Large language models are powerful but sealed off from the external digital world: their knowledge freezes at training time and they cannot take actions. This thesis argues that wiring them to application programming interfaces fixes that, and it distills the wiring process into seven steps—model selection, tool-knowledge enhancement, multi-stage pipeline design, API selection, API-call generation, task decomposition, and feedback-driven improvement. Its sharper proposal is an on-device architecture: reusable tasks are stored locally as macros that map a user description to a fixed series of API calls, so the agent answers recurring requests by classifying them into the right macro instead of generating a fresh response. If correct, this would give phones and tablets a fast, offline, privacy-preserving assistant for frequent jobs, with the cloud model reserved for new or complex ones.","feed_headline":"Seven steps to give LLMs API-powered hands","feed_subtitle":"Recurring requests become quick offline classifications, keeping data on-device and cutting latency","key_machinery":"The load-bearing mechanism is the locally stored custom database of macro-functions: each entry pairs a task title and description with the ordered series of API calls that fulfils the task, so the agent can remember exactly what to execute. Matching a user's phrasing to the right entry is done in a semantic vector space—a representation in which similar meanings are close together—using word embeddings and cosine similarity. That turns every recurring request into a classification problem, replacing a slow generative step with a fast lookup.","core_discovery":"The central discovery is a high-level recipe for turning a text-only LLM into an API-calling AI agent, plus a concrete architectural proposal for running that agent on a portable device. The seven steps walk a builder from choosing a suitable model and injecting external tool knowledge, through a multi-stage pipeline that identifies intent, selects and calls an API, and processes the response, to task decomposition and iterative user feedback. The on-device architecture replaces the generative step with a classification step: a locally stored custom database holds macro-functions—each with a title, a description, and an ordered series of API calls—and at run time the agent matches the user's request to an entry through a semantic vector space of word embeddings. In the proof of concept, one hundred synthetic user requests were classified into the right macro category by all three tested models, and the two generative models could also combine macros or offer extra explanations; the thesis itself describes the experiment as a strong indication rather than a solid proof.","pith_inferences":["Beyond the paper: testing the semantic matcher on real user utterances—including multilingual paraphrases, typos, and voice transcription—would likely lower the reported 100 percent classification rate, and that drop would mark the architecture's true reliability boundary.","Beyond the paper: a hybrid design that falls back to a cloud LLM when no local macro scores above a threshold would trade a little privacy for much lower failure rates on novel requests, extending the architecture's reach without abandoning its speed.","Beyond the paper: a direct latency and energy benchmark comparing classification-only service with on-device generation for the same task list would make the claimed speed advantage quantitative and testable."],"forward_implications":["Recurring requests can be answered with minimal latency, because a semantic lookup over a small local database is far cheaper than generating a response from a cloud LLM.","The agent keeps working offline for tasks that do not require live data, since both the database and the small models live on the device.","Users can personalize the agent by teaching it new tasks through the training interface, and the macro library can grow through sharing.","Large, multi-step goals can be split into dependent API calls, so a single request can trigger a coordinated sequence such as booking flights and hotels in order.","Iterative feedback and retraining allow the agent to adapt its API choices and call formats to observed success rates."],"supporting_citations":[{"why":"Provides the powerful LLM used to generate synthetic user inputs and training data, and to demonstrate tool-augmented responses such as weather lookup.","marker":"[7]"},{"why":"Supplies the multi-stage method for creating good-quality training data for APIs with a strong LLM, and the depth-first search reasoning technique for task decomposition.","marker":"[5]"},{"why":"Contributes the annotated API-dialogue benchmark and the multi-stage pipeline structure for API call generation and evaluation.","marker":"[3]"},{"why":"Offers the model-selection criteria and the two-level API selection mechanism based on a loss-function metric.","marker":"[18]"},{"why":"Provides the embedding model used for semantic-vector matching and classification in the proof of concept.","marker":"[8]"},{"why":"Supplies the categorized API hub used for domain-first API selection.","marker":"[31]"},{"why":"The music-notation API used in the piano-sheet example to turn generated notation into a printed score.","marker":"[6]"},{"why":"Provides the community leaderboard used to compare and select open-source LLMs during model selection.","marker":"[23]"}],"fun_headline_variants":["7-step blueprint for API-empowered LLM agents","On-device AI agents: match requests to macros, not generation","Turn LLMs into API callers with this 7-step guide","Macro-match, not generate: on-device LLM agent recipe","Small models, wide API reach: an on-device agent recipe"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The architecture's speed and offline availability rest on the assumption that matching the user's wording to a stored task description by meaning will always pick the correct macro, and that users can reliably teach the agent new macros through the training interface.","fun_headline_variants_meta":{"raw":{"variants":["7-step blueprint for API-empowered LLM agents","On-device AI agents: match requests to macros, not generation","Turn LLMs into API callers with this 7-step guide","Macro-match, not generate: on-device LLM agent recipe","Small models, wide API reach: an on-device agent recipe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000662,"raw_usage":{"total_tokens":3070,"prompt_tokens":1036,"completion_tokens":2034,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":1944}},"tokens_in":652,"tokens_out":2034,"duration_ms":16190,"temperature":1.0,"reasoning_tokens":1944,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:33:54.717520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take genuinely new user requests—including paraphrases, multilingual inputs, and voice-transcribed queries—run them through the local matcher, and count how often the correct macro or API sequence is selected; a substantial drop in accuracy, or frequent user failures while teaching new macros, would refute the paper's central premise.","supporting_citations":[{"cited_title":"Toolformer: Language models can teach themselves to use tools,","cited_arxiv_id":null,"evidence_quote":"Offers the model-selection criteria and the two-level API selection mechanism based on a loss-function metric."},{"cited_title":"Rapid’s api hub","cited_arxiv_id":null,"evidence_quote":"Supplies the categorized API hub used for domain-first API selection."},{"cited_title":"Musescore","cited_arxiv_id":null,"evidence_quote":"The music-notation API used in the piano-sheet example to turn generated notation into a printed score."},{"cited_title":"Hugging face: Open source machine learning tools","cited_arxiv_id":null,"evidence_quote":"Provides the community leaderboard used to compare and select open-source LLMs during model selection."}],"review_version":1}