{"id":"463e7d9e-c3bd-47b3-a5b2-46689ece8e48","arxiv_id":"2412.16597","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An LLM-based voice interface for surgical AR navigation reduced task completion time and cognitive load versus keyword speech commands in simulated pancreatic surgery tasks.","lead":"This paper builds a voice-controlled interface for an augmented-reality surgical navigation system that uses a large language model to turn natural spoken requests into calls to the system's existing functions, and compares it with a conventional keyword-command voice interface. In a simulator study with nine surgeons, the LLM interface produced faster task completion and lower reported cognitive load, while two real surgeries suggested it is usable in practice.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparison is confounded: the LLM arm receives patient-specific decision support and macro-command generation that the speech-command arm lacks, so the TCT/cognitive-load advantage may not be attributable to the voice interface.","rationale":"The reader's verdict is CONDITIONAL and already flags missing LLM accuracy and patient-specific decision context. My stress-test sharpens the same weakness: even under perfect LLM function-call accuracy, the reported TCT/NASA-RTLX advantage cannot be attributed to the voice interaction paradigm because the LLM arm includes decision support and compound-command execution that the baseline lacks. This is load-bearing because the Abstract's 'compared to speech commands' phrasing invites the reader to conclude the voice modality is superior, whereas the experiment actually compares a context-aware assistant-plus-voice system to a bare keyword interface. The concern is not an internal inconsistency: the paper explicitly frames context-awareness as a benefit of LLMs and acknowledges limitations. It is a boundary condition on the central claim. A third, information-balanced condition would settle whether the effect is due to the LLM's understanding or to the added task intelligence. The reader also notes no code/data and a two-surgery case study with no quantitative comparison; those reinforce conditionality but are secondary. I would retain CONDITIONAL: the simulator data are internally consistent and the qualitative results are plausible, but the headline comparison should be re-scoped as 'LLM-based context-aware VCUI' rather than 'voice interaction method,' and the missing prompt/accuracy data should be reported before generalization. Hence UNCHANGED from the reader's verdict, with the condition made explicit: balanced information access and released LLM decision logs.","tokens_in":18468,"tokens_out":7479,"duration_ms":66185,"concrete_test":"Run a third simulator condition: the same keyword-based speech-command VCUI augmented with equivalent context-aware decision support, e.g., a precomputed 'show infiltrated/resectable structures' macro per task, so the only difference is keyword invocation vs. natural-language invocation. If TCT and NASA-RTLX for this speech+support condition are statistically indistinguishable from the LLM condition on Tasks 5-6, the reported advantage is attributable to added decision support, not to the LLM voice interface. Additionally, publish the exact initial prompt and per-query interaction logs, and compute exact-match accuracy of LLM function calls against ground truth; if accuracy is below ~95% or the heuristic examples paraphrase task language, the generalization claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract) is that an LLM-based VCUI yields significantly lower task completion time and cognitive workload than speech commands. For this to be a claim about the voice-control method, the two arms must differ only in input interaction. They do not. In Section 4.2.2, the LLM arm is initialized with a per-patient prompt containing organ distance data, diagnosis, surgical resection guidelines, and 'sentencesAndResultsExamples', while the speech-command arm (Section 4.2.1) is a fixed set of 34 atomic keywords. Tasks 5 and 6 deliberately omit the names of structures to visualize and require clinical inference; the LLM performs that inference from its private prompt, whereas the speech-command surgeon must reason unaided. The experiment therefore varies two factors at once: input grammar (natural language vs. keywords) and system intelligence (context-aware compound function calls vs. no decision support). The significantly larger TCT gap on Tasks 5-6 and the qualitative quote 'I simply asked and that showed' (Section 5.5.2) are consistent with the advantage coming from decision support/macro execution, not from the voice modality. The paper's assertion (Section 4.2.2) that no direct annotation to study tasks was included in the prompt cannot be audited because the actual prompt and interaction logs are not released. Without an information-balanced baseline or per-query LLM function-call accuracy, the headline comparative result remains confounded.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an LLM-based voice-controlled user interface (VCUI) for an augmented-reality surgical assistance system (ARAS) used in pancreatic surgery. The authors compare this interface against a conventional speech-command interface in a within-subject simulated study with nine experienced surgeons (six standardized tasks, NASA-RTLX, SUS, semi-structured interviews) and in two real pancreatic surgeries with post-operative interviews. Quantitative results show significantly lower task completion times for all six tasks and significantly lower NASA-RTLX total, mental demand, physical demand, and effort scores for the LLM condition; qualitative findings indicate a strong surgeon preference for the LLM-based interface. The central claim, stated in the Abstract, is that the LLM-based VCUI achieves significantly lower task completion time and cognitive workload compared to speech commands.","tokens_in":18753,"tokens_out":4629,"duration_ms":40705,"significance":"If the comparison were clean, this would be a valuable contribution as one of the few evaluations of LLM-based voice control in a surgical AR setting, with a rare real-surgery case study and mixed-methods data. The authors give credit for involving surgeons throughout the design process, reporting both simulated and intraoperative deployments, and acknowledging limitations such as the specific functionality scope. However, the significance is conditional on resolving the confounding between the voice modality and the additional decision-support capabilities provided only to the LLM arm. As presented, the headline comparative claim cannot be attributed specifically to the voice interface, and the inability to audit the initial prompt weakens the reproducibility of the result.","major_comments":[{"comment":"The two experimental arms differ in two factors simultaneously: the input grammar (natural language versus fixed keywords) and the availability of a patient-specific prompt that provides the LLM with diagnosis, distance data, surgical guidelines, and sentence-result examples. The speech-command arm contains none of this information. Tasks 5 and 6 were explicitly designed to require clinical inference about which structures to visualize, and the significantly larger TCT benefit on these tasks (Section 5.5.1, t(8) = -4.34 and -4.16) is consistent with the advantage coming from the LLM's decision support and compound function calling rather than from the voice-control modality itself. The Abstract's claim of 'significantly lower task completion time and cognitive workload compared to speech commands' is therefore a system-level comparison, not a comparison of voice interaction methods. The authors should add an information-matched baseline (for example, a speech-command interface with pre-programmed macros for the compound tasks) or substantially reframe the claim and the research questions.","section":"Section 4.2.2 vs. 4.2.1; Appendix A"},{"comment":"This assertion cannot be audited because the actual initial prompt is not released. Moreover, Appendix A shows that the prompt includes a 'sentencesAndResultsExamples' list, one entry of which is the sentence 'Show me the infiltrated vessels' with the corresponding result being a set of function calls. This example is structurally identical to Task 5 (Tumor infiltration), in which participants had to enable the structures infiltrated by the tumor. The authors must either make the full initial prompt available or provide evidence that the examples were finalized before the study tasks were designed and that they do not overlap with the tasks. Without this, the statement that no bias was introduced cannot be independently verified.","section":"Section 4.2.2, 'no direct annotation to the study tasks (C5.1)'"},{"comment":"The paper's central mechanism is context-aware function calling: the LLM infers which functions to invoke from natural language and patient-specific context. However, the paper reports no objective accuracy measure for these function-call decisions. The only success criterion is the participant's verbal confirmation that the visualized structures were correct (Section 5.3), and the reset function is described as a safeguard (Section 4.2.2). The narrative in Section 7.1 that the LLM 'correctly decided' in complex cases (e.g., P2's 'Can you show me what should be resected?') is illustrative but does not quantify correctness, resets, or unrecognized errors. Without per-query decision accuracy or at least reset counts, the measured TCT savings might include time spent recovering from incorrect actions, which would change the interpretation of the efficiency benefit. Please report these figures.","section":"Sections 5.5.1 and 7.1"}],"minor_comments":[{"comment":"The reference to the JSON format appears as '??' in the text; please cite Appendix A explicitly.","section":"Section 4.2.2"},{"comment":"There is a typo: 'cogntitive' should be 'cognitive'.","section":"Introduction"},{"comment":"The table header contains stray spaces and typos ('V oice Keywords', 'F unctionality'), and one row appears to have misaligned content: 'Activates/Deactivates rendering of the associated structure' should specify which structure.","section":"Table 2"},{"comment":"The two surgeons who participated in the case study are first referred to as P1 and P2, but the quotes in Section 6.3 are attributed to S1 and S2. Please clarify whether these are the same participants and use consistent labels.","section":"Section 6.2"},{"comment":"The word 'frustration' is misspelled as 'frustation' in the figure and surrounding text.","section":"Figure 8 and Section 5.5.1"},{"comment":"Please report effect sizes (e.g., Cohen's d) for the TCT and NASA-RTLX comparisons to help readers assess practical significance. The multiple subscale tests are reported without any familywise-error correction; although the number of significant subscales exceeds what would be expected by chance, this should be acknowledged with effect sizes and corrected p-values or a note about the exploratory nature of the subscale analyses.","section":"Section 5.5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal's interests and reports a rare real-surgery evaluation. My main concern is the confound between the voice-control modality and the decision-support functionality: the LLM arm receives information and inference capabilities that the speech-command arm lacks, which directly affects the headline claim. The inability to audit the initial prompt is also a reproducibility risk, especially since the appendix contains an example resembling a study task. I believe these issues are fixable within the manuscript's scope: the authors could reframe the contribution as a system-level evaluation, add a matched baseline, release the full prompt, and report LLM decision accuracy. If the authors choose to reframe rather than add a baseline, they should also adjust the Abstract and research questions so that the claim does not overstate the voice-modality comparison. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the quick take: the headline claim — an LLM-based voice interface yields significantly lower task completion time and cognitive load than speech commands — is not cleanly supported by the study design. The LLM arm receives a patient-specific prompt containing organ distance data, diagnosis, surgical guidelines, and sentence-result examples, while the speech-command arm is just 34 keywords. On tasks 5 and 6, where structure names are withheld, the LLM can infer what to show from that prompt; the speech-command surgeon must reason unaided. The advantage is therefore attributable to added decision support and macro-command generation, not to the voice modality alone. The paper's assertion that the prompt contains no direct annotation to study tasks cannot be audited because the prompt is not released.\n\nThat is the load-bearing flaw. The rest of the paper is solid. The integration — a dynamic prompt generator, distance calculations, context-aware function calling, and a reset mechanism — is a real contribution. The simulator study (N=9 expert surgeons, within-subject) reports faster completion on all six tasks and lower NASA-RTLX total and three subscales, with consistent interview quotes. The authors are candid about limitations and about the system's simple functionality, and the real-surgery case study is appropriately presented as qualitative.\n\nMinor concerns: one-tailed tests with no multiple-comparison correction or effect sizes, N=9 (though typical for this domain), no code or data release, and no per-query LLM function-call accuracy. The \"significantly lower\" in the abstract applies to the simulator study, not the clinical sessions.\n\nWho benefits: researchers designing LLM-based interaction for medical AR and anyone studying voice control in high-stakes environments. It is a useful existence proof and a good discussion piece. I would cite it with a caveat, and I'd bring it to a reading group.\n\nRecommendation: send it to peer review. It deserves referee time, and the reviewers should ask for an information-balanced baseline, reported LLM decision accuracy, and the actual prompt and interaction logs.\n\nBest,\n\nA.","headline":"A worthwhile feasibility study of an LLM-based surgical AR voice interface, but the comparative claim is confounded by unequal decision support between conditions.","tokens_in":19295,"tokens_out":3908,"would_cite":true,"duration_ms":32984,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A language-model voice interface for augmented-reality surgical navigation outperforms keyword speech commands on task time and cognitive workload, in simulation and in the operating room.","keywords":["augmented reality surgery","voice-controlled user interface","large language models","surgical navigation","cognitive workload","speech commands","pancreatic surgery","context-aware function calling"],"falsifier":"Audit every LLM function call during the six study tasks against a surgeon-defined ground-truth answer for each task; if the LLM's action-level accuracy is at or below the speech-command baseline's recognition accuracy while task completion times remain faster, the claim that the advantage comes from correct context-aware calls would be refuted.","tokens_in":18263,"feed_emoji":"🏥","tokens_out":7379,"duration_ms":57373,"temperature":0.7,"pith_summary":"This paper argues that a voice-controlled interface for an augmented-reality surgical navigation system works better when a large language model, rather than fixed keyword commands, turns the surgeon's speech into system actions. In a simulated pancreatic-surgery study with nine experienced surgeons, the LLM interface produced significantly lower task completion times across all six tasks and a significantly lower overall cognitive workload score than a conventional speech-command interface, while usability ratings were high for both. Interviews with the surgeons and a two-case clinical deployment in real pancreatic surgeries supported the quantitative finding, with surgeons describing the natural phrasing as less stressful and better suited to time-critical decision moments. The point of the work is that context-aware function calling lets a small set of simple system functions be composed on the fly, which matters because surgical hands are occupied, the AR view must not be occluded, and every decision happens under time pressure.","feed_headline":"LLM voice control cuts time and cognitive load in AR surgery","feed_subtitle":"Speaking naturally beat memorizing 34 keywords: faster tasks, lower mental workload in simulator and operating room.","key_machinery":"The carrier of the argument is the LLM-based VCUI's function-calling loop. A dynamic initial prompt is generated per patient, containing the patient-specific 3D model's inter-structure distance data, diagnosis, resection guidelines, the list of executable ARAS functions, and heuristic sentence-to-function examples; the prompt instructs the model to return only appropriate function calls. The paper's key safeguard is an auto-repeat mechanism that resends this initial prompt after every user utterance to prevent the model from drifting out of context, plus a reset function that terminates the chat and stores the user's correction as a new example, which is a concrete mitigation against LLM hallucination. This context-aware function calling is what lets one natural sentence invoke several visualization functions in parallel, and it is the mechanism that the study's task-time and cognitive-load advantages are attributed to.","core_discovery":"On the paper's own terms, the central discovery is that an LLM-based voice-controlled user interface (VCUI) is a feasible and measurably more efficient control modality for an AR-based surgical assistance system than keyword speech commands. In a within-subject simulated study with nine experienced surgeons, the LLM condition had significantly lower task completion time on all six surgically grounded tasks and a significantly lower total NASA-RTLX score, with significant reductions in mental demand, physical demand, and effort; SUS was rated 'excellent' for the LLM interface (87.78) and 'good' for speech commands (79.17), although that difference was not statistically significant. The authors also report qualitative agreement from post-study interviews and from a case study in which each interface was used during an actual pancreatic tumor resection. The explanation they offer is that the LLM decides which system functions to invoke from the context of a natural request and from patient-specific data, so the surgeon no longer has to translate a clinical judgement into a sequence of keywords.","pith_inferences":["Beyond the paper, the same 'prompt with context and function list' pattern could transfer to other hands-busy, high-stress AR settings such as industrial maintenance or emergency response, provided the LLM's function calls are constrained to a small verified set.","A direct testable extension would be to measure the LLM's action-level accuracy against a surgeon-annotated ground truth for each task; high accuracy would confirm the TCT gain is real, while low accuracy would imply part of the saving comes from faster-but-wrong calls that users must catch and reset.","The paper describes but does not implement a hybrid interface; an experiment comparing keyword commands, LLM-only, and a switchable hybrid on task time and workload would quantify the trade-off between direct control and context-aware flexibility.","Because tasks 5 and 6 were deliberately not annotated in the system prompt, repeating them with paraphrased requests would test whether the LLM's success is robust to phrasing or depends on the particular sentence forms used in the study."],"forward_implications":["In time-critical intraoperative moments, a surgeon can ask for structures relevant to the current decision in plain language, and the system composes the required visualization immediately instead of the surgeon issuing several keyword commands.","Because the LLM composes simple system functions on the fly, the ARAS function vocabulary can stay small; adding new clinical behaviours does not force an ever-growing list of unique spoken keywords.","The reset-and-remember interaction gives users a way to correct the LLM in situ, and the stored correction becomes an example in the regenerated prompt, so the system's behaviour can be adapted per patient or surgeon.","The reported qualitative data imply that a hybrid interface, speech commands for direct deterministic actions and an LLM for complex or time-pressed decisions, would preserve the transparency of keywords while keeping the flexibility of natural language."],"supporting_citations":[{"why":"Supplies the clinically evaluated speech-command voice control that the LLM interface is compared against.","marker":"[40]"},{"why":"Second real-world speech-command voice-control baseline, also using a head-mounted AR display in surgery.","marker":"[9]"},{"why":"Provides the NASA-RTLX cognitive-workload measure used for the workload comparison.","marker":"[17]"},{"why":"Provides the SUS usability scale used to rate both interfaces.","marker":"[5]"},{"why":"Supplies the pragmatic qualitative-analysis method used for the post-study interviews.","marker":"[3]"},{"why":"Provides the adjective rating scale that turns SUS scores into 'good' versus 'excellent' classifications.","marker":"[2]"}],"fun_headline_variants":["LLM voice control speeds AR surgery and cuts load","Natural language voice beats keyword commands in AR surgery","LLM-driven voice UI reduces time and workload in AR surgery","Context-aware LLM voice trims AR surgery time and burden","Surgeons find LLM voice interface more intuitive for AR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured time saving counts as a genuine advantage only if the LLM's function-call decisions are actually correct, but the paper reports no quantitative accuracy for those decisions, relying instead on a reset function and participants' verbal confirmation.","fun_headline_variants_meta":{"raw":{"variants":["LLM voice control speeds AR surgery and cuts load","Natural language voice beats keyword commands in AR surgery","LLM-driven voice UI reduces time and workload in AR surgery","Context-aware LLM voice trims AR surgery time and burden","Surgeons find LLM voice interface more intuitive for AR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1797,"prompt_tokens":916,"completion_tokens":881,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":801}},"tokens_in":532,"tokens_out":881,"duration_ms":9334,"temperature":1.0,"reasoning_tokens":801,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:26:27.318167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit every LLM function call during the six study tasks against a surgeon-defined ground-truth answer for each task; if the LLM's action-level accuracy is at or below the speech-command baseline's recognition accuracy while task completion times remain faster, the claim that the advantage comes from correct context-aware calls would be refuted.","supporting_citations":[{"cited_title":"Augmented reality during parotid surgery: real-life evaluation of voice control of a head mounted display","cited_arxiv_id":null,"evidence_quote":"Supplies the clinically evaluated speech-command voice control that the LLM interface is compared against."},{"cited_title":"Augmented reality during parotid surgery: Real-life evaluation of voice control and user-experience","cited_arxiv_id":null,"evidence_quote":"Second real-world speech-command voice-control baseline, also using a head-mounted AR display in surgery."},{"cited_title":"Nasa-task load index (nasa-tlx); 20 years later","cited_arxiv_id":null,"evidence_quote":"Provides the NASA-RTLX cognitive-workload measure used for the workload comparison."},{"cited_title":"Sus-a quick and dirty usability scale.Usability evaluation in industry , 189(194):4–7, 1996","cited_arxiv_id":null,"evidence_quote":"Provides the SUS usability scale used to rate both interfaces."},{"cited_title":"Qualitative HCI research: Going behind the scenes","cited_arxiv_id":null,"evidence_quote":"Supplies the pragmatic qualitative-analysis method used for the post-study interviews."},{"cited_title":"Determining what in- dividual sus scores mean: Adding an adjective rating scale","cited_arxiv_id":null,"evidence_quote":"Provides the adjective rating scale that turns SUS scores into 'good' versus 'excellent' classifications."}],"review_version":1}