Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A modular AI assistant turns spoken commands into real beamline experiments.

desk verdict A real, working voice-controlled beamline demo with an honest but under-powered evaluation; worth refereeing, but the quantitative claims need a cleaner benchmark. read the letter →

arxiv 2412.18161 v1 pith:BWX2NB26 submitted 2024-12-24 cs.AI

Shray Mathur , Noah van der Vleuten , Kevin Yager , Esther Tsai This is my paper · ORCID
classification cs.AI
keywords naturallanguageinstrumentcontrollargemodelssynchrotronbeamlinesvoice-controlledexperimentcognitiveblocksprogramsynthesisX-rayscatteringhuman-instrumentinteraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VISION is a modular voice and text assistant that lets a scientist at an X-ray scattering beamline say or type commands such as "measure the sample for five seconds" and have them turned into executable Python that drives the instruments. The paper's central claim is that a deterministic ensemble of small specialized LLM components, called cognitive blocks or cogs, can route a spoken request to the right tool, generate the correct data-acquisition or analysis code, and run a real experiment with low latency on an ordinary beamline workstation while the models run on a separate GPU server. The authors report a recorded voice-controlled beamtime at a synchrotron beamline as the first of its kind, together with per-cog evaluations of speech transcription, command classification, and code generation. If the claim holds, researchers no longer need a developer at the instrument or deep command-line expertise: natural language becomes the interface, and new instrument capabilities can be added simply by describing them.

What carries the argument

The central object is the cognitive block, or cog: one LLM scaffolded with domain-specific prompts or tools to perform a single task, with cogs assembled in a deterministic workflow to form the assistant. A classifier cog routes each user utterance to the operator, analyst, or another tool; the operator cog generates Python code from natural-language commands; a speech-transcription cog is fine-tuned with synthetic audio to recognize beamline jargon; and a refiner cog converts descriptions of new functions into JSON that extends the dynamic system prompts at inference time. The integration that connects the generated code to the hardware is keystroke injection into the interactive data-acquisition terminal, which keeps the conventional command line usable but is the channel the paper itself flags as not robust.

What would settle it

Run the Operator cog's generated code for its sequential and structured test cases inside a mocked instrument environment that actually simulates motor moves and detector triggers, and count how many runs perform the intended experiment; the paper reports exact-match accuracy around 77 to 83 percent and explicitly says functional-equivalence testing is under development, so this would directly settle whether the code executing at the beamline really does what the user asked.

Watch

Extended reading notes

Core claim

The paper sets out to show that the gap between a scientist's intent and a beamline's hardware can be closed by an ensemble of specialized LLM components rather than by one general assistant. VISION transcribes speech, classifies the request, produces the Python that moves motors, triggers the detector, sets temperature, or runs an analysis, shows the code to the user for confirmation, and then executes it. The reported live demonstration at an X-ray scattering beamline is the paper's evidence: a user spoke commands and the system moved a sample motor, controlled the detector, varied the sample stage temperature, and performed basic data reduction and visualization, with LLM workloads running on a separate GPU server so the beamline workstation stayed light-duty. On that evidence the authors conclude that VISION is the first voice-controlled beamline system, capable of low-latency LLM operation while preserving the conventional command-line workflow.

Load-bearing premise

The voice-controlled beamtime rests on keystroke injection into an interactive terminal as the connection between the assistant and the hardware, and the paper concedes this integration is not robust; if that channel cannot be made dependable across beamlines, the central demonstration remains a one-off rather than a trustworthy system.

Editorial extensions

If this is right

  • A new user could perform a beamtime by speaking instructions, while the conventional command-line interface remains available since the same terminal still receives the commands.
  • The add-custom-function workflow lets instrument scientists extend VISION to new hardware or protocols by describing them in natural language rather than writing integration code.
  • The modular cog design allows models to be swapped per task, so the architecture can adapt to other beamlines or other complex instruments beyond X-ray scattering.
  • The speech fine-tuning pipeline shows a transcription model can learn domain jargon from roughly forty synthetic examples per term in about a minute per term on the GPU server.
  • The architecture keeps a human confirmation step before any generated code is executed, making the system a supervised assistant rather than a fully autonomous agent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The keystroke-injection channel is the true bottleneck: if a structured queue or shared namespace replaced it, as the paper suggests, the same architecture would likely transfer to other instrument-control frameworks, and a direct test would be a deployment on a second, differently configured beamline.
  • The reported code-generation accuracies, based on string and syntax similarity rather than functional equivalence, may not reflect whether the executed commands actually perform the intended measurement, so a mock-environment test would sharpen or revise the headline claim.
  • The same cog ensemble pattern could plausibly be reused for microscopy, spectroscopy, or neutron scattering facilities where scientists face the same barrier of command-level instrument interfaces.
  • The "first voice-controlled beamline" claim should be read as a demonstration of feasibility rather than of dependable everyday operation, since the paper explicitly describes the integration method as not robust.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents VISION, a modular AI assistant for scientific user facilities, composed of LLM-based 'cognitive blocks' (cogs) for speech transcription, command classification, beamline control code generation (Operator), data analysis (Analyst), function refinement, and chatbot queries. The system uses dynamic prompts built from JSON files and communicates with the Bluesky data acquisition framework through keystroke injection into an interactive IPython terminal. The authors report per-cog evaluations across multiple LLMs and demonstrate a voice-controlled GIWAXS experiment at the NSLS-II 11-BM beamline, claiming this is the first voice-controlled beamline system with low-latency LLM-based operation on a light-duty workstation.

Significance. If substantiated, VISION would be a useful step toward natural-language control of complex scientific instruments. The paper's strengths are its clearly described modular architecture, the reproducible dynamic-prompt design, the detailed supplementary material (including full prompts), the efficient fine-tuning pipeline for Whisper on beamline jargon, and a real beamline demonstration with user confirmation. The authors also deserve credit for unusually candid acknowledgment of evaluation limitations. However, the central claim that VISION reliably translates natural language and speech into correct beamline control code is not yet supported by the reported quantitative evaluation: the evaluation dataset was used to refine the very prompts being tested, the ground truth was partially generated by two of the evaluated models, the best reference implementation is chosen per metric, and the metrics are string-similarity measures rather than execution-based functional equivalence tests. The single live demonstration, executed through a keystroke-injection channel that the paper itself describes as 'not robust,' is not a reliability measurement.

major comments (4)
  1. [§3.1 and S1.2.5 (Tables 3, S4)] The Operator evaluation does not substantiate the claim that VISION generates correct beamline control code. The paper states in S1.2.5 that the dataset 'should not be viewed as a testing dataset since the it was used to guide the prompts,' that the prompt is 'heavily geared towards Qwen2.5-coder,' that 'the ground truth solutions were partially constructed from outputs of Qwen2.5-Coder and Claude-3.5-Sonnet,' and that the best reference implementation is independently selected per metric. The metrics reported (exact match, Levenshtein distance, CodeBLEU) compare strings, not execution outcomes. These choices can inflate reported scores and do not measure whether the generated code would actually perform the intended beamline action. I recommend either implementing the functional-equivalence mocking that the authors note is 'under development' or explicitly reframing Tables 3 and S4 as prompt-engineering quality indicators rather than system accuracy.
  2. [§3.2 and §4] The abstract and conclusion claim VISION is 'the first voice-controlled beamline system' and that the beamtime demonstration establishes this milestone. However, the demonstration is a single user-confirmed run, and Section 4 concedes that the keystroke-injection integration used to control Bluesky 'is not robust and an enhanced integration is needed.' A single successful demonstration through a non-robust channel does not support a general claim about dependable voice-controlled beamline operation. I suggest softening the claims to 'demonstrated a voice-controlled experiment' and adding a discussion of the integration channel's reliability, including failure modes and any repeated trials.
  3. [§3.1 (Tables 2, 3, S3, S4) and S1.2.5] The cross-model comparisons are confounded by prompt overfitting and uneven quantization. The Operator prompt was iteratively improved on the evaluation dataset specifically for Qwen2.5-Coder-32B-Instruct, so the statement that this model 'seems to punch above its weight' is not a general conclusion about model capability. Additionally, models use different Ollama quantizations (Q4_0, Q4_K_M, FP16), and the 'best ground truth per metric' selection makes scores non-comparable across models. These tables should be presented as results for the current prompt set and configuration, not as general model rankings, or the confounds should be controlled.
  4. [§1 and Conclusion] The 'first voice-controlled beamline system' claim needs a precise baseline. The paper cites a prior VISION prototype [30] and other LLM-based facility systems, so it is unclear what 'first' excludes: previous voice transcription? LLM-generated control code? End-to-end voice to hardware control at a synchrotron? Please define the novelty claim explicitly relative to [30] and the cited related work, or remove the 'first' wording if the comparison cannot be established.
minor comments (5)
  1. [Title/Abstract] The title as typeset contains extra spacing ('M ODULAR AI A SSISTANT'); please correct this in the camera-ready version.
  2. [§1] In the sentence 'Advances in AI and machine learning (ML) should be be utilized for', the word 'be' is duplicated; please fix the typo.
  3. [§3.1, Table 2 note] The table note says 'Phi-3.5 showed a standard deviation of ±1.44 on only the ONE_WORD prompt type' in the context of F1 scores, but F1 is bounded by [0,1] and the value 1.44 appears in the execution-time column. The note likely refers to the time standard deviation; please clarify.
  4. [§3.1] The sentence 'Cog classification for identifying a suitable specialized cog is important as, depending on the tasks, some cogs must be able to reproduce the same outcome, while some cogs can be allowed more freedom' is grammatically awkward; please rephrase for clarity.
  5. [§2.2 and §3.2] The keystroke-injection mechanism is described at a high level but the paper does not explain how synchronization with the IPython terminal state is managed or how errors in execution are detected. A more detailed description would help others assess the robustness of the integration and reproduce the demo.

Circularity Check

2 steps flagged · score 6.0 of 10

Operator-cog accuracy is prompt-fitted and ground-truth-biased, so the quantitative code-correctness claim is partially circular by construction.

  1. fitted input called prediction [Section 3, 'Cog Performance' intro; Section 3.1 'Operator Cog' (Table 3); Supplementary S1.2.5 'Dataset']
    "These datasets were not directly included in system prompts, but the evaluation results were used in the development process to refine the system prompts for the specific LLM used."

    The reported Operator accuracy in Table 3 is computed on the same dataset whose evaluation results were used to iteratively refine the system prompt. The paper itself says in S1.2.5 that this small dataset 'should not be viewed as a testing dataset since it was used to guide the prompts.' Consequently, the high accuracy numbers, especially for Qwen2.5-coder, measure how well the prompt was fitted to those examples rather than an independent predictive result. The 'prediction' of strong performance is therefore forced by the fitting procedure, not by an out-of-sample test.

  2. fitted input called prediction [Section 3.1 'Operator Cog'; Supplementary S1.2.5 'Dataset' and 'Metrics']
    "The ground truth solutions were partially constructed from outputs of Qwen2.5-Coder and Claude-3.5-Sonnet, which may bias the metrics in their favor. ... If there were multiple ground truth answers, the best performing ground truth for each metric is chosen individually when calculating the averages."

    The ground-truth programs used to score the models were partially generated by the same models being scored, and when several references exist the paper selects, per metric, the reference that gives the best score. This makes the reported CodeBLEU, Levenshtein, and exact-match numbers favorable to those models by construction. The admitted bias means the quantitative evidence for the central 'correct beamline control code' claim is self-referential rather than an independent measurement of code correctness.

full rationale

The central derivation chain is the claim that VISION converts natural language and speech into correct beamline-control code. The one executed beamline demonstration is an independent, non-circular existence proof, and the Whisper fine-tuning and classifier evaluations use separate held-out test templates. However, the quantitative support for the Operator cog, which is load-bearing for the code-correctness claim, is partially circular in two ways: the evaluation dataset was used to refine the prompts, and the ground-truth solutions were partly generated by the evaluated models with per-metric best-reference selection. The paper explicitly says the dataset 'should not be viewed as a testing dataset.' The absence of functional-equivalence mocking (S1.2.5, Section 4) is a correctness limitation rather than a circularity, and the admitted keystroke-injection fragility is a robustness issue, not a logical one. The real demo prevents total collapse, but the headline quantitative accuracy is partly self-referential, giving a score of 6.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The system's central claims rest on the representativeness of small hand-built evaluation sets, the validity of synthetic TTS audio as a proxy for real human speech, the reliability of keystroke injection as a control channel, and the stability of third-party LLM behavior. The paper explicitly acknowledges the evaluation-data issue in S1.2.5 and the keystroke robustness issue in Section 4.

assumptions (4)
  • domain assumption The hand-built evaluation datasets (99, 120, 37, and 6 cases) are representative of real beamline user commands.
    These datasets are used to claim classifier, operator, and analyst performance, yet the same datasets guided prompt refinement (Section 3, S1.2.5), so they are not independent test sets.
  • domain assumption Synthetic TTS audio behaves like real human speech for Whisper fine-tuning and evaluation.
    The fine-tuning and test audio are generated with Google Translate TTS (S1.1); the paper claims general applicability to real-world voices without a human-voice test set.
  • domain assumption Keystroke injection into an interactive IPython Bluesky terminal is a sufficiently reliable integration channel.
    The voice demo relies on this mechanism, and Section 4 states that the implementation is not robust and needs enhanced integration.
  • domain assumption Third-party LLMs and ASR models (Qwen, GPT-4o, Claude, Whisper) behave consistently with their documented versions at temperature 0.
    The system performance is tied to specific proprietary or hosted models; any model update could change behavior, and version pinning is only partially specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities." pith.science (2026). https://pith.science/paper/BWX2NB26

@misc{pith2026241218161,
  author       = {Pith},
  title        = {Pith review of: VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWX2NB26}},
  note         = {Machine review of arXiv:2412.18161}
}
read the original abstract

Scientific user facilities, such as synchrotron beamlines, are equipped with a wide array of hardware and software tools that require a codebase for human-computer-interaction. This often necessitates developers to be involved to establish connection between users/researchers and the complex instrumentation. The advent of generative AI presents an opportunity to bridge this knowledge gap, enabling seamless communication and efficient experimental workflows. Here we present a modular architecture for the Virtual Scientific Companion (VISION) by assembling multiple AI-enabled cognitive blocks that each scaffolds large language models (LLMs) for a specialized task. With VISION, we performed LLM-based operation on the beamline workstation with low latency and demonstrated the first voice-controlled experiment at an X-ray scattering beamline. The modular and scalable architecture allows for easy adaptation to new instrument and capabilities. Development on natural language-based scientific experimentation is a building block for an impending future where a science exocortex -- a synthetic extension to the cognition of scientists -- may radically transform scientific practice and discovery.

Figures

Figures reproduced from arXiv: 2412.18161 by the authors.

Figure 1
Figure 1. Virtual scientific companion (VISION) aims to lead the NL-controlled scientific expedition with joint [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of the VISION architecture, with each cog described in Table 1. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Word Error Rate (WER) performance comparison of fine-tuned Whisper models. The plot shows the WER [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Confusion matrix for Athene-v2-Agent with system prompt type–ONE_WORD, with failed cases given in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: VISION deployment at NSLS-II 11-BM CMS: GUI was launched at the beamline workstation, with backend [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.

Reference graph

Works this paper leans on

76 extracted references · 60 canonical work pages · cited by 1 Pith paper

  1. [30]

    Virtual scientific companion for synchrotron beamlines: A prototype

    Daniel Potemkin, Carlos Soto, Ruipeng Li, Kevin Yager, and Esther Tsai. Virtual scientific companion for synchrotron beamlines: A prototype. arxiv, 2023

    work page 2023

  2. [1]

    Methods and Applications of Autonomous Experimentation

    Marcus Noack and Daniela Ushizima. Methods and Applications of Autonomous Experimentation. CRC Press, 2024

    work page 2024

  3. [2]

    Scientific exploration with expert knowledge (seek) in autonomous scanning probe microscopy with active learning

    Utkarsh Pratiush, Hiroshi Funakubo, Rama Vasudevan, Sergei V Kalinin, and Yongtao Liu. Scientific exploration with expert knowledge (seek) in autonomous scanning probe microscopy with active learning. Digital Discovery, 2025

    work page 2025

  4. [3]

    Self-driving laboratory for polymer electronics

    Aikaterini Vriza, Henry Chan, and Jie Xu. Self-driving laboratory for polymer electronics. Chemistry of Materials, 35(8):3046–3056, 2023. 11

    work page 2023

  5. [4]

    Autonomous discovery of emergent morphologies in directed self-assembly of block copolymer blends

    Gregory S Doerk, Aaron Stein, Suwon Bae, Marcus M Noack, Masafumi Fukuto, and Kevin G Yager. Autonomous discovery of emergent morphologies in directed self-assembly of block copolymer blends. Science Advances, 9(2):eadd3687, 2023

    work page 2023

  6. [5]

    Autonomous x-ray scattering

    Kevin G Yager, Pawel W Majewski, Marcus M Noack, and Masafumi Fukuto. Autonomous x-ray scattering. Nanotechnology, 34(32):322001, 2023

    work page 2023

  7. [6]

    Improved understanding of carbon nanotube growth via autonomous jump regression targeting of catalyst activity

    Robert Waelder, Chiwoo Park, Arthur Sloan, Jennifer Carpena-Núñez, Joshua Yoho, Stephane Gorsse, Rahul Rao, and Benji Maruyama. Improved understanding of carbon nanotube growth via autonomous jump regression targeting of catalyst activity. Carbon, 228:119356, 2024

    work page 2024

  8. [7]

    An autonomous laboratory for the accelerated synthesis of novel materials

    Nathan J Szymanski, Bernardus Rendy, Yuxing Fei, Rishi E Kumar, Tanjin He, David Milsted, Matthew J McDermott, Max Gallant, Ekin Dogus Cubuk, Amil Merchant, Haegyeom Kim, Anubhav Jain, Christopher J Bartel, Kristin Persson, Yan Zeng, and Gerbrand Ceder. An autonomous laboratory for the accelerated synthesis of novel materials. Nature, 624(7990):86–91, Dec...

    work page 2023

Show all 76 references
  1. [8]

    A robotic platform for flow synthesis of organic compounds informed by ai planning

    Connor W Coley, Dale A Thomas III, Justin AM Lummiss, Jonathan N Jaworski, Christopher P Breen, Victor Schultz, Travis Hart, Joshua S Fishman, Luke Rogers, Hanyu Gao, et al. A robotic platform for flow synthesis of organic compounds informed by ai planning. Science, 365(6453):...

    2019

  2. [9]

    An introduction to synchrotron radiation: techniques and applications

    Philip Willmott. An introduction to synchrotron radiation: techniques and applications. John Wiley & Sons, 2019

    2019

  3. [10]

    X-ray scattering for soft matter research at nsls-ii

    Andi Barbour, Yong Q Cai, Andrei Fluerasu, Guillaume Freychet, Masafumi Fukuto, Oleg Gang, Eliot Gann, Ricarda Laasch, Ruipeng Li, Benjamin M Ocko, et al. X-ray scattering for soft matter research at nsls-ii. Synchrotron radiation news, 36(2):24–30, 2023

    2023

  4. [11]

    Structural, electrochemical, and (de) lithiation mechanism investigation of cation-disordered rocksalt and spinel hybrid nanomaterials in lithium-ion batteries

    Arun Kingan, Cynthia Huang, Zachary R Mansley, Ryan C Hill, Zhongling Wang, David C Bock, Lei Wang, Lu Ma, Steven N Ehrlich, Jianming Bai, et al. Structural, electrochemical, and (de) lithiation mechanism investigation of cation-disordered rocksalt and spinel hybrid nanomateri...

    2024

  5. [12]

    Concurrent inhibition of oncogenic and wild-type RAS-GTP for cancer therapy

    Matthew Holderfield, Bianca J Lee, Jingjing Jiang, Aidan Tomlinson, Kyle J Seamon, Alessia Mira, Enrico Patrucco, Grace Goodhart, Julien Dilly, Yevgeniy Gindin, et al. Concurrent inhibition of oncogenic and wild-type RAS-GTP for cancer therapy. Nature, pages 1–8, 2024

    2024

  6. [13]

    Mapping the 3d orientation of nanocrystals and nanostructures in human bone: Indications of novel structural features

    Tilman A Grünewald, Marianne Liebi, Nina K Wittig, Andreas Johannes, Tanja Sikjaer, Lars Rejnmark, Zirui Gao, Martin Rosenthal, Manuel Guizar-Sicairos, Henrik Birkedal, et al. Mapping the 3d orientation of nanocrystals and nanostructures in human bone: Indications of novel str...

    2020

  7. [14]

    Three-dimensional imaging of integrated circuits with macro-to nanoscale zoom

    Mirko Holler, Michal Odstrcil, Manuel Guizar-Sicairos, Maxime Lebugle, Elisabeth Müller, Simone Finizio, Gemma Tinti, Christian David, Joshua Zusman, Walter Unglaub, et al. Three-dimensional imaging of integrated circuits with macro-to nanoscale zoom. Nature Electronics, 2(10)...

    2019

  8. [15]

    Synchrotron x-ray analytical techniques for studying materials electrochem- istry in rechargeable batteries

    Feng Lin, Yijin Liu, Xiqian Yu, Lei Cheng, Andrej Singer, Oleg G Shpyrko, Huolin L Xin, Nobumichi Tamura, Chixia Tian, Tsu-Chien Weng, et al. Synchrotron x-ray analytical techniques for studying materials electrochem- istry in rechargeable batteries. Chemical reviews, 117(21):...

    2017

  9. [16]

    Encyclopedia of human computer interaction

    Claude Ghaoui. Encyclopedia of human computer interaction. IGI global, 2005

    2005

  10. [17]

    The virtual lab: AI agents design new SARS-CoV-2 nanobodies with experimental validation

    Kyle Swanson, Wesley Wu, Nash L Bulaong, John E Pak, and James Zou. The virtual lab: AI agents design new SARS-CoV-2 nanobodies with experimental validation. bioRxiv, page 2024.11.11.623004, November 2024

    2024

  11. [18]

    From intention to implementation: Automating biomedical research via llms

    Yi Luo, Linghang Shi, Yihao Li, Aobo Zhuang, Yeyun Gong, Ling Liu, and Lin Chen. From intention to implementation: Automating biomedical research via llms. arXiv preprint arXiv:2412.09429, 2024

    2024 arXiv

  12. [19]

    The application of large language models in medicine: A scoping review

    Xiangbin Meng, Xiangyu Yan, Kuo Zhang, Da Liu, Xiaojuan Cui, Yaodong Yang, Muhan Zhang, Chunxia Cao, Jingjia Wang, Xuliang Wang, et al. The application of large language models in medicine: A scoping review. Iscience, 27(5), 2024

    2024

  13. [20]

    Autonomous chemical research with large language models

    Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570–578, December 2023

    2023

  14. [21]

    ChemCrow: Augmenting large-language models with chemistry tools

    Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. ChemCrow: Augmenting large-language models with chemistry tools. arXiv [physics.chem-ph], April 2023

    2023

  15. [22]

    Leveraging large language models for predictive chemistry

    Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrero, and Berend Smit. Leveraging large language models for predictive chemistry. Nature Machine Intelligence, 6(2):161–169, 2024

    2024

  16. [23]

    Assessment of chemistry knowledge in large language models that generate code

    Andrew D White, Glen M Hocky, Heta A Gandhi, Mehrad Ansari, Sam Cox, Geemi P Wellawatte, Subarna Sasmal, Ziyue Yang, Kangxin Liu, Yuvraj Singh, et al. Assessment of chemistry knowledge in large language models that generate code. Digital Discovery, 2(2):368–376, 2023. 12

    2023

  17. [24]

    Chatgpt chemistry assistant for text mining and the prediction of mof synthesis

    Zhiling Zheng, Oufan Zhang, Christian Borgs, Jennifer T Chayes, and Omar M Yaghi. Chatgpt chemistry assistant for text mining and the prediction of mof synthesis. Journal of the American Chemical Society, 145(32):18048– 18062, 2023

    2023

  18. [25]

    Large language model-guided prediction toward quantum materials synthesis

    Ryotaro Okabe, Zack West, Abhijatmedhi Chotrattanapituk, Mouyang Cheng, Denisse Córdova Carrizales, Weiwei Xie, Robert J Cava, and Mingda Li. Large language model-guided prediction toward quantum materials synthesis. arXiv preprint arXiv:2410.20976, 2024

    arXiv

  19. [26]

    Llmatdesign: Autonomous materials discovery with large language models

    Shuyi Jia, Chao Zhang, and Victor Fung. Llmatdesign: Autonomous materials discovery with large language models. arXiv preprint arXiv:2406.13163, 2024

    2024 arXiv

  20. [27]

    LLaMP: Large language model made powerful for high-fidelity materials knowledge retrieval and distillation

    Yuan Chiang, Elvis Hsieh, Chia-Hong Chou, and Janosh Riebesell. LLaMP: Large language model made powerful for high-fidelity materials knowledge retrieval and distillation. arXiv preprint arXiv:2401.17244, 2024

    2024 arXiv

  21. [28]

    Large language models design sequence-defined macromolecules via evolutionary optimization

    Wesley F Reinhart and Antonia Statt. Large language models design sequence-defined macromolecules via evolutionary optimization. npj Computational Materials, 10(1):262, 2024

    2024

  22. [29]

    Rapid and automated alloy design with graph neural network-powered llm-driven multi-agent systems

    Alireza Ghafarollahi and Markus J Buehler. Rapid and automated alloy design with graph neural network-powered llm-driven multi-agent systems. arXiv preprint arXiv:2410.13768, 2024

    2024 arXiv

  23. [31]

    Opportunities for retrieval and tool augmented large language models in scientific facilities

    Michael H Prince, Henry Chan, Aikaterini Vriza, Tao Zhou, Varuni K Sastry, Yanqi Luo, Matthew T Dearing, Ross J Harder, Rama K Vasudevan, and Mathew J Cherukara. Opportunities for retrieval and tool augmented large language models in scientific facilities. npj Computational Ma...

    2024

  24. [32]

    Esac (eq-sans assisting chatbot): Application of large language models and retrieval-augmented generation for enhanced user experience at eq-sans

    Changwoo Do, Gergely Nagy, and William T Heller. Esac (eq-sans assisting chatbot): Application of large language models and retrieval-augmented generation for enhanced user experience at eq-sans. arXiv preprint arXiv:2407.19075, 2024

    2024 arXiv

  25. [33]

    Synergizing human expertise and ai efficiency with language model for microscopy operation and automated experiment design

    Yongtao Liu, Marti Checa, and Rama K Vasudevan. Synergizing human expertise and ai efficiency with language model for microscopy operation and automated experiment design. Machine Learning: Science and Technology, 2024

    2024

  26. [34]

    Paperqa: Retrieval-augmented generative agent for scientific research

    Jakub Lála, Odhran O’Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G Rodriques, and Andrew D White. Paperqa: Retrieval-augmented generative agent for scientific research. arXiv preprint arXiv:2312.07559, 2023

    2023 arXiv

  27. [35]

    Kevin G. Yager. Domain-specific chatbots for science using embeddings. Digital Discovery, 2:1850–1861, 2023

    2023

  28. [36]

    Language agents achieve superhuman synthesis of scientific knowledge

    Michael D Skarlinski, Sam Cox, Jon M Laurent, James D Braza, Michaela Hinks, Michael J Hammerling, Manvitha Ponnapati, Samuel G Rodriques, and Andrew D White. Language agents achieve superhuman synthesis of scientific knowledge. arXiv preprint arXiv:2409.13740, 2024

    2024 arXiv

  29. [37]

    ORGANA: A robotic assistant for automated chemistry experimentation and characterization

    Kourosh Darvish, Marta Skreta, Yuchi Zhao, Naruki Yoshikawa, Sagnik Som, Miroslav Bogdanovic, Yang Cao, Han Hao, Haoping Xu, Alán Aspuru-Guzik, Animesh Garg, and Florian Shkurti. ORGANA: A robotic assistant for automated chemistry experimentation and characterization. arXiv [c...

    2024

  30. [38]

    Accelerating computational materials discovery with artificial intelligence and cloud high-performance computing: from large-scale screening to experimental validation

    Chi Chen, Dan Thien Nguyen, Shannon J Lee, Nathan A Baker, Ajay S Karakoti, Linda Lauw, Craig Owen, Karl T Mueller, Brian A Bilodeau, Vijayakumar Murugesan, and Matthias Troyer. Accelerating computational materials discovery with artificial intelligence and cloud high-performa...

    2024

  31. [39]

    Specifications: The missing link to making the development of llm systems an engineering discipline

    Ion Stoica, Matei Zaharia, Joseph Gonzalez, Ken Goldberg, Hao Zhang, Anastasios Angelopoulos, Shishir G Patil, Lingjiao Chen, Wei-Lin Chiang, and Jared Q Davis. Specifications: The missing link to making the development of llm systems an engineering discipline. arXiv preprint ...

    2024 arXiv

  32. [40]

    Kevin G. Yager. Towards a Science Exocortex. Digital Discovery, 3:1933–1957, 2024

    1933

  33. [41]

    Bluesky’s Ahead: A Multi-Facility Collaboration for an a la Carte Software Project for Data Acquisition and Management

    Daniel Allan, Thomas Caswell, Stuart Campbell, and Maksim Rakitin. Bluesky’s Ahead: A Multi-Facility Collaboration for an a la Carte Software Project for Data Acquisition and Management. Synchrotron Radiation News, 32(3):19–22, 2019

    2019

  34. [42]

    https://github.com/CFN-softbio/SciAnalysis [Accessed: 2024-12]

    SciAnalysis, 2024. https://github.com/CFN-softbio/SciAnalysis [Accessed: 2024-12]

    2024

  35. [43]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  36. [44]

    The claude 3 model family: Opus, sonnet, haiku

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1, 2024. 13

    2024

  37. [45]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  38. [46]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    2024 arXiv

  39. [47]

    Qwen2.5- coder technical report, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024

  40. [48]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  41. [49]

    GPT-4 technical report, 2023

    OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report, 2023

    2023

  42. [50]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  43. [51]

    https://github.com/lbl-camera/tsuchinoko [Accessed: 2024-12]

    Tsuchinoko, 2024. https://github.com/lbl-camera/tsuchinoko [Accessed: 2024-12]

    2024

  44. [52]

    Whisper: Automatic speech recognition

    OpenAI. Whisper: Automatic speech recognition. https://github.com/openai/whisper, 2022. Accessed: 2024-12-23

    2022

  45. [53]

    https://github.com/Xi-CAM/Xi-cam [Accessed: 2024-12]

    Xi-cam, 2024. https://github.com/Xi-CAM/Xi-cam [Accessed: 2024-12]

    2024

  46. [54]

    Xi-cam: a versatile interface for data visualization and analysis

    Ronald J Pandolfi, Daniel B Allan, Elke Arenholz, Luis Barroso-Luque, Stuart I Campbell, Thomas A Caswell, Austin Blair, Francesco De Carlo, Sean Fackler, Amanda P Fournier, et al. Xi-cam: a versatile interface for data visualization and analysis. Journal of synchrotron radiat...

    2018

  47. [55]

    Codebleu: a method for automatic evaluation of code synthesis, 2020

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis, 2020

    2020

  48. [56]

    Lora-whisper: Parameter- efficient and extensible multilingual asr

    Zheshu Song, Jianheng Zhuo, Yifan Yang, Ziyang Ma, Shixiong Zhang, and Xie Chen. Lora-whisper: Parameter- efficient and extensible multilingual asr. arXiv preprint arXiv:2406.06619, 2024

    2024 arXiv

  49. [57]

    Towards rehearsal- free multilingual asr: A lora-based case study on whisper

    Tianyi Xu, Kaixun Huang, Pengcheng Guo, Yu Zhou, Longtao Huang, Hui Xue, and Lei Xie. Towards rehearsal- free multilingual asr: A lora-based case study on whisper. arXiv preprint arXiv:2408.10680, 2024

    2024 arXiv

  50. [58]

    Hüseyin Polat, Alp Kaan Turan, Cemal Koçak, and Hasan Basri Ula¸ s. Implementation of a whisper architecture- based turkish automatic speech recognition (asr) system and evaluation of the effect of fine-tuning with a low-rank adaptation (lora) adapter on its performance. Elect...

    2024

  51. [59]

    When whisper meets tts: Domain adaptation using only synthetic speech data

    Juan Camilo Vásquez-Correa, Haritz Arzelus, Juan M Martin-Doñas, Joaquin Arellano, Ander Gonzalez-Docasal, and Aitor Álvarez. When whisper meets tts: Domain adaptation using only synthetic speech data. In International Conference on Text, Speech, and Dialogue, pages 226–238. S...

    2023

  52. [60]

    The term [X] is used in

    Generic Sentence Templates: Approximately 50 sentence templates were crafted to provide diverse syntactic contexts for the target jargon. These templates were adaptable to any specialized term, with examples like "The term [X] is used in..." or "[X] plays a critical role in......

  53. [61]

    By spelling jargon terms phonetically, the generated audio closely mimicked the pronunciation used by domain specialists, ensuring alignment with real-world usage

    Synthetic Audio Generation: The modified sentences were fed into Google Translate’s Text-to-Speech (TTS) API to generate synthetic audio clips. By spelling jargon terms phonetically, the generated audio closely mimicked the pronunciation used by domain specialists, ensuring al...

  54. [62]

    For a vocabulary of n new words and k sentences per word, the dataset contained n × k text-audio pairs

    Dataset Assembly: For each new domain-specific term or word, approximately 50 text-audio pairs were used for fine-tuning and 30 text-audio pairs for testing. For a vocabulary of n new words and k sentences per word, the dataset contained n × k text-audio pairs. This efficient ...

  55. [63]

    LoRA’s ability to minimize computational overhead while effectively adapting models has been demonstrated in prior work [56, 57, 58]

    Fine-Tuning via LoRA: The assembled dataset was used to fine-tune Whisper using LoRA, a parameter- efficient approach that integrates new knowledge without requiring full model retraining. LoRA’s ability to minimize computational overhead while effectively adapting models has ...

  56. [66]

    e x a m p l e _ i n p u t s

    Strictly provide the output only in the fol lo wi ng format : 16 - Output only one word i n d i c a t i n g the class : - Op - Ana - Notebook - gpcam - xicam Use the f ol low in g examples to learn about how to generate your outputs : ’’’ ) Box S2: Examples JSON [ { " e x a m ...

  57. [70]

    Always output one word c o r r e s p o n d i n g to the i d e n t i f i e d class . Use the f ol low in g examples to learn about how to generate your outputs : Examples : Example 1: User Prompt : Measure the sample for 5 seconds and increase the t e m p e r a t u r e by 10 de...

    1989

  58. [71]

    CodeBLEU Score [55]: A composite metric that combines four components to evaluate code similarity: • N-gram match score: Standard BLEU score for surface-level similarity • Weighted n-gram match score: Modified BLEU score with weighted keywords • Syntax match score: Similarity ...

  59. [72]

    Levenshtein Distance (LD): The minimum number of single-character edits (insertions, deletions, or substitu- tions) required to change one string into another: LD(s1, s2) =LevenshteinDistance(s1, s2) (2) Calculated by using the levenshtein package6

  60. [73]

    Normalized Levenshtein Distance (NLD): A normalized version of the Levenshtein distance that scales the result to a value between 0 and 1: NLD(s1, s2) = LD(s1, s2) max(|s1|, |s2|) (3)

  61. [74]

    Exact Match Accuracy: Binary metric for perfect string matches: Accuracy = number of exact matches with at least one ground truth example total dataset entries (4) For cases with multiple valid ground truth implementations, we independently select the best reference implementa...

    2008

  62. [75]

    different iterator variable names: theta or th instead of angle

  63. [76]

    spacing or no spacing in loop arithmetic: 1.5 + 0.02 or 1.5+0.02

  64. [77]

    Scan incident angle from 0.05 to 1.5 degree (with 0.02 step), exposure time 0.5s

    explicit or implicit parameter naming: (exposure_time=0.5) or (0.5) Therefore, these differences are purely cosmetic and would result in the same measurements being done at the beamline. Table S5 additionally shows metrics in a where there is functional equivalence but no exac...

  65. [78]

    D et erm in e the Command Type : - Operator ( Op ) : - Any task that involves hardware control - Analyst ( Ana ) : - Data analysis tasks - Notebook : - Logging tasks , or writing tasks , or general o b s e r v a t i o n s - gpCAM ( gpcam ) : - Only predict gpcam if you see it ...

  66. [79]

    Analyze the user prompt and d et er mi ne whether the command is an Op , Ana , Notebook , gpcam , or xicam

  67. [80]

    Strictly provide the output only in the fol lo wi ng format : - Output only one word i n d i c a t i n g the class : - Op - Ana - Notebook - gpcam - xicam

  68. [81]

    Measure 5 seconds every minute , for 10 minutes

    Always output one word c o r r e s p o n d i n g to the i d e n t i f i e d class . Use the f ol low in g examples to learn about how to generate your outputs : Examples : Example 1: 25 User Prompt : Measure the sample for 5 seconds and increase the t e m p e r a t u r e by 10...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.