Agentic Language-to-Objective Synthesis for Optofluidic Assembly
Pith reviewed 2026-06-29 16:53 UTC · model grok-4.3
The pith
An LLM pipeline turns spoken commands into differentiable objectives that assemble microparticles via laser-driven fluid flows.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Speak-to-Objective uses a perceive-compose-propose-act-report-and-learn loop in which a conditioned LLM generates composite objective functions from geometry, spacing, and assignment terms; these functions are executable by SLSQP and, when paired with laser-induced thermoviscous flows, produce natural-language-programmable microscale particle assemblies in a microfluidic environment that remain robust to incomplete information and external disturbances.
What carries the argument
The Speak-to-Objective pipeline, which composes geometry, spacing, and assignment/topology terms into composite differentiable objectives that interface user intent with an inverse solver and physical actuation.
If this is right
- Objectives generated by the pipeline can assemble particle patterns from partial traces without requiring complete initial specifications.
- The same objectives enable recovery of the target pattern after external perturbations during assembly.
- Explicit objectives produced by the pipeline support precise placement of individual particles in addition to descriptive patterning.
- Because the objectives are actuator-agnostic, the same language-to-objective translation can be reused with different physical actuation methods.
- The closed loop that incorporates user feedback allows the system to refine objectives over repeated interactions.
Where Pith is reading between the lines
- The separation of intent specification from actuation details could allow the same pipeline to control assembly in other light-based or robotic manufacturing settings.
- Repeated user feedback in the report-and-learn step may gradually improve the LLM's ability to produce objectives that require less solver iteration.
- If the objective composition generalizes, the approach could reduce the expert knowledge needed to program new microscale assembly tasks.
Load-bearing premise
A conditioned large language model can reliably output fully differentiable objective functions that the SLSQP solver can execute to produce successful assemblies on the optofluidic platform without any post-hoc manual adjustments.
What would settle it
Run the pipeline on several distinct natural-language commands, feed the generated objectives directly to the SLSQP solver, and check whether the resulting particle patterns on the experimental platform match the commanded geometries and placements with no manual editing of the objectives.
Figures
read the original abstract
Light-based advanced manufacturing increasingly requires programmable, closed-loop tools that translate human design intent into executable operations at small length scales. Yet a key bottleneck persists across robotic and manufacturing modalities: turning user intent into machine-readable objectives that are reliably executable. While micro-robotics offers versatile manipulation via optical actuation of fluids, mathematically tractable goal specification remains manual and hard to reuse. Here, we introduce Speak-to-Objective, a modular agentic pipeline that uses a conditioned Large Language Model (LLM) to translate spoken or written commands into fully differentiable objective functions for assembling microparticles in a constraint-aware inverse solver (SLSQP) and on an experimental optofluidic platform. The approach employs a compact loop - perceive -> compose -> propose -> act -> report & learn - that treats the objective as the interface between intent and actuation, separating what to assemble or pattern from how to actuate, while learning from user feedback. The pipeline composes geometry, spacing, and assignment/topology terms to generate robust descriptive objectives that assemble from partial traces and recover after perturbations, as well as explicit objectives for precise placement, all in an actuator-agnostic fashion. Using laser-induced thermoviscous flows as the physical actuation modality, we demonstrate natural-language-programmable, light-based microscale assembly of particle patterns in a microfluidic environment. Beyond its immediate impact on programmable microassembly, and using laser-induced optofluidic actuation as a reduced-complexity experimental platform, our work points toward self-driving, AI-assisted optical manufacturing platforms in which natural language, differentiable objectives, and laser-based actuation are coupled into a reusable digital workflow.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Speak-to-Objective, a modular agentic pipeline that uses a conditioned large language model to translate spoken or written commands into fully differentiable objective functions. These functions are optimized via a constraint-aware SLSQP inverse solver for microparticle assembly and validated experimentally on an optofluidic platform using laser-induced thermoviscous flows. The pipeline employs a perceive-compose-propose-act-report&learn loop that separates user intent from actuation; it composes geometry, spacing, and assignment/topology terms to produce objectives claimed to support assembly from partial traces, recovery after perturbations, precise placement, and actuator-agnostic operation, with a demonstration of natural-language-programmable microscale assembly.
Significance. If the claims hold with supporting quantitative evidence, the work could advance programmable micro-manufacturing by providing a reusable natural-language interface to differentiable objectives and physical actuation. The explicit separation of intent from actuation, combined with LLM-based composition and closed-loop feedback, offers a pathway toward self-driving optical manufacturing platforms. The reduced-complexity optofluidic testbed supplies a concrete experimental anchor, though the absence of metrics in the abstract limits evaluation of the robustness assertions.
major comments (2)
- [Abstract] Abstract: The central claims that the composed objectives are 'robust' and enable 'assembly from partial traces and recover after perturbations' are presented without any quantitative results, success rates, error bars, validation data, or statistical analysis, which is load-bearing for the assertion that an LLM can reliably produce fully executable objectives for the SLSQP solver without post-hoc manual adjustments.
- [Abstract] Abstract: No experimental details are supplied on trial counts, perturbation types, recovery metrics, or comparisons to manual objective specification, undermining assessment of the actuator-agnostic and perturbation-recovery claims that form the core of the pipeline's contribution.
minor comments (1)
- [Abstract] The abstract would benefit from inclusion of at least one key quantitative performance metric to ground the robustness claims.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on the abstract. We agree that quantitative support for the robustness and recovery claims is needed in the abstract itself to strengthen the presentation. We will revise the abstract accordingly while preserving the manuscript's core contributions. Point-by-point responses are below.
read point-by-point responses
-
Referee: [Abstract] Abstract: The central claims that the composed objectives are 'robust' and enable 'assembly from partial traces and recover after perturbations' are presented without any quantitative results, success rates, error bars, validation data, or statistical analysis, which is load-bearing for the assertion that an LLM can reliably produce fully executable objectives for the SLSQP solver without post-hoc manual adjustments.
Authors: We agree that the abstract would be strengthened by including quantitative metrics. The full manuscript reports experimental results with trial counts, success rates, and recovery metrics in the results section. We will revise the abstract to add a concise summary of these quantitative findings (e.g., success rates for partial-trace assembly and perturbation recovery) to better support the claims. revision: yes
-
Referee: [Abstract] Abstract: No experimental details are supplied on trial counts, perturbation types, recovery metrics, or comparisons to manual objective specification, undermining assessment of the actuator-agnostic and perturbation-recovery claims that form the core of the pipeline's contribution.
Authors: We acknowledge that the abstract lacks these experimental details. We will update the abstract to incorporate key information on trial counts, perturbation types tested, and recovery metrics drawn from the experimental validation in the manuscript. This revision will address the concern without requiring changes to the underlying results or methods. revision: yes
Circularity Check
No significant circularity detected
full rationale
The paper describes an agentic LLM-based pipeline (Speak-to-Objective) that composes geometry/spacing/assignment terms into differentiable objectives for SLSQP optimization and experimental optofluidic actuation. No equations, fitted parameters, or derivation steps are presented that reduce to self-definition, fitted-input-as-prediction, or self-citation load-bearing. The central claim rests on external experimental validation with thermoviscous flows rather than internal reduction to inputs. The pipeline is framed as separating intent from actuation with reported robustness to partial traces and perturbations, making the work self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Institute of Biological and Chemical Systems, Karlsruhe Institute of Technology, Germany
-
[2]
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK
-
[3]
Department of Mathematics, Imperial College London, UK
-
[4]
action-at-a-distance
Institute of Anthropomatics and Robotics (IAR), Karlsruhe Institute of Technology, Germany *e-mail: moritz.kreysing@kit.edu Abstract Light-based advanced manufacturing increasingly requires programmable, closed- loop tools that translate human design intent into executable operations at small length scales. Yet a key bottleneck persists across robotic and...
2082
-
[5]
M.; Bjorkholm, J
Ashkin, A.; Dziedzic, J. M.; Bjorkholm, J. E.; Chu, S., Observation of a single-beam gradient force optical trap for dielectric particles. Opt. Lett. 1986, 11 (5), 288-290
1986
-
[6]
G., A revolution in optical manipulation
Grier, D. G., A revolution in optical manipulation. Nature 2003, 424 (6950), 810-816
2003
-
[7]
G., Optical forces arising from phase gradients
Roichman, Y.; Sun, B.; Roichman, Y.; Amato-Grill, J.; Grier, D. G., Optical forces arising from phase gradients. Phys. Rev. Lett. 2008, 100 (1), 013602
2008
-
[8]
T., Optical pulling force
Chen, J.; Ng, J.; Lin, Z.; Chan, C. T., Optical pulling force. Nat. Photon. 2011, 5 (9), 531-534
2011
-
[9]
M.; Jones, P
Maragò, O. M.; Jones, P. H.; Gucciardi, P. G.; Volpe, G.; Ferrari, A. C., Optical trapping and manipulation of nanostructures. Nat. Nanotechnol. 2013, 8 (11), 807-819
2013
-
[10]
Light Sci
Gao, D.; Ding, W.; Nieto-Vesperinas, M.; Ding, X.; Rahman, M.; Zhang, T.; Lim, C.; Qiu, C.-W., Optical manipulation from the microscale to the nanoscale: fundamentals, advances and prospects. Light Sci. Appl. 2017, 6 (9), e17039-e17039
2017
-
[11]
Li, X.; Dan, D.; Zavatski, S.; Gao, W.; Zhang, Q.; Zhou, Y.; Qian, J.; Yang, Y.; Yu, X.; Yan, S.; Xu, X.; Martin, O. J. F.; Yao, B., Optical tweeze-sectioning microscopy for 3D imaging and manipulation of suspended cells. Sci. Adv. 2025, 11 (27), eadx3900
2025
-
[12]
C.; Pané, S.; Zhang, L.; Pumera, M., Magnetically Driven Micro and Nanorobots
Zhou, H.; Mayorga-Martinez, C. C.; Pané, S.; Zhang, L.; Pumera, M., Magnetically Driven Micro and Nanorobots. Chemical Reviews 2021, 121 (8), 4999-5041
2021
-
[13]
J.; Choi, H., Fabrication and Characterization of Magnetic Microrobots for Three- Dimensional Cell Culture and Targeted Transportation
Kim, S.; Qiu, F.; Kim, S.; Ghanbari, A.; Moon, C.; Zhang, L.; Nelson, B. J.; Choi, H., Fabrication and Characterization of Magnetic Microrobots for Three- Dimensional Cell Culture and Targeted Transportation. Advanced Materials 2013, 25 (41), 5863-5868
2013
-
[14]
G.; Gibson, G
Būtaitė, U. G.; Gibson, G. M.; Ho, Y.-L. D.; Taverne, M.; Taylor, J. M.; Phillips, D. B., Indirect optical trapping using light driven micro-rotors for reconfigurable hydrodynamic manipulation. Nat. Commun. 2019, 10 (1), 1215
2019
-
[15]
B.; Padgett, M
Phillips, D. B.; Padgett, M. J.; Hanna, S.; Ho, Y. L. D.; Carberry, D. M.; Miles, M. J.; Simpson, S. H., Shape-induced force fields in optical trapping. Nature Photonics 2014, 8 (5), 400-405
2014
-
[16]
Light: Science & Applications 2020, 9 (1), 84
Li, D.; Liu, C.; Yang, Y.; Wang, L.; Shen, Y., Micro-rocket robot with all-optic actuating and tracking in blood. Light: Science & Applications 2020, 9 (1), 84
2020
-
[17]
K.; Liu, L.; Huang, Q.; Wu, M
Zhang, S.; Xu, B.; Elsayed, M.; Nan, F.; Liang, W.; Valley, J. K.; Liu, L.; Huang, Q.; Wu, M. C.; Wheeler, A. R., Optoelectronic tweezers: a versatile toolbox for nano-/micro-manipulation. Chem. Soc. Rev. 2022, 51 (22), 9203-9242
2022
-
[18]
The Journal of Physical Chemistry C 2025, 129 (11), 5502-5510
Dara, P.; Käll, M., Bubble Dynamics and Directional Marangoni Flow Induced by Laser Heating of Silicon Nanodisk Arrays. The Journal of Physical Chemistry C 2025, 129 (11), 5502-5510
2025
-
[19]
The Journal of Physical Chemistry C 2023, 127 (35), 17454-17459
Dara, P.; Shanei, M.; Jones, S.; Käll, M., Directional Control of Transient Flows Generated by Thermoplasmonic Bubble Nucleation. The Journal of Physical Chemistry C 2023, 127 (35), 17454-17459
2023
-
[20]
S.; Anderson, L
Nedev, S.; Carretero-Palacios, S.; Kühler, P.; Lohmüller, T.; Urban, A. S.; Anderson, L. J. E.; Feldmann, J., An Optically Controlled Microscale Elevator Using Plasmonic Janus Particles. ACS Photonics 2015, 2 (4), 491-496
2015
-
[21]
Science Robotics 2017, 2 (4), eaam6431
Li, J.; Esteban-Fernández de Ávila, B.; Gao, W.; Zhang, L.; Wang, J., Micro/ nanorobots for biomedicine: Delivery, surgery, sensing, and detoxification. Science Robotics 2017, 2 (4), eaam6431
2017
-
[22]
M., Optofluidic microsystems for chemical and biological analysis
Fan, X.; White, I. M., Optofluidic microsystems for chemical and biological analysis. Nature Photonics 2011, 5 (10), 591-597
2011
-
[23]
M.; Braun, D., Optically driven fluid flow along arbitrary microscale patterns using thermoviscous expansion
Weinert, F. M.; Braun, D., Optically driven fluid flow along arbitrary microscale patterns using thermoviscous expansion. J. Appl. Phys. 2008, 104 (10)
2008
-
[24]
M.; Kraus, J
Weinert, F. M.; Kraus, J. A.; Franosch, T.; Braun, D., Microscale fluid flow induced by thermoviscous expansion along a traveling wave. Phys. Rev. Lett. 2008, 100 (16), 164501
2008
-
[25]
W.; Iserman, C.; Kar, M.; Munder, M.; Voigt, A.; Alberti, S.; Grill, S
Mittasch, M.; Gross, P.; Nestler, M.; Fritsch, A. W.; Iserman, C.; Kar, M.; Munder, M.; Voigt, A.; Alberti, S.; Grill, S. W.; Kreysing, M., Non-invasive perturbations of intracellular flow reveal physical principles of cell organization. Nat. Cell Biol. 2018, 20 (3), 344-351
2018
-
[26]
D.; Klykov, S.; Maghelli, N.; Kreysing, M., Feedback-based positioning and diffusion suppression of particles via optical control of thermoviscous flows
Erben, E.; Seelbinder, B.; Stoev, I. D.; Klykov, S.; Maghelli, N.; Kreysing, M., Feedback-based positioning and diffusion suppression of particles via optical control of thermoviscous flows. Opt. Express 2021, 29 (19), 30272-30283
2021
-
[27]
D.; Seelbinder, B.; Erben, E.; Maghelli, N.; Kreysing, M., Highly sensitive force measurements in an optically generated, harmonic hydrodynamic trap
Stoev, I. D.; Seelbinder, B.; Erben, E.; Maghelli, N.; Kreysing, M., Highly sensitive force measurements in an optically generated, harmonic hydrodynamic trap. eLight 2021, 1 (1), 7
2021
-
[28]
Liao, W.; Erben, E.; Kreysing, M.; Lauga, E., Theoretical model of confined thermoviscous flows for artificial cytoplasmic streaming. Phys. Rev. Fluids 2023, 8 (3), 034202
2023
-
[29]
Light Sci
Erben, E.; Liao, W.; Minopoli, A.; Maghelli, N.; Lauga, E.; Kreysing, M., Opto-fluidically multiplexed assembly and micro-robotics. Light Sci. Appl. 2024, 13 (1), 59
2024
-
[30]
Small 2025, e01039
Erben, E.; Saraev, I.; Liao, W.; Nan, F.; Lauga, E.; Kreysing, M., Optical Micromanipulations Based on Model Predictive Control of Thermoviscous Flows. Small 2025, e01039
2025
-
[31]
P.; Parlane, F
MacLeod, B. P.; Parlane, F. G. L.; Morrissey, T. D.; Häse, F.; Roch, L. M.; Dettelbach, K. E.; Yunker, L. P. E.; Hein, J. E.; Berlinguette, C. P., Self-driving laboratory for accelerated discovery of thin-film materials. _Sci. Adv._ 2020, _6_ (20), eaaz8867
2020
-
[32]
M.; Gusev, V
Burger, B.; Maffettone, P. M.; Gusev, V. V.; Aitchison, C. M.; Bai, Y.; Wang, X.; Li, X.; Alston, B. M.; Li, B.; Clowes, R.; Rankin, N.; Harris, B.; Sprick, R. S.; Cooper, A. I., A mobile robotic chemist. _Nature_ 2020, _583_, 237-241
2020
-
[33]
M.; Donina, L.; Dragone, V.; Long, D.-L.; Cronin, L., Controlling an organic synthesis robot with machine learning to search for new reactivity
Granda, J. M.; Donina, L.; Dragone, V.; Long, D.-L.; Cronin, L., Controlling an organic synthesis robot with machine learning to search for new reactivity. _Nature_ 2018, _559_, 377-381
2018
-
[34]
Kalhor, P.; Jung, N.; Bräse, S.; Wöll, C.; Tsotsalas, M.; Friederich, P., Functional Material Systems Enabled by Automated Data Extraction and Machine Learning. _Adv. Funct. Mater._ 2023, _34_ (20), 2302630
2023
-
[35]
W.; Thomas, D
Coley, C. W.; Thomas, D. A.; Lummiss, J. A. M.; Jaworski, J. N.; Breen, C. P.; et al., A robotic platform for flow synthesis of organic molecules informed by AI planning. _Science_ 2019, _365_ (6453), 557-561
2019
-
[36]
F.; Aspuru-Guzik, A., Autonomous chemical research with large language models
Boiko, D.; Chan, B.; Lu, C.; Jensen, K. F.; Aspuru-Guzik, A., Autonomous chemical research with large language models. Nature 2023, 624, 570–578
2023
-
[37]
Agentic Exploration of Physics Models
Nägele, M.; Marquardt, F., Agentic exploration of physics models. arXiv preprint arXiv:2509.24978 (2025)
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[38]
Eureka: Human-Level Reward Design via Coding Large Language Models
Ma, Yecheng Jason; Liang, William; Wang, Guanzhi; Huang, De-An; Bastani, Osbert; Jayaraman, Dinesh; Zhu, Yuke; Fan, Linxi (“Jim”); Anandkumar, Anima. Eureka: Human-Level Reward Design via Coding Large Language Models. In: Proceedings of the 12th International Conference on Learning Representations (ICLR 2024)
2024
-
[39]
DrEureka: Language Model Guided Sim-To-Real Transfer
Ma, Yecheng Jason; Liang, William; Wang, Hung-Ju; Wang, Sam; Zhu, Yuke; Fan, Linxi; Bastani, Osbert; Jayaraman, Dinesh. DrEureka: Language Model Guided Sim-To-Real Transfer. In: Proceedings of the Robotics: Science and Systems (RSS) Conference 2024
2024
-
[40]
arXiv preprint arXiv:2508.17921 (2025)
Paul, D.; Milosevic, N.; Scherf, N.; Cichos, F., Physical embodiment enables information processing beyond explicit sensing in active matter. arXiv preprint arXiv:2508.17921 (2025)
- [41]
-
[42]
M.; Häse, F.; Kreisbeck, C.; Tamayo-Mendoza, T.; Yunker, L
Roch, L. M.; Häse, F.; Kreisbeck, C.; Tamayo-Mendoza, T.; Yunker, L. P. E.; Hein, J. E.; Aspuru-Guzik, A., ChemOS: An orchestration software to democratize autonomous discovery. _Chem. Sci._ 2018, _9_, 6091-6098
2018
-
[43]
Flows inside Cells: From Natural Cytoplasmic Streaming to Microfluidic Systems
Liao, Weida. Flows inside Cells: From Natural Cytoplasmic Streaming to Microfluidic Systems. Apollo - University of Cambridge Repository, 2025, doi:10.17863/CAM.122076
-
[44]
E.; Chang, B.; Mitchener, L.; Yiu, A.; Szostkiewicz, C
Ghareeb, A. E.; Chang, B.; Mitchener, L.; Yiu, A.; Szostkiewicz, C. J.; Shved, D.; Gyimesi, G. J.; Laurent, J. M.; Wright, S. M.; Razzak, M. T.; White, A. D.; Finnemann, S. C.; Hinks, M. M.; Rodriques, S. G. A multi-agent system for automating scientific discovery. Nature 2026, doi:10.1038/s41586-026-10652-y
-
[45]
Y.; Norgaard, P.; Shamsi, Z.; Smalling, D.; Thompson, J.; Venugopalan, S.; Williams, B
Aygün, E.; Belyaeva, A.; Comanici, G.; Coram, M.; Cui, H.; Garrison, J.; Johnston, R.; Kast, A.; McLean, C. Y.; Norgaard, P.; Shamsi, Z.; Smalling, D.; Thompson, J.; Venugopalan, S.; Williams, B. P.; et al. An AI system to help scientists write expert-level empirical software. Nature 2026, doi:10.1038/s41586-026-10658-6
-
[46]
Accelerating scientific discovery with Co- Scientist
Gottweis, J.; Weng, W.-H.; Daryin, A.; Tu, T.; Sirkovic, P.; Myaskovsky, A.; Glowaty, G.; Weissenberger, F.; Orlandi, A.; Tomašev, N.; Zverinski, D.; Rendulic, I.; Vedadi, E.; Hasler, F.; Rimanic, L.; et al. Accelerating scientific discovery with Co- Scientist. Nature 2026, doi:10.1038/s41586-026-10644-y
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.