Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

This paper claims that LLM-generated reward functions, extended to tactile sensing, can beat expert-tuned rewards for in-hand rotation on a real robot hand with a fraction of the code.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

First demonstration that LLM-generated reward functions using tactile sensing can outperform a human-engineered baseline for real-world in-hand rotation.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection First real LLM reward design for tactile in-hand rotation; the performance gain is plausible, but the human-supplied success signal does much of the work, so the 'autonomous design' claim is softer than the title suggests. the 3 major comments →

arxiv 2509.07445 v1 pith:YFXZMVKK submitted 2025-09-09 cs.RO cs.AI

Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions

Harrison Field , Max Yang , Yijiong Lin , Efi Psomopoulou , David Barton , Nathan F. Lepora This is my paper
classification cs.RO cs.AI
keywords tactile sensingin-hand manipulationLLM reward designreinforcement learningsim-to-real transferdexterous robot handprompt engineeringteacher-student distillation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a large language model can write the reward function that teaches a robot hand to rotate objects in the air using touch — a task where human experts currently spend considerable effort tuning reward terms. Text2Touch's answer is yes: with careful prompt engineering and an iterative generate-train-critique-refine loop, LLMs produced reward code that trained policies which outperform a human-engineered baseline in simulation and on a real tactile Allegro Hand. The generated rewards are dramatically simpler — roughly one-tenth the environment variables, one-quarter the lines of code, and one-eighth the volume of the human version — and still transfer to hardware, rotating ten household objects about all three axes in palm-up and palm-down orientations, including heavier and novel objects never seen in training. If the claim holds, it removes much of the expert effort usually required to go from a task description to a deployable dexterous skill, the bottleneck that keeps tactile robotics slow to scale.

Core claim

The central claim is that LLM-designed reward functions, brought to tactile sensing for the first time, can beat a carefully tuned human-engineered reward on gravity-invariant multi-axis in-hand rotation, both in simulation and on real hardware. The method pairs an Eureka-style iterative loop — the LLM proposes reward code, a PPO policy trains under it in simulation, component-level training statistics feed back, and the model refines — with two prompt interventions that make the loop viable in an environment exposing more than seventy variables. First, the success bonus and early-reset penalty are given to the LLM as scalable tensor inputs inside its reward expression rather than as fixed s

What carries the argument

The load-bearing device is the scalable bonus/penalty prompting strategy. The original Eureka pipeline adds a fixed scalar success bonus outside the LLM's reward; Text2Touch instead hands the LLM a success_bonus and an early_reset_penalty_value as named tensor inputs and lets the model scale them inside its own reward sum. Table 1 shows this separates total failure from success: no candidate reward without it produced a task success. Around it sits the modified prompt template, which replaces Eureka's generic signature with an explicit typed list of the roughly seventy environment variables — contact forces, fingertip positions, keypoint errors, joint states — credited with eliminating most

Load-bearing premise

The comparison treats the human-coded success_bonus and early_reset_penalty values, which the LLM is told to scale, as neutral scaffolding rather than the main driver of task success — the paper states that no LLM-generated reward produced any success without them, so if that hand-written function is doing the real work, the LLM's contribution shrinks to tuning a human-designed signal.

What would settle it

Run the same reward-discovery loop with the human-written success-bonus/fall-penalty function removed — replace it with a constant, or require the LLM to derive its own success signal from raw observations — and train the identical teacher-student pipeline. If rotations per episode collapse or fall below the human baseline, the hand-written scaffolding is the operative cause. The mirror experiment: train a policy under the human baseline reward stripped to just the scaled success bonus and early-reset penalty, and check whether it approaches the LLM rewards' rotation counts on the same hardwar

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • LLM-generated tactile rewards can replace expert reward tuning for a real dexterous hand task, shortening the route from a natural-language task description to a working policy.
  • Simplicity and hardware performance moved together: rewards touching 7–12 environment variables with 24–43 lines of code beat a human reward using 66 variables and 111 lines, so interpretability did not come at the cost of performance.
  • The scalable bonus/penalty prompt device extends LLM reward design to high-complexity environments, where the original prompt structure produced zero successful rewards across five LLMs.
  • Distilled student policies generalized to out-of-distribution object masses and shapes without retraining, with every LLM-based student exceeding the human baseline.
  • Real tactile feedback favored the LLM-trained policies: the paper hypothesizes that smoother physical contact signals let them detect instability and regrasp, reversing their disadvantage from simulation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The success_bonus and early_reset_penalty inputs are computed by a human-written function, and the paper states no LLM reward succeeded without them; the 'LLM-designed' claim is conditional on that scaffolding, since the model's role could be read as scaling a human-supplied success signal.
  • A decisive ablation would remove or replace that human-written function — for example, require the LLM to design its own success detector — and retrain the same pipeline; if rotations collapse, the scaffolding, not the language model, is carrying the result.
  • Because reward correlation with the human baseline was low (0.30–0.47), the LLM rewards likely encode genuinely different manipulation strategies rather than approximations of the human one; dissecting the finger-gaiting patterns of the winning policies could reveal a novel rotation style.
  • The pattern that aggressive, contact-sensitive policies failed in rigid-body simulation but succeeded on smooth real tactile sensors suggests a practical rule for future LLM reward prompts: if the hardware's tactile signal is richer than the simulator's, favor compact and fast-rotating rewards.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents Text2Touch, an LLM-based reward design pipeline for tactile in-hand object rotation with a four-fingered Allegro Hand. It extends the Eureka iterative reward-generation loop by (i) giving the LLM an explicit reward-function signature containing more than 70 environment variables and (ii) exposing two scalar variables, a success bonus B and an early-reset penalty P, for the LLM to scale inside its reward expression. Teacher policies are trained in simulation with privileged observations and distilled to students using only proprioception and tactile inputs. The authors report that across five LLMs, the best LLM-generated rewards outperform a published human-engineered baseline in simulation, and that three deployed policies outperform the baseline on real-world average rotations per episode and time to termination. The paper also reports code-complexity metrics showing that the LLM rewards are much shorter and use fewer environment variables than the human baseline.

Significance. If the central attribution claim is accepted, the paper is a significant systems contribution: it is the first demonstration of LLM-designed rewards for tactile in-hand manipulation with real-world sim-to-real transfer, and it shows that a carefully designed prompt can scale Eureka-style reward search to a high-dimensional tactile environment. Strengths include the breadth of LLMs tested (five models, including open/closed and reasoning/non-reasoning), the consistent simulation results across models, the explicit code-quality metrics, and the honest limitation section. The main risk is that the 'LLM-designed reward' framing may overstate the LLM's role: task success is impossible without human-coded success/penalty scaffolding, and the LLM's main contribution may be re-scaling that scaffolding. The real-world evidence also lacks trial counts and error bars. These concerns are load-bearing for the paper's headline claims, but they are addressable with additional analysis or a qualified reframing.

major comments (3)
  1. [Section 3.2; Listing 7; Listing 9] The central attribution claim that LLMs 'designed' the reward functions is weakened by the paper's own scaffolding. Section 3.2 states that no LLM-generated reward produced task success without the scalable success bonus B and early-reset penalty P, and Listing 7 (lines 191-203) shows that these values are produced by a human-written function compute_success_bonus_fall_penalty with human-set thresholds such as success_tolerance, fall_reset_dist, and reach_goal_bonus. The example rewards in Listing 9 and Appendix D are largely weighted combinations of these externally computed signals (e.g., success_bonus * 35.0, early_reset_penalty_value * 0.01, temperature_success * success_bonus). Thus the comparison is not 'LLM-written reward vs. human-written reward' on equal footing: both sides receive the same human-coded success/termination oracle, and the LLM's learned component is closer to tuni
  2. [Table 4; Section 4.3] The real-world comparison rests on averages with no measure of variability. Table 4 reports a single value per condition and a 'Total Avg' row, but no standard deviations, no number of trials per object/condition, and no significance tests. This is particularly important because several per-condition entries are worse than or close to the baseline (e.g., GPT-4o Palm Down Y: 0.46 rotations vs. baseline 0.67; GPT-4o Palm Down X: 0.73 vs. 0.65; Gemini Palm Down Y: 0.61 vs. 0.67). The aggregate '38% increase' for Deepseek-R1-671B is therefore not substantiated as a robust effect. Please report per-condition trial counts and dispersion, and perform a significance test or clearly label the results as indicative single-robot demonstrations. Without this, the claim of 'significantly outperforms' in the abstract is not supported by the real-world data.
  3. [Section 4.1; Section 4.3] Selection bias appears to inflate the reported gains. In Stage 1, the final reward for each LLM is selected as the best among 100 candidates per cell (Section 4.1), and the final training run is repeated five times with the highest rotation score reported (Table 2). In Stage 3, only three LLM policies are chosen for hardware deployment, selected after Stage 2 as the highest-rotation, highest-solve-rate, and longest-episode models (Section 4.3). Because the real-world comparison is performed only on these selected models, the reported superiority over the human baseline is a maximum over many candidate rewards, seeds, and models rather than a typical result. The authors should provide the full distribution of real-world results for all five LLM student policies, or otherwise account for selection in the statistical comparison. This is load-bearing for the 'LLMs consistently outperform' na
minor comments (4)
  1. [Appendix E.2] Typo: 'eluded to' should be 'alluded to'.
  2. [Listing 7 / Appendix A] Several typos in the listing: 'obesrvation', 'workframe' for 'worldframe', and inconsistent use of 'pivot axel' vs. 'axis'. These do not affect the method but should be cleaned up.
  3. [Section 6] The limitation 'we evaluated fewer real-world hand orientations (palm up and palm down) than were considered originally in the baseline study' should be moved into the main results discussion or at least cross-referenced in Table 4, since the comparison is to AnyRotate and this reduces comparability.
  4. [Appendix C] The additional LLM results in Table 6 are reported without specifying the number of reward-discovery runs or seeds, making it hard to know whether the differences among LLMs are within noise. A brief note on variance would help.

Circularity Check

1 steps flagged

LLM reward advantage largely re-scales a human-written success bonus; the 'LLM-designed rewards' attribution is partially circular.

specific steps
  1. fitted input called prediction [Section 3.1-3.2; Listing 7 (lines 191-203); Listing 9]
    "Empirically, Table 1 shows task success through LLM-generated rewards proved impossible without providing a scalable B, P. ... self.success_bonus, self.early_reset_penalty_value = compute_success_bonus_fall_penalty(...) ... scaled_success_bonus = torch.where(success_mask, success_bonus * 35.0, progressive_bonus * 10.0)"

    The fitness F used to select rewards is the running successes count (Section 3.1: 'We track these as a running successes count, which forms the fitness F'). The success_bonus input the LLM is told to scale is computed by the human-written compute_success_bonus_fall_penalty from the same sub-goal success condition (mean keypoint distance <= success_tolerance) and human-set thresholds. Best LLM rewards, e.g. Deepseek's 'success_bonus * 35.0' and o3-mini's '50.0 * success_bonus', therefore embed the human success oracle in the optimized reward, and the reported 'LLM-designed' advantage is in part the same human signal being re-scaled. The paper concedes no task success was possible without B,P, so the LLM contribution is largely automated tuning of shaping weights around a human success/termi

full rationale

The sim-to-real pipeline, teacher-student distillation, and real-world evaluations are external and not circular; if the claim were merely 'LLM-composed reward code can outperform a human baseline,' the experiments support it. However, the headline 'LLM-designed rewards' is undermined because the necessary success/termination signal is human-written. The paper itself states no task success was possible without the human-provided scalable B,P. The fitness function used for selection is the same sub-goal success count that the human-written success_bonus encodes, so the LLM reward functions that add scaled success_bonus are partly optimizing the human's success oracle rather than an independently LLM-designed objective. This is a partial circularity in the central attribution. No self-citation is load-bearing here: the AnyRotate baseline and pipeline are used as a fixed external benchmark, and the paper freezes those components for fair comparison. The measured performance gains and the concise LLM-generated shaping terms retain independent content, so the paper is not wholly circular, but the core 'LLM-designed reward' claim is overstated.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central result depends on many inherited constants from AnyRotate, on LLM-chosen scalings of a human-coded success bonus, and on several domain assumptions about simulation, selection, and transfer. No new physical entities are introduced.

free parameters (4)
  • LLM-chosen reward scales and temperatures = Examples: success bonus multiplier 35.0, kp_temp 0.15, weight_success 50.0 (Listings 9, 13-16)
    Each generated reward function contains component weights and temperatures selected by the LLM through iterative feedback. These values materially change policy behavior and are fitted to task-score feedback.
  • Scalable success bonus B and early reset penalty P = B and P are computed by a human-coded function (Listing 7); final scale chosen by the LLM, e.g., 35x or 0.01x
    Section 3.2 reports that no LLM reward succeeded without these inputs, so their presence and scaling are free parameters on which the main result depends.
  • Goal update tolerance = 0.15 (Teacher), 0.25 (Student)
    These values from AnyRotate (Appendix E.1) affect how often sub-goal successes are counted, which is the fitness used to select rewards.
  • Environment thresholds = Fall reset distance, axis deviation reset distance, success tolerance, contact threshold
    Inherited from the AnyRotate environment configuration; they define success and termination and therefore shape both the fitness and the supplied success bonus.
axioms (4)
  • domain assumption PPO with the listed hyperparameters reliably optimizes the LLM-written rewards in Isaac Gym
    Section 3.1 and Appendix E.1; all simulation and downstream real-world results depend on standard RL training working as expected.
  • domain assumption 150M-step reward discovery scores predict which rewards will be best after 8B steps and on hardware
    Section 3.4 states full 8B-step training for every candidate is infeasible, so short runs select R*. If rankings change with more training, the selection is unreliable.
  • domain assumption Rigid-body simulated tactile contact is an adequate training ground for real TacTip sensors after teacher-student distillation
    Section 3.5 and the Limitations admit a gap between simulated rigid-body contacts and real vision-based tactile sensors; successful transfer is assumed and only shown for a few selected policies.
  • domain assumption The natural language task description accurately specifies the rotation task without injecting human bias
    Section 3.1 says the description was chosen after experiments and closely aligns with the human-engineered baseline task. If it encodes the baseline objective too directly, the LLM is not fully independent of human design.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions." pith.science (2026). https://pith.science/paper/YFXZMVKK

@misc{pith2026250907445,
  author       = {Pith},
  title        = {Pith review of: Text2Touch: Tactile In-Hand Manipulation with LLM-Designed Reward Functions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YFXZMVKK}},
  note         = {Machine review of arXiv:2509.07445}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs) are beginning to automate reward design for dexterous manipulation. However, no prior work has considered tactile sensing, which is known to be critical for human-like dexterity. We present Text2Touch, bringing LLM-crafted rewards to the challenging task of multi-axis in-hand object rotation with real-world vision based tactile sensing in palm-up and palm-down configurations. Our prompt engineering strategy scales to over 70 environment variables, and sim-to-real distillation enables successful policy transfer to a tactile-enabled fully actuated four-fingered dexterous robot hand. Text2Touch significantly outperforms a carefully tuned human-engineered baseline, demonstrating superior rotation speed and stability while relying on reward functions that are an order of magnitude shorter and simpler. These results illustrate how LLM-designed rewards can significantly reduce the time from concept to deployable dexterous tactile skills, supporting more rapid and scalable multimodal robot learning. Project website: https://hpfield.github.io/text2touch-website

Figures

Figures reproduced from arXiv: 2509.07445 by David Barton, Efi Psomopoulou, Harrison Field, Max Yang, Nathan F. Lepora, Yijiong Lin.

Figure 1
Figure 1. Figure 1: Text2Touch improves upon previous reward function design methods to increase the performance of robotic in-hand object rotation in rotation speed and grasp stability. We evaluate the performance of LLM-generated reward functions using only tactile and proprioceptive in￾formation in the real world. Designing reinforcement learning (RL) reward functions for dexterous in-hand manipulation remains a formidable… view at source ↗
Figure 2
Figure 2. Figure 2: Text2Touch training and deployment pipeline. The left and middle columns comprise the reward generation pipeline with our novel prompting strategy components in green, and the right column denoting the model distillation and deployment phase. The teacher model (yellow) is the final output of the reward generation phase and the student (lilac) is the distilled teacher using only real-world observations. dir… view at source ↗
Figure 3
Figure 3. Figure 3: Real-world deployment using various objects in palm-up & palm-down configurations. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Top: Objects used to evaluate dis￾tilled Student tactile model in simulation. Bottom: Objects used to evaluate distilled models in the real world. The improved real-world stability of LLM-generated policies likely stems from differences between sim￾ulated and physical tactile sensing. Simulated con￾tact uses rigid-body models, producing brittle sig￾nals, whereas real-world vision-based tactile sensors prov… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. TactiDex: A Real-World Tactile-Guided Benchmark for Human-Like Dexterous Manipulation

    cs.RO 2026-07 conditional novelty 6.0

    A tactile-rich HOI dataset plus a tri-component force reward improves contact fidelity and success of human-to-robot dexterous transfer over kinematic imitation alone.

Reference graph

Works this paper leans on

65 extracted references · 42 canonical work pages · cited by 1 Pith paper · 4 internal anchors

  1. [1]

    Kober, J

    J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey.The International Journal of Robotics Research, 32(11):1238–1274, 2013

    2013

  2. [2]

    Hadfield-Menell, S

    D. Hadfield-Menell, S. Milli, P. Abbeel, S. J. Russell, and A. Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017

    work page 2017

  3. [3]

    Booth, W

    S. Booth, W. B. Knox, J. Shah, S. Niekum, P. Stone, and A. Allievi. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5920–5929, 2023

    work page 2023

  4. [4]

    Liang, W

    J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023

    work page 2023

  5. [5]

    Y . J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y . Zhu, L. Fan, and A. Anandkumar. Eureka: Human-level reward design via coding large language models.arXiv preprint arXiv:2310.12931, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al. Language to rewards for robotic skill synthesis.arXiv preprint arXiv:2306.08647, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    T. Xie, S. Zhao, C. H. Wu, Y . Liu, Q. Luo, V . Zhong, Y . Yang, and T. Yu. Text2reward: Reward shaping with language models for reinforcement learning.arXiv preprint arXiv:2309.11489, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    Q. Guo, X. Liu, J. Hui, Z. Liu, and P. Huang. Utilizing large language models for robot skill reward shaping in reinforcement learning. InInternational Conference on Intelligent Robotics and Applications, pages 3–17. Springer, 2024

    work page 2024

  9. [9]

    Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation with Large Language Models

    N. Turcato, M. Iovino, A. Synodinos, A. D. Libera, R. Carli, and P. Falco. Towards autonomous reinforcement learning for real-world robotic manipulation with large language models.arXiv preprint arXiv:2503.04280, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  10. [10]

    Y . J. Ma, W. Liang, H.-J. Wang, S. Wang, Y . Zhu, L. Fan, O. Bastani, and D. Jayaraman. Dreureka: Language model guided sim-to-real transfer.arXiv preprint arXiv:2406.01967, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Hansen, F

    J. Hansen, F. Hogan, D. Rivkin, D. Meger, M. Jenkin, and G. Dudek. Visuotactile-rl: Learning multimodal manipulation policies with deep reinforcement learning. In2022 International Conference on Robotics and Automation (ICRA), pages 8298–8304. IEEE, 2022

    2022

  12. [12]

    M. Yang, C. Lu, A. Church, Y . Lin, C. Ford, H. Li, E. Psomopoulou, D. A. Barton, and N. F. Lepora. Anyrotate: Gravity-invariant in-hand object rotation with sim-to-real touch.arXiv preprint arXiv:2405.07391, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    H. Qi, B. Yi, S. Suresh, M. Lambeta, Y . Ma, R. Calandra, and J. Malik. General in-hand object rotation with vision and touch. InConference on Robot Learning, pages 2549–2564. PMLR, 2023. 10

    work page 2023

  14. [14]

    H. Qi, A. Kumar, R. Calandra, Y . Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. InConference on Robot Learning, pages 1722–1732. PMLR, 2023

    2023

  15. [15]

    S. J. Russell and P. Norvig.Artificial intelligence: a modern approach. Pearson, 2016

    work page 2016

  16. [16]

    URLhttps://research.google/blog/language-to-rewards-for-robotic-skill-synthesis/

    work page

  17. [17]

    Automated Rewards via LLM-Generated Progress Functions

    V . Sarukkai, B. Shacklett, Z. Majercik, K. Bhatia, C. Ré, and K. Fatahalian. Automated rewards via llm-generated progress functions.arXiv preprint arXiv:2410.09187, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    G. Xie, J. Xu, Y . Yang, Y . Ding, and S. Zhang. Large language models as efficient reward func- tion searchers for custom-environment multi-objective reinforcement learning.arXiv preprint arXiv:2409.02428, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, K.-H. Lee, S. Levine, Y . Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J. Rettingh...

    Pith/arXiv arXiv 2022

  20. [20]

    Mon-Williams, G

    R. Mon-Williams, G. Li, R. Long, W. Du, and C. G. Lucas. Embodied large language mod- els enable robots to complete complex tasks in unpredictable environments.Nature Machine Intelligence, pages 1–10, 2025

    work page 2025

  21. [21]

    Driess, F

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, et al. Palm-e: An embodied multimodal language model. 2023

    work page 2023

  22. [22]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    Pith/arXiv arXiv 2022

  23. [23]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    K. Ryu, Q. Liao, Z. Li, K. Sreenath, and N. Mehr. Curricullm: Automatic task curric- ula design for learning complex robot skills using large language models.arXiv preprint arXiv:2409.18382, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  25. [25]

    Liang, S

    W. Liang, S. Wang, H.-J. Wang, O. Bastani, D. Jayaraman, and Y . J. Ma. Environment cur- riculum generation via large language models. In8th Annual Conference on Robot Learning, 2024

    work page 2024

  26. [26]

    W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force.Sensors, 17(12):2762, 2017

    2017

  27. [27]

    Lambeta, P.-W

    M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V . R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation.IEEE Robotics and Automation Letters, 5(3):3838–3845, 2020

    2020

  28. [28]

    N. F. Lepora. Soft biomimetic optical tactile sensing with the tactip: A review.IEEE Sensors Journal, 21(19):21131–21143, 2021

    work page 2021

  29. [29]

    Y . Lin, J. Lloyd, A. Church, and N. Lepora. Tactile gym 2.0: Sim-to-real deep reinforcement learning for comparing low-cost high-resolution robot touch. volume 7 ofProceedings of Machine Learning Research, pages 10754–10761. IEEE, August 2022. doi:10.1109/LRA. 2022.3195195. URLhttps://ieeexplore.ieee.org/abstract/document/9847020. 11

    work page arXiv 2022

  30. [30]

    Y . Lin, A. Church, M. Yang, H. Li, J. Lloyd, D. Zhang, and N. F. Lepora. Bi-touch: Bi- manual tactile manipulation with sim-to-real deep reinforcement learning.IEEE Robotics and Automation Letters, 8(9):5472–5479, 2023

    work page 2023

  31. [31]

    J. W. James and N. F. Lepora. Slip detection for grasp stabilization with a multifingered tactile robot hand.IEEE Transactions on Robotics, 37(2):506–519, 2020

    work page 2020

  32. [32]

    URLhttps://www.borntoengineer.com/ tactile-robotic-hand-breakthrough-rotating-objects-in-any-direction

    July 2024. URLhttps://www.borntoengineer.com/ tactile-robotic-hand-breakthrough-rotating-objects-in-any-direction

    work page 2024

  33. [33]

    O. M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020

    2020

  34. [34]

    X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In2018 IEEE international conference on robotics and automation (ICRA), pages 3803–3810. IEEE, 2018

    work page 2018

  35. [35]

    Akkaya, M

    I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik’s cube with a robot hand.arXiv preprint arXiv:1910.07113, 2019

    Pith/arXiv arXiv 1910

  36. [36]

    Chebotar, A

    Y . Chebotar, A. Handa, V . Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In2019 International Conference on Robotics and Automation (ICRA), pages 8973–8979. IEEE, 2019

    work page 2019

  37. [37]

    T. Chen, M. Tippur, S. Wu, V . Kumar, E. Adelson, and P. Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes.Science Robotics, 8(84):eadc9244, 2023

    2023

  38. [38]

    W. Chen, J. Xu, F. Xiang, X. Yuan, H. Su, and R. Chen. General-purpose sim2real protocol for learning contact-rich manipulation with marker-based visuotactile sensors.IEEE Transactions on Robotics, 40:1509–1526, 2024

    work page 2024

  39. [39]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  40. [40]

    J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain.Science robotics, 5(47):eabc5986, 2020

    2020

  41. [41]

    D. Chen, B. Zhou, V . Koltun, and P. Krähenbühl. Learning by cheating. InConference on robot learning, pages 66–75. PMLR, 2020

    work page 2020

  42. [42]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Hurst, A

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    URLhttps://openai.com/index/openai-o3-mini/

    work page

  46. [46]

    G. Team, P. Georgiev, V . I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024. 12

    Pith/arXiv arXiv 2024

  47. [47]

    M. H. Halstead.Elements of Software Science (Operating and programming systems series). Elsevier Science Inc., 1977

    work page 1977

  48. [48]

    Laskin, K

    M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data.Advances in neural information processing systems, 33:19884–19895, 2020

    work page 2020

  49. [49]

    Y . Lin, J. Huang, M. Zimmer, Y . Guan, J. Rojas, and P. Weng. Invariant transform experience replay: Data augmentation for deep reinforcement learning.IEEE Robotics and Automation Letters, 5(4):6615–6622, 2020. doi:10.1109/LRA.2020.3013937

    work page arXiv 2020

  50. [50]

    Makoviychuk, L

    V . Makoviychuk, L. Wawrzyniak, Y . Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning.arXiv preprint arXiv:2108.10470, 2021. 13 A Prompting Prompting StructureThe prompting structure is composed of several component parts. As re- ferred to in S...

    Pith/arXiv arXiv 2021

  51. [51]

    System prompt: High level overview of the context and expectations with tips on providing good reward code as well as an example reward function template to adhere to

    work page

  52. [52]

    For iterations>0, we frame the prompt as a conversation in memory where we include:

    Initial user prompt: Inclusion of environment code as contextMand natural language task descriptionl. For iterations>0, we frame the prompt as a conversation in memory where we include:

    work page

  53. [53]

    Best LLM response (as determined via the fitness functionF)

    work page

  54. [54]

    Policy feedback: Generated metrics that track optimisation of reward components defined by the LLM in the previous iteration

    work page

  55. [55]

    ‘ ‘ ‘ python ... ‘ ‘ ‘

    Code feedback: Instructions to use the previous reward function and returned metrics to inform the next reward function. Apart from our innovations, the prompt scaffolding is taken from Eureka [5]. Separately, the com- ponents look like: Listing 1: System Prompt. The task reward signature is incorporated later and varies depending on whether the prompt st...

    work page

  56. [60]

    ‘ ‘ ‘ python ... ‘ ‘ ‘

    ** Penalty R e d u c t i o n s ** - Non - tip penalty reduced from -0.1 to -0.05 per contact - Early reset penalty impact reduced by 50% ** E xpe ct ed I m p r o v e m e n t s :** - 5 -8 x inc re as e in k p _ r e w a r d values ( from mean 0.01 to ~0.3 -0.5) - More fre qu en t success bonuses through p r o g r e s s i v e scaling - Better contact m a n a...

    work page

  57. [61]

    ** Ke yp oi nt Reward O ver ha ul ** - I n c r e a s e d weight from 3.0 x to s t a n d a l o n e c o m p o n e n t (3.0 base value ) - Reduced t e m p e r a t u r e from 0.3 to 0.15 for sharper re sp on se - Changed error n o r m a l i z a t i o n to mean instead of sum - Removed c o m p e t i n g p o s e _ r e w a r d that was d o m i n a t i n g rewards

    work page

  58. [62]

    ** Success Bonus E n h a n c e m e n t ** - Added p r o g r e s s i v e bonus scaling with po si tio n / o r i e n t a t i o n ac cu ra cy - I n c r e a s e d max bonus m u l t i p l i e r from 25 x to 35 x - Relaxed o r i e n t a t i o n t h r e s h o l d from 0.15 to 0.2 (~11.5 ◦ )

    work page

  59. [63]

    ** Contact System Changes ** - Reduced ideal co nt act s from 3 to 2 for better gaiting - I n c r e a s e d c o n t a c t _ s t a b i l i t y weight to 1.5 x - Halved va ri an ce t e m p e r a t u r e d e n o m i n a t o r for s tr ic te r s t a b i l i t y

    work page

  60. [64]

    ** Ve lo ci ty Penalty R e l a x a t i o n ** - Reduced linear penalty c o e f f i c i e n t from 0.15 to 0.1 - I n c r e a s e d t e m p e r a t u r e from 0.6 to 1.0 for s mo oth er penalty - Lowered weight from 0.8 x to 0.5 x

    work page

  61. [65]

    ‘ ‘ ‘ python ... ‘ ‘ ‘

    ** Penalty R e d u c t i o n s ** - Non - tip penalty reduced from -0.1 to -0.05 per contact - Early reset penalty impact reduced by 50% ** E xpe ct ed I m p r o v e m e n t s :** - 5 -8 x inc re as e in k p _ r e w a r d values ( from mean 0.01 to ~0.3 -0.5) - More fre qu en t success bonuses through p r o g r e s s i v e scaling - Better contact m a n a...

    work page

  62. [66]

    - To better d i f f e r e n t i a t e even small improvements , we de cr eas e the t e m p e r a t u r e

    P os it ion Reward ( p o s _ r e w a r d ) : - O bs erv ed values were in a narrow range (0.37 - 0.70) . - To better d i f f e r e n t i a t e even small improvements , we de cr eas e the t e m p e r a t u r e . - New f o r m u l a t i o n uses an e x p o n e n t i a l decay with a lower po s_t em p so that errors are p e n a l i z e d more

    work page

  63. [67]

    - New f o r m u l a t i o n l in ea rly maps the a bso lu te a l i g n m e n t of the q u a t e r n i o n ’s scalar part , i

    O r i e n t a t i o n Reward ( o r n _ r e w a r d ) : - P re vio us o r i e n t a t i o n reward using exp ( - error ^2/ o rn _te mp ) yielded nearly co nst an t values . - New f o r m u l a t i o n l in ea rly maps the a bso lu te a l i g n m e n t of the q u a t e r n i o n ’s scalar part , i . e . using w = | q_w |. When w is low ( <0.5) it outputs 0 ...

    work page

  64. [68]

    Contact Reward ( c o n t a c t _ r e w a r d ) : - Prior contact reward squared the ratio ; its values spanned a small range . - We now use a cubic t r a n s f o r m a t i o n on the ratio of good co nt ac ts to better reward robust contacts , so that fewer c on ta cts result in very low rewards and only high - quality contact yields a strong bonus

    work page

  65. [69]

    p o s _ r e w a r d

    Success Bonus and Early Reset Penalty : - We keep the scaling for these sparse reward c o m p o n e n t s but re - balance the overall sum . The total reward is a we ig hte d sum of these c o m p o n e n t s . " " " # T e m p e r a t u r e / scaling p a r a m e t e r s for e x p o n e n t i a l / linear t r a n s f o r m s : po s_ te mp : float = 0.0002 #...

    work page

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.