REVIEW 5 major objections 4 minor 30 references
Large Language Models as Autonomous Spacecraft Operators in Kerbal Space Program
T0 review · 5 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A pure LLM agent, using prompt engineering and fine-tuning, controlled a pursuit spacecraft well enough to place second in the KSPDG challenge.
desk verdict A credible engineering demonstration of LLM-based spacecraft control in a public competition, with a real confound between the hand-computed prograde feature and the prompt that needs an ablation before the scientific claims fully land. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the LLM-in-the-loop control loop: each KSPDG observation is serialized into a user prompt, the model outputs a short reasoning trace followed by a call to a `perform_action` function, and the returned discrete thrust is applied to the spacecraft. Two augmentations carry most of the work. The first is the computed prograde vector, the direction of the pursuer's velocity relative to the evader expressed in the vessel frame; it converts orbital geometry into a simple left/right/up/down hint the model can reason about. The second is chain-of-thought prompting, which guides the model to state whether each component of the prograde vector is positive or negative before choosing the opposite thrust. For the fine-tuned variants, the training corpus is small—two human gameplay logs for GPT, roughly 10 to 50 synthetic navball-bot logs for LLaMA—and a sliding window over past actions lets the model learn short sequences with less data.
What would settle it
A decisive experiment: fine-tune the best LLaMA model on the same 50 gameplay logs but with the prograde vector removed from every prompt, then run it on scenario E3; if the best closest-approach distance rises toward the naive baseline of 225 meters, the controller is the engineer's mapping, not the model's.
Extended reading notes
Core claim
On the paper's own terms, the core claim is that a pure LLM agent—no value network, no policy gradient, no trajectory optimizer—can solve the KSPDG Pursuer-Evader scenario well enough to rank second in an open competition. The agent works by formatting the current orbital state as a natural-language observation, augmenting it with a computed prograde vector that tells which way the pursuer is moving relative to the evader, and asking the model to reason step-by-step before calling a function that returns a discrete thrust. Chain-of-thought prompting reduces execution failures to zero across the evaluated scenarios and brings the best closest-approach distance to 5.63 meters in scenario E3. Fine-tuning on human and synthetic gameplay improves the policy further and cuts response latency, with the best LLaMA fine-tune reaching a best distance of 11.86 meters and outperforming the bot that generated its training data. The authors present this as evidence that LLMs are a viable, sample-efficient alternative to reinforcement learning for spacecraft control in simulation.
Load-bearing premise
The load-bearing premise is that the hand-written prompt and the precomputed prograde vector already encode the correct control law, so the model itself is not independently discovering the policy from telemetry.
Editorial extensions
If this is right
- LLM agents for a spacecraft control benchmark can be built and fine-tuned in about two to four months, with tens of training episodes rather than the thousands typically required by reinforcement learning.
- A hand-computed prograde vector plus chain-of-thought prompting reduces action-call failures to 0% and brings the pursuer within meters of the evader in all four Pursuer-Evader scenarios.
- Fine-tuning an open-weight model on synthetic navball-bot logs yields a controller that outperforms the bot that generated the data, suggesting the model contributes its own prior knowledge on top of imitation.
- Latency, not just accuracy, is a first-order constraint: the fine-tuned LLaMA agents fly clunky trajectories because of 3-second response delays, so deployment decisions must trade model size against control frequency.
- The KSPDG environment, despite being designed as a test-time benchmark, supports developing LLM agents, opening the door to a wider LLM-versus-classical-control comparison in space guidance, navigation, and control.
Reading between the lines
- A fair test of LLM autonomy would remove or vary the hand-crafted prograde hint, since the paper's own overshooting example shows that an incorrect hint degrades performance and therefore part of the controller lives in the prompt engineer's mapping.
- A natural extension is to replace the manually computed prograde vector with a learned or automatically derived feature and measure how much performance drops, which would separate the model's reasoning from the engineer's control law.
- The same prompt-and-fine-tune recipe could transfer to other orbital rendezvous or docking simulators, and to the other KSPDG scenarios (target guarding, sun-blocking) that this paper leaves untested.
- If scaling the training corpus from 50 to several hundred logs continues the observed trend, fine-tuned LLM agents may approach or beat the differential-equations approach that took first place in the competition.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents an LLM-based agent for the Kerbal Space Program Differential Games (KSPDG) pursuit-evasion challenge. The authors convert spacecraft telemetry into textual prompts, augment the observations with a computed prograde vector, and use GPT-3.5 with chain-of-thought (CoT) prompting and fine-tuning, as well as LLaMA-3 fine-tuned on logs from a rule-based 'Navball bot'. They report that the GPT-based agent ranked 2nd in the KSPDG challenge, and that fine-tuned LLaMA reaches a best distance of 11.86 m in scenario E3, outperforming the teacher bot (34.34 m). The central claim is that LLMs are a sample-efficient alternative to RL for spacecraft control, requiring only tens of demonstration logs.
Significance. If the central claim were fully established, this would be a noteworthy result in space guidance and control, showing that a pre-trained language model can be adapted to a closed-loop control task with a small dataset. The paper is commendable for open-sourcing code, models, and data, for using a public competition with external baselines (PPO, iLQGames, Lambert-MPC), and for reporting clear competition rankings. However, the central claim is currently confounded: the prograde vector (Eq. 1) and the CoT template (Fig. 4) together encode a near-complete sign-based control policy, so the experiments as presented do not distinguish LLM-driven reasoning from a hand-crafted lookup rule plus a text parser. The lack of ablations, the absence of error bars and seed variation, and an apparent data error in Table 10 prevent the paper from supporting the broader claim; the contribution is better characterized at present as an integration study of LLMs as action parsers in a simulated space environment.
major comments (5)
- [§4.2, Eq. (1), Fig. 4] The prompt with CoT encodes a fixed sign-based lookup rule: positive x prograde implies 'move left', negative y implies 'forward', negative z implies 'up'. Since the prograde vector is computed externally (Eq. 1) and injected into the text, the LLM is only required to parse three numbers and emit the corresponding discrete action. To support the claim that the LLM is the controller, the authors should include a scripted baseline that reads the same prograde vector and applies the same mapping directly (including appropriate deadbands), and compare its performance against the GPT and LLaMA results in Tables 4, 7, and 10. The paper's own observation in §6.8 and Fig. D.11, that changing the prompt hint changes behavior (overshooting), confirms that the prompt text is carrying the policy; an agnostic-prompt condition should be reported quantitatively.
- [Table 10 vs. Table 6] The 'Best Score' column in Table 10 for the GPT experiments contains values identical to the latency values in Table 6 (e.g., GPT Baseline 759.69, GPT Default LRM 977.32, GPT + LRM 0.2 749.02, GPT + system prompt 684.13, GPT + 2 files 468.98). Since Eq. (7) defines the score as a function of distance, velocity, fuel, and time, these numbers cannot be scores. This appears to be a copy-paste error, but it undermines the scoring comparison in Table 10. Please recompute the scores using Eq. (7) with the recorded variables, or remove the column and report only the individual components.
- [§6.4, §6.5, §6.6] All performance results are reported as single 'best' and 'average' values without the number of evaluation episodes, random seeds, or confidence intervals. For example, Table 4 gives a 0% failure rate for the CoT agent and Table 7 gives 0.00% failure rates for several LLaMA models, but no sample size is stated anywhere. The conclusion that fine-tuning 'reduces run variance' (§6.5) is not supportable without repeated runs. Please provide at least 5–10 independent evaluation runs per configuration, state the exact number of episodes per scenario, and report mean ± standard deviation (or confidence intervals) for the distance, latency, and failure-rate metrics.
- [§5.1, §6.6] The LLaMA models are fine-tuned on Navball bot logs collected with Algorithm 3 and evaluated on scenario E3. Since the teacher bot itself is a hand-coded prograde/braking controller (Algorithm 2), the fine-tuned model may be fitting a policy that is already encoded in its inputs, and the reported 'outperforms the teacher' result (11.86 m vs. 34.34 m best distance) may reflect distribution overlap rather than generalization. Please report whether any training orbits resemble E3, evaluate on held-out orbit parameters that are excluded from training, and provide the teacher bot's performance under the same evaluation protocol (including the same number of runs and the same success criterion).
- [§6.6, §6.8, Fig. D.11] The manuscript concedes that CoT 'hacks' the language model via prompting, and Fig. D.11 shows that an incorrect prompt hint forces an accelerate-only behavior leading to overshooting. Together with the external prograde computation, these admissions indicate that the hand-crafted prompt template, not the LLM's pre-trained reasoning or acquired control knowledge, is responsible for the reported qualitative success. To separate the factors, please include ablations where (a) the LLM is given raw observations without the prograde vector and without the CoT hint, (b) the prograde vector is given but the CoT hint is replaced by an instruction to ignore it, and (c) the full CoT prompt is used. The differences in performance will quantify the contribution of the language model's reasoning versus the template.
minor comments (4)
- [§6.5] The sentence 'fine-tuning significantly decreases GPT-3.5 response latency, as demonstrated in Table 4' should reference Table 6, because Table 4 reports distances, not latencies.
- [Table 5] Table 5 does not state which scenario was used for the GPT fine-tuning experiments; please add a scenario label (presumably E3) to match Table 7, or clarify if the averages are across all E1–E4 scenarios.
- [§4.1] There is a typo: 'ASISSTANT' should be 'ASSISTANT' in the description of the GPT fine-tuning format.
- [Algorithm 1] The indentation of 'If response includes function call Then' appears to place it outside the 'Try' block; please fix the pseudocode indent so the control flow is unambiguous.
Circularity Check
No circular derivation: the central evidence is an externally scored public benchmark; the prompt-embedded prograde policy is an attribution confound, not a circularity.
full rationale
The paper's central evidence is the KSPDG public competition, with a fixed scoring function (Eq. 7) and external baselines (naive, PPO, iLQGames, Lambert-MPC) reported from SpaceGym. The performances of the CoT-prompted GPT agent and the fine-tuned LLaMA agents are empirical outcomes, not quantities derived from a fitted parameter renamed as a prediction. The fine-tuning results are evaluated on validation data and on simulation trajectories, so the reported generalization is externally checkable rather than forced by construction. The self-citations are not load-bearing: Rodriguez-Fernandez et al. (2024) is used as background for the design choice to retain CoT over rule-based prompting, and the paper's own Table 4 independently demonstrates the CoT effect; Carrasco et al. (2025) is a side expectation about GPT-4, not the basis of the central claim. There is, however, a genuine attribution confound: Equation (1) computes the prograde vector outside the LLM, and Figure 4 embeds a sign-based throttle rule in the CoT prompt, so the reported performance cannot cleanly be credited to autonomous LLM reasoning. The paper itself concedes in Section 6.6 that CoT 'hacks' the language model via prompting. This is a correctness and ablation concern, not a circular derivation, because the benchmark, scoring, and baselines are external and the result is not equivalent to the paper's inputs by construction. No circular step meets the evidentiary standard required here.
Assumptions & free parameters
free parameters (7)
- GPT learning rate multiplier (LRM) =
0.2
- LoRA rank r =
16
- LoRA alpha =
8
- LoRA dropout =
0.05
- Sliding window size =
3
- Navball bot rotation threshold =
not stated
- Navball bot approach speed =
not stated
assumptions (5)
- domain assumption KSPDG's two-body physics and discrete thrust model approximate real orbital pursuit-evasion.
- standard math The rotation matrix R in Eq. (2)-(5) correctly transforms relative velocity into the vessel frame.
- ad hoc to paper Textual prompts plus the hand-written CoT template are sufficient for the LLM to select effective actions.
- domain assumption The Navball bot's logged actions are a valid teacher for fine-tuning.
- ad hoc to paper Fine-tuned LLaMA models generalize beyond the teacher's behavior.
Cite this review
Pith. "Pith review of Large Language Models as Autonomous Spacecraft Operators in Kerbal Space Program." pith.science (2026). https://pith.science/paper/5TED3IFT
@misc{pith2026250519896,
author = {Pith},
title = {Pith review of: Large Language Models as Autonomous Spacecraft Operators in Kerbal Space Program},
year = {2026},
howpublished = {\url{https://pith.science/paper/5TED3IFT}},
note = {Machine review of arXiv:2505.19896}
}
read the original abstract
Recent trends are emerging in the use of Large Language Models (LLMs) as autonomous agents that take actions based on the content of the user text prompts. We intend to apply these concepts to the field of Control in space, enabling LLMs to play a significant role in the decision-making process for autonomous satellite operations. As a first step towards this goal, we have developed a pure LLM-based solution for the Kerbal Space Program Differential Games (KSPDG) challenge, a public software design competition where participants create autonomous agents for maneuvering satellites involved in non-cooperative space operations, running on the KSP game engine. Our approach leverages prompt engineering, few-shot prompting, and fine-tuning techniques to create an effective LLM-based agent that ranked 2nd in the competition. To the best of our knowledge, this work pioneers the integration of LLM agents into space research. The project comprises several open repositories to facilitate replication and further research. The codebase is accessible on \href{https://github.com/ARCLab-MIT/kspdg}{GitHub}, while the trained models and datasets are available on \href{https://huggingface.co/OhhTuRnz}{Hugging Face}. Additionally, experiment tracking and detailed results can be reviewed on \href{https://wandb.ai/carrusk/huggingface}{Weights \& Biases
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[2]
Claude: An ai assistant by anthropic. URL:https://www. anthropic.com/index/claude. accessed: 2024-07-03. Arslanian, H., Fischer, F.,
work page 2024
-
[4]
arXiv preprint arXiv:2005.14165
Language models are few-shot learners. arXiv preprint arXiv:2005.14165 . Carrasco, A., Nedungadi, M., Rodriguez-Fernandez, V., Linares, R.,
arXiv 2005
-
[6]
stanford.edu/2023/03/13/alpaca.html
Alpaca: An instruction-following llama model.https://crfm. stanford.edu/2023/03/13/alpaca.html. Accessed: 2024-07-15. Dao, T.,
work page 2023
-
[7]
URL:https://arxiv.org/abs/2307.08691, arXiv:arXiv:2307.08691.arXiv preprint arXiv:2307.08691
Flashattention-2: Faster attention with better paral- lelism and work partitioning. URL:https://arxiv.org/abs/2307.08691, arXiv:arXiv:2307.08691.arXiv preprint arXiv:2307.08691. Dao, T., Fu, D.Y., Ermon, S., Rudra, A., Ré, C.,
-
[9]
URL:https: //www.deepmind.com/publications/gemini
Gemini: A general-purpose transformer model. URL:https: //www.deepmind.com/publications/gemini. accessed: 2024-07-03. DeepSeek-AI,
work page 2024
-
[11]
URL:https://arxiv.org/abs/2310.06825,arXiv:2310.06825
Mistral 7b. URL:https://arxiv.org/abs/2310.06825,arXiv:2310.06825. Engadget,
-
[13]
Effi- cient iterative linear-quadratic approximations for nonlinear multi-player general- sum differential games, in: 2020 IEEE international conference on robotics and automation (ICRA), IEEE. pp. 1475–1481. 36 Gaudet, B., Linares, R., Furfaro, R.,
work page 2020
-
[14]
URL:https: //arxiv.org/abs/2402.12354,arXiv:2402.12354
Lora: Low-rank adaptation of large language models. URL:https: //arxiv.org/abs/2402.12354,arXiv:2402.12354. Ji, Z., et al.,
Show all 30 references
-
[15]
arXivpreprintarXiv:2305.14390URL:https://arxiv.org/ abs/2305.14390
Hallucinations in large language models: Prevalence, causes, and mitigationstrategies. arXivpreprintarXiv:2305.14390URL:https://arxiv.org/ abs/2305.14390. Kerbal Space Program Wiki,
-
[17]
URL:https:// arxiv.org/abs/2402.09353,arXiv:2402.09353
Dora: Weight-decomposed low-rank adaptation. URL:https:// arxiv.org/abs/2402.09353,arXiv:2402.09353. Mao, J., Qian, Y., Zhao, H., Wang, Y.,
-
[18]
arXiv preprint arXiv:2310.01415
Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415 . Nagel, M., van Baalen, M., Blankevoort, T., Welling, M.,
-
[19]
Accessed on 03/29/2024
Introducing chatgpt.https://openai.com/blog/chatgpt. Accessed on 03/29/2024. OpenAI,
2024
-
[20]
arXiv preprint arXiv:2303.08774
Gpt-4 technical report. arXiv preprint arXiv:2303.08774 . OpenAI,
-
[21]
Accessed: 2024-03-28
Iterating on hyperparameters - fine-tuning guide.https://platform.openai.com/docs/guides/fine-tuning/ iterating-on-hyperparameters. Accessed: 2024-03-28. 37 OpenAI,
2024
-
[22]
URL:https://openai.com/index/ introducing-openai-o1-preview/
Introducing openai o1-preview. URL:https://openai.com/index/ introducing-openai-o1-preview/. accessed: 2025-02-06. Piaget, J.,
2025
-
[24]
arXiv preprint arXiv:2404.00413 URL:https://arxiv.org/abs/2404.00413
Language models are spacecraft operators. arXiv preprint arXiv:2404.00413 URL:https://arxiv.org/abs/2404.00413. Russell, S., Norvig, P.,
-
[25]
arXiv preprint arXiv:2302.13971
Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 . Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.,
-
[26]
arXiv preprint arXiv:2308.11432
A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432 . Wei, J., et al.,
-
[27]
arXiv preprint arXiv:2201.11903 URL:https://arxiv.org/abs/2201
Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 URL:https://arxiv.org/abs/2201. 11903. Wittgenstein, L.,
-
[29]
arXiv preprint arXiv:2305.15486
Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning. arXiv preprint arXiv:2305.15486 . Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., Ma, Y.,
-
[30]
Llamafac- tory: Unified efficient fine-tuning of 100+ language models, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Association for Computational Linguistics, Bangkok, Thailand. p. N/A. URL:http:...
-
[2014]
URL:https://www.engadget.com/ 2014-04-02-kerbal-space-program-asteroid-add-on.html
Kerbal space program asteroid redirect mis- sion add-on now available. URL:https://www.engadget.com/ 2014-04-02-kerbal-space-program-asteroid-add-on.html. accessed: 2024- 07-02. Fridovich-Keil, D., Ratner, E., Peters, L., Dragan, A.D., Tomlin, C.J.,
2014
-
[2018]
URL:https://wiki
Navball. URL:https://wiki. kerbalspaceprogram.com/wiki/Navball. accessed: 2025-02-23. Lee, K., Firat, O., Agarwal, A., Fannjiang, C., Sussillo, D.,
2025
-
[2019]
URL:https://arxiv.org/abs/1910.03771,arXiv:1910.03771
Transformers: State-of-the-art natural language processing. URL:https://arxiv.org/abs/1910.03771,arXiv:1910.03771. Wu, Y., Min, S.Y., Prabhumoye, S., Bisk, Y., Salakhutdinov, R., Azaria, A., Mitchell, T., Li, Y.,
1910 arXiv
-
[2020]
arXiv preprint arXiv:2004.05150
Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150 . 35 Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.,
2004 arXiv
-
[2021]
URL:https://arxiv.org/abs/2102.07350
Prompt programming for large language mod- els: Beyond the few-shot paradigm. URL:https://arxiv.org/abs/2102.07350. accessed: 2024-07-02. Rodriguez-Fernandez, V., Carrasco, A., Cheng, J., Scharf, E., Siew, P.M., Linares, R.,
2024 arXiv
-
[2022]
URL:https://arxiv.org/ abs/2205.14135,arXiv:2205.14135
Flashattention: Fast and memory-efficient exact attention with io-awareness. URL:https://arxiv.org/ abs/2205.14135,arXiv:2205.14135. DeepMind,
-
[2023]
Spacegym: Discrete and differential games in non-cooperative space operations, in: 2023 IEEE Aerospace Conference, pp. 1–12. doi:10.1109/AERO55745.2023. 10115968. Anthropic,
2023
-
[2024]
arXiv preprint arXiv:2412.19437 arXiv:2412.19437
Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 arXiv:2412.19437. available athttps://github.com/deepseek-ai/ DeepSeek-V3. Dettmers, T., Touvron, H., Joulin, A., Grave, E., Raileanu, R., Ott, M.,
-
[2025]
Visual language models as operator agents in the space do- main, in: AIAA SCITECH 2025 Forum, AIAA. p. N/A. URL:https: //arc.aiaa.org/doi/abs/10.2514/6.2025-1543, doi:10.2514/6.2025-1543, arXiv:https://arc.aiaa.org/doi/pdf/10.2514/6.2025-1543. CRFM, S.,
2025 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.