pith. sign in

arxiv: 2606.17986 · v1 · pith:XZBBQZUAnew · submitted 2026-06-16 · 💻 cs.CR

ShellGames: Speculative LLM-Driven SSH Deception

Umberto Salviati , Fabio De Gaspari , Mauro Conti , Luigi Vincenzo Mancini This is my paper

Pith reviewed 2026-06-27 00:09 UTC · model grok-4.3

classification 💻 cs.CR
keywords cyber deceptionLLM shell simulatorSSH honeypotmoving target defensesubversion detectionstateful interactionspeculative executioncommand consistency
0
0 comments X

The pith

ShellGames shows an LLM can run a credible SSH shell by combining chain-of-thought prompting, memory management, speculative execution, sandbox routing, and subversion detection.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that LLMs can overcome their core weaknesses and sustain long interactive deception sessions inside an SSH environment. It does this by layering five techniques that handle correctness, state coherence, response speed, complex command handling, and detection of attempts to expose the fake system. A new benchmark suite tests these aspects directly, and the results plus a twenty-person user study indicate the combined system produces output that users rate as realistic as a genuine shell while covering more commands than standard honeypots. If the approach holds, defenders could maintain believable sessions that keep adversaries uncertain for longer periods. The work focuses on making deception practical rather than on theoretical improvements to language models alone.

Core claim

ShellGames is an LLM-based SSH shell simulator that addresses the limitations of persistent state, output inconsistencies, hallucinations, latency, and behavioral subversion. It does so through automatic chain-of-thought and few-shot learning for command correctness, memory management for system-state coherency, speculative command execution to cut response time, smart routing of interactive commands into a sandbox, and subversion detection that exploits the constrained shell input-output domain. The system is evaluated on a new standardized benchmark covering correctness, consistency, state tracking, and robustness tasks, and a user study confirms that participants find its realism comparab

What carries the argument

The five complementary techniques—chain-of-thought prompting, memory management, speculative execution, sandbox routing, and subversion detection—working together to produce coherent, low-latency, and subversion-resistant shell responses.

If this is right

  • Longer credible sessions become feasible for cyber deception and moving target defense.
  • Adversaries face sustained uncertainty during reconnaissance and exploitation.
  • Benchmark protocols enable direct comparison of future deception systems on the same tasks.
  • User-rated realism can reach levels that match genuine shells under free exploration.
  • Command coverage perceived by users exceeds that of traditional honeypots.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same five-technique pattern could be tested on other interactive command environments such as databases or network devices.
  • Real-world deployment would require measuring whether attackers remain engaged across hours rather than short user-study sessions.
  • Combining the approach with automated system reconfiguration might further raise the cost of successful attacks.
  • The subversion-detection method may apply to any domain with a narrow, well-defined input language.

Load-bearing premise

The five techniques can be combined without introducing new inconsistencies or latency that would undermine the deception in real attacker interactions.

What would settle it

A prolonged live session in which an adversary triggers inconsistent output or evades the subversion detector and correctly labels the system as simulated.

Figures

Figures reproduced from arXiv: 2606.17986 by Fabio De Gaspari, Luigi Vincenzo Mancini, Mauro Conti, Umberto Salviati.

Figure 1
Figure 1. Figure 1: ShellGames high-level overview. (v) subversion detection, which detects subversion attempts and returns a shell￾compliant error response. By combining these components, ShellGames provides a consistent, responsive, and credible environment while laying the groundwork toward agentic cyber deception systems. We present the architecture of Shell￾Games in Section 4.1 and defer implementation details to Appendi… view at source ↗
Figure 2
Figure 2. Figure 2: Consistency comparison across varying sequence lengths. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cumulative response latency over command sequences of varying length. [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: User study responses on a 5-point Likert scale. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
read the original abstract

Cyber deception and Moving Target Defense are promising strategies that aim to disrupt adversaries by increasing uncertainty. However, sustaining long-lived, credible interactive sessions with adversaries remains an open challenge. Large Language Models (LLMs) offer a promising path toward more dynamic deception systems, but suffer from key limitations that fundamentally limit their applicability, including: lack of persistent state, output inconsistencies, hallucinations, latency, and susceptibility to behavioral subversion that may reveal the deception. We propose ShellGames, an SSH shell simulator based on LLM designed to address these limitations. ShellGames combines five complementary techniques: (i) Automatic Chain-of-Thought and few-shot learning to improve correctness; (ii) memory management to maintain system state coherency; (iii) speculative command execution to reduce response latency; (iv) smart routing of complex interactive commands to a sandboxed environment; and (v) subversion detection leveraging the constrained input-output domain of shell environments. To enable systematic evaluation, we introduce a standardized benchmarking protocol and dataset spanning correctness, consistency, state tracking, and robustness tasks. ShellGames achieves $0.898$ command accuracy on correctness ($+5.3pp$ over baselines), $0.918$ sequence-level accuracy on consistency ($+36pp$), $0.98$ state tracking accuracy ($+18.3pp$), and $0.95$ accuracy on robustness ($+37pp$). A user study with $n=20$ participants confirms that ShellGames achieves realism comparable to a real shell under free exploration and outperforms traditional honeypots on perceived command coverage.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes ShellGames, an LLM-based SSH shell simulator for cyber deception that integrates five techniques—automatic Chain-of-Thought/few-shot learning, memory management, speculative command execution, smart routing to sandboxes, and subversion detection—to address LLM limitations like lack of persistent state, inconsistencies, hallucinations, latency, and behavioral subversion. It introduces a new benchmark spanning correctness, consistency, state tracking, and robustness tasks, reporting specific numeric gains over baselines, plus a user study (n=20) claiming realism comparable to real shells and better command coverage than traditional honeypots.

Significance. If the central performance and realism claims hold under more rigorous adversarial testing, the work would represent a meaningful step toward practical, long-lived LLM-driven deception systems in moving target defense, with the five-technique combination and standardized benchmark as potentially reusable contributions. The empirical focus on measurable improvements in command accuracy, sequence consistency, and robustness is a strength relative to purely conceptual proposals.

major comments (2)
  1. [Benchmark protocol and results sections] Benchmark protocol and results sections: the headline claims (0.898 command accuracy +5.3pp, 0.918 sequence accuracy +36pp, 0.98 state tracking +18.3pp, 0.95 robustness +37pp) are presented without reported details on baseline implementations, exact dataset construction, statistical significance tests, or controls against post-hoc metric selection; these omissions are load-bearing because the paper's contribution rests on demonstrating that the five techniques produce verifiable gains rather than artifacts of evaluation design.
  2. [User study section] User study section: the n=20 free-exploration study is described as confirming realism comparable to a real shell, but does not appear to include conditions where participants know the system is deceptive and deliberately probe for inconsistencies, latency artifacts, or state violations over sustained sessions; this directly undercuts the robustness and long-lived deception claims that the five techniques are intended to support.
minor comments (2)
  1. [Abstract/Introduction] The abstract and introduction would benefit from a brief table summarizing the five techniques and the exact limitations each targets.
  2. [Benchmark protocol] Notation for the benchmark tasks (correctness, consistency, etc.) should be defined more explicitly before the numeric results are reported.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which highlight important aspects of evaluation rigor. We respond point-by-point to the major comments below.

read point-by-point responses

  1. Referee: [Benchmark protocol and results sections] Benchmark protocol and results sections: the headline claims (0.898 command accuracy +5.3pp, 0.918 sequence accuracy +36pp, 0.98 state tracking +18.3pp, 0.95 robustness +37pp) are presented without reported details on baseline implementations, exact dataset construction, statistical significance tests, or controls against post-hoc metric selection; these omissions are load-bearing because the paper's contribution rests on demonstrating that the five techniques produce verifiable gains rather than artifacts of evaluation design.

    Authors: We agree that the current presentation would benefit from greater detail to support reproducibility and rule out evaluation artifacts. In the revised manuscript we will expand the Benchmark Protocol and Results sections to include: explicit descriptions of each baseline implementation (models, prompting variants, and hyperparameters); the precise dataset construction process (task generation, validation, and split); results of statistical significance tests (e.g., McNemar or Wilcoxon signed-rank tests with p-values); and an explicit statement that the four metric families were defined a priori from the task taxonomy before any experiments were run. These additions directly address the load-bearing concerns raised. revision: yes

  2. Referee: [User study section] User study section: the n=20 free-exploration study is described as confirming realism comparable to a real shell, but does not appear to include conditions where participants know the system is deceptive and deliberately probe for inconsistencies, latency artifacts, or state violations over sustained sessions; this directly undercuts the robustness and long-lived deception claims that the five techniques are intended to support.

    Authors: The user study was intentionally scoped to free exploration to measure perceived realism and command coverage under naturalistic interaction, which is a necessary first step for deception systems. We acknowledge that it does not incorporate deliberate adversarial probing over sustained sessions. In the revision we will add an explicit limitations paragraph in the User Study section that states this scope limitation and its implications for long-lived deception claims. We will also strengthen the connection to the separate robustness benchmark (0.95 accuracy), which quantifies resistance to subversion attempts, and note planned future adversarial studies. This is a partial revision because the existing study design cannot be altered retroactively, but the discussion and framing can be improved. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical measurements on new benchmark and user study

full rationale

The paper describes an LLM-based SSH simulator evaluated via a new standardized benchmark (correctness, consistency, state tracking, robustness tasks) and an n=20 user study. Reported accuracies (0.898 command accuracy, 0.918 sequence accuracy, etc.) are direct empirical measurements of the combined techniques, not outputs of any derivation, equation, or fitted parameter that reduces to the inputs by construction. No equations, ansatzes, uniqueness theorems, or self-citations appear in the provided abstract or description that would create self-definitional, fitted-prediction, or load-bearing circularity. The work is self-contained as an engineering evaluation against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no free parameters, axioms, or invented entities are identifiable from the given text. The five techniques are engineering choices rather than new postulated entities.

pith-pipeline@v0.9.1-grok · 5814 in / 1424 out tokens · 38398 ms · 2026-06-27T00:09:29.342488+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 1 canonical work pages

  1. [1]

    Springer, 2016

    Sushil Jajodia, V Subrahmanian, Vipin Swarup, and Cliff Wang.Cyber deception, volume 6. Springer, 2016

    2016

  2. [2]

    Springer Science & Business Media, 2011

    Sushil Jajodia, Anup K Ghosh, Vipin Swarup, Cliff Wang, and X Sean Wang.Mov- ing target defense: creating asymmetric uncertainty for cyber threats, volume 54. Springer Science & Business Media, 2011

    2011

  3. [3]

    Toward proactive, adaptive defense: A survey on moving target defense.IEEE Communications Surveys & Tutorials, pages 709–745, 2020

    Jin-Hee Cho, Dilli P Sharma, Hooman Alavizadeh, Seunghyun Yoon, Noam Ben- Asher, Terrence J Moore, Dong Seong Kim, Hyuk Lim, and Frederica F Nelson. Toward proactive, adaptive defense: A survey on moving target defense.IEEE Communications Surveys & Tutorials, pages 709–745, 2020

    2020

  4. [4]

    Mov- ing target defense techniques: A survey.Security and Communication Networks, 2018, 2018

    Cheng Lei, Hong-Qi Zhang, Jing-Lei Tan, Yu-Chen Zhang, and Xiao-Hu Liu. Mov- ing target defense techniques: A survey.Security and Communication Networks, 2018, 2018

    2018

  5. [5]

    Dolos: A novel architecture for moving target defense

    Giulio Pagnotta, Fabio De Gaspari, Dorjan Hitaj, Mauro Andreolini, Michele Cola- janni, and Luigi V Mancini. Dolos: A novel architecture for moving target defense. IEEE Transactions on Information Forensics and Security, 18:5890–5905, 2023

    2023

  6. [6]

    Ctibench: A benchmark for evaluating llms in cyber threat intelligence.Advances in Neural Information Processing Systems, 37:50805–50825, 2024

    Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating llms in cyber threat intelligence.Advances in Neural Information Processing Systems, 37:50805–50825, 2024

    2024

  7. [7]

    Llama-3.1-foundationai- securityllm-base-8b technical report, 2025

    Paul Kassianik, Baturay Saglam, Alexander Chen, Blaine Nelson, Anu Vellore, Massimo Aufiero, Fraser Burch, Dhruv Kedia, Avi Zohary, Sajana Weerawardhena, et al. Llama-3.1-foundationai-securityllm-base-8b technical report.arXiv preprint arXiv:2504.21039, 2025

    work page arXiv 2025

  8. [8]

    Auto- mated tactics planning for cyber attack and defense based on large language model agents.Neural Networks, page 107842, 2025

    Yimo Ren, Jinfa Wang, Zhihui Zhao, Hui Wen, Hong Li, and Hongsong Zhu. Auto- mated tactics planning for cyber attack and defense based on large language model agents.Neural Networks, page 107842, 2025

    2025

  9. [9]

    A survey of large language models (llms) for cybersecurity: Opportunities and directions

    Md Abdur Rahman, Guillermo Francia, Hossain Shahriar, Alfredo Cuzzocrea, Atef Mohamed, Muhammad Umair Khan, and Sheikh Iqbal Ahamed. A survey of large language models (llms) for cybersecurity: Opportunities and directions. In2025 IEEE International Conference on Big Data (BigData), pages 4333–4342. IEEE, 2025

    2025

  10. [10]

    Cyberllminstruct: A pseudo-malicious dataset revealing safety-performance trade-offs in cyber security llm fine-tuning

    Adel ElZemity, Budi Arief, and Shujun Li. Cyberllminstruct: A pseudo-malicious dataset revealing safety-performance trade-offs in cyber security llm fine-tuning. InProceedings of the 18th ACM Workshop on Artificial Intelligence and Security, pages 77–88, 2025

    2025

  11. [11]

    Memory mat- ters: The need to improve long-term memory in llm-agents

    Kostas Hatalis, Despina Christou, Joshua Myers, Steven Jones, Keith Lambert, Adam Amos-Binks, Zohreh Dannenhauer, and Dustin Dannenhauer. Memory mat- ters: The need to improve long-term memory in llm-agents. InProceedings of the AAAI Symposium Series, volume 2, pages 277–280, 2023

    2023

  12. [12]

    Hallulens: Llm hallucination bench- mark

    Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. Hallulens: Llm hallucination bench- mark. InProceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 24128–24156, 2025

    2025

  13. [13]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. InProceedings of the 16th ACM workshop on artificial intelligence and security, pages 79–90, 2023

    2023

  14. [14]

    Automatic chain of thought prompting in large language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. InThe Eleventh International Con- ference on Learning Representations, 2023. 18 Salviati et al

    2023

  15. [15]

    Language models are few-shot learners.Advances in neural information pro- cessing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Pra- fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information pro- cessing systems, 33:1877–1901, 2020

    1901

  16. [16]

    A survey of contemporary open-source honeypots, frameworks, and tools.Journal of Network and Computer Applications, 220:103737, 2023

    Niclas Ilg, Paul Duplys, Dominik Sisejkovic, and Michael Menth. A survey of contemporary open-source honeypots, frameworks, and tools.Journal of Network and Computer Applications, 220:103737, 2023

    2023

  17. [17]

    Spitzner

    L. Spitzner. Honeypots: catching the insider threat. In19th Annual Computer Security Applications Conference, 2003. Proceedings., pages 170–179, 2003

    2003

  18. [18]

    Honeypots: concepts, approaches, and chal- lenges

    Iyatiti Mokube and Michele Adams. Honeypots: concepts, approaches, and chal- lenges. InProceedings of the 45th Annual ACM Southeast Conference, ACMSE ’07, page 321–326, New York, NY, USA, 2007. Association for Computing Machinery

    2007

  19. [19]

    Holz and F

    T. Holz and F. Raynal. Detecting honeypots and other suspicious environments. In Proceedings from the Sixth Annual IEEE SMC Information Assurance Workshop, pages 29–36, 2005

    2005

  20. [20]

    van Eeten, Katsunari Yoshioka, and Tsutomu Matsumoto

    Shun Morishita, Takuya Hoizumi, Wataru Ueno, Rui Tanabe, Carlos Gañán, Michel J.G. van Eeten, Katsunari Yoshioka, and Tsutomu Matsumoto. Detect me if you... oh wait. an internet-wide view of self-revealing honeypots. In2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), pages 134–143, 2019

    2019

  21. [21]

    https://github.com/cowrie/cowrie

    Cowrie. https://github.com/cowrie/cowrie

  22. [22]

    Aiipot: Adaptive intelligent-interaction honeypot for iot de- vices

    Volviane Saphir Mfogo, Alain Zemkoho, Laurent Njilla, Marcellin Nkenlifack, and Charles Kamhoua. Aiipot: Adaptive intelligent-interaction honeypot for iot de- vices. In2023 IEEE 34th Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC), pages 1–6, Sep. 2023

    2023

  23. [23]

    Mysql-pot: A llm-based honeypot for mysql threat protection

    YuqiHu,SiyuCheng,YuanyiMa,ShuangwuChen,FengruiXiao,andQuanZheng. Mysql-pot: A llm-based honeypot for mysql threat protection. In2024 9th Inter- national Conference on Big Data Analytics (ICBDA), pages 227–232, 2024

    2024

  24. [24]

    Llm in the shell: Generative honeypots

    Muris Sladić, Veronica Valeros, Carlos Catania, and Sebastian Garcia. Llm in the shell: Generative honeypots. In2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pages 430–435. IEEE, 2024

    2024

  25. [25]

    Honeyllm: Enabling shell honey- pots with large language models

    Chongqi Guan, Guohong Cao, and Sencun Zhu. Honeyllm: Enabling shell honey- pots with large language models. In2024 IEEE Conference on Communications and Network Security (CNS), pages 1–9. IEEE, 2024

    2024

  26. [26]

    Ai-enhanced hon- eypots: Leveraging llm for adaptive cybersecurity responses

    Jason Aljenova Christli, Charles Lim, and Yevonnael Andrew. Ai-enhanced hon- eypots: Leveraging llm for adaptive cybersecurity responses. In2024 16th Inter- national Conference on Information Technology and Electrical Engineering (ICI- TEE), pages 451–456, 2024

    2024

  27. [27]

    Otal and M

    Hakan T. Otal and M. Abdullah Canbaz. Llm honeypot: Leveraging large language models as advanced interactive honeypot systems. In2024 IEEE Conference on Communications and Network Security (CNS), pages 1–6, 2024

    2024

  28. [28]

    Weber, Marc Feger, and Michael Pilgermann

    Simon B. Weber, Marc Feger, and Michael Pilgermann. Don’t stop believin’: A unified evaluation approach for llm honeypots.IEEE Access, 12:144579–144587, 2024

    2024

  29. [29]

    Ignore previous prompt: Attack techniques for lan- guage models

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for lan- guage models. InNeurIPS ML Safety Workshop, 2022

    2022

  30. [30]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks

    Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. In13th International Conference on Learning Representations, 2025. ShellGames: Speculative LLM-Driven SSH Deception 19

    2025

  31. [31]

    Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics, 12:157–173, 2024

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics, 12:157–173, 2024. A Appendix A.1 Implementation Details As ShellGames is designed to operate as a honeypot, comprehensive data c...

    2024