Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

LLMHoney: A Real-Time SSH Honeypot with Large Language Model-Driven Dynamic Response Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMHoney is an SSH honeypot that combines an LLM's dynamic shell replies with an instant dictionary cache; its 13-model benchmark finds a workable accuracy-latency balance.

desk verdict A useful but under-measured engineering report: the hybrid cache+LLM design makes sense, but the evaluation benchmarks raw LLM inference, not the actual system. read the letter →

arxiv 2509.01463 v1 pith:6ZPGKEVL submitted 2025-09-01 cs.CR

Pranjay Malhotra This is my paper
classification cs.CR
keywords SSHhoneypotlargelanguagemodelsshellemulationdeceptionvirtualfilesystemlatencybenchmarkdynamicresponsegenerationrealism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports the design and benchmarking of LLMHoney, an SSH honeypot that passes attacker commands to a large language model instead of a static script, while a dictionary cache serves common commands instantly. It claims this hybrid keeps responses realistic enough for an interactive shell and makes the common-command path fast and exact. On a testbed of 138 Linux commands across 13 LLM backends, it finds Gemini-2.0 and moderate-size open models (Qwen2.5-1.5B, Phi3-3.8B) produce the most plausible replies with mean latencies near 3 seconds, while smaller models hallucinate at rates that would give them away. If true, it shows operational LLM honeypots are feasible without fine-tuning, at a cost of 5-20x compute and reduced concurrency relative to static honeypots. The paper itself limits the claim: validation is a laboratory benchmark, not a live-attacker test.

What carries the argument

The load-bearing element is the command-processing loop: a dictionary-based virtual filesystem and command-state tracker serves known commands instantly, while an LLM engine generates replies for cache misses. The dictionary cache makes exactness and latency separable—known commands are deterministic and immediate, novel commands get flexible, context-aware generation. State tracking updates the prompt with files and process information, which guides the LLM and reduces contradictions with earlier outputs.

What would settle it

A blind transcript test would settle the realism claim: collect 50 LLMHoney sessions and 50 real shell sessions over equivalent commands, and ask experienced Linux users to label each. If judges identify the honeypot at well above chance, the central realism claim is falsified; if they cannot, it is supported.

Watch

Extended reading notes

Core claim

The central discovery is that a pre-trained LLM, steered only by prompt engineering and held to a tracked virtual filesystem state, can stand in for a Linux shell well enough that most of a 138-command testbed produces outputs similar to ground truth, and that model choice dominates realism. Gemini-2.0-flash led with BLEU 0.245, cosine similarity 0.405, and Jaro-Winkler 0.711 at 3.17 s latency; Phi3-3.8B had the lowest hallucination rate at 5.8%. A dictionary cache removes the common-command critical path entirely, so the LLM is only exercised on novel inputs. The paper further claims the LLM route beats static honeypots in coverage and adaptivity, while conceding 5-20x compute overhead and

Load-bearing premise

The central claim depends on treating similarity to 138 curated reference outputs as a stand-in for deceiving real attackers; the paper concedes it tested no live adversaries and that the command set is far smaller than needed.

Editorial extensions

If this is right

  • Common commands such as uname -a, ls, and cat /etc/passwd are served instantaneously and exactly by the dictionary cache, so the LLM's multi-second latency only affects novel or unusual inputs.
  • Model choice materially changes deception quality: top models like Gemini-2.0 achieve the highest similarity scores, while sub-1B models hallucinate 15.8-25.9% of the time, likely enough for detection.
  • A workable, untuned LLM honeypot can be assembled from pre-trained models with prompt engineering plus state tracking, without fine-tuning on honeypot logs.
  • Deployment costs are 5-20x a static honeypot and concurrent sessions drop to roughly 5-10 on standard hardware, so LLM honeypots fit targeted engagement more than mass-scale scanning.
  • No single backend dominates: Gemini-2.0 maximizes fidelity, Phi3-3.8B minimizes hallucinations, so the right choice depends on whether engagement quality or consistency safety matters more.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark suggests a production version should treat the dictionary cache as a learning component: as real attack logs accumulate, frequently seen novel commands can be promoted into the cache, shifting the LLM's workload toward genuinely novel probes and cutting average latency well below 3 seconds.
  • The reported hallucination rates make a post-generation filter (regex or a small classifier) look necessary for deployment; the paper lists it as future work, but the measured 5.8-25.9% rates imply it is load-bearing, not optional.
  • A session-level 'Turing test'—having experienced users or a judge model distinguish LLMHoney transcripts from real shell transcripts—would be a stronger test than the similarity metrics, and the paper's single-command benchmark cannot rule out failure on sequential multi-step attacks.
  • The memory-delta measurements may undercount long-session accumulation because the pipeline restarts the inference server on timeout; sustained sessions need a separate peak-memory test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents LLMHoney, an SSH honeypot that places an LLM in the command-response loop and combines it with a dictionary-based virtual filesystem cache. The authors benchmark 13 LLM backends (0.36B–3.8B, plus Gemini API models) on a fixed set of 138 shell commands, reporting BLEU, cosine, Jaro–Winkler, latency, memory, and hallucination rates. They conclude that Gemini-2.0 and moderately sized models such as Qwen2.5-1.5B and Phi3-3.8B give the best trade-off, and that the dictionary cache makes common commands 'instantaneous and exact'. The paper positions the work as a step toward LLM-based honeypots that can engage attackers more realistically than static systems like Cowrie, while acknowledging that only laboratory validation was performed.

Significance. If properly supported, the paper would offer a useful comparative evaluation of LLMs for shell emulation and a concrete prototype architecture. The authors are commendably explicit about limitations (Section VII) and provide a claimed reproducible evaluation pipeline. However, as it stands, the central claim that the hybrid cache+LLM system 'achieves a balance between authenticity and performance' is not empirically demonstrated, and the headline reliability metric (hallucination rate) is undefined. The contribution is therefore a promising prototype plus a partial benchmark, not yet a validated system. The acknowledged gaps — small command set, no real attackers, no baseline comparison — further limit the strength of the conclusions.

major comments (4)
  1. [Table III / Section IV.D] The hallucination rate is a headline metric (5.8%–25.9%), but the detection procedure is never defined. Section IV.D describes only similarity metrics; Section VIII lists automated hallucination detection as future work, contradicting its use in the results. Without a precise operational definition (e.g., whether it is based on substring matching, a classifier, or manual inspection), the reliability claims are not reproducible or interpretable. This is load-bearing because model selection is driven by hallucination rates.
  2. [Section IV.D / Section V] The evaluation pipeline does not exercise the dictionary cache. The core loop 'Execute each command' records wall-clock latency and memory delta with no reference to a cache lookup, so Table II latencies are raw LLM inference times. Section V's assertion that the cache makes common commands 'instantaneous and exact' is unsupported: no cache-hit rate, end-to-end session latency, or fraction of commands served by the cache is measured. Consequently, the claimed 'balance between authenticity and performance' for the hybrid system is not tested.
  3. [Section IV.D] The success flag is defined as 'cosine similarity > 0.4 or jaro winkler similarity > 0.4'. This threshold is arbitrary and unanchored to any external criterion for what a human attacker would accept as a plausible shell response. Since the paper's accuracy conclusions (e.g., 'most reliable and accurate responses') rest on this threshold, the reported success rates are not interpretable. The authors should either justify the threshold by calibration or report distributions without a binary success cut.
  4. [Section VI.F / Section VII.A] The paper explicitly concedes that 'without direct baseline evaluation, performance comparisons remain speculative' (VI.F) and that the 138-command, isolated-command evaluation 'falls substantially short of comprehensive validation requirements' (VII.A). These admissions undermine the central claim that LLMHoney improves attacker engagement and realism over traditional honeypots. A comparison with Cowrie or another standard honeypot, even on the same command set, is necessary to support the comparative claims in the abstract and conclusion.
minor comments (4)
  1. [Abstract / Section IV] The abstract lists exact-match, Levenshtein similarity, and other metrics that are not reported in the results tables or figures. Either present these metrics or remove them from the abstract.
  2. [Table II / Table III] The model ordering differs between the tables (e.g., Gemini-1.5 appears first in Table II but third in Table III). Consistent ordering would aid readability.
  3. [Section V / Figure 2] 'Token Accuracy' is shown in Figure 2 but is not defined in the methodology beyond 'Token-level accuracy via NLTK’s word_tokenize'. Specify how token accuracy is computed and report it in a table.
  4. [Section IV.A / Table II] Memory overhead for API-based Gemini models reflects only the client-side Python process, not the server-side cost; comparing this to local model memory usage is apples-to-oranges. This should be acknowledged or the metric redefined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark is anchored to external ground-truth shell outputs and standard similarity metrics.

full rationale

The paper's central empirical claims are derived by comparing raw LLM outputs against an external ground-truth dataset (138 Linux commands with expected output from a real shell) using standard similarity metrics (BLEU, cosine, Jaro-Winkler, Levenshtein, exact match). No equation or fitted parameter is defined in terms of the target outcome. The success threshold (cosine > 0.4 or Jaro-Winkler > 0.4) is an arbitrary evaluation criterion, not a fitted input, so it does not make the reported accuracy results true by construction. The claim that the dictionary cache makes common commands 'instantaneous and exact' is a design property—storing exact outputs and returning them on lookup is definitional, not a statistically forced prediction—and it is not presented as a benchmark result. There are no load-bearing self-citations: references [1]–[3] are prior work by other authors, and the paper does not invoke a uniqueness theorem or adopt an ansatz via citation. The paper's own limitations (Section VII: small command set, laboratory-only validation, lack of direct Cowrie comparison) are validity gaps, not circularity. Thus no circular step can be exhibited with a quote showing a reduction of the claimed result to its inputs.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

All load-bearing inputs are either the author-curated command dataset, the unstated hallucination counting rule, or the self-defined success threshold. The strongest external anchor is the ground truth of real shell outputs; the weakest is the untested link between similarity scores and actually fooling an attacker.

free parameters (2)
  • success threshold (cosine or Jaro-Winkler > 0.4) = 0.4
    Section IV.D defines "Success flag if cosine similarity > 0.4 or jaro winkler similarity > 0.4". The threshold is hand-chosen with no justification and determines the per-model success summaries.
  • hallucination detection procedure = not described
    Table III reports Halluc.% for every model, and the abstract cites these rates, yet Section IV never states how a hallucination is detected or counted; the reported rates are therefore not reproducible from the text.
assumptions (5)
  • domain assumption Ground-truth outputs in commands.csv accurately represent a real Linux shell.
    Section IV.B states each row provides "the ground-truth response from a real shell"; no description of how these were collected or validated.
  • domain assumption Lexical/semantic similarity to ground truth is a valid proxy for deception effectiveness.
    Used throughout Section V to rank models; Section VII.B concedes the experiments "cannot validate deception effectiveness against sophisticated adversaries".
  • domain assumption The 138 hand-curated commands are representative of attacker behavior.
    Section IV.B selects commands across categories; Section VII.A admits this "falls substantially short of comprehensive validation requirements" and that 1,000+ variations are typical.
  • ad hoc to paper The success criterion (cosine or Jaro-Winkler > 0.4) is an acceptable definition of a successful emulation.
    Section IV.D; no external standard supports the 0.4 cutoffs.
  • standard math Standard implementations of BLEU, Jaro-Winkler, Levenshtein and TF-IDF cosine are applied correctly.
    Section IV.D lists NLTK, Jellyfish, difflib and scikit-learn; no unit checks or sample verification are shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMHoney: A Real-Time SSH Honeypot with Large Language Model-Driven Dynamic Response Generation." pith.science (2026). https://pith.science/paper/6ZPGKEVL

@misc{pith2026250901463,
  author       = {Pith},
  title        = {Pith review of: LLMHoney: A Real-Time SSH Honeypot with Large Language Model-Driven Dynamic Response Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ZPGKEVL}},
  note         = {Machine review of arXiv:2509.01463}
}
read the original abstract

Cybersecurity honeypots are deception tools for engaging attackers and gather intelligence, but traditional low or medium-interaction honeypots often rely on static, pre-scripted interactions that can be easily identified by skilled adversaries. This Report presents LLMHoney, an SSH honeypot that leverages Large Language Models (LLMs) to generate realistic, dynamic command outputs in real time. LLMHoney integrates a dictionary-based virtual file system to handle common commands with low latency while using LLMs for novel inputs, achieving a balance between authenticity and performance. We implemented LLMHoney using open-source LLMs and evaluated it on a testbed with 138 representative Linux commands. We report comprehensive metrics including accuracy (exact-match, Cosine Similarity, Jaro-Winkler Similarity, Levenshtein Similarity and BLEU score), response latency and memory overhead. We evaluate LLMHoney using multiple LLM backends ranging from 0.36B to 3.8B parameters, including both open-source models and a proprietary model(Gemini). Our experiments compare 13 different LLM variants; results show that Gemini-2.0 and moderately-sized models Qwen2.5:1.5B and Phi3:3.8B provide the most reliable and accurate responses, with mean latencies around 3 seconds, whereas smaller models often produce incorrect or out-of-character outputs. We also discuss how LLM integration improves honeypot realism and adaptability compared to traditional honeypots, as well as challenges such as occasional hallucinated outputs and increased resource usage. Our findings demonstrate that LLM-driven honeypots are a promising approach to enhance attacker engagement and collect richer threat intelligence.

Figures

Figures reproduced from arXiv: 2509.01463 by the authors.

Figure 1
Figure 1. shows the overall architecture of LLMHoney. At a high level, we have five main components: configuration, network listener, authentication manager, session handler & LLM engine, and logging & storage [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Average similarity metrics for each model: Cosine Similarity, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Mean response latency (red bars, left axis) and absolute memory [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Trade-off between average response time and Cosine Similarity. Each [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots

    cs.CR 2026-05 unverdicted novelty 7.0 of 10

    Honeyval evaluates LLM HTTP honeypots with AI attackers and shows they produce longer interactions, lower detection rates, and cost advantages over rule-based baselines.

  2. SoK: Honeypots & LLMs, More Than the Sum of Their Parts?

    cs.CR 2025-10 unverdicted novelty 7.0 of 10

    A systematization of knowledge paper that taxonomizes honeypot detection vectors, synthesizes LLM-honeypot literature into canonical architecture and evaluation methods, and proposes a roadmap for autonomous deception...

  3. AdvancedShelLM: A Stateful Multi-Agent LLM Honeypot for SSH Deception

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    AdvancedShelLM deploys a manager-worker multi-LLM architecture and stateful filesystem for SSH honeypots, reporting up to 99% unit-test pass rates and evidence that its outputs alter real attacker behavior in deployment.

Reference graph

Works this paper leans on

7 extracted references · 5 canonical work pages · cited by 3 Pith papers

  1. [1]

    H. T. Otal and M. A. Canbaz, “LLM Honeypot: Leveraging Large Language Models as Advanced Interactive Honeypot Systems,“ arXiv preprint arXiv:2409.08234, 2024. [Online]. Available: https://arxiv.org/ abs/2409.08234

    arXiv 2024

  2. [2]

    LLM in the Shell: Generative Honeypots

    M. Sladi ´c, V . Valeros, C. Catania, and S. Garc ´ıa, ““LLM in the Shell: Generative Honeypots,”“ in Proc. IEEE EuroSP Workshops , 2024 (to appear). [Online]. Available: arXiv:2309.00155

    work page Pith review arXiv 2024

  3. [3]

    “HoneyGPT: Breaking the Trilemma in Terminal Honeypots with Large Language Models,

    Z. Wang, J. You, H. Wang, T. Yuan, S. Lv, Y . Wang, and L. Sun, ““HoneyGPT: Breaking the Trilemma in Terminal Honeypots with Large Language Models,”“ arXiv:2406.01882, 2023

    arXiv 2023

  4. [4]

    “Cowrie 2.5.0 Documentation,

    M. Oosterhof, ““Cowrie 2.5.0 Documentation,”“ 2021. [Online]. Avail- able: https://cowrie.readthedocs.io

    work page 2021

  5. [5]

    “Transformers: State-of-the-Art Natural Language Pro- cessing,

    T. Wolf et al., ““Transformers: State-of-the-Art Natural Language Pro- cessing,”“ in Proc. EMNLP (Systems Demonstrations), 2020, pp. 38–45

    work page 2020

  6. [6]

    “Ollama: Run & deploy LLMs easily,

    ““Ollama: Run & deploy LLMs easily,”“ 2023. [Online]. Available: https://ollama.ai

    work page 2023

  7. [7]

    “BLEU: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, ““BLEU: a method for automatic evaluation of machine translation,”“ in Proc. 40th ACL , 2002, pp. 311–318

    work page 2002

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.