Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

Neural Linguistic Steganography

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Arithmetic coding with a language model makes hidden messages statistically indistinguishable from the model's own text.

desk verdict A competent engineering paper that applies arithmetic-coding steganography to GPT-2 cleanly, but the abstract's 'preserving security' overstates what the evidence actually supports. read the letter →

arxiv 1909.01496 v1 pith:BVEXPJRZ submitted 2019-09-03 cs.CL cs.CRcs.LG

Zachary M. Ziegler , Yuntian Deng , Alexander M. Rush This is my paper · ORCID
classification cs.CLcs.CRcs.LG
keywords linguisticsteganographyarithmeticcodingneurallanguagemodelsinformationhidingKullback-Leiblerdivergencetextgenerationsteganalysiscovertcommunication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that a classical compression algorithm, arithmetic coding, when run in reverse over the next-word probabilities of a strong neural language model, turns a secret bit string into natural-looking text while making the generated text statistically indistinguishable from the model's own output. If true, this closes the gap between edit-based linguistic steganography, which hides little data, and generation-based schemes, which are detectable in principle. The authors report a measured Kullback-Leibler divergence of about $4\times 10^{-8}$ nats between the cover distribution and the language model distribution at full modulation, and human raters accept the generated next sentences about as often as true next sentences at low compression. The practical payoff is a concrete, efficient way to conceal the existence of a message inside ordinary prose.

What carries the argument

The mechanism is arithmetic coding on $[0,1)$: the secret message is a uniform bit string read as a binary fraction, and at each generation step the language model's conditional distribution partitions the current interval into token bins; the cover text is the sequence of bins containing the fraction. Decoding reverses the walk to recover the bits. The paper adds two knobs, temperature $\tau$ and top-$k$ truncation, to modulate the distribution away from the exact $p_{\mathrm{LM}}$ for quality. The load-bearing identity is the optimality result for arithmetic steganography: for long sequences this procedure gives $q = p_s$, so $H(q) = H(p_s)$, meaning the encoding is simultaneously entropy-optimal and distribution-matching.

What would settle it

Train a binary classifier on long samples of genuine human-written news text versus cover text produced by the unmodulated arithmetic-coding system, held out from the language model's training data; if the classifier's accuracy exceeds chance with high confidence, the near-zero KL to the language model is not translating into indistinguishability from natural language.

Watch

Extended reading notes

Core claim

The central claim is that generation-based linguistic steganography can simultaneously achieve near-perfect statistical security and human-level fluency by using arithmetic coding as the invertible map between uniform message bits and cover text. With the unmodulated language model, the induced cover distribution $q$ has KL divergence $4\times 10^{-8}$ nats from $p_{\mathrm{LM}}$, which is effectively zero and consistent with the theory that arithmetic coding yields $q = p_{\mathrm{LM}}$. With temperature and top-$k$ truncation, the method trades a small amount of this statistical fidelity for more conservative generations; crowdsourced evaluators then rate its one-bit-per-word output statistically indistinguishably from human-written next sentences. The same construction also lets a natural-language message be compressed to bits and re-encoded in unrelated cover text.

Load-bearing premise

The security claim collapses if the pretrained language model's distribution differs detectably from the true distribution of natural language, because the paper measures and optimises only closeness to the model, not to human text.

Editorial extensions

If this is right

  • A statistical adversary who knows only the sender's language model cannot distinguish steganographic cover text from the model's own samples, because the divergence is measured at $4\times 10^{-8}$ nats.
  • The hidden-message capacity is essentially the entropy of the cover distribution, so encoding is nearly as efficient as the language model's own token prediction, around 1 to 5 bits per word across the settings tested.
  • Human readers, even with a news article as context, accept the generated next sentences about as often as true human-written next sentences when the compression is around 1 bit per word.
  • Because arithmetic coding is invertible, natural-language messages can first be compressed to uniform bits and then re-expressed as unrelated cover text, so arbitrary secret text, not just random bits, can be sent this way.
  • Any future improvement in language modeling directly improves both the statistical fidelity and the fluency of the hidden channel, since the coding mechanism is already at its theoretical optimum.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would not equate $D_{\mathrm{KL}}(q\|p_{\mathrm{LM}})$ with $D_{\mathrm{KL}}(q\|P_{\mathrm{true}})$: if a steganalyst uses a language model better than the sender's, the near-zero KL to the sender's model would not protect the message; an explicit cross-model test would decide whether the practical security claim extends beyond a fixed model.
  • The same arithmetic-coding construction could be turned into a public benchmark for linguistic steganography by holding out a strong language model and testing whether any detector can separate samples of $q$ from human text; the current $4\times 10^{-8}$ nats result suggests such a benchmark would need very sensitive detectors.
  • The text-to-text variant (message text to bits to cover text) points toward untraceable messaging over ordinary channels, since the cover text carries no obvious link to the intended content.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes a generation-based linguistic steganography method built on arithmetic coding with a pretrained language model, specifically GPT-2. The sender treats a uniformly random secret bit string as a binary fraction and maps it to a sequence of tokens by repeatedly reading intervals defined by the model's conditional distributions; decoding inverts the mapping. The authors compare this approach to Huffman- and block-based steganographic baselines, report KL divergence between the generated distribution q and the language model distribution pLM, and run an Amazon Mechanical Turk study of human plausibility of generated sentences conditioned on news article contexts. The central reported result is that unmodulated arithmetic coding (tau=1, k=50256) yields DKL(q||pLM) around 4e-8 nats, which the paper interprets as near-perfect statistical security by matching the language model distribution.

Significance. The paper's core technical contribution is sound and useful: arithmetic coding is the correct construction for making a steganographic cover distribution match a chosen model distribution, and the near-machine-epsilon KL value is a strong implementation check that the scheme behaves as theory predicts. The comparison against Huffman and block baselines under the same GPT-2 model is informative, and the release of code and a demo is valuable. However, the security claim is scoped too broadly: the measured KL is an internal consistency check between q and the very model used to generate, not evidence about DKL(q||Ptrue). The authors explicitly acknowledge this limitation in Section 4.2, but the abstract and conclusion use language such as 'preserving security' that overstates the evidence. The paper is therefore a valuable demonstration of a construction and its properties, rather than a demonstration that the generated text is undetectable in the true distribution of natural language.

major comments (3)
  1. [Section 4.2 and Abstract] The abstract claims the method 'preserving security by matching the cover message distribution with the language model distribution,' but the only statistical evidence is DKL(q||pLM) where pLM is the same GPT-2 model used to construct q. Arithmetic coding is designed by construction to drive q toward pLM, so a KL of 4e-8 nats is an internal consistency check, not a measure of security against natural language. The paper acknowledges in Section 4.2 that 'pLM might not match the distribution of natural language' and that the remaining gap is 'a function of language model performance,' but the framing of the headline result should be corrected. Please either (a) explicitly scope all security claims to 'security relative to the chosen language model distribution,' or (b) add an independent evaluation, such as a steganalysis classifier trained on human-written text versus generated text, or a KL estimate against a different large language model than the one used for encoding.
  2. [Figures 3 and 4] No single operating point is shown to achieve both near-zero KL and high human plausibility. The unmodulated configuration with tau=1 and k=50256, which gives the 4e-8 nats KL, is not evaluated in the human study; the human evaluation uses k=300 with temperatures 0.4, 0.7, 1.0, and 1.2, and the configuration that is most human-plausible (tau=0.4, around 1 bit/word) has non-negligible KL on the curve in Figure 3. Thus the conclusion that the system 'can fool both statistical adversaries who analyze the full generative distribution, and human adversaries who analyze a single sample' is not supported by a single reported configuration. The authors should either report human evaluations for the unmodulated configuration, report KL for the human-evaluated configurations in a directly comparable way, or explicitly state that the two properties are demonstrated at different operating points and the system can be tuned between them.
  3. [Section 4.3] The claim that 'at 1 bit/word both methods lead to cover texts statistically indistinguishable from human-written sentences' is not backed by any reported statistical test. The MTurk protocol yields binary judgments, and the paper reports only the mean human judgment score per condition. Without confidence intervals, a significance test against the true-data condition, or a model-based comparison, the phrase 'statistically indistinguishable' is unsupported. Please provide the statistical analysis, or rephrase this claim as a descriptive statement about the mean scores.
minor comments (3)
  1. [Section 1] There is a typo in the sentence 'Traditionally, most practical stenography systems are edit-based'; 'stenography' should be 'steganography'.
  2. [Section 3] The sentence 'yields a distribution q = ps or equivalently DKL(q||ps) = 0 for long sequences' conflates exact equality with asymptotic equality; arithmetic coding with finite precision has small but nonzero quantization error, so the equality holds only in the limit. Please clarify the asymptotic nature of the claim.
  3. [Section 4.4 and Table 1] The qualitative examples are helpful, but some generated cover texts contain obvious factual errors or artifacts (e.g., 'kim ju-ha-teh' in Table 1). Since the human evaluation is central to the paper's claims, a sentence or two discussing how often such artifacts occur and whether they affect the human plausibility judgments would strengthen the presentation.

Circularity Check

1 steps flagged · score 6.0 of 10

The near-zero KL is a constructional identity: security is measured against the same pLM that arithmetic coding is designed to match, so the abstract's 'preserving security' is not tested against natural language.

  1. self definitional [Abstract; Section 4.2 (Metrics); Section 4.3 (Quantitative results)]
    "Since we do not have access to Ptrue we resort to DKL(q||pLM). pLM might not match the distribution of natural language, but it allows for relative comparisons of steganography algorithms. The remaining gap is a function of language model performance, which is not the focus of this work. [...] Most striking, arithmetic coding with the unmodulated language model induces a q distribution with a KL of 4e-8 nats."

    Section 2 defines security as DKL(q||Ptrue), the divergence from the true distribution of natural language. Section 4.2 replaces this with DKL(q||pLM), and Section 3 cites Sallee's theorem that arithmetic coding with goal distribution ps yields q = ps, hence DKL(q||ps) = 0. The reported 4e-8 nats is therefore not a measurement of security against a natural-language adversary; it is an identity forced by the arithmetic-coding construction. The abstract's claim of 'preserving security by matching the cover message distribution with the language model distribution' depends entirely on the unmeasured assumption that GPT-2's pLM approximates Ptrue, which the paper explicitly declines to evaluate. The headline statistic is an internal consistency check, not independent evidence.

full rationale

The paper contains no load-bearing self-citation: the central construction relies on Sallee's external theorem and on the pretrained GPT-2 model, and the human evaluation is an independent external benchmark. The circular step is definitional rather than citation-based. The security objective is restated as closeness to pLM because Ptrue is unavailable, and the method is then constructed to make q equal to pLM; consequently, the headline DKL(q||pLM) ≈ 4e-8 nats is an identity of the scheme, not evidence that generated text is indistinguishable from human text. The paper's own caveat that 'the remaining gap is a function of language model performance' is honest, but it does not rescue the abstract's 'preserving security' framing, since that framing silently treats pLM as if it were Ptrue. The human-fooling results are independent and valid, so the circularity is partial rather than total, reflected in the midpoint score of 6.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the standard arithmetic coding theorem, on the shared-model assumption, and on the unmeasured proxy that pLM approximates natural language. The hand-chosen temperature and top-k parameters affect the trade-off between fluency and bit rate but are not fitted to maximize the headline security claim.

free parameters (2)
  • Temperature tau = 0.4, 0.7, 1.0, 1.2
    Chosen by hand to modulate the language model distribution and trade generation fluency against bits per word. The value changes the measured KL and human judgments.
  • Top-k truncation k = 300 for most experiments, 50256 for the unmodulated model
    Chosen by hand to remove the long tail of the token distribution. The unmodulated setting k=50256 gives the near-zero KL result; the truncated setting affects fluency and the human evaluation.
assumptions (4)
  • standard math Arithmetic coding with a uniformly distributed input produces exactly the target distribution ps, so DKL(q||ps) tends to 0 for long sequences.
    Theorem from Sallee (2004), used in Section 3 to assert q equals ps and that H(q) equals the entropy of ps.
  • standard math An arbitrary message can be losslessly compressed to a uniformly distributed bit string before steganographic encoding.
    Invoked in Section 2 with a citation to Han (2005); this justifies assuming m is uniform without loss of generality.
  • domain assumption The sender and receiver share the exact same language model pLM and the same context, since decoding inverts the arithmetic coding mapping.
    Section 2 defines pLM as part of the key shared by Alice and Bob; any mismatch would break decoding.
  • domain assumption The language model distribution pLM is a good proxy for the true natural language distribution Ptrue in the security metric.
    Section 4.2 substitutes DKL(q||pLM) for DKL(q||Ptrue) because Ptrue is unavailable, and the security interpretation in the abstract tacitly depends on this substitution being reasonable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural Linguistic Steganography." pith.science (2026). https://pith.science/paper/BVEXPJRZ

@misc{pith2026190901496,
  author       = {Pith},
  title        = {Pith review of: Neural Linguistic Steganography},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BVEXPJRZ}},
  note         = {Machine review of arXiv:1909.01496}
}
read the original abstract

Whereas traditional cryptography encrypts a secret message into an unintelligible form, steganography conceals that communication is taking place by encoding a secret message into a cover signal. Language is a particularly pragmatic cover signal due to its benign occurrence and independence from any one medium. Traditionally, linguistic steganography systems encode secret messages in existing text via synonym substitution or word order rearrangements. Advances in neural language models enable previously impractical generation-based techniques. We propose a steganography technique based on arithmetic coding with large-scale neural language models. We find that our approach can generate realistic looking cover sentences as evaluated by humans, while at the same time preserving security by matching the cover message distribution with the language model distribution.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Subliminal Learning: Language models transmit behavioral traits via hidden signals in data

    cs.LG 2025-07 conditional novelty 7.0 of 10

    Teacher language models transfer behavioral traits to students fine-tuned on semantically unrelated number, code, and reasoning traces, especially when the two models share initialization.

  2. Semantic Steganography: A Framework for Robust and High-Capacity Information Hiding using Large Language Models

    cs.CR 2024-12 reject novelty 6.0 of 10

    A new LLM-based steganography framework encodes secret bits in the choice of entities mentioned in generated text, claiming higher capacity and robustness, but its indistinguishability claim is not supported and is co...

Reference graph

Works this paper leans on

23 extracted references · 18 canonical work pages · cited by 2 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Muhalim Mohamed Amin, Mazleena Salleh, Subariah Ibrahim, Mohd Rozi Katmin, and MZI Shamsuddin. 2003. Information hiding using steganography. In 4th National Conference of Telecommunication Technology, 2003. NCTT 2003 Proceedings., pages 21--25. IEEE

    work page 2003

  4. [4]

    Krista Bennett. 2004. Linguistic steganography: Survey, analysis, and robustness concerns for hiding information in text

    work page 2004

  5. [5]

    Ingemar J Cox, Ton Kalker, Georg Pakura, and Mathias Scheel. 2005. Information transmission and steganography. In International Workshop on Digital Watermarking, pages 15--29. Springer

    work page 2005

  6. [6]

    Falcon Dai and Zheng Cai. 2019. Towards near-imperceptible steganographic text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4303--4308, Florence, Italy. Association for Computational Linguistics

    work page 2019

  7. [7]

    Tina Fang, Martin Jaggi, and Katerina Argyraki. 2017. Generating steganographic text with lstms. arXiv preprint arXiv:1705.10742

    arXiv 2017

  8. [8]

    Te Sun Han. 2005. Folklore in source coding: Information-spectrum approach. IEEE Trans. Inf. Theor., 51(2):747--753

    work page 2005

Show all 23 references
  1. [9]

    PR Hefei. 2009. Steganalysis of synonym-substitution based natural language watermarking. International Journal of Multimedia and Ubiquitous Engineering, 4(2)

    2009

  2. [10]

    Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693--1701

    2015

  3. [11]

    Arvind Kumar and Km Pooja. 2010. Steganography-a data hiding technique. International Journal of Computer Applications, 9(7):19--23

    2010

  4. [12]

    Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023

    2016 arXiv

  5. [13]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  6. [14]

    Jorma Rissanen and Glen G Langdon. 1979. Arithmetic coding. IBM Journal of research and development, 23(2):149--162

    1979

  7. [15]

    Frank Rubin. 1979. Arithmetic stream coding using fixed precision registers. IEEE Transactions on Information Theory, 25(6):672--675

    1979

  8. [16]

    P. Sallee. 2004. Model based steganography. Int. Workshop on Digital Watermarking

    2004

  9. [17]

    Mohammad Shirali-Shahreza and M Hassan Shirali-Shahreza. 2007. Text steganography in sms. In 2007 International Conference on Convergence Information Technology (ICCIT 2007), pages 2260--2265. IEEE

    2007

  10. [18]

    Andreas Westfeld and Andreas Pfitzmann. 1999. Attacks on steganographic systems. In International workshop on information hiding, pages 61--76. Springer

    1999

  11. [19]

    Alex Wilson and Andrew D Ker. 2016. Avoiding detection on twitter: embedding strategies for linguistic steganography. Electronic Imaging, 2016(8):1--9

    2016

  12. [20]

    Lingyun Xiang, Yan Li, Wei Hao, Peng Yang, and Xiaobo Shen. 2018. Reversible natural language watermarking using synonym substitution and arithmetic coding. Comput., Mater. Continua, 55(3):541--559

    2018

  13. [21]

    Lingyun Xiang, Xinhui Wang, Chunfang Yang, and Peng Liu. 2017. A novel linguistic steganography based on synonym run-length encoding. IEICE transactions on Information and Systems, 100(2):313--322

    2017

  14. [22]

    Zhong-Liang Yang, Xiao-Qing Guo, Zi-Ming Chen, Yong-Feng Huang, and Yu-Jin Zhang. 2019. Rnn-stega: Linguistic steganography based on recurrent neural networks. IEEE Transactions on Information Forensics and Security, 14(5):1280--1295

    2019

  15. [23]

    Barret Zoph, Marjan Ghazvininejad, and Kevin Knight. 2015. How much information does a human translator add to the original? In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 889--898

    2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.