REVIEW 3 major objections 5 minor 8 references
Using LSTMs to Model the Java Programming Language
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A standard LSTM predicts the next Java statement more accurately than it predicts English words.
desk verdict A small, honest empirical datapoint: a standard LSTM does reach perplexities around 11–22 on a stripped AST-token stream of Java, but the paper's main comparison to English is not head-to-head and the numbers are sold as more meaningful than they are. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is statement-level AST tokenization, which converts each Java statement's abstract syntax tree into a flat, space-delimited token string: underscores join node labels and parentheses encode parent-child relationships, while curly braces are kept as tokens in method bodies. This turns the task of predicting the next statement into ordinary next-token prediction, so the same two-layer dropout-regularized LSTM with a 10,000-token vocabulary used for English can be trained and evaluated on Java. Perplexity, computed as the exponential of mean cross-entropy over the test set, and top-k accuracy are the metrics that carry the comparison.
What would settle it
Train a trigram or another simple frequency-based sequence model on the exact same flattened AST token sequences and compare perplexity and top-1 accuracy with the reported LSTM numbers; if the simple model matches or beats them, or if removing curly-brace tokens collapses the LSTM's accuracy, the claimed effectiveness is an artifact of the restricted token stream.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that after flattening Java method bodies into space-delimited strings of abstract syntax tree node labels, with variable names, function names, and literal values removed but tree structure preserved through parentheses and underscores, an LSTM learns to predict the next statement token with much lower perplexity and much higher top-k accuracy than the same model achieves on English text. The reported Java perplexities range from about 11 to 22, while the English corpus gives 85; reported top-1 accuracies range from about 0.47 to 0.58 for Java, versus 0.269 for English. From this the paper concludes that LSTMs can effectively model the syntactic structure of Java code and that such modeling is suitable for next-statement prediction in method bodies.
Load-bearing premise
The load-bearing premise is that abstract syntax tree node labels with variable names, function names, and literal values removed still constitute Java code in the sense needed to compare modeling Java with modeling English.
Editorial extensions
If this is right
- A standard two-layer dropout LSTM can predict the next statement token in Java method bodies with top-1 accuracy above 0.47 and top-10 accuracy above 0.71 on all four Java repositories tested.
- The same model is more accurate on stripped Java syntax than on English text by the paper's numbers, with perplexity 11-22 versus 85 and top-1 accuracy 0.47-0.58 versus 0.269.
- Because the tokenization preserves tree structure, predicted tokens correspond to syntactic statement templates that can be reconstructed into code skeletons, supporting syntactic template suggestion.
- The model produces similar results across different Java repositories, suggesting it captures a general structural regularity of Java method bodies rather than repository-specific quirks.
Reading between the lines
- Because the token vocabulary is mostly AST node types plus braces, the reported gap over English likely tracks the lower entropy of that restricted alphabet rather than deeper program understanding; a direct entropy or n-gram baseline on the same token stream would separate those explanations.
- Restoring variable names, types, and literal values would probably raise perplexity and lower accuracy, so the practical ceiling for code-completion tools lies in combining structural prediction with a separate mechanism for naming; the paper's own future-work note points along that direction.
- If the structural predictability generalizes across repositories, syntactic template suggestion could be built from a small model and a parser without needing project-specific identifiers, which would make it cheap to retrain on new codebases.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper trains a standard two-layer LSTM with the Zaremba et al. "medium" configuration on preprocessed Java method bodies from four repositories (JDK, Guava, ElasticSearch, Spring Framework) and on the Penn Treebank (PTB) English corpus. Java source is flattened into sequences of Eclipse AST node IDs with braces inserted to preserve method-body structure, discarding identifiers and literals. The paper reports test-set perplexities of 11.3-21.8 for Java versus 85.3 for PTB, and top-1/5/10 accuracies of roughly 0.47-0.58 / 0.65-0.74 / 0.72-0.78 for Java versus 0.27 / 0.47 / 0.55 for PTB. It concludes that LSTMs can model Java code effectively and better than English. The central claim of the abstract is the Java-vs-English comparison, supported by Table 2 and the tokenization description in Sections 2.1-2.4.
Significance. If the central comparison were valid, the paper would show a clean, reproducible result: a generic LSTM predicts next syntactic statement tokens in Java method bodies much more accurately than it predicts next words in English. The experimental setup is standard, the metrics are conventional, and the reported PTB perplexity is consistent with prior work, which lends initial credibility to the raw measurements. The preprocessing pipeline is described in enough detail to be reimplemented, and the idea of syntactic template prediction has practical relevance for code completion and patch generation. However, the significance is contingent on the comparison being controlled for tokenization differences. As it stands, the paper conflates the difficulty of the language with the difficulty of the artificially reduced token-prediction task, so the headline conclusion needs substantial additional evidence before it can be accepted.
major comments (3)
- [§2.2, §2.4, Eq. (1), Table 2] The central Java-vs-English comparison is confounded by a systematic token-level bias. Section 2.2 inserts "{" and "}" into the Java sequences as part of the model input, and Section 2.4 reports that these braces comprise about 35% of all Java tokens; they are removed only from the frequency-distribution analysis, not from the corpus fed to the LSTM. Since perplexity in Eq. (1) averages over all test tokens, a large block of nearly deterministic brace tokens mechanically lowers the Java perplexity and inflates the top-k accuracies in Table 2. The paper should report perplexity and accuracy separately for brace and non-brace tokens, or report a version of Eq. (1) computed after excluding braces, before the abstract's "improvement over English" can be taken as established.
- [§2.1, §2.3, Table 2] The two tasks being compared are not head-to-head. The Java stream is a small alphabet of Eclipse AST node IDs plus parentheses and braces, whereas PTB is full natural language with a 10,000-word lexical vocabulary. Perplexity and top-k accuracy are strongly sensitive to vocabulary size and to the proportion of tokens whose identity is syntactically forced, so the numbers in Table 2 reflect task ease as much as model quality. A fair comparison requires either a syntax-only English stream (for example, PTB tagged with POS tags or flattened CFG productions) or, at minimum, an n-gram baseline on the same Java token stream so the reader can separate "Java templates are predictable" from "LSTMs are good at this task." Without such a control, the claim that the LSTM's Java performance is an improvement over its English performance does not follow.
- [Abstract and §5] The claim that the LSTM "models Java code" and outperforms English overstates what was actually measured. The tokenization in Section 2.1 discards variable names, function names, and literal values, and Section 5 acknowledges that variable types are not included either. The paper therefore demonstrates prediction of syntactic statement templates in a stripped AST representation, not modeling of Java code as written. The abstract and conclusion should be reworded to describe syntactic template prediction, or the experiments should retain identifiers and literals to support the broader claim.
minor comments (5)
- [§3.2] The sentence "We chose word-level perplexity was chosen as the metric" is grammatically garbled and should be rewritten, for example as "We chose word-level perplexity as the metric."
- [Figure 1 caption] The caption reads "AST representation of of the Java statement" with a duplicated "of"; please correct the typo.
- [§2.2] The sentence "The sequence of these tokens forms a 'sentence' which the represents body of a Java method" contains a stray "the"; it should read "which represents the body of a Java method."
- [§5] The conclusion contains the duplicated phrase "Results indicate that indicate that LSTMs can achieve..." and should be edited.
- [§2.3, §3.1] The paper does not report the number of method bodies or tokens per corpus, the random seed, or the number of runs, and it provides no error bars for the numbers in Table 2. Adding these details, or at least a statement that each corpus was run once, would improve reproducibility and help readers judge the significance of the perplexity gap.
Circularity Check
No circularity: perplexities and accuracies are direct held-out test measurements with a fixed external model configuration.
full rationale
This paper does not derive its Java modeling claim from the claim itself. The LSTM uses the standard 'medium' configuration from Zaremba et al. with only the epoch count changed based on validation plateauing, so no parameter is fitted to the Java test set and then relabeled as a prediction. Perplexity and top-k accuracy are empirical quantities computed from test-set cross-entropy using Equations 1-3. The Java corpora are independently built from four open-source repositories via AST tokenization, and PTB is an external English benchmark. The strongest criticism is that the Java token stream is not directly comparable to full PTB English because it discards identifiers/literals and contains highly predictable braces, but that is a validity objection about task equivalence, not a circularity: the reported numbers would still be measurements rather than restatements of the inputs. Citations to Zaremba et al. supply the architecture and the English baseline but are not the source of the Java results. No load-bearing step reduces by construction to its own input, so no circular step is identified.
Assumptions & free parameters
free parameters (2)
- Vocabulary size =
10,000
- Training epochs =
15
assumptions (4)
- domain assumption Flattened AST tokens with variable names and literal values removed retain enough logical structure to represent Java code for prediction.
- domain assumption Statements in a method body are analogous to words in a sentence, so a word-level LSTM can model method bodies as sentences.
- domain assumption A fixed vocabulary of 10,000 tokens with an <unk> metatoken is adequate for both English and Java corpora.
- domain assumption The Penn Treebank is a suitable English baseline against which Java modeling performance can be compared.
Cite this review
Pith. "Pith review of Using LSTMs to Model the Java Programming Language." pith.science (2026). https://pith.science/paper/6FW7346D
@misc{pith2026190811685,
author = {Pith},
title = {Pith review of: Using LSTMs to Model the Java Programming Language},
year = {2026},
howpublished = {\url{https://pith.science/paper/6FW7346D}},
note = {Machine review of arXiv:1908.11685}
}
read the original abstract
Recurrent neural networks (RNNs), specifically long-short term memory networks (LSTMs), can model natural language effectively. This research investigates the ability for these same LSTMs to perform next "word" prediction on the Java programming language. Java source code from four different repositories undergoes a transformation that preserves the logical structure of the source code and removes the code's various specificities such as variable names and literal values. Such datasets and an additional English language corpus are used to train and test standard LSTMs' ability to predict the next element in a sequence. Results suggest that LSTMs can effectively model Java code achieving perplexities under 22 and accuracies above 0.47, which is an improvement over LSTM's performance on the English language which demonstrated a perplexity of 85 and an accuracy of 0.27. This research can have applicability in other areas such as syntactic template suggestion and automated bug patching.
Figures
Reference graph
Works this paper leans on
-
[1]
Mining source co de repositories at massive scale using language modeling
Miltiadis Allamanis and Charles Sutton. Mining source co de repositories at massive scale using language modeling. In Proceedings of the 10th Working Conference on Mining Software Repositories, MSR ’13, pages 207–216, Piscataway, NJ, USA, 2013. IEEE Press
work page 2013
-
[2]
Anh Tuan Nguyen and Tien N. Nguyen. Graph-based statistic al language model for code. In Proceedings of the 37th International Conference on Softwa re Engineering - Volume 1 , ICSE ’15, pages 858–868, Piscataway, NJ, USA, 2015. IEEE Pr ess
work page 2015
-
[3]
Muhammad Asaduzzaman, Chanchal K. Roy, Kevin A. Schneide r, and Daqing Hou. A simple, efficient, context-sensitive approach for code com pletion. Journal of Soft- ware: Evolution and Process , 28(7):512–541, 2016. JSME-15-0030.R3
work page 2016
-
[4]
A utomatic patch generation learned from human-written patches
Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. A utomatic patch generation learned from human-written patches. In Proceedings of the 2013 Inter- national Conference on Software Engineering , ICSE ’13, pages 802–811, Piscataway, NJ, USA, 2013. IEEE Press. Using LSTMs to Model the Java Programming Language 9
work page 2013
-
[5]
Rec urrent neural network regularization
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Rec urrent neural network regularization. CoRR, abs/1409.2329, 2014
arXiv 2014
-
[6]
Eclipse documentation on the AST cla ss
Eclipse Foundation. Eclipse documentation on the AST cla ss. http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2Fdom%2FAST.html,
-
[7]
Fro m feedforward to re- current lstm neural networks for language modeling
Martin Sundermeyer, Hermann Ney, and Ralf Schl¨ uter. Fro m feedforward to re- current lstm neural networks for language modeling. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP) , 23(3):517–529, 2015
work page 2015
-
[8]
A par allel-fusion rnn-lstm architecture for image caption generation
Minsi Wang, Li Song, Xiaokang Yang, and Chuanfei Luo. A par allel-fusion rnn-lstm architecture for image caption generation. In Image Processing (ICIP), 2016 IEEE International Conference on , pages 4448–4452. IEEE, 2016
work page 2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.