Pith. sign in

REVIEW 4 major objections 5 minor 24 references

Neural Poetry: Learning to Generate Poems using Syllables

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A syllable-level neural language model, pretrained on modern Italian and the poet's other works, produces tercets that non-expert judges mark as genuine Dante 28% of the time.

desk verdict A clean, modestly novel system for Italian tercet generation; the perplexity transfer results are credible, but the headline human-eval claim is entangled with a hand-designed selection filter and needs a baseline/ablation before it carries weight. read the letter →

arxiv 1908.08861 v2 pith:LMTAETNX submitted 2019-08-23 cs.CL

classification cs.CL
keywords PoemGenerationTransferLearningLanguageModelsRecurrentNeuralNetworksNaturalsyllable-basedmodelDanteAlighierihumanevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a neural language model operating on syllables, rather than words or characters, can generate new tercets in the style of Dante's Divine Comedy with plausible form. After pretraining on a large modern Italian corpus and then on Dante's non-poetic prose and minor poems before fine-tuning on the Divine Comedy, the model's output was judged to be real Dante by non-expert readers 28% of the time, versus 64% for genuine Dante tercets—a relative difference of 56.25%. Expert judges rated the generated rhymes highly and perceived traces of Dante's style, though they found the text weak in meter and emotional content. The paper thus demonstrates that form-focused, stylized poem generation is feasible with a relatively simple LSTM, and that syllable-level tokens carry much of the stylistic signal.

What carries the argument

The machinery is the syllable-based language model (sy-LM): text is split into syllables by Italian hyphenation rules, special tokens mark word boundaries, verse ends, and tercet ends, and an LSTM with tied input/output embeddings predicts the next syllable. Because the vocabulary is syllables rather than words, the model naturally tracks hendecasyllabic meter and rhyme and tolerates vocabulary drift between modern Italian and Dante's Tuscan. Training proceeds in stages: a large modern Italian web corpus first, then Dante's prose and minor poems, then the Divine Comedy. Generation samples tercets stochastically and ranks them with a four-part score—correct tercet length, three 11-syllable verses, chained ABA rhyme, and use of Divine Comedy vocabulary—so the selected output respects terza rima form.

What would settle it

Run a preregistered Turing test with a representative sample of, say, 100 Italian-speaking non-specialists, each classifying 20 balanced tercets in random order, and check agreement statistics; if sy-LM's real-mark rate is not near 28% or Dante's not near 64% with the same relative gap, the central authenticity claim is disconfirmed.

Watch

Extended reading notes

Core claim

The central claim is that a syllable-token LSTM language model, trained in a multi-stage transfer procedure, produces Dantean tercets that a generic population of judges frequently cannot distinguish from Dante's own. On a per-judge test with 13 non-expert judges, sy-LM output was labeled as real 28% of the time and real Dante tercets 64% of the time; a post-hoc split shows that less-capable judges actually preferred the generated tercets over Dante's. Four expert judges, scoring 20 tercets each, gave the generated text an average rhyme score of 3.37 out of 5, compared with 4.50 for Dante, and a style score of 2.49 versus 4.34, while emotion and meter scored low (1.73 and 1.90). The overall picture is that the model captures surface form and some stylistic resemblance, but not strong semantic or emotional coherence.

Load-bearing premise

The authenticity percentages rest on a small human panel—13 non-expert judges for the 28% versus 64% comparison—and the paper's own post-hoc split shows the panel is heterogeneous, so the headline rates may not match a broader population.

Editorial extensions

If this is right

  • The same syllable-based pipeline can be pointed at other syllabic poetries, including English verse, by swapping the hyphenation rules and the target corpus.
  • Multi-stage transfer—general language, then the author's non-poetic prose and minor poems, then the target poem—is a reusable recipe for low-resource stylistic imitation.
  • Enforcing meter, rhyme, and tercet length with a lightweight post-hoc scoring function yields recognizable poetic form without separate learned meter or rhyme modules.
  • Because syllable counts are explicit in the model, the scoring criteria could directly become reward signals for reinforcement learning, the direction the authors outline as future work.
  • The model's ceiling is form and style: low expert emotion and meter scores indicate that pure next-syllable prediction does not provide semantic or emotional coherence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the gap between expert rhyme scores (3.37) and expert style scores (2.49) suggests the syllable-level signal is mostly phonological rather than lexical-semantic, so adding a word-sense or content-conditioning layer would be the natural next experiment.
  • My inference: because the published 28% authenticity rate measures the model plus its scoring filter, the language model alone might perform differently; ablating the scoring function would isolate the filter's contribution.
  • My inference: the post-hoc split into less- and more-capable judges implies the headline relative difference is panel-dependent, so the result is an existence proof rather than a stable population estimate.
  • My inference: for a new target poet, the first two pretraining stages could likely be reused unchanged since they are author-agnostic, requiring only a final fine-tuning run on the new poet's corpus.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a syllable-based LSTM language model (sy-LM) for Italian poetry generation, trained on the Divine Comedy with a multi-stage transfer-learning pipeline that first pre-trains on modern Italian web text (PAISA') and then on Dante's other prose and poems before fine-tuning on the Divine Comedy. Generation is performed by Monte Carlo sampling of 2,000 tercets, followed by a hand-designed scoring filter R(x) that averages four scores: tercet form (R1), hendecasyllabic meter (R2), ABA rhyme (R3), and Dante-vocabulary word usage (R4). The authors report perplexity results showing that transfer learning improves the language model, and they present human evaluations: 13 non-expert judges marked generated tercets as 'real' 28% of the time versus 64% for real Dante tercets, while 4 expert judges rated generated tercets on readability, emotion, meter, rhyme, and style, with particularly positive rhyme scores.

Significance. If the reported results hold, the paper makes a useful contribution to low-resource poetic style generation: the syllable-level representation is a sensible inductive bias for Italian prosody, and the multi-stage transfer-learning result (PAISA' -> DP -> DC) is clearly demonstrated by the perplexity numbers in Table 1. The main architectural idea is simple, reproducible, and well aligned with the formal properties of terza rima, and the authors should be credited for reporting a human evaluation with both non-expert and expert judges rather than relying on automated metrics alone. The qualitative claims, however, are only as strong as the evaluation protocol: the authenticity rate in Table 2 is computed on a heavily filtered subset of generated tercets without an equally filtered set of real Dante tercets, and the small judge panels and possible denominator inconsistencies make the headline percentages less reliable. The central idea is defensible, but the load-bearing evaluation needs substantial strengthening before the qualitative claims can be accepted at face value.

major comments (4)
  1. [Section 4.1, Table 2 and Table 4] The human authenticity test is asymmetric: the 2,000 generated tercets are filtered by the scoring function R(x), and only the highest-scoring outputs are shown to judges, whereas the real Dante tercets in the comparison are never passed through R1-R4 or through the same selection procedure. Consequently, the 28% versus 64% result conflates the quality of the neural model with the quality of the hand-designed scorer, since a trivial or even random generator could in principle reach high scores on R2 (counting eleven syllables), R3 (ABA rhyme), and R4 (Dante vocabulary). To identify the contribution of sy-LM, the authors should evaluate (i) unfiltered sampled tercets from sy-LM, (ii) real Dante tercets that have been passed through the same scoring filter and selected in the same way, and (iii) a non-neural baseline such as random syllable sequences or a simple n-gram model subjected to the same R-scoring and selection. Without these controls, the central qualitative claim measures the full pipeline rather than the model, and the contribution of syllable-based transfer learning is not identifiable.
  2. [Section 4.1, Table 2] The reported percentages are not consistent with the stated experimental setup. With 13 non-expert judges each evaluating 10 tercets (5 generated and 5 real), the number of generated judgments is 65 and the number of real judgments is 65. The reported sy-LM rate of 28% and Poet rate of 64% correspond to 18.2 and 41.6 judgments, respectively, which are not integers. Likewise, Table 4 reports per-tercet fooling rates of 88.00%, 55.56%, and 45.45%, none of which is a multiple of 1/13. The authors should clarify how these percentages are computed, state the exact denominators for each reported rate, and identify which judge group (non-expert, expert, or combined) produced the per-tercet values in Table 4. If a per-judge or per-item count was averaged, that should be explicitly described; otherwise the numerical claims are not reproducible.
  3. [Section 4.1, Table 3] The expert evaluation is based on only four judges, each providing scores for 10 generated tercets and 10 real tercets, but the paper reports only average scores for the generated tercets and a single 'Poet (Average)' row, without per-judge scores for the real tercets, without variance, and without any significance test. The conclusion that 'experts perceived Dante's style and rhymes in the generated text' is supported only by a rhyme average of 3.37 against a real-poem average of 4.50 and a style average of 2.49 against 4.34; these differences are large, and the claim should be phrased with appropriate caution. At minimum, the authors should report the full score distributions, including the real tercet scores for each expert, and perform a paired or mixed-effects analysis given the repeated-measures design.
  4. [Section 4.1, Table 4] The examples in Table 4 reveal very high variance in per-tercet human judgments: the first tercet was marked as real by 88.00% of evaluators, while the last one never fooled anyone. Reporting only the average of such a skewed distribution, as in Table 2, hides the fact that a small subset of high-scoring outputs may drive the headline result. The authors should report the distribution over generated tercets, for example the median and interquartile range of the authenticity rates, and should avoid presenting the selected examples as representative without this context. This issue is directly connected to the scoring-filter asymmetry raised above, because the selected examples are exactly those that maximize R(x).
minor comments (5)
  1. [Section 1] The text contains a typographical error: 'by reading the Divide Comedy' should read 'by reading the Divine Comedy'.
  2. [Section 3.2] The notation in Eq. (8) is slightly inconsistent: R4(x) is defined as a sum over words w in x, but f_w(x_i) is written as an elementwise function of word tokens; please align the notation so that the scoring function is clearly defined for a word token.
  3. [Section 3] The syllabification procedure is described only as 'the most common Italian hyphenation rules' with 'rare exceptions'; since the entire model operates on syllables, the authors should provide more detail on the segmentation algorithm and, ideally, a small quantitative evaluation of its accuracy on an Italian test sample.
  4. [Section 4] The paper reports hyperparameter values (embedding size 300, LSTM hidden size 1024, dropout 0.3, batch size 32, learning rate 0.001) but does not specify the optimizer, the number of training epochs, or the early-stopping criterion used for the PAISA' and DP pre-training stages; these details are needed for reproducibility.
  5. [Section 4.1] The authors should state exactly how many generated tercets were selected by the scoring procedure and shown to judges in each of the two experiments (e.g., whether each non-expert judge saw a unique set of five generated tercets or whether the same five were reused). Without this information, the number of distinct generated tercets evaluated, and hence the effective sample size, remains unclear.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the generation pipeline, hand-set scoring filter, transfer learning evaluation, and human judgments are not constructed to reproduce their own inputs.

full rationale

The paper's derivation chain is: train a syllable-level LSTM on the Divine Comedy with transfer learning from PAISA' and Dante's other works; sample tercets; filter them with explicit, hand-set scores R1-R4 for tercet shape, hendecasyllable count, ABA rhyme, and Divine-Comedy vocabulary; and then evaluate the filtered outputs with non-expert and expert judges. None of these steps defines its output in terms of the claimed conclusion. The scoring constants (a=0.05, b=1) and rules in Eqs. (5)-(8) are stated in the paper and are not fitted to the human ratings, so the human-judgment results are external evidence rather than a renaming of the filter. The perplexity results in Table 1 are measured on held-out Divine Comedy tercets, so the transfer-learning claim is independently testable. The only self-citation is reference [12] in related work, used as background on character/subword representations, and it is not load-bearing. The selective scoring of 2,000 sampled tercets means the reported authenticity rates characterize the full generation-plus-selection system rather than the raw LSTM alone; that is an evaluation-scope caveat, not circularity. No step reduces by construction to its inputs.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new conceptual entities such as particles or forces. It relies on standard deep learning components and a set of domain assumptions about syllable segmentation, cross-era transfer, the sufficiency of the language modeling objective, and the validity of human authorship attribution. The main free parameters are architectural and training hyperparameters plus the hand-set scoring constants in the selection mechanism.

free parameters (6)
  • Syllable embedding dimension d = 300
    Selected by validation perplexity in Section 4; the central model depends on this capacity.
  • LSTM state size = 1024
    Selected by validation perplexity in Section 4.
  • Dropout probability = 0.3
    Set for state neurons to regularize; selected during validation.
  • Batch size = 32
    Validated during pre-training; best result reported in Section 4.
  • Learning rate = 0.001
    Validated during pre-training; best result reported in Section 4.
  • R4 scoring weights (a, b) = a=0.05, b=1
    Hand set in Eq. (8) to encourage Divine Comedy vocabulary words and strongly discourage invalid words; these constants affect which tercets are selected for evaluation.
assumptions (4)
  • domain assumption Italian hyphenation rules correctly segment words into syllables for the poetry corpus.
    Section 3 states a module 'follows the most common Italian hyphenation rules that, apart from rare exceptions, correctly divides words into syllables.' The model and scoring depend on these divisions, and errors would propagate to meter counting.
  • domain assumption Syllable-level regularities are sufficiently stable from modern Italian to Dante's language for transfer to be beneficial.
    Section 3.1 justifies pre-training on PAISA' with the claim that 'there are a number of inherent regularities at the syllable level that have not changed that much.' This is asserted, not measured.
  • domain assumption A LSTM language model trained with cross-entropy on a target author's text will capture author-specific style and content relevant to poetry.
    The whole approach assumes that next-syllable prediction is a sufficient objective for style transfer. The paper does not justify this from theory.
  • domain assumption Human judgments of authorship ('real Dante' vs not) are a valid proxy for successful style imitation.
    The main qualitative measure in Table 2 uses binary non-expert attribution; its reliability is not analyzed, and judges may be guessing based on superficial cues.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural Poetry: Learning to Generate Poems using Syllables." pith.science (2026). https://pith.science/paper/LMTAETNX

@misc{pith2026190808861,
  author       = {Pith},
  title        = {Pith review of: Neural Poetry: Learning to Generate Poems using Syllables},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LMTAETNX}},
  note         = {Machine review of arXiv:1908.08861}
}
read the original abstract

Motivated by the recent progresses on machine learning-based models that learn artistic styles, in this paper we focus on the problem of poem generation. This is a challenging task in which the machine has to capture the linguistic features that strongly characterize a certain poet, as well as the semantics of the poet's production, that are influenced by his personal experiences and by his literary background. Since poetry is constructed using syllables, that regulate the form and structure of poems, we propose a syllable-based neural language model, and we describe a poem generation mechanism that is designed around the poet style, automatically selecting the most representative generations. The poetic work of a target author is usually not enough to successfully train modern deep neural networks, so we propose a multi-stage procedure that exploits non-poetic works of the same author, and also other publicly available huge corpora to learn syntax and grammar of the target language. We focus on the Italian poet Dante Alighieri, widely famous for his Divine Comedy. A quantitative and qualitative experimental analysis of the generated tercets is reported, where we included expert judges with strong background in humanistic studies. The generated tercets are frequently considered to be real by a generic population of judges, with relative difference of 56.25\% with respect to the ones really authored by Dante, and expert judges perceived Dante's style and rhymes in the generated text.

Figures

Figures reproduced from arXiv: 1908.08861 by the authors.

Figure 1
Figure 1. Sketch of the Syllable LM. Input tercets are first pre-processed by a module that splits words into syllables and introduces some special tokens: word-separator <sep>, begin-of-tercet <go>, end-of-verse <eov>, end-of-tercet <eot>. Orange blocks are syllable embeddings, each blue block depicts the network of Eq. (2-4). The system predicts the next syllable, i.e. the one associated to the largest probability in yˆt . … view at source ↗
Figure 2
Figure 2. Results of [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 20 canonical work pages

  1. [1]

    In: Proceedings of the 27th International Conference on Computational Linguistics

    Akbik, A., Blythe, D., Vollgraf, R.: Contextual string embeddings for sequence labeling. In: Proceedings of the 27th International Conference on Computational Linguistics. pp. 1638–1649 (2018)

  2. [2]

    Oxford Uni- versity Press, Oxford University Press (1998)

    Alighieri, D., Sisson, C., Sisson, C., Higgins, D.: The Divine Comedy. Oxford Uni- versity Press, Oxford University Press (1998)

  3. [3]

    Journal of machine learning research 3(Feb), 1137–1155 (2003)

    Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. Journal of machine learning research 3(Feb), 1137–1155 (2003)

  4. [4]

    In: Proceedings of the 2016 Conference of the NAACL: Human Language Technologies

    Chopra, S., Auli, M., Rush, A.M.: Abstractive sentence summarization with at- tentive recurrent neural networks. In: Proceedings of the 2016 Conference of the NAACL: Human Language Technologies. pp. 93–98 (2016)

  5. [5]

    In: ICCC

    Colton, S., Goodwin, J., Veale, T.: Full-face poetry generation. In: ICCC. pp. 95– 102 (2012) 12 A. Zugarini, S. Melacci, M. Maggini

  6. [6]

    In: Advances in neural information processing systems

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in neural information processing systems. pp. 2672–2680 (2014)

  7. [7]

    In: International Conference on Computational Linguistics: Technical Papers

    Hasan, S.A., Lee, K., Datla, V., Qadir, A., Liu, J., Farri, O., et al.: Neural para- phrase generation with stacked residual lstm networks. In: International Conference on Computational Linguistics: Technical Papers. pp. 2923–2934 (2016)

  8. [8]

    In: Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers)

    Hopkins, J., Kiela, D.: Automatically generating rhythmic verse with neural net- works. In: Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers). vol. 1, pp. 168–178 (2017)

Show all 24 references
  1. [9]

    In: Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on

    Hwang, K., Sung, W.: Character-level language modeling with hierarchical recur- rent neural networks. In: Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on. pp. 5720–5724. IEEE (2017)

  2. [10]

    Lau, J.H., Cohn, T., Baldwin, T., Brooke, J., Hammond, A.: Deep-speare: A joint neural model of poetic language, meter and rhyme (2018)

  3. [11]

    In: 9th Web as Corpus Workshop (WaC-9)@ EACL 2014

    Lyding, V., Stemle, E., Borghetti, C., Brunello, M., Castagnoli, S., Dell’Orletta, F., Dittmann, H., Lenci, A., Pirrelli, V.: The paisa’ corpus of italian web texts. In: 9th Web as Corpus Workshop (WaC-9)@ EACL 2014. pp. 36–43. EACL (2014)

  4. [12]

    In: International Conference on Artificial Neural Networks

    Marra, G., Zugarini, A., Melacci, S., Maggini, M.: An unsupervised character-aware neural approach to word and context representation learning. In: International Conference on Artificial Neural Networks. pp. 126–136. Springer (2018)

  5. [13]

    In: Eleventh annual conference of the inter- national speech communication association (2010)

    Mikolov, T., Karafi´ at, M., Burget, L., ˇCernock` y, J., Khudanpur, S.: Recurrent neural network based language model. In: Eleventh annual conference of the inter- national speech communication association (2010)

  6. [14]

    In: Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Pro- cessing

    Miyamoto, Y., Cho, K.: Gated word-character recurrent language model. In: Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Pro- cessing. pp. 1992–1997 (2016)

  7. [15]

    Cambridge university press (2000)

    Reiter, E., Dale, R.: Building natural language generation systems. Cambridge university press (2000)

  8. [16]

    The Journal of Machine Learning Research 15(1), 1929–1958 (2014)

    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15(1), 1929–1958 (2014)

  9. [17]

    In: Proceedings of the 2nd Workshop on Representation Learning for NLP

    Subramanian, S., Rajeswar, S., Dutil, F., Pal, C., Courville, A.: Adversarial gener- ation of natural language. In: Proceedings of the 2nd Workshop on Representation Learning for NLP. pp. 241–251 (2017)

  10. [18]

    In: Thirteenth annual conference of the international speech communication association (2012)

    Sundermeyer, M., Schl¨ uter, R., Ney, H.: Lstm neural networks for language mod- eling. In: Thirteenth annual conference of the international speech communication association (2012)

  11. [19]

    In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence

    Wang, Q., Luo, T., Wang, D., Xing, C.: Chinese song iambics generation with neural attention-based model. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. pp. 2943–2949. AAAI Press (2016)

  12. [20]

    In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

    Wen, T.H., Gasic, M., Mrkˇ si´ c, N., Su, P.H., Vandyke, D., Young, S.: Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pp. 1711–1721 (2015)

  13. [21]

    In: Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pp

    Yi, X., Li, R., Sun, M.: Generating chinese classical poems with rnn encoder- decoder. In: Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data, pp. 211–223. Springer (2017)

  14. [22]

    In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing

    Yi, X., Sun, M., Li, R., Li, W.: Automatic poetry generation with mutual rein- forcement learning. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3143–3153 (2018) Neural Poetry: Learning to Generate Poems using Syllables 13

  15. [23]

    In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

    Yu, L., Zhang, W., Wang, J., Yu, Y.: Seqgan: Sequence generative adversarial nets with policy gradient. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

  16. [24]

    In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)

    Zhang, X., Lapata, M.: Chinese poetry generation with recurrent neural networks. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 670–680 (2014)

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.