{"id":"45ad2768-c60f-451f-a916-02e55e24b95d","arxiv_id":"1908.08861","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A syllable-level LSTM language model with multi-stage transfer learning generates Italian tercets that non-expert judges attribute to Dante about 28% of the time versus 64% for real Dante, with a relative difference of 56.25%.","lead":"This paper trains a syllable-based LSTM language model to generate new verses of Dante's Divine Comedy in Italian. It reports that non-expert judges sometimes mistake the generated tercets for Dante's own work, and that expert judges perceive the rhyme and style.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 28% authenticity figure may be an artifact of the hand-designed scorer applied to 2,000 sampled tercets; unfiltered sy-LM outputs are never evaluated against real Dante, so the claim conflates model with selection.","rationale":"The reader correctly flags the human evaluation as the weakest link, and I agree it is the critical area. My stress-test sharpens it: the most concrete defect is the confounding of the neural sampler with the explicit scorer. This is directly load-bearing for the abstract's sentence that the generated tercets are frequently considered real. The transfer-learning PPL numbers in Table 1 are credible and provide independent support for the training procedure, but they do not show that syllable unitization is essential; no comparison against char/word LMs is reported. Nonetheless, absence of an ablation is a novelty concern rather than a correctness defect. The selection confound, by contrast, threatens the validity of the headline qualitative result. Because the paper's verdict is already CONDITIONAL, my analysis does not move the verdict; it adds a precise condition that should be met before the authenticity claim is accepted.","tokens_in":9266,"tokens_out":5932,"duration_ms":62683,"concrete_test":"Run a controlled human evaluation with the same 13 non-expert judges and the same 5-real/5-generated protocol using four conditions: (i) unfiltered random sy-LM samples, (ii) top-scored sy-LM outputs selected by R1–R4, (iii) real Dante tercets, and (iv) a baseline that emits random words from the Divine Comedy vocabulary satisfying 11 syllables per line and ABA rhyme, then scores and selects by R1–R4. If unfiltered sy-LM shows a much lower real-mark rate than 28%, or if the baseline reaches a comparable real-mark rate, the headline claim must be reattributed to the selection mechanism. Also report the exact per-tercet denominators for Table 4 and recompute aggregate rates with confidence intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is not judge heterogeneity but output selection. In Section 3.2 the authors generate 2,000 tercets and keep only those with the highest score R(x), the average of R1 (tercet form), R2 (11-syllable verses), R3 (ABA rhyme), and R4 (Dante vocabulary). Section 4.1 then reports human judgments on 'generated tercets', and Table 4 displays selected examples whose fooling rates range from 88.00% to 0%, indicating that a small subset drives the average. If the 28% sy-LM rate in Table 2 is computed on top-scored outputs only, it measures the quality of the scorer as much as of the neural model. Real Dante tercets were not passed through R1–R4, so the comparison is not between equal pipelines. Moreover, because R2–R4 explicitly enforce hendecasyllables, terza rima, and the Divine Comedy vocabulary, a trivial Monte Carlo generator could receive high scores; without a baseline using the same scorer, the contribution of syllable-based transfer learning is unidentified. Table 4 also reports per-tercet rates that are not representable with the stated 13 non-expert judges (88.00%, 55.56%, and 45.45% are not multiples of 1/13), so the evaluation denominators need clarification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a syllable-based LSTM language model (sy-LM) for Italian poetry generation, trained on the Divine Comedy with a multi-stage transfer-learning pipeline that first pre-trains on modern Italian web text (PAISA') and then on Dante's other prose and poems before fine-tuning on the Divine Comedy. Generation is performed by Monte Carlo sampling of 2,000 tercets, followed by a hand-designed scoring filter R(x) that averages four scores: tercet form (R1), hendecasyllabic meter (R2), ABA rhyme (R3), and Dante-vocabulary word usage (R4). The authors report perplexity results showing that transfer learning improves the language model, and they present human evaluations: 13 non-expert judges marked generated tercets as 'real' 28% of the time versus 64% for real Dante tercets, while 4 expert judges rated generated tercets on readability, emotion, meter, rhyme, and style, with particularly positive rhyme scores.","tokens_in":9577,"tokens_out":3598,"duration_ms":38031,"significance":"If the reported results hold, the paper makes a useful contribution to low-resource poetic style generation: the syllable-level representation is a sensible inductive bias for Italian prosody, and the multi-stage transfer-learning result (PAISA' -> DP -> DC) is clearly demonstrated by the perplexity numbers in Table 1. The main architectural idea is simple, reproducible, and well aligned with the formal properties of terza rima, and the authors should be credited for reporting a human evaluation with both non-expert and expert judges rather than relying on automated metrics alone. The qualitative claims, however, are only as strong as the evaluation protocol: the authenticity rate in Table 2 is computed on a heavily filtered subset of generated tercets without an equally filtered set of real Dante tercets, and the small judge panels and possible denominator inconsistencies make the headline percentages less reliable. The central idea is defensible, but the load-bearing evaluation needs substantial strengthening before the qualitative claims can be accepted at face value.","major_comments":[{"comment":"The human authenticity test is asymmetric: the 2,000 generated tercets are filtered by the scoring function R(x), and only the highest-scoring outputs are shown to judges, whereas the real Dante tercets in the comparison are never passed through R1-R4 or through the same selection procedure. Consequently, the 28% versus 64% result conflates the quality of the neural model with the quality of the hand-designed scorer, since a trivial or even random generator could in principle reach high scores on R2 (counting eleven syllables), R3 (ABA rhyme), and R4 (Dante vocabulary). To identify the contribution of sy-LM, the authors should evaluate (i) unfiltered sampled tercets from sy-LM, (ii) real Dante tercets that have been passed through the same scoring filter and selected in the same way, and (iii) a non-neural baseline such as random syllable sequences or a simple n-gram model subjected to the same R-scoring and selection. Without these controls, the central qualitative claim measures the full pipeline rather than the model, and the contribution of syllable-based transfer learning is not identifiable.","section":"Section 4.1, Table 2 and Table 4"},{"comment":"The reported percentages are not consistent with the stated experimental setup. With 13 non-expert judges each evaluating 10 tercets (5 generated and 5 real), the number of generated judgments is 65 and the number of real judgments is 65. The reported sy-LM rate of 28% and Poet rate of 64% correspond to 18.2 and 41.6 judgments, respectively, which are not integers. Likewise, Table 4 reports per-tercet fooling rates of 88.00%, 55.56%, and 45.45%, none of which is a multiple of 1/13. The authors should clarify how these percentages are computed, state the exact denominators for each reported rate, and identify which judge group (non-expert, expert, or combined) produced the per-tercet values in Table 4. If a per-judge or per-item count was averaged, that should be explicitly described; otherwise the numerical claims are not reproducible.","section":"Section 4.1, Table 2"},{"comment":"The expert evaluation is based on only four judges, each providing scores for 10 generated tercets and 10 real tercets, but the paper reports only average scores for the generated tercets and a single 'Poet (Average)' row, without per-judge scores for the real tercets, without variance, and without any significance test. The conclusion that 'experts perceived Dante's style and rhymes in the generated text' is supported only by a rhyme average of 3.37 against a real-poem average of 4.50 and a style average of 2.49 against 4.34; these differences are large, and the claim should be phrased with appropriate caution. At minimum, the authors should report the full score distributions, including the real tercet scores for each expert, and perform a paired or mixed-effects analysis given the repeated-measures design.","section":"Section 4.1, Table 3"},{"comment":"The examples in Table 4 reveal very high variance in per-tercet human judgments: the first tercet was marked as real by 88.00% of evaluators, while the last one never fooled anyone. Reporting only the average of such a skewed distribution, as in Table 2, hides the fact that a small subset of high-scoring outputs may drive the headline result. The authors should report the distribution over generated tercets, for example the median and interquartile range of the authenticity rates, and should avoid presenting the selected examples as representative without this context. This issue is directly connected to the scoring-filter asymmetry raised above, because the selected examples are exactly those that maximize R(x).","section":"Section 4.1, Table 4"}],"minor_comments":[{"comment":"The text contains a typographical error: 'by reading the Divide Comedy' should read 'by reading the Divine Comedy'.","section":"Section 1"},{"comment":"The notation in Eq. (8) is slightly inconsistent: R4(x) is defined as a sum over words w in x, but f_w(x_i) is written as an elementwise function of word tokens; please align the notation so that the scoring function is clearly defined for a word token.","section":"Section 3.2"},{"comment":"The syllabification procedure is described only as 'the most common Italian hyphenation rules' with 'rare exceptions'; since the entire model operates on syllables, the authors should provide more detail on the segmentation algorithm and, ideally, a small quantitative evaluation of its accuracy on an Italian test sample.","section":"Section 3"},{"comment":"The paper reports hyperparameter values (embedding size 300, LSTM hidden size 1024, dropout 0.3, batch size 32, learning rate 0.001) but does not specify the optimizer, the number of training epochs, or the early-stopping criterion used for the PAISA' and DP pre-training stages; these details are needed for reproducibility.","section":"Section 4"},{"comment":"The authors should state exactly how many generated tercets were selected by the scoring procedure and shown to judges in each of the two experiments (e.g., whether each non-expert judge saw a unique set of five generated tercets or whether the same five were reused). Without this information, the number of distinct generated tercets evaluated, and hence the effective sample size, remains unclear.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a machine-learning venue interested in creative generation and transfer learning, and the perplexity portion of the experiments is solid. My main concern is that the qualitative claims, which are the paper's headline contribution, rest on an evaluation protocol that conflates model and filter, with small panels and apparent numerical inconsistencies. These issues are fixable in a revision: re-running the human evaluation with appropriate controls and clear denominators would substantially strengthen the manuscript. I would not reject the paper, but the current version requires major revisions before the qualitative claims can be considered reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core of this paper is straightforward: a syllable-level LSTM language model, trained in stages on modern Italian (PAISA'), Dante's prose and minor poems, and then the Divine Comedy, and sampled to produce tercets. Perplexity on the Divine Comedy test set drops from 12.39 to 10.55 with staged pretraining; that part is internally consistent and credible. Table 1 is the strongest evidence in the paper. The novelty is narrow but real: Italian poetry, Dante specifically, and syllable tokens are an underexplored choice. Related work is covered adequately, and the citation pattern looks normal rather than self-promotional.\n\nThe soft spots are in the qualitative claims, and one is bigger than the authors acknowledge. They sample 2,000 tercets, keep only the best-scored ones under R1-R4 (three hendecasyllables, ABA rhyme, Divine Comedy vocabulary), and then ask judges to compare those selected outputs against real Dante tercets. Real tercets are not passed through the same scorer. So the 28% versus 64% result measures the generation-plus-selection pipeline, not the syllable language model by itself. A trivial generator could score well on form and vocabulary once R2-R4 are applied. Without a random or alternative baseline run through the same filter, the contribution of sy-LM is unidentified. This is fixable and not fatal to the engineering claim, but it should be addressed before the human evaluation is taken as evidence about syllables.\n\nThere is also no ablation comparing syllable tokens against character or word tokens. The central motivation, that syllables matter for poetry, is asserted rather than demonstrated. The human evaluation is small: 13 non-expert and 4 expert judges, with no error bars or significance tests. The authors are honest about weak emotion and meter, which helps, but Table 4 reports per-tercet fooling rates of 88.00%, 55.56%, and 45.45%, percentages that cannot all come from 13 judges, so the denominators need clarification. Minor, but sloppy. No code or data are released, which limits reproducibility.\n\nBottom line: this deserves a serious referee for a computational creativity or digital humanities venue, not a desk reject. The quantitative transfer result is solid; the evaluation claims need a same-pipeline baseline for real tercets, a no-scorer baseline, and an ablation of the syllable tokenization. I would accept for peer review with expectation of heavy revision.","headline":"A clean, modestly novel system for Italian tercet generation; the perplexity transfer results are credible, but the headline human-eval claim is entangled with a hand-designed selection filter and needs a baseline/ablation before it carries weight.","tokens_in":10065,"tokens_out":4194,"would_cite":true,"duration_ms":43499,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A syllable-level neural language model, pretrained on modern Italian and the poet's other works, produces tercets that non-expert judges mark as genuine Dante 28% of the time.","keywords":["Poem Generation","Transfer Learning","Language Models","Recurrent Neural Networks","Natural Language Generation","syllable-based language model","Dante Alighieri","human evaluation"],"falsifier":"Run a preregistered Turing test with a representative sample of, say, 100 Italian-speaking non-specialists, each classifying 20 balanced tercets in random order, and check agreement statistics; if sy-LM's real-mark rate is not near 28% or Dante's not near 64% with the same relative gap, the central authenticity claim is disconfirmed.","tokens_in":9087,"feed_emoji":"📜","tokens_out":7276,"duration_ms":72610,"temperature":0.7,"pith_summary":"The paper claims that a neural language model operating on syllables, rather than words or characters, can generate new tercets in the style of Dante's Divine Comedy with plausible form. After pretraining on a large modern Italian corpus and then on Dante's non-poetic prose and minor poems before fine-tuning on the Divine Comedy, the model's output was judged to be real Dante by non-expert readers 28% of the time, versus 64% for genuine Dante tercets—a relative difference of 56.25%. Expert judges rated the generated rhymes highly and perceived traces of Dante's style, though they found the text weak in meter and emotional content. The paper thus demonstrates that form-focused, stylized poem generation is feasible with a relatively simple LSTM, and that syllable-level tokens carry much of the stylistic signal.","feed_headline":"Generated Dante-style tercets pass as real 28% of the time","feed_subtitle":"A syllable-level Italian language model, pretrained in stages, writes tercets that experts praise for rhyme and style.","key_machinery":"The machinery is the syllable-based language model (sy-LM): text is split into syllables by Italian hyphenation rules, special tokens mark word boundaries, verse ends, and tercet ends, and an LSTM with tied input/output embeddings predicts the next syllable. Because the vocabulary is syllables rather than words, the model naturally tracks hendecasyllabic meter and rhyme and tolerates vocabulary drift between modern Italian and Dante's Tuscan. Training proceeds in stages: a large modern Italian web corpus first, then Dante's prose and minor poems, then the Divine Comedy. Generation samples tercets stochastically and ranks them with a four-part score—correct tercet length, three 11-syllable verses, chained ABA rhyme, and use of Divine Comedy vocabulary—so the selected output respects terza rima form.","core_discovery":"The central claim is that a syllable-token LSTM language model, trained in a multi-stage transfer procedure, produces Dantean tercets that a generic population of judges frequently cannot distinguish from Dante's own. On a per-judge test with 13 non-expert judges, sy-LM output was labeled as real 28% of the time and real Dante tercets 64% of the time; a post-hoc split shows that less-capable judges actually preferred the generated tercets over Dante's. Four expert judges, scoring 20 tercets each, gave the generated text an average rhyme score of 3.37 out of 5, compared with 4.50 for Dante, and a style score of 2.49 versus 4.34, while emotion and meter scored low (1.73 and 1.90). The overall picture is that the model captures surface form and some stylistic resemblance, but not strong semantic or emotional coherence.","pith_inferences":["My inference: the gap between expert rhyme scores (3.37) and expert style scores (2.49) suggests the syllable-level signal is mostly phonological rather than lexical-semantic, so adding a word-sense or content-conditioning layer would be the natural next experiment.","My inference: because the published 28% authenticity rate measures the model plus its scoring filter, the language model alone might perform differently; ablating the scoring function would isolate the filter's contribution.","My inference: the post-hoc split into less- and more-capable judges implies the headline relative difference is panel-dependent, so the result is an existence proof rather than a stable population estimate.","My inference: for a new target poet, the first two pretraining stages could likely be reused unchanged since they are author-agnostic, requiring only a final fine-tuning run on the new poet's corpus."],"forward_implications":["The same syllable-based pipeline can be pointed at other syllabic poetries, including English verse, by swapping the hyphenation rules and the target corpus.","Multi-stage transfer—general language, then the author's non-poetic prose and minor poems, then the target poem—is a reusable recipe for low-resource stylistic imitation.","Enforcing meter, rhyme, and tercet length with a lightweight post-hoc scoring function yields recognizable poetic form without separate learned meter or rhyme modules.","Because syllable counts are explicit in the model, the scoring criteria could directly become reward signals for reinforcement learning, the direction the authors outline as future work.","The model's ceiling is form and style: low expert emotion and meter scores indicate that pure next-syllable prediction does not provide semantic or emotional coherence."],"supporting_citations":[{"why":"Supplies the Divine Comedy text, the target corpus of tercets and the vocabulary used in scoring.","marker":"[2]"},{"why":"Provides the large modern Italian web corpus used in the initial pretraining stage.","marker":"[11]"},{"why":"Is the LSTM language modeling result that motivates and supports the sy-LM architecture.","marker":"[18]"},{"why":"Is the closest comparison point, a deep model explicitly combining language, meter, and rhyme modules, which the present work replaces with syllable tokens plus lightweight scoring.","marker":"[10]"},{"why":"Provides the Monte Carlo sampling procedure used to generate candidate tercets.","marker":"[23]"}],"fun_headline_variants":["AI writes Dante-style tercets that fool 28% of readers","Syllable-based AI churns out Dante-ish verse, fools quarter of judges","Fake Dante: AI poems pass as real for 1 in 4 readers","AI Dante: 28% of humans can't tell machine verse from the original","Machine poet mimics Dante's rhyme, but experts see gaps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The authenticity percentages rest on a small human panel—13 non-expert judges for the 28% versus 64% comparison—and the paper's own post-hoc split shows the panel is heterogeneous, so the headline rates may not match a broader population.","fun_headline_variants_meta":{"raw":{"variants":["AI writes Dante-style tercets that fool 28% of readers","Syllable-based AI churns out Dante-ish verse, fools quarter of judges","Fake Dante: AI poems pass as real for 1 in 4 readers","AI Dante: 28% of humans can't tell machine verse from the original","Machine poet mimics Dante's rhyme, but experts see gaps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3238,"prompt_tokens":976,"completion_tokens":2262,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2164}},"tokens_in":592,"tokens_out":2262,"duration_ms":19778,"temperature":1.0,"reasoning_tokens":2164,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:27:10.417333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a preregistered Turing test with a representative sample of, say, 100 Italian-speaking non-specialists, each classifying 20 balanced tercets in random order, and check agreement statistics; if sy-LM's real-mark rate is not near 28% or Dante's not near 64% with the same relative gap, the central authenticity claim is disconfirmed.","supporting_citations":[{"cited_title":"Oxford Uni- versity Press, Oxford University Press (1998)","cited_arxiv_id":null,"evidence_quote":"Supplies the Divine Comedy text, the target corpus of tercets and the vocabulary used in scoring."},{"cited_title":"In: 9th Web as Corpus Workshop (WaC-9)@ EACL 2014","cited_arxiv_id":null,"evidence_quote":"Provides the large modern Italian web corpus used in the initial pretraining stage."},{"cited_title":"In: Thirteenth annual conference of the international speech communication association (2012)","cited_arxiv_id":null,"evidence_quote":"Is the LSTM language modeling result that motivates and supports the sy-LM architecture."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the closest comparison point, a deep model explicitly combining language, meter, and rhyme modules, which the present work replaces with syllable tokens plus lightweight scoring."}],"review_version":1}