{"id":"eab2cb6f-dc02-46e4-8a82-dbac12820b7d","arxiv_id":"1908.04332","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A case study comparing LSTM, GRU and Bidirectional RNN for character-level TV script generation, with internally inconsistent reported results.","lead":"Three character-level neural networks (LSTM, GRU, and Bidirectional RNN) were trained on Game of Thrones scripts to generate new dialogue. The paper compares their speed and loss, but the reported numbers contradict each other and no proper evaluation is given.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central efficiency and loss rankings are contradicted by the paper's own Table II and Fig. 5, so the Section V conclusion is unsupported.","rationale":"The reader's verdict is REJECT with high confidence. I read the full manuscript and identified the same load-bearing condition: the Section V ranking of efficiency and loss must be supported by the quantitative results in Section IV. That condition is not met because the paper's own numbers are internally inconsistent. Table II reports per-step training times of 28/24/17 ms for LSTM and 24/17/14 ms for GRU, making GRU strictly faster in every architecture, while the Section IV text says LSTM took the least time and GRU took slightly greater time. The loss numbers also conflict: Table II lists log-likelihood values for LSTM, GRU, and BiRNN that do not match the final loss values described for Fig. 5 (e.g., BiRNN at 32E-3 in Fig. 5 text versus -2.928 to -3.216 in Table II). Since the conclusion's rankings are drawn directly from these numbers, a conflict in the numbers removes the support for the central claim. Additional weaknesses—no held-out set, no error bars, no repeated runs, and the degenerate 'JOJO' output from the lowest-loss model—corroborate that the reported comparison is not a reliable basis for the claimed rankings. I agree with the reader's weakest assumption; my concrete test would settle whether Table II matches the actual training histories. Because the central claim is contradicted by the paper's own evidence, the REJECT verdict stands unchanged.","tokens_in":9159,"tokens_out":4084,"duration_ms":39259,"concrete_test":"Run the released training code and saved histories ([23]/[25]) for all nine configurations with the same batch size, optimizer, and hardware; record the per-step wall-clock time and final training loss from the saved pickle histories. Then verify whether (a) Table II matches the histories, and (b) the per-step times put GRU below LSTM in all three depths, as the table states. If the histories disagree with Table II, or if the Fig. 5 loss descriptions disagree with Table II, the conclusion's efficiency and loss rankings are contradicted by the paper's own data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section V) is that \"LSTM generates text in the most efficient way followed by GRU and then Bidirectional RNN while loss is least in Bidirectional RNN followed by LSTM and it is most in GRU.\" The only quantitative support is Table II and the Fig. 5 descriptions, and these sources contradict the claim. For Training Step Time (ms), Table II gives LSTM single/bi/quad = 28/24/17 and GRU = 24/17/14, i.e., GRU is faster in every configuration, while the Section IV text says LSTM took the least time and GRU took slightly greater time than LSTM. For loss, Table II reports log-likelihood values (LSTM -0.064/0.170/0.394, GRU 0.080/0.216/0.423, BiRNN -2.928/-3.216/-2.535), whereas the Fig. 5 text reports final losses of LSTM 88E-2/107E-2/126E-2, GRU 116E-2, and BiRNN 32E-3/26E-3/37E-3 — a different scale and different numbers. No error bars, repeated runs, or held-out evaluation are provided. Because the stated rankings are derived directly from these numbers, the central conclusion is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a case study comparing LSTM, GRU, and Bidirectional RNN models for character-level script generation, using a Game of Thrones script dataset. The authors train single-, bi-, and quad-layer variants of each architecture, report training loss and per-step training time, and provide qualitative examples of generated text. The central claim, stated in Section V, is that LSTM is the most efficient in text generation, followed by GRU and then Bidirectional RNN, while loss is lowest for Bidirectional RNN, then LSTM, and highest for GRU. The paper also provides implementation details, including model architecture, preprocessing, and training setup.","tokens_in":9403,"tokens_out":1669,"duration_ms":17683,"significance":"If the claimed rankings were supported by rigorous evidence, this would be a modest but useful engineering comparison for character-level sequence generation. However, the central comparative conclusion is directly contradicted by the paper's own reported data, and the evaluation is confined to training loss with no held-out set, no repeated runs, and no uncertainty quantification. The qualitative outputs themselves include degenerate repeated-character sequences for the Bidirectional RNN (Fig. 7(c)), which the text does not acknowledge. The paper's potential contribution is therefore not realized in its current form; the contradictions are internal and load-bearing, not merely presentation issues.","major_comments":[{"comment":"The conclusion's efficiency ranking ('LSTM generates text in a most efficient way followed by GRU and then Bidirectional RNN') is contradicted by Table II, which reports Training Step Time (ms) of 28/24/17 for LSTM single/bi/quad, 24/17/14 for GRU, and 108/62/49 for Bidirectional RNN. In every configuration, GRU is faster than LSTM, so the statement in Section IV that 'LSTM based neural networks took the least time... GRU took slightly greater time than LSTM' is unsupported by the table it cites. Since the central claim depends directly on these numbers, this is a load-bearing inconsistency that cannot be resolved by rewording alone.","section":"Section V and Table II"},{"comment":"The loss values in Table II and the loss values described for Fig. 5 are on different scales and cannot be reconciled. For example, Table II reports Log-Likelihood of -0.064 for single-layer LSTM and -2.928 for single-layer Bidirectional RNN, while the Fig. 5 text reports final losses of 88E-2 for LSTM and 32E-3 for BiRNN. The magnitude and even the sign conventions differ, and no explanation is given for the discrepancy. Because the paper's loss ranking is derived from these conflicting numbers, the claim that 'loss is least in Bidirectional RNN followed by LSTM and most in GRU' is not supported by the evidence presented.","section":"Table II vs. Fig. 5 descriptions"},{"comment":"All quantitative comparisons are based on training loss and training step time, with no held-out validation or test data, no repeated random seeds, and no error bars or confidence intervals. The models are evaluated on the same data they were trained on, so the reported loss values measure fit to the training corpus rather than generalization to unseen script data. The qualitative generated-text evaluation is also questionable: the single-layer Bidirectional RNN output in Fig. 7(c) is a long repetition of 'JOJOJO...', yet the text in Section IV claims 'the results generated are understandable.' These issues undermine any comparative claim about model quality and should be addressed with a proper evaluation protocol before the conclusions can be accepted.","section":"Section IV and experimental protocol"}],"minor_comments":[{"comment":"The sentence 'LSTM generates text a in most efficient way' contains a typo ('a in' should be 'in the').","section":"Section V"},{"comment":"The Related Work section includes reference [5] as 'Alan' and later says 'Alan Ritter's paper [5] applies deep reinforcement learning to stimulate dialogues,' but the actual reference [5] is Li et al. (2016) on deep reinforcement learning for dialogue generation. The citation appears mislabeled; also, Ritter et al. is already cited as [6]. The reference list needs careful checking.","section":"Section II and References"},{"comment":"The loss function is described as 'Sparse Categorical Cross Entropy' but the text repeatedly misspells 'categorical' as 'categorial'. Also, the loss formula is written with a subscript style that is not rendered cleanly; please format it properly.","section":"Section III"},{"comment":"The generated text for the single-layer Bidirectional RNN model consists almost entirely of the repeated string 'JOJOJOJO...', which suggests the model collapsed during generation. The text should discuss this failure rather than claiming that the results are understandable.","section":"Fig. 7(c)"}],"recommendation":"reject","confidential_remarks":"The manuscript is not publishable in its current form: the central conclusion is contradicted by the paper's own Table II, the loss numbers in Table II and Fig. 5 are mutually inconsistent, and the evaluation lacks any held-out data. These are not fixable by local edits; a meaningful revision would require re-running the experiments with a proper evaluation protocol and reconciling the reported numbers. Given the scope of the paper (a short comparison study), this does not seem feasible without a substantially new empirical contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know up front: the empirical heart of this paper does not hold. The authors claim LSTM is the most efficient and BiRNN has lowest loss, but Table II shows GRU with lower training step times than LSTM in every configuration (e.g., single-layer 24 ms vs 28 ms), and the Fig. 5 text reports loss values on a different scale and with different numbers than Table II (e.g., LSTM single-layer final loss 88E-2 in text vs -0.064 log-likelihood in table). The conclusion in Section V is unsupported by the paper's own data.\n\nWhat's actually here: a small case study applying standard char-level LSTM, GRU, and BiRNN to Game of Thrones scripts, with Keras implementation, code and generated text on GitHub. That is the extent of the novelty. No new architecture, no theoretical derivation, no held-out evaluation. It does include a reasonable description of the preprocessing and the training setup, and the authors do acknowledge that the generated text is not coherent. That honesty is worth something, but it doesn't rescue the results.\n\nThe soft spots are load-bearing. The central ranking is not just missing error bars; it is contradicted by the paper's own tables. The loss values in Table II and the Fig. 5 descriptions disagree, and the Fig. 7(c) output is a degenerate string of repeated 'JOJO'. The evaluation is on training loss only, so overfitting is uncontrolled. There are no repeated runs, no validation split, no human evals, and no baselines beyond the three architypes. Under these conditions, claims about which model is 'most efficient' have little meaning. The paper reads like a student project report written up before the numbers were checked.\n\nThe citation pattern is ordinary but not problematic; the related work is a bit scattered but covers the main dialogue-generation pointers. The main issue is internal consistency.\n\nWho is this for? Maybe someone who wants a very quick look at why char-level RNN text generation is finicky, but the contradictions make it unreliable even for that. I would not recommend peer review in current form. If the authors re-ran the experiments with a proper train/validation split, reported per-run variance, and corrected the table/text discrepancies, a trimmed version could be a modest reproducibility note. As it stands, it is not a sound source.\n\nRecommendation: desk reject, or at most major revision with a demand to fix the data reporting.","headline":"A routine char-level RNN case study whose central efficiency/loss ranking is contradicted by its own Table II and Fig. 5 text.","tokens_in":9976,"tokens_out":2784,"would_cite":false,"duration_ms":28211,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims LSTM is the most efficient of three recurrent networks for script generation, while Bidirectional RNN gives the lowest loss.","keywords":["text generation","script generation","LSTM","GRU","Bidirectional RNN","sequence-to-sequence model","recurrent neural networks","character-level language model"],"falsifier":"Re-train the three models on the same annotated script using the hyperparameters reported in the paper and record the mean log-likelihood loss and wall-clock time per batch after the same number of epochs; if GRU's per-step time is not greater than LSTM's in every layer configuration, the paper's efficiency ranking is contradicted. Also check the single-layer LSTM loss, because the paper's results table reports $-0.064$ while the text reports a final value near $0.88$, so only one of these numbers can be right.","tokens_in":8918,"feed_emoji":"🎬","tokens_out":8044,"duration_ms":72679,"temperature":0.7,"pith_summary":"This paper is a case study asking which recurrent architecture—LSTM, GRU, or Bidirectional RNN—is best suited to generating new TV-script dialogue from a character-level corpus. It claims that LSTM generates text most efficiently, followed by GRU and then Bidirectional RNN, while Bidirectional RNN achieves the lowest predictive loss, followed by LSTM, with GRU highest. The authors train single-, bi-, and quad-layered variants of each model on a serialized script of a popular TV series and compare their training-loss curves and per-step times. If the ranking holds, it gives practitioners a concrete trade-off between generation speed and predictive loss when choosing an architecture for script or dialogue generation.","feed_headline":"LSTM beats GRU and BiRNN on script generation speed","feed_subtitle":"A three-way character-level RNN comparison on a TV-script corpus: LSTM leads on efficiency, BiRNN on loss.","key_machinery":"The central object is the shifted-character prediction task: each input sequence of $n$ characters is paired with the same sequence shifted one character to the right, so the model learns the probability of the next character given the previous characters. Each model stacks an embedding layer, one or more recurrent layers drawn from {LSTM, GRU, Bidirectional RNN} in uni-, bi-, or quad-layer configurations, dropout at 0.4, a dense layer, and RMSprop optimization with sparse categorical cross-entropy loss; at generation time, the next character is chosen by highest probability after temperature scaling. This setup lets the authors attribute differences in loss and step time to the recurrent architecture itself.","core_discovery":"On the paper's own terms, the central discovery is a three-way performance ranking for character-level script generation. Training LSTM, GRU, and Bidirectional RNN variants on the serialized script, with input sequences paired to one-character-right-shifted targets, the authors report that LSTM takes the least time per training step, GRU slightly more, and Bidirectional RNN the most; in mean log-likelihood loss, Bidirectional RNN is best, LSTM second, and GRU worst. Within each architecture, the single-layer configuration tends to reach lower loss than deeper ones for LSTM and GRU, while all Bidirectional RNN configurations converge quickly to very small loss values. Generated samples resemble English and character names but lack coherent plot-level connectivity.","pith_inferences":["The paper's own generated sample from the single-layer Bidirectional RNN collapses into a repeated string despite that model reporting the lowest loss, which suggests loss alone is a poor predictor of output quality; a natural follow-up would be to compare diversity metrics or human ratings instead of log-likelihood only.","The step times reported in the paper's results table show GRU faster than LSTM in some layer configurations, while the text claims LSTM is fastest; a direct re-measurement of wall-clock time per epoch under identical batching would settle which ordering is operational.","The comparison treats the three architectures as flat alternatives; a hybrid encoder-decoder with a Bidirectional RNN encoder and an LSTM decoder might combine the low loss of the former with the speed of the latter, but this is not tested in the paper.","Because the dataset is a single TV series, testing on multilingual or multi-show corpora with different dialogue density would reveal whether the ranking is an artifact of this one script."],"forward_implications":["If LSTM is indeed the fastest per training step, it is the natural default for interactive or resource-limited script-generation tools.","If Bidirectional RNN has the lowest loss but the highest step time, quality-oriented offline generation could justify its higher cost.","Deeper configurations generally do not improve loss for LSTM and GRU, suggesting that additional layers mainly add computation for this data scale.","The temperature-1 sampling procedure produces locally grammatical but plot-incoherent text, indicating that character-level loss alone is not a proxy for narrative quality.","The same experimental recipe can be applied to any serialized script corpus to choose an architecture before investing in a larger model."],"supporting_citations":[{"why":"The paper cites this work as the basis for the Bidirectional RNN architecture used in the comparison.","marker":"[1]"},{"why":"Cited as the origin of the LSTM architecture; supplies the gated recurrent formulation the experiments compare.","marker":"[2]"},{"why":"Cited as the basis for GRU and for the recurrent-neural-network background of the study.","marker":"[3]"},{"why":"The serialized pickle version of the TV-script corpus on which all three models are trained.","marker":"[4]"},{"why":"The neural conversational model that motivates the dialogue-generation task and the sequence-to-sequence framing.","marker":"[7]"},{"why":"The sequence-to-sequence learning formulation the paper adopts for building input-output character pairs.","marker":"[9]"},{"why":"The publicly available annotated script data that provides the structured text and speaker labels used for training.","marker":"[24]"}],"fun_headline_variants":["LSTM trains fastest, BiRNN has best loss on scripts","Script generation: single-layer LSTM/GRU beat deeper nets","BiRNN converges fast, LSTM trains quickest","Character-level script models: speed vs loss trade-off","LSTM wins speed, BiRNN wins loss on script AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire ranking stands or falls with the accuracy and comparability of the reported loss values and training step times for the three models; if those numbers are not measured consistently or are not reproducible, the ordering of models does not follow.","fun_headline_variants_meta":{"raw":{"variants":["LSTM trains fastest, BiRNN has best loss on scripts","Script generation: single-layer LSTM/GRU beat deeper nets","BiRNN converges fast, LSTM trains quickest","Character-level script models: speed vs loss trade-off","LSTM wins speed, BiRNN wins loss on script AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001551,"raw_usage":{"total_tokens":6166,"prompt_tokens":877,"completion_tokens":5289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":5203}},"tokens_in":493,"tokens_out":5289,"duration_ms":33210,"temperature":1.0,"reasoning_tokens":5203,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:44:26.891811+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train the three models on the same annotated script using the hyperparameters reported in the paper and record the mean log-likelihood loss and wall-clock time per batch after the same number of epochs; if GRU's per-step time is not greater than LSTM's in every layer configuration, the paper's efficiency ranking is contradicted. Also check the single-layer LSTM loss, because the paper's results table reports $-0.064$ while the text reports a final value near $0.88$, so only one of these numbers can be right.","supporting_citations":[{"cited_title":"Learning long -term dependencies with gradient descent is difficult,","cited_arxiv_id":null,"evidence_quote":"The paper cites this work as the basis for the Bidirectional RNN architecture used in the comparison."},{"cited_title":"Got.pkl,","cited_arxiv_id":null,"evidence_quote":"The serialized pickle version of the TV-script corpus on which all three models are trained."},{"cited_title":"Sequence to sequence learning with neural networks,","cited_arxiv_id":null,"evidence_quote":"The sequence-to-sequence learning formulation the paper adopts for building input-output character pairs."},{"cited_title":"Game of Thrones Data,","cited_arxiv_id":null,"evidence_quote":"The publicly available annotated script data that provides the structured text and speaker labels used for training."}],"review_version":1}