Pith. sign in

REVIEW 3 major objections 4 minor 25 references

LSTM vs. GRU vs. Bidirectional RNN for script generation

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims LSTM is the most efficient of three recurrent networks for script generation, while Bidirectional RNN gives the lowest loss.

desk verdict A routine char-level RNN case study whose central efficiency/loss ranking is contradicted by its own Table II and Fig. 5 text. read the letter →

arxiv 1908.04332 v1 pith:WPLL72YH submitted 2019-08-12 cs.CL cs.LG

Sanidhya Mangal , Poorva Joshi , Rahul Modak This is my paper · ORCID
classification cs.CLcs.LG
keywords textgenerationscriptLSTMGRUBidirectionalRNNsequence-to-sequencemodelrecurrentneuralnetworkscharacter-levellanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a case study asking which recurrent architecture—LSTM, GRU, or Bidirectional RNN—is best suited to generating new TV-script dialogue from a character-level corpus. It claims that LSTM generates text most efficiently, followed by GRU and then Bidirectional RNN, while Bidirectional RNN achieves the lowest predictive loss, followed by LSTM, with GRU highest. The authors train single-, bi-, and quad-layered variants of each model on a serialized script of a popular TV series and compare their training-loss curves and per-step times. If the ranking holds, it gives practitioners a concrete trade-off between generation speed and predictive loss when choosing an architecture for script or dialogue generation.

What carries the argument

The central object is the shifted-character prediction task: each input sequence of $n$ characters is paired with the same sequence shifted one character to the right, so the model learns the probability of the next character given the previous characters. Each model stacks an embedding layer, one or more recurrent layers drawn from {LSTM, GRU, Bidirectional RNN} in uni-, bi-, or quad-layer configurations, dropout at 0.4, a dense layer, and RMSprop optimization with sparse categorical cross-entropy loss; at generation time, the next character is chosen by highest probability after temperature scaling. This setup lets the authors attribute differences in loss and step time to the recurrent architecture itself.

What would settle it

Re-train the three models on the same annotated script using the hyperparameters reported in the paper and record the mean log-likelihood loss and wall-clock time per batch after the same number of epochs; if GRU's per-step time is not greater than LSTM's in every layer configuration, the paper's efficiency ranking is contradicted. Also check the single-layer LSTM loss, because the paper's results table reports $-0.064$ while the text reports a final value near $0.88$, so only one of these numbers can be right.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a three-way performance ranking for character-level script generation. Training LSTM, GRU, and Bidirectional RNN variants on the serialized script, with input sequences paired to one-character-right-shifted targets, the authors report that LSTM takes the least time per training step, GRU slightly more, and Bidirectional RNN the most; in mean log-likelihood loss, Bidirectional RNN is best, LSTM second, and GRU worst. Within each architecture, the single-layer configuration tends to reach lower loss than deeper ones for LSTM and GRU, while all Bidirectional RNN configurations converge quickly to very small loss values. Generated samples resemble English and character names but lack coherent plot-level connectivity.

Load-bearing premise

The entire ranking stands or falls with the accuracy and comparability of the reported loss values and training step times for the three models; if those numbers are not measured consistently or are not reproducible, the ordering of models does not follow.

Editorial extensions

If this is right

  • If LSTM is indeed the fastest per training step, it is the natural default for interactive or resource-limited script-generation tools.
  • If Bidirectional RNN has the lowest loss but the highest step time, quality-oriented offline generation could justify its higher cost.
  • Deeper configurations generally do not improve loss for LSTM and GRU, suggesting that additional layers mainly add computation for this data scale.
  • The temperature-1 sampling procedure produces locally grammatical but plot-incoherent text, indicating that character-level loss alone is not a proxy for narrative quality.
  • The same experimental recipe can be applied to any serialized script corpus to choose an architecture before investing in a larger model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own generated sample from the single-layer Bidirectional RNN collapses into a repeated string despite that model reporting the lowest loss, which suggests loss alone is a poor predictor of output quality; a natural follow-up would be to compare diversity metrics or human ratings instead of log-likelihood only.
  • The step times reported in the paper's results table show GRU faster than LSTM in some layer configurations, while the text claims LSTM is fastest; a direct re-measurement of wall-clock time per epoch under identical batching would settle which ordering is operational.
  • The comparison treats the three architectures as flat alternatives; a hybrid encoder-decoder with a Bidirectional RNN encoder and an LSTM decoder might combine the low loss of the former with the speed of the latter, but this is not tested in the paper.
  • Because the dataset is a single TV series, testing on multilingual or multi-show corpora with different dialogue density would reveal whether the ranking is an artifact of this one script.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents a case study comparing LSTM, GRU, and Bidirectional RNN models for character-level script generation, using a Game of Thrones script dataset. The authors train single-, bi-, and quad-layer variants of each architecture, report training loss and per-step training time, and provide qualitative examples of generated text. The central claim, stated in Section V, is that LSTM is the most efficient in text generation, followed by GRU and then Bidirectional RNN, while loss is lowest for Bidirectional RNN, then LSTM, and highest for GRU. The paper also provides implementation details, including model architecture, preprocessing, and training setup.

Significance. If the claimed rankings were supported by rigorous evidence, this would be a modest but useful engineering comparison for character-level sequence generation. However, the central comparative conclusion is directly contradicted by the paper's own reported data, and the evaluation is confined to training loss with no held-out set, no repeated runs, and no uncertainty quantification. The qualitative outputs themselves include degenerate repeated-character sequences for the Bidirectional RNN (Fig. 7(c)), which the text does not acknowledge. The paper's potential contribution is therefore not realized in its current form; the contradictions are internal and load-bearing, not merely presentation issues.

major comments (3)
  1. [Section V and Table II] The conclusion's efficiency ranking ('LSTM generates text in a most efficient way followed by GRU and then Bidirectional RNN') is contradicted by Table II, which reports Training Step Time (ms) of 28/24/17 for LSTM single/bi/quad, 24/17/14 for GRU, and 108/62/49 for Bidirectional RNN. In every configuration, GRU is faster than LSTM, so the statement in Section IV that 'LSTM based neural networks took the least time... GRU took slightly greater time than LSTM' is unsupported by the table it cites. Since the central claim depends directly on these numbers, this is a load-bearing inconsistency that cannot be resolved by rewording alone.
  2. [Table II vs. Fig. 5 descriptions] The loss values in Table II and the loss values described for Fig. 5 are on different scales and cannot be reconciled. For example, Table II reports Log-Likelihood of -0.064 for single-layer LSTM and -2.928 for single-layer Bidirectional RNN, while the Fig. 5 text reports final losses of 88E-2 for LSTM and 32E-3 for BiRNN. The magnitude and even the sign conventions differ, and no explanation is given for the discrepancy. Because the paper's loss ranking is derived from these conflicting numbers, the claim that 'loss is least in Bidirectional RNN followed by LSTM and most in GRU' is not supported by the evidence presented.
  3. [Section IV and experimental protocol] All quantitative comparisons are based on training loss and training step time, with no held-out validation or test data, no repeated random seeds, and no error bars or confidence intervals. The models are evaluated on the same data they were trained on, so the reported loss values measure fit to the training corpus rather than generalization to unseen script data. The qualitative generated-text evaluation is also questionable: the single-layer Bidirectional RNN output in Fig. 7(c) is a long repetition of 'JOJOJO...', yet the text in Section IV claims 'the results generated are understandable.' These issues undermine any comparative claim about model quality and should be addressed with a proper evaluation protocol before the conclusions can be accepted.
minor comments (4)
  1. [Section V] The sentence 'LSTM generates text a in most efficient way' contains a typo ('a in' should be 'in the').
  2. [Section II and References] The Related Work section includes reference [5] as 'Alan' and later says 'Alan Ritter's paper [5] applies deep reinforcement learning to stimulate dialogues,' but the actual reference [5] is Li et al. (2016) on deep reinforcement learning for dialogue generation. The citation appears mislabeled; also, Ritter et al. is already cited as [6]. The reference list needs careful checking.
  3. [Section III] The loss function is described as 'Sparse Categorical Cross Entropy' but the text repeatedly misspells 'categorical' as 'categorial'. Also, the loss formula is written with a subscript style that is not rendered cleanly; please format it properly.
  4. [Fig. 7(c)] The generated text for the single-layer Bidirectional RNN model consists almost entirely of the repeated string 'JOJOJOJO...', which suggests the model collapsed during generation. The text should discuss this failure rather than claiming that the results are understandable.

Circularity Check

1 steps flagged · score 6.0 of 10

The central model-quality ranking reduces to the in-sample training loss that the models were optimized to minimize, with no held-out evaluation.

  1. fitted input called prediction [Section III.A (loss definition), Section IV Table II, Section V Conclusion]
    "In order to check how well or poorly a model is behaving after each optimization or training iteration, a loss value is used and, in this case, Sparse Categorical Cross Entropy loss is used... The loss function is defined as: Loss = -1/N ... [Table II:] Log-Likelihood Training Step Time (ms)... [Conclusion:] loss is least in Bidirectional RNN followed by LSTM and it is most in GRU."

    The quantity compared in Table II as 'Log-Likelihood' is the same Sparse Categorical Cross-Entropy objective defined in Section III.A that each model minimized during training. Consequently, the conclusion's ranking by 'loss is least in Bidirectional RNN...' is not an independent predictive result; it restates the in-sample training objective that the optimizer drove down. The paper reports no held-out split, validation set, or external benchmark in Section IV, and the generated text samples are presented qualitatively without scoring. Thus the central model-quality comparison reduces to reporting the fitted loss values rather than to any prediction on unseen data.

full rationale

The paper's load-bearing comparison is the Section V ranking of LSTM, GRU, and Bidirectional RNN by efficiency and by loss. The loss ranking is circular in the narrow sense that the table values are the very objective minimized during training: the lowest training loss is, by construction, the best fit to the training set, and no generalization target is introduced. The efficiency ranking is not circular, but it is internally contradicted by Table II, which shows GRU step times (24/17/14 ms) faster than LSTM step times (28/24/17 ms) in every configuration, so that claim fails on the paper's own numbers rather than by circularity. The self-citations ([4], [23], [25]) are to the authors' own data and code repository and are not used to justify the ranking, so they are not load-bearing circularity. Overall, the central loss-based conclusion is a restatement of the fitted training objective, meriting partial circularity, while the remaining claims are unsupported correctness issues rather than additional circular steps.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new mathematical or physical entities, but the comparison rests on several hand-chosen hyperparameters and unstated assumptions about the dataset and evaluation. The central ranking is therefore contingent on these choices and assumptions.

free parameters (6)
  • Embedding dimension = 256
    Hand-chosen in Section III.B; affects model capacity and is not derived.
  • Recurrent layer sizes = 1024 / 512,256 / 512,256,128,64
    Hand-selected configurations in Table I with no justification for the specific sizes.
  • Dropout rate = 0.4
    Set in Section III.A after Moon et al.; hand-chosen and not tuned.
  • Learning rate = 1e-3
    RMSprop setting in Section III.A; hand-chosen.
  • Sampling temperature = 1
    Set in Section III.B; affects text generation randomness and is not justified.
  • Context length = 100 characters
    Models retain up to 100-character sequences, stated in Section III.A; hand-chosen.
assumptions (3)
  • domain assumption Character-level next-character prediction on a script corpus captures enough structure to generate dialogue.
    The whole method uses local character statistics; no discourse-level or semantic modeling is attempted.
  • domain assumption Training loss is a valid measure of generation quality.
    The comparisons in Section IV use training loss only; there is no held-out set or human evaluation.
  • domain assumption The annotated dataset from GitHub [24] is accurate and sufficient.
    The paper gives no dataset size, preprocessing details, or validation of the annotation quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LSTM vs. GRU vs. Bidirectional RNN for script generation." pith.science (2026). https://pith.science/paper/WPLL72YH

@misc{pith2026190804332,
  author       = {Pith},
  title        = {Pith review of: LSTM vs. GRU vs. Bidirectional RNN for script generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WPLL72YH}},
  note         = {Machine review of arXiv:1908.04332}
}
read the original abstract

Scripts are an important part of any TV series. They narrate movements, actions and expressions of characters. In this paper, a case study is presented on how different sequence to sequence deep learning models perform in the task of generating new conversations between characters as well as new scenarios on the basis of a script (previous conversations). A comprehensive comparison between these models, namely, LSTM, GRU and Bidirectional RNN is presented. All the models are designed to learn the sequence of recurring characters from the input sequence. Each input sequence will contain, say "n" characters, and the corresponding targets will contain the same number of characters, except, they will be shifted one character to the right. In this manner, input and output sequences are generated and used to train the models. A closer analysis of explored models performance and efficiency is delineated with the help of graph plots and generated texts by taking some input string. These graphs describe both, intraneural performance and interneural model performance for each model.

Figures

Figures reproduced from arXiv: 1908.04332 by the authors.

Figure 1
Figure 1. A table describing all the configuration on which the experiment was [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The image describes architectural design used in this experiment, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. shows a comparative study between all the variants of the neural layers to train and generate text sequence for the scripts, as well as the average time required to train each of these steps on Google Colab running over GCP. While training different models, it was observed that LSTM based neural networks took the least time to execute a training epoch, Bidirectional RNN took the most time and GRU took slightly great… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: An image describing all the losses for all intraneural layers, the [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: The image describes text generated by the trained model after taking “JON: “as an input. (a) represents text generated by single-layered LSTM model. (b) shows text generated by single-layered GRU model and (c) is for single-layered BidirectionalRNN model. A complete li…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 20 canonical work pages

  1. [1]

    Learning long -term dependencies with gradient descent is difficult,

    Y. Bengio, P. Simard and P. Frasconi, "Learning long -term dependencies with gradient descent is difficult," IEEE Transactions on Neural Networks, vol. 5, no. 2, pp. 157-166, 1994

    work page 1994

  2. [2]

    Learning phrase representations using RNN encoder -decoder for statistical machine translation,

    K. a. V. M. B. a. G. C. a. B. D. a. B. F. a. S. H. a. B. Y. Cho, "Learning phrase representations using RNN encoder -decoder for statistical machine translation," arXiv preprint arXiv:1406.1078, 2014

    arXiv 2014

  3. [3]

    A Critical Review of Recurrent Neural Networks,

    Z. C. a. B. J. a. E. C. Lipton, "A Critical Review of Recurrent Neural Networks," arXiv preprint arXiv:1506.00019, 2015

    arXiv 2015

  4. [4]

    Got.pkl,

    S. Mangal, "Got.pkl," GitHub, June 2019. [Online]. Available: https://github.com/sanidhyamangal/gru_lstm_tv_script/blob/master/go t.pkl. [Accessed June 2019]

    work page 2019

  5. [5]

    Deep reinforcement learning for dialogue generation,

    J. a. M. W. a. R. A. a. G. M. a. G. J. a. J. D. Li, "Deep reinforcement learning for dialogue generation," arXiv preprint arXiv:1606.01541, 2016

    arXiv 2016

  6. [6]

    Data -driven response generation in social media,

    A. a. C. C. a. D. W. B. Ritter, "Data -driven response generation in social media," Proceedings of the conference on empirical methods in natural language processing, pp. 583-593, 2011

    work page 2011

  7. [7]

    A neural conversational model,

    O. a. L. Q. Vinyal s, "A neural conversational model," arXiv preprint arXiv:1506.05869, 2015

    arXiv 2015

  8. [8]

    Learning from delayed rewards,

    C. J. C. H. Watkins, "Learning from delayed rewards," King's College, Cambridge, 1989

    work page 1989

Show all 25 references
  1. [9]

    Sequence to sequence learning with neural networks,

    I. a. V. O. a. L. Q. V. Sutskever, "Sequence to sequence learning with neural networks," Advances in neural information processing systems, pp. 3104-3112, 2014

    2014

  2. [10]

    Building end -to-end dialogue systems using generative hierarchical neural ne twork models,

    I. V. a. S. A. a. B. Y. a. C. A. a. P. J. Serban, "Building end -to-end dialogue systems using generative hierarchical neural ne twork models," Thirtieth AAAI Conference on Artificial Intelligence, 2016

    2016

  3. [11]

    Using Markov decision process for learning dialogue strategies,

    R. P. a. W. E. Esther Levin, "Using Markov decision process for learning dialogue strategies," Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181), vol. 1, pp. 201-204, 1998

    1998

  4. [12]

    A Markovian decision process,

    R. Bellman, "A Markovian decision process," Journal of mathematics and mechanics, pp. 679-684, 1957

    1957

  5. [13]

    Training a Real -world POMDP-based Dialogue System,

    B. a. S. J. a. W. K. a. Y. H. a. Y. S. Thomson, "Training a Real -world POMDP-based Dialogue System," Association for Computational Linguistics, pp. 9-16, 2007

    2007

  6. [14]

    Continuously learning neural dialogue management,

    P.-H. a. G. M. a. M. N. a. R.-B. L. a. U. S. a. V. D. a. W. T.-H. a. Y. S. Su, "Continuously learning neural dialogue management," arXiv preprint arXiv:1606.02689, 2016

    2016 arXiv

  7. [15]

    Magenta,

    "Magenta," Google, [Online]. Available: https://magenta.tensorflow.org/. [Accessed 29 October 2018]

    2018

  8. [16]

    Word embedding by Keras,

    "Word embedding by Keras," Learn Neural Networks, [Online]. Available: https://learn -neural-networks.com/world-embedding-by- keras/. [Accessed July 2019]

    2019

  9. [17]

    Dropout: A Simple Way to Prevent Neural Networks from Overfitting,

    G. H. A. K. I. S. R. S. Nitish Srivastava, "Dropout: A Simple Way to Prevent Neural Networks from Overfitting," Journal of Machine Learning Research, vol. 15, pp. 1929-1958, 2014

    1929

  10. [18]

    Densely connected convolutional networks.,

    G. L. Z. V. D. M. L. &. W. K. Q. Huang, " Densely connected convolutional networks.," CVPR, vol. 1, p. 3, 2017

    2017

  11. [19]

    Rnndrop: A novel dropout for rnns in asr,

    C. H. L. H. S. I. Moon T, "Rnndrop: A novel dropout for rnns in asr," Automatic Speech Recognition and Understanding (ASRU), pp. 65-70, 2015

    2015

  12. [20]

    Lecture 6.5 -rmsprop: Divide the gradient by a running average of its recent magnitude,

    T. a. H. G. Tieleman, "Lecture 6.5 -rmsprop: Divide the gradient by a running average of its recent magnitude," COURSERA: Neural networks for machine learning, vol. 4, no. 2, pp. 26-31, 2012

    2012

  13. [21]

    pickle — Python object serialization,

    "pickle — Python object serialization," Python Software Foundation, [Online]. Available: https://docs.python.org/3/library/pickle.html. [Accessed June 2019]

    2019

  14. [22]

    Learning representations by back-propagating errors,

    D. E. a. H. G. E. a. W. R. J. a. o. Rumelhart, "Learning representations by back-propagating errors," Cognitive modeling, vol. 5, no. 3, p. 1, 1988

    1988

  15. [23]

    gru_lstm_tv_script,

    R. M. P. J. Sanidhya Mangal, "gru_lstm_tv_script," GitHub, [Online]. Available: https://github.com/sanidhyamangal/gru_lstm_tv_script. [Accessed June 2019]

    2019

  16. [24]

    Game of Thrones Data,

    S. Koirala, "Game of Thrones Data," GitHub, [Online]. Available: https://github.com/shekharkoirala/Game_of_Thrones/tree/master/Data . [Accessed June 2019]

    2019

  17. [25]

    generated_text,

    R. M. P. J. Sanidhya Mangal, "generated_text," GitHub, [Online]. Available: https://github.com/sanidhyamangal/gru_lstm_tv_script/tree/master/ge nerated_text. [Accessed July 2019]

    2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.