Pith. sign in

REVIEW 4 major objections 5 minor 26 references

The paper introduces NewsReX, a Keras 3 / JAX library that re-implements classic news recommendation models and reports that it cuts total training time on a consumer GPU by up to 41 percent compared with an existing library, without sacrif

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A JAX-based news recommendation library claims 37-41% total training-time speedups over NewsRecLib for NRMS and LSTUR on MIND-small, with additional ablation studies.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful library with a strong practical angle, but the NAML speedup math and an apparent test-set exposure sentence make the current draft unreliable. the 4 major comments →

arxiv 2508.21572 v1 pith:NUUC3GQM submitted 2025-08-29 cs.IR

NewsReX: A More Efficient Approach to News Recommendation with Keras 3 and JAX

Igor L.R. Azevedo , Toyotaro Suzumura , Yuichiro Yasui This is my paper
classification cs.IR
keywords news recommendationreproducibilityKeras 3JAXJIT compilationneural news modelsnegative samplingMIND dataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents NewsReX, an open-source library for neural news recommendation built on Keras 3 with the JAX backend. Its claim is that re-implementing the standard models NRMS, NAML, and LSTUR on this stack makes reproducing benchmark experiments much faster, especially on consumer hardware such as an 8GB RTX 3060 Ti. The headline numbers are a 41.0% cut in total experiment time for NRMS on MIND-small (66.2 to 39.0 minutes) and a 37.0% cut for LSTUR (73.5 to 46.3 minutes), with no reported loss in accuracy. Alongside the speed claim, the paper reports ablations on validation-split strategy, negative sampling order, batch size, and epoch count, plus a demonstration on a proprietary Japanese news dataset. A sympathetic reading is that the field's reproducibility problem is partly a compute problem, and this library is offered as a faster, accessible path.

Core claim

On the paper's own terms, the discovery is that a performance-oriented reimplementation of standard neural news recommendation models using Keras 3's JAX backend reproduces the accuracy of established implementations while consuming substantially less wall-clock time. The difference is attributed to JIT compilation (with warmup), vectorized JAX metric computation, and precomputed user/news embeddings cached during evaluation. The paper validates this on MIND-small and a proprietary Japanese news dataset for NRMS, NAML, and LSTUR, and reports that the speed advantage persists on a low-memory consumer GPU.

What carries the argument

The library's architecture pairs the standard news-encoder/user-encoder decomposition with a JAX backend that uses Just-In-Time compilation and XLA optimization, so the first epoch pays a one-time compilation cost and later epochs run on an optimized graph. A custom JAX metrics calculator uses jit and vmap for vectorized batch evaluation, and fast evaluation precomputes and caches news and user embeddings. The combination of JIT warmup, precomputation, and vectorization is what the paper credits for the measured speedups.

Load-bearing premise

The speed comparison presupposes that the two frameworks being compared use identical model architectures, hyperparameters, and data pipelines, so the measured difference reflects the JAX backend rather than differences in configuration or evaluation frequency.

What would settle it

Run the same published configuration (MIND-small, batch size 16, 5 epochs, 32-bit precision) for NRMS, NAML, and LSTUR with the proposed library and the leading existing library on the same RTX 3060 Ti, logging total wall-clock time and the same checkpoints per epoch; if total times match within noise, the claimed backend speedup fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Reproducing standard news recommendation experiments on consumer GPUs becomes practical, since total experiment time drops to under an hour for NRMS and LSTUR on MIND-small.
  • The JIT warmup profile means longer multi-epoch runs amortize the one-time compilation cost, so the speedup grows with training length.
  • The ablation tables provide concrete defaults and warnings: random validation splits scored better than chronological ones in their tests, unshuffled negatives scored slightly better than shuffled, and NRMS plateaued after 10-20 epochs.
  • The framework includes visualization tools that surface dataset skew and model bias, e.g., a treemap showing over-recommendation of popular subcategories.
  • Custom multilingual datasets can be trained with the provided scripts, demonstrated on a proprietary Japanese news dataset.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the speedup transfers to other models and larger MIND splits, JAX-backed implementations could become the default baseline environment, making cross-paper comparisons cheaper and shifting the reproducibility bottleneck from compute to configuration reporting.
  • The ablation result that random validation splits outperform chronological ones is a warning for temporal-split benchmarks: it suggests test-set contamination or distribution drift may be conflated with model quality in existing comparisons.
  • A direct, independent replication that documents the baseline's exact negative-sampling order, evaluation cadence, and data-loading settings would determine how much of the 41%/37% gap is backend efficiency versus undocumented setup differences.
  • The negative-sampling order effect (unshuffled positives-first scoring slightly higher) could interact with the common practice of shuffling candidate lists at inference, and is worth testing on larger datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces NewsReX, an open-source library for neural news recommendation built on Keras 3 and JAX. The main claimed contribution is computational efficiency: on a consumer GPU (RTX 3060 Ti), NewsReX is reported to reduce total experiment time for NRMS on MIND-small from 66.2 to 39.0 minutes (41.0% speedup) and for LSTUR from 73.5 to 46.3 minutes (37.0% speedup) relative to NewsRecLib. The paper also reports accuracy results (AUC, MRR, nDCG@5/10) for NRMS, NAML, and LSTUR on MIND-small and a proprietary Nikkei dataset, offers ablation studies on validation-set splitting, negative-sampling strategies, batch size, and number of epochs, and describes visualization tools for dataset and model analysis.

Significance. If the efficiency claims are validated, NewsReX would be a useful contribution to the news-recommendation community by lowering the hardware barrier for reproducing standard experiments. The library is open source, includes a modular design, and provides multilingual support, which are concrete strengths. The ablation studies, while modest in scope, could serve as practical reference points. However, the paper's central claims rest on timing comparisons that are not rigorously reported, and an explicit statement about test-set exposure undermines the accuracy results. The potential significance is real, but the evidence as presented is not yet reliable.

major comments (4)
  1. [§3.2.2, Figure 7] The NAML timing result is internally inconsistent. The paper reports NewsRecLib averaging 11.66 min/epoch and NewsReX 7.82 min/epoch over 5 epochs, a per-epoch training-time reduction of ~32.9%, yet the total-experiment speedup for NAML is only 3.1%. No NAML total times are given, and 'total experiment time' is never defined. If the per-epoch metric is the main driver, a 3.1% total speedup implies that the vast majority of wall-clock time is spent outside training; this must be explained, and the same decomposition should be provided for NRMS and LSTUR. Without this, the headline speedups cannot be interpreted.
  2. [§4.1, Table 2] The paper states: 'It is important to highlight that under any scenario the model has been exposed to samples from the test dataset.' Taken literally, this means the accuracy and ablation results in Tables 1–5 are invalid because test samples contaminated training or validation. This is a load-bearing issue for the empirical claims. The authors must either correct this statement, clarify the intended wording, or rerun all experiments with a clean split. As written, the accuracy comparisons do not support the conclusions.
  3. [§3.2] The timing benchmark lacks essential experimental rigor. No error bars, standard deviations, or number of repeated runs are reported for any timing measurement, in contrast to the accuracy experiments which state 'at least three' runs. Additionally, the NewsRecLib configuration is not documented: exact model architecture, negative sampling setting, evaluation cadence, data-loading pipeline, and any framework-specific overhead are unspecified. Without evidence that the two frameworks perform the same work per epoch, the reported speedup may reflect configuration differences rather than backend efficiency. This concern applies directly to the central efficiency claim.
  4. [§3.2.2] The comparison with Microsoft Recommenders is only described as 'around 60 minutes per epoch' with no hardware, model, or dataset details. This is too vague to be a meaningful benchmark and should be either removed or properly quantified with the same methodology as the NewsRecLib comparison.
minor comments (5)
  1. [Abstract / §6] The abstract states 'Code available at https://github.com/igor17400/NewsReX', but §6 says 'will be uploaded to GitHub upon conference acceptance.' Please reconcile this discrepancy and provide a working repository link at submission time.
  2. [§2.5] The 'intelligent JIT warmup strategy' is mentioned but not described in enough detail to be reproducible. A precise explanation of how warmup is triggered and measured would strengthen the paper.
  3. [Figure 5] The per-epoch timing figure would benefit from error bars or at least a note on the number of repetitions. Also, the y-axis units are not explicitly stated in the caption.
  4. [§4.4, Table 5] The NRMS results for 10, 20, 40, and 80 epochs are identical to several decimal places. This is plausible if the model converges early, but it would be helpful to report the actual loss curves or a small variance measure to confirm that the values are not artifacts of rounding or a fixed random seed.
  5. [Global] There are several typos and grammar issues (e.g., 'Inherintance' in Figure 1, 'factured' in Figure 2, 'the speed up it also achieved'). A careful proofread is needed.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims are empirical measurements, not derived predictions.

full rationale

This is an empirical systems paper. The central claims are wall-clock training-time comparisons (Section 3.2.2) and accuracy numbers (Table 1), both obtained by running NewsReX and NewsRecLib under stated settings. There is no derivation chain in which an output is defined in terms of an input, no fitted parameter is renamed as a prediction, and no load-bearing uniqueness theorem is imported from the authors' prior work. The only self-citation is the mention of [22], a GNN-based user-encoder paper by one co-author, used as background and not as justification for any result. The manuscript does contain a serious methodological admission in Section 4.1 that the model 'has been exposed to samples from the test dataset' and an apparent internal inconsistency in the NAML timing numbers (11.66 vs 7.82 min/epoch but only 3.1% total speedup), but these are correctness and reproducibility concerns about experimental measurement, not circularity of the kind this review targets. Because nothing is predicted from fitted values and no argument reduces to a self-citation, the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 0 invented entities

The central claims rely on empirical measurement rather than theoretical derivation. The main free parameters are the experimental hyperparameters chosen by the authors. The authors assume fair comparison with NewsRecLib and no data leakage, both of which are questioned by the text.

free parameters (4)
  • learning_rate = 0.0001
    Chosen by hand for all experiments (Section 3.1). The efficiency and accuracy results may depend on this value.
  • batch_size = 16 (default), varied 16-256 in Section 4.3
    Chosen by hand for the main comparisons; the ablation shows batch size affects accuracy, so timing comparisons may also be affected.
  • number_of_epochs = 5 (main), 10-80 in Section 4.4
    Selected by the authors; the ablation claims a plateau after 10 epochs, but the main efficiency numbers use 5.
  • negative_sampling_k = not stated
    The number of negative samples per positive is not reported in the text, yet it is a key component of the training setup and could affect both speed and accuracy.
axioms (3)
  • domain assumption NewsRecLib and NewsReX are configured with equivalent models, hyperparameters, and data pipelines for the timing comparison.
    Section 3.2 claims a standardized setup but does not specify NewsRecLib's exact configuration, so the speedup is only valid if the comparison is fair.
  • domain assumption The MIND-small test set was not used during training or validation.
    Section 4.1 states the opposite: 'under any scenario the model has been exposed to samples from the test dataset'. If this is true, all accuracy results are invalid.
  • standard math The attention mechanisms described in Section 2.1 (Equations 1-4) are implemented correctly and are standard.
    The framework's correctness depends on standard attention and softmax operations, which are not formally verified.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of NewsReX: A More Efficient Approach to News Recommendation with Keras 3 and JAX." pith.science (2026). https://pith.science/paper/NUUC3GQM

@misc{pith2026250821572,
  author       = {Pith},
  title        = {Pith review of: NewsReX: A More Efficient Approach to News Recommendation with Keras 3 and JAX},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUUC3GQM}},
  note         = {Machine review of arXiv:2508.21572}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reproducing and comparing results in news recommendation research has become increasingly difficult. This is due to a fragmented ecosystem of diverse codebases, varied configurations, and mainly due to resource-intensive models. We introduce NewsReX, an open-source library designed to streamline this process. Our key contribution is a modern implementation built on Keras 3 and JAX, which provides an increase in computational efficiency. Experiments show that NewsReX is faster than current implementations. To support broader research, we provide a straightforward guide and scripts for training models on custom datasets. We validated this functionality using a proprietary Japanese news dataset from Nikkei News, a leading Japanese media corporation renowned for its comprehensive coverage of business, economic, and financial news. NewsReX makes reproducing complex experiments faster and more accessible to a wider range of hardware making sure the speed up it also achieved for less powerful GPUs, like an 8GB RTX 3060 Ti. Beyond the library, this paper offers an analysis of key training parameters often overlooked in the literature, including the effect of different negative sampling strategies, the varying number of epochs, the impact of random batching, and more. This supplementary analysis serves as a valuable reference for future research, aiming to reduce redundant computation when comparing baselines and guide best practices. Code available at https://github.com/igor17400/NewsReX.

Figures

Figures reproduced from arXiv: 2508.21572 by Igor L.R. Azevedo, Toyotaro Suzumura, Yuichiro Yasui.

Figure 1
Figure 1. Figure 1: Model Inherintance and Component Architecture [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Dataset Class Hierarchy The system handles language-specific preprocessing, such as tokenization and normalization, and supports various embedding strategies like GloVe for English and BPEmb for multilingual sce￾narios. To optimize I/O and loading times, the data pipeline uses a pickle-based caching system that serializes processed datasets, vocabularies, and embedding matrices for quick retrieval. 2.4 Exp… view at source ↗
Figure 5
Figure 5. Figure 5: Training time per epoch comparison for the NAML, NRMS, and LSTUR models, comparing [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Average training time per epoch for the NAML, [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Total experiment time and time saved by NewsReX compared to NewsRecLib for the NRMS, NAML, and LSTUR models on the MIND-small dataset. test set, where our models are evaluated for the MIND-small and proprietary Nikkei News dataset. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Heatmap visualizations of article performance on the test sets for the MIND-small and proprietary Nikkei News [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Treemap visualization comparing the distribution of user’s Ground Truth Clicks with the Top Recommendations [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 13 canonical work pages · 4 internal anchors

  1. [1]

    Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, San- jay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Mur...

  2. [2]

    Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019. Neural News Recommendation with Long- and Short-term User Representations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Anna Korhonen, David Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, Florence, Ital...

    work page 2019

  3. [3]

    Lukas Biewald. 2020. Experiment Tracking with Weights and Biases. https: //www.wandb.com/ Software available from wandb.com

    2020

  4. [4]

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: composable transformations of Python+NumPy programs. http://github.com/jax-ml/jax

    work page 2018

  5. [5]

    François Chollet et al. 2015. Keras. https://keras.io

    work page 2015

  6. [6]

    William Falcon and The PyTorch Lightning team. 2019. PyTorch Lightning. doi:10.5281/zenodo.3828935

    doi:10.5281/zenodo.3828935 2019

  7. [7]

    Shen Gao, Jiabao Fang, Quan Tu, Zhitao Yao, Zhumin Chen, Pengjie Ren, and Zhaochun Ren. 2024. Generative News Recommendation. arXiv:2403.03424 [cs.IR] https://arxiv.org/abs/2403.03424

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [8]

    Scott Graham, Jun-Ki Min, and Tao Wu. 2019. Microsoft recommenders: tools to accelerate developing recommender systems. In Proceedings of the 13th ACM Conference on Recommender Systems (Copenhagen, Denmark) (Rec- Sys ’19). Association for Computing Machinery, New York, NY, USA, 542–543. doi:10.1145/3298689.3346967

    arXiv 2019

  9. [9]

    Benjamin Heinzerling and Michael Strube. 2018. BPEmb: Tokenization-free Pre- trained Subword Embeddings in 275 Languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Nico- letta Calzolari (Conference chair), Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi ...

    work page 2018

  10. [10]

    Andreea Iana, Goran Glavaš, and Heiko Paulheim. 2023. NewsRecLib: A PyTorch- Lightning Library for Neural News Recommendation. arXiv:2310.01146 [cs.IR] https://arxiv.org/abs/2310.01146

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  11. [11]

    Yunyong Ko, Seongeun Ryu, and Sang-Wook Kim. 2025. CROWN: A Novel Approach to Comprehending Users’ Preferences for Accurate Personalized News Recommendation. In Proceedings of the ACM on Web Conference 2025 (WWW ’25). ACM, 1911–1921. doi:10.1145/3696410.3714752

    arXiv 2025

  12. [12]

    Haokai Ma, Ruobing Xie, Lei Meng, Xin Chen, Xu Zhang, Leyu Lin, and Jie Zhou

  13. [13]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Alessandro Mos- chitti, Bo Pang, and Walter Daelemans (Eds.). Association for Computational Linguistics, Doha, Qatar, 1532–1543. doi:10.3115/v1/D14-1162

    doi:10.3115/v1/d14-1162 2014

  14. [14]

    Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2021. Pp-rec: News recommendation with personalized user interest and time-aware news popularity. arXiv preprint arXiv:2106.01300 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  15. [15]

    Tao Qi, Fangzhao Wu, Chuhan Wu, and Yongfeng Huang. 2022. News rec- ommendation with candidate-aware user modeling. In Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 1917–1921

    work page 2022

  16. [16]

    Wentao Shi, Jiawei Chen, Fuli Feng, Jizhi Zhang, Junkang Wu, Chongming Gao, and Xiangnan He. 2023. On the theories behind hard negative sampling for recommendation. In Proceedings of the ACM Web Conference 2023 . 812–822

    2023

  17. [17]

    Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019. Neural News Recommendation with Attentive Multi-View Learning. arXiv:1907.05576 [cs.CL] https://arxiv.org/abs/1907.05576

    Pith/arXiv arXiv 2019

  18. [18]

    Chuhan Wu, Fangzhao Wu, Mingxiao An, Jianqiang Huang, Yongfeng Huang, and Xing Xie. 2019. NPA: neural news recommendation with personalized attention. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining . 2576–2584

    work page 2019

  19. [19]

    Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie

    work page

  20. [20]

    Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020. MIND: A Large-scale Dataset for News Recommendation. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.)....

    doi:10.18653/v1/2020.acl-main.331 2020

  21. [21]

    Omry Yadan. 2019. Hydra - A framework for elegantly configuring complex applications. Github. https://github.com/facebookresearch/hydra

    2019

  22. [22]

    Boming Yang, Dairui Liu, Toyotaro Suzumura, Ruihai Dong, and Irene Li. 2023. Going Beyond Local: Global Graph-Enhanced Personalized News Recommenda- tions. In Proceedings of the 17th ACM conference on recommender systems . 24–34

    work page 2023

  23. [23]

    Yang Yu, Fangzhao Wu, Chuhan Wu, Jingwei Yi, and Qi Liu. 2022. Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation. arXiv:2112.00944 [cs.IR] https://arxiv.org/abs/2112.00944 9

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  24. [2015]

    https://www.tensorflow.org/ Software available from tensorflow.org

    TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/ Software available from tensorflow.org

  25. [2019]

    Neural News Recommendation with Multi-Head Self-Attention. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNLP), Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, Hong Ko...

    doi:10.18653/v1/d19-1671 2019

  26. [2023]

    In Proceedings of the 17th ACM conference on recommender systems

    Exploring false hard negative sample in cross-domain recommendation. In Proceedings of the 17th ACM conference on recommender systems . 502–514

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.