{"id":"5e5af950-8a9d-466b-bb4e-fc21339ed71a","arxiv_id":"2508.21572","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A JAX-based news recommendation library claims 37-41% total training-time speedups over NewsRecLib for NRMS and LSTUR on MIND-small, with additional ablation studies.","lead":"NewsReX is a new open-source library that re-implements three classic news recommendation models in Keras 3 with a JAX backend, claiming faster training on consumer GPUs. The paper also reports ablation experiments on the MIND dataset and a Japanese news dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NAML timing inconsistency and admitted test-set exposure undermine the reliability of the central efficiency and accuracy claims.","rationale":"The reader's weakest_assumption was that NewsReX and NewsRecLib implement equivalent configurations, so the timing difference is attributable to the JAX backend. This is a valid concern, but I found a more concrete and load-bearing problem: the NAML timing numbers in §3.2.2 are arithmetically incompatible under any reasonable interpretation, which directly undermines the central efficiency claim. The admitted test-set exposure in §4.1 independently invalidates all accuracy results and strengthens the case that the experimental methodology is not sound. I agree with the reader's REJECT verdict; the paper's contribution is empirical, and the empirical support is unreliable. My recommendation remains REJECT because the central claims cannot be trusted without reproducing the benchmark with a controlled, documented setup and without clarifying the test-set exposure.","tokens_in":10872,"tokens_out":3602,"duration_ms":42003,"concrete_test":"Reproduce the §3.2 benchmark with the released NewsReX code and the exact NewsRecLib configuration used for the comparison, instrumenting each stage (JIT warmup, per-epoch training, validation, test) over at least 5 runs. Verify whether 5×(7.82)+overhead and 5×(11.66)+overhead can produce the reported 3.1% NAML total speedup; if not, the published timing numbers are arithmetically inconsistent. Separately, inspect the released data pipeline to confirm whether MIND-small test impressions appear in the training or validation split, which would confirm the §4.1 leakage.","verdict_should_be":"REJECT","load_bearing_attack":"The headline efficiency claim rests on the wall-clock comparison in §3.2.2. The NAML result is internally inconsistent: NewsRecLib averaged 11.66 min/epoch and NewsReX 7.82 min/epoch over 5 epochs, a per-epoch training-time reduction of ~32.9%, yet the reported total-experiment speedup is only 3.1%. A 3.1% total speedup implies a total runtime of roughly 620 minutes, meaning ~90% of the experiment time lies outside the reported per-epoch training time. The paper never defines what 'total experiment time' includes (evaluation? data loading? multiple runs?), reports no NAML total times, no error bars, and no NewsRecLib configuration details. This discrepancy suggests the per-epoch metric is not the driver of total time, or that the two frameworks are not doing comparable work. If the metric is misleading for NAML, the headline NRMS (41%) and LSTUR (37%) speedups are also called into question. Additionally, §4.1 states the model was 'exposed to samples from the test dataset' under any scenario, which invalidates the accuracy tables if training used test samples; this also indicates a data pipeline that may differ from NewsRecLib's, further confounding the timing comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces NewsReX, an open-source library for neural news recommendation built on Keras 3 and JAX. The main claimed contribution is computational efficiency: on a consumer GPU (RTX 3060 Ti), NewsReX is reported to reduce total experiment time for NRMS on MIND-small from 66.2 to 39.0 minutes (41.0% speedup) and for LSTUR from 73.5 to 46.3 minutes (37.0% speedup) relative to NewsRecLib. The paper also reports accuracy results (AUC, MRR, nDCG@5/10) for NRMS, NAML, and LSTUR on MIND-small and a proprietary Nikkei dataset, offers ablation studies on validation-set splitting, negative-sampling strategies, batch size, and number of epochs, and describes visualization tools for dataset and model analysis.","tokens_in":11133,"tokens_out":2399,"duration_ms":25065,"significance":"If the efficiency claims are validated, NewsReX would be a useful contribution to the news-recommendation community by lowering the hardware barrier for reproducing standard experiments. The library is open source, includes a modular design, and provides multilingual support, which are concrete strengths. The ablation studies, while modest in scope, could serve as practical reference points. However, the paper's central claims rest on timing comparisons that are not rigorously reported, and an explicit statement about test-set exposure undermines the accuracy results. The potential significance is real, but the evidence as presented is not yet reliable.","major_comments":[{"comment":"The NAML timing result is internally inconsistent. The paper reports NewsRecLib averaging 11.66 min/epoch and NewsReX 7.82 min/epoch over 5 epochs, a per-epoch training-time reduction of ~32.9%, yet the total-experiment speedup for NAML is only 3.1%. No NAML total times are given, and 'total experiment time' is never defined. If the per-epoch metric is the main driver, a 3.1% total speedup implies that the vast majority of wall-clock time is spent outside training; this must be explained, and the same decomposition should be provided for NRMS and LSTUR. Without this, the headline speedups cannot be interpreted.","section":"§3.2.2, Figure 7"},{"comment":"The paper states: 'It is important to highlight that under any scenario the model has been exposed to samples from the test dataset.' Taken literally, this means the accuracy and ablation results in Tables 1–5 are invalid because test samples contaminated training or validation. This is a load-bearing issue for the empirical claims. The authors must either correct this statement, clarify the intended wording, or rerun all experiments with a clean split. As written, the accuracy comparisons do not support the conclusions.","section":"§4.1, Table 2"},{"comment":"The timing benchmark lacks essential experimental rigor. No error bars, standard deviations, or number of repeated runs are reported for any timing measurement, in contrast to the accuracy experiments which state 'at least three' runs. Additionally, the NewsRecLib configuration is not documented: exact model architecture, negative sampling setting, evaluation cadence, data-loading pipeline, and any framework-specific overhead are unspecified. Without evidence that the two frameworks perform the same work per epoch, the reported speedup may reflect configuration differences rather than backend efficiency. This concern applies directly to the central efficiency claim.","section":"§3.2"},{"comment":"The comparison with Microsoft Recommenders is only described as 'around 60 minutes per epoch' with no hardware, model, or dataset details. This is too vague to be a meaningful benchmark and should be either removed or properly quantified with the same methodology as the NewsRecLib comparison.","section":"§3.2.2"}],"minor_comments":[{"comment":"The abstract states 'Code available at https://github.com/igor17400/NewsReX', but §6 says 'will be uploaded to GitHub upon conference acceptance.' Please reconcile this discrepancy and provide a working repository link at submission time.","section":"Abstract / §6"},{"comment":"The 'intelligent JIT warmup strategy' is mentioned but not described in enough detail to be reproducible. A precise explanation of how warmup is triggered and measured would strengthen the paper.","section":"§2.5"},{"comment":"The per-epoch timing figure would benefit from error bars or at least a note on the number of repetitions. Also, the y-axis units are not explicitly stated in the caption.","section":"Figure 5"},{"comment":"The NRMS results for 10, 20, 40, and 80 epochs are identical to several decimal places. This is plausible if the model converges early, but it would be helpful to report the actual loss curves or a small variance measure to confirm that the values are not artifacts of rounding or a fixed random seed.","section":"§4.4, Table 5"},{"comment":"There are several typos and grammar issues (e.g., 'Inherintance' in Figure 1, 'factured' in Figure 2, 'the speed up it also achieved'). A careful proofread is needed.","section":"Global"}],"recommendation":"major_revision","confidential_remarks":"The test-set exposure admission in §4.1 is the most serious issue and should be flagged prominently to the authors. If it is a wording error, it must be corrected; if it is literal, the empirical section needs to be redone. The NAML timing inconsistency also suggests that the efficiency comparison is not yet trustworthy. The paper has a useful systems contribution, but in its current form the core claims are not substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, it ships a real artifact: NewsReX, a Keras 3/JAX library covering the standard NNR models (NRMS, NAML, LSTUR), with Hydra configs, multilingual embeddings, visualizations, and a set of ablations on MIND-small plus a proprietary Nikkei dataset. The ablations—batch size, epochs, negative sampling order, random-vs-chronological validation—are genuinely useful reference points for anyone running these experiments. The efficiency angle, making NNR training feasible on an 8GB consumer GPU, is a meaningful practical goal.\n\nSecond, the paper's central speedup claim has a hole. For NAML, NewsReX averages 7.82 min/epoch vs NewsRecLib's 11.66, a ~33% reduction in training time, yet the total experiment speedup is reported as 3.1%. With NRMS (41%) and LSTUR (37%), the total speedups roughly match the per-epoch reductions, which implies the two frameworks have similar non-training overhead. For NAML to come out at 3.1%, either the experiment is dominated by non-training time or the numbers aren't measuring the same thing. The paper never defines \"total experiment time,\" gives no error bars or repeated runs, and doesn't document NewsRecLib's configuration (negative sampling k, evaluation cadence, data pipeline). That's a fixable but serious omission in an otherwise empirical paper.\n\nThe more alarming item is §4.1's sentence: \"under any scenario the model has been exposed to samples from the test dataset.\" If that is literal, it is test-set leakage and invalidates Tables 1–5. I suspect a dropped \"not,\" but the preprint cannot stand with that sentence as written. There is also a minor contradiction between the abstract (code is at GitHub) and the conclusion (code will be uploaded upon acceptance).\n\nWhat the paper does well: the library design seems sensible, the JIT warmup and vectorized metrics are concrete engineering choices, and the ablation results are in the right ballpark for MIND-small. The per-epoch timing figures, apart from NAML, are plausible.\n\nWho is this for? Practitioners reproducing NNR baselines on consumer hardware, and people building on NewsRecLib. Not theorists. The accuracy tables are only usable once the test-exposure ambiguity is cleared up.\n\nMy recommendation: send to peer review, not desk reject. The artifact is concrete and the problems are addressable. A good referee should demand a definition of total experiment time, error bars, and a direct confirmation that no test samples touched training or validation. If the §4.1 sentence is literal, it's a reject; otherwise, this could become a useful library paper.","headline":"Useful library with a strong practical angle, but the NAML speedup math and an apparent test-set exposure sentence make the current draft unreliable.","tokens_in":11628,"tokens_out":8616,"would_cite":false,"duration_ms":74674,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces NewsReX, a Keras 3 / JAX library that re-implements classic news recommendation models and reports that it cuts total training time on a consumer GPU by up to 41 percent compared with an existing library, without sacrif","keywords":["news recommendation","reproducibility","Keras 3","JAX","JIT compilation","neural news models","negative sampling","MIND dataset"],"falsifier":"Run the same published configuration (MIND-small, batch size 16, 5 epochs, 32-bit precision) for NRMS, NAML, and LSTUR with the proposed library and the leading existing library on the same RTX 3060 Ti, logging total wall-clock time and the same checkpoints per epoch; if total times match within noise, the claimed backend speedup fails.","tokens_in":10767,"feed_emoji":"⚡","tokens_out":5781,"duration_ms":57422,"temperature":0.7,"pith_summary":"The paper presents NewsReX, an open-source library for neural news recommendation built on Keras 3 with the JAX backend. Its claim is that re-implementing the standard models NRMS, NAML, and LSTUR on this stack makes reproducing benchmark experiments much faster, especially on consumer hardware such as an 8GB RTX 3060 Ti. The headline numbers are a 41.0% cut in total experiment time for NRMS on MIND-small (66.2 to 39.0 minutes) and a 37.0% cut for LSTUR (73.5 to 46.3 minutes), with no reported loss in accuracy. Alongside the speed claim, the paper reports ablations on validation-split strategy, negative sampling order, batch size, and epoch count, plus a demonstration on a proprietary Japanese news dataset. A sympathetic reading is that the field's reproducibility problem is partly a compute problem, and this library is offered as a faster, accessible path.","feed_headline":"JAX-backed news recommender library cuts training time by 41%","feed_subtitle":"Standard benchmarks become reproducible on 8GB consumer GPUs in under an hour.","key_machinery":"The library's architecture pairs the standard news-encoder/user-encoder decomposition with a JAX backend that uses Just-In-Time compilation and XLA optimization, so the first epoch pays a one-time compilation cost and later epochs run on an optimized graph. A custom JAX metrics calculator uses jit and vmap for vectorized batch evaluation, and fast evaluation precomputes and caches news and user embeddings. The combination of JIT warmup, precomputation, and vectorization is what the paper credits for the measured speedups.","core_discovery":"On the paper's own terms, the discovery is that a performance-oriented reimplementation of standard neural news recommendation models using Keras 3's JAX backend reproduces the accuracy of established implementations while consuming substantially less wall-clock time. The difference is attributed to JIT compilation (with warmup), vectorized JAX metric computation, and precomputed user/news embeddings cached during evaluation. The paper validates this on MIND-small and a proprietary Japanese news dataset for NRMS, NAML, and LSTUR, and reports that the speed advantage persists on a low-memory consumer GPU.","pith_inferences":["If the speedup transfers to other models and larger MIND splits, JAX-backed implementations could become the default baseline environment, making cross-paper comparisons cheaper and shifting the reproducibility bottleneck from compute to configuration reporting.","The ablation result that random validation splits outperform chronological ones is a warning for temporal-split benchmarks: it suggests test-set contamination or distribution drift may be conflated with model quality in existing comparisons.","A direct, independent replication that documents the baseline's exact negative-sampling order, evaluation cadence, and data-loading settings would determine how much of the 41%/37% gap is backend efficiency versus undocumented setup differences.","The negative-sampling order effect (unshuffled positives-first scoring slightly higher) could interact with the common practice of shuffling candidate lists at inference, and is worth testing on larger datasets."],"forward_implications":["Reproducing standard news recommendation experiments on consumer GPUs becomes practical, since total experiment time drops to under an hour for NRMS and LSTUR on MIND-small.","The JIT warmup profile means longer multi-epoch runs amortize the one-time compilation cost, so the speedup grows with training length.","The ablation tables provide concrete defaults and warnings: random validation splits scored better than chronological ones in their tests, unshuffled negatives scored slightly better than shuffled, and NRMS plateaued after 10-20 epochs.","The framework includes visualization tools that surface dataset skew and model bias, e.g., a treemap showing over-recommendation of popular subcategories.","Custom multilingual datasets can be trained with the provided scripts, demonstrated on a proprietary Japanese news dataset."],"supporting_citations":[{"why":"Serves as the main baseline for all per-epoch and total-time comparisons.","marker":"[10]"},{"why":"Defines the NRMS architecture whose implementation and speedup are reported.","marker":"[19]"},{"why":"Defines the LSTUR architecture used in the second benchmarked speedup.","marker":"[2]"},{"why":"Defines the NAML architecture, the third model benchmarked against the baseline.","marker":"[17]"},{"why":"Supplies the public MIND-small dataset used for the efficiency and ablation experiments.","marker":"[20]"},{"why":"The JAX backend whose JIT compilation and vectorization are credited with the speedups.","marker":"[4]"},{"why":"The Keras 3 API that the library's models and encoders are built on.","marker":"[5]"},{"why":"The earlier library from which the fast-evaluation/precomputation idea is adapted.","marker":"[8]"},{"why":"Provides the multilingual subword embeddings that enable the Japanese dataset experiments.","marker":"[9]"},{"why":"Provides the English word embeddings used by default in the MIND experiments.","marker":"[13]"}],"fun_headline_variants":["JAX-backed NewsReX cuts news recommender training time by 41%","NewsReX: Reproduce news recommender benchmarks on 8GB GPUs in under an hour","Faster news recommendation training with NewsReX on Keras 3 and JAX","NewsReX library reproduces accuracy with 41% lower training time"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The speed comparison presupposes that the two frameworks being compared use identical model architectures, hyperparameters, and data pipelines, so the measured difference reflects the JAX backend rather than differences in configuration or evaluation frequency.","fun_headline_variants_meta":{"raw":{"variants":["JAX-backed NewsReX cuts news recommender training time by 41%","NewsReX: Reproduce news recommender benchmarks on 8GB GPUs in under an hour","Faster news recommendation training with NewsReX on Keras 3 and JAX","NewsReX library reproduces accuracy with 41% lower training time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00036,"raw_usage":{"total_tokens":1784,"prompt_tokens":749,"completion_tokens":1035,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":945}},"tokens_in":493,"tokens_out":1035,"duration_ms":8662,"temperature":1.0,"reasoning_tokens":945,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:10:58.132870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same published configuration (MIND-small, batch size 16, 5 epochs, 32-bit precision) for NRMS, NAML, and LSTUR with the proposed library and the leading existing library on the same RTX 3060 Ti, logging total wall-clock time and the same checkpoints per epoch; if total times match within noise, the claimed backend speedup fails.","supporting_citations":[{"cited_title":"NewsRecLib: A PyTorch-Lightning Library for Neural News Recommendation","cited_arxiv_id":"2310.01146","evidence_quote":"Serves as the main baseline for all per-epoch and total-time comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the NRMS architecture whose implementation and speedup are reported."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the LSTUR architecture used in the second benchmarked speedup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The JAX backend whose JIT compilation and vectorization are credited with the speedups."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Keras 3 API that the library's models and encoders are built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multilingual subword embeddings that enable the Japanese dataset experiments."}],"review_version":1}