{"id":"b219789f-b653-4827-a088-79b7490c618d","arxiv_id":"2411.16422","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"On a validation split of NASA's CMAPSS FD001 data, a two-layer bidirectional LSTM with dropout predicts remaining useful life with RMSE 26.68, with no official test set evaluation.","lead":"This paper compares several neural network models, especially bidirectional LSTM, for predicting how many more cycles a turbofan engine will run before failure, using NASA's simulated CMAPSS dataset. The best model achieves a validation RMSE of about 26.7 cycles, but the paper only tests on a split of the training data and does not compare with published state-of-the-art results.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claim that BLSTM+Dropout is best is unsupported because all reported metrics come from the same validation split used for early stopping and model selection; the official test_FD001/RUL_FD001 sets are never evaluated.","rationale":"The paper is a straightforward application of BLSTM to the CMAPSS FD001 benchmark. It gives a clear architecture description and a standard dataset, but the empirical claim rests on validation-split metrics that were also used for early stopping, learning-rate reduction, and model selection. The reader's weakest assumption identifies exactly this issue: the validation split is treated as a proxy for the official test set without any evaluation on test_FD001.txt/RUL_FD001.txt. This is a load-bearing correctness concern because the central contribution is the empirical comparison, not the architecture itself. A concrete re-run on the official test set would settle whether the reported advantage of BLSTM+Dropout is real or an artifact of selection bias. I did not find a more fundamental internal inconsistency; the main problem is the missing held-out evaluation. Therefore the reader's REJECT verdict is appropriate, and no change to the verdict is needed.","tokens_in":8319,"tokens_out":3531,"duration_ms":33924,"concrete_test":"Retrain all five model variants described in Section V with the same preprocessing, but choose hyperparameters and early stopping only on a separate validation split (e.g., 80% of train_FD001 engines for training, 20% for validation). Then evaluate each final model once on test_FD001.txt against RUL_FD001.txt and report RMSE, MAE, and R2. If BLSTM+Dropout is not the best on the official test set, or if its RMSE differs substantially from 26.68, the central claim is an artifact of selecting on the validation split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-A describes test_FD001.txt and RUL_FD001.txt as the held-out test data, but Section V reports metrics only from train_FD001.txt, which the paper splits into training and validation sets (Section IV-B). Section IV-D uses validation loss for early stopping and learning-rate reduction, and Section IV-E describes hyperparameter tuning on this same pipeline. Therefore the headline RMSE/MAE/R2 values in Table II are selected on the same split that also chose the architecture and stopped training. This selection bias can inflate the apparent advantage of BLSTM+Dropout (26.68 RMSE, 20.56 MAE, 0.59 R2) over LSTM (27.74, 21.45, 0.55) and plain BLSTM (27.38, 20.99, 0.57). No evaluation on the official test trajectories is reported, so the central claim that BLSTM+Dropout is the most effective model is not established, and the reported numbers are not comparable to existing CMAPSS benchmark results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Bi-Directional Long Short-Term Memory (BLSTM) models for remaining useful life (RUL) prediction on the NASA CMAPSS FD001 turbofan engine dataset. The authors pre-process train_FD001.txt by dropping low-variability/high-correlation features, apply Min-Max scaling and a power transformation, then train and compare several models: linear regression, LSTM, BLSTM, BLSTM with dropout, and BLSTM with dropout plus batch normalization. The paper reports that the BLSTM+Dropout architecture performs best, with RMSE 26.68, MAE 20.56, and R2 0.59 on a validation split of the training data. The manuscript includes architecture diagrams, loss curves, residual plots, and a discussion of early stopping and learning-rate reduction.","tokens_in":8506,"tokens_out":2879,"duration_ms":27474,"significance":"If the reported results were obtained on the official held-out test set, the paper would provide a modest but useful benchmark of LSTM/BLSTM variants for a standard prognostics dataset, with a clear description of model architectures and standard error metrics. The paper's strengths include its explicit comparison of multiple baselines (linear regression, LSTM, BLSTM, BLSTM with dropout, and a deeper normalized variant) and its use of widely accepted metrics (RMSE, MAE, R2). However, the central empirical claim is currently not established because all metrics come from the same validation split used for early stopping, learning-rate reduction, and model selection, while the official test_FD001.txt and RUL_FD001.txt files are never evaluated. The contribution is therefore primarily a negative result about evaluation practice rather than a validated RUL prediction method.","major_comments":[{"comment":"The headline results (RMSE 26.68, MAE 20.56, R2 0.59 for BLSTM+Dropout) are computed on a validation split of train_FD001.txt, not on the official held-out test data. Section IV-A explicitly describes test_FD001.txt and RUL_FD001.txt as the test set, but Section V never reports any evaluation on these files. Because Section IV-D uses the same validation split for early stopping and learning-rate reduction, and Section IV-E uses it for hyperparameter selection, the numbers in Table II are in-sample relative to the model selection procedure and are not comparable to existing CMAPSS benchmark results. The paper must evaluate the final models on the official test trajectories and report those metrics.","section":"Section V / Table II"},{"comment":"The model selection is circular: the best model is chosen by lowest validation loss during training and hyperparameter tuning, and the same validation loss is later reported as evidence of predictive performance. This selection bias can inflate the apparent advantage of BLSTM+Dropout over LSTM (27.74 vs 26.68 RMSE) and BLSTM (27.38 vs 26.68 RMSE). To support the central claim, the authors need either an independent test set evaluation or a nested cross-validation procedure that separates model selection from final performance estimation.","section":"Sections IV-D and V"}],"minor_comments":[{"comment":"The table caption says 'Comparison of error metrics for 4 models' but the table lists five models (Linear Regression, LSTM, BLSTM, BLSTM+Dropout, BLSTM+Dropout+Normalization).","section":"Table II"},{"comment":"Figure 1's caption reads 'Mean of All Features in the Validation Set,' but the figure appears to be computed from the training data described in Section IV-A. Please clarify which dataset is plotted.","section":"Figure 1"},{"comment":"The feature-removal criterion is incomplete in the manuscript: after the displayed equation, the text reads 'Where ...' and then lists definitions for sigma, mu, and N_unique, but the equation itself is garbled and the threshold 'sigma < 0.005 * mu and N_unique < 5' is not integrated into a readable formula. Please rewrite this equation with proper notation and variable definitions.","section":"Section IV-B, Eq. (1)"},{"comment":"The Min-Max scaling formula is missing parentheses and is ambiguous as printed. It should be x' = (x - min(X)) / (max(X) - min(X)). Please correct the typesetting.","section":"Section IV-B, Eq. (2)"},{"comment":"Reference [12] has a typo in the URL ('hdps' instead of 'https'). Please also check all reference entries for completeness; several list only partial author names and venues.","section":"References"},{"comment":"The paper does not state the random seed used for the train-validation split, the specific split ratio, or the number of time steps used for the input sequences, making the experiments difficult to reproduce.","section":"Section IV-D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early-stage technical report rather than a polished journal submission. The main technical flaw—evaluation on the validation split rather than the official test set—is fixable by re-running the experiments on test_FD001.txt and RUL_FD001.txt, but the authors should also address the incomplete equations and inconsistent table/figure captions. If the re-evaluation does not confirm BLSTM+Dropout superiority on the official test set, the central claim would need to be substantially revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a cleanly written but routine benchmark of BLSTM variants on the CMAPSS FD001 dataset, and the headline result (BLSTM+Dropout, RMSE 26.68) is not comparable to the literature because it is computed on the same validation split that was used for early stopping, learning-rate scheduling, and model selection. The official test_FD001 and RUL_FD001 files are described but never evaluated.\n\nWhat the paper does well: the preprocessing steps are described concretely (feature removal criteria, min-max scaling, power transform), the model architectures are specified, and Figure 12 honestly shows that the best model overfits. The internal comparison across five models is a reasonable exercise for a course project or industrial memo.\n\nThe soft spots are the load-bearing ones. All metrics in Table II come from train_FD001.txt split into training and validation. Section IV-D explicitly uses validation loss for early stopping and learning-rate reduction, and Section IV-E describes tuning on that same pipeline. So the reported RMSE/MAE/R2 are in-sample relative to the model selection procedure. They cannot be compared to published CMAPSS results, most of which report on the official test set. The differences between LSTM, BLSTM, and BLSTM+Dropout are small (27.74 vs 27.38 vs 26.68 RMSE) and without error bars or multiple seeds it is unclear if the ordering is even stable. There is no code release, no seed information, and the hyperparameter search is described only qualitatively. The claim of novelty — 'numerous changes to hyperparameters' — is not a scientific contribution, and the conclusion's statement that BLSTM+Dropout is 'the most effective' is only valid for this particular split, not for the CMAPSS benchmark.\n\nI would not cite this paper, and I would not send it to peer review in its current form. The evaluation protocol flaw is fixable: if the author retrains on the full train_FD001 and reports metrics on the official test set (with the standard piecewise-linear RUL labeling or at least a clearly stated labeling), the paper could become a useful data point. As it stands, it is a competent write-up of a model comparison with an invalid benchmark comparison.","headline":"Routine BLSTM benchmark on CMAPSS FD001; headline RMSE is from the validation split used for early stopping and model selection, so the numbers are not comparable to the literature.","tokens_in":9077,"tokens_out":2582,"would_cite":false,"duration_ms":22654,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that a bi-directional LSTM with dropout layers gives the best remaining-useful-life predictions among the five models it benchmarks on the CMAPSS FD001 turbofan dataset, with RMSE 26.68, MAE 20.56, and R-squared 0.59.","keywords":["remaining useful life","turbofan engine","CMAPSS dataset","bi-directional LSTM","prognostics and health management","time-series prediction","dropout regularization"],"falsifier":"Run the same BLSTM-with-dropout model on the official test_FD001.txt sensor trajectories and compare its predictions with the RUL_FD001.txt targets; if RMSE, MAE, or R-squared differ materially from 26.68, 20.56, and 0.59, the paper's reported best-model claim is not supported by held-out data.","tokens_in":8061,"feed_emoji":"🛩️","tokens_out":11215,"duration_ms":97017,"temperature":0.7,"pith_summary":"The paper aims to show that a Bi-Directional Long Short-Term Memory (BLSTM) network with dropout layers gives the most accurate remaining useful life (RUL) predictions among the models it compares on the CMAPSS FD001 turbofan dataset. It benchmarks linear regression, LSTM, BLSTM, BLSTM with dropout, and BLSTM with dropout plus batch normalization, and reports that BLSTM with dropout reaches RMSE 26.68, MAE 20.56, and R-squared 0.59, the best of the five. The motivation is that better RUL forecasts support predictive maintenance, flight safety, and lower aircraft-on-ground costs. The reported evaluation is carried out on a validation split of the training file rather than on the official test files.","feed_headline":"BLSTM with dropout leads RUL prediction benchmark on CMAPSS","feed_subtitle":"The author reports RMSE 26.68 and R-squared 0.59 on the CMAPSS FD001 split, a step toward catching engine failure earlier.","key_machinery":"The central mechanism is the Bi-Directional Long Short-Term Memory (BLSTM) layer, a recurrent unit that processes the input sequence both forward and backward through two LSTM layers, so the hidden state at each time step carries context from past and future sensor observations. The paper pairs two BLSTM layers with dropout at 20 percent to reduce overfitting, and uses early stopping and learning-rate reduction during training. This bidirectional temporal memory is what lets the model turn multivariate sensor history into a remaining-cycle prediction.","core_discovery":"The central claim is that reading the sensor sequence in both temporal directions and applying dropout between two BLSTM layers captures engine degradation better than any of the alternative architectures tested. The winning architecture is Input, Dropout(0.2), BLSTM(128 units), Dropout(0.2), BLSTM(128 units), then a Dense output layer. After dropping 12 low-variability or highly correlated features, applying min-max scaling to [0,1], and applying a power transform, the model is trained on a linear RUL target that counts remaining cycles for each engine unit. The author reports that this model achieves RMSE 26.68, MAE 20.56, and R-squared 0.59, outperforming linear regression, LSTM, BLSTM without dropout, and the batch-normalized variant on the validation split.","pith_inferences":["Editorial inference: the model's real test is on the official test_FD001.txt trajectories against the RUL_FD001.txt labels; the paper does not run that evaluation, so the reported metrics are not yet comparable to published CMAPSS results.","Editorial inference: applying the same BLSTM-with-dropout architecture to the FD002, FD003, and FD004 datasets, which add different operating conditions and fault modes, would show whether the reported advantage generalizes.","Editorial inference: the paper assigns RUL as a linear countdown to failure; a piecewise or health-threshold target is a plausible alternative that could change the error metrics, and it is not tested here."],"forward_implications":["On the paper's validation split, BLSTM with dropout produces a lower RMSE and MAE than linear regression, LSTM, BLSTM without dropout, and BLSTM with batch normalization.","The winning combination pairs two BLSTM layers with 20 percent dropout, and the paper credits this pairing with capturing degradation trends while limiting overfitting.","A validation R-squared of 0.59 means the model explains about 59 percent of the variance in remaining cycles for the engines in the split.","The paper concludes that sequence-aware neural models such as BLSTM with dropout perform better than linear regression on this RUL task."],"supporting_citations":[{"why":"supplies the CMAPSS FD001 training, test, and RUL files that define the dataset and evaluation setup.","marker":"[13]"},{"why":"provides the RNN and LSTM module diagrams used to define the model's cell-state and gate structure.","marker":"[15]"},{"why":"gives the deep bidirectional LSTM background that justifies the two-directional recurrent layers.","marker":"[16]"},{"why":"explains the difference between bidirectional and unidirectional LSTM, the comparison at the heart of the paper.","marker":"[17]"},{"why":"offers the hybrid 1D-CNN-LSTM baseline that the paper's RUL models build on in the literature review.","marker":"[10]"},{"why":"gives an earlier deep-learning RUL model on the CMAPSS dataset that the paper positions itself against.","marker":"[5]"},{"why":"defines the MSE, RMSE, MAE, and R-squared metrics used to score all models.","marker":"[20]"}],"fun_headline_variants":["BLSTM with dropout beats benchmarks on turbofan RUL prediction","Two-layer BLSTM with dropout nails RUL on CMAPSS","Dropout-enhanced BLSTM improves turbofan RUL predictions","BLSTM with dropout achieves RMSE 26.68 on turbofan RUL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the validation split of train_FD001.txt is a fair stand-in for the official test set, so the RMSE, MAE, and R-squared measured on that split would transfer to unseen engines.","fun_headline_variants_meta":{"raw":{"variants":["BLSTM with dropout beats benchmarks on turbofan RUL prediction","Two-layer BLSTM with dropout nails RUL on CMAPSS","Dropout-enhanced BLSTM improves turbofan RUL predictions","BLSTM with dropout achieves RMSE 26.68 on turbofan RUL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000542,"raw_usage":{"total_tokens":2626,"prompt_tokens":1001,"completion_tokens":1625,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":1547}},"tokens_in":617,"tokens_out":1625,"duration_ms":11741,"temperature":1.0,"reasoning_tokens":1547,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:07:36.067529+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same BLSTM-with-dropout model on the official test_FD001.txt sensor trajectories and compare its predictions with the RUL_FD001.txt targets; if RMSE, MAE, or R-squared differ materially from 26.68, 20.56, and 0.59, the paper's reported best-model claim is not supported by held-out data.","supporting_citations":[{"cited_title":"Srivastava","cited_arxiv_id":null,"evidence_quote":"defines the MSE, RMSE, MAE, and R-squared metrics used to score all models."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the CMAPSS FD001 training, test, and RUL files that define the dataset and evaluation setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the RNN and LSTM module diagrams used to define the model's cell-state and gate structure."},{"cited_title":"Hybrid speech recognition with deep bidirectional lstm,","cited_arxiv_id":null,"evidence_quote":"gives the deep bidirectional LSTM background that justifies the two-directional recurrent layers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"explains the difference between bidirectional and unidirectional LSTM, the comparison at the heart of the paper."},{"cited_title":"Remaining useful life estima- tion of turbofan engines with deep learning using change-point detection based labeling and feature engineering,","cited_arxiv_id":null,"evidence_quote":"offers the hybrid 1D-CNN-LSTM baseline that the paper's RUL models build on in the literature review."},{"cited_title":"A deep learning model for remaining useful life prediction of aircraft turbofan engine on c-mapss dataset,","cited_arxiv_id":null,"evidence_quote":"gives an earlier deep-learning RUL model on the CMAPSS dataset that the paper positions itself against."}],"review_version":1}