{"id":"dfee1d3b-04e8-40cb-869e-d5533edf5ff9","arxiv_id":"2506.19698","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Training predictive maintenance models on downstream maintenance cost instead of prediction accuracy alone lowers average maintenance regret in turbofan experiments, by up to about 22% in the long-term setting.","lead":"The paper proposes an integrated estimate-optimize framework for predictive maintenance that fine-tunes remaining-useful-life models to directly minimize maintenance costs, and tests it on simulated turbofan engine data. It reports that this approach cuts average maintenance regret by up to roughly 22% compared with the standard estimate-then-optimize pipeline, with the largest gains when the maintenance policy ignores cost information.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantile-policy regret comparison is unfair: IEO-Q violates the α=1% reliability target, so the 21.92% reduction in Table 4 may reflect constraint violation rather than better decisions.","rationale":"The reader's weakest assumption concerned the mismatch between Corollary 1, which is proved for an exact empirical risk minimizer, and the implemented 100-step Adam fine-tuning. That is a real gap, but it only weakens the theoretical support; the empirical comparison would still be informative. The concern I identify is more load-bearing because it directly targets the paper's strongest quantitative claim: the 21.92% regret reduction with the quantile policy in the long-term experiment. If the reliability target α=0.01 is taken seriously, IEO-Q's out-of-sample failure frequency of 0.025 means it is not operating within the stated policy constraint, so comparing its regret to ETO-Q is comparing a policy that violates a safety constraint against one that satisfies it. The paper reports the failure frequency transparently, which is good, but it does not condition the regret comparison on the reliability target. A concrete test that re-calibrates both frameworks to a hard 1% failure cap would settle whether the reported advantage survives an apples-to-apples comparison. Because this concern is testable and addressable, and because the CSO result provides some independent support for the central claim, I would keep the verdict at CONDITIONAL rather than reject. The theory-algorithm gap remains a secondary issue that the authors should also address by either analyzing the actual fine-tuning procedure or clearly stating that Corollary 1 applies only to the idealized ERM solution.","tokens_in":20739,"tokens_out":8026,"duration_ms":91988,"concrete_test":"Re-run the long-term experiment (Table 4) for ETO-Q and IEO-Q under a hard out-of-sample reliability cap: for each of the 100 repeats, independently calibrate each framework's decision rule (e.g., tune α or shift the chosen z for each test instance) so that the realized failure frequency on the 20 test engines is ≤ 0.01, then compare average regret. If IEO-Q no longer beats ETO-Q, or the gap shrinks materially below the reported 21.92%, the headline quantile result is an artifact of exceeding the safety constraint. As a secondary check, report the same analysis for the CSO policy, where IEO-C does not have a failure-frequency advantage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline empirical claim for the quantile policy (21.92% regret reduction, Table 4) is not apples-to-apples. The quantile policy in Eq. (4) is defined to choose the latest maintenance window z such that the estimated failure probability before z is at most α=0.01; α is the decision-maker's stated reliability requirement. In the long-term out-of-sample experiment, IEO-Q has a realized failure frequency of 0.025 ± 0.008, about 2.5 times the target α, whereas ETO-Q has 0.001 ± 0.002 and is over-conservative. The regret metric (i) in Section 3.4 does not incorporate the α constraint, so it treats the cost of a failure purely through cc and cd, which are not the same as a hard reliability cap. IEO-Q appears to achieve much of its cost reduction by making the learned distribution sufficiently overconfident that the same quantile rule schedules maintenance later, converting early preventive cost into occasional failure cost. Whether this is a legitimate improvement depends on whether α is a hard constraint or merely a tuning parameter. If α is a reliability requirement, IEO-Q is inadmissible in the long-term experiment, and the 21.92% figure overstates the benefit. The CSO-policy result (6.56% reduction, with failure frequency also decreasing) is less exposed to this concern, but the strongest numerical claim in the paper is the IEO-Q number.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an integrated estimate-optimize (IEO) framework for predictive maintenance, in which a probabilistic remaining-useful-life model is fine-tuned on the expected maintenance cost rather than on a purely predictive loss. The authors contrast this with the estimate-then-optimize (ETO) baseline, provide a formal consistency analysis via a non-asymptotic generalization bound in Theorem 1 and Corollary 1, and introduce a stochastic perturbation gradient estimator to handle the non-differentiability of discrete maintenance decisions. Experiments on the CMAPSS turbofan dataset are run across three settings: an in-sample base case, a short-term out-of-sample case, and a long-term out-of-sample case with artificially induced stronger misspecification. The paper reports average regret reductions up to 21.92% for the quantile policy and 6.56% for the contextual stochastic optimization policy in the long-term experiment.","tokens_in":21056,"tokens_out":4415,"duration_ms":49361,"significance":"If the central claims hold, the paper makes a useful contribution to decision-focused learning and predictive maintenance by connecting statistical consistency of prognostic models to downstream maintenance costs. The strengths include a clearly formulated empirical decision-risk minimization problem, an unbiased score-function gradient estimator in Eq. (13) that avoids differentiating through a discrete argmin, a non-asymptotic bound built on Natarajan dimension, and an experimental design that varies the degree of misspecification and repeats each experiment 100 times. The main novelty claim, namely, adapting IEO to discrete maintenance decision spaces, is plausible and relevant. However, the significance is currently moderated by a gap between the proven guarantee and the implemented algorithm, and by a headline empirical comparison in which the quantile policy violates its own reliability constraint. These issues are addressable, so the contribution is worth pursuing in a major revision.","major_comments":[{"comment":"Corollary 1 is stated for omega_erm, an exact minimizer of the empirical decision risk bRL(omega), but the implemented IEO procedure runs only 100 steps of Adam from the ETO-trained parameter omega_eto. The paper gives no convergence guarantee for this fine-tuning scheme and no experiment measuring how close the 100-step iterate is to omega_erm. As written, the finite-sample guarantee therefore applies to a quantity that the deployed algorithm is not shown to produce. The authors should either prove a convergence result for the fine-tuning procedure, report the distance between the fine-tuned iterate and a candidate empirical minimizer, or explicitly state that Theorem 1 and Corollary 1 characterize exact ERM rather than the 100-step algorithm.","section":"Section 4.2, Eq. (11), and Table 1"},{"comment":"The headline claim of a 21.92% regret reduction for IEO-Q is not an apples-to-apples comparison. The quantile policy in Eq. (4) is defined by a target failure probability alpha=0.01, but IEO-Q attains a realized failure frequency of 0.025 +/- 0.008 in the long-term experiment, about 2.5 times the target, whereas ETO-Q is over-conservative at 0.001 +/- 0.002. The regret metric (i) does not penalize violation of the alpha constraint, so part of IEO-Q's cost reduction may come from shifting preventive maintenance cost into occasional failures. If alpha is a hard reliability requirement, IEO-Q is inadmissible in this experiment and the 21.92% figure overstates the framework's benefit. The authors should compare policies at matched realized failure frequencies, add a constraint-aware metric with a penalty for alpha violation, or clearly state whether alpha is a hard constraint or merely a tuning parameter.","section":"Section 5.4, Table 4, and the regret metric (i) in Section 3.4"},{"comment":"Proposition 1 asserts that the relative optimality gap delta_o is not Lipschitz continuous in the estimation error delta_e and states that this holds for all C>0. The proposition is presented as a formal statement but no proof is provided in the main text or in Appendix A, which contains only the proofs of Theorem 1 and Corollary 1. Since the motivating example is described as being formally captured by this proposition, a complete proof or a precise construction of the distributions and cost parameters realizing the failure of Lipschitz continuity should be supplied.","section":"Section 2.3 and Appendix A"},{"comment":"The non-asymptotic bound is stated in terms of the Natarajan dimension d of the integrated policy class G and assumes d <= n, but d is never computed or bounded for the Weibull-plus-argmin neural network used in Section 5. Consequently, the bound is not instantiated for the actual model class deployed in the experiments, and it is unclear whether the regime d <= n holds there. In addition, Remark 1 concedes that the i.i.d. assumption underlying Assumption 1 is likely violated by the sliding-window construction, and no dependent-data guarantee is derived. The paper should either compute or upper-bound d for the implemented architecture, or present Theorem 1 explicitly as an abstract existence result and supply additional analysis or experiments for the dependent-data setting.","section":"Theorem 1, Definition 4, and Remark 1"},{"comment":"The long-term experiment is described as introducing a 'two-phase degradation process' by capping RUL values greater than 125 at 125. This is a manual label transformation rather than a two-phase degradation model, and the resulting 'stronger misspecification' is an artifact of the capping rule. The interpretation of the comparison would be clearer if the paper stated this explicitly and discussed how the capping changes the label distribution relative to the original FD001 test set, which is not used in the out-of-sample experiments.","section":"Section 5.4"}],"minor_comments":[{"comment":"The list of framework variants says 'ETO-C, ETO-C, IEO-C, and IEO-Q'; the second entry should presumably be ETO-Q.","section":"Section 5.1"},{"comment":"The sentence 'The smaller delta_e(bP), the better decision is incurred by bP' is garbled; it should be 'the smaller delta_o(bP), the better the decision incurred by bP'.","section":"Section 2.3"},{"comment":"The proof of Proposition 2 is definitional because RL(omega)=R(omega) by construction. The authors should state explicitly that this is an identity rather than a substantive consistency result, otherwise the proposition reads as circular.","section":"Section 3.2, Proposition 2"},{"comment":"The phrase 'well-calibrated failure frequency, closely matching the preset risk margin of 1%' is accurate for the base case but should not be generalized, since the long-term experiment in Table 4 shows a failure frequency of 0.025, which is 2.5 times the target.","section":"Section 5.2, Table 2"},{"comment":"The out-of-sample experiments use held-out engines from the FD001 training set with an added cap on RUL labels, rather than the original CMAPSS test set. The paper should state this limitation clearly and discuss whether the conclusions are expected to transfer to the original test distribution.","section":"Section 5.3 and Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely of interest to the journal's audience in decision-focused learning and maintenance optimization. The main concerns are a mismatch between the theoretical guarantee (exact ERM) and the implemented algorithm (100 Adam steps), and the quantile-policy comparison in which the reported gain is achieved partly by violating the stated reliability target. Both issues are fixable with additional analysis or by restricting the claims, so major revision is appropriate rather than rejection. The authors should also be asked to supply a proof for Proposition 1 and to be transparent about the fact that the non-asymptotic bound is not instantiated for the actual model class."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a genuine and reasonably careful application of decision-focused learning to RUL-based predictive maintenance. The Weibull parameterization plus perturbation gradient is a sensible way to make end-to-end training tractable, and the empirical comparison of ETO vs IEO under controlled misspecification is a useful addition to the PdM literature. But the headline number — the 21.92% regret reduction for the quantile policy in Table 4 — is not trustworthy as a measure of better decisions. The stress-test note is right: IEO-Q's realized failure frequency is 0.025 ± 0.008 against a stated α = 0.01 target, about 2.5 times the reliability requirement, while ETO-Q is over-conservative at 0.001. The regret metric has no penalty for violating α, so part of IEO-Q's cost advantage is just shifting from early preventive cost to occasional failure cost. If α is a hard requirement, IEO-Q is inadmissible in that experiment. The paper itself notes in the base case that IEO-Q failure frequency 'closely matches' the 1% margin — that is true in-sample but not in long-term, so the calibration claim does not survive distribution shift. The CSO result (6.56%) is much less exposed, and I would trust that direction.\n\nThe theory is a mixed bag. Corollary 1 is a standard Natarajan-dimension bound, but it is proven for an exact empirical risk minimizer of the decision loss, while the implemented method is 100 steps of Adam fine-tuning starting from the ETO model. Remark 1 concedes the i.i.d. assumption is likely violated but does not address the ERM mismatch. Proposition 1 is stated without proof — probably straightforward, but it should not be left as an assertion. Proposition 2 is definitional, so it should not be oversold as a substantive consistency result.\n\nOther soft spots: the base case is in-sample; the long-term experiment caps all RUL labels above 125 at 125, which is a real modification of the data-generating process; no code or seeds are provided. These are addressable.\n\nBottom line: the paper deserves a serious referee. The authors should close the theory-algorithm gap, either by proving a bound for the actual fine-tuning procedure or by stating plainly what the theory does and does not cover, and they should re-examine whether the quantile comparison is apples-to-apples under the α constraint. Even if the quantile claim cannot be defended, the CSO result alone leaves a reasonable paper.","headline":"A legitimate application of decision-focused learning to RUL-based maintenance, but the headline quantile-policy gain may largely be a reliability-constraint violation, and the theory covers a different algorithm than the one actually run.","tokens_in":21564,"tokens_out":2175,"would_cite":true,"duration_ms":20606,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a remaining-useful-life model directly on expected maintenance cost, rather than on prediction error, reduces average maintenance regret by up to about 22% relative to the standard estimate-then-optimize approach in the paper's…","keywords":["Machine Learning","Predictive Maintenance","Estimate-then-Optimize","Integrated Estimate-Optimize","remaining useful life","decision-focused learning","model misspecification","C-MAPSS turbofan"],"falsifier":"Replicate the long-term experiment on the FD003 subset of C-MAPSS (two fault modes, same sliding-window features, Weibull head, 200-step ETO pretraining plus 100-step IEO fine-tuning, RUL capped at 125, 20 test engines, 100 repetitions): if the integrated fine-tuning does not reduce average maintenance regret below estimate-then-optimize in this independently generated misspecification regime, the claimed structural advantage of decision-loss training fails to generalize beyond the single-mode FD001 setting.","tokens_in":20503,"feed_emoji":"🔧","tokens_out":16958,"duration_ms":152967,"temperature":0.7,"pith_summary":"The paper claims that how well a remaining-useful-life (RUL) model predicts is not the same as how well it supports maintenance decisions: under model misspecification a more accurate forecast can produce costlier maintenance, because the discrete set of maintenance windows makes decision cost jump discontinuously with small prediction errors. To fix this, the authors propose an integrated estimate-optimize (IEO) framework that starts from a forecast model trained on prediction error and fine-tunes it directly on expected maintenance cost, using a stochastic perturbation gradient method to handle the non-differentiability of discrete decisions. They prove a finite-sample consistency guarantee: with enough data, minimizing the decision loss drives excess decision risk toward zero at rate $O(\\sqrt{\\log n/n})$, covering both stochastic-optimization and quantile maintenance policies. On C-MAPSS turbofan data they report that IEO reduces average maintenance regret by up to about 22% relative to estimate-then-optimize, with the gains persisting as misspecification intensifies. The practical point: prognostic models should be tuned for the decisions they feed, not for the accuracy of their predictions alone.","feed_headline":"Train on cost, not accuracy: 22% lower maintenance regret","feed_subtitle":"Fine-tuning failure-prediction models on expected maintenance cost beats accuracy-only training.","key_machinery":"The central object is the decision loss $L(m(X;\\omega),Y) = c(\\pi(m(X;\\omega);\\phi),Y)$, which scores a prognostic model by the maintenance cost of the decision its predicted distribution triggers; it is consistent with the decision objective by construction, so minimizing it is equivalent to minimizing expected cost. The model produces a Weibull-type distribution for the remaining useful life, outputting just two parameters (scale $\\lambda$ and shape $k$) that are truncated and renormalized onto a discrete support, keeping the prediction low-dimensional while retaining flexibility. Because the policy's argmin over discrete maintenance windows is not differentiable, the gradient of the decision loss with respect to the Weibull parameters is estimated by a stochastic perturbation scheme: Gaussian noise is added to the parameters and the score-function identity $\\nabla_\\theta \\tilde{L}_i = \\Sigma^{-1}\\mathbb{E}_\\eta[L(\\theta_i+\\Sigma\\eta,y_i)\\eta]$ is evaluated by Monte-Carlo sampling, with a REINFORCE-style baseline to reduce variance. The generalization argument rests on the Natarajan dimension $d$ of the integrated policy class, which enters a Sauer-Massart bound to yield the finite-sample excess-risk guarantee.","core_discovery":"The paper's central claim is that prediction accuracy and decision quality are different objectives in predictive maintenance, and that training should target the latter. The authors first show that the optimality gap of a maintenance policy is not Lipschitz continuous in the estimation error of the RUL distribution, so two estimates with identical accuracy can lead to different expected costs and a worse prediction can lead to better decisions. They then define the decision loss $L(m(X;\\omega),Y) = c(\\pi(m(X;\\omega);\\phi),Y)$, which scores a model by the maintenance cost of the decision its predicted distribution triggers, and note that this loss is consistent with the decision objective by construction. The main theoretical result is a PAC-type guarantee: an empirical minimizer of this decision loss has excess decision risk bounded by $2C_1\\sqrt{\\log(2/\\delta)/2n} + 2C_1\\sqrt{(2d\\log(en/d)+4d\\log K)/n}$, where $d$ is the Natarajan dimension of the integrated policy class and $K$ the number of maintenance windows, so the learned model converges to the best-in-class decision-oriented model as the sample size grows. Empirically, on the C-MAPSS FD001 turbofan data with a Weibull-type RUL model, IEO fine-tuning reduces average regret by 4.3% (CSO) and 27.8% (quantile) in the in-sample base case, and by 6.56% and 21.92% respectively in the long-term out-of-sample setting under strong misspecification.","pith_inferences":["The non-Lipschitz mechanism behind Proposition 1 is not specific to maintenance: any forecast that feeds a discrete choice problem (inventory orders, scheduling slots, capacity decisions) can exhibit the same divergence between prediction accuracy and decision quality, so the decision-loss fine-tuning recipe should transfer to those settings.","The sharp drop in regret variance suggests that decision-oriented fine-tuning also acts as a regularizer, pulling different random initializations toward the same decision regions; a direct test would be to measure the variance of the chosen maintenance windows, not just costs, across seeds.","Because the paper's own Remark 1 concedes that sliding-window time series violate the i.i.d. assumption behind Theorem 1, the theoretical guarantee is not literally in force for the experiments; re-deriving the bound under a mixing assumption and checking the regret reduction under block-bootstrap resampling would close that gap.","The Weibull head can only represent unimodal RUL distributions, leaving multi-modal degradation (the two-fault-mode data the paper sets aside) untested; a mixture-of-Weibull extension, which the paper names as future work, would show whether decision-loss training can compensate for that restriction."],"forward_implications":["Prognostic models should be selected and tuned on maintenance cost: in the reported experiments, the model with the lowest prediction error is not the model with the lowest maintenance regret, so accuracy-based model comparison can select the wrong model for operations.","The advantage of decision-oriented training persists as misspecification intensifies: for the stochastic-optimization policy the relative regret reduction rises from 4.3% in the base case to 6.56% in the long-term setting, and for the quantile policy worst-case regret over 100 trials drops from 51.1 to 30.8.","The quantile policy benefits most when its risk tolerance is misaligned with the cost structure: with a fixed tolerance, IEO corrects the over-conservatism that inflates regret, calibrating the observed failure frequency toward the preset risk margin.","The finite-sample guarantee means the approach is usable on the small run-to-failure datasets typical of maintenance practice, since excess decision risk vanishes at rate $O(\\sqrt{\\log n/n})$ as the number of run-to-failure samples grows.","IEO stabilizes decisions across training runs: maximum regret over 100 trials falls by 12.9% (CSO) and 34.1% (quantile) in the base case, so decision-oriented fine-tuning also reduces run-to-run variability."],"supporting_citations":[{"why":"Supplies the smart predict-then-optimize paradigm that the IEO framework adapts to discrete maintenance decisions.","marker":"(Elmachtoub and Grigas, 2022)"},{"why":"Supplies the P-consistency definition used to characterize when accuracy-oriented training can support decisions.","marker":"(Ho-Nguyen and Kılınç-Karzan, 2022)"},{"why":"Supplies the Natarajan-dimension generalization-bound technique that Theorem 1 adapts to finite discrete decision sets.","marker":"(El Balghiti et al., 2023)"},{"why":"Supplies the contextual stochastic optimization formulation behind the CSO policy, and the mixing-process result cited in Remark 1.","marker":"(Bertsimas and Kallus, 2020)"},{"why":"Supplies the perturbed-optimizer Monte-Carlo gradient estimator on which the stochastic perturbation algorithm is based.","marker":"(Berthet et al., 2020)"},{"why":"Supplies the REINFORCE baseline technique used to reduce the variance of the gradient estimator.","marker":"(Williams, 1992)"},{"why":"Supplies the sliding-window feature engineering and turbofan maintenance modeling that the experiments build on.","marker":"(Mitici et al., 2023)"},{"why":"Supplies the C-MAPSS turbofan run-to-failure dataset used in the case study.","marker":"(Saxena et al., 2008)"}],"fun_headline_variants":["Train on cost, not accuracy: up to 22% less maintenance regret","Cost-aware training cuts maintenance regret by up to 22%","Why accurate predictions can still lead to costly choices","Optimize for cost, not error: up to 22% lower maintenance regret","Decision-first ML: up to 22% less regret by training on cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's guarantee applies to a model that exactly minimizes the decision loss on the training data and presupposes that the capacity measure of the model class stays below the sample size, while the experiments use a 100-step fine-tune and never compute that capacity measure, so the proof covers an idealized training procedure rather than the one whose regret reductions are reported.","fun_headline_variants_meta":{"raw":{"variants":["Train on cost, not accuracy: up to 22% less maintenance regret","Cost-aware training cuts maintenance regret by up to 22%","Why accurate predictions can still lead to costly choices","Optimize for cost, not error: up to 22% lower maintenance regret","Decision-first ML: up to 22% less regret by training on cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000943,"raw_usage":{"total_tokens":4133,"prompt_tokens":1151,"completion_tokens":2982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":767,"completion_tokens_details":{"reasoning_tokens":2889}},"tokens_in":767,"tokens_out":2982,"duration_ms":22587,"temperature":1.0,"reasoning_tokens":2889,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:28:08.883959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replicate the long-term experiment on the FD003 subset of C-MAPSS (two fault modes, same sliding-window features, Weibull head, 200-step ETO pretraining plus 100-step IEO fine-tuning, RUL capped at 125, 20 test engines, 100 repetitions): if the integrated fine-tuning does not reduce average maintenance regret below estimate-then-optimize in this independently generated misspecification regime, the claimed structural advantage of decision-loss training fails to generalize beyond the single-mode FD001 setting.","supporting_citations":[{"cited_title":"predict, then optimize","cited_arxiv_id":null,"evidence_quote":"Supplies the smart predict-then-optimize paradigm that the IEO framework adapts to discrete maintenance decisions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the P-consistency definition used to characterize when accuracy-oriented training can support decisions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Natarajan-dimension generalization-bound technique that Theorem 1 adapts to finite discrete decision sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the contextual stochastic optimization formulation behind the CSO policy, and the mixing-process result cited in Remark 1."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the perturbed-optimizer Monte-Carlo gradient estimator on which the stochastic perturbation algorithm is based."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the REINFORCE baseline technique used to reduce the variance of the gradient estimator."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the C-MAPSS turbofan run-to-failure dataset used in the case study."}],"review_version":2}