{"id":"5e0c871c-4f86-49c9-a6d9-777acb279411","arxiv_id":"2501.14152","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A softmax policy network trained on estimated counterfactual rewards reports large estimated clinical improvements on two multimodal datasets.","lead":"The authors introduce Prescriptive Neural Networks, feedforward networks that learn which treatment to recommend by minimizing estimated outcomes from a counterfactual model. They report estimated improvements of 32% for heart valve complications and over 40% for liver trauma mortality when clinical notes are added, but the evaluation relies on the same estimated outcomes used to train the models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains (32%/40%) are computed from the same estimated counterfactual matrix Γ used to train PNNs; without ground-truth validation the absolute improvements may be artifacts of Γ bias.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: the estimated counterfactual reward matrix Γ is used both to train the PNN (Eq. 4) and to measure its improvement (Eq. 5). The paper provides no evidence—synthetic, observational, or prospective—that Γ is an unbiased estimate of true counterfactual outcomes. This makes the headline 32% and over-40% reductions, and to a lesser extent the multimodal advantage, unsupported. I agree with the reader's assessment and their conditional verdict: the architecture and pipeline are coherent, and the baseline comparisons are meaningful under the shared Γ, but the absolute real-world significance claims require external validation. I do not see a reason to move the verdict; the conditional verdict already captures the needed remedy. My proposed synthetic ground-truth experiment is a concrete and feasible check that would distinguish a genuine prescriptive gain from an artifact of counterfactual estimation bias.","tokens_in":17042,"tokens_out":4356,"duration_ms":42874,"concrete_test":"Construct a synthetic multimodal dataset with known outcome model y(x,t), known propensity π(t|x) with confounding, and known text/embedding features. Apply the paper's pipeline: estimate Γ via doubly robust estimator (Eq. 1), train PNN on Eq. 4, evaluate with Eq. 5, and compare to the true expected outcome of the same prescriptions under y(x,t). Vary outcome-model and propensity-model misspecification. If the estimated improvement overstates true improvement by more than roughly 10 percentage points, or if the policy chosen as best under Γ is not best under true y, then the headline improvements should be rephrased as conditional on Γ, and external validation (e.g., a randomized or quasi-experimental check) is required before deployment claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claims (abstract; §1.2) are that PNNs reduce TAVR complications by 32% and liver-trauma mortality by over 40% in 'estimated outcomes.' Both the training objective (Eq. 4) and the evaluation metric (Eq. 5) operate on the same doubly-robust reward matrix Γ (Eq. 1). A policy trained to minimize Eq. 4 will preferentially choose treatments whose Γ entries are underestimated, and Eq. 5 will then report exactly those gains—so if Γ is biased, the 32%/40% figures are artifacts. The paper never states or tests the unconfoundedness and positivity conditions required for Eq. 1 to recover true counterfactuals (§2.3), and §3.1 does not specify whether Γ for the test set is estimated out-of-fold or on the test data itself; §3.4 even concedes that 'test-set reward matrix entries lack natural meaning' for doubly robust estimates, yet uses them as the headline metric. The baselines in Table 4 are all scored on the same Γ, so rank comparisons are less vulnerable; the vulnerable claims are the absolute improvements and the multimodal advantage. No synthetic ground truth, factual-outcome calibration, or code/data release is provided. This is an external-validity gap, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Prescriptive Neural Networks (PNNs), a feedforward neural network trained on multimodal (tabular and text) embeddings to output outcome-optimizing treatment prescriptions. The training objective (Eq. 4) minimizes a softmax-weighted sum of entries from a counterfactual reward matrix Γ estimated by doubly robust or direct methods (Section 2.3). The authors report that PNNs reduce estimated postoperative complication rates by 32% in TAVR and estimated mortality by over 40% in liver trauma, and that on four tabular datasets PNNs outperform or match Regress & Compare, Causal Forests, and Optimal Policy Trees. They also fit Mirrored Optimal Classification Trees on PNN prescriptions to recover interpretability, and report stability and realism metrics across 5 data splits and 5 model runs.","tokens_in":17277,"tokens_out":3036,"duration_ms":29293,"significance":"If the empirical claims were reliable, the paper would make a useful contribution: it is a flexible prescriptive framework that handles multimodal inputs, spans all four treatment scenarios (binary, multiple discrete, single continuous, multiple continuous), and offers an interpretability recovery route via knowledge distillation. The stability and treatment-realism analyses are thoughtful additions. However, the central empirical evidence rests entirely on estimated counterfactual rewards, with no synthetic ground-truth validation, no observed-outcome calibration, and no formal statement or test of the unconfoundedness and positivity assumptions required by the doubly robust estimator. The 32% and 40% headline gains are therefore not established as real-world effects. The paper is internally consistent in its modeling, but its external validity claims are substantially overreaching.","major_comments":[{"comment":"The evaluation metric in Eq. (5) is computed from the same estimated reward matrix Γ that appears in the training objective Eq. (4). A policy minimizing Eq. (4) will preferentially choose treatments whose Γ entries are underestimated, and Eq. (5) will mechanically report those gains. The paper acknowledges in Section 3.4 that 'test-set reward matrix entries lack natural meaning' for doubly robust estimates, yet the abstract and Section 1.2 claim 32% and 40% improvements in estimated outcomes as the paper's headline results. This circularity means the absolute improvements in Table 1 are not credible evidence of real-world benefit. The authors should validate Γ against either synthetic ground truth with known counterfactuals or observed factual outcomes in an overlap region, and should state and test unconfoundedness and positivity assumptions for Eq. (1). Without such validation, the absolute improvement claims should be explicitly framed as 'improvements in estimated rewards under the chosen estimator,' and the abstract should be tempered accordingly.","section":"Section 3.4, Eq. (5), and Section 2.4, Eq. (4)"},{"comment":"The procedure for estimating the test-set reward matrix Γ is underspecified. It is not stated whether the test-set counterfactuals are estimated with models trained only on the 50% training split (out-of-fold) or on the full data including the test set. If the latter, the evaluation leaks information and the reported improvement numbers, including the multimodal-versus-tabular comparisons in Table 1, are optimistically biased. This concern affects all results, not only the absolute numbers, and should be clarified and corrected if needed.","section":"Section 3.1 and Section 3.5, Table 1"},{"comment":"The claim that multimodal PNNs significantly outperform tabular-only PNNs is based on reward matrices that, in the 'Tabular & Notes' rows, are themselves estimated using the multimodal data. Since the PNN is trained on the same embedding representations used by the reward estimator, the multimodal advantage may partly reflect shared representation bias rather than genuine prescriptive gains from additional modalities. The paper reports results under both tabular-only and multimodal reward estimators, which is helpful, but the central multimodal claim would be considerably strengthened by a synthetic experiment where the true counterfactual outcomes are known, or by a factual-outcome validation under an overlap assumption.","section":"Section 3.5, Table 1"}],"minor_comments":[{"comment":"The manuscript contains several typos and formatting issues, including 'replcement' in Section 3.5, 'V asiliki' in the author line, and 'T ranscatheter aortic valve replcement' in Section 3.5. A careful proofread is needed.","section":"Throughout"},{"comment":"The hyperparameter choices (number of layers, nodes, learning rate, weight decay, batch size) are described qualitatively but no actual values are reported for any dataset. A table listing the tuned hyperparameters per dataset would substantially improve reproducibility.","section":"Section 3.2"},{"comment":"The revenue improvement formula for groceries uses the notation '¯pr' twice for different quantities; this should be disambiguated (e.g., one is the mean revenue under prescribed treatments and the other is the mean revenue under actual treatments).","section":"Table 4, footnote 1"},{"comment":"Several figures referenced in the appendix (e.g., Figures 3-12) appear in the manuscript as 'Force Appendix above figures' placeholder text rather than as embedded figures; readers are directed to an external Google Drive link. The manuscript should either include the figures or clearly state that they are available as supplementary material.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the self-referential evaluation: the models are trained and evaluated on the same estimated reward matrix, and the paper even concedes that test-set doubly robust reward entries 'lack natural meaning.' The paper would be much stronger if the authors added a synthetic-data experiment with known ground truth, or at least calibrated the estimated counterfactual rewards against factual outcomes under overlap. Without such validation, the 32% and 40% claims in the abstract are likely to be overinterpreted by readers, and the novelty claim of being the 'first prescriptive method to handle multimodal data' should also be supported by a more systematic literature search and, ideally, a comparison with a multimodal baseline. The paper does not currently provide code or data, which may also be an expectation for a methods paper in this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, well-written application paper, but the headline numbers are not evidence of real clinical impact yet. The PNN is a feedforward network trained to minimize the softmax-weighted estimated reward matrix Γ, and the reported improvement is computed from the same Γ. If Γ is biased, the 32% TAVR and 40% liver-trauma reductions are artifacts. The paper never states or tests the unconfoundedness and positivity conditions needed for the doubly-robust estimator to recover true counterfactuals.\n\nWhat's genuinely new: applying prescriptive policy learning to multimodal data (tabular plus clinical notes) with pretrained embeddings is a useful extension, and I don't know of earlier work doing exactly that. The empirical package is above average for the genre: four tabular datasets covering all four treatment types (binary, discrete, single continuous, multiple continuous), comparison against Regress & Compare, Causal Forest, and Optimal Policy Tree, plus a careful stability analysis across splits. The Mirrored OCT distillation is a nice touch, and the authors are honest in Section 3.4 and the Discussion that reward-matrix evaluation has limitations. That honesty makes the missing validation more frustrating, not less.\n\nThe soft spots, in rough order:\n1. Self-referential evaluation. Eq. 4 trains on Γ; Eq. 5 measures on Γ. A policy that prefers treatments whose Γ entries are underestimated will look great on Eq. 5 regardless of real outcomes. Rank comparisons against baselines are less vulnerable, but the absolute improvements, and the multimodal advantage, rest entirely on Γ being unbiased.\n2. No external grounding. No synthetic experiments with known ground truth, no calibration against observed outcomes, no unconfoundedness test. It is also unclear whether the test-set Γ is estimated out-of-fold; Section 3.1 says counterfactual estimation is needed for the test set but doesn't specify the protocol.\n3. Reproducibility. No code or data release. Hyperparameters are listed but not their chosen values; some discretization grids are missing.\n4. Overclaim: \"first prescriptive method to handle multimodal data\" depends on a narrow reading of the literature; concatenating embeddings is not a deep methodological leap.\n\nWho should read this: people working on prescriptive analytics from observational data, especially medical applications, and anyone teaching evaluation pitfalls. It deserves a serious referee; the question is important and the paper is coherent. But a major revision with validation against known ground truth or observed outcomes is needed before the headline numbers can be trusted.","headline":"Useful but self-referential: headline gains are measured on the same estimated reward matrix used to train the policies, so the 32% and 40% numbers are not real-world evidence until validated.","tokens_in":17822,"tokens_out":3547,"would_cite":false,"duration_ms":30856,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes Prescriptive Neural Networks (PNNs), a feedforward neural network that learns an outcome-optimizing treatment policy from multimodal data—tabular features plus text embeddings—and reports estimated complication…","keywords":["prescriptive analytics","multimodal deep learning","counterfactual estimation","doubly robust estimator","neural network policy learning","knowledge distillation","optimal classification trees","healthcare decision-making"],"falsifier":"Give the same PNN pipeline a synthetic dataset whose true outcome functions are known, and compare the expected true outcome of its prescriptions with the improvement estimated from the reward matrix; alternatively, run a prospective study in which TAVR and liver trauma patients are treated according to PNN prescriptions and compare actual complication and mortality rates with historical practice. If the synthetic or prospective true outcomes show no improvement, the central claim fails.","tokens_in":16788,"feed_emoji":"🩺","tokens_out":10215,"duration_ms":82240,"temperature":0.7,"pith_summary":"The paper's central claim is that a feedforward neural network can be trained to prescribe treatments by minimizing an estimated counterfactual reward, and that this framework is the first prescriptive method able to consume multimodal data. PNNs take tabular features and embeddings from pretrained clinical language models, estimate a doubly robust reward matrix over all treatments, and then train a softmax output layer to pick, for each patient, the treatment with the best estimated outcome. On two real-world medical datasets, the authors report that multimodal PNNs reduce estimated postoperative complication rates by 32% in TAVR and estimated mortality by over 40% in liver trauma. On four tabular datasets spanning binary, discrete, single-continuous, and multiple-continuous treatments, PNNs outperform or match established prescriptive baselines. The paper also shows that these neural prescriptions can be distilled into interpretable Optimal Classification Trees with little loss, which matters for deployment where decisions must be explained.","feed_headline":"Multimodal deep learning prescribes treatments, cuts complications 32%","feed_subtitle":"It reads clinical notes and tables to pick treatments, cutting estimated complications by 32% and mortality over 40%.","key_machinery":"The load-bearing object is the Prescriptive Neural Network (PNN), a feedforward network whose output layer has one neuron per treatment and uses a softmax to turn output logits into prescription probabilities. During training it minimizes the relaxed objective $\\frac{1}{n}\\sum_{i=1}^n \\sum_{t\\in T} \\sigma_t(z(x_i))\\Gamma_{i,t}$, where $\\Gamma_{i,t}$ is the doubly robust estimated outcome of assigning treatment $t$ to observation $i$. This objective is the soft relaxation of the policy objective proposed by Amram et al. (2022), and it is what makes the prescription task differentiable and trainable by backpropagation. Multimodality enters through the input representation: embeddings extracted from Clinical Longformer text models are reduced by PCA and concatenated with normalized tabular features. Interpretability is then recovered by fitting an Optimal Classification Tree to the PNN's final prescriptions, yielding a Mirrored OCT.","core_discovery":"The discovery claimed is that the non-differentiable prescriptive objective $\\min_\\tau \\sum_i \\mathbf{1}\\{\\tau(x_i)=t\\}\\Gamma_{i,t}$ can be replaced by the softmax-relaxed objective $\\min_\\tau \\frac{1}{n}\\sum_i \\sum_t \\sigma_t(z(x_i))\\Gamma_{i,t}$, making the prescription problem trainable end-to-end with backpropagation. Here $\\Gamma_{i,t}$ is the doubly robust estimate of the outcome under treatment $t$ for observation $i$, and $\\sigma_t(z)$ is the softmax probability the network assigns to treatment $t$. With embeddings from Clinical Longformer concatenated to tabular features, the same network handles multimodal inputs. The authors claim this is the first prescriptive method for multimodal data, and that the resulting policies improve estimated outcomes by 32% and over 40% in the TAVR and liver trauma datasets while matching or beating four established baselines on tabular data.","pith_inferences":["Because the prescription problem is framed as softmax classification over treatments, the same architecture should transfer to other multimodal decision settings—dynamic pricing with text reviews, ad targeting with images or video, and personalized maintenance scheduling—without changing the model.","The use of frozen pretrained embeddings means the framework could act as a lightweight prescriptive head on top of any large language or vision model, pointing toward a general recipe for turning generative models into decision policies.","Distilling a black-box policy into an OCT also provides an audit mechanism: inspecting the tree's splits could reveal whether the learned policy depends on clinically or ethically questionable features, helping to certify the policy before deployment.","Because the paper's evaluation metric uses the same reward matrix that defines the training target, the most informative next experiment is external validation—synthetic ground-truth counterfactuals or a prospective comparison of PNN prescriptions against clinician decisions."],"forward_implications":["Incorporating clinical notes alongside tabular features improved estimated outcome gains by more than 10 percentage points on average in the TAVR and liver trauma datasets, suggesting multimodal information is what drives the largest gains.","Across diabetes, groceries, splenic injuries, and REBOA datasets, PNNs matched or beat Regress & Compare, Causal Forests, and Optimal Policy Trees on estimated outcome improvement.","Fitting Mirrored OCTs to PNN prescriptions preserves most of the estimated improvement, with an average loss of only 1.38% across tabular datasets, while returning an interpretable decision tree.","PNNs prescribe a comparable share of the available treatments and remain stable across randomized splits, with standard deviations similar to the other prescriptive methods.","Because the network's final layer outputs a probability for each treatment, users can apply thresholds or penalties to control how much the prescribed policy deviates from current practice."],"supporting_citations":[{"why":"Supplies the prescriptive objective that PNNs soften into a differentiable loss, and provides the Optimal Policy Tree baseline used in the tabular comparisons.","marker":"Amram et al. (2022)"},{"why":"Provides the doubly robust policy evaluation and learning ideas used to construct the counterfactual reward matrix from observational data.","marker":"Dudik et al. (2011)"},{"why":"Defines the Optimal Classification Trees used to distill PNN prescriptions into interpretable Mirrored OCTs.","marker":"(Bertsimas & Dunn, 2017; 2019)"},{"why":"Clinical Longformer supplies the 768-dimensional clinical text embeddings that the multimodal PNN takes as input.","marker":"Li et al. (2022)"},{"why":"Defines the Regress & Compare baseline from predictive to prescriptive analytics and is the main comparison for tabular outcome improvement.","marker":"Bertsimas & Kallus (2020)"},{"why":"Introduces causal forests, one of the state-of-the-art prescriptive baselines PNNs are compared against on tabular datasets.","marker":"Wager & Athey (2018)"}],"fun_headline_variants":["First prescriptive deep learning for multimodal data","Softmax relaxation enables end-to-end prescription learning","Multimodal PNNs cut TAVR complications by 32%","AI prescribes better treatments by reading notes and tables","Prescriptive networks match baselines, boost multimodal outcomes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the estimated counterfactual reward matrix $\\Gamma$, built from historical observational data with doubly robust estimation, accurately represents what each patient's outcome would actually have been under every treatment; if that matrix is biased, the reported 32% and 40% improvements and the favorable comparisons against baselines may not reflect real-world gains.","fun_headline_variants_meta":{"raw":{"variants":["First prescriptive deep learning for multimodal data","Softmax relaxation enables end-to-end prescription learning","Multimodal PNNs cut TAVR complications by 32%","AI prescribes better treatments by reading notes and tables","Prescriptive networks match baselines, boost multimodal outcomes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":3004,"prompt_tokens":947,"completion_tokens":2057,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":1979}},"tokens_in":563,"tokens_out":2057,"duration_ms":14972,"temperature":1.0,"reasoning_tokens":1979,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:20:39.793845+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the same PNN pipeline a synthetic dataset whose true outcome functions are known, and compare the expected true outcome of its prescriptions with the improvement estimated from the reward matrix; alternatively, run a prospective study in which TAVR and liver trauma patients are treated according to PNN prescriptions and compare actual complication and mortality rates with historical practice. If the synthetic or prospective true outcomes show no improvement, the central claim fails.","supporting_citations":[{"cited_title":"Optimal policy trees","cited_arxiv_id":null,"evidence_quote":"Supplies the prescriptive objective that PNNs soften into a differentiable loss, and provides the Optimal Policy Tree baseline used in the tabular comparisons."}],"review_version":1}