{"id":"93495149-3ac7-4631-84ea-46cc0b9cff33","arxiv_id":"2508.05677","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"On the NHIS mortality task, standard white-box attacks flip the final prediction of the AdaptiveFS RL questionnaire model in 33.1% (FGSM) to 64.7% (AutoAttack) of tested correctly classified cases.","lead":"An independent researcher applies six standard white-box adversarial attacks to a reinforcement-learning medical questionnaire system and reports that 33 to 65 percent of tested predictions can be flipped. The paper also introduces a 247-rule medical constraint filter intended to keep perturbed answers clinically plausible, but the evidence for that second claim is not supported by the reported data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 97.6% clinical-plausibility rate is computed per configuration (41/42), not per sample, and constraints on glucose/SBP reference features absent from the 50-feature input; sample-level medical constraint validity is therefore unestablished.","rationale":"I read the paper as attempting two claims: first, that RL-based medical questionnaire systems are vulnerable to white-box input perturbations, and second, that this remains true when perturbations are constrained to be clinically plausible by a 247-rule medical framework. The first claim is plausible and consistent with the adversarial ML literature; the paper also reports a baseline accuracy of 89% and AUC of 0.86, which is consistent with the original AdaptiveFS paper, and it uses standard attack libraries, lending some credibility to the experimental setup. However, the second, more distinctive claim is the load-bearing one for the paper's novelty, and it is not supported by the reported evidence. The validation statistics in Section 6.2 are aggregated at the level of 42 configurations, while attack success rates are computed over 1,000 samples per configuration. A configuration-level pass does not imply sample-level compliance; a configuration could pass a bounds check on a subset of samples while most of its 1,000 samples violate constraints. More seriously, the constraint framework as described cannot act on the actual model input: the age-adjusted systolic blood pressure rule and the diabetic-glucose rules reference features that are not among the 50 listed in Table 9, and Appendix D states explicitly that all perturbations and validations were applied to those 50 features. This is an internal inconsistency, not merely a disagreement with external consensus. The paper also lacks a random-perturbation baseline, so even the raw attack success rates are not interpretable as adversarial effects, and the statistical section contains contradictory observation counts (42 versus 105 configurations across Table 3). None of these issues require questioning the authors' integrity; they are checkable properties of the manuscript. Because the headline contribution rests on the claim of clinically plausible attacks, and that claim lacks sample-level validation and references out-of-input features, the reader's REJECT verdict stands. No change is needed.","tokens_in":19182,"tokens_out":6133,"duration_ms":67976,"concrete_test":"Reconstruct the 50-feature AdaptiveFS model and attack pipeline; for a representative configuration (e.g., AutoAttack at epsilon=1.0), enumerate all 1,000 perturbed samples and apply each of the 247 rules pointwise. Verify whether any rule references a feature absent from the 50 input features (e.g., glucose, systolic blood pressure); if so, determine whether that rule is evaluated on hidden variables or skipped. Report (a) the per-sample fraction of fully compliant adversarial inputs rather than the per-configuration fraction, and (b) attack success rate recomputed only on fully compliant samples. If the per-sample pass rate is materially below 97.6% or the constrained attack success rate drops sharply, the headline claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that each adversarial sample used to compute the reported attack success rates actually lies in C_medical. That is not shown. Section 6.2 reports constraint validation as configuration-level counts: 42/42 configurations generated valid examples, 97.6% (41/42) passed physiological bounds, 83.3% (35/42) preserved correlations, and 95.2% (40/42) were fully compliant. Since 42 is 6 methods times 7 epsilon values and each configuration uses 1,000 samples, these counts do not establish that any particular perturbed sample, let alone the vast majority, satisfies the constraints. The per-sample pass rate is never reported. Additionally, several constraints are defined over variables that are not in the model input. Section 3.3.1 defines age-dependent systolic blood pressure bounds, and Eq. 16 plus Appendix A define a diabetic-glucose conditional rule, but Table 9, which Appendix D identifies as the complete 50-feature set on which all adversarial perturbations and medical constraint validations were applied, contains neither blood pressure nor glucose. Thus the constraint check either operates on features the classifier never sees, making it ineffective for the actual attack, or the feature list is incomplete. Either way, the headline claim of vulnerabilities under strict medical constraints is not supported. The qualitative point that white-box perturbations can flip predictions is plausible, but the paper's distinctive contribution, clinically plausible attacks at a 97.6% success rate, collapses without sample-level, in-input-space validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript evaluates six white-box adversarial attack methods against a re-implemented AdaptiveFS RL-based medical questionnaire system for NHIS 4-year mortality prediction. The attack is formalized in Eq. (5) as maximizing the loss subject to an epsilon norm bound and a medical constraint set C_medical. The authors introduce a 247-rule medical constraint framework, report attack success rates between 33.08% (FGSM) and 64.70% (AutoAttack), and conclude that the system remains vulnerable under strict medical constraints.","tokens_in":19346,"tokens_out":8295,"duration_ms":95387,"significance":"If the central claim were properly supported, the paper would fill a genuine gap in the adversarial-robustness literature by moving from medical image classifiers to RL-driven questionnaire systems and by attempting to enforce clinical plausibility during attack generation. The explicit constrained optimization formulation and the use of standard attack libraries are strengths, and the quantitative claim of label flips on a re-implemented AdaptiveFS baseline is a plausible direction. However, the load-bearing evidence for clinical plausibility is configuration-level rather than sample-level, several constraints reference variables outside the actual feature set, and no random-perturbation control is included; at present the headline conclusion is not established.","major_comments":[{"comment":"The reported 97.6% clinical-plausibility rate is a configuration-level statistic: 41 of 42 configurations passed the physiological-bounds check, not 97.6% of individual adversarial samples. Because each configuration uses 1,000 attack samples and ASR is reported as a per-configuration aggregate (Tables 2 and 3), this count does not show that the samples whose labels were flipped actually satisfy the medical constraints. Please report per-sample pass rates for each method and epsilon, and ideally the joint rate \"attack succeeds and all constraints hold.\"","section":"Section 6.2; Abstract"},{"comment":"The constraint set cannot be evaluated on the actual attack vector as described. Eq. (11) imposes age-dependent systolic blood-pressure bounds and Eq. (16) imposes a diabetic-glucose rule, but the 50 features in Table 9, which Appendix D states are the complete set on which all perturbations and validations were applied, include neither blood pressure nor glucose. Either the constraints were checked on variables outside the model input (so they cannot constrain x+delta in Eq. (5)), or the feature set is incomplete. The revision must align the constraint vocabulary with the actual feature set and verify satisfaction on the perturbed samples.","section":"Section 3.3.1 and 3.3.3; Appendix D, Table 9"},{"comment":"No random-perturbation baseline is provided. With epsilon values up to 2.0 in the normalized [-1,1] feature space, many perturbations are large enough to move a feature across nearly its full range (Section 4.4 even gives the example of age changing from 20 to 60 at epsilon=2.0). Without a random or non-adversarial perturbation control at matched epsilon and with the same C_medical projection, the reported label-flip rates cannot be attributed to gradient-based adversarial direction. Add such a baseline and report the per-epsilon delta in ASR relative to it.","section":"Section 5.2 and 6.5; Eq. (5)"},{"comment":"Algorithm 2 is asserted to converge to a clinically plausible vector while preserving the adversarial effect, but this is not demonstrated. The projection and consistency-propagation steps in Algorithm 2, and the fallback \"minimize the distance to the original perturbed vector subject to the constraints,\" can in principle produce a vector that is no longer adversarial. The paper reports only the final ASR after the full pipeline. Please report, per configuration, the ASR before and after constraint satisfaction, or at least the per-sample joint success of \"label flip and all constraints satisfied.\"","section":"Section 3.3.4 and 6.2; Algorithm 2"},{"comment":"The observation counts in the statistical analysis are inconsistent with the stated experimental design. Section 5.3 defines 42 configurations (6 methods x 7 epsilon values), but Table 3(a) sums to 105 observations (AutoAttack N=7, C&W N=14, others N=21) and Table 3(b) sums to 105 (ART 42, Foolbox 35, Custom 28). The ANOVA in Table 4 uses F(5,99), corresponding to 105 observations. The manuscript must reconcile these numbers and state explicitly how implementation libraries enter the sample; otherwise the ANOVA and post-hoc tests are not auditable.","section":"Section 6.6; Tables 3 and 4"}],"minor_comments":[{"comment":"The abstract says six attack methods, while Section 8 says \"7 distinct major attack strategies\" and Table 2 lists six; please correct the count.","section":"Abstract vs. Section 8"},{"comment":"The AutoAttack row in Table 3(a) has a formatting error (\"Auto764.70\").","section":"Table 3"},{"comment":"Table 8 lists both PGD (L∞) and PGD (L2), but the text and Table 2 describe a single PGD row; clarify which norm is used in the reported results.","section":"Table 8"},{"comment":"Section 6.3 refers to \"our comprehensive statistical analysis (detailed in Section 5.5)\", but the paper has no Section 5.5; the statistical methods are explained only later in Section 6.6.","section":"Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The central issues are fixable in a revision: per-sample constraint validation, aligning the constraint vocabulary with the 50 features, and adding a random-perturbation baseline. As currently written, however, the abstract's \"97.6% success rate\" is a configuration-level statistic, and the claim of \"strict medical constraints\" is not supported. I recommend major revision rather than outright rejection because the underlying attack-evaluation framework is a reasonable and publishable direction if these load-bearing points are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's broad point is probably right — a white-box attacker can flip the predictions of an RL-based questionnaire model by editing answers. That is consistent with a lot of prior adversarial-ML work. But the paper's distinctive contribution, the claim that this can be done under strict medical constraints 97.6% of the time, is not backed by the evidence as written. The stress-test note is on target: constraint success is reported per configuration (41/42 passed physiological bounds), not per sample, and the constraints cite features that are not in the 50-feature input vector. So we never actually see that the specific adversarial samples used for the ASR numbers are clinically plausible.\n\nCredit where it's due: this is the first systematic evaluation of adversarial attacks on RL-based medical questionnaire systems I know of. The 247-rule medical constraint framework is a sensible attempt to address a real gap — prior work has often ignored clinical plausibility. The authors also re-implemented AdaptiveFS and ran six standard attacks across seven epsilons, and the related-work coverage is decent. As a vulnerability scan of one re-implemented model, it is a reasonable starting point.\n\nThe soft spots are real, though. Beyond the per-configuration vs per-sample issue, there is no random-perturbation baseline, so the ASR numbers are not interpretable as adversarial effect. The statistical tables contradict each other: Table 3 reports N=21 for several methods despite only 7 epsilon values, and the '42 experimental configurations' doesn't reconcile with the library counts. Appendix C describes AutoAttack as a variable ensemble of FGSM+PGD+C&W, which is not what AutoAttack is. These are fixable, but they erode confidence.\n\nWho should read this: people working on medical-AI security and robustness evaluation. The paper is not ready as is. I'd still engage it — send it to review, because the direction is right and the flaws are mostly about reporting and validation, not a fundamentally impossible question. A serious referee should demand sample-level constraint validation (per perturbed sample, in the actual input space), a random-perturbation baseline, corrected statistics, and code or a precise replication protocol.\n\nBottom line: the qualitative finding is believable, the distinctive claim is not established. Worth one round of major revision, not desk reject.","headline":"Plausible qualitative point, but the headline clinical-plausibility claim is not supported by the reported validation statistics.","tokens_in":19952,"tokens_out":2521,"would_cite":false,"duration_ms":29678,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that RL-based medical questionnaire systems are vulnerable to adversarial input perturbations even when those perturbations must pass a 247-rule medical constraint framework.","keywords":["adversarial attacks","reinforcement learning","medical questionnaire","Markov decision process","medical constraints","NHIS","mortality prediction","white-box attacks"],"falsifier":"Re-run the attack pipeline and, for each of the 1,000 test samples, check whether the final adversarial example after constraint projection satisfies all 247 constraints when applied to the actual 50 features listed in Appendix D; if the per-sample pass rate is substantially below 97.6%, or if constraints such as glucose-for-diabetics and blood-pressure bounds reference features absent from the feature table, the claim of clinically plausible attacks is not established.","tokens_in":1427,"feed_emoji":"🩺","tokens_out":2488,"duration_ms":61056,"temperature":0.7,"pith_summary":"The paper attempts to establish that reinforcement-learning-based adaptive medical questionnaire systems, which ask patients questions and then issue a diagnosis, can be manipulated by white-box adversarial attacks even when the altered answers must remain clinically plausible under a strict medical constraint framework. It reports attack success rates between 33.08% (FGSM) and 64.70% (AutoAttack) against the AdaptiveFS system predicting 4-year mortality from the NHIS dataset, with a claimed 97.6% success rate in generating clinically plausible adversarial samples. If true, the result matters because it suggests a patient's answers could be changed within medically reasonable ranges and still flip the system's risk classification, potentially leading to delayed or incorrect treatment.","feed_headline":"Attacks flip up to 65% of AI medical risk calls","feed_subtitle":"Even with 247 clinical plausibility checks, white-box tweaks to patient answers change mortality predictions.","key_machinery":"The central object is the adversarial perturbation problem posed as maximizing the loss toward a target label subject to an $\\ell_p$ bound and a medical feasibility constraint set $C_{\\text{medical}}$, where $C_{\\text{medical}}$ is defined by 247 rules across physiological bounds, feature correlations, and conditional constraints (e.g., diabetic patients must have elevated glucose). The constraint satisfaction algorithm projects perturbed features back into the feasible region; this projection is what supposedly makes the adversarial examples clinically plausible while preserving attack effectiveness.","core_discovery":"On its own terms, the paper's central discovery is that the AdaptiveFS RL questionnaire system, a DQN-plus-Guesser architecture trained on NHIS data for binary 4-year mortality prediction, can be reliably attacked by six white-box methods (FGSM, PGD, BIM, DeepFool, C&W, AutoAttack) across seven perturbation magnitudes, and that a constraint-satisfaction-based medical validation layer of 247 rules does not prevent these attacks. The paper argues that the medically plausible adversarial examples produced by this pipeline reveal a systemic vulnerability in RL-based medical questionnaire systems, not just a weakness of this particular model.","pith_inferences":["The paper reports constraint compliance at the configuration level (42/42 configurations, 97.6% of bounds checks) rather than per-sample pass rates; if per-sample validation yields lower compliance, the 33-65% success rates may not all come from genuinely clinically plausible inputs.","Several constraints described in Section 3.3 and Appendix A, such as glucose bounds for diabetics and systolic blood pressure limits, reference features that are not among the 50 NHIS features listed in Appendix D, so those particular rules may be vacuous in the actual experiments.","The same attack-and-constraint recipe could transfer to other adaptive testing domains (educational testing, mental-health screeners) where a 'plausibility' layer replaces medical constraints, so the vulnerability may be broader than clinical questionnaires.","A stronger test of the paper's claim would be to have clinicians blind-rate the perturbed samples for plausibility, since automated constraint checking may miss semantic implausibilities."],"forward_implications":["If the paper is correct, RL-based medical questionnaire systems cannot be assumed safe merely because inputs are validated against clinical rules.","Deployment of such systems should include adversarial robustness testing and input-level detection mechanisms beyond basic plausibility checks.","The 247-rule constraint framework could serve as a shared benchmark for generating and evaluating clinically plausible adversarial examples in tabular medical AI.","The monotonic increase of success rate with perturbation strength implies that even moderate, hard-to-detect changes can shift diagnoses, so regulators may need to specify allowable perturbation budgets.","The white-box results motivate follow-up work on whether black-box transfer attacks achieve similar success rates against questionnaire systems."],"supporting_citations":[{"why":"Supplies the AdaptiveFS RL questionnaire framework and NHIS preprocessing methodology that the attacks target.","marker":"[2]"},{"why":"Provides the NHIS dataset of 182,630 observations used for training and evaluation.","marker":"[1]"},{"why":"Defines the FGSM attack method that the paper adapts for medical questionnaires.","marker":"[23]"},{"why":"Defines the PGD iterative attack method used in the evaluation.","marker":"[24]"},{"why":"Defines the C&W optimization-based attack method used in the evaluation.","marker":"[41]"},{"why":"Defines AutoAttack, the ensemble method that achieves the highest success rate and also motivates the need for robust evaluation.","marker":"[21]"},{"why":"Provides the baseline comparison for medical adversarial attacks and the context for why clinical plausibility matters.","marker":"[10]"}],"fun_headline_variants":["Medical AI questionnaires fooled despite 247 clinical checks","Adversarial tweaks slip past 247 medical constraints to fool RL triage","Even clinically plausible answer edits flip 65% of AI mortality predictions","RL medical questionnaire vulnerable even with 247 constraint checks","White-box attacks on medical Q&A system succeed despite 247 rules"],"cache_read_input_tokens":22016,"weakest_assumption_plain":"The claim collapses if the 247-rule medical constraint framework does not actually validate each perturbed sample on the features that were perturbed; the paper reports configuration-level pass rates rather than per-sample pass rates.","fun_headline_variants_meta":{"raw":{"variants":["Medical AI questionnaires fooled despite 247 clinical checks","Adversarial tweaks slip past 247 medical constraints to fool RL triage","Even clinically plausible answer edits flip 65% of AI mortality predictions","RL medical questionnaire vulnerable even with 247 constraint checks","White-box attacks on medical Q&A system succeed despite 247 rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2699,"prompt_tokens":976,"completion_tokens":1723,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1635}},"tokens_in":592,"tokens_out":1723,"duration_ms":13326,"temperature":1.0,"reasoning_tokens":1635,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:31:38.866582+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the attack pipeline and, for each of the 1,000 test samples, check whether the final adversarial example after constraint projection satisfies all 247 constraints when applied to the actual 50 features listed in Appendix D; if the per-sample pass rate is substantially below 97.6%, or if constraints such as glucose-for-diabetics and blood-pressure bounds reference features absent from the feature table, the claim of clinically plausible attacks is not established.","supporting_citations":[{"cited_title":"Learning to Ask Medical Questions using Reinforcement Learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the AdaptiveFS RL questionnaire framework and NHIS preprocessing methodology that the attacks target."},{"cited_title":"National health interview survey,","cited_arxiv_id":null,"evidence_quote":"Provides the NHIS dataset of 182,630 observations used for training and evaluation."},{"cited_title":"Towards evaluating the robustness of neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines the C&W optimization-based attack method used in the evaluation."},{"cited_title":"Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,","cited_arxiv_id":null,"evidence_quote":"Defines AutoAttack, the ensemble method that achieves the highest success rate and also motivates the need for robust evaluation."},{"cited_title":"Adversarial attacks on medical machine learning,","cited_arxiv_id":null,"evidence_quote":"Provides the baseline comparison for medical adversarial attacks and the context for why clinical plausibility matters."}],"review_version":1}