{"id":"3abe9887-41f4-4efd-8d7f-1b3093bda344","arxiv_id":"2411.16790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ProbChecklist learns predictive checklists end to end from images, time series, and text by treating learned concept probabilities as probabilistic facts in a checklist logic program.","lead":"ProbChecklist learns discrete checklists directly from images, time series, and clinical text by turning neural network outputs into probabilistic checklist items. It gives clinicians a short list of binary checks instead of a black-box score, while keeping accuracy close to deep learning baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 3 computes the checklist probability under an unstated independence assumption; the concept extractors produce correlated concepts, so the training objective may not match the actual binarized checklist.","rationale":"The reader's weakest assumption and mine coincide: the load-bearing weakness is the unstated independence of concepts in the probabilistic checklist objective. I considered the Table 1 inconsistencies and the possible feature-selection leakage, but those are reporting and reproducibility issues that could be fixed without changing the method. The independence/misspecification issue is more fundamental because it sits at the center of the proposed training objective: if Eq. (3) is not the probability of the actual binarized checklist, the 'probabilistic logic programming' framing is not doing the claimed work. The empirical results could still hold as a useful heuristic, so I do not recommend rejection; the paper should state the independence assumption explicitly and validate it empirically. Since the reader already judged the paper CONDITIONAL for essentially this reason, my read does not change the verdict.","tokens_in":30620,"tokens_out":11310,"duration_ms":244728,"concrete_test":"Train ProbChecklist on a dataset with a held-out validation split (e.g., MIMIC-III) and build a reliability diagram for Eq. (3): for each validation sample i, record p_i, the binarized concept vector c_i = 1[p_i > tau], and q_i = P(sum_j c_i[j] >= T) from Eq. (3). Bin the q_i values into deciles and compare the mean q_i in each bin with the empirical frequency of [sum_j c_i[j] >= T]; also compute the mean absolute pairwise phi-correlation of the binarized concepts. If the calibration error exceeds roughly 0.1 or the mean absolute correlation is large (say >0.2), Eq. (3) is misspecified for the actual checklist. The decisive follow-up is to retrain using a likelihood that respects dependencies (e.g., a direct Gumbel-softmax estimator of the discrete count) and check whether the test-set accuracy or the recovered ground-truth checklist on the synthetic MNIST task changes materially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4, Proposition 4.1, Equation (3) defines the training objective as P(sum_j c_i[j] >= T), computed by summing products of concept probabilities over all subsets. This expression is exact only when the binary concept indicators c_i[j] are conditionally independent given the input x_i. The paper never states this independence assumption, and it is unlikely to hold: the concept extractors in Section 4.3 are neural networks operating on the same input or modality, so their outputs—and the binarized concepts obtained by thresholding with tau in Section 4.5—will generally be correlated. With correlated concepts, the product of marginals is not the joint probability, so Eq. (3) is not P(sum >= T) for the actual checklist. This matters because every learned checklist is trained against this quantity; the reported accuracies depend on a surrogate objective whose relationship to the discrete classifier is never checked. The proof's reference to a 'binomial distribution' also conflates the Poisson-binomial tail with the true distribution of correlated concept counts. This does not make the empirical results meaningless, but it means the central probabilistic claim—that the architecture computes the probability of the checklist query—is unverified as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ProbChecklist, a framework for learning predictive checklists from high-dimensional data modalities. For each modality, a neural concept extractor outputs soft concept probabilities; these are passed to a probabilistic logic module that computes the probability that at least T of M concepts fire, yielding a differentiable training objective. At inference, the concept probabilities are thresholded to form a hard checklist. The method is evaluated on a synthetic MNIST checklist task, PhysioNet sepsis prediction (tabular and time series), MIMIC-III mortality prediction, and Medical Abstracts neoplasm detection, with comparisons to ILP/MIP checklist baselines and standard neural baselines. The paper also introduces interpretability regularization, a fairness penalty, and an extension to checklists of decision trees. The central claim is that ProbChecklist is the first approach that learns predictive checklists directly from images, time series, and text while remaining interpretable and competitive with deep classifiers.","tokens_in":30841,"tokens_out":12152,"duration_ms":118828,"significance":"If the result holds, the paper addresses a genuine gap: existing checklist-learning methods are limited to Boolean or tabular inputs, whereas ProbChecklist provides an end-to-end differentiable alternative to integer programming for high-dimensional modalities. The paper has real strengths: it includes a synthetic ground-truth checklist for validation, experiments across images, time series, and text, a fairness analysis, an interpretability study of the learned concepts, and an anonymous code link. The main caveat is that the training probability in Eq. (3) depends on an independence assumption that is not stated; the significance of the empirical results is conditional on that assumption being made explicit and on its robustness being examined. The checklist-of-trees extension is a useful flexibility demonstration, although its experiments are limited to synthetic data.","major_comments":[{"comment":"Equation (3) is exact only under an independence assumption that is never stated: the binary indicators c_i[j] must be conditionally independent given x_i, with margins p_i[j]. The text defines p_i[j] = P(c_i[j] = 1), which gives only marginal probabilities; a joint distribution is needed to compute P(sum_j c_i[j] >= T). If the independence is intended through the probabilistic-logic-programming convention that probabilistic facts are independent, this should be stated explicitly near Eq. (3) and the consequences of correlated concept extractors should be discussed; if it is not intended, Eq. (3) is not the probability of the checklist query. The proof's reference to a 'binomial distribution' is also inaccurate, since the product formula is the Poisson-binomial tail for independent but non-identically distributed Bernoulli variables. Because Eq. (3) defines the training loss, this missing assumption is load-bearing for the claim that the architecture computes the probability of the checklist.","section":"Section 4.4, Proposition 4.1, Eq. (3)"},{"comment":"The synthetic MNIST experiment is presented as a validation that ProbChecklist recovers a known checklist, but no quantitative recovery result is reported. Table 1 gives accuracy (96.8%) but not the correspondence between the 16 learned concepts (d'_k = 4 per image, T about 8.4) and the four ground-truth item rules (one per image, T = 3). The qualitative gradient analysis in Figure 4b is suggestive, but it does not establish that the learned checklist matches the intended rule set. Please report a recovery measure, such as per-concept rule match rates, the binarized checklist actually deployed, or an alignment score between learned and ground-truth concepts.","section":"Section 5.1 and Appendix E.2"},{"comment":"Training optimizes the soft probability in Eq. (3), while deployment uses hard thresholded concepts c_i[j] = I[p_i[j] > tau], with tau chosen after training and not appearing in the loss. Appendix E.8 documents the resulting gap in 'Model' versus 'Checklist' rows, but the paper does not analyze whether the soft objective is a reliable surrogate for the discrete checklist. Since the central deliverable is the discrete checklist, please either incorporate the binarization into the training objective through a differentiable relaxation, or provide a direct analysis of when the gap is small and how the tau selection procedure affects the final checklist's properties.","section":"Section 4.5 and Appendix E.8"}],"minor_comments":[{"comment":"In the PhysioNet tabular row, the entry '0.815 ± 0.185S' contains a stray 'S', and the row layout makes the values of d'_k, M, and T difficult to parse; please reformat.","section":"Table 1"},{"comment":"The symbol T is used both for the checklist threshold and for the number of trees ('a total of T trees, with tau the checklist thresholding parameter'), redefining a symbol already introduced in Eq. (1); please use a distinct symbol.","section":"Section 4.7 and Figure 7"},{"comment":"The depth notation is inconsistent: Eq. (6) says L = 2 but uses two leaf probabilities p[2,1] and p[2,2], whereas Appendix A.4 defines L layers with 2^{L-1} leaf nodes; please align the definitions.","section":"Section 4.7 and Appendix A.4"},{"comment":"The abstract and main text refer to 'clinical notes', but the Medical Abstracts TC Corpus consists of medical abstracts; please align the terminology.","section":"Abstract and Section 5.1"},{"comment":"Several claims of 'significantly improved performance' are made without significance tests, and on the PhysioNet tabular benchmark ProbChecklist's accuracy (62.579 ± 2.58) is not higher than the MIP baseline's (63.688 ± 2.437); please add statistical tests or soften the wording.","section":"Section 5.1 and Table 1"},{"comment":"The denominator '||a_j(x_i)||2,||a_l(x_i)||2' appears malformed and should presumably be the product of the two norms; please fix the typo.","section":"Appendix F.1, Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a machine learning journal, and I do not see grounds for rejection. The main technical concern is the unstated independence assumption behind Eq. (3); it is fixable by stating the PLP convention explicitly and, preferably, by adding an empirical check of how sensitive the learned checklists are to violations of that assumption. The synthetic recovery claim also needs to be quantified before the paper can fully support its central message. The novelty relative to prior multimodal checklist work by Jin et al. (2022) should be clarified in one sentence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this paper takes predictive checklist learning beyond tabular data, which is genuinely new relative to Zhang et al. and Makhija et al. The architecture—neural concept extractors feeding a differentiable Poisson-binomial tail—is simple but effective, and the synthetic MNIST benchmark with a ground-truth checklist is a good validation. I think it deserves serious peer review, with two caveats.\n\nWhat’s good: the extension to images, time series, and text is not just a wrapper; the concept extractors are trained end-to-end with the checklist objective, and the paper shows the learned concepts can be inspected (gradient heatmaps, token lists). The checklist-of-decision-trees extension is a nice bonus. The paper is also unusually honest in the appendix about the exponential memory cost and the need for manual concept interpretation.\n\nSoft spots, in rough order of importance:\n\nFirst, the independence assumption. Proposition 4.1 computes the checklist probability by multiplying concept probabilities, which is exact only if the concepts are independent given the input. That is never stated, and the concept extractors are neural networks on the same input, so their outputs will be correlated. The proof even calls the distribution 'binomial' when it is Poisson-binomial. This does not sink the method—the MNIST recovery experiment shows the training signal is useful in practice—but the paper should say the loss is a surrogate and ideally test how much correlation hurts.\n\nSecond, the headline performance claim is overbroad. Table 1 shows ProbChecklist does not beat MIP or ILP on the PhysioNet tabular task (62.6 vs 63.7 and 63.0). It wins on image and text tasks, but the abstract’s 'outperforms various explainable techniques' needs qualification.\n\nThird, the fairness story is more concerning than the main text suggests. Figure 5 reports a reduction in ΔFNR for gender, but Appendix Table 12 shows FNR for female patients went from 0.10 to 0.70 and for male patients from 0.29 to 0.85. That is not closing a gap in a meaningful sense; it is making both groups worse. The regularizer reduces disparity, but the paper should report and discuss the absolute error rates.\n\nMinor: the feature selection step in the appendix does not say whether the logistic-regression weights come from the training set only, and the anonymous code link has no commit hash. The latter is normal for double-blind; the former should be clarified.\n\nBottom line: the core method is sound and the empirical evidence, especially the synthetic MNIST recovery, supports the claim that checklist learning can be extended to high-dimensional modalities. The independence issue and fairness framing are fixable in revision. I’d send it out; a careful referee will make it better.","headline":"Extends checklist learning to images, time series, and text via a differentiable Poisson-binomial objective; the independence assumption is unstated and the fairness framing overreaches, but the core idea is sound and deserves peer review.","tokens_in":31384,"tokens_out":4683,"would_cite":true,"duration_ms":52489,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes ProbChecklist, which learns predictive checklists—'at least T of M binary conditions' rules—directly from images, time series, and text by treating the checklist as a probabilistic logic program.","keywords":["predictive checklists","probabilistic logic programming","interpretable machine learning","concept learning","clinical decision support","fairness","multimodal learning","decision trees"],"falsifier":"Take a synthetic dataset where concepts are generated with known correlation, such as two concepts that always share the same value, train ProbChecklist, and compare the predicted probability of a positive label on held-out data with the empirical fraction after binarization at τ. If Eq. (3) is at fault, the predicted probability will be misestimated whenever correlated concepts fire together.","tokens_in":30402,"feed_emoji":"📋","tokens_out":5775,"duration_ms":50323,"temperature":0.7,"pith_summary":"This paper proposes ProbChecklist, a method for learning predictive checklists—simple decision rules of the form 'at least T of M binary conditions are met'—directly from high-dimensional data such as images, clinical time series, and text. Previous checklist-learning algorithms required Boolean or tabular inputs or hand-crafted summaries; ProbChecklist instead learns soft binary concepts with neural networks and treats the checklist as a probabilistic logic program, so the whole pipeline can be trained by gradient descent. The authors show the method matches or exceeds explainable baselines and approaches deep-model accuracy on MNIST image sequences, sepsis prediction, ICU mortality, and neoplasm detection, and that fairness constraints can shrink error-rate gaps across protected groups. If right, it makes a highly interpretable clinical decision tool applicable to the data modalities actually collected in hospitals.","feed_headline":"Checklists can now be learned from raw images and time series","feed_subtitle":"ProbChecklist turns raw inputs into binary concepts with probabilistic logic, matching deep models on clinical tasks.","key_machinery":"The central object is a probabilistic logic program whose logical rule is the checklist: each learned concept is a probabilistic fact, and the query 'at least T concepts are true' is evaluated by Eq. (3), summing over all binary selection vectors. This equation carries the argument because it converts a discrete combinatorial rule into a differentiable probability: under the assumption that concepts are independent Bernoulli variables, the probability of exactly d true concepts is the sum of products ∏ pi[j]^σ(j)(1−pi[j])^(1−σ(j)), and the positive-class probability is the tail sum d≥T. The same logical-rule machinery is reused for decision trees, with branch probabilities expressed as products over node outcomes, and balanced-tree regularizers are added to keep splits informative.","core_discovery":"The central claim is that the discrete structure of a checklist can be learned end-to-end from arbitrary continuous inputs by reformulating it as a probabilistic logic program. Concept extractors map each modality to probabilities pi[j]=P(concept j true), and the checklist rule 'predict positive if at least T concepts are true' becomes a query whose probability is the sum over all subsets of concepts of size T or more of the product of concept probabilities. This query probability is used in a log-likelihood loss, so gradients flow through the concept extractors; at inference the probabilities are binarized with threshold τ and unused concepts are pruned, yielding a discrete checklist. The paper reports that on MNIST synthetic sequences, PhysioNet sepsis, MIMIC-III mortality, and medical abstracts, ProbChecklist outperforms checklist baselines such as ILP and MIP and is competitive with non-interpretable CNN, LSTM, and BERT models, and that a fairness regularizer substantially reduces ΔFPR and ΔFNR across gender and ethnicity subgroups. It also frames decision trees as logical rules and learns a checklist of decision trees as a proof-of-concept.","pith_inferences":["The independence assumption in Eq. (3) is untested; a direct extension would be to add pairwise correlation terms or a copula to the concept probabilities and check whether calibrated probability improves, since inference would remain a discrete checklist.","Because concepts are learned from data, the method can surface candidate clinical findings—such as a rise in heart rate around hours 12–17—that clinicians might not have specified in advance; this hypothesis-generation use is not developed in the paper.","A reader could test scalability by replacing the exponential 2^M enumeration with relaxed k-subset sampling; the paper mentions this as a future direction, so it is a natural next step rather than a demonstrated result.","The fairness analysis suggests a broader claim: that checklist classifiers, due to their simplicity, allow fairness gaps to be audited item-by-item; the paper does not make this claim explicitly."],"forward_implications":["If ProbChecklist generalizes as reported, predictive checklists can be trained end-to-end on raw clinical time series and images, removing the need to hand-engineer summary features or solve integer programs.","The threshold τ provides an explicit sensitivity-specificity dial after training, so the same learned concepts can be deployed for conservative screening or specific confirmation.","Fairness regularization can reduce across-group FPR and FNR gaps, making checklist-based decisions more equitable without abandoning interpretability.","Because the same probabilistic-logic machinery learns decision trees and checklists of trees, the approach extends the discrete-model family beyond a single linear checklist.","The reported performance on clinical notes suggests text-based checklists can be generated from documentation alone, easing checklist design in data-rich settings."],"supporting_citations":[{"why":"Defines the predictive checklist framework on Boolean inputs that ProbChecklist extends to arbitrary modalities, and supplies the ILP baseline.","marker":"[Zhang et al., 2021]"},{"why":"Extends checklist learning to continuous tabular data by learning thresholds, and supplies the MIP baseline ProbChecklist must beat.","marker":"[Makhija et al., 2022]"},{"why":"Introduces the neural probabilistic logic programming paradigm that ProbChecklist adapts for differentiable checklist learning.","marker":"[Manhaeve et al., 2018]"},{"why":"Provides the formal background of probabilistic logic programming that underlies the checklist-as-logical-rule formulation.","marker":"[De Raedt & Kimmig, 2015]"},{"why":"The PhysioNet sepsis dataset benchmarks ProbChecklist against checklist and deep baselines on clinical time series and tabular inputs.","marker":"[Reyna et al., 2019]"},{"why":"MIMIC-III supplies the ICU mortality time series data and the fairness evaluation across gender and ethnicity subgroups.","marker":"[Johnson et al., 2016]"},{"why":"TANGOS regularization for sparse and decorrelated concept attributions is adopted to keep learned concepts interpretable.","marker":"[Jeffares et al., 2023]"},{"why":"Clinical BERT embeddings are used as the concept extractor for neoplasm detection from medical abstracts.","marker":"[Alsentzer et al., 2019]"},{"why":"Defines the fairness metrics ΔFPR and ΔFNR that the fairness regularizer penalizes.","marker":"[Corbett-Davies & Goel, 2018]"}],"fun_headline_variants":["ProbChecklist: learning checklists from continuous data","Checklists from images, time series: probabilistic logic","Probabilistic logic learns discrete checklists from raw data","Checklists beyond Boolean: learn from images and time series"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the learned binary conditions behave independently when the model combines their probabilities into the chance that at least T are true; if the conditions are actually correlated, the checklist's training probability is misspecified.","fun_headline_variants_meta":{"raw":{"variants":["ProbChecklist: learning checklists from continuous data","Checklists from images, time series: probabilistic logic","Probabilistic logic learns discrete checklists from raw data","Checklists beyond Boolean: learn from images and time series"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001084,"raw_usage":{"total_tokens":4534,"prompt_tokens":949,"completion_tokens":3585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":3520}},"tokens_in":565,"tokens_out":3585,"duration_ms":28216,"temperature":1.0,"reasoning_tokens":3520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:23:03.044579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a synthetic dataset where concepts are generated with known correlation, such as two concepts that always share the same value, train ProbChecklist, and compare the predicted probability of a positive label on held-out data with the empirical fraction after binarization at τ. If Eq. (3) is at fault, the predicted probability will be misestimated whenever correlated concepts fire together.","supporting_citations":[{"cited_title":"Learning optimal predictive checklists","cited_arxiv_id":null,"evidence_quote":"Defines the predictive checklist framework on Boolean inputs that ProbChecklist extends to arbitrary modalities, and supplies the ILP baseline."},{"cited_title":"Learning predictive checklists from continuous medical data","cited_arxiv_id":"2211.07076","evidence_quote":"Extends checklist learning to continuous tabular data by learning thresholds, and supplies the MIP baseline ProbChecklist must beat."},{"cited_title":"Deepproblog: Neural probabilistic logic programming","cited_arxiv_id":null,"evidence_quote":"Introduces the neural probabilistic logic programming paradigm that ProbChecklist adapts for differentiable checklist learning."},{"cited_title":"Probabilistic (logic) programming concepts","cited_arxiv_id":null,"evidence_quote":"Provides the formal background of probabilistic logic programming that underlies the checklist-as-logical-rule formulation."},{"cited_title":"Early prediction of sepsis from clinical data: the physionet/computing in cardiology challenge 2019","cited_arxiv_id":null,"evidence_quote":"The PhysioNet sepsis dataset benchmarks ProbChecklist against checklist and deep baselines on clinical time series and tabular inputs."},{"cited_title":"TANGOS : Regularizing tabular neural networks through gradient orthogonalization and specialization","cited_arxiv_id":null,"evidence_quote":"TANGOS regularization for sparse and decorrelated concept attributions is adopted to keep learned concepts interpretable."},{"cited_title":"The measure and mismeasure of fairness: A critical review of fair machine learning, 2018","cited_arxiv_id":null,"evidence_quote":"Defines the fairness metrics ΔFPR and ΔFNR that the fairness regularizer penalizes."}],"review_version":1}