{"id":"ac169df5-7f87-4163-8ed1-5061e331ec7e","arxiv_id":"2411.08821","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"CLIQUE defines local feature importance as the average change in model error when a feature is replaced by values from its quantile grid, emphasizing locally dependent effects.","lead":"CLIQUE is a new way to measure how much each input feature matters for a single prediction, by replacing the feature with several quantile values and recording how the model's error changes. It highlights local and interacting effects that SHAP and LIME often blur, and handles multi-class classification directly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 1 leaves the loss L unspecified for classification, so the reported CLIQUE results—and the multi-class/P1 claims built on them—are not reproducible. Specify L and re-check.","rationale":"The reader's weakest_assumption focused primarily on whether replacing a feature with M quantile values adequately represents the perturbation space, with the unspecified classification loss listed as a second fragility. I agree that the quantile-grid choice is empirically justified rather than derived, but I see the unspecified loss as the more load-bearing issue because it affects the well-definedness of the algorithm itself. Every quantitative claim about CLIQUE for classification is expressed through L, and L is absent from Algorithm 1 and from all classification experiments. This makes the headline results non-reproducible and creates an internal tension between P1 and P2: a 0-1 loss ignores probability shifts, while a probabilistic loss can assign nonzero importance in regions where the class is invariant. Since this is fixable by specifying a default loss and re-running the experiments, it does not change the reader's conditional verdict; it sharpens the condition that must be met before the multi-class and zero-importance claims can be accepted. My agreement is partial because the reader identified a different primary weakness, though we overlap on the loss ambiguity.","tokens_in":15337,"tokens_out":5964,"duration_ms":63791,"concrete_test":"Re-run the AND Gate (Section 3.1) and the MNIST x4y6/x3y6 analyses (Section 4.2) with M=25 and the same RF/GBM fits, using three explicit losses: (a) 0-1 misclassification error, (b) multiclass log loss, and (c) Brier score over one-hot labels. Record the fraction of observations in the irrelevant region (v2 < -1/3 for the AND gate; for MNIST, the 'low importance' subgroups identified in Figures 13-14) whose CLIQUE value exceeds a noise floor set by repeated simulations. If the near-zero-importance property and the digit-group separations change materially across losses, then Eq. 1 requires a specified default loss, and the paper's classification claims must be re-stated as conditional on that loss.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The definition of Vij in Eq. 1 is the average over M quantile replacements of [L(modified prediction, yi) - L(original prediction, yi)]. In regression, L is squared error, but for every classification experiment (AND gate, Corners, Regression Interaction, Lichen, MNIST), L is never specified, and Algorithm 1 has no loss parameter. This is not a minor omission: the paper's central claim is that CLIQUE \"can be directly applied to multi-class classification problems\" and satisfies P2, where importance should register changes in predicted classes or probabilities. If L is 0-1 misclassification loss, then probability changes that do not alter the predicted class contribute zero, so P2 is not honored. If L is multiclass log loss or Brier score, then P1 (zero importance for irrelevant variables) is not guaranteed by construction: in the v2 < -1/3 region of the AND gate, a tree ensemble can exhibit small probability fluctuations when v1 is replaced, and the averaged loss difference may be nonzero even though the class label is invariant. The figures therefore depend on an undisclosed loss choice, and the reader cannot determine whether the near-zero importances shown are a property of CLIQUE or of that choice. The multi-class MNIST results, which are the main demonstration of P2, inherit the same ambiguity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CLIQUE, a model-agnostic local variable importance measure defined as the average change in a loss function when a feature value is replaced by M quantile values from that feature's training distribution. The method uses cross-validated predictions and is designed to satisfy three properties: P1 (near-zero importance when a variable is irrelevant for an observation), P2 (direct applicability to multi-class problems, with importance assigned whenever perturbing a variable changes predicted classes or probabilities), and P3 (importance depends on model errors rather than predictions). The authors compare CLIQUE with SHAP and LIME on synthetic AND-gate, Corners, and regression-interaction data, and on the Lichen, MNIST, and Concrete datasets, reporting that CLIQUE better highlights locally dependent relationships and is more stable than a permutation-based variant (CLIP). They also provide a sensitivity analysis of the hyperparameter M and runtime comparisons.","tokens_in":15601,"tokens_out":5511,"duration_ms":182866,"significance":"If the claims hold, CLIQUE offers a useful and simple alternative to SHAP and LIME for local importance that extends naturally to multi-class problems and focuses on the loss space rather than the prediction space. The paper has clear strengths: the method is easy to describe and implement, the synthetic experiments directly target the claimed conditional behavior, the sensitivity analysis of M in Appendix A.3 is informative, and the runtime study in Appendix A.4 is carefully executed with Monte Carlo repetitions. The main weaknesses are that the loss function for classification is never specified, making the central experiments non-reproducible, and the paper provides no formal or even heuristic analysis of when the quantile-grid procedure guarantees the claimed zero/nonzero behavior. These issues are load-bearing for the central claims, so the significance is currently conditional on resolving them.","major_comments":[{"comment":"The loss function L in Eq. (1) is never specified for any classification experiment. Algorithm 1 has no loss parameter, yet Eq. (1) is applied to the AND-gate (Section 3.1), Corners (Section 3.2), Lichen (Section 4.1), and MNIST (Section 4.2) data. The choice of L is not a minor detail: if L is 0-1 misclassification loss, then probability changes that do not alter the predicted class contribute zero, so property P2 ('new predicted classes or probabilities' imply importance) is not honored; if L is multiclass log loss or Brier score, then property P1 can fail because small probability fluctuations in an invariant region can produce nonzero average loss differences. The manuscript must state which loss is used in each classification experiment and explain how P1 and P2 depend on that choice.","section":"Section 2, Eq. (1); Algorithm 1"},{"comment":"The central claim that CLIQUE yields near-zero importance for irrelevant variables is only demonstrated empirically on synthetic examples. No conditions are given under which Eq. (1) provably or approximately satisfies P1. In particular, replacing a feature with quantile values drawn from its marginal distribution can move a point far from the observed conditional distribution of the other features, and the effect of such off-manifold replacements on the loss difference is not analyzed (the paper cites Hooker et al. [12] but does not address their extrapolation concern). A formal statement or explicit assumptions—for example, that the model output or loss is invariant to the feature in the region of interest, or that the quantile replacements remain in the support of the relevant conditional distribution—are needed to support the general claim.","section":"Section 2, Eq. (1); Sections 3.1-3.3"},{"comment":"The evaluation of property P1 in the simulations is qualitative. The paper states that CLIQUE values are 'practically zero' in irrelevant regions, but it does not report numerical summaries such as the mean absolute importance in the irrelevant region versus the relevant region, or any separation measure. Because the zero-versus-nonzero distinction is the central contribution, quantitative comparisons should accompany the figures so that the reader can assess the magnitude and variability of the claimed near-zero values, rather than relying on visual inspection alone.","section":"Sections 3.1-3.3, Figures 2, 4, 5, 7"}],"minor_comments":[{"comment":"In line 10, the assignment 'W:j n×1 ← m' is unclear; it should read 'W:j ← grid[m]' or equivalent, since m is an index into the quantile grid.","section":"Algorithm 1"},{"comment":"The x-axis label 'MinTempAve > 65' is confusing because the text says the data are split on whether MinTempAve is below 65. Please align the label with the actual grouping used in the figure (e.g., 'MinTempAve > 65: FALSE/TRUE' or 'below 65').","section":"Figure 9"},{"comment":"The statement that M = 25 was chosen based on 'desired qualities of the Central Limit Theorem' is not a valid justification for a quantile grid; the Central Limit Theorem concerns averages of random variables, not deterministic quantile replacements. Please rephrase the rationale and rely on the stability analysis in Appendix A.3.","section":"Section 3, paragraph on M"},{"comment":"Please clarify whether the quantile grid of M values includes the original observed value, and how duplicate quantiles (which can occur with discrete features or ties) are handled when computing the average in Eq. (1).","section":"Section 2, discussion before Eq. (1)"},{"comment":"The demonstration of property P2 is presented by grouping CLIQUE values by class label, but P2 concerns whether perturbing a variable changes predicted classes or probabilities. Please clarify how these plots establish P2 rather than merely describing class-specific importance values.","section":"Figures 14 and 15"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a statistics/machine-learning journal, and the core idea is simple and potentially useful. The main blocker is the unspecified classification loss, which makes the experiments non-reproducible and weakens the P2 claim. I would also encourage the authors to add at least a basic theoretical discussion of when the quantile-replacement average yields zero importance, as the current empirical evidence alone is thin for the paper's central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CLIQUE is a genuinely new local importance method, and the empirical comparison against SHAP and LIME is the right kind of evidence for the claim that it captures locally dependent relationships. The basic idea—replace a feature with quantile-grid values, average the change in loss over a CV model—is simple, transparent, and different from attribution-based methods. The AND gate, Corners, and regression-interaction simulations make the case clearly: SHAP and LIME leave nonzero importance in regions where the true response is invariant, while CLIQUE stays near zero. The stability advantage over permutation-based CLIP is also real and well illustrated.\n\nThe main problem is reproducibility of the classification experiments. Equation 1 defines V_ij with an unspecified loss L. For regression we know it's squared error, but for every classification experiment (AND, Corners, Lichen, MNIST) the paper never says what L was used. Algorithm 1 has no loss parameter either. That is not a nitpick: it changes what the numbers mean. If L is 0-1 error, then probability changes that don't flip the class contribute zero, so property P2 as stated—'new predicted classes or probabilities'—is not honored. If L is log loss or Brier, small probability fluctuations in invariant regions can produce small positive importances, so the 'near zero' results are load-bearing on an unstated choice. The reader simply cannot tell from the text whether the reported zeros are a feature of CLIQUE or an artifact of that choice. The fix is straightforward: specify L for each experiment and put a loss parameter in the algorithm pseudocode.\n\nOther issues are minor by comparison. The property statements P1–P3 are informal; there is no analysis of when CLIQUE will actually recover conditional structure. The real-data subgroup comparisons (MinTempAve > 65, x3y6 > 7.5) look post-hoc and would benefit from pre-specified thresholds or a sensitivity check. The paper also overstates the multi-class limitation of SHAP and LIME—they can be applied per class—though CLIQUE's single importance per feature is a legitimate design choice. No formal theorem, but that is fine; the synthetic evidence is the main support.\n\nWho is this for? Researchers and practitioners who want a local importance measure that focuses on prediction error and is aware of conditional dependence. After the loss issue is fixed and the empirical claims are re-verified, it would be a useful method to have in the toolbox.\n\nRecommendation: send it to peer review. The missing loss is a major, but addressable, revision—not a fatal flaw. A good referee will push for the specification, repeated simulations with error bars, and clearer handling of the property claims.","headline":"Genuinely new local importance method with a real conditioning advantage, but the classification experiments never specify the loss L in Eq. 1, which blocks verification of the headline claims until fixed.","tokens_in":16094,"tokens_out":3569,"would_cite":false,"duration_ms":31795,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIQUE assigns each feature of each observation a local importance equal to the average change in model loss when the feature is replaced by quantile values from its training distribution, and it stays near zero for variables that are…","keywords":["local variable importance","model-agnostic interpretation","quantile grid","conditional dependence","SHAP","LIME","multi-class classification","permutation importance"],"falsifier":"Take a regression where the true effect of $x_j$ on $y$ is exactly zero for all rows, train a flexible model, and run CLIQUE: if the median absolute CLIQUE value across rows is not near zero compared to the model's loss scale, the claim that CLIQUE produces near-zero importance for irrelevant variables fails. A second check is to compare CLIQUE with the exhaustive replacement of every observed value of the feature; large discrepancies would show the quantile grid itself is the source of the signal.","tokens_in":15176,"feed_emoji":"🎯","tokens_out":5847,"duration_ms":49204,"temperature":0.7,"pith_summary":"CLIQUE gives every row of a dataset a number per feature: the average change in model loss when that feature is replaced by M quantile values from its own training distribution. The paper claims this score is near zero exactly where a variable has no local effect on the outcome, even when SHAP and LIME still report nonzero importance, and that it exposes interactions such as a feature mattering only when another feature is above a threshold. Because the score is built on loss rather than predicted probabilities, it transfers to multi-class classification without per-class summaries. If true, CLIQUE is a drop-in local explanation tool that answers 'does changing this variable hurt the model here?' rather than 'which direction does it push the prediction?'","feed_headline":"Local feature importance goes quiet when the variable doesn't matter","feed_subtitle":"New CLIQUE score averages loss changes over quantile replacements, exposing interactions SHAP and LIME miss.","key_machinery":"The defining object is Eq. (1), $$V_{ij} = \\frac{1}{M}\\sum_{m=1}^{M} \\left[ L\\big(\\hat f(\\tilde x_i(j,m)), y_i\\big) - L\\big(\\hat f(x_i), y_i\\big)\\right],$$ where $\\hat f$ is a cross-validated model that did not train on row $i$, $\\tilde x_i(j,m)$ is row $i$ with feature $j$ replaced by the $m$-th quantile value of that feature, and $L$ is the loss. The recommendation to replace a value with all other possible values is approximated by the quantile grid, which removes the need for many random permutations and stabilizes the estimate. The conditional character comes from holding all other features at the observed row while sweeping the target feature: a feature is locally important only if perturbing it changes the model's error on that row. CLIP, the permutation variant, is the same formula with random draws instead of quantiles and serves as the comparison that isolates the grid's contribution.","core_discovery":"On its own terms, the paper's central discovery is that local importance can be computed as a conditional error differential: for observation $i$ and feature $j$, replace $x_{ij}$ with each of $M$ quantile values of that feature and average the resulting change in loss. In experiments on an AND-gate, a corners-shaped decision boundary, and a regression with a switching interaction, these CLIQUE values are essentially zero in the region where the feature is known to be irrelevant and strictly positive elsewhere, whereas LIME tracks only marginal trends and SHAP assigns many nonzero values in the null region. The same mechanism gives interpretable per-class importance on MNIST pixels and splits the lichen data by an interacting covariate, and CLIQUE matches SHAP in speed while scaling linearly in sample size. The punchline is therefore that error-based quantile replacement, not prediction-based attribution, is what lets local importance reflect local dependence.","pith_inferences":["Beyond the paper, CLIQUE's per-row error differentials can be averaged into a global importance measure that inherits the conditional behavior, something the paper leaves for future work.","Beyond the paper, restricting the quantile grid to a neighborhood of the observed value would turn CLIQUE into a genuinely local-in-feature-space explanation, since the full-range grid mixes far-away values with nearby ones.","Beyond the paper, reversing the sign convention could support counterfactual queries: finding which feature changes increase loss the most points to the variables whose perturbation would most alter the prediction.","Beyond the paper, because CLIQUE depends on cross-validated predictions, its reliability inherits the stability of the base model and the CV scheme; comparing importances across folds would test how much of the signal is model noise."],"forward_implications":["CLIQUE gives practitioners a per-observation ranking of features by error impact, so a feature can be reported as unimportant for a specific prediction even if it is globally important.","The multi-class property means a single importance matrix can be produced for a K-class problem without building one explanation per class.","Because only loss values are required, CLIQUE applies to any model with a loss, including regression, binary classification, and multi-class classification.","The quantile-grid estimator is more stable than permutation-based local importance at the same M, so users can use fewer perturbations to obtain a given level of variance.","CLIQUE's linear scaling in sample size and features positions it as a faster and more local alternative to LIME for large tabular data."],"supporting_citations":[{"why":"Supplies the all-values replacement recommendation that CLIQUE approximates with a quantile grid.","marker":"[19]"},{"why":"Defines SHAP, the prediction-based attribution baseline that CLIQUE is compared against.","marker":"[17]"},{"why":"Defines LIME, the local surrogate baseline that the paper shows captures mostly marginal information.","marker":"[24]"},{"why":"Introduces permutation importance for random forests, which CLIP localizes and CLIQUE replaces with a quantile grid.","marker":"[4]"},{"why":"Introduces ICE, the grid-replacement local visualization that CLIQUE turns into a scalar importance.","marker":"[9]"},{"why":"Generalizes permutation importance to a model-agnostic framework that CLIQUE converts to per-observation errors.","marker":"[8]"},{"why":"Provides the TreeSHAP complexity result used to compare CLIQUE's scaling with SHAP.","marker":"[18]"}],"fun_headline_variants":["CLIQUE: local importance via quantile-based error changes","Zero importance when features don't matter: CLIQUE","Quantile replacement exposes true local feature effects","New error-based importance zeros out irrelevant features","Local interactions in loss space: CLIQUE beyond LIME and SHAP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on the assumption that replacing a feature with M quantile values from its training distribution is a faithful surrogate for every perturbation that could affect a given observation, so that a zero average loss change really means the feature is locally irrelevant.","fun_headline_variants_meta":{"raw":{"variants":["CLIQUE: local importance via quantile-based error changes","Zero importance when features don't matter: CLIQUE","Quantile replacement exposes true local feature effects","New error-based importance zeros out irrelevant features","Local interactions in loss space: CLIQUE beyond LIME and SHAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000873,"raw_usage":{"total_tokens":3735,"prompt_tokens":858,"completion_tokens":2877,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":2800}},"tokens_in":474,"tokens_out":2877,"duration_ms":20152,"temperature":1.0,"reasoning_tokens":2800,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:17:39.326500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a regression where the true effect of $x_j$ on $y$ is exactly zero for all rows, train a flexible model, and run CLIQUE: if the median absolute CLIQUE value across rows is not near zero compared to the model's loss scale, the claim that CLIQUE produces near-zero importance for irrelevant variables fails. A second check is to compare CLIQUE with the exhaustive replacement of every observed value of the feature; large discrepancies would show the quantile grid itself is the source of the signal.","supporting_citations":[{"cited_title":"Interpretable machine learn- ing","cited_arxiv_id":null,"evidence_quote":"Supplies the all-values replacement recommendation that CLIQUE approximates with a quantile grid."},{"cited_title":"”why should i trust you?” explaining the predictions of any classifier","cited_arxiv_id":null,"evidence_quote":"Defines LIME, the local surrogate baseline that the paper shows captures mostly marginal information."},{"cited_title":"Peeking inside the black box: Visualizing statistical learning with plots of indi- vidual conditional expectation","cited_arxiv_id":null,"evidence_quote":"Introduces ICE, the grid-replacement local visualization that CLIQUE turns into a scalar importance."},{"cited_title":"All models are wrong, but many are use- ful: Learning a variable’s importance by studying an entire class of prediction models simultane- ously","cited_arxiv_id":null,"evidence_quote":"Generalizes permutation importance to a model-agnostic framework that CLIQUE converts to per-observation errors."}],"review_version":1}