{"id":"33c12769-b446-4f67-9fa5-f04de6dcdf20","arxiv_id":"2412.16252","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"iKF is an iterative random-forest variant that ranks variables and interaction paths by permutation importance, and the authors report it recovers known interactions better than its main baseline, iRF, in simulations and on Drosophila enhancer data.","lead":"This paper introduces iKF, a method that repeatedly pins one variable at the root of random forests to find pairs and higher-order interactions among variables, and labels each interaction as accompanied, synergistic, or hierarchical. A generalist might read it to see whether tree-based interpretability tools can actually rediscover known mechanisms, here tested on simulated functions and a Drosophila gene-regulation dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The post-hoc, any-model claim is undefined and untested: Algorithm 1 never specifies which model supplies the predictions in Eq. (2), and the experiments appear to use the King's forests themselves, which would reduce iKF to a self-contained RF variant rather than a model-agnostic explainer.","rationale":"I focused on the stated weakest assumption because it is the most load-bearing: the paper's novelty and title rest on being post-hoc, yet the algorithm is underspecified with respect to the model, and no experiment exercises the post-hoc setting. This is not a manufactured concern; it is visible in the pseudocode (Algorithm 1, lines 1-3) and in the absence of any model in the experimental protocol. If the authors intend the King's forests as the model, then iKF is essentially an interaction-detection variant of random forests and the comparison to iRF is less meaningful; if they intend an external model, then Algorithm 1 is missing a required input and the experiments are missing. I also note the simulation equation mismatch between Section 4.1 and Appendix A, which further undermines reproducibility, but I treat it as secondary because the post-hoc issue alone is sufficient to support the current REJECT verdict. I credit the Drosophila rediscovery and the simulation recovery rates as positive evidence that the underlying forest-based procedure can surface known interactions; this is why a revision that adds a true post-hoc experiment and reconciles the simulation definitions could change the verdict. The suggested test directly settles whether the post-hoc mechanism works on a separately trained model, which is the exact condition needed for the central claim to hold.","tokens_in":22754,"tokens_out":7393,"duration_ms":64131,"concrete_test":"Use simulation (b2): y = 2*x1*sign(1+x3)*sin(x5) + e with n=200, p=500, and a held-out test set. Train a separate predictive model, e.g., XGBoost or an MLP, on the training split. Run the iKF procedure with PVIM computed from this pre-trained model's predictions on the test set: for each candidate King forest, permute the King variable and compute Eq. (2) using the pre-trained model's y-hat. Update variable weights exactly as in Algorithm 1. Record whether the top depth-3 paths include (x1,x3,x5) and compute the overall recovery rate over 100 replications. If the active third-order interaction is not recovered at a rate comparable to Table 1, the any-model claim fails; if it is recovered, the paper still needs to report this experiment to substantiate the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central contribution claimed in the abstract and Section 1 is a post-hoc tool that can be applied to any model without retraining. This requires a precisely specified external predictor: Eq. (2) defines PVIM using predictions of 'any given model', and the whole weighting scheme in Algorithm 1 depends on PVIM. However, Algorithm 1's input is only forest size, maximum depth, and a King; it never accepts or names a model, and the experimental section never states which model produced the y-hat values in Eq. (2). The only coherent reading of the text is that the King's forests themselves, grown on the same data, serve as the predictor. Under that reading, iKF is a new way to screen and detect interactions from raw data, not a method for interpreting a separately trained black-box model. The claimed advantage over iRF and the promise for large-scale pre-trained models are therefore unsupported. A secondary but related flaw is the mismatch between the simulation formulas in Section 4.1 and the appendix (e.g., (a2) and (b2) differ in functional form), so the empirical results in Table 1 cannot be tied to a uniquely specified ground truth. The Drosophila rediscovery is genuine positive evidence, but it too was run on data with iKF's own forests, so it does not validate the post-hoc claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Iterative Kings' Forests (iKF), a method that, given a designated 'King' variable, builds forests with that variable fixed at the root, uses King's permutation variable importance to reweight variables, and outputs ranked depth-d path shortlists together with three interaction types: Accompanied, Synergistic, and Hierarchical. The authors claim that iKF is a post-hoc interpretability tool applicable to any model without retraining, and they evaluate it on simulated functions against DC-SIS and iRF, reporting lower minimum recovery sizes and higher interaction recovery rates. They also apply iKF to Drosophila enhancer data and claim rediscovery of biologically verified transcription-factor interactions involving Zld.","tokens_in":23056,"tokens_out":8145,"duration_ms":69110,"significance":"If the post-hoc claim were supported, iKF would be a convenient tool for extracting multi-order interaction candidates from arbitrary trained models, with clear value for genomics and other scientific domains. The paper has genuine strengths: it compares against two relevant baselines across numerous simulation settings, reports quantile-based recovery sizes and recovery rates, and grounds the Drosophila findings in specific biological literature. However, the central claim is not supported by the algorithm as written or by the experiments, and the simulation protocol is internally inconsistent, so the significance of the contribution is not currently established.","major_comments":[{"comment":"Equation (2) defines PVIM using predictions ŷ_i of 'any given model', but Algorithm 1 never accepts or names such a model, and no experiment states which model produced the predictions used in Eq. (2). The only coherent reading is that the King's forests themselves predict the response, which makes iKF a standalone random-forest-based screening method rather than a post-hoc explainer for an arbitrary trained model. The abstract's and Section 1's central claim that iKF 'can be applied to any model ... without retraining' is therefore untested, and the per-tree King's PVIM used in Eq. (3) is not even defined when the predictor is a separate global model. The paper needs either an explicit model input with a model-agnostic PVIM definition and experiments on a separately trained model, or a reframing of the contribution; as written, the advertised post-hoc property is unsupported.","section":"§3.2, Eq. (2), Algorithm 1"},{"comment":"The simulation ground-truth functions are not stated consistently. Section 4.1 defines (a2) as y = 4 x1 sin(x3) − 4 x5 cos(x7) and (b2) as y = 2 x1 sign(1 + x3) sin(x5), while Appendix A defines (a2) as y = 2s x1 sin(x3) + 2s x5 cos(x7 + π/2) and (b2) as y = s x1 log(5|1+x3|) sin(x5). The appendix also has five settings per part while Section 4.1 has three, and the scaling parameter s is never defined. Consequently the ORR and IRR numbers in Table 1 cannot be tied to a uniquely specified simulation, and the experiments are not reproducible as written.","section":"§4.1 vs Appendix A"},{"comment":"The selection of the first King x(1) is an input to Algorithm 2, but the experimental protocol never reports how it was chosen in the simulations or in the Drosophila study. Section 3.2 allows a random choice when no prior knowledge is available, but the results in Figure 2 and Tables 1, 9–15 contain no seed or initialization details. Since every subsequent King and weight update depends on x(1), this is a load-bearing underspecification of the method.","section":"§4.1, Algorithm 2"},{"comment":"The criterion for interaction order and type is a 'sharp increase' in King's PVIM across depths, but no threshold, null distribution, or replication-based uncertainty is given. In Table 2, x7's PVIM rises from −0.21 to 1.79 and is treated as evidence of an interaction, while x10's rise from −0.19 to 0.33 is dismissed; in Table 4, x1's change from 1.11 to 1.33 is described as 'keeping increasing'. In Section 5.3, conclusions about Synergistic versus Hierarchical interactions separate PVIM values around 0.02 from values near 1e−16 with no error bars or formal comparison. The interaction-type labels therefore rest on an unstated decision rule rather than a defined statistical criterion.","section":"§3.3, Tables 2, 4, 7, 8"}],"minor_comments":[{"comment":"The text says 'a smaller S indicates greater recovery power'; the definition just introduced calls this quantity MRS, and S is later used for the survived variable set, so the notation should be corrected.","section":"§4.2"},{"comment":"The text states that PVIMs of 'variables 1 and 5' increase sharply and then says 'This indicates that 3 and 5 are involved'; the latter should refer to variables 1 and 5.","section":"§4.3, Case 2"},{"comment":"The hyperparameter Nc is written as 'Nc =⌊n/ 2 log(n)⌋', which is ambiguous between n/(2 log n) and (n/2) log n; the intended formula should be disambiguated.","section":"§4.1"},{"comment":"No code or data release is mentioned; given the many simulation tables and the need to reconstruct the exact protocol, a reproducibility statement or code link would be needed in any revision.","section":"General"}],"recommendation":"reject","confidential_remarks":"The Drosophila literature-based validation is a genuine strength, and the broad simulation comparison shows effort. However, the core claim of model-agnostic post-hoc interpretability is not operationalized in Algorithm 1 or tested in any experiment, and the simulation protocol is internally inconsistent. This is a load-bearing problem rather than a local presentation issue. If the authors later submit a version that explicitly reframes iKF as a standalone random-forest interaction screening method, fixes the simulation definitions, and reports the first-King selection, the contribution could be reassessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine extension of iRF, not a paradigm shift. The King-at-root construction, the PVIM-based reweighting, and the three interaction types are new relative to the cited iRF/DC-SIS baselines, and the simulations plus the Drosophila rediscovery show the method finds things iRF misses. But the abstract's central claim that iKF is a post-hoc tool for any model is never actually tested. Algorithm 1 takes no model argument, and the experiments appear to use iKF's own forests as the predictor in Eq. (2). So what's delivered is a data-level interaction screener, not a model-agnostic interpreter.\n\nThe good parts are real. Forcing a King at the root and weighting variables by King's PVIM is a sensible way to pool evidence across trees, and the three-type taxonomy (accompanied, synergistic, hierarchical) maps naturally onto biological ideas like dominant versus nested transcription factors. The Drosophila case study rediscovers several verified Zld interactions and a third-order (Zld, Kni, Tll) that iRF missed—that is independent biological validation, even if it doesn't support the post-hoc framing.\n\nThe soft spots are serious but not all fatal. The simulation formulas in Section 4.1 conflict with Appendix A (e.g., (a2) and (b2) differ in functional form), so the recovery rates in Table 1 can't be tied to a uniquely specified ground truth. The first King selection in experiments is never described. More importantly, the paper treats a sharp King's PVIM jump from depth d to d+1 as evidence of an order-(d+1) interaction, with no significance threshold or error bars; the Drosophila values in Tables 7-8 are near machine precision (1e-16), so it's unclear what \"large\" means. No code or data are provided.\n\nMy judgment: the kernel is worth a serious referee, but the current submission overclaims. I'd send it to review and ask for major revision—re-specify the predictor (or honestly reframe iKF as a data-level interaction discovery method), align the simulation equations, add the first-King selection rule, and ship code. Under that revision the contribution could be solid. As is, it's a promising but underspecified tool.","headline":"A real extension of iRF whose headline 'post-hoc for any model' claim is untested; the King-root idea and three-type taxonomy are worth debating, but the paper needs major revision, not rejection.","tokens_in":23617,"tokens_out":1960,"would_cite":false,"duration_ms":18209,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Iterative Kings' Forests (iKF) is a post-hoc tree method that produces ranked, typed multi-order interaction candidates from a trained model without retraining.","keywords":["Iterative Kings' Forests","post-hoc interpretability","variable interactions","variable importance","random forests","Drosophila enhancers","feature screening","scientific discovery"],"falsifier":"Generate data from a function with only pairwise interactions, $y = x_1 x_3 + x_5 x_7 + \\text{noise}$, run iKF at maximum depth 3 over 100 replicates, and count how often a depth-3 path containing $x_1$, $x_3$, and an irrelevant variable appears in the top-ranked lists; if false triples are common, the depth-jump rule does not establish interaction order.","tokens_in":22540,"feed_emoji":"🌳","tokens_out":10228,"duration_ms":77212,"temperature":0.7,"pith_summary":"The paper proposes Iterative Kings' Forests (iKF), a post-hoc interpretability method that discovers multi-order interactions among variables. iKF repeatedly names one variable the \"King,\" builds a forest in which every tree is rooted at that King, and uses a permutation-based importance measure, King's PVIM, to reweight variables and rank depth-specific tree paths as interaction candidates. On simulated functions with known pairwise and third-order ground truth, iKF recovers the active interactions more often than iterative random forests and ranks the active variables ahead of distance-correlation screening. On Drosophila embryo enhancer data, iKF rediscovers 14 of the 16 transcription-factor interactions previously verified in the literature and adds three further Zld-centred pairwise interactions and one third-order interaction that also have literature support. The authors claim this makes iKF usable on any trained model, without retraining, to generate ranked shortlists of interaction hypotheses and to label each as Accompanied, Synergistic, or Hierarchical.","feed_headline":"Root each tree in one 'King' variable to find 3-way interactions","feed_subtitle":"A post-hoc tool rediscovers verified gene-regulation links in Drosophila and beats standard baselines on simulations.","key_machinery":"The load-bearing object is the King's Forest: a forest of trees with the chosen King variable fixed at every root. King's PVIM, defined as the change in prediction error when the King is permuted, is used both to select informative trees and to update variable weights, so the King's interaction partners accumulate weight and migrate toward the root. From the resulting forests, iKF extracts all depth-d root-to-leaf paths and ranks them by two metrics: summed King's PVIM and path reproduction count. The rule for assigning interaction type is that a substantial jump in King's PVIM from depth d to d+1 signals an order-(d+1) interaction, path-direction symmetry distinguishes Synergistic from Hierarchical interactions, and the presence of a marginal effect marks an Accompanied interaction. These ranked paths and type labels are the output a scientist would use to generate mechanistic hypotheses.","core_discovery":"The central claim is that forcing a chosen \"King\" variable to sit at the root of every tree converts a random forest into a targeted interaction detector: the variables that share a path with the King and are rewarded by King's PVIM are exactly the variables that interact with it, and the depth at which King's PVIM jumps reveals the interaction order. In simulations, iKF outperforms iRF in overall interaction recovery, for example 0.60 versus 0.09 overall recovery rate in case (a1) and 0.68 and 0.48 recovery in the third-order settings (b1) and (b2) where iRF recovers none. On the Drosophila data, iKF identifies Zld as a dominant transcription factor whose interactions with gap-gene and anteroposterior-patterning TFs are hierarchical, while the Zld-Twi interaction is synergistic, matching the documented biology that Zld licenses downstream genes. The paper therefore presents iKF as a general post-hoc scientific-discovery tool that returns ranked, typed candidate interactions rather than a single importance score.","pith_inferences":["The paper's own caveat is that iKF cannot identify all interactions, and its (b3) simulation with p=500, where the all-active screening rate at model size d2 is only 0.08, is a concrete reminder that the output should be read as ranked hypotheses rather than a complete census.","A sharper version of iKF would replace the qualitative \"sharp increase\" rule for interaction order with a significance threshold or bootstrap interval over replications, which the 100-replication simulation protocol makes straightforward to build.","The input to Algorithm 1 never names the predictor model, and the experiments appear to use the King's forests themselves; applying iKF to a fixed black-box model's predictions is the direct test of the strongest \"any model\" reading.","Path-direction asymmetry is a cheap, interpretable signature of hierarchical interactions that could be benchmarked directly on synthetic functions whose true generative order is known."],"forward_implications":["A researcher can run iKF on a trained model and receive ranked shortlists of candidate variable interactions at several orders, with no retraining step.","The method attaches a type to each candidate interaction, so a biologist can immediately separate synergistic pairs from nested hierarchical regulation.","In simulations, iKF finds third-order interactions that the iRF baseline never recovers, suggesting it extends interaction discovery beyond pairwise effects.","On Drosophila enhancer data, iKF recovers known Zld-centred regulatory interactions and identifies Zld as the dominant factor, matching the experimentally documented hierarchy.","Because Eq. (2) is defined on predictions of any given model, iKF could provide the same interaction discovery for neural networks and other black boxes if the post-hoc claim holds."],"supporting_citations":[{"why":"Defines random forests and the permutation variable importance measure that Eq. (2) uses as the King's PVIM.","marker":"Breiman, 2001"},{"why":"Supplies the iterative random forests baseline that iKF extends and compares against, and the Drosophila embryo dataset used in the biological case study.","marker":"Basu et al., 2018"},{"why":"Provides the DC-SIS feature-screening baseline that iKF is compared with on minimum recovery size.","marker":"Li et al., 2012"},{"why":"Documents Zelda binding targets used to verify the rediscovered Zld-centred pairwise transcription-factor interactions.","marker":"Harrison et al., 2011"},{"why":"Documents Zld's temporal coordination of gap-gene expression, used to verify hierarchical interactions and the third-order Zld-Kni-Tll interaction.","marker":"Nien et al., 2011"},{"why":"Supplies the biological background that enhancers act hierarchically, motivating the Hierarchical Interaction type.","marker":"Levine, 2010"}],"fun_headline_variants":["Crown one 'King' per tree to expose hidden gene interactions","Turn random forests into targeted interaction detectors","Post-hoc tool ranks and types multi-order interactions","From Drosophila data, a method that distinguishes interaction types","Root trees in a King variable to rediscover gene-regulation links"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a sharp rise in a King's importance score when trees are allowed to grow deeper is a reliable sign of a true higher-order interaction, and that the model being scored has actually learned the interaction.","fun_headline_variants_meta":{"raw":{"variants":["Crown one 'King' per tree to expose hidden gene interactions","Turn random forests into targeted interaction detectors","Post-hoc tool ranks and types multi-order interactions","From Drosophila data, a method that distinguishes interaction types","Root trees in a King variable to rediscover gene-regulation links"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1830,"prompt_tokens":928,"completion_tokens":902,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":825}},"tokens_in":544,"tokens_out":902,"duration_ms":7645,"temperature":1.0,"reasoning_tokens":825,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:20:15.041365+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate data from a function with only pairwise interactions, $y = x_1 x_3 + x_5 x_7 + \\text{noise}$, run iKF at maximum depth 3 over 100 replicates, and count how often a depth-3 path containing $x_1$, $x_3$, and an irrelevant variable appears in the top-ranked lists; if false triples are common, the depth-jump rule does not establish interaction order.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines random forests and the permutation variable importance measure that Eq. (2) uses as the King's PVIM."},{"cited_title":"Kumbier, J","cited_arxiv_id":null,"evidence_quote":"Supplies the iterative random forests baseline that iKF extends and compares against, and the Drosophila embryo dataset used in the biological case study."},{"cited_title":"M., X.-Y","cited_arxiv_id":null,"evidence_quote":"Documents Zelda binding targets used to verify the rediscovered Zld-centred pairwise transcription-factor interactions."},{"cited_title":"Liang, S","cited_arxiv_id":null,"evidence_quote":"Documents Zld's temporal coordination of gap-gene expression, used to verify hierarchical interactions and the third-order Zld-Kni-Tll interaction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the biological background that enhancers act hierarchically, motivating the Hierarchical Interaction type."}],"review_version":1}