{"id":"1d8de818-f603-4ef8-88da-e963129b4b29","arxiv_id":"2412.10312","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"GenSPP uses genetic search to train the generator separately from the predictor, reporting higher rationale F1 than FR, MGR, MCD, and G-RAT while classification accuracy is roughly maintained.","lead":"This paper introduces GenSPP, a rationalization model that trains its explanation generator with a genetic algorithm instead of backpropagation, aiming to avoid the 'interlocking' failure of select-then-predict systems. The authors report better highlight quality than four baselines on a synthetic benchmark and on HateXplain, at the cost of much higher training time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fitness function in Eq. 8 is inverted: individuals violating the Eq. 7 gate receive maximal fitness, so the genetic search is driven toward generators whose predictors fail the task, undermining the central interlocking-free claim.","rationale":"The reader's verdict is REJECT, and my independent read reaches the same destination, but through a slightly different route. The reader's weakest_assumption focuses on Eq. 6 and the hand-set tolerance l + epsilon permitting degenerate masks; the reader's rationale also notes that Eq. 8 rewards the rejection branch of Eq. 7. I agree with the latter as the single most load-bearing problem. The inversion is internal inconsistency, not a disagreement with field consensus: for any individual with Lt < l + epsilon, h is bounded by 1/(1 − L) <= 2 roughly, whereas for any individual with Lt >= l + epsilon, h is approximately 1/epsilon_hat, which can be orders of magnitude larger. Since genetic selection is explicitly fitness-proportional and survival selects the highest fitness, the search prefers exactly the individuals that the constrained optimization in Eq. 6 forbids. This does not require sensitivity analysis or assumptions about the data; it follows from the formulas. It also explains why the reported behavior on HateXplain — selecting no tokens on many examples — is plausibly an optimized solution under the inverted objective: an empty mask can drive Lt above threshold while making Omega small, and that combination is rewarded. The paper contains genuinely useful components: the synthetic dataset is controlled, the disjoint-training motivation is clear, and the computational comparison in Table 9 is honest about overhead. But the core mechanism, as written, cannot support the claims of interlocking-free optimization. A corrected fitness direction might still yield competitive rationalization, but the current manuscript does not establish that. I therefore recommend REJECT, consistent with the reader, with the path to revision being a corrected fitness function and a re-run of all experiments.","tokens_in":19913,"tokens_out":2694,"duration_ms":28096,"concrete_test":"Instrument Algorithm 1 on the Toy dataset: for one generation, log Lt, tilde_h, h, and the roulette-wheel selection probability for every individual. Then rerun the full benchmark with the fitness corrected to h = tilde_h + epsilon_hat (same seeds, population size, and hyperparameters). If Table 1 Hl-F1 or Clf-F1 changes materially, or if the selected generator degenerates to empty or trivial masks, the published results depend on the inverted fitness. If results are statistically identical, the inversion is not the load-bearing failure.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing flaw is the fitness direction in Eqs. 7–8. GenSPP is a maximization-based genetic algorithm: roulette-wheel selection and half-elitist survival selection retain individuals with the largest h. Eq. 7 defines tilde_h = 1 − L when Lt < l + epsilon, and 0 otherwise. Eq. 8 then defines h = 1 / (tilde_h + epsilon_hat). Consequently, an individual whose trained predictor violates the classification constraint (Lt >= l + epsilon) receives tilde_h = 0 and h ≈ 1/epsilon_hat, the maximum possible fitness, while an individual that satisfies the constraint receives h ≈ 1/(1 − L), which is smaller whenever L > 0. The genetic search therefore actively selects against the constraint that Eq. 6 is meant to enforce. This is not a subtle tuning issue: the reward direction is exactly backwards. Since the paper's central claim is that GenSPP is interlocking-free because it optimizes the disjoint formulation in Eq. 6, and since the reported Hl-F1 improvements are attributed to that formulation, the empirical results in Tables 1 and 2 may be artifacts of rewarding failure. The reader's concern about a degenerate mask satisfying the constraint is secondary; the inversion is visible directly from the equations as written and invalidates the proposed optimization objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes GenSPP, a selective rationalization framework that replaces the end-to-end select-then-predict objective with a disjoint two-stage optimization solved by a genetic algorithm. Each individual in the population encodes a generator, is evaluated by training a fresh predictor on the masked inputs, and receives a fitness combining classification loss and highlight sparsity/contiguity. The authors claim GenSPP is the first interlocking-free architecture that requires no heuristics or additional regularizers, and report superior rationale quality with comparable classification accuracy on a new synthetic toy dataset and HateXplain. The paper also introduces a synthetic benchmark and provides code/data.","tokens_in":20236,"tokens_out":7106,"duration_ms":62328,"significance":"The idea of breaking interlocking by genetic search over generator parameters is novel, and the paper is commendable for releasing code and data, constructing a controlled synthetic dataset, and reporting five-seed runs with Wilcoxon significance tests. If the proposed fitness function were correct, the method would be a nontrivial advance over existing mitigation-based approaches. However, the central optimization objective as written is inverted, so the empirical claims are not currently supported; the significance of the contribution therefore depends on whether the objective can be corrected and the experiments redone.","major_comments":[{"comment":"The fitness function h in Eq. (8) is defined as h = 1/(tilde_h + hat_epsilon). Because Eq. (7) sets tilde_h = 0 whenever Lt >= l + epsilon, an individual whose predictor violates the classification constraint receives the maximum possible fitness h = 1/hat_epsilon, while an individual satisfying the constraint receives the smaller value h = 1/(1 - L + hat_epsilon). Since the genetic algorithm maximizes h (roulette-wheel selection and half-elitism, Section 5.3), the search is actively driven toward generators whose predictors fail the task. This reverses the intended effect of the constraint in Eq. (6), and consequently the results in Tables 1 and 2 may be artifacts of rewarding constraint violation rather than evidence for the proposed method.","section":"Section 5.2, Eq. (8)"},{"comment":"The reformulation in Eq. (6) is presented as being 'equivalent' to Eq. (4), but no derivation or formal statement is given. Eq. (4) minimizes the classification loss alone, while Eq. (6) minimizes Omega(m) subject to a loss bound l + epsilon. The equivalence would require that the set of masks with loss below l + epsilon coincides with the relevant minimizers of Eq. (4), and that l = L(f_{omega*}(x), y) is the right benchmark; neither is established. The threshold l + epsilon is hand-set to 0.1 (Toy) and 0.6 (HateXplain) with no sensitivity analysis, so a near-empty mask may satisfy the constraint vacuously. This affects the central claim that GenSPP optimizes the disjoint formulation and hence that it is interlocking-free in the intended sense.","section":"Section 4, Eq. (6)"},{"comment":"Table 1 reports that GenSPP selects on average 0.75 tokens on HateXplain (selection rate 6.51%), whereas baselines select about 3 tokens. The paper interprets this as a learned preference to select no highlight for negative examples, but with the inverted fitness this is also consistent with degenerate near-empty selections that still achieve Clf-F1 around 69.7. The manuscript does not report the distribution of selection sizes, the number of examples with zero selected tokens, or the held-out Lt relative to the threshold l + epsilon. Without such evidence, the high Hl-F1 and comparable Clf-F1 do not demonstrate that meaningful rationales are learned, rather than a trivial bias exploited through empty inputs.","section":"Section 7, Table 1"}],"minor_comments":[{"comment":"The phrase 'does not require any learning overhead' is misleading given the roughly 5-10x wall-clock increase reported in Section 8 and Table 9; clarify that it refers to additional modules or regularizers, not computational cost.","section":"Abstract and Section 8"},{"comment":"omega* is not defined; specify that it is the predictor trained on the unmasked input and describe how it is obtained in practice.","section":"Section 4, Eq. (5)"},{"comment":"The summation for Ls runs from i=0 while Lc runs from i=1; align the indexing with the token definition x_1..x_n.","section":"Section 2.1, Eq. (2)"},{"comment":"The number of predictor training epochs during evaluation is set to 3 without sensitivity analysis; since fitness scores depend critically on this choice, a small study of its effect would strengthen the paper.","section":"Appendix A.2"},{"comment":"The Wilcoxon test is reported only against the 'best baseline'; state whether multiple-comparison correction was applied and report the exact test statistic or p-value range.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is salvageable, but the fitness inversion is a serious technical error, and the reported experiments would need to be redone with a corrected objective. I would therefore recommend major revision rather than outright rejection, provided the authors can fix Eq. (8) and re-run the comparisons. If, after correction, the qualitative conclusions change, rejection would be the appropriate outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the short version: this paper proposes GenSPP, a genetic-algorithm alternative to SGD for training select-then-predict rationalization. The idea is new and the code/data are released, but the fitness function in Eq. 8 has the reward direction inverted. Written as h = 1/(tilde_h + epsilon_hat), an individual whose predictor violates the task constraint (tilde_h = 0) gets h = 1/epsilon_hat, the maximum fitness, while a compliant individual gets a smaller h as long as its loss is positive. Since selection and survival both favor larger h, the genetic search is pushed toward the exact failure Eq. 6 is meant to rule out. This is not a subtle tuning issue; it is the opposite of the stated objective. The reported Hl-F1 gains in Tables 1-2 may largely be artifacts of rewarding predictors that fail the classification task.\n\nWhat is genuinely new: evolving generator parameters with a GA, and evaluating each candidate via a predictor trained from scratch. This is a sensible way to decouple generator and predictor co-adaptation. The paper is transparent about computational overhead and hyperparameters, and the synthetic toy dataset is a useful controlled probe. The lower seed variance relative to baselines is also worth noting.\n\nThe soft spots extend beyond Eq. 8. The 'first interlocking-free' claim is overstated: Jain et al. 2020 and Li et al. 2022 already break the joint training loop; the delta here is the evolutionary search, not the disjoint structure. The HateXplain comparison is also uneven: GenSPP selects 0.75 tokens on average versus ~3.4 for baselines, so raw Hl-F1 is apples-to-oranges, and the 4-point Clf-F1 gap to G-RAT is called 'comparable' when it is a real gap. Li et al. 2022, the closest related work, is not directly compared.\n\nIf the fitness direction is corrected and experiments rerun, the idea could still hold. As written, the central mechanism contradicts its own equations, so my own verdict is reject. But I would send it to a serious referee: the idea is novel, the artifacts are detailed, and the flaw is concrete and fixable. The referee should ask for a corrected objective, a matched-sparsity comparison on HateXplain, and a direct comparison against Li et al. 2022.","headline":"Novel genetic-search rationalization idea, but Eq. 8 inverts the fitness reward direction, undermining the central claim; the paper still deserves review because the idea and artifacts are concrete.","tokens_in":20757,"tokens_out":3072,"would_cite":false,"duration_ms":28177,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GenSPP eliminates interlocking in select-then-predict rationalization by evolving the generator with a genetic algorithm and training a fresh predictor for every candidate, so no module can dominate the other during training.","keywords":["selective rationalization","interlocking","genetic algorithm","neuroevolution","rationale extraction","explainable NLP","hate speech detection"],"falsifier":"On HateXplain, set the tolerance $l+\\epsilon=0.8$ so that an empty mask predicting the majority class (cross-entropy near 0.64) satisfies the constraint, then run the full GenSPP search; if high-fitness individuals select zero tokens on most examples while test highlight F1 collapses toward chance, the fitness criterion is insufficient to enforce meaningful highlights.","tokens_in":19623,"feed_emoji":"🧬","tokens_out":7029,"duration_ms":60863,"temperature":0.7,"pith_summary":"Select-then-predict rationalization couples a generator that picks highlight tokens with a predictor that classifies the masked input; when trained jointly, the two modules can lock into a suboptimal equilibrium called interlocking. This paper proposes GenSPP, a framework that claims to be the first interlocking-free solution that needs no heuristics, weight sharing, or auxiliary regularizers: the generator is evolved by a genetic algorithm, and every candidate generator is evaluated by training a fresh predictor from scratch on its masked inputs. The evaluation is driven by a fitness function that requires masked-input classification to stay within a tolerance of an unmasked-input predictor's loss while minimizing sparsity and contiguity penalties. On a controlled synthetic dataset and on HateXplain, GenSPP reports significantly higher highlight F1 than four SPP baselines (76.0 versus 65.7 on Toy, 42.6 versus 36.2 on HateXplain), comparable classification F1, lower selection ratios, and lower variance across seeds. The paper also shows GenSPP recovers when the initial population is deliberately skewed toward a degenerate generator.","feed_headline":"Genetic search breaks rationalization's interlocking trap","feed_subtitle":"Separate training of generator and predictor lifts highlight F1 by 6 to 10 points on two benchmarks.","key_machinery":"The carrying mechanism is the Eq. (6) reformulation of select-then-predict as a constrained generator-only optimization, solved by genetic global search. Each individual is a full set of generator parameters; roulette-wheel selection, one-point crossover, Gaussian mutation, and half-elitism survival produce new candidate generators, and the fitness function of Eqs. (7)-(8) gives nonzero score only when masked-input classification loss falls below the hand-set tolerance $l+\\epsilon$ and then rewards low sparsity and contiguity regularization. Disjointness is enforced by training a fresh predictor from scratch for every individual with the generator frozen.","core_discovery":"GenSPP's central discovery is that interlocking disappears once the generator and predictor are never co-adapted: the joint minimization $\\min_\\theta \\min_\\omega L(f_\\omega(g_\\theta(x)\\odot x),y)$ is replaced by the constrained problem $\\min_\\theta \\Omega(m)$ subject to $\\min_\\omega L(f_\\omega(g_\\theta(x)\\odot x),y)\\le l+\\epsilon$, where $l$ is the loss of a predictor trained on the unmasked input. A genetic population of generator parameter vectors is searched globally, and each individual is scored by a fitness $\\tilde h$ that combines normalized classification loss and mask regularization; predictors are re-initialized and trained from scratch for each evaluation. The author's claim is that this makes interlocking structurally impossible rather than merely unlikely, because the predictor cannot overfit to a stale mask and the generator is not updated through predictor gradients. The reported results support the claim on two benchmarks by showing better highlight quality and recovery from a deliberately skewed generator initialization.","pith_inferences":["GenSPP's disjointness argument is not tied to genetic algorithms specifically; any non-differentiable global search, such as evolutionary strategies or random search with enough budget, should exhibit the same interlocking-free property, so the core claim is testable by swapping the search operator.","The hand-set tolerance $l+\\epsilon$ is the main hidden dial: the paper fixes it at 0.1 (Toy) and 0.6 (HateXplain) with no sensitivity analysis, so an immediate extension is to measure how highlight quality degrades as the tolerance widens toward the loss of a trivial predictor.","On longer inputs with multiple rationales, selective rationalization typically moves to sentence-level selection; applying GenSPP there would test whether the genetic search overhead and contiguity penalty scale, and whether the disjointness advantage survives when the mask space is much larger.","The reported ability to select no tokens on negative examples in HateXplain suggests GenSPP can learn per-example sparsity rather than a global ratio; a testable extension is whether this emerges only because the fitness threshold is loose enough to let empty masks pass on non-hate posts."],"forward_implications":["If GenSPP is correct, interlocking is not a pathology that must be smoothed, guided, or regularized away; it is an artifact of co-adaptation, and disjoint training eliminates it at the architectural level.","Rationalization models no longer need a sparsity threshold $\\alpha$ or surrogate differentiable losses; the fitness function can directly encode sparsity and contiguity preferences, removing dataset-specific hyperparameter tuning.","The method inherits a robustness property: a deliberately bad generator placed in the initial population does not trap the search, so recovery from interlocking is a population-level property rather than a gradient-escape event.","The cost of these properties is computational: a single GenSPP seed run takes roughly 36 minutes on Toy and 78 minutes on HateXplain versus roughly 8 and 4 minutes for baselines, so gains in highlight quality are bought with search budget."],"supporting_citations":[{"why":"Introduces the select-then-predict architecture and the sparsity/contiguity regularizers that GenSPP replaces with its fitness function.","marker":"Lei et al. (2016)"},{"why":"Diagnoses interlocking and shows the predictor overfits to suboptimal masks; this motivates the Eq. (6) split into disjoint training.","marker":"Yu et al. (2021)"},{"why":"Relaxes the sparsity constraint to a target sparsity level and defines the $\\alpha$ setting that GenSPP avoids requiring.","marker":"Chang et al. (2020)"},{"why":"Provides the FR baseline and the skewed-generator training protocol used in the synthetic skewing experiments.","marker":"Liu et al. (2022)"},{"why":"Provides the MGR multi-generator baseline, which GenSPP claims is unnecessary under disjoint training.","marker":"Liu et al. (2023b)"},{"why":"Provides the MCD guidance-based baseline that GenSPP is compared against.","marker":"Liu et al. (2023c)"},{"why":"Provides the G-RAT guidance-based baseline, a recent competitor that still mitigates rather than eliminates interlocking.","marker":"Hu and Yu (2024)"},{"why":"Supplies the HateXplain dataset with human rationales, the real-world benchmark for evaluating highlight quality.","marker":"Mathew et al. (2021)"}],"fun_headline_variants":["GenSPP: first interlocking-free rationalization with genetic search","Genetic search stops interlocking in AI rationalization","Split training kills interlocking: GenSPP tops benchmarks","Interlocking gone: GenSPP uses genetic global search","No more interlocking: GenSPP's disjoint training wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that matching the loss of an unmasked-input predictor within a hand-set tolerance is the right test of a good highlight; if the tolerance is loose enough, a degenerate mask such as selecting almost nothing can satisfy the constraint and receive high fitness, decoupling the search from genuine rationalization.","fun_headline_variants_meta":{"raw":{"variants":["GenSPP: first interlocking-free rationalization with genetic search","Genetic search stops interlocking in AI rationalization","Split training kills interlocking: GenSPP tops benchmarks","Interlocking gone: GenSPP uses genetic global search","No more interlocking: GenSPP's disjoint training wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00138,"raw_usage":{"total_tokens":5552,"prompt_tokens":874,"completion_tokens":4678,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":4596}},"tokens_in":490,"tokens_out":4678,"duration_ms":29852,"temperature":1.0,"reasoning_tokens":4596,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:58:09.540443+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On HateXplain, set the tolerance $l+\\epsilon=0.8$ so that an empty mask predicting the majority class (cross-entropy near 0.64) satisfies the constraint, then run the full GenSPP search; if high-fitness individuals select zero tokens on most examples while test highlight F1 collapses toward chance, the fitness criterion is insufficient to enforce meaningful highlights.","supporting_citations":[{"cited_title":"Jaakkola","cited_arxiv_id":null,"evidence_quote":"Relaxes the sparsity constraint to a target sparsity level and defines the $\\alpha$ setting that GenSPP avoids requiring."}],"review_version":1}