{"id":"55699ea7-558a-4b93-b4fc-e977cbeb09b1","arxiv_id":"2510.19698","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Logistic regression over LLM-generated natural-language rules outperformed letting the LLM reason with those rules on six binary text tasks.","lead":"The paper introduces RLIE, a four-stage framework in which an LLM proposes natural-language rules, logistic regression learns weights for them, and hard training errors drive iterative rule refinement. On six text-classification datasets, the direct logistic model outperformed feeding the same rules and weights back into the LLM.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on unverified reliability of LLM ternary rule judgments; if z_{i,j} is noisy or prompt-sensitive, E1's advantage and the division-of-labor conclusion are artifacts.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the LLM ternary judgments are unexamined despite being the sole input to the probabilistic combiner. My stress-test confirms this is the most critical point. The paper's empirical support is real but narrow: one LLM (gpt-4o-mini), small 200-sample training splits, no significance tests, no released code or full prompts, and no analysis of the local judgments' reliability. The central comparison E1 vs E2-E4 would be decisive only if z were stable and accurate; without that, we cannot distinguish a genuine division-of-labor principle from prompt-specific noise. Because the paper presents an interesting, plausible framework and the missing evidence is obtainable, the appropriate verdict remains CONDITIONAL. No change from the reader's verdict is needed, but the proposed concrete test should be part of the revision requirements.","tokens_in":14791,"tokens_out":3988,"duration_ms":41239,"concrete_test":"On a held-out 100-sample subset per dataset, take the final learned rule set H* and rerun the local judgment prompt used in Section 3.1 ten times per sample at temperature 0.3 and also with five reworded prompt templates. Compute per-rule agreement (Cohen's kappa) and the accuracy of z against true labels. Then refit the Section 3.2 logistic regression using the majority-vote z and compare E1 accuracy to the single-pass result. If kappa falls below 0.8, or if majority-vote E1 accuracy shifts by more than 2 points on any dataset, the linear-only superiority is not robust to LLM judgment instability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that linear-only inference (E1) outperforms LLM-augmented inference (E2-E4), supporting a division of labor where LLMs handle local rule judgment and logistic regression handles global aggregation. The foundation of this claim is the per-sample ternary judgment z_{i,j} = LLM(x_i, h_j) introduced in Section 3.1. These judgments are the only features used to train the elastic-net logistic regression in Section 3.2, and they also drive hard-example selection in Section 3.3 and the final E1 predictions in Section 3.4. If these judgments are noisy, biased, or unstable across repeated calls or prompt phrasings, then the learned weights, the rule pruning, and the comparison between E1 and E2-E4 all become artifacts of a particular LLM and a particular prompt template rather than evidence for a general neuro-symbolic design principle. The paper reports no agreement statistics, no parse-error rates, no calibration of z against true labels, and no prompt-variation experiments. Because the entire pipeline—including the 'surprising' degradation of LLM-augmented inference—depends on the quality and consistency of these local judgments, the absence of any reliability analysis is the most load-bearing gap. This concern does not by itself invalidate RLIE, but it means the headline conclusion is currently under-supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RLIE, a four-stage framework for learning weighted natural-language rule sets with LLMs: (1) an LLM generates candidate rules from training examples; (2) elastic-net logistic regression learns weights over LLM-produced ternary rule judgments; (3) iterative refinement selects hard examples and prompts the LLM to generate new or revised rules; (4) evaluation compares direct inference via the logistic model (E1) with three LLM-augmented inference strategies (E2: rules only, E3: rules + weights, E4: rules + weights + linear prediction). Experiments on six HypoBench datasets with gpt-4o-mini report accuracy and macro-F1. The central empirical claim is that E1 (Linear-only) achieves the best performance on 'nearly all' datasets, while injecting the learned rules and weights into the LLM degrades accuracy, supporting a division of labor in which LLMs handle local semantic judgments and a probabilistic combiner handles global aggregation. The paper also reports comparisons with zero-shot inference, zero-shot generation, IO Refinement, and HypoGeniC, and claims superior or competitive performance with lower variance.","tokens_in":15173,"tokens_out":8327,"duration_ms":71422,"significance":"If the central claim holds, the paper would provide a useful and counterintuitive empirical result: a hybrid neuro-symbolic pipeline with a simple logistic-regression combiner outperforms LLM-augmented reasoning on these tasks, and LLMs are best kept at local rule application. The framework is clear, the experiments use held-out test sets with fixed splits and repeated runs, and the paper reports mean and standard deviation for all methods. However, the evidence as presented is not yet sufficient to support the strength of the claims. The load-bearing gaps are (a) the absence of any reliability analysis for the LLM ternary judgments that serve as the only input features, (b) the lack of statistical significance testing given that several headline differences are within one standard deviation, (c) the absence of reproducible artifacts (code, full prompts, data splits, learned rule examples) and an ablation of the iterative-refinement stage, and (d) a comparison limited to LLM-based baselines that does not substantiate the claim of 'superior over all performance.' The central idea is promising, but the current manuscript needs substantial additional analysis before the conclusions ca","major_comments":[{"comment":"The entire pipeline rests on the LLM's ternary judgment z_{i,j}=LLM(x_i,h_j) in {−1,0,+1}. These judgments are used as the only features for logistic regression, for coverage filtering, for hard-example selection, and for E1 predictions. Yet the paper reports no reliability analysis: no repeated-call agreement, no parse-error rate, no calibration of z against true labels, and no prompt-variation experiments. If these judgments are noisy or prompt-sensitive, the learned weights, the pruning decisions, and the E1-vs-E2/E3/E4 comparison could be artifacts of one LLM and one prompt template. A minimal analysis (e.g., rerunning with alternate phrasings, reporting agreement and per-rule accuracy of z) is necessary before the 'division of labor' conclusion is supported.","section":"Section 3.1 (Eq. 1), Section 3.2 (Eq. 2), Section 5.2"},{"comment":"The headline claim 'Linear-only achieves the best performance on nearly all datasets' is not backed by significance tests. In Table 2, several differences are within one standard deviation: Dreaddit accuracy (E1=69.11±1.17 vs E4=69.00±1.33), Headlines (65.56±1.50 vs 65.33±1.86), Citations (identical 59.38±3.12), and LLM Detect (58.67±2.65 vs 56.67±3.51). In Table 1, IO Refinement outperforms RLIE on Dreaddit accuracy (74.89±3.47 vs 69.11±1.17) and LLM Detect accuracy (60.33±17.32 vs 58.67±2.65). Paired bootstrap tests or repeated-measures ANOVA are needed to support 'superior performance' and 'degradation' claims.","section":"Section 5.2, Table 2"},{"comment":"The reproducibility statement says code 'will be made publicly available upon publication,' and Appendix C states that prompts 'can be access after this paper is accepted.' Thus the manuscript under review provides no code, no full prompts for all six datasets, no exact data splits, and no example of a learned rule set or its weights. For a paper whose contributions are interpretability and rule quality, the absence of any concrete learned rules or weights is a substantial gap. Please include the actual prompts, data splits, and representative learned rules/weights in the appendix or supplement.","section":"Section 9, Appendix C"},{"comment":"The claim that RLIE 'achieves superior over all performance compared to a range of LLM-based methods' is not supported by the presented baselines. Only two LLM-based rule learners and two zero-shot LLM baselines are compared; no classical rule learning method (e.g., RIPPER, RuleFit) or standard supervised classifier is included. Even among the included methods, IO Refinement achieves higher accuracy and F1 on Dreaddit and on LLM Detect, and HypoGeniC achieves higher F1 on Retweets. The novelty claim of being 'the first' to combine LLMs with probabilistic rule learning is also not substantiated with a literature survey. Please moderate these claims or add the missing comparisons.","section":"Section 1, Section 5.1, Table 1"},{"comment":"The contribution of the iterative refinement stage is not ablated. The main results compare the full RLIE pipeline with external baselines, but there is no comparison to RLIE without refinement (e.g., one-shot rule generation followed by logistic regression). Since the paper claims that error-driven refinement improves rule quality and final performance, an ablation isolating this stage is necessary to validate its role.","section":"Section 3.3, Section 5"}],"minor_comments":[{"comment":"Typo: 'Generated rules are the evaluated' should be 'Generated rules are then evaluated'; 'surprising degrade' should be 'surprisingly degrades.'","section":"Abstract"},{"comment":"Superscript inconsistency: z^{(1)}_{i,j} = LLM(x_i, h^{(i)}_j) should use h^{(1)}_j rather than h^{(i)}_j.","section":"Section 3.1, Eq. (1)"},{"comment":"Typo: 'valuated' should be 'evaluated.'","section":"Table 2 caption"},{"comment":"The appendix only shows prompts for the Retweets task. Provide prompts for all datasets or state in the main text that they are included in a supplement. Also 'can be access' should be 'can be accessed.'","section":"Appendix C"},{"comment":"Minor language issue: 'you can used the weighted patterns' should be 'you can use the weighted patterns.'","section":"Figure 7, Figure 8 prompts"},{"comment":"The statement that LLMs 'can be inconsistent when required to strictly adhere to complex, explicit instructions' is a general claim without a citation. Add a reference or soften the wording.","section":"Section 5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope. The central idea is plausible and the pipeline is clearly described, but the two most important issues are the absence of reliability analysis for the LLM ternary judgments (the sole feature source) and the lack of significance testing for the headline comparisons. I recommend requiring both, as well as an ablation of iterative refinement and inclusion of reproducible artifacts (code, full prompts, learned rules). The authors should also temper the 'superior over all' and 'first' claims, which are not supported by the current limited comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: RLIE is a clean hybrid — LLM proposes rules and judges rule satisfaction per example, elastic-net logistic regression learns the weights. The sharpest result is that feeding learned rules (and the linear model's prediction) back into the LLM usually hurts accuracy. That is a real, useful observation.\n\nWhat's new: the systematic E1–E4 comparison. Linear-only inference beats LLM-augmented inference on most datasets, which makes a concrete design point: keep LLMs on local semantic judgments and let a probabilistic combiner do global aggregation. The framework pieces — coverage filtering, hard-example mining, early stopping — are standard, but the integration and the ablation are well done.\n\nSoft spots, in order:\n1. The whole pipeline depends on the LLM's ternary judgments z as features. No agreement, parse-error, or prompt-variation analysis. If z is noisy, the learned weights and the E1-vs-E2–E4 comparison are artifacts of one LLM and one template. This is the load-bearing gap.\n2. No significance tests. Several headline differences are within one standard deviation (Dreaddit: 69.11±1.17 vs 69.00±1.33). 'Linear-only is best' is often true in the table but not statistically pinned.\n3. 200 training samples, one LLM, and code/full prompts withheld until publication. The reproducibility statement is a promise, not a deliverable.\n4. The 'first' novelty claim outruns the related-work coverage.\n\nNone of this sinks the paper. The central idea is plausible and the empirical pattern is consistent across six datasets. With significance testing, artifact release, and a reliability analysis of the LLM judgments, this could be a solid contribution. As written, it is a promising preprint with overstated conclusions.\n\nWorth a reading-group slot; I'd cite only after a revision. I'd send to peer review, with a clear message that the z-reliability gap has to be addressed.","headline":"Clean hybrid pipeline and an interesting negative result on LLM rule injection, but the load-bearing LLM judgment step is uncharacterized and the headline differences aren't significance-tested.","tokens_in":15625,"tokens_out":3522,"would_cite":false,"duration_ms":32920,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the best way to use LLM-generated natural-language rules is to score each rule with the LLM and let a regularized logistic regression combine the scores; handing the rules back to the LLM for reasoning degrades accurac","keywords":["rule learning","large language models","logistic regression","interpretable rules","neuro-symbolic reasoning","iterative refinement","natural language hypotheses","ternary judgments"],"falsifier":"Replace the LLM's ternary judgments with random labels on the same datasets, or repeat the protocol with a different LLM and prompt template; if the ordering of the four inference strategies changes or the Linear-only advantage disappears, the central division-of-labor claim is specific to the particular LLM and prompt rather than a general principle.","tokens_in":14650,"feed_emoji":"⚖️","tokens_out":5716,"duration_ms":41407,"temperature":0.7,"pith_summary":"The paper tries to establish that a hybrid pipeline—LLMs write natural-language rules and make per-sample rule judgments, while a regularized logistic regression learns global rule weights—outperforms asking the LLM to reason with the rules directly. Across six text-classification datasets, the simplest strategy, applying the learned weights directly, ranks in the top two on accuracy and achieves the best F1 on all six tasks, while injecting rules, weights, and the linear model's prediction into the LLM context fails to help and often hurts. The authors interpret this as evidence for a division of labor: LLMs are reliable for semantic generation and local interpretation, but unreliable for fine-grained probabilistic integration. If correct, the paper offers a practical, transparent way to build interpretable classifiers from LLM-proposed rules and a caution against using LLMs as probabilistic reasoners.","feed_headline":"Weighted LLM rules beat handing the rules back to the LLM","feed_subtitle":"Letting an LLM write and judge rules while a logistic model does the voting wins on accuracy and stability across six text tasks.","key_machinery":"The central object is the ternary judgment matrix Phi^{(t)}(x_i) = (z_{i,1},...,z_{i,m}), where each z_{i,j} is the LLM's local judgment of whether rule j applies to sample i, valued +1 (positive), -1 (negative), or 0 (abstain). The mechanism is to treat these judgments as features in an elastic-net logistic regression, which provides global weighting, sparse rule selection, calibration, and a thresholded prediction. The abstention option makes rule coverage explicit, and the iterative refinement loop selects hard examples by prediction error |p_i - y_i| to drive new rule generation.","core_discovery":"RLIE learns a compact set of weighted natural-language rules through four stages: LLM-based rule generation with coverage filtering; feature construction where each sample becomes a ternary vector of rule judgments in {-1,0,+1}, with 0 marking abstention; elastic-net logistic regression that learns global weights and selects rules; and iterative refinement that feeds prediction-error hard examples back to the LLM. The central finding is that the Linear-only inference mode—using the learned weights directly as the classifier—performs best on nearly every dataset, while the three LLM-augmented modes, which receive the rules, the weights, and sometimes the model's own prediction as a reference,","pith_inferences":["The authors do not measure the consistency or calibration of the LLM's ternary judgments; a plausible editorial extension is that the ceiling of the whole framework is set by that local judgment quality, so improving per-rule judgment reliability should transfer directly to final accuracy.","The comparison is conducted on fixed small splits of 200 training and 200 validation samples; on larger datasets the LLM-based inference modes might close the gap, so the Linear-only conclusion may be regime-specific.","A testable extension suggested by the abstention design is to calibrate the abstain output probabilistically (e.g., via Platt scaling) rather than treating it as a hard 0, and to check whether that lifts the linear combiner further."],"forward_implications":["For LLM-generated rule sets, the recommended inference mode is direct weighted application rather than feeding the rules back to the LLM.","Providing learned rule weights to an LLM does not yield stable accuracy gains, so LLM internalization of probabilistic importance is unreliable.","Even a correct reference prediction from the linear model can be overridden by the LLM under complex prompting, so LLM-augmented reasoning should not be assumed to be a safe post-hoc layer.","The iterative refinement loop yields compact rule sets (capacity 10) that are transparent and auditable, suitable for explanation and human review.","Because the local LLM interface is fixed, the global combiner can be swapped for more expressive probabilistic models without changing the rule-generation stage."],"fun_headline_variants":["LLM proposes rules, logistic model does the math","Weighted rules from LLM outperform LLM-augmented inference","Hand rules to logistic model, not LLM, for best accuracy","RLIE: LLM writes rules, logistic model scores them"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the LLM's per-text judgments about whether each rule applies—yes, no, or not applicable—are accurate and consistent enough to be used as features, and the paper does not measure their reliability.","fun_headline_variants_meta":{"raw":{"variants":["LLM proposes rules, logistic model does the math","Weighted rules from LLM outperform LLM-augmented inference","Hand rules to logistic model, not LLM, for best accuracy","RLIE: LLM writes rules, logistic model scores them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1202,"prompt_tokens":762,"completion_tokens":440,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":506,"tokens_out":440,"duration_ms":4000,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T08:36:02.119767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the LLM's ternary judgments with random labels on the same datasets, or repeat the protocol with a different LLM and prompt template; if the ordering of the four inference strategies changes or the Linear-only advantage disappears, the central division-of-labor claim is specific to the particular LLM and prompt rather than a general principle.","supporting_citations":[],"review_version":1}