{"id":"47072660-af5a-472f-b853-8a6bb79d7add","arxiv_id":"2508.19780","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"EUREKA uses LLM pairwise comparisons to rank features by interestingness and trains logistic regression on the top-ranked features, producing non-obvious yet above-chance classifiers on six tabular datasets.","lead":"A new machine learning pipeline asks an LLM to rank dataset features by 'interestingness', then builds simple classifiers from only the most interesting features, even when they are not the most accurate. The paper shows such classifiers can still beat chance and can surface surprising rules, such as using humidity to detect room occupancy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM interestingness judgments are validated only by self-agreement, so the 'interestingness' of EUREKA's rules is not established; an external anchor is needed.","rationale":"The reader's weakest_assumption identifies exactly the same gap: Section 2.3 treats the LLM's pairwise comparisons as ground truth for interestingness, and Table 1 only checks same-model stability. I agree this is the most load-bearing vulnerability. The paper's other components—predictive accuracy above chance, likelihood-ratio significance tests, and the Borda count justification—are reasonable, and the limitations section is candid about subjectivity. However, the central construct 'interesting' is never anchored to anything outside the LLM, so the headline claim is not yet empirically established. The proposed concrete test (human rater agreement and cross-LLM replication) would settle whether the concern lands. Since the paper is a proposal with a plausible demonstration rather than a falsified result, the appropriate verdict remains conditional: accept the framework as a promising direction but require external validation before treating interestingness as a discoverable property.","tokens_in":13605,"tokens_out":4264,"duration_ms":48756,"concrete_test":"Run a pre-registered human evaluation: for Occupancy Detection, Twin Papers, and Adult, sample the top-3 EUREKA features and the top-3 conventional-baseline features; have at least 50 human raters choose the more interesting feature for each of the resulting 15 pairs per dataset. Compute Kendall's tau between the aggregate human ranking and the LLM's ranking. Separately, re-run EUREKA with a different LLM family (e.g., a Llama-3-70B or Claude model) on the same prompts. If the human–LLM tau is below 0.3 on any dataset, or if the two LLMs' top-3 features overlap on fewer than 2 of the 3 datasets, the interestingness ranking is model-specific rather than a reliable property of the rules.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EUREKA 'consistently identifies features that are non-obvious yet still predictive'. The predictive component is tested, but the interestingness component rests entirely on pairwise judgments from a single LLM, gpt-5-nano, prompted with feature names and task descriptions (Sec. 2.3). The only validation is within-model stability across 20 runs (Table 1). Stability is compatible with a ranking that is stable but arbitrary or driven by the LLM's memorized benchmark associations; it does not establish that the selected rules are genuinely non-obvious or interesting to humans. Because interestingness is operationalized as 'whatever the LLM ranks high', the examples in Section 3 could be artifacts of the specific model rather than properties of the rules. This is compounded by the Limitations section, which explicitly declines to use any objective or human-grounded measure. Without external validation, EUREKA is indistinguishable in practice from random feature selection followed by a post-hoc narrative about whichever predictive features emerge, so the paper's signature claim is unfalsifiable as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EUREKA, a framework for building 'interestingness-first' classifiers: features are ranked by LLM-based pairwise comparisons of how interesting each feature would make a prediction rule, and an interpretable logistic regression is trained on the top-K features. Experiments on six tabular datasets show that EUREKA selects features different from accuracy-driven baselines (e.g., humidity instead of light for occupancy, colon-in-title instead of reference-list length for citation counts) and that classifiers using these features achieve above-chance accuracy. The paper positions this as a new objective for supervised learning, prioritizing surprise and communicability over raw accuracy.","tokens_in":13895,"tokens_out":4337,"duration_ms":55989,"significance":"If the central claim holds, the paper introduces a genuinely different objective for classifier construction and a simple, reproducible pipeline. The method is easy to understand, the pairwise Borda-count ranking has a theoretical grounding in the ranking-from-pairwise-comparisons literature, and several of the discovered rules are concrete and thought-provoking. The predictive component is partially tested: selected features do yield above-chance accuracy on held-out data in several datasets, and the authors run a significance test against an intercept-only model. However, the paper's signature claim—that the selected rules are 'non-obvious yet still predictive'—is only established for the predictive half. The interestingness half is operationalized as 'whatever gpt-5-nano ranks highly' and is validated only by within-model stability, not by any human or independent rater. The paper's own Limitations section explicitly declines to use objective or human-grounded measures. As a result, the current evidence supports a weaker claim: LLM-judged feature rankings can be stable and can point to features that remain predictive. Whether these are interesting to humans remains unverified.","major_comments":[{"comment":"The central construct 'interestingness' has no external anchor. The only validation is stability of gpt-5-nano rankings across 20 runs (Table 1), but stability does not imply validity: a ranking can be stable and still arbitrary, or driven by the LLM's prior knowledge of benchmark datasets and their documentation. The paper's Limitations section says 'we deliberately refrain from relying on such objective metrics' and regards subjectivity as essential, but the abstract and Section 3 present examples as objectively 'non-obvious' and 'interesting'. Under the current operationalization, any feature ranked high by the LLM is, by definition, interesting, so no disconfirming evidence can arise on that axis. To support the abstract's claim, the authors should either (a) obtain human interestingness ratings for the top-ranked and bottom-ranked rules, (b) show agreement with an independent LLM or","section":"§2.3, Table 1, §5"},{"comment":"The K* selection rule is underspecified to the point of possible post-hoc selection. Section 2.6 says to increase K until the classifier is 'predictive enough', but no threshold is defined. In Table 2, when the top-ranked feature lacks predictive power, the authors 'also report the second-ranked feature'—e.g., HumidityRatio and Humidity for Occupancy, Density and Age for Mammographic Mass. This means the headline feature set is not the output of a single well-defined algorithm; it is selected after looking at performance. A reader cannot tell whether the reported features would have been chosen by a fixed rule. Please specify the criterion precisely (e.g., validation accuracy above chance plus a margin, or a p-value threshold on a validation likelihood-ratio test), apply it before inspecting test results, and report the automatically selected K* together with the accuracy at that K*. As","section":"§2.6, §3.3, Table 2"},{"comment":"The empirical support for 'consistently identifies features that are non-obvious yet still predictive' rests on a single stratified 80/20 split per dataset, with no error bars, confidence intervals, or multiple splits. This is especially problematic for the Twin Papers result, where the advantage over the chance rate is about 2 percentage points; even with 17,000 test instances, the reader needs a confidence interval and a description of how the significance stars were computed for each point. The 'chance rate' is defined as majority-class accuracy, which is a weak benchmark for class-imbalanced problems; a classifier that is statistically significant against an intercept-only model can still have test accuracy below the majority-class baseline (as the paper itself notes for HumidityRatio). Reporting balanced accuracy, AUC, or calibration alongside accuracy, and repeating the evaluation","section":"Figure 2, §3.3"},{"comment":"The method is not fully reproducible as described. The exact prompt used for pairwise comparisons is not given, the chosen model is only 'gpt-5-nano', and no temperature, sampling parameters, or API settings are stated. Since the LLM's judgment is the entire interestingness signal, the prompt and sampling details are essential experimental configuration, not an implementation detail. Similarly, the validation-based baseline is described as 'split the training data into validation sets' without specifying the number of folds or the selection rule. Please include the full prompt and the exact validation protocol in an appendix or supplementary material.","section":"§3.1, §2.3"}],"minor_comments":[{"comment":"The sentence about HumidityRatio being 'statistically significant' but 'test accuracy does not reach the chance baseline' is confusing as written. Clarify how a significant effect can fall below the majority-class baseline, for example by reporting the effect direction and the class imbalance, and explain why the star notation in Figure 2 does not mean 'above chance'.","section":"§3.3"},{"comment":"The Adult subplot's legend order appears inconsistent with the K values shown on the x-axis; the listed feature sets are not in monotone inclusion order. Please reorder the legend to correspond to the plotted curves.","section":"Figure 2"},{"comment":"The related work discussion mentions InterFeat and HypoBench as 'concurrent studies' treating interestingness as an evaluation criterion, but there is no direct technical comparison or positioning against their metrics. A brief comparison of the underlying notion of interestingness would help.","section":"§4.2"},{"comment":"The spurious-correlations paragraph says 'we believe spurious correlations are not a major concern in our problem setting', yet immediately invokes the Nicolas Cage/pool-drowning example as a valuable discovery. That example is exactly a spurious correlation, so the paragraph undercuts its own point. Please rewrite to separate 'spurious as causal' from 'interesting as narrative'.","section":"§5"},{"comment":"The claim that pairwise Borda count is 'up to constant factors, an information-theoretically optimal procedure' relies on Shah and Wainwright's theorem, but the setting here uses LLM judgments with possible transitive inconsistency. A sentence noting the assumptions of that theorem would prevent overgeneralization.","section":"§2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is readable and the idea is timely, but the gap between the title/abstract and the evidence is substantial. The most serious issue is that 'interestingness' is never anchored to anything outside the LLM's own rankings, which makes the flagship claim untestable as stated. The K* selection rule is also dangerously close to post-hoc selection. Both are fixable: the authors could add a small human-rating study (even with 10-20 raters on the discovered rules), specify a fixed selection criterion, and report variance over splits. If the authors are unwilling to add any external anchor, the paper would be better reframed as 'LLM-preferred-feature classifiers' and the claims downgraded accordingly. I would not reject: the predictive component is real and the central idea is worth pursuing, but the present manuscript is not yet publishable in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper defines a new objective—build classifiers that maximize interestingness rather than accuracy—and that is genuinely new. The pipeline is simple and clearly described: LLM pairwise comparisons of features, Borda count ranking, top-K logistic regression. The examples are vivid and the writing is honest. Credit where due: the predictive part mostly works. On held-out data, the selected features do beat chance, and the \"colon in title\" rule is a nice non-obvious finding. The comparison against standard feature selection shows EUREKA picks different features, and the stability analysis of the ranking is a reasonable first check.\n\nBut the central construct—\"interesting\"—is never validated. The LLM's ranking is the only measure, and self-agreement across runs (Table 1) does not show that the ranking corresponds to what humans find interesting. Stability could just mean the LLM consistently prefers its own pet features, or features it remembers from benchmark documentation. The Limitations section openly says they deliberately avoid objective or human-grounded measures. That is acceptable for a position paper, but the abstract claims EUREKA \"consistently identifies features that are non-obvious\"—that claim is not supported. Without human raters or at least a second, independent LLM, interestingness is whatever the LLM says.\n\nTwo more soft spots. The K* selection rule is underspecified: \"predictive enough\" is never defined, and since Figure 2 reports test accuracy across K, it is unclear whether the choice of K uses the test set. That is a potential leakage. There are also no error bars, no code, and single 80/20 splits.\n\nThese flaws are fixable. A small human evaluation of interestingness, or a cross-LLM agreement check, plus a defined K selection rule, would substantially strengthen the paper. It deserves a serious referee, but the signature claim needs external validation before it is treated as established.","headline":"A genuinely new objective and a clean pipeline, but 'interestingness' is validated only by the LLM itself, so the central claim is not yet supported.","tokens_in":14293,"tokens_out":2404,"would_cite":true,"duration_ms":28230,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EUREKA ranks features by LLM-judged interestingness and trains interpretable classifiers on the top-ranked ones; the resulting rules—humidity for occupancy, colons for citations—beat chance and significance tests on six benchmarks.","keywords":["interestingness-first classifiers","feature selection by LLM","pairwise comparison ranking","Borda count","interpretable classifiers","hypothesis generation","tabular benchmarks","surprising rules"],"falsifier":"Present the same pairwise 'which rule is more interesting?' prompts to a different LLM family and to a panel of human raters on the Occupancy Detection and Twin Papers tasks. If the Borda rankings across LLMs disagree substantially (Kendall τ < 0.5) or if the LLM ranking correlates no better than chance with human interestingness ratings, the claim that EUREKA captures interestingness rather than model-specific quirks would fail.","tokens_in":13547,"feed_emoji":"💡","tokens_out":11524,"duration_ms":108143,"temperature":0.7,"pith_summary":"This paper tries to establish that classifiers can be built for surprise rather than accuracy, and that doing so yields rules that are both non-obvious and still predictive. The proposed framework, EUREKA, uses an LLM to compare features pairwise—asking which would make the more interesting prediction rule—then aggregates those judgments into a ranking and trains a simple logistic-regression classifier on the top-ranked features. Across six tabular benchmarks, the selected features are exactly the ones accuracy-driven methods ignore: humidity instead of light for office occupancy, colon-in-title instead of reference-list length for citations, capital-loss instead of capital-gain for income. The resulting classifiers exceed the majority-class baseline and pass a significance test, while remaining easy to state and interpret. If this is right, classification becomes a mechanism for hypothesis discovery: surprising signals worth investigating, not just predictions to be scored.","feed_headline":"Surprising features still predict; EUREKA beats chance on six datasets","feed_subtitle":"Humidity, title colons, and capital losses beat the majority-class baseline—modest accuracy, real insight.","key_machinery":"The engine of the method is pairwise comparison by a large language model feeding a Borda count. For every pair of features, the LLM is asked which one would make the more interesting prediction rule for the target label; each win gives the feature one vote, and the features are ranked by total votes—a scheme the paper invokes as information-theoretically near-optimal for ranking from noisy comparisons. On top of this ranking, EUREKA builds a logistic-regression classifier using only the top-K interesting features, increasing K until the classifier first beats the baseline; interpretability of the base model is treated as a necessary condition, because an uninterpretable rule could not be co","core_discovery":"The paper's central claim is that interestingness can be the primary objective in building a classifier, not an afterthought. Concretely, it claims that asking an LLM to compare features by 'which would make the more interesting rule' yields rankings whose top features are consistently non-obvious yet predictive: Humidity and HumidityRatio outrank Light and CO2 for occupancy; 'title contains a colon' outranks 'longer reference list' for citation outcome; capital-loss outranks capital-gain for income. The resulting rules achieve accuracy above the chance rate—85% for humidity-based occupancy, 52% for the colon rule on a 17,000-instance test set—and pass a likelihood-ratio significance test. T","pith_inferences":["Editorial extension: EUREKA is effectively an automated hypothesis generator, but the paper stops at showing the rules are statistically detectable; a natural next step is to design confirmatory studies for rules like 'colon in title ⇒ more citations' before treating them as knowledge.","Editorial extension: because the LLM sees feature names and task descriptions, its 'surprise' judgments may come from prior beliefs about the world rather than from the data; comparing EUREKA's rankings with data-only surprisal measures would separate these sources.","Editorial extension: the framework could be extended to rank pairs of features or whole candidate classifiers, which the paper names as a limitation and future direction but does not test.","Editorial extension: the paper's tolerance of spurious correlations—justified by the Nicolas Cage example—suggests that an interestingness-first tool would benefit from an explicit 'spurious but fun' label so automated discoveries are not mistaken for causal findings."],"forward_implications":["EUREKA is ready to apply to any labeled tabular dataset with meaningful column names: the pipeline returns a short, human-readable rule that is both non-obvious and significantly better than the majority-class baseline.","The K-selection procedure gives a principled stopping rule: it returns the smallest prefix of the interestingness ranking whose classifier becomes predictive, making clear which features are interesting-but-weak and which are both interesting and predictive.","The discovered rules are interpretable by construction, so they can be communicated to non-experts—e.g., 'high humidity means the room is occupied'—without needing post-hoc explanation.","The significance checks (likelihood-ratio test at α=0.05 with Bonferroni correction) mean that even a 2% edge over chance on 17,000 samples, as in the colon-in-title rule, is reported as a real signal rather than noise.","Because EUREKA avoids features that accuracy-driven baselines rank first (light, reference-list length, BI-RADS score, capital gain), it explores a complementary region of the feature space that conventional feature selection leaves untouched."],"supporting_citations":[{"why":"Supplies the Borda-count aggregation whose optimality justifies turning pairwise LLM judgments into a global feature ranking.","marker":"[49]"},{"why":"Supplies the Twin Papers dataset and the paired-citation task used in the experiments.","marker":"[47]"},{"why":"Provides the active-ranking baseline that EUREKA is compared against in Section 2.4.","marker":"[17]"},{"why":"Provides the Group LASSO method used as an accuracy-driven baseline in the feature-selection comparison.","marker":"[32]"},{"why":"Defines the grouped-variable model-selection approach behind the Group LASSO baseline.","marker":"[65]"},{"why":"Provides the meta-analytic estimate (r≈0.13) showing the first-person pronoun rule is low-accuracy but considered interesting—the paper's core motivating example.","marker":"[9]"},{"why":"Supplies the psychological finding that first-person pronoun use flags depressive tendency, the key 'interesting but weak' rule that motivates the framework.","marker":"[46]"}],"fun_headline_variants":["LLM picks odd features that still predict","Colon in title predicts citations, says EUREKA","Humidity beats CO2 for occupancy insights","Interesting classifiers: lower accuracy, real insight","EUReKA finds unexpected yet useful features"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The entire method rests on the premise that an LLM's answer to 'which feature would make the more interesting rule?' is a trustworthy proxy for human interestingness; the paper only checks that the same LLM is stable across repeated runs, not that its rankings match what people actually find surprising.","fun_headline_variants_meta":{"raw":{"variants":["LLM picks odd features that still predict","Colon in title predicts citations, says EUREKA","Humidity beats CO2 for occupancy insights","Interesting classifiers: lower accuracy, real insight","EUReKA finds unexpected yet useful features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00013,"raw_usage":{"total_tokens":964,"prompt_tokens":749,"completion_tokens":215,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":146}},"tokens_in":493,"tokens_out":215,"duration_ms":2875,"temperature":1.0,"reasoning_tokens":146,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:27:47.431235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Present the same pairwise 'which rule is more interesting?' prompts to a different LLM family and to a panel of human raters on the Occupancy Detection and Twin Papers tasks. If the Borda rankings across LLMs disagree substantially (Kendall τ < 0.5) or if the LLM ranking correlates no better than chance with human interestingness ratings, the claim that EUREKA captures interestingness rather than model-specific quirks would fail.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Borda-count aggregation whose optimality justifies turning pairwise LLM judgments into a global feature ranking."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Twin Papers dataset and the paired-citation task used in the experiments."},{"cited_title":"Meier, S","cited_arxiv_id":null,"evidence_quote":"Provides the Group LASSO method used as an accuracy-driven baseline in the feature-selection comparison."},{"cited_title":"Yuan and Y","cited_arxiv_id":null,"evidence_quote":"Defines the grouped-variable model-selection approach behind the Group LASSO baseline."},{"cited_title":"Edwards and N","cited_arxiv_id":null,"evidence_quote":"Provides the meta-analytic estimate (r≈0.13) showing the first-person pronoun rule is low-accuracy but considered interesting—the paper's core motivating example."},{"cited_title":"Rude, E.-M","cited_arxiv_id":null,"evidence_quote":"Supplies the psychological finding that first-person pronoun use flags depressive tendency, the key 'interesting but weak' rule that motivates the framework."}],"review_version":1}