{"id":"73ee4d68-8bc6-4028-b702-3a76d54a2264","arxiv_id":"2412.16177","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs can generate original, simple conjectures about the solubilizer, but about two-thirds of their GAP code fails to execute.","lead":"This paper tests whether large language models can propose new math conjectures about a niche group theory concept, the solubilizer, and then write computer algebra code to check them for counterexamples. It finds the models do produce some original and plausible conjectures, but most of their code fails to run.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'plausible' label rests on a simple-group-only search: Listing 1 checks SimpleGroupsIterator, so the 40 'No Counter-examples' outputs do not test universal claims over non-solvable groups.","rationale":"The paper's strongest claim is deliberately modest, and the falsifiable half is well supported by the 109 counterexample findings with explicit examples (PSL(3,2), A5). The vulnerable half is 'plausible', which is operationalized as 'no counterexamples found'. The reader's weakest-assumption analysis correctly targets the search space. This is also the place where the pipeline's own prompt is weakest: the canonical GAP loop in Listing 1 uses SimpleGroupsIterator, not all non-solvable groups, so the evidence base for universal statements is narrower than the statement's quantifier. This is an internal-design issue, not merely a disagreement with consensus: the paper itself notes in A.3.1 that some runs used only simple groups. The appendix also shows that the category can include vacuous conjectures (A.9), meaning 'No Counter-examples' is a weak positive signal even within the searched class. A concrete re-test over non-simple non-solvable groups would settle whether this concern lands. Since the reader already judged the paper CONDITIONAL based on this same issue, my read does not move the verdict; it sharpens the stated condition and the test that would meet it.","tokens_in":12893,"tokens_out":14510,"duration_ms":150251,"concrete_test":"Use a human-written GAP script (no LLM-generated code) to re-test every one of the 40 'No Counter-examples' conjectures against a targeted family of non-simple non-solvable groups of order up to 10^6: direct products S×C_p for each simple group S in the original search and each prime p with |S|·p ≤ 10^6, plus a sample of split extensions of small irreducible modules by S (e.g., A5 acting on F_2^4, PSL(2,7) acting on F_2^3). Record which conjectures fail on these groups; if any fail, the simple-only pruning under-supports the plausibility claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LLM-generated conjectures are 'plausible' is supported mainly by the 40 outputs with 'No Counter-examples' (Section 4.1). For that evidence to be load-bearing, the counterexample search must cover a meaningful portion of the quantified domain, which is 'all non-solvable groups' in the conjectures. The paper's own code-generation prompt (Listing 1, line 32) instructs models to iterate over SimpleGroupsIterator(1, 10^6), i.e., only simple groups. Appendix A.3.1 confirms the runs checked 'all non-solvable (or in some cases just simple) groups of order up to 1,000,000', but the provided template defaults to simple groups. A universal statement can hold for every simple group up to 10^6 and still fail on a non-simple non-solvable group, e.g., a direct product S×C_p or a non-split extension. The solubilizer depends on normal and radical subgroups, so simple-group checks are not a sound reduction. Additionally, the 'No Counter-examples' category can include vacuous statements: Conjecture A.9 has an unsatisfiable hypothesis (y=1 gives ⟨x,x⟩ solvable), so the code has no instance to test. Thus the pruning step does not currently establish plausibility for the stated domain; a few non-simple counterexamples would collapse the 'plausible' half of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an LLM-based conjecture mining pipeline for the solubilizer Sol_G(x) in finite group theory. ChatGPT-4, Claude Sonnet, and Gemini 1.5 are prompted to generate conjectures and GAP code to test them; falsified conjectures are fed back into later prompts. From 757 raw outputs the authors identify 420 unique conjectures: 109 were falsified by counterexamples, 40 were labeled 'No Counter-examples', and 271 could not be executed. The paper concludes that LLMs can produce original, albeit simple, conjectures that are either plausible or falsifiable, while being limited in code execution.","tokens_in":156,"tokens_out":5723,"duration_ms":90917,"significance":"The pipeline is described concretely and transparently: full prompts, a GAP code template, and a manual novelty check by an author of the original solubilizer papers are included. The use of an external computational algebra system (GAP) means the counterexample tests are not circular. The 109 falsified conjectures provide solid evidence for the 'falsifiable via counterexamples' half of the central claim. The contribution would be a useful small-scale demonstration of LLM-assisted conjecture screening if the 'No Counter-examples' category were properly scoped. As written, however, the plausibility claim is overreaching because the counterexample search is not over the universal domain stated in the conjectures, and at least one 'No Counter-examples' output is vacuous. The paper's value is therefore conditional on rescoping the claims and removing vacuous entries.","major_comments":[{"comment":"The 'No Counter-examples' label is based on checking only simple groups of order up to 10^6 in the default template (SimpleGroupsIterator(1,1000000)), and Appendix A.3.1 states that the runs checked 'all non-solvable (or in some cases just simple) groups of order up to 1,000,000'. The conjectures themselves quantify over all non-solvable groups, e.g., Conjecture 4.1. A property can hold for every simple group up to that bound yet fail on a non-simple non-solvable group such as a direct product S×C_p or a non-split extension, and the solubilizer's dependence on subgroups and radical subgroups means simple-group checks are not a sound reduction. The 40 'No Counter-examples' outputs therefore do not establish plausibility for the stated domain. The paper should either restrict all claims to 'no counterexamples in the tested simple groups' or extend the verification to all non-solvable groups of order at most 10^6.","section":"Section 4.1, Listing 1, Appendix A.3.1"},{"comment":"Conjecture A.9 has an unsatisfiable hypothesis. Setting y=1 gives ⟨x, xy⟩ = ⟨x,x⟩ = ⟨x⟩, which is cyclic and hence always solvable, directly contradicting the assumption '⟨x, xy⟩ is not solvable for any element y ∈ G'. The reported 'No Counter-examples' output is therefore vacuous, as the test code has no instance to check. This entry is counted among the 40 'No Counter-examples' results, suggesting that the total may include other vacuous statements. Each 'No Counter-examples' conjecture should be manually checked for satisfiability before being used as evidence of plausibility.","section":"Appendix A.4.3, Conjecture A.9"},{"comment":"The proof of Conjecture 4.1 is logically invalid. From the fact that each maximal subgroup M of H = Sol_G(x) is contained in some maximal subgroup N of G, it does not follow that the intersection of all maximal subgroups of H is contained in the intersection of all maximal subgroups of G. A concrete counterexample to this inference is G = S_4 with H = C_4 embedded in the maximal subgroup D_8: Φ(C_4) is non-trivial while Φ(S_4) is trivial. The conjecture may still be true, but the appendix's proof does not establish it, and the paper's presentation of this as a successful model-generated proof is therefore overstated.","section":"Appendix A.5.1"}],"minor_comments":[{"comment":"The phrase 'solubilizer subset' is unclear; it should likely be 'solubilizer subfield' or 'solubilizer construction', since the solubilizer is a set-valued construct, not a subset in the sense used here.","section":"Section 3.1"},{"comment":"There are numerous typographical errors, e.g., 'dispuited', 'rememeber', 'similary', 'futher', and 'exampeled'. The manuscript would benefit from a careful proofreading pass.","section":"Throughout"},{"comment":"The sentence describing ChatGPT's conjectures contains a blank: 'the solubilizer couldn't be bigger than or that it couldn't be exactly equal to any of the following numbers'. The missing value or bound should be supplied.","section":"Section 5.2"},{"comment":"The OpenAI o1 data are based on only 51 samples and are explicitly preliminary. The statement that 'o1 is the best one to use for this task in all aspects' is too strong for such a small sample and should be softened or clearly labeled as preliminary.","section":"Appendix A.6"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is modest and the pipeline is reproducible, but the 'plausible' half of the claim is not supported by the verification that was actually performed. The fix is local: rescope the 'No Counter-examples' category, remove or test vacuous conjectures, and replace the flawed proof in Appendix A.5.1. If the authors make these changes, the paper could be acceptable as a small contribution to the LLM-for-mathematics literature; in its current form, the overstatement would likely be a reviewer concern at a serious venue. The paper's fit is stronger for a workshop or a specialized journal than for a flagship AI conference, though the decision is the editor's."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, honest empirical paper. It shows LLMs can generate simple conjectures about a fresh construct (the solubilizer) and then write GAP code to prune some of them via counterexamples. The headline number is 40 'no counterexamples' out of 420 unique conjectures, but that label is weaker than the abstract suggests.\n\nWhat's actually new: applying the guess-and-check loop to a relatively unexplored object, comparing three models plus o1, and giving a transparent breakdown of success, failure, and code-execution rates. The authors are candid about the limitations—high duplication, poor code generation, simple results—and they show concrete failure examples, including some in A5. I also appreciate that they include a vacuous case in A.9 rather than hiding it.\n\nThe load-bearing problem is the search space. The conjectures are stated for all non-solvable groups, but the prompt in Listing 1 instructs models to use SimpleGroupsIterator(1, 10^6), and Appendix A.3.1 confirms that many runs only checked simple groups. So 'No Counter-examples' really means 'no counterexamples among the groups actually tested'—often only simple groups up to order one million. A direct product S × C_p could falsify a universal statement even when every simple group is fine. A.9 is a clear example of a vacuous statement: the hypothesis '⟨x, xy⟩ is not solvable for any y' fails when y=1, so the code has nothing to check and reports no counterexample. That means the pruning step does not, by itself, make these conjectures 'plausible' over the stated domain.\n\nThe originality check is also thin: one author with ties to the solubilizer literature judged novelty, and the code and full dataset are not released. That is fixable, but as-is the empirical claims are hard to verify.\n\nOverall, this is a reasonable workshop-level contribution about LLM behavior and a possible pipeline for exploring a niche construction. I would not cite it as evidence of any mathematical truth, but it is useful as a data point on LLM conjecture generation. It deserves a serious referee, but only with revision: the authors need to either broaden the counterexample search to non-simple non-solvable groups (or at least state the limited domain in the abstract), and they should report vacuous cases separately. I'd bring it to a reading group on AI for mathematics.","headline":"A modest, honest study of LLM-generated conjectures on the solubilizer, but the 'no counterexample' label overstates the search coverage.","tokens_in":13703,"tokens_out":3907,"would_cite":false,"duration_ms":36652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["20D10","20D60","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs paired with a computational checker can produce original but simple math conjectures, though they often fail to write runnable code.","keywords":["LLM conjecture generation","solubilizer","finite group theory","GAP computer algebra","counterexample pruning","automated mathematical discovery","guess-and-check pipeline","ChatGPT Gemini Claude"],"falsifier":"Take the surviving conjectures—for instance Conjecture 4.1 or Conjecture A.1—and run the same GAP search over all non-solvable groups of order up to 1,000,000, not just the simple ones, then extend beyond that bound; a single counterexample would show the 'No Counter-examples' label was an artifact of the restricted search space rather than evidence of plausibility.","tokens_in":12673,"feed_emoji":"🧮","tokens_out":7268,"duration_ms":69399,"temperature":0.7,"pith_summary":"This paper tries to establish a modest empirical point: large language models can generate original mathematical conjectures in a specialized, unfamiliar area if each conjecture is immediately checked by computational algebra. Focusing on the solubilizer—the set of elements that generate a solvable subgroup together with a given element—the authors run a generate-and-prune loop in which ChatGPT, Gemini, and Claude propose conjectures, write GAP code to test them, and any falsified conjecture is fed back into the prompt. Of 420 unique outputs, 40 withstood the counterexample search and 109 were falsified by code that ran; the rest failed because the generated code did not execute. The upshot is that LLMs can act as a plausible brush-clearing tool for new mathematical constructs, even though they lean on known ideas and cannot yet be expected to produce major new mathematics.","feed_headline":"LLMs mint math conjectures that survive counterexample checks","feed_subtitle":"A group-theory experiment found 40 of 420 LLM-made conjectures survived a computational counterexample search.","key_machinery":"The object is the solubilizer, $\\operatorname{Sol}_G(x) = \\{y \\in G \\mid \\langle x, y\\rangle \\text{ is solvable}\\}$, a set that is not generally a subgroup. The method is a generate-and-prune loop: the LLM receives the definition, true results from the literature, and the current list of falsified conjectures; it outputs a conjecture plus GAP code; the code is executed; if it finds a counterexample, that conjecture is added to the prompt's false list and the loop repeats. This guess-and-check machinery lets the model's creativity be filtered by computation without a theorem prover, and the deliberately obscure subject reduces the chance that the model is regurgitating training data.","core_discovery":"The central discovery is that a generic LLM, given the definition of a relatively recent group-theoretic object and some literature, can produce conjectures that are distinct from the prompt material and that survive a computational counterexample search—provided the search is implemented as a loop rather than a one-shot generation. The paper reports that 40 of 420 unique conjectures returned 'No Counter-examples' under GAP checks on groups up to order 1,000,000, while 109 were refuted by concrete counterexamples and 271 could not be tested because the model's code did not run. The authors argue this shows LLMs are capable of original, simple conjecture generation, with ChatGPT producing the most surviving conjectures and OpenAI o1, in a small auxiliary sample, producing executable code far more often. The claim is deliberately modest: the value is the pipeline, not any single conjecture.","pith_inferences":["A conjecture labeled 'No Counter-examples' is only as strong as the search space; extending the GAP search to non-simple non-solvable groups of order up to 1,000,000, or beyond that bound, could falsify some of the 40 survivors that the paper currently treats as plausible.","The cosine-similarity measure mostly captures syntactic wording, not mathematical content; two conjectures can be near-duplicates semantically while scoring as distinct, so the reported uniqueness fractions likely overstate diversity.","A natural testable extension would be to feed each model the other models' surviving conjectures and measure whether the pool's diversity grows; the paper's own suggestion of multi-model interaction could be evaluated quantitatively.","The 'interestingness' of a conjecture is left unmeasured; a concrete proxy—number of surviving conjectures that later become theorems, or that require a new proof technique—could make the pipeline's output more comparable across models."],"forward_implications":["If the pipeline works as reported, LLMs can usefully clear out the space around a newly invented mathematical object, generating simple plausible conjectures that human experts can then prove or refine.","The method transfers in principle to other computational mathematics domains—number theory, geometry, combinatorics—where a solver like SageMath or MAGMA can play the role GAP plays here.","The high rate of non-executable code (64.5% of unique outputs, and 81.9% for Gemini) means code generation, not conjecture generation, is the current bottleneck.","Because the models tend to reproduce known patterns and duplicates, the approach is best seen as fast exploration rather than a source of deep or fundamentally new mathematics.","Adding a formal theorem prover or proof engine to the loop could turn surviving conjectures into proved statements, making the system end-to-end."],"supporting_citations":[{"why":"Supplies the baseline concern that LLM conjectures may be training-data regurgitations and prone to redundancy, which motivates the choice of the obscure solubilizer topic.","marker":"[6]"},{"why":"One of the solubilizer papers included in the prompt, providing the literature basis the models are asked to build on.","marker":"[10]"},{"why":"Provides group-structure results about the solubilizer that appear in the prompt and give the models material for conjecture generation.","marker":"[11]"},{"why":"Contains the theorem that the solubilizer order cannot be $p^2$, which GPT-4 re-conjectured, anchoring the paper's originality check.","marker":"[12]"},{"why":"Original source of the solubilizer definition and the result that it is a subgroup for every element exactly when the group is soluble.","marker":"[14]"},{"why":"Thompson's theorem that a finite group is soluble iff every two-generated subgroup is soluble, which motivates the construct's significance.","marker":"[23]"},{"why":"Characterizes the soluble radical via the condition that the solubilizer equals the whole group, a fact used in the prompt and in example conjectures.","marker":"[24]"}],"fun_headline_variants":["LLM conjectures pruned by their own counterexamples","40 of 420 AI math conjectures pass counterexample sweep","Mining math conjectures with an LLM prune-and-test loop","LLMs propose math conjectures, then filter by counterexample"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that a conjecture is plausible rests on the assumption that checking groups up to order 1,000,000—often only the simple groups in that range—is enough to reveal counterexamples if they exist; if that search space is too small, some of the 40 'No Counter-examples' results could be false in larger or non-simple groups.","fun_headline_variants_meta":{"raw":{"variants":["LLM conjectures pruned by their own counterexamples","40 of 420 AI math conjectures pass counterexample sweep","Mining math conjectures with an LLM prune-and-test loop","LLMs propose math conjectures, then filter by counterexample"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000838,"raw_usage":{"total_tokens":3582,"prompt_tokens":799,"completion_tokens":2783,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":415,"completion_tokens_details":{"reasoning_tokens":2724}},"tokens_in":415,"tokens_out":2783,"duration_ms":23982,"temperature":1.0,"reasoning_tokens":2724,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:17:46.377566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the surviving conjectures—for instance Conjecture 4.1 or Conjecture A.1—and run the same GAP search over all non-solvable groups of order up to 1,000,000, not just the simple ones, then extend beyond that bound; a single counterexample would show the 'No Counter-examples' label was an artifact of the restricted search space rather than evidence of plausibility.","supporting_citations":[{"cited_title":"Exploring mathematical conjecturing with large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline concern that LLM conjectures may be training-data regurgitations and prone to redundancy, which motivates the choice of the obscure solubilizer topic."},{"cited_title":"More on the non-solvable graphs and solvabilizers,","cited_arxiv_id":null,"evidence_quote":"One of the solubilizer papers included in the prompt, providing the literature basis the models are asked to build on."},{"cited_title":"The solubility graph associated with a finite group,","cited_arxiv_id":null,"evidence_quote":"Provides group-structure results about the solubilizer that appear in the prompt and give the models material for conjecture generation."},{"cited_title":"On the solubilizer of an element in a finite group,","cited_arxiv_id":null,"evidence_quote":"Contains the theorem that the solubilizer order cannot be $p^2$, which GPT-4 re-conjectured, anchoring the paper's originality check."},{"cited_title":"Non-solvable graph of a finite group and solvabilizers,","cited_arxiv_id":null,"evidence_quote":"Original source of the solubilizer definition and the result that it is a subgroup for every element exactly when the group is soluble."},{"cited_title":"Nonsolvable finite groups all of whose local subgroups are solvable,","cited_arxiv_id":null,"evidence_quote":"Thompson's theorem that a finite group is soluble iff every two-generated subgroup is soluble, which motivates the construct's significance."},{"cited_title":"Thompson-like characterization of the solvable radical,","cited_arxiv_id":null,"evidence_quote":"Characterizes the soluble radical via the condition that the solubilizer equals the whole group, a fact used in the prompt and in example conjectures."}],"review_version":1}