{"id":"0443a8c1-a311-4697-a493-18e54fe5b46e","arxiv_id":"2412.13645","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"LLMs' hypotheses for real-world classification tasks are driven mostly by task priors, and in-context demonstrations, even with flipped labels, do little to change them.","lead":"This paper tests whether large language models generate hypotheses from the examples in their prompt or from knowledge they already have. Across five real-world classification tasks, removing the examples barely changed the quality of the hypotheses, suggesting the models lean mainly on prior knowledge.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The accuracy metric is an LLM applying the hypothesis; if GPT-4o classifies from its own priors rather than the pattern, the w/ vs w/o invariance reflects the evaluator, not the generator.","rationale":"The single most load-bearing concern is the validity of the hypothesis-based inference metric as a measure of hypothesis quality. The paper's headline quantitative result—Table 1, overall best accuracy 68.62 without demos vs 68.56 with demos—is produced by having GPT-4o apply a generated hypothesis to test examples. The evaluator is the same model family as the generator and the prompts contain task names and the hypotheses contain domain vocabulary; it can rely on its own priors to label the test item. If this happens, the accuracy gap is compressed toward the evaluator's prior accuracy regardless of the hypothesis text, so near-equal accuracy does not imply near-equal hypothesis quality. The label-flipping analyses in Section 5.2 are even more directly vulnerable: invariance of downstream predictions under flipped demo labels is exactly what you would expect if the classifier (the evaluator) does not use the generated hypothesis. Appendix C.1 attempts to control this by removing task-specific wording, but it still uses an LLM to judge 'alignment' and the hypothesis text itself still contains the task vocabulary; the control is therefore incomplete. The case study in Section 6.3 provides genuine evidence that the generator itself resists flipped labels, but it is limited to a single synthetic sentiment task with IO-prompting and does not evaluate hypothesis quality on the five benchmark datasets. Human evaluation in Section 5.4 is pairwise preference rather than accuracy and covers only three text datasets. Consequently, if the evaluator confound is real, the paper's central claim is not established by its main experiments. This is exactly the reader's weakest assumption, and I agree. The proposed null-hypothesis control would settle it: if GPT-4o scores well above chance on test data even when given a semantically unrelated pattern, the metric is not measuring hypothesis quality. I concur with the CONDITIONAL verdict: the concern is concrete and testable, and the paper provides enough auxiliary evidence (case study, human preference) that a prompt rejection is not warranted, but the main quantitative claims need this control before acceptance.","tokens_in":16955,"tokens_out":7015,"duration_ms":61104,"concrete_test":"Run the Section 4.2 hypothesis-based inference with a 'null hypothesis' that is semantically unrelated to the task (e.g., 'positive examples contain the word banana' for the three text datasets; 'the X-ray image shows a cat' for PneumoniaMNIST), using the exact evaluation prompts from Appendix F and the same GPT-4o settings. A faithful pattern follower would predict mostly 'no', giving accuracy near the negative-class proportion (50% in these balanced test sets). If observed accuracy is substantially above 50% (say, >55-60%) and comparable to the real-hypothesis accuracy in Table 1, the evaluator is classifying from its own priors, not from the provided pattern, so Table 1 cannot distinguish generator quality from evaluator prior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that hypothesis quality is unaffected by removing demonstrations rests on the hypothesis-based inference metric (Section 4.2). That metric is implemented by prompting GPT-4o to apply a generated pattern to a test example. The evaluator shares the generator's task-specific priors, and the evaluation prompts (Appendix F) explicitly name the task (e.g., 'hallucination', 'pneumonia') and the hypotheses themselves contain domain vocabulary (e.g., 'pleural effusion'). Consequently, GPT-4o can answer from its own knowledge of what a hallucinating or pneumonia-positive case looks like, independent of the supplied pattern. If so, accuracy will be high and nearly constant across w/ and w/o demo conditions even when the hypotheses differ in quality, so Table 1's small gaps (68.62 vs 68.56) do not establish that the generator's output is unchanged. The label-flipping experiments (Section 5.2) share this defect: if the evaluator ignores the pattern, flipping labels in the generator's demonstrations will not change downstream predictions. Appendix C.1 rewrites the question as 'Does the provided text/image align with the given pattern?' but still uses an LLM to judge alignment, and the pattern text itself still encodes the task, so the confound remains. The case study (Section 6.3) directly examines generation and supports prior dominance, but it is a single synthetic sentiment task with IO-prompting only and cannot carry the full quantitative claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper asks whether LLM-generated natural-language hypotheses for real-world classification tasks are driven primarily by task-specific model priors or by in-context demonstrations. The authors compare three hypothesis-generation baselines (IO-prompting, iterative refinement, and HypoGeniC) with and without demonstrations on five datasets across three LLMs, measuring hypothesis quality via LLM-based inference, LLM-based scoring, human pairwise comparison, and a case study with flipped labels. They report that removing demonstrations causes minimal loss in downstream accuracy, that label flipping in demonstrations rarely changes predictions, and that a sentiment case study shows the model continues to generate hypotheses consistent with its prior. They conclude that task-specific model prior dominates hypothesis generation in real-world inductive reasoning.","tokens_in":17236,"tokens_out":8478,"duration_ms":66579,"significance":"If the central empirical claim holds, the paper makes a valuable contribution: it challenges the common assumption that labeled demonstrations are the primary driver of in-context inductive reasoning, and it offers a practical message about when prompt-data curation may be unnecessary. The study is reasonably broad in design (three baselines, three LLMs, multiple modalities, human evaluation) and is transparent about its limitations (classification only, limited case study). The case study in Section 6.3 provides direct process-level evidence that the model can ignore flipped-label demonstrations. However, the main accuracy result depends on an LLM-based evaluator that may share the generator's priors, so the central quantitative claim needs a control before it can be accepted.","major_comments":[{"comment":"The hypothesis-based inference metric used for the headline result in Table 1 is vulnerable to a prior-confounding: the evaluator LLM is given the task name and the hypothesis text, which itself contains domain vocabulary (e.g., 'pleural effusion' in Table 10), so GPT-4o can answer from its own task knowledge even if it ignores the provided pattern. The near-identical w/ and w/o demo accuracies (68.62 vs 68.56) are therefore consistent with an evaluator that is invariant to the hypothesis. Appendix C.1 replaces the task framing with a generic alignment question but still uses an LLM judge on the pattern text, so the confound persists. I request a control condition that uses a pattern from a different task or a scrambled pattern, or a non-LLM pattern matcher; without such a control, the claim that removing demonstrations does not degrade hypothesis quality is not established.","section":"4.2, Appendix C.1, Appendix F"},{"comment":"The 'w/o demos' conditions for iterative refinement and HypoGeniC are not 'model prior only' conditions. In iterative refinement without demonstrations, the model receives ranked hypotheses with their validation accuracies; in HypoGeniC without demonstrations, it receives reward-score rankings computed from the validation set. These are data-derived feedback signals that can shape the generated hypotheses. Hence the comparison in Table 1 for these two baselines conflates the absence of in-context demonstrations with the presence of validation-set feedback, and the overall averages do not isolate the role of model prior. The zero-shot interpretation is clean only for IO-prompting; please either remove the validation feedback in the w/o demos arms or restrict the prior-dominance conclusion to the IO-prompting baseline.","section":"4.1, 5.1"},{"comment":"The case study in Section 6.3 is the only direct, generation-level evidence that the model ignores flipped-label demonstrations, and it is limited to a single synthetic sentiment task with IO-prompting. This evidence, together with the human preference results (Section 5.4), supports a qualitative version of the claim, but it does not by itself carry the dataset-level and model-level quantitative generalization in Sections 5.1 and 7. Please either add generation-level analyses for additional datasets/baselines or soften the conclusion accordingly.","section":"6.3, 7"}],"minor_comments":[{"comment":"The word 'Accuraccy' should be 'Accuracy' throughout the table captions, and Table 9 contains the typo 'satefy' for 'safety'.","section":"Tables 1, 2, 7, 8, 9"},{"comment":"Section 6.1 refers to 'Figure 5' when presenting accuracy differences between label formats, which appears to be Table 5; Section 6.2 refers to 'Table 5' for the ACR/BCR results, which appears to be Figure 5. The cross-references should be corrected.","section":"6.1, 6.2"},{"comment":"The last row of Table 5 ('w/o demos Best: 68.62 Average: 62.12') is ambiguous because it mixes reference values from Table 1 without explaining that these are the no-demonstration baselines; please clarify its meaning.","section":"Table 5"},{"comment":"The human evaluation reports nine participants but gives no inter-annotator agreement or participant background information; adding these details would strengthen the reliability assessment.","section":"5.4"},{"comment":"The method name is spelled inconsistently as 'HypoGeniC' and 'HypoGenic' in different places; please standardize.","section":"4.1, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an interesting and practically relevant question, and the authors have made a serious effort with multiple models, datasets, and evaluation layers. The main blocker is the evaluator confound: the accuracy metric may not measure the generated hypothesis at all. A control condition with scrambled or cross-task patterns would be straightforward and would substantially raise confidence in the central claim. The second issue, that iterative refinement and HypoGeniC use validation-set feedback in their 'no demos' arms, also needs to be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's core contribution is a systematic ablation: across three baselines, five real-world tasks, and three LLMs, removing in-context demonstrations barely changes hypothesis-based classification accuracy. That's a useful, well-structured empirical result. The label-flipping analysis showing prior resistance is also a nice addition, and the case study in Section 6.3 directly examining generation—where the model's supported demonstrations ignore flipped labels—is the most convincing part of the paper. That direct evidence supports the qualitative claim that priors dominate, even if the quantitative headline needs caveats.\n\nThe main soft spot is exactly what the stress-test note flags: the hypothesis-based inference metric (Section 4.2) uses GPT-4o to apply the generated hypothesis to test examples. The evaluator shares the generator's task priors, and the evaluation prompts explicitly name the task (e.g., 'pneumonia', 'hallucination'), while the hypotheses themselves contain domain vocabulary like 'pleural effusion'. So GPT-4o can answer correctly from its own knowledge, independent of the supplied pattern. That makes the accuracy nearly invariant to demonstration removal regardless of hypothesis quality. Appendix C.1 strips task-specific wording but still uses an LLM and still shows the pattern text, which encodes domain knowledge, so the confound remains. The same issue applies to the label-flipping experiments: if the evaluator ignores the pattern, flipping labels in the generator's demonstrations won't change predictions. The human evaluation and case study are real external checks, but the human eval is pairwise preference on three text datasets, not an accuracy measurement, and the case study is a single synthetic sentiment task with IO-prompting only.\n\nOther, smaller issues: no significance testing (just means and standard deviations, and some differences are within noise), and no released code, so reproducibility is limited. The limitations section is honest about scope but doesn't mention the evaluator confound.\n\nOn balance, the paper is worth engaging with. The finding, if true, is important: LLM-generated 'novel' hypotheses in real-world tasks may be pre-trained knowledge retrieval rather than data-driven insight. The direct generation evidence and the breadth of the ablation make it a solid contribution even with the evaluator issue. I'd send it to peer review, but with a clear request to fix the main metric—either by using a non-LLM classifier for hypothesis application or by validating the LLM evaluator against human judgments on the test set—and to report significance tests and release the code.","headline":"Systematic study of LLM hypothesis generation with a plausible central claim, but the main accuracy metric is confounded by the LLM evaluator's own priors, so the strong quantitative conclusion is not fully supported.","tokens_in":17762,"tokens_out":2064,"would_cite":true,"duration_ms":19987,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs generate hypotheses from prior knowledge, not examples, on real-world inductive reasoning tasks, so removing demonstrations costs little accuracy.","keywords":["inductive reasoning","hypothesis generation","model prior","in-context demonstrations","large language models","zero-shot hypothesis proposal","real-world classification","natural language hypotheses"],"falsifier":"Give the LLM evaluator a deliberately wrong hypothesis (for example, 'hallucinations happen only for blue objects') on one of these datasets and measure test accuracy; if accuracy stays high, the evaluator is supplying the task knowledge itself, which would break the inference that generation relies on priors.","tokens_in":16743,"feed_emoji":"🧠","tokens_out":8181,"duration_ms":73559,"temperature":0.7,"pith_summary":"On five real-world classification tasks spanning text, image, and image-text inputs, the paper tests whether LLM-generated natural-language hypotheses come from in-context demonstrations or from task-specific knowledge the model already has. Across three generation strategies and three LLMs, deleting the demonstrations produces nearly identical hypothesis quality and downstream classification accuracy. The paper interprets this as evidence that task-specific model prior dominates hypothesis generation in real-world inductive reasoning. This matters because data-driven hypothesis generation is supposed to feed scientific discovery; if the model priors are doing the work, the apparent discovery may be recycled pretraining knowledge rather than insight extracted from the data.","feed_headline":"Across five tasks, deleting examples barely dents LLM hypotheses","feed_subtitle":"Five real-world datasets and three LLMs show in-context demonstrations add almost nothing over pretrained priors.","key_machinery":"The central mechanism investigated is the task-specific model prior, operationalized as the hypotheses an LLM produces from task instructions alone, compared against hypotheses produced when demonstrations with ground-truth labels are added. The comparison runs through three baselines—input-output prompting, iterative refinement with ranking, and HypoGeniC-style update from mistakes—and is evaluated by hypothesis-based classification, LLM scoring and pairwise comparison, and human pairwise comparison. Removing demonstrations isolates the prior's contribution; flipping, randomizing, or reformatting labels tests whether demonstrations can override it.","core_discovery":"The paper's central claim is that in real-world inductive reasoning, LLM hypothesis generation is primarily driven by task-specific model prior, not by the provided input-label mappings. Concretely, for the best single hypothesis averaged over five datasets and three baselines, accuracy without demonstrations is 68.62% versus 68.56% with demonstrations; for multiple-hypothesis classification, it is 66.32% versus 65.74%. The invariance persists when demonstration labels are flipped or randomized, when the label format changes, and when hypotheses are scored by LLMs or humans. The paper therefore characterizes LLMs as zero-shot hypothesis generators on tasks where pretraining has supplied strong priors, with demonstrations acting as weak guidance that cannot override those priors.","pith_inferences":["The paper leaves implicit a caution for scientific-discovery workflows: when models generate hypotheses from familiar datasets, those hypotheses may recycle pretraining knowledge, so novelty should be tested against evidence that the hypothesis is not already in the training distribution.","One testable extension is to repeat the with/without-demonstrations comparison on a non-public dataset from a domain with no relevant prior; the paper's account predicts demonstrations would then matter, while a null result would point to a different mechanism.","Another extension is to probe the evaluator directly by supplying a deliberately wrong hypothesis; if test accuracy barely moves, the invariance would be located in the evaluation model's prior rather than in the generator.","The findings suggest prompt design may be better spent eliciting and constraining priors, for example by requesting several divergent hypotheses, than by adding more labeled examples."],"forward_implications":["In knowledge-rich real-world classification tasks, labeled demonstrations are not necessary for LLMs to propose usable hypotheses; zero-shot generation with task instructions alone yields comparable downstream accuracy.","Data still helps for hypothesis selection: iterative refinement with validation-set ranking consistently outperforms the other baselines, indicating that the useful role of data is choosing among hypotheses rather than shaping their content.","Flipped or random label demonstrations barely change predictions, so the hypotheses LLMs produce on familiar tasks reflect pretrained knowledge more than the supplied label mapping; example labels are weak steering signals.","The conclusion holds across GPT-4o, Qwen2-VL, and Gemini-1.5-pro and across single- and multiple-hypothesis evaluation, so it is not tied to one model or evaluation mode.","LLM-based scoring and human pairwise comparisons also fail to prefer hypotheses generated with demonstrations, and in some cases slightly prefer zero-shot hypotheses."],"supporting_citations":[{"why":"Defines what counts as a valid natural-language hypothesis, the object being studied throughout.","marker":"Zhong et al., 2022"},{"why":"Supplies the input-output prompting and iterative-refinement baselines and the LLM-as-hypothesis-proposer framing.","marker":"Qiu et al., 2023"},{"why":"Provides the HypoGeniC baseline and the truthful hotel review dataset.","marker":"Zhou et al., 2024"},{"why":"Provides the HypoGeniC update-from-mistakes mechanism and the hypothesis-based inference evaluation.","marker":"Liu et al., 2024"},{"why":"Supplies the unhealthy comments and funny Reddit datasets.","marker":"Zhong et al., 2023"},{"why":"Supplies the object-hallucination pattern induction dataset.","marker":"Li et al., 2023"},{"why":"Supplies the PneumoniaMNIST chest X-ray dataset.","marker":"Yang et al., 2023a"},{"why":"Frames the expectation that demonstrations convey the task distribution, which the paper's finding tests.","marker":"Min et al., 2022"},{"why":"Establishes in-context learning's reliance on labeled examples, the backdrop for the surprising invariance.","marker":"Brown, 2020"}],"fun_headline_variants":["LLM reasoning leans on pretrained priors, not examples","Removing examples barely hurts LLM inductive reasoning","Model priors trump in-context examples for LLM tasks","Demonstrations are optional: LLMs rely on priors","LLM hypotheses driven by priors, not provided examples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion assumes that the LLM used to apply a hypothesis to new examples judges the hypothesis's pattern rather than relying on its own task knowledge when labeling.","fun_headline_variants_meta":{"raw":{"variants":["LLM reasoning leans on pretrained priors, not examples","Removing examples barely hurts LLM inductive reasoning","Model priors trump in-context examples for LLM tasks","Demonstrations are optional: LLMs rely on priors","LLM hypotheses driven by priors, not provided examples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1494,"prompt_tokens":864,"completion_tokens":630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":548}},"tokens_in":480,"tokens_out":630,"duration_ms":5466,"temperature":1.0,"reasoning_tokens":548,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:55:40.899981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the LLM evaluator a deliberately wrong hypothesis (for example, 'hallucinations happen only for blue objects') on one of these datasets and measure test accuracy; if accuracy stays high, the evaluator is supplying the task knowledge itself, which would break the inference that generation relies on priors.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines what counts as a valid natural-language hypothesis, the object being studied throughout."}],"review_version":1}