{"id":"17f9b55d-637c-4255-bc64-d30fbf5485e1","arxiv_id":"2505.21427","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper claims an LLM with an inductively learned prompt policy achieves 20x precision over random chance in startup success prediction, but test-set leakage and weak baselines invalidate the headline claim.","lead":"An LLM-based system using iteratively refined natural language policies is claimed to predict startup success with 40.5% precision, over 20 times the base rate. The study's experimental design contaminates the test set and compares against a strawman baseline, so the headline result is not credible as stated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.3 says the 100/1000 test set was used for policy comparison, so the final precision is an in-sample selection result, not an out-of-sample estimate; this invalidates the 20x claim unless a fresh holdout confirms it.","rationale":"The reader identified the same load-bearing weakness: the 100-success/1000-failure set was used for policy comparison, so the headline precision may be an in-sample selection result. The paper's own Section 3.3 is explicit, and Section 6.1's claim that scoring used the training set does not resolve the contradiction. No separate frozen holdout is described for the final o3 policy, so even the Table 5 result is vulnerable if Table 4 informed the choice of policy. This is a correctness risk, not merely a difference in baselines or evaluation philosophy; it directly undermines the paper's strongest quantitative claim. A single fresh holdout evaluation would settle the question. The reader's REJECT verdict remains appropriate, and no verdict adjustment is needed.","tokens_in":8203,"tokens_out":3700,"duration_ms":43077,"concrete_test":"Freeze the exact final policy from Appendix A.2. Construct a fresh held-out set of 40 successful and 2,000 failed founders from the same population and extraction pipeline, none of whose members appear in any set used during policy development or selection. Run the final policy once on this set, without any further policy edits, and compute precision with a bootstrap confidence interval. If the precision is materially below 0.40 (for example, in the 0.08-0.17 range matching the vanilla baselines or Table 4's weaker rows), the 20x claim is an artifact of test-set-contaminated selection. If it stays near 0.40, the concern is weakened and the central claim would gain genuine out-of-sample support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the final policy achieves about 40% precision on realistic test sets, a 20x lift over random chance. For this to be an out-of-sample estimate, the 100/1000 and 40/2000 test sets must not have informed policy selection. Section 3.3 states the opposite for the former: the 100-success/1000-failure configuration was 'used for preliminary assessment and comparison across different policies.' Section 5.2 and 5.3 then report the 'best policy's' precision on exactly this type of set, with no statement that the final policy was frozen before any comparison. Section 6.1 says policy scoring used the training set, but that does not neutralize the explicit language in Section 3.3; it creates an internal inconsistency about which data influenced the choice of the final policy. If the best policy was selected by comparing candidates on the 100/1000 set, the precision values in Tables 2 and 3 are in-sample numbers, and their average of 0.467 is an artifact of selection. The same circularity threatens Section 5.4/5.5: if the o3-generated policy was chosen after inspecting Table 4, then Table 5's mean precision of 0.405 is not a fresh evaluation. The paper never describes a pre-registered, untouched holdout, so the strongest claim lacks a valid out-of-sample basis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Policy Induction, a framework in which a natural-language decision policy is embedded in an LLM prompt and iteratively refined through an in-context learning loop using precision-based scoring on a training set. The authors evaluate the approach on founder-success prediction: they report vanilla LLM baselines, policy-guided results on 100-success/1000-failure test sets, evaluations on four 40-success/2000-failure sets, and a final o3-generated policy with a mean precision of 40.5%, which they translate into a 20x lift over random precision and a 7.1x improvement over a claimed top-tier VC success rate. The central claim is that this precision is an out-of-sample estimate of a deployable policy's performance.","tokens_in":1457,"tokens_out":1537,"duration_ms":53209,"significance":"If the reported precision were a valid out-of-sample estimate, the paper would be a useful contribution: the method is transparent, data-efficient, cheap, and does not require gradient-based fine-tuning, and the authors make a genuine effort to separate model logic from model weights. The interpretability of the policy and the explicit emphasis on human auditability are real strengths. However, the paper's headline numbers currently rest on test sets that the manuscript itself says were used for policy selection, and the comparison baselines are not commensurate with the claimed benchmarks. Once the evaluation is placed on a proper held-out basis, the empirical contribution may be more modest, so the significance depends on the outcome of a re-evaluation.","major_comments":[{"comment":"Section 3.3 states that the 100-success/1000-failure test configuration was \"used for preliminary assessment and comparison across different policies,\" yet Sections 5.2 and 5.3 present the precision of the \"best-performing policy\" on exactly this configuration, with no statement that the policy was frozen before any comparison. The scoring on the training set described in Section 6.1 does not resolve this issue, because the test set itself is acknowledged as a device for comparing policies. Consequently, the mean precision of 0.467 in Table 3 is an in-sample selection result rather than an out-of-sample estimate, and the 20x claim in Section 5.5 and the Conclusions does not currently have a valid empirical basis. The authors should either evaluate the final policy on a fresh, untouched holdout or explicitly present these numbers as development-set results and remove the out-of-sample claims.","section":"3.3, 5.2, 5.3"},{"comment":"The four 40-success/2000-failure test sets are used in Table 4 to evaluate the best GPT-4o-mini policy, and Table 5 then reports an o3-generated policy on the same four sets. The paper does not state whether the o3 policy was finalized before inspecting Table 4. If the o3 policy was selected or further tuned after seeing these results, the mean precision of 0.405 in Table 5 is also a selection artifact. Please clarify the temporal ordering of policy generation and evaluation, and, ideally, add an additional untouched test set to support the 20x claim.","section":"5.4, 5.5"},{"comment":"The headline comparisons are not apples-to-apples. Random precision is determined by the composition of the test set, so a \"20x lift over random precision\" is largely a statement about the chosen base rate rather than a comparison against a predictive baseline. The 5.6% figure attributed to top-tier VC firms is not defined or sourced as a precision measure, and the paper provides no evidence that VC firms evaluate the same population, labels, and time horizon used here. Please replace these comparisons with a trained classifier or another predictive baseline on the same features, and substantially temper the claims about beating top-tier VCs.","section":"5.1, 7"},{"comment":"No uncertainty intervals are reported. Table 3 shows precision ranging from 0.205 to 0.652 across subsets, Table 4 from 0.068 to 0.308, and Table 5 from 0.250 to 0.600, yet the paper reports only means and draws conclusions about robustness and practical utility. Please report per-subset confidence intervals or a variance estimate, and discuss how selection effects and the small number of success cases (40 or 100 per set) affect the stability of the means.","section":"5.3, 5.4, 5.5"}],"minor_comments":[{"comment":"The claim that Test Sets 6 and 7 are \"inherently weaker\" is supported only by vanilla GPT-4o runs, not by the same model used in the policy evaluation; please run the vanilla comparison with GPT-4o-mini or qualify the claim.","section":"5.3, A.3"},{"comment":"\"raised more than$500M\" is missing a space, and the wording of the success definition is awkward; please copy-edit this passage.","section":"3.1"},{"comment":"Footnote 1 contains the typo \"outlier1 rates,\" and the use of \"outlier\" as a synonym for \"unicorn\" should be defined at first use.","section":"5.4"},{"comment":"The paper never clarifies whether the \"standard test set\" used for Tables 1 and 2 is one of the eight subsets in Table 3 or a separate set; this relationship should be stated explicitly.","section":"3.3, 5.2"},{"comment":"Table 2 reports a single run each for the initial and best policies; a small number of repetitions would help assess the variability of the improvement.","section":"5.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central evaluation is undermined by the explicit statement in Section 3.3 that the 100/1000 test configuration was used for policy comparison, while the same configuration underlies the reported precision of the best policy. This is a fixable problem if the authors provide a fresh holdout evaluation and recalibrate the claims; however, the current version overstates both the out-of-sample validity and the comparison to top-tier VC performance. I would not recommend acceptance without those changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim is unsupported because the test sets did the policy selection. Section 3.3 says the 100/1000 set was used for \"preliminary assessment and comparison across different policies,\" and Section 5.2 then reports the \"best policy's\" precision on that same configuration. Section 5.5 appears to have selected o3 after looking at Table 4, so the 40% average on the realistic sets is a selected number, not a fresh evaluation. The stress-test note is correct.\n\nWhat is actually new: the iterative in-context policy induction loop—generate a plain-text policy from few-shot examples, update it through a precision-scored loop, and optionally add reflections or expert edits. That is a concrete, cheap, transparent way to adapt LLMs to a domain with little data and a real need for auditability. The appendix gives an 18-rule policy that a domain expert could actually argue with. That is a genuine strength.\n\nThe paper does some things right: it explicitly strips out company and founder names to reduce contamination, it reports eight test subsets to show variability, and the conclusions section honestly flags nondeterminism, prompt sensitivity, and possible latent exposure. That is more honest than many papers in this space.\n\nThe soft spots are real and they land on the central claim. The baseline is the base rate, not a competitive model: comparing to 1.9% random precision is a low bar. Precision varies from 0.6 to 0.07 across the four realistic sets in Table 4, so the mean is unstable and there are no confidence intervals. The paper says the training set was only used for scoring in Section 6.1, but that does not neutralize the explicit statement in Section 3.3. So the 20x claim is not out-of-sample.\n\nStill, the problem is fixable with a clean pre-registered holdout. The method itself is not broken; the evaluation is. I would not cite this paper for the performance numbers, but I would point to it as an interesting design pattern for explainable LLM policies.\n\nWho is this for? Researchers working on interpretable LLM decision support, and anyone thinking about evaluation design for ICL-style methods. It deserves a serious referee—the idea is worth testing—but the revision bar is high: a fresh holdout, a competitive baseline like GPT-4o with an explicit prompt, and error bars.\n\nRecommendation: send to peer review with the expectation of major revision; desk rejection would be defensible but would lose a potentially useful method.","headline":"The policy-induction loop is a real idea, but the evaluation circularity kills the headline 20x claim; the method might survive a clean holdout.","tokens_in":8998,"tokens_out":2443,"would_cite":false,"duration_ms":26283,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a large language model carrying a plain-text policy in its prompt can predict startup success with about 40% precision on realistic test sets, a 20-fold gain over random chance.","keywords":["startup success prediction","in-context learning","large language models","explainable artificial intelligence","policy induction","venture capital","precision","memory-augmented"],"falsifier":"Run the entire policy-induction pipeline on a fresh, pre-registered test set that is never touched during policy selection, using the same 40-success/2,000-failure construction, and check whether mean precision remains near 0.40 rather than dropping toward the 0.17 average of the first realistic-set runs (Table 4) or toward the vanilla o3 baseline of 0.229.","tokens_in":8015,"feed_emoji":"📈","tokens_out":7422,"duration_ms":67556,"temperature":0.7,"pith_summary":"The paper tries to show that a large language model can learn to identify successful startup founders by reading a short, human-readable policy that is embedded in its prompt and iteratively refined from a small labeled set of examples. The authors report that the best policy, trained on 120 successful and 120 failed founders with no gradient-based updates, reaches roughly 40% precision on test sets that mimic the real-world rarity of startup success, a more than 20-fold improvement over random chance. The value of the claim, if true, is that high-stakes investment screening can be made both data-efficient and auditable: the policy is plain text, so experts can see, edit, and port the decision logic to other domains. The method also suggests that small, cheap models can match or approach far larger models when guided by a good induced policy.","feed_headline":"Plain-text policy predicts startup success at 40% precision","feed_subtitle":"With 120 examples and no fine-tuning, an editable written policy beats random chance by 20x.","key_machinery":"The central object is the natural-language policy: a structured list of heuristics, capped at 20 lines, that is inserted into the LLM prompt and acts as the model's symbolic memory. The mechanism is the in-context learning loop: for each training example, the model is asked to summarize the case into an updated policy, candidate versions are scored by precision on the training set, and the best version becomes the next policy. Parallel and sequential update strategies, plus optional reflection sentences and expert edits, feed into this loop. The policy does two jobs: it is the only learned representation, so no weights change, and it is the explanation, since the rules are directly readable.","core_discovery":"The authors' central claim is that a decision policy expressed in natural language and carried in the LLM prompt is a sufficient mechanism for learning to predict startup success. Starting from 20 success and 20 failure examples, the system prompts an LLM to write a policy, then runs an in-context learning loop: each new training example is summarized into a revised policy, old and new versions are scored by their precision on the training set, and the better version is kept. After four rounds of refinement with 120 successes and 120 failures, the final policy, generated with the o3 model but evaluated with GPT-4o-mini, achieves a mean precision of 0.405 across four test sets with 40 successes and 2,000 failures each, about 20 times the random baseline of 1.96% and 7.1 times the typical 5.6% success rate attributed to top-tier venture capital firms. The stronger model is used only for policy writing; all scoring and inference stay on the cheap model.","pith_inferences":["The four realistic test sets in Table 5 are fixed draws, not a random sample of all possible 40/2000 sets; the true out-of-sample precision could be lower if those sets happen to be easier than average, especially since the same pipeline was used to choose among policies.","The policy's rules read like a checklist of diligence signals; an obvious extension is to test whether the induced rules survive when applied to a completely different cohort (e.g., non-US or pre-2010 startups), which the paper does not do.","The claimed 7.1x advantage over top-tier VC precision depends on the external estimate of 5.6% VC success, which is not derived in the paper; treating that number as a moving baseline would change the comparison.","Because policy selection used training-set precision (Section 6.1), and the paper says this matched validation-set trends, an independent validation-set check could confirm whether the training-set score is a safe proxy or an optimistic one."],"forward_implications":["A policy learned from 120 positive and 120 negative examples can reach roughly 40% precision, implying the approach is usable when labeled data are scarce.","Because the policy is plain text, the same pipeline can be applied to other screening tasks, such as grant evaluation or academic hiring, without retraining the model.","Using a stronger LLM only for policy generation while keeping inference on a cheap model gives a large precision gain, suggesting compute can be concentrated at the policy-writing stage.","On imbalanced test sets matching the real-world 1.96% success rate, precision remains well above random, suggesting the method is not just fitting the balanced training distribution.","The mean F0.5 score of 0.294 on realistic sets indicates the method prioritizes precision over recall, which suits investment settings where false positives are costly."],"supporting_citations":[{"why":"Survey of in-context learning; supplies the paradigm the method builds on.","marker":"Dong et al., 2023"},{"why":"Memory-augmented neural networks; conceptual basis for using the policy as external symbolic memory.","marker":"Graves et al., 2016"},{"why":"Taxonomy of data contamination; motivates removing founder and company names from the inputs so the model cannot memorize known successes.","marker":"Palavalli et al., 2024"},{"why":"Prior LLM-based venture capital decision approach (GPTree) that this work positions itself against and complements.","marker":"Xiong et al., 2024"}],"fun_headline_variants":["LLM policy from 120 examples hits 40% startup precision","Editable text policy predicts startups 20x better than random","No fine-tuning: prompt policy beats top VCs by 7x","Plain English policy yields 40% startup prediction precision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the realistic test sets were held out from policy selection, and Section 3.3 says the 100-success/1,000-failure set was used for preliminary assessment across policies, so the reported 40% precision may overstate true out-of-sample performance if that set influenced the final choice.","fun_headline_variants_meta":{"raw":{"variants":["LLM policy from 120 examples hits 40% startup precision","Editable text policy predicts startups 20x better than random","No fine-tuning: prompt policy beats top VCs by 7x","Plain English policy yields 40% startup prediction precision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000398,"raw_usage":{"total_tokens":2086,"prompt_tokens":955,"completion_tokens":1131,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1059}},"tokens_in":571,"tokens_out":1131,"duration_ms":9919,"temperature":1.0,"reasoning_tokens":1059,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:28:41.135497+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the entire policy-induction pipeline on a fresh, pre-registered test set that is never touched during policy selection, using the same 40-success/2,000-failure construction, and check whether mean precision remains near 0.40 rather than dropping toward the 0.17 average of the first realistic-set runs (Table 4) or toward the vanilla o3 baseline of 0.229.","supporting_citations":[],"review_version":1}