Random Rule Forest (RRF): Interpretable and Manageable Ensembles of LLM-Generated Questions for Predicting Success from Unstructured Data

Aaron Ontoyin Yin; Ben Griffin; Diego Vidaurre; Fuat Alican; Joseph Ternasky; Ugur Koyluoglu; Yigit Ihlamur

REVIEW 4 major objections 6 minor 1 cited by

Random Rule Forest (RRF): Interpretable and Manageable Ensembles of LLM-Generated Questions for Predicting Success from Unstructured Data

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-generated yes/no questions, voted together, beat direct prompting at startup screening.

desk verdict The abstract promises a clinical-trial result that does not exist in the body; the startup results are plausible but under-supported, so the paper as submitted falls short of its claims. read the letter →

arxiv 2505.24622 v3 pith:EYNFW7UG submitted 2025-05-30 cs.AI cs.LG

Ben Griffin , Aaron Ontoyin Yin , Diego Vidaurre , Ugur Koyluoglu , Joseph Ternasky , Fuat Alican , Yigit Ihlamur This is my paper

classification cs.AIcs.LG

keywords randomruleforestLLM-generatedquestionsensemblelearninginterpretableAIstartupsuccesspredictionweaklearnerslow-base-rateclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Random Rule Forest (RRF) claims that a large language model can be used as a generator of simple YES/NO questions rather than as an end-to-end predictor, and that combining those questions with a plain unit-weight voting rule produces an interpretable predictor that beats direct prompting in low-base-rate tasks. The paper demonstrates this on early-stage startup screening: on 9,892 founder profiles, RRF reaches 13.1% precision versus a 1.9% base rate, a 6.9x improvement, and outperforms zero- and few-shot LLM baselines on F0.5. Adding a handful of expert-written questions raises precision to 15.3%. If right, this matters because high-stakes screening decisions, where errors are costly and outcomes must be auditable, can get transparent, editable scorecards without giving up predictive performance.

What carries the argument

Random Rule Forest (RRF) is an ensemble whose weak learners are natural-language YES/NO questions generated by an LLM rather than trained classifiers. The pipeline has three stages: (1) an LLM generates 250 candidate questions from balanced founder profiles; (2) a filtering stage uses sentence-embedding cosine similarity to remove semantically identical questions, discards questions with precision below chance (10%), removes predictively redundant questions via a Hamming-distance threshold, and ranks survivors by F0.5; (3) the top N questions vote, predicting success when at least T answer YES. The plain unit-weight vote is the load-bearing design choice: it keeps the model fully auditable, with all decisions traceable to plain-language green flags.

What would settle it

On a fixed set of founders and a fixed question pool, vary the model version, temperature, and prompt phrasing used for the answering step and measure per-question agreement; if the ensemble's top-N ranking or the reported F0.5 changes by more than the gap between RRF and the best baseline, the LLM-answer assumption is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that an LLM's ability to generate plain-language YES/NO questions can be turned into a strong, transparent predictor by treating each question as a weak learner and combining them with threshold voting. On the founder dataset, RRF filters hundreds of generated questions down to a ranked pool, keeps the top N questions, and predicts success when at least T of them answer YES, yielding a precision of 13.1% against a 1.9% base rate (6.9x) and an F0.5 of 0.124 that beats all zero- and few-shot prompting baselines tested across three LLM architectures. Replacing several LLM questions with expert-crafted ones raises precision to 15.3%. The paper also reports that on an established Phase I clinical-trial benchmark, RRF beats published baselines on PR-AUC and ROC-AUC, evidence that the mechanism transfers to another low-base-rate text-screening domain.

Load-bearing premise

The paper assumes that the LLM's YES/NO answers to each generated question are reliable and stable enough that a count of YES votes carries genuine predictive signal; if answer noise or prompt sensitivity dominates, the reported precision gains could be artifacts.

Editorial extensions

If this is right

The voting threshold and ensemble size give operators direct control over the precision-recall trade-off, so the same question pool can be tuned for precision, F1, F2, or F0.5 without retraining.
Experts can audit, edit, or add individual rules, and a small set of expert-crafted questions raises precision from 13.1% to 15.3%, showing human-LLM collaboration works inside the ensemble.
Decision traces reduce to a short list of YES/NO answers, so every prediction can be justified in natural language to stakeholders.
The pipeline transfers to a second low-base-rate domain (Phase I clinical-trial screening), where the paper reports superior PR-AUC and ROC-AUC over published baselines.
The method recovers classical weak-learner theory with conceptual diversity replacing algorithmic diversity, because each question encodes a different hypothesis about success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test implied by the paper is to measure answer stability across LLM versions and prompt wordings; high variance would require voting-based calibration or answer aggregation.
The approach should transfer to other low-base-rate text screening tasks, such as grant review, hiring, or underwriting, where explanations are obligatory, though geographies and label definitions would need local data.
The forward-split result (F0.5 = 0.267 versus 0.124 in nested cross-validation) hints at temporal generalization, but the authors flag the small positive cohort; a larger temporal holdout would settle whether the gain is real.
Unit-weight voting may be more robust than learned weights when positives are scarce, but that claim needs an explicit comparison against a logistic-regression meta-learner on the same question outputs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The first thing to know: the abstract claims RRF beats published baselines on an established Phase I clinical-trial benchmark with PR-AUC and ROC-AUC, but the body contains no clinical-trial experiment, no dataset, no metrics, and no baselines. This is not a small omission; it is the paper's headline contribution. As written, a reader can only conclude that the claim is either unsupported or refers to a version that was not submitted. That alone undermines the paper.

What the paper does well is the core RRF idea. Using an LLM to generate a large pool of yes/no questions, then filtering for semantic and predictive redundancy and combining via threshold voting, is simple, interpretable, and genuinely distinct from GPTree, Tree Prompting, and RAISE. The expert-in-the-loop variant is a sensible extension. The dataset is split into generation, filtering, and evaluation sets, which shows more care than many similar papers, and the ablation studies on similarity metrics and ranking criteria are a fair attempt to probe robustness.

The soft spots go beyond the missing clinical experiment. The main performance table has no error bars—we see point estimates for precision, recall, and F0.5 but no variance across the 10x10 CV folds that are mentioned later. The compute comparison is lopsided: RRF takes 67 sequential hours and about $21, while the cheapest baseline runs in 1.3 hours for $2.90. That is roughly a 50x cost gap, and it is not discussed as a trade-off. The dataset is selectively labeled, excluding companies that raised between $4M and $500M, which inflates the separation between success and failure and makes the 6.9x precision gain look larger than it would be in a realistic screening pool. No code or data are released, so none of this is reproducible. Finally, the method leans entirely on the LLM's yes/no answers being stable and meaningful, but the paper reports no test of inter-run consistency, prompt sensitivity, or human agreement on those answers. If the answering step is noisy in a way that correlates with the outcome, the whole ensemble signal could be an artifact.

This is a clever method in search of better evidence. The startup results are plausible but not convincing as presented, and the clinical-trial claim is what would have made it a strong paper. I would send it to review, because the RRF idea deserves a serious look and the flaws are fixable, but I would expect the referee report to demand removal of the unsupported claim, proper uncertainty quantification, matched-compute baselines, and at least a reproducibility statement.

Referee Report

4 major / 6 minor

Summary. The paper proposes Random Rule Forest (RRF), an ensemble method in which an LLM generates candidate yes/no questions about unstructured founder profiles; questions are filtered for semantic and predictive redundancy, ranked, and combined by unit-weight threshold voting. The model is evaluated on a proprietary dataset of 9,892 U.S. founders (1.9% success rate), reporting held-out precision of 13.1% (6.9x base rate), F0.5 of about 0.124 vs 0.088 for the best direct-prompting baseline, and an expert-in-the-loop variant at 15.3% precision. The supplied abstract additionally claims RRF outperforms published baselines on an established Phase I clinical-trial benchmark on PR-AUC and ROC-AUC, but no such experiment appears in the body. The paper also includes ablations over similarity metrics, thresholds, ranking criteria, and a temporal split.

Significance. The startup-screening result, if it survives additional validation, is a useful contribution: RRF is genuinely interpretable, requires only two tunable hyperparameters, and the pipeline separates question generation/filtering from evaluation, which is a sound design choice in a low-base-rate setting. The ablations (Section 5.5, Appendices H and K) and the explicit cost table are welcome transparency. However, the contribution is currently overstated: the clinical-trial result named in the abstract is absent, and the core mechanism (LLM yes/no answers) is not tested for reliability. The paper's value therefore depends on additional experiments that are within scope.

major comments (4)

[Abstract] The abstract supplied for this submission states that RRF 'outperforms published baselines on the threshold-independent metrics PR-AUC and ROC-AUC' on an established Phase I clinical-trial benchmark. The body text (Sections 1–7) and Appendices A–K contain no Phase I clinical-trial dataset, no PR-AUC or ROC-AUC results, and no comparison to published clinical-trial baselines. This is not a minor omission: the clinical-trial benchmark is presented as the second of two low-base-rate domains and supports the general claim in Section 6 that LLMs can serve as auditable feature generators for high-stakes decisions. The authors must either add the experiment or remove the claim from the abstract and discussion; the full-text abstract, which mentions only startup success, must be reconciled with the submitted abstract.
[§4.3, §5.3] The test-time prediction in §5.3, and the question scoring in §4.3, rely entirely on LLM yes/no answers to the generated questions, but no experiment assesses the reliability or stability of these answers. There is no inter-run consistency check (e.g., repeated sampling with different temperatures/seeds), no sensitivity to prompt wording or model version, and no human agreement study on a subsample. Because the ensemble's signal is mediated by this answering step, noisy or systematically biased answers could produce the reported precision gains even if the questions are informative. Please add such sensitivity analyses.
[Table 1] Table 1 reports each model's precision, recall, and F0.5 as single point estimates, while §5.5 reports that ablation results are averaged over 100 random seeds. The headline comparison (RRF F0.5 = 0.124 vs best baseline 0.088) is therefore presented without any uncertainty quantification. Please report confidence intervals or standard deviations across seeds/repeats, and, if possible, a statistical test of the difference, for the main results in Table 1 as well as the expert-in-the-loop comparison.
[Appendix J / §5.3] The cost comparison in Appendix J (Table 3) shows that RRF's sequential evaluation takes 67–72 hours versus 0.9–4.1 hours for the prompting baselines, roughly a 20–70x gap. The paper acknowledges this in passing but still frames the comparison as a practical improvement. To support the claim that RRF is a practical method, please add a cost-controlled comparison (e.g., RRF restricted to a similar inference budget, or cost-normalized performance) or explicitly discuss the trade-off in the main text.

minor comments (6)

[Title/Abstract] The supplied abstract mentions the Phase I clinical-trial benchmark and 'and Manageable' in the title, while the full-text title and abstract are startup-only; please unify the versions.
[Abstract, Table 1, Appendix H] F0.5 values are inconsistent: the submitted abstract says 0.121, Table 1 reports 0.124, and Appendix H reports 0.1233; please reconcile the reported numbers.
[Section 7] Section 7 states that expert-in-the-loop refinement achieves 'an 9× improvement,' but 15.3% precision against a 1.9% base rate is 8.05×, and the full-text abstract says 8×; correct the arithmetic and the typo.
[Section 3.1] Section 3.1 says the dataset 'covers nearly a third of this population'; 9,892/35,000 is 28%, so please either provide exact denominators or soften the claim.
[Appendix K] Appendix K reports F0.5 = 0.267 on the forward split with only 35 positive test cases; the text acknowledges instability, but the claim that the model 'generalises well across time' should be tempered or accompanied by bootstrap confidence intervals.
[Section 4.3] The filtering thresholds (semantic similarity 0.9, precision filter 10%, Hamming 0.15) are chosen without sensitivity analysis except for the Hamming threshold; please add a brief discussion or ablation for the fixed thresholds.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: RRF's question generation, filtering, and ensemble tuning use separate data splits with nested cross-validation; the reported 13.1% precision is an out-of-sample measurement. The abstract's Phase I clinical-trial PR-AUC/ROC-AUC claim is absent from the body, a serious evidence gap but not circularity.

full rationale

The core RRF pipeline is not circular: question generation uses a dedicated 500-founder set, question filtering uses a separate 500-founder set, and ensemble hyperparameters (N, T) are chosen by nested cross-validation on the 8,381-founder evaluation set, with test predictions made on held-out folds. The reported precision of 13.1% is an empirical out-of-sample measurement, not a fitted quantity renamed as a prediction. The only self-citation is the contextual Y Combinator (3.2%) and tier-1 VC (5.6%) rates from Mu et al. (2025), which shares authors with this paper; however, the paper explicitly labels these comparisons "illustrative only" in Section 5.1, footnote 3, so they are not load-bearing to the RRF derivation. No uniqueness theorem or ansatz is imported from prior work to force the model choice. Separately from circularity, the arXiv abstract claims RRF outperforms published baselines on a Phase I clinical-trial benchmark with PR-AUC and ROC-AUC, but no such experiment, dataset, or baseline appears anywhere in the full text; Sections 5 and Appendices A-K contain only startup experiments. This is a serious missing-support and evidence gap, but it is not a circularity. Accordingly, the circularity score is 1, indicating no significant circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claims depend on several hand-chosen thresholds and tuning hyperparameters (N, T) plus assumptions about LLM answer reliability, profile quality, and representativeness. No new entities are introduced.

free parameters (6)

Ensemble size N = not stated; grid-searched
Number of top-ranked questions included in the voting ensemble; tuned on the evaluation set via nested cross-validation (§4.4).
Voting threshold T = not stated; grid-searched
Minimum number of YES votes required for a positive prediction; tuned jointly with N on the evaluation set (§4.4).
Semantic similarity threshold = 0.9
Cosine similarity cutoff for removing near-duplicate questions; hand-chosen default (§4.2).
Hamming distance threshold = 0.15
Answer-pattern redundancy cutoff; selected as best in ablation (§5.5, Figure 2c).
Precision filter threshold = 0.10
Questions with precision below 10% on the filtering set are discarded; chosen to match the filtering set's success rate (§4.3).
Number of generated questions = 250
Candidate pool size; chosen conservatively based on a saturation analysis in Appendix B.

assumptions (6)

domain assumption The LLM answers generated yes/no questions reliably and consistently for new founder profiles
Used at §4.3 for scoring and at §5.3 for test-time voting; no human agreement or repeat-run consistency check is reported.
domain assumption Founder profiles contain sufficient, unbiased information available at the time of the investment decision
§3.1 claims profiles are built from Crunchbase/LinkedIn metadata; the paper also notes in §6.1 that profiles may omit key details or contain unverified claims.
domain assumption The curated evaluation set is representative of the screening population
§3.1 and Appendix F: founders raising between $4M and $500M are excluded, which changes the base rate and limits representativeness.
domain assumption Nested cross-validation on a fixed, pre-selected question pool gives an unbiased estimate of RRF performance
§4.1: question generation and filtering are performed once outside the CV folds; the paper argues this is sound because heuristics are not retrained.
domain assumption LLM-generated questions behave like classical weak learners (slightly better than chance, diverse)
§1 invokes ensemble theory; the filtering step enforces precision >10% on the filtering set, but diversity is only approximated via Hamming distance on response vectors.
domain assumption The relationship between founder characteristics and success is approximately stationary over time
Stated in Limitations §6.1; needed for the random-split cross-validation to be meaningful.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Random Rule Forest (RRF): Interpretable and Manageable Ensembles of LLM-Generated Questions for Predicting Success from Unstructured Data." pith.science (2026). https://pith.science/paper/EYNFW7UG

@misc{pith2026250524622,
  author       = {Pith},
  title        = {Pith review of: Random Rule Forest (RRF): Interpretable and Manageable Ensembles of LLM-Generated Questions for Predicting Success from Unstructured Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EYNFW7UG}},
  note         = {Machine review of arXiv:2505.24622}
}

read the original abstract

Many high-stakes screening tasks require predicting rare outcomes from unstructured text, where errors are costly and decisions must be auditable. We introduce Random Rule Forest (RRF), an interpretable ensemble that uses a large language model (LLM) not as an end-to-end predictor but as a generator of simple YES/NO questions. Each question acts as a weak learner, and their responses are combined by a plain unit-weight vote into an auditable ``green-flags'' scorecard: enough independent positive signals indicate a higher chance of success. We argue this deliberate simplicity is a robust default when positives are scarce and learned weights are hard to estimate. We evaluate RRF in two low-base-rate domains. On early-stage startup screening from founder profiles, RRF produces a transparent scorecard whose precision is several times the base rate (with light expert input raising it further) and, unlike direct prompting, its operating point can be controlled directly. On an established Phase~I clinical-trial benchmark, RRF outperforms published baselines on the threshold-independent metrics PR-AUC and ROC-AUC. Together these show that LLMs can serve as auditable feature generators for high-stakes text-based decisions, combining transparency with competitive predictive performance.

Figures

Figures reproduced from arXiv: 2505.24622 by the authors.

**Figure 1.** Schematic overview of the RRF pipeline. (a) LLM-guided question generation: An LLM is prompted with founder descriptions from the Question Generation Set to produce 250 candidate YES/NO questions. Semantically similar ones are removed using cosine similarity. (b) Filtering and ranking: Questions are scored on the Question Filtering Set, filtered for prediction redundancy, and ranked by accuracy. Gold bars indicate e… view at source ↗

**Figure 2.** Expert-in-the-loop comparison and ablation analysis. Panels (a)–(e) report mean F0.5 scores across 100 random seeds, each ablation varying one component of the RRF pipeline while holding others fixed. (a) Expert input. Adding expert-curated questions raises precision and F0.5 relative to LLM-only ensembles, although recall declines; expert-only ensembles underperform overall. (b) Similarity metric. Hamming distance … view at source ↗

**Figure 3.** Effect of question pool size on the number of distinct questions that survive semantic similarity filtering (threshold = 0.9). Beyond 200 generated questions, the number of unique heuristics begins to plateau, showing diminishing returns from further generation. C. Example Question Evaluation Prompt You are a VC analyst that evaluates startup founders. Your task is to assess whether the following question applies t… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: shows a rank-ordered view of every (N, T) configuration explored in the grid search. Each point is coloured by ensemble size N, revealing that the highest F0.5 scores concentrate within a narrow band of N values, while most settings yield much lower performance. 0 200…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

From Limited Data to Rare-event Prediction: LLM-powered Feature Engineering and Multi-model Learning in Venture Capital
cs.LG 2025-09 conditional novelty 4.0 of 10

An LLM-feature-driven ensemble predicts billion-dollar startup outcomes with 9.8X to 11.1X the precision of a random classifier, but the label and the model's intermediate target are both funding, so the result partly...

Reference graph

Works this paper leans on

20 extracted references · 18 canonical work pages · cited by 1 Pith paper

[1]

Has the founder ever worked in a role related to marketing?

work page
[2]

Is the founder actively engaged in commu- nity service or social impact initiatives?

work page
[3]

Did the founder have experience in the mar- keting field?

work page
[4]

Did the founder complete their education in the last 10 years?

work page
[5]

As a result, they were excluded from the final ensemble

Has the founder been involved in any non- profit organizations? Many of these low-performing questions were likely influ- enced by patterns present in the small training subset of 20 founders fed to the LLM, and their narrow focus failed to generalize to the broader validation set. As a result, they were excluded from the final ensemble. 10 Random Rule Fo...

work page arXiv 2019
[6]

Is the founder’s educational background rel- evant to their startup’s industry?

work page
[7]

Has the founder successfully led a startup to a profitable exit in the past?

work page
[8]

Has the founder held a senior leadership role in a startup that achieved significant revenue growth?

work page

Show all 20 references

[9]

Does the founder have a proven track record of mentoring other entrepreneurs in their field?
[10]

Compute Resources and Cost Table 3 details the sequential runtime and API cost for evalu- ating 8,500 founder profiles across different models

Is the founder’s educational background di- rectly relevant to their startup’s technology? J. Compute Resources and Cost Table 3 details the sequential runtime and API cost for evalu- ating 8,500 founder profiles across different models. While results are shown for transparenc...

2025
[11]

Has the founder been involved in any venture capital or private equity firms?
[12]

Has the founder ever worked in a role related to product management?
[13]

Is the founder’s university ranked among the top 50 globally?
[14]

Did the founder have experience in the biotechnology sector?
[15]

Has the founder successfully raised funding for their startup from reputable investors?
[2005]

Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schu- bert, Michael Petrov, and Chris Olah

Diversity creation methods: a survey and categori- sation.Information fusion6, 1 (2005), 5–20. Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schu- bert, Michael Petrov, and Chris Olah. 2020. Curve detec- tors.Distill5, 6 (2020), e00024–003. Dominik Dellermann, Nikolaus Lipu...

2005 arXiv
[2014]

Jan Ruben Zilke, Eneldo Loza Menc´ıa, and Frederik Janssen

Models and tools for technology start-up compa- nies success analysis.Economic Alternatives3 (2014), 15–24. Jan Ruben Zilke, Eneldo Loza Menc´ıa, and Frederik Janssen

2014
[2016]

InDiscovery Science: 19th International Conference, DS 2016, Bari, Italy, October 19–21, 2016, Proceedings 19

Deepred–rule extraction from deep neural networks. InDiscovery Science: 19th International Conference, DS 2016, Bari, Italy, October 19–21, 2016, Proceedings 19. Springer, 457–473. 9 Random Rule Forest (RRF): Interpretable Ensembles of LLM-Generated Questions for Predicting St...

2016
[2018]

InProceed- ings of the 27th ACM international conference on infor- mation and knowledge management

Web-based startup success prediction. InProceed- ings of the 27th ACM international conference on infor- mation and knowledge management. 2283–2291. Sichao Xiong and Yigit Ihlamur. 2023. Founder-GPT: Self- play to evaluate the Founder-Idea fit.arXiv preprint arXiv:2312.12037(2...

2023 arXiv
[2025]

John X Morris, Chandan Singh, Alexander M Rush, Jian- feng Gao, and Yuntian Deng

A Fused Large Language Model for Predicting Startup Success.European Journal of Operational Re- search322 (2025), 198–214. John X Morris, Chandan Singh, Alexander M Rush, Jian- feng Gao, and Yuntian Deng. 2023. Tree prompting: effi- cient task adaptation without fine-tuning.ar...

2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Has the founder ever worked in a role related to marketing?

work page

[2] [2]

Is the founder actively engaged in commu- nity service or social impact initiatives?

work page

[3] [3]

Did the founder have experience in the mar- keting field?

work page

[4] [4]

Did the founder complete their education in the last 10 years?

work page

[5] [5]

As a result, they were excluded from the final ensemble

Has the founder been involved in any non- profit organizations? Many of these low-performing questions were likely influ- enced by patterns present in the small training subset of 20 founders fed to the LLM, and their narrow focus failed to generalize to the broader validation set. As a result, they were excluded from the final ensemble. 10 Random Rule Fo...

work page arXiv 2019

[6] [6]

Is the founder’s educational background rel- evant to their startup’s industry?

work page

[7] [7]

Has the founder successfully led a startup to a profitable exit in the past?

work page

[8] [8]

Has the founder held a senior leadership role in a startup that achieved significant revenue growth?

work page