{"id":"b61d8953-5484-4ab7-bc42-ba4aae595236","arxiv_id":"2501.03892","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LEAP, an LLM-based library, automatically selects ML functions and writes SQL-like code to answer 92% of 120 social science queries over unstructured data on the first attempt, and 100% within three attempts.","lead":"This paper builds LEAP, a library that lets researchers ask social science questions in plain language and get answers computed from unstructured data such as tweets or documents. It also releases QUIET-ML, a benchmark of 120 such questions, where LEAP succeeds on 92 percent at first try and all questions within three tries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"QUIET-ML's ground-truth construction is undocumented, and the paper's own cost figures make human-generated gold implausible; the 100% pass@3 and 92% pass@1 numbers may therefore measure LEAP's fidelity to its own ML functions rather than substantive answer correctness.","rationale":"The paper's central claim is empirical—92% pass@1, 100% pass@3, $1.06 per query on QUIET-ML—so the load-bearing condition is that QUIET-ML's gold answers are independently produced and substantively correct. Section 2 never says how they were made, and the paper's own economics rule out human labeling at scale: 120 queries × ~22,323 records × 2.0 annotations ≈ 5.4M labels, which the authors themselves price at $1,736–$2,266 per query in the §5.2 cost comparison. The plausible alternative is that gold tables were generated by the same published ML models that LEAP internally supports, which would make pass@k a measure of orchestration fidelity—a legitimate but much narrower claim than 'ground-truth answers.' The vague-query subset shows this most sharply: a vague query like Q2 has no determinate gold until disambiguated, and the paper's success criterion ties that gold to LEAP's own recommended alternative. The §4.2.1 few-shot exemplar carrying 'ground-truth function selections' adds a residual leakage risk: the paper does not state the exemplar is disjoint from the 120 test queries. I still credit the paper's reproducible artifact, component ablations, and transparent cost breakdown, all of which survive this critique; the concern is about the framing of the headline numbers, not the engineering. The reader's CONDITIONAL verdict already targets this; I would keep it, tightening the condition to require disclosure of the gold-generation scripts, an independent validation subsample (e.g., human labels on a small random subset of records for a few queries), and confirmation that the §4.2.1 exemplar is not in QUIET-ML.","tokens_in":23099,"tokens_out":16240,"duration_ms":149709,"concrete_test":"Inspect the released repository (github.com/uiuc-kang-lab/leap) for the QUIET-ML construction scripts and trace how gold answers were generated for representative queries Q2, Q3, and Q11. If the scripts produce gold tables by invoking the same ML functions LEAP internally supports (f_emotion, f_dw/f_p/i/f_dw-type, f_trigger), with no human-validation subsample and no independent external labels, then the benchmark evidences orchestration fidelity only, and the paper's 'ground-truth answers' claim (Section 2) is unsupported, requiring re-scoped claims. If the gold was instead produced by human annotation or by independent published model outputs with documented provenance, the concern fails and the conditional can be relaxed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim—92% pass@1, 100% pass@3 on all 120 QUIET-ML queries (Section 5.2)—rests entirely on QUIET-ML's 'ground-truth query results' (Section 2), yet the paper never documents how those reference answers were produced. Internal evidence makes independent human or third-party construction implausible: 120 queries × ~22,323 records × 2.0 annotations ≈ 5.4M labels, which the paper itself prices at $1,736–$2,266 per query (§5.2). The only feasible route is automated annotation, i.e., running the same published ML models that constitute LEAP's 'internally supported functions.' If so, each gold table is the output of the very functions LEAP's LLM is asked to select, so pass@k measures whether LEAP replays a reference pipeline (function chain, parameters, SQL), not whether answers are semantically valid. For the 33 vague queries the circularity is tighter: a vague query such as Q2 ('Is the public mood correlated with...') has no well-defined answer until disambiguated, and the success criterion is that one of LEAP's recommended alternatives 'yields a result that matches the ground truth' (§5.2); if Q2's gold is the emotion distribution, that distribution is f_emotion's output—again a LEAP internal. A related risk: the table-generation prompt embeds a worked example with 'ground-truth function selections' (§4.2.1), and the paper never states that this exemplar is disjoint from the 120 test queries. The vague-query comparison (96.97% vs ≤41% baselines, Figure 6) may also be asymmetric: baselines must answer an ill-posed question directly while LEAP is allowed to substitute a disambiguated query whose gold the authors chose. The orchestration contribution is real and survives this critique; the 'ground-truth answers / reliable performance' framing (§1, §2, §7) does not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces QUIET-ML, a new benchmark of 120 natural-language social science queries over unstructured data (text, PDFs, videos), together with claimed ground-truth answers. It also proposes LEAP, an LLM-powered pipeline that first uses a forward planning filter to detect vague queries and suggest disambiguated alternatives, then selects and applies ML functions (from a function tree with doubly linked dependency lists) to extend the data into an annotated table, and finally generates and executes code (mostly SQL) to answer the query. The paper reports 92% pass@1, 100% pass@3, and a cost of $1.06 per query on QUIET-ML, with ablations showing the importance of each system component.","tokens_in":23453,"tokens_out":6495,"duration_ms":56488,"significance":"If the performance and cost claims are supported by a sound, independent benchmark, this is a significant step toward making ML-based analysis of unstructured data accessible to social scientists. The paper's strengths are its broad benchmark coverage (9 domains, 68 sources, 33 vague queries), its modular system design, the systematic evaluation with 5 runs per query and clear pass@k definitions, the thorough ablations (forward planning filter, doubly linked lists, function tree), and the public artifact. However, the benchmark's ground-truth construction is not documented, and the evaluation metrics for vague queries are defined in a way that risks circularity. These issues must be resolved before the headline numbers can be trusted.","major_comments":[{"comment":"The paper states that QUIET-ML \"includes the ground-truth query results on the provided unstructured data\" (Section 2) but never documents how these reference answers were produced. Given the average data size of 22,323 records and an average of 2.0 annotations per record (Section 2), manual construction at the paper's own cost estimates of $1,736–$2,266 per query (Section 5.2) would be prohibitively expensive, implying that the gold answers were generated automatically, most plausibly by the same or similar ML models that constitute LEAP's \"internally supported functions.\" If that is the case, the reported 92% pass@1 and 100% pass@3 measure whether LEAP can replay a reference pipeline (function chain, parameters, SQL) rather than whether the answers are semantically valid. The authors must report the ground-truth generation procedure, including the models, parameters, and any manual verification; without this, the central performance claim is not independently verifiable.","section":"Section 2, Section 5.2"},{"comment":"For the 33 vague queries, a run is scored as successful if \"LEAP correctly rejects the query and recommends alternatives, one of which yields a result that matches the ground truth.\" Since a vague query such as Q2 (\"Is the public mood correlated with...\") has no unique answer until disambiguated, the ground truth itself must encode a specific disambiguation. If that disambiguation was generated by one of LEAP's internally supported ML functions (e.g., the emotion distribution), then the evaluation only verifies that LEAP can propose the particular alternative that was arbitrarily selected as gold. The paper should specify how the ground-truth disambiguation for each vague query was chosen, report the set of acceptable alternative queries, and state whether success requires the first recommendation or allows any recommendation.","section":"Section 5.2"},{"comment":"The table generation prompt includes a worked example that states the \"ground-truth function selections\" for the query \"I want to count the number of positive paragraphs in the PDF document,\" including the exact function chain (OCR, paragraph separator, sentiment analyzer, stopper) and parameter selections. The paper does not state whether this exemplar is one of the 120 QUIET-ML test queries or whether the test set contains queries with the same function chain. If the exemplar is not disjoint from the test set, the prompt leaks the correct answer for those queries and inflates pass@k. Please disclose the exemplar and confirm its disjointness from the test queries, or remove it from the evaluation.","section":"Section 4.2.1"}],"minor_comments":[{"comment":"The label \"LogicBeam\" in the horizontal bar chart should be \"LogicalBeam\" to match the text and reference [8].","section":"Figure 6"},{"comment":"The sentence \"LEAP achieves an accuracy of 93.3% on the 33 vague queries when also provided with unstructured data\" is confusing because the immediately preceding comparison uses structured tables; please clarify which numbers correspond to which setting in the main results.","section":"Section 5.2"},{"comment":"The forward planning filter's third case mentions \"generating an alternative query list Q based on F, D, and q,\" but the success criterion in Section 5.2 requires that one of the recommended alternatives yields a result that matches the ground truth; please state explicitly whether the alternative list is scored on the annotated table or on the original unstructured data.","section":"Section 4.1"},{"comment":"The statement that \"over half (61) of them require executing two or more ML models\" is technically 61/120 = 50.8%; consider writing \"61 (50.8%)\" for precision.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The key risk is the undocumented ground truth; I recommend that the editor require full disclosure of QUIET-ML's construction before considering acceptance. The authors should also clarify the relationship between the prompt exemplar in Section 4.2.1 and the test set. The paper is otherwise within scope for a database conference, given its focus on query processing over unstructured data, but the evaluation's foundation needs to be made explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: LEAP is a genuinely useful piece of systems work, but the evaluation's foundation is shaky. The 92% pass@1 / 100% pass@3 numbers only mean something if QUIET-ML's ground truth is trustworthy, and the paper never says how it was produced.\n\nWhat's new and good: The idea of wrapping ML functions behind a function tree, with doubly linked lists for dependencies and alias checks to avoid redundant executions, is a real engineering contribution. The vague-query filter is a nice touch. The ablation studies are the strongest part: removing the function tree drops pass@1 from 92% to 16%, removing doubly linked lists drops it from 87.7% to 20%, and the forward planning filter doubles accuracy. Those are clear, quantitative demonstrations that the components matter. The cost analysis is also transparent and reasonable.\n\nThe soft spots, in order of severity. First, QUIET-ML's ground truth construction is undocumented. Section 2 just says the dataset includes the ground-truth query results. But the paper's own numbers make independent human annotation implausible: 120 queries × ~22,323 records × 2.0 annotations ≈ 5.4 million labels, which at the paper's own pricing would cost somewhere in the mid-six figures. The only realistic way those gold tables were produced is by running the same ML functions LEAP is designed to select. If so, the benchmark measures whether LEAP can replay a reference pipeline, not whether the answers are semantically correct.\n\nSecond, the vague-query evaluation is asymmetric. LEAP succeeds if it rejects the query and suggests any alternative that matches ground truth; baselines have to answer the ill-posed query directly. That explains the 96.97% vs. 41% gap but makes the comparison less informative. Third, the table-generation prompt embeds a worked example with ground-truth function selections, and the paper never states that this exemplar is disjoint from the 120 test queries.\n\nNone of this kills the paper. The orchestration contribution is real and the ablations are solid. But the performance claims need re-framing as \"fidelity to an internally defined pipeline\" until the ground truth is properly documented and independently validated.\n\nRecommendation: send it to peer review, but the reviewers should insist on a major revision. The dataset construction needs to be specified, ideally with a human-annotated subset, and the evaluation claims need to be scaled back accordingly.","headline":"The engineering is real and the ablations are solid, but the headline performance numbers rest on a benchmark whose ground truth is never explained, and the paper's own cost arithmetic makes independent human annotation implausible.","tokens_in":24020,"tokens_out":1681,"would_cite":false,"duration_ms":18239,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LEAP, an LLM-powered library, turns natural-language social science queries over unstructured data into ML annotations and SQL analysis, achieving 100% pass@3 and 92% pass@1 on a new 120-query benchmark at an average…","keywords":["natural language to SQL","unstructured data","machine learning annotation","social science queries","vague query handling","LLM function calling","benchmark dataset","end-to-end query cost"],"falsifier":"A direct check: take a sample of QUIET-ML queries, compute answers with independently trained models or human annotators, and compare those answers to LEAP's outputs; if agreement is substantially below the reported 92% pass@1, the benchmark would be measuring reuse of the library's own functions rather than correct social-science answers.","tokens_in":22906,"feed_emoji":"📊","tokens_out":5680,"duration_ms":49721,"temperature":0.7,"pith_summary":"Social scientists increasingly want to ask research questions directly over raw text, video, and PDFs—questions like which posts are persuasive or whether public mood predicts economic indicators—but the semantic information they need is not stored in the data and must be extracted by machine-learning models. The paper introduces QUIET-ML, a benchmark of 120 real social-science queries with ground-truth answers, and LEAP, an end-to-end library that takes a natural-language query and raw data as input and returns a result. LEAP first decides whether the query is vague, plans a chain of ML functions, annotates the data into a structured table, then generates and executes SQL-style code over that table. The paper reports 92% pass@1 and 100% pass@3 on QUIET-ML at an average end-to-end cost of $1.06 per query, of which code generation costs $0.02. If true, this would let domain experts get reliable answers to social-science questions on unstructured data without manually selecting models or writing analysis code.","feed_headline":"LLM pipeline answers vague social science queries 92% of the time","feed_subtitle":"It selects ML models, annotates raw posts and PDFs, then runs the query—for about $1.06 per question.","key_machinery":"The load-bearing mechanism is a four-part LLM pipeline. The forward planning filter uses chain-of-thought prompting to decide whether a query is vague and to produce a planned ML function chain; it also returns reformulated queries for vague cases. The stage selector runs table generation, code generation, execution, and display as function calls, tracking progress. Within table generation, the supported ML function list is organized as a function tree so only one leaf node's functions are passed to the LLM at a time—this fits the token limit and cuts cost by 55%. Doubly linked lists connect functions with mutual output dependencies (for example, emotion classification output feeds emotion-trigger identification), raising accuracy on implicit-dependency queries from 20% to 87.7%, and alias check blocks reuse existing columns instead of rerunning expensive ML functions. The code generator is an NL2SQL step over the annotated table.","core_discovery":"The central claim, stated on the paper's own terms, is that the barrier to ML-based social science analysis is not the ML models themselves but the glue: choosing the right models, ordering dependent calls, noticing vague queries, and translating results into queries. LEAP packages that glue into an automatic pipeline. Given raw unstructured data and a natural-language query, a forward planning filter decides whether the query is deterministic and, if it is vague, stops and proposes specified alternatives; otherwise it plans a function chain. A stage selector then extends the data with ML annotations—selecting functions from a tree so the prompt fits the token limit, following doubly linked dependency edges for implicit calls like emotion classification before emotion-trigger extraction—and generates and executes the final query. On the 120-query QUIET-ML benchmark, LEAP achieves 92% pass@1, 100% pass@3 and pass@5, and 96.97% success on the 33 vague queries, compared with 41.21% for the best baseline.","pith_inferences":["Editorial inference: the QUIET-ML benchmark's ground truth is described as 'ground-truth query results' but the paper does not say how it was produced; if the same internally supported ML functions generated those answers, then LEAP's high scores show it can reconstruct its own function outputs, not that the outputs are semantically correct for new data.","Editorial inference: the vague-query filter inherits the judgment of the underlying LLM, so the 96–98% component accuracies are likely to drift as models or prompts change; a deterministic check on data sufficiency and value ranges would make the guarantees more stable.","Editorial inference: the same architecture could extend beyond text to images, audio, or video as long as ML functions exist that can annotate those modalities; QUIET-ML already includes videos and PDFs, so the main requirement is function coverage."],"forward_implications":["Social science researchers can issue queries in plain language over raw Tweets, PDFs, or videos and receive an answer plus a structured, annotated table, without writing ML code.","Queries that are too vague to be answered deterministically are rejected with suggested reformulations, so users learn what is missing instead of getting wrong answers.","For non-vague queries with unspecified numeric thresholds, LEAP warns and then picks a data-driven value, preserving the query's intent.","End-to-end cost per query is about $1.06, under 0.1% of the estimated $1,700–$2,300 traditional annotation cost, so exploratory analyses become affordable.","User-defined ML functions can be dropped into the same pipeline, allowing researchers to reuse their own models."],"supporting_citations":[{"why":"Defines the social-science topics that QUIET-ML is designed to cover, grounding the benchmark's scope.","marker":"[126]"},{"why":"Supplies the emotion-trigger task (Q11) and vague-query examples that motivate LEAP's function-dependency handling.","marker":"[120]"},{"why":"Provides the dog-whistle dataset and functions behind Q3, a non-vague multi-annotation use case.","marker":"[75]"},{"why":"Provides the persuasion-effect task (Q17) used to illustrate queries with unspecified numerical values.","marker":"[108]"},{"why":"Supplies the 'public mood and economic indicators' query (Q2), the paper's example of a vague query.","marker":"[9]"},{"why":"Gives the chain-of-thought prompting technique used in LEAP's forward planning filter.","marker":"[114]"},{"why":"Gives the few-shot learning technique used for generating alternative query suggestions.","marker":"[13]"},{"why":"Supplies the LLM function-calling interface that LEAP uses to select stages and ML functions.","marker":"[80]"},{"why":"Is the baseline designed for schema-level vague NL2SQL queries that LEAP compares against.","marker":"[8]"},{"why":"Defines the pass@k metric used to measure LEAP's accuracy.","marker":"[59]"}],"fun_headline_variants":["LEAP auto-pipelines vague social science queries with 92% accuracy","One pipeline turns raw posts and PDFs into answered social science queries","For $1.06, this library answers vague social science queries with 92% hit rate","Glue not models is the barrier; LEAP automates the glue for 92% accuracy","Automatic ML pipeline for social science queries hits 92% on first try"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the QUIET-ML ground-truth answers are correct and were computed independently of the ML functions LEAP selects, so that matching them demonstrates semantic correctness rather than self-consistency with its own annotation functions.","fun_headline_variants_meta":{"raw":{"variants":["LEAP auto-pipelines vague social science queries with 92% accuracy","One pipeline turns raw posts and PDFs into answered social science queries","For $1.06, this library answers vague social science queries with 92% hit rate","Glue not models is the barrier; LEAP automates the glue for 92% accuracy","Automatic ML pipeline for social science queries hits 92% on first try"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2909,"prompt_tokens":1019,"completion_tokens":1890,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":1783}},"tokens_in":635,"tokens_out":1890,"duration_ms":13380,"temperature":1.0,"reasoning_tokens":1783,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:44:20.582549+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check: take a sample of QUIET-ML queries, compute answers with independently trained models or human annotators, and compare those answers to LEAP's outputs; if agreement is substantially below the reported 92% pass@1, the benchmark would be measuring reuse of the library's own functions rather than correct social-science answers.","supporting_citations":[{"cited_title":"Why Do You Feel This Way? Summarizing Triggers of Emotions in Social Media Posts","cited_arxiv_id":"2210.12531","evidence_quote":"Supplies the emotion-trigger task (Q11) and vague-query examples that motivate LEAP's function-dependency handling."}],"review_version":1}