{"id":"75d6eb88-e1f3-4f87-b3c3-04bec95f3495","arxiv_id":"2607.19375","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Using O*NET and 4.5M chatbot conversations plus synthetic prompts, EconEvals measures LM performance on U.S. work activities and predicts substantial time savings in 47% of occupations, with usage lagging.","lead":"EconEvals builds language-model benchmarks and a simulation-based 'exposure' measure to estimate how much time AI could save across U.S. occupations, using the O*NET job taxonomy. It finds broad potential time savings in nearly half of occupations, but little observed usage in most of those tasks, with privacy and proprietary systems identified as key bottlenecks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exposure headline depends on an unvalidated self-simulation: the LM generates both the task step times and the savings estimates, so the 46.6%/79.4% figures need a human time-use validation before they can support the central claim.","rationale":"The paper has real independent support for its benchmark infrastructure: a detailed pipeline with labeled precision (0.91), cost estimates, and a 0.67 average Spearman correlation with GDPval. The occupation-level benchmark validation is noisy (per-occupation correlations range from -0.40 to 1.00), but that concern is secondary. The load-bearing weakness is the exposure measure, exactly as the reader's weakest_assumption states. My concern does not change the overall verdict: the paper is a plausible infrastructure contribution, but the headline exposure numbers should not be reported as established until at least one external validation of the simulated time savings is performed. The test I propose would settle whether the concern lands. If it fails, the exposure portion should be relabeled as a synthetic/narrative measure rather than an empirical estimate; if it passes, the headline is substantially strengthened. The lack of released code/data artifacts is additional reason for conditionality, but the core issue is the missing ground truth for the time-savings estimates.","tokens_in":26803,"tokens_out":6438,"duration_ms":62511,"concrete_test":"Take a stratified sample of 40 O*NET tasks across the exposure distribution (high/moderate/low and across the 857 occupations used in Fig. 6). For each task, conduct a small time-and-motion study: have 3–5 real workers perform the task with and without a current chatbot, measuring elapsed time; or reuse an existing human time-use/job-analysis dataset (e.g., BLS American Time Use Survey or O*NET job analyst ratings) as a proxy. Compare the measured median per-task time savings with the simulation's predicted median. If the rank correlation is below ~0.3 or if mean absolute error in percentage savings exceeds 20 points—or if the occupation-level 46.6% share changes by more than a few points when recomputed with measured savings—the central exposure claim is not supported. Releasing the per-task simulation traces alongside would make the comparison auditable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim—current LMs could save substantial time on at least half of the tasks in 46.6% of U.S. occupations—rests entirely on the simulation-based exposure measure in §4/§D.1. The measure works by having an LM roleplay a worker, first generating a step-by-step task decomposition with baseline times per step, then estimating how much of each step a chatbot would save. These estimates are produced without any human time-study, direct observation, or external time-use data. The input prompt to the 'worker' is itself a synthetic prompt generated and verified by LMs (Appendix B), so the entire pipeline is a closed loop: LMs generate task steps and times, and LMs judge their own usefulness. The paper reports no precision, recall, or calibration for these time estimates, unlike the query-mapping pipeline which has labeled precision. The only external comparison is to Anthropic's Claude usage shares, but that compares predictions to adoption, not to actual time savings. Figure 6 also excludes 159/1,016 occupations due to 'data generation errors,' so the denominator of the headline is not the full occupation set. If the self-simulated times are systematically optimistic—e.g., the model overestimates baseline step times or underestimates overhead—the 46.6% and 79.4% figures are unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EconEvals, an open-source evaluation infrastructure for measuring language-model capabilities on U.S. labor-economy tasks. It builds DWA-level benchmarks from real user–chatbot conversations (143 DWAs) and augments these with synthetically generated queries for a broader task space, reporting 226 benchmark evaluations in total, including 43 occupation-level benchmarks aligned with OpenAI's GDPval. The paper also proposes a 'whitebox' simulation-based exposure measure in which an LM roleplays a worker, decomposes each O*NET task into steps with baseline times, and estimates per-step time savings from current chatbot capabilities. The headline empirical claims are that current models could save substantial time on at least half of tasks in 46.6% of U.S. occupations, that 79.4% of predicted-high-exposure tasks show little current Claude usage, and that the synthetic occupation-level benchmarks predict GDPval scores with mean Spearman correlation 0.67.","tokens_in":27123,"tokens_out":4408,"duration_ms":42719,"significance":"If the claims hold, the paper would be a substantial contribution: it provides the first open benchmark suite attempting to cover the full O*NET taxonomy, a transparent per-task exposure accounting with reasoning traces, and a much cheaper synthetic-data alternative to human-sourced GDPval-style benchmarks. The paper's strengths include its detailed pipeline documentation, explicit cost reporting, a labeled precision estimate (0.91 on n=50) for the real-query mapping, correlation checks against GDPval, and the release of per-task exposure justifications. However, two load-bearing claims are currently unsupported: the assertion that benchmarks exist for all 2,087 DWAs / 1,016 occupations, and the validity of the simulation-based exposure headline. These need substantial revision or re-scoping before the central conclusions can be accepted.","major_comments":[{"comment":"The Introduction states that the pipeline 'result[s] in benchmarks for all 2,087 DWAs, spanning all 1,016 occupations,' and the abstract implies full-coverage evaluation. But §3 reports results for only 226 benchmarks: 143 real DWA-level, 40 synthetic DWA-level, and 43 occupation-level. The other ~2,000 DWAs have synthetically generated queries, but no model scores are reported. The paper should clearly distinguish 'query generation coverage' from 'evaluated benchmark coverage' and revise the claims that ECONEVALS 'provides benchmarks for all 2,087 DWAs.' This is load-bearing because full coverage is a central advertised advantage over GDPval.","section":"§2.2, §3, and Introduction"},{"comment":"The exposure headline (46.6% of occupations with at least half of tasks substantially exposed) rests entirely on an unvalidated self-simulation. In Appendix D.1, the LM generates the worker background, the step decomposition, the baseline time per step, and the time saved per step; there is no human time-use study, no direct observation, and no calibration against measured time savings. Unlike the query-mapping pipeline, no precision or recall is reported for these time estimates. Figure 6 also excludes 159 of 1,016 occupations due to 'data generation errors,' so the 46.6% figure is computed on 857 occupations, not the full set. The paper needs either external validation (e.g., human time studies, comparison to actual task-completion times) or a clear reframing of these numbers as 'simulation-based potential' with the denominator limitation stated in the abstract and main text.","section":"§4 and Appendix D.1"},{"comment":"The synthetic data generation and the exposure simulation share the same conditioning on a time-savings ladder: the worker roleplay is asked to produce a prompt that saves a target percentage of time, and a verifier checks only realism and answerability, not the truth of the time-savings claim. The exposure simulation then uses that same synthetic prompt and a model response to estimate time savings. This creates a potential circularity: the LM is asked to estimate savings for a prompt it generated to justify a target savings level. The paper should provide a sensitivity analysis with independently constructed prompts, or demonstrate that estimated time savings are not inflated by the generation procedure.","section":"Appendix B and Appendix D.1"},{"comment":"The abstract's '500× lower cost' claim is based on a rough, lower-bound estimate. Appendix C.1 uses the GDPval mean review time (109 minutes) as a proxy for per-reviewer task-creation time, explicitly excludes drafting and editing time, and rates the estimate as low-to-medium confidence. The comparison also mixes LM-generated synthetic queries against human worker queries, which the paper itself acknowledges are higher quality along several dimensions. The cost factor should be qualified as a very rough estimate, not stated as a precise 500× advantage in the abstract and introduction.","section":"§3 and Appendix C.1"}],"minor_comments":[{"comment":"The abstract reports '47% of occupations' while §4 reports 46.6%; the rounding is acceptable, but the paper should be consistent and should state the effective denominator (857 occupations) near the headline.","section":"Abstract / §4"},{"comment":"The sentence 'We prioritize recall to improve task coverage' appears to contradict the immediately preceding reported recall of 0.18. This is likely a typo (perhaps 'coverage' was meant), but as written it is confusing.","section":"§2.1"},{"comment":"The table caption refers to 'lines in grey' that are filtered out, but in the provided text all rows appear identical. Please ensure the grey shading is visible in the final version.","section":"Table 3"},{"comment":"The text says 'We present results for 226 benchmarks' and then lists 143 + 40 + 43 = 226, but the synthetic DWA evaluation is only for 40 DWAs. The paper should explicitly note that the remaining DWAs have generated queries but no reported benchmark scores, to avoid the impression of full evaluation.","section":"§3"},{"comment":"The paper promises open-source release but no repository or data URL is provided in the text. Please include a link or state the planned release venue.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a promising infrastructure paper with a genuinely useful benchmark pipeline and a novel whitebox exposure trace. However, the two headline claims—full benchmark coverage and simulation-based exposure percentages—are currently overstated relative to what is actually evaluated and validated. The benchmark overclaim is easily fixed by rephrasing. The exposure claim requires either hard validation or a substantially more cautious presentation; without that, the central '46.6% / 79.4%' results should not be promoted as estimates of actual worker time savings. I recommend major revision rather than rejection because the underlying data release and pipeline are valuable and the authors have demonstrated ability to provide detailed methodological appendixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the benchmark pipeline is a real contribution: the authors build DWA-level evaluations from 4.5M real chat queries, supplement with verified synthetic prompts, and show their occupation-level scores correlate with GDPval at 0.67. Second, the headline exposure claim — current models could save substantial time on at least half of tasks in 46.6% of occupations — is produced by an LLM roleplaying a worker, generating its own step-by-step task decomposition with baseline times, then estimating the time savings from a chatbot. There is no human time-use validation anywhere in that loop. Treat the exposure numbers as a plausible simulation, not a measurement.\n\nThe paper does several things well. The retrieval-and-classify pipeline is transparent, with precision 0.91 on a modest labeled sample, an honest recall estimate of 0.18, and a breakout of costs. The synthetic data generation uses a time-savings ladder and verifier models, which is more careful than most. The whitebox simulation at least produces reasoning traces, so you can see why a task is or isn't exposed. Comparing against Claude usage data is a sensible sanity check, and the skill-importance regressions add context.\n\nThe soft spots are real but specific. The exposure measure is the load-bearing flaw: the LLM is both the worker and the estimator, with no external anchor for step times or savings. The Claude comparison is about adoption, not realized savings. Also, 159 of 1,016 occupations are dropped from Figure 6 due to \"data generation errors,\" so the denominator behind the headline is not the full set. The abstract says benchmarks cover all 2,087 DWAs, but only 226 benchmark results are actually reported; the rest are synthetic generation claims without evaluated scores. The 500x cost comparison is limited to query generation and is based on a rough back-of-the-envelope estimate of GDPval costs. Finally, the preprint says \"open-source\" but ships no code or data links — that needs fixing for the claims to be checkable.\n\nThis is a paper for people building or using economic benchmarks for AI. The benchmark contribution is solid enough to cite and build on. The exposure headline should be flagged as simulated until someone validates it against human time-use studies or at least grounds it in observed productivity gains. I'd send it to peer review, but the revision should either add external validation or soften the exposure framing. The infrastructure is the value; the self-simulation is a hypothesis in need of a test.","headline":"Useful benchmark infrastructure paper whose headline exposure claim rests on an unvalidated LLM self-simulation; the benchmark work deserves attention, the 47% number does not.","tokens_in":27644,"tokens_out":1972,"would_cite":true,"duration_ms":20887,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that language models' economic value is best assessed task by task, and its simulation-based estimates indicate current chatbots could save substantial time on at least half the tasks in nearly half of U.S. occupations, wi","keywords":["language models","economic evaluation","labor market impact","occupational exposure","task-level benchmarks","synthetic data","time savings","AI adoption"],"falsifier":"Run a controlled time study: have workers in a sample of high-exposure occupations perform their real tasks with and without a current chatbot, with independent time measurement. If the measured minutes saved come out substantially below the simulated step-level savings, or if the steps the simulation marks as automatable prove to require human judgment or integration overhead, then the 46.6% occupation estimate and the bottleneck rankings would be unsupported.","tokens_in":26656,"feed_emoji":"💼","tokens_out":7514,"duration_ms":74713,"temperature":0.7,"pith_summary":"The paper replaces hand-waving about AI's labor impact with concrete, task-level measurement. It introduces EconEvals, an open evaluation suite with benchmark queries for every work activity and occupation in the U.S. government's occupational taxonomy, grounded in real user-chatbot conversations where possible and supplemented by synthetic roleplay queries. Alongside the benchmarks, it introduces a 'whitebox' simulation in which a language model roleplays a worker and itemizes, step by step, how much time a chatbot would save on each task. Using that simulation, the paper estimates that current language models could save substantial time on at least half of the tasks in 46.6% of occupations. It also finds that 79.4% of task opportunities judged highly time-saving see little real chatbot use, and that privacy and proprietary systems are the main fixable bottlenecks beyond physical or interactive limits.","feed_headline":"AI could save time in 46.6% of U.S. occupations","feed_subtitle":"A task-level evaluation of language models finds real chatbot use lags the potential.","key_machinery":"The load-bearing mechanism is the whitebox simulation-based exposure measure. For each occupation and task, a language model roleplays a worker answering a labor economist's questions: first describing the job, then listing the steps of the task with baseline minutes per step, then considering a chatbot response to a synthetic query and marking which steps a chatbot would speed up and by how much. A post-processing filter removes steps that belong to other tasks; three simulation runs are median-averaged; and the aggregate savings are converted into exposure categories. This per-step accounting does two jobs at once: it produces the numeric time-savings estimates behind the 46.6% headline, a","core_discovery":"The paper's central claim is that the connection between language-model capability and labor-market value can be measured directly, task by task, rather than inferred from broad capability lists. It contributes an open evaluation suite with benchmark queries for all 2,087 work activities and all 1,016 occupations in the U.S. occupational taxonomy, built partly from real public user-chatbot conversations and partly from synthetic roleplay queries, at roughly a 500-fold lower query-generation cost than the existing worker-sourced benchmark that covered fewer than 5% of occupations. On the exposure side, it simulates a worker using a chatbot, decomposes each task into timed steps, aggregates sa","pith_inferences":["Inference: If the simulated per-step time accounting is validated, the practical unit of AI planning shifts from whole occupations to individual steps, letting firms and workers target the steps where chatbots help most while leaving interactive or privacy-bound steps to humans.","Inference: The bottleneck taxonomy invites a direct experiment: in organizations that add chatbot access to private or proprietary systems (for example, internal data and compliance-approved workflows), the usage gap on currently exposed-but-unused tasks should measurably shrink, which would confirm the paper's causal story rather than just its correlation.","Inference: The same roleplay-and-itemize simulation could be run on other countries' job taxonomies or on an individual firm's task lists, producing localized exposure maps at low cost; the method is not tied to the U.S. taxonomy.","Inference: Because the simulation is produced by the very kind of model being evaluated, it may be systematically optimistic about step-level savings; comparing its estimates to randomized field measurements is the natural next test."],"forward_implications":["Because the benchmarks cover the full U.S. work taxonomy and cost roughly 500x less to build than worker-sourced ones, model rankings for economically relevant work could be refreshed as new models are released without repeated expensive data collection.","The usage-exposure gap implies that capability alone does not determine productivity impact: even if current models could save time on 79.4% of the tasks judged exposed, realized gains depend on complementary integration and adoption.","The bottleneck analysis suggests that fixing privacy and proprietary-system constraints, rather than waiting for better models, could unlock a large share of predicted time savings.","Synthetic benchmarks that predict worker-grounded scores with an average correlation of 0.67 can serve as a cheap screening tool for deciding which occupations deserve expensive human evaluation.","Simulation-based exposure being lower than rubric-based exposure implies that capability-list rubrics systematically overestimate how much current chatbots can help workers."],"fun_headline_variants":["AI could save time in 47% of jobs, but usage lags","New benchmark measures AI's labor impact task by task","Language models could slash half of tasks in 47% of occupations","AI potential vs. reality: usage lags in 79% of tasks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The simulation assumes that a language model roleplaying a worker can accurately estimate how long each task step really takes and how much time a chatbot would save, and no human time-study is used to check those estimates.","fun_headline_variants_meta":{"raw":{"variants":["AI could save time in 47% of jobs, but usage lags","New benchmark measures AI's labor impact task by task","Language models could slash half of tasks in 47% of occupations","AI potential vs. reality: usage lags in 79% of tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1102,"prompt_tokens":741,"completion_tokens":361,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":285}},"tokens_in":485,"tokens_out":361,"duration_ms":4413,"temperature":1.0,"reasoning_tokens":285,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T09:46:16.146802+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled time study: have workers in a sample of high-exposure occupations perform their real tasks with and without a current chatbot, with independent time measurement. If the measured minutes saved come out substantially below the simulated step-level savings, or if the steps the simulation marks as automatable prove to require human judgment or integration overhead, then the 46.6% occupation estimate and the bottleneck rankings would be unsupported.","supporting_citations":[],"review_version":1}