{"id":"8a8be61c-dba4-44a9-a638-6977bd537df3","arxiv_id":"2608.00038","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Analyzing 14.65M Gemini interactions from April 2026, AI covers 68% of occupations (88% of US employment) yet only a median 21% of tasks per occupation, is rarely used for end-to-end automation, and is dominated by non-work use.","lead":"Using 15 million de-identified Gemini interactions, Google's ATLAS maps how people use AI across occupations, household tasks, countries, and languages. It finds AI use is broad but shallow and mostly collaborative, and that home use is large and missing from GDP.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The '<10% automation' claim for non-routine cognitive work rests on a cluster-level intent classifier validated only on synthetic data; a human-labeled conversation-level test is needed before accepting this headline.","rationale":"The reader's weakest assumption already identifies the pipeline's classification accuracy—including the intent classifiers—as the key uncertainty. My stress-test sharpens this to the specific cluster-level intent classifier behind the <10% automation figure, which is the most policy-relevant and least externally validated component. The paper's own caveats in Section 3.6 and Section 2.3 support this concern: clusters can nest multiple intents, and paid enterprise usage is excluded. A human-labeled validation test would directly settle whether the automation share is robust or an artifact of classifier design. Since the reader already issued a CONDITIONAL verdict, my concern does not change the verdict; it reinforces the need to release validation data or an independent benchmark before the headline should be taken as established.","tokens_in":43543,"tokens_out":3545,"duration_ms":40788,"concrete_test":"Draw a stratified random sample of ~5,000 raw work conversations from the same surfaces (or, if privacy rules prohibit, use cluster summaries together with per-cluster conversation counts), have independent human annotators label each conversation/cluster with the five intent categories defined in Section 3.6, and compare against the pipeline's cluster-level labels. Recompute the automation share for non-routine cognitive work using human labels, weighted by cluster size. If the human-labeled share is materially above 10%, or if human annotators frequently disagree with the automated label on mixed-intent clusters, the central 'mostly assistive' conclusion is not robust. Ideally also release the human-labeled validation set and classifier code for independent reproduction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most consequential claim—that end-to-end task automation represents less than 10% of AI conversations in non-routine cognitive work, implying AI is mostly assistive—depends entirely on the bespoke intent classifier in Section 3.6. The classifier does not label individual conversations; it labels cluster summaries, and the paper itself notes that clusters 'could nest multiple user intents.' With one label per cluster, any mixed-intent cluster will be assigned to whichever intent dominates the auto-generated summary. If those summaries are optimized for O*NET/ATUS taxonomy assignment rather than for intent, automation signals could be systematically underrepresented. Validation (Section 2.1, Appendix A.2.2) uses synthetic ground truth generated by the same model family used for classification, so it cannot detect model-specific biases in the 'automation' construct. Additionally, the sample is a two-week window from consumer surfaces plus free API; paid enterprise API logs are excluded (Section 2.3), so the work statistics may miss the most automation-intensive usage. The paper is carefully hedged, but the quantitative '<10%' figure is not yet established at the precision required to support the central policy claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces ATLAS v1.0, a fully automated pipeline that maps 14.65 million de-identified Gemini interactions (App, AI Mode, free API) to O*NET occupations/tasks, ATUS activities, and other attributes. The authors report three headline findings: (i) AI usage has diffused broadly across occupations (68% of detailed occupations, representing 88.4% of US employment) but remains shallow (median 21% task saturation in occupations with observed use); (ii) work-related usage is predominantly collaborative rather than automating, with end-to-end automation accounting for less than 10% of non-routine cognitive conversations; and (iii) outside work, conversational AI covers activities spanning ~98% of waking hours, with over-representation of high-friction services and potentially large unmeasured household productivity gains. The paper also documents a strong positive correlation between GDP per capita and AI usage, and broad multilingual adoption. The authors are unusually transparent about limitations: they note the absence of paid/enterprise API logs, the two-week sample window, the probabilistic nature of classifications, the lack of outcome data, and the preliminary nature of intent and expertise classifiers.","tokens_in":43882,"tokens_out":5310,"duration_ms":65244,"significance":"If the measurement pipeline is valid, this is one of the largest and most detailed empirical maps of real-world AI usage, directly informing debates on AI and labor, task-based models, household production, and the global digital divide. It complements existing industry studies (Anthropic, OpenAI) with a wider set of surfaces and introduces a household time-use taxonomy, a penetration-correction method for cross-country comparisons, and new evidence on language usage. The paper explicitly frames its findings as early observations and provides extensive methodological detail and privacy safeguards. However, the central quantitative claims—especially the <10% automation share and the 21% median task saturation—rest on proprietary LLM classifiers validated only on synthetic data and internal human ratings, with no public benchmark or released validation items. The economic correlations (wage elasticity, GDP elasticity) are descriptive and are not yet backed by causal identification. Thus the paper's significance is conditional on substantial external or human validation of the classification pipeline.","major_comments":[{"comment":"The claim that end-to-end automation is less than 10% of non-routine cognitive conversations is load-bearing and rests on an intent classifier applied to cluster summaries, not individual conversations. The paper itself acknowledges that clusters 'could nest multiple user intents'; assigning one label per cluster means mixed-intent clusters will be labeled by the dominant content of an auto-generated summary, potentially undercounting automation. Validation in §2.1 and Appendix A.2.2 uses synthetic data generated by the same model family and reports no human-labeled conversation-level test for the intent construct. Please provide precision/recall against human raters on real conversations, or restrict the headline to a preliminary, classifier-relative share.","section":"§3.6, Fig. 7"},{"comment":"The occupational coverage (68%) and median task saturation (21%) are defined relative to arbitrary thresholds—50 unique users per occupation and 25 unique users per task—and no sensitivity analysis is reported. Because the 'broad but shallow' conclusion is central, these thresholds should be varied (e.g., 10/25/100 users) or replaced with a continuous concentration measure to show that the qualitative finding is not an artifact of the chosen cutoffs. Without this, the 'shallow penetration' inference is partly built into the threshold choice.","section":"§3.4, Figs. 3–4"},{"comment":"The sample is a two-week window (April 6–19, 2026) from consumer surfaces and free API only; paid/enterprise API is excluded. The paper correctly lists this as a limitation, but the <10% automation figure in the executive summary is not consistently qualified. Enterprise usage is likely the most automation-intensive, and a two-week window may be seasonally atypical (tax-related queries are noted in §3.3). Please restate all 'automation is rare' conclusions as applying to the included consumer/free surfaces and period, and provide robustness checks on alternative windows or partial enterprise data if available.","section":"§2.3, §3.6"},{"comment":"The accuracy of the work/non-work classifier, the O*NET/ATUS mapping, and the bespoke intent/expertise classifiers is validated only with synthetic ground truth and internal human ratings; no public validation item set or external benchmark is provided. Because all headline percentages inherit any systematic misclassification (e.g., a bias toward 'collaborative' labels), the authors should release a de-identified validation sample with human labels, or at minimum report per-category precision/recall against a held-out human-annotated set of real conversations. Without this, the mapping accuracy cannot be independently assessed.","section":"§2.1, App. A.2.2, App. B"}],"minor_comments":[{"comment":"The Executive Summary states household time savings of 'just 30 minutes per week' while the model in §4.3 uses a range of 0.5–5% (≈5.4–53 minutes/week). Clarify that 30 minutes is an illustrative scenario within the range, not the conservative lower bound.","section":"Abstract & §4.3"},{"comment":"The abstract says '68% of occupations globally' and '88% of US employment' without noting that the 68% is an unweighted occupation share while the 88% is employment-weighted; please make this distinction explicit in the abstract.","section":"Abstract & §3.4"},{"comment":"The upper/lower bounds on employment coverage at different task-saturation thresholds are informative; consider presenting them in the main text rather than a footnote, since they qualify the 'shallow diffusion' claim.","section":"§3.4, fn. 14"},{"comment":"Several references have formatting issues: 'BLS , 2025' contains a stray space; Comin and Mestieri is listed as 'None 2014'; some NBER working papers and arXiv items are missing complete identifiers. Please standardize.","section":"References"},{"comment":"No data or code availability statement is provided. Even if the full data cannot be released, a statement about what ancillary materials (e.g., synthetic validation prompts, human rating instruments) will be made available would aid replicability.","section":"General"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Top of mind: this is the most detailed real-usage map of a consumer AI assistant we have, and it is a genuinely careful descriptive study. But the headline 'less than 10% automation' number is softer than the abstract implies, because it comes from a cluster-level intent classifier validated mostly on synthetic data. The paper is more hedged than its own press would suggest; still, that one number should not be treated as measured.\n\nWhat is actually new and valuable: the scale (15M interactions across Gemini App, AI Mode, and API), the mapping to O*NET occupations and ATUS activities, the penetration-adjusted cross-country comparisons, and the language-switching analysis. The distinction between occupation coverage (68%) and task saturation (21% median) is a genuinely useful way to frame shallow diffusion. The wage-intensity gradient and the ATUS time-use regressions are plausible and, importantly, do not depend on the contested intent classifier. The authors are transparent about the pipeline and compare themselves to Anthropic and OpenAI rather than ignoring them.\n\nThe soft spots, in proportion: the stress-test concern about the intent classifier lands. The classifier labels cluster summaries, and clusters can nest multiple user intents; a single label per cluster will push mixed clusters toward the dominant surface intent. Validation on synthetic data generated by the same model family cannot catch model-specific bias in the 'automation' construct. So the '<10%' figure is a directional signal, not a firm estimate. The two-week window and the exclusion of paid/enterprise API usage mean work-related automation-intensive use is likely underrepresented. The 68% and 21% numbers also depend on arbitrary user thresholds (50 and 25), which the paper acknowledges but which make point estimates fragile. I would push back on the 'circularity' framing in the Pith Report: every classifier-based study defines its outcome; the real issue is validation, not self-reference. The household $100B value is properly presented as a scenario range, so it is less concerning than the automation claim.\n\nBottom line: this is a valuable descriptive contribution for labor economists and anyone studying AI diffusion. It deserves a serious referee, and I would bring it to a reading group. The main revision I would ask for is a human-labeled conversation-level validation of the intent classifier, or a clear downgrade of the automation claim to 'preliminary cluster-based estimate.'","headline":"Big, careful descriptive snapshot of real AI usage, but the headline '<10% automation' figure rests on a cluster-level intent classifier validated only on synthetic data.","tokens_in":44349,"tokens_out":2700,"duration_ms":25540},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mapping 15 million real AI conversations onto official economic taxonomies, this paper finds that generative AI has reached most occupations and daily activities but is used for only a minority of tasks and mostly as a collaborator rather t","keywords":["generative AI adoption","task automation vs augmentation","O*NET","American Time Use Survey","household production","global diffusion","Gemini usage data","labor market impact"],"falsifier":"Take a held-out sample of several thousand real, de-identified Gemini-style conversations, have independent human annotators assign O*NET tasks, work/non-work status, and intent labels, and re-run the ATLAS pipeline on the same sample. If the end-to-end automation share in non-routine cognitive work exceeds roughly 30%, or the median task-saturation estimate moves by more than 10 percentage points, the paper's central 'broad but shallow, assistive' conclusion is contradicted. A weaker public signal would be an equivalent audit of another major AI assistant's logs showing automation-heavy usage","tokens_in":43497,"feed_emoji":"📊","tokens_out":5864,"duration_ms":60700,"temperature":0.7,"pith_summary":"The paper builds a privacy-preserving census of AI use by mapping 15 million de-identified Gemini conversations to standard economic categories: occupations and tasks (O*NET/SOC), daily activities (ATUS), countries, and languages. Its central claim is that generative AI has diffused remarkably widely—touching 68% of occupations and activities covering 98% of Americans' waking hours—but has penetrated shallowly: the median occupation uses AI for only 21% of its tasks, and end-to-end automation accounts for under 10% of conversations in non-routine cognitive work. A sympathetic reader would see this as the first large-scale behavioral evidence that, at this stage, AI is a complement to human work and household life rather than a substitute. The paper also argues that home use, especially in high-friction bureaucratic tasks, generates value that GDP misses, and that global adoption tracks national wealth closely. Why it matters: it moves the debate from exposure projections to observed usage, giving policymakers an empirical foundation for questions about displacement, productivity, and the digital divide.","feed_headline":"AI reaches 68% of jobs, automates under 10% of cognitive work","feed_subtitle":"A 15-million-conversation map of Gemini use shows AI is a collaborator, not a replacement, for most tasks and households.","key_machinery":"The load-bearing machinery is the ATLAS measurement pipeline: 14.65 million de-identified interactions (Gemini App, AI Mode, and API, April 6–19, 2026) are PII-redacted, summarized, clustered by the OCTO tool, and then assigned to the O*NET/SOC work taxonomy and the ATUS time-use taxonomy, plus bespoke intent and expertise classifiers. A penetration adjustment using web-traffic referral shares is applied for cross-country comparisons. This machinery converts unstructured chat into statistically comparable task- and activity-level counts, which is what makes the headline percentages possible; every one of those percentages inherits the pipeline's classification accuracy.","core_discovery":"On its own terms, the paper's discovery is that actual AI usage is broad but shallow and overwhelmingly collaborative. At work, usage spans 68% of detailed occupations representing 88% of US employment, yet only 3% of occupations show AI usage for more than three-quarters of their tasks, and the median occupation shows usage for 21% of tasks. For non-routine cognitive work—the category that dominates interactions—the intent classifier attributes end-to-end automation to fewer than 10% of conversations, with the rest falling into partial drafting, review, ideation, or information retrieval. Outside work, over 86% of conversational interactions occur, and AI is over-represented in exactly the","pith_inferences":["If the shallow-usage pattern reflects current capability rather than intrinsic demand, the 21% median task saturation is a lower bound; one testable extension is to track whether saturation rises as model capabilities improve in the same panel.","The paper's own limitation—no outcome data—suggests a natural extension: link usage clusters to downstream task completion or time-use diaries to estimate actual productivity gains rather than conversation counts.","The language-symmetry result (work and non-work code-switching nearly identical) could be tested against other providers' data; if it holds, it implies that perceived multilingual quality gaps are not driving task-sorting behavior.","The 'interest-adoption gap' between search interest and usage in emerging markets invites a causal study: does lowering access costs (e.g., cheaper data plans, SMS-based AI) convert interest into usage, as suggested by some cited pilots?"],"forward_implications":["If correct, exposure-based projections of massive white-collar displacement are not matched by current real-world usage: the gap between 'affected' and 'automated' is large and measurable.","Household AI use in planning, research, and bureaucracy implies a flow of non-market time savings (the authors' scenario range is roughly $15–150 billion per year in the US) that standard GDP accounting omits.","The steep income and education gradient of usage implies that, absent deliberate diffusion measures, AI benefits may concentrate among higher-wage workers and wealthier countries, widening existing divides.","The 9–12% higher turn counts and 18–20% more tokens for non-primary-language interactions suggest investments in multilingual capability carry a measurable efficiency dividend for users.","The government-services over-representation (roughly 20 times its time share) and off-hours usage suggest AI is already functioning as a de facto extension of public-service access, with implications for digital-government design."],"fun_headline_variants":["AI spans 68% of jobs, automates under 10% of cognitive work","Google's 15M Gemini chats reveal AI's broad but shallow use","AI is a collaborator, not a replacer, per 15M Gemini chats","88% of US employment touched by AI, but automation is rare"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire statistical picture rests on the automated classifiers—work/non-work triage, O*NET task and occupation assignment, intent, and expertise—being accurate on real conversations; they were validated only on synthetic ground truth and internal human ratings, with no public benchmark, and the sample is a two-week window from one provider's consumer surfaces.","fun_headline_variants_meta":{"raw":{"variants":["AI spans 68% of jobs, automates under 10% of cognitive work","Google's 15M Gemini chats reveal AI's broad but shallow use","AI is a collaborator, not a replacer, per 15M Gemini chats","88% of US employment touched by AI, but automation is rare"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000987,"raw_usage":{"total_tokens":4052,"prompt_tokens":802,"completion_tokens":3250,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":3178}},"tokens_in":546,"tokens_out":3250,"duration_ms":68920,"temperature":1.0,"reasoning_tokens":3178,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:33:54.743600+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.3.0-alltime-deepseek","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Take a held-out sample of several thousand real, de-identified Gemini-style conversations, have independent human annotators assign O*NET tasks, work/non-work status, and intent labels, and re-run the ATLAS pipeline on the same sample. If the end-to-end automation share in non-routine cognitive work exceeds roughly 30%, or the median task-saturation estimate moves by more than 10 percentage points, the paper's central 'broad but shallow, assistive' conclusion is contradicted. A weaker public signal would be an equivalent audit of another major AI assistant's logs showing automation-heavy usage","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}