{"id":"25fa59bf-9dd8-45d8-8909-1400d22fee4f","arxiv_id":"2608.05008","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"In ChatGPT's first ad rollout, lower-income simulated accounts received ads more often than higher-income accounts, with no significant race difference.","lead":"This paper reports the first large-scale audit of ads shown inside ChatGPT, using 91 fake user accounts to test how ad delivery varies by race and income. It finds lower-income accounts were more likely to receive ads, while race showed no detectable effect in the first month of rollout.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Income gradient may be an artifact of staggered account creation: Section 5.1 omits account age/date, and Section 3.1 shows creation spanned late January to mid-March; re-running with creation-date controls would settle it.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: without controlling for account age or exposure opportunity, the income gradient in Section 5.1 cannot be attributed to income-based targeting. This is not a disagreement with consensus; it is an internal identification problem in the regression. The paper deserves credit for releasing the ad archive, standardizing prompts, and documenting limitations, and the descriptive content findings (sector mix, prompt-level ad rates, OpenAI-excluded topics) are independent of this confound. However, the paper's own Section 3.1 states creation spanned late January to mid-March, and Section 5.2 shows ads began March 8, so the binary ever-exposed outcome is exquisitely sensitive to when each account was created. Since the regression has no timing covariate, the central empirical claim is conditional on an untested assumption. If the proposed covariate adjustment preserves a significant negative income odds ratio, the finding would be substantially stronger; if it does not, the headline should be reframed as a descriptive baseline rather than evidence of income-based ad targeting. Either way, the current verdict of CONDITIONAL is the right level, so no change is needed.","tokens_in":13524,"tokens_out":4055,"duration_ms":46327,"concrete_test":"Re-run the Section 5.1 logistic regression with the same 91 accounts and ZIP median income, adding account creation date (or days active between March 8 and 31, plus completed prompt-days) as covariates. If the income odds ratio attenuates toward 1.0 or its 95% CI crosses 1, the income gradient is not separable from study timing. As a secondary check, stratify by creation batch (pre- vs post-March 8) and by the low-completion days March 15/24; if the income effect appears only in early-created or high-completion subsamples, the timing/missingness artifact is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is Section 5.1's finding that lower-income accounts are more likely to receive ads. The logistic regression of binary ad exposure on ZIP-level median income includes no control for account creation date or active days in the March 8-31 ad window. Section 3.1 says accounts were created in small batches from late January to mid-March, and Section 5.2 reports a 14-day median from activation to first ad, with ads beginning only on March 8. An account created in February had several weeks of eligible exposure before the March 31 cutoff; an account created on March 15 had at most 16 days. If creation batch is correlated with assigned income tercile, the odds ratio in Section 5.1 could reflect differential opportunity rather than income-based targeting. The robustness checks there (quadratic term, Cook's-distance deletion) test functional form and leverage, not this confound. Section 6.3's reported incomplete runs (65% of prompts succeeded; completion below 35% on March 15 and 24) add a second route to the same bias: if missingness correlates with income, the observed exposure difference could be an artifact of data collection, not ad delivery. Because this is the headline contribution, the omitted timing variable is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the first empirical audit of advertising on ChatGPT, using 91 sock puppet accounts stratified by three racial/ethnic groups and three income terciles via ZIP-code residential proxies. The authors collected over 3,600 ads between February and May 2026, characterize the advertiser ecosystem, and release a public ad library. Their headline findings are that accounts begin receiving ads roughly 14 days after creation, that lower-income accounts are more likely to be exposed to ads, and that race shows no significant association with ad delivery. They also report that ads are mostly from retail and information sectors and that OpenAI's stated content exclusions appear to hold in their data.","tokens_in":13761,"tokens_out":5919,"duration_ms":62616,"significance":"If the income-gradient finding survives scrutiny, this paper provides a valuable early empirical baseline for a new and consequential advertising channel. The public ad archive, the detailed and reproducible sock-puppet methodology, and the candid discussion of infrastructure fragility are genuine strengths. The paper is also careful to frame its null race finding as underpowered. However, the central income-exposure claim currently rests on an account-level regression with a marginal p-value and no control for account creation timing, and the paper's own limitations section discloses substantial missing data that is not accounted for in the main analysis. These issues make the headline result fragile and need to be addressed before the finding can be considered established.","major_comments":[{"comment":"The logistic regression of ad exposure on ZIP income omits account creation date or days active during the March 8–31 ad window. Section 3.1 states accounts were created in small batches from late January to mid-March, so an account created in February had weeks of exposure opportunity before March 31, while an account created on March 15 had at most 16 days. If creation batch is correlated with assigned income cell, the reported odds ratio (OR = 0.98 per $1,000, p = 0.0438) could reflect differential opportunity rather than income-based targeting. The robustness checks reported in this section (quadratic income term, Cook's distance deletion) test functional form and leverage, not this confound. Please add creation date or days-active as a control, or restrict to accounts active for the full ad window, and report the relationship between creation date and income cell.","section":"Section 5.1 / Figure 6a"},{"comment":"The limitations section reports that only 65% of prompts were successfully run overall, with mean completion below 35% on March 15 and March 24. Missingness of this magnitude can bias the binary exposure outcome if failures correlate with income cell (e.g., through proxy stability or account flagging). The income regression in Section 5.1 uses exposure measured over the study window, but the analysis does not test whether missingness is balanced across income and race cells or across account age. Please report missingness by cell and by creation date, and conduct a sensitivity analysis (e.g., per-protocol analysis or multiple imputation) to show that the income finding is not an artifact of differential data collection.","section":"Section 6.3"},{"comment":"The claim that accounts begin receiving ads 14 days after account creation is confounded by the fixed March 8 ad rollout date. If ads became available on March 8, any account created about two weeks earlier would show a 14-day gap to first ad regardless of platform behavior. The paper should compare observed time-to-first-ad against a null model that incorporates creation dates and the rollout date, or restrict to accounts created before the rollout and report the conditional distribution. In addition, the sentence reporting 'median time ... 14.0 days (mean = 17.6, SD = 7.6, range: 13–16)' is internally inconsistent: a range of 13–16 cannot produce a mean of 17.6 and SD of 7.6; please clarify whether the bracketed numbers are the IQR or the range.","section":"Section 5.2"},{"comment":"The primary income result depends on a marginal p-value of 0.0438 from an account-level regression with n = 91, with no adjustment for the multiple outcomes examined in the paper (exposure, ad rate, race-stratified Spearman correlations, topic-level ad rates). The inclusion of several related tests raises the risk that this result is a chance finding. Please report the sensitivity of the conclusion to the exposure-window definition (Figure 6a is labeled 'full-study February 6 – May 20, 2026,' while Section 5 defines the study period as March 8–31), and provide confidence intervals or a permutation-based p-value that does not rely on asymptotic normality at this sample size.","section":"Section 5.1 / Section 5.3"}],"minor_comments":[{"comment":"The abstract reports 186 unique advertisers, while Sections 1.1 and 5 report 191 unique advertisers; please reconcile these numbers.","section":"Abstract / Section 1.1 / Section 5"},{"comment":"The Figure 9 caption reports 3,303 confirmed ad impressions, while Section 5 reports 3,573 ads for March 8–31 and 3,602 total; please reconcile these counts.","section":"Figure 9"},{"comment":"There is a duplicated article in 'top Reddit post titles by upvote the the past year'; please fix the typo.","section":"Section 3.2"},{"comment":"'Februrary' is misspelled in both places; please correct.","section":"Sections 1.1 and 3.2"},{"comment":"The contradictory 'range: 13–16' and 'interquartile range spanning only 13 to 16 days' should be clarified: one of these is likely the IQR and the other should be the actual minimum and maximum.","section":"Section 5.2"},{"comment":"Section 1.1 states 8–12 accounts per demographic group, while Section 5.1 mentions 16–19 exposed accounts per racial group; please clarify whether these are different quantities (cell sizes vs. exposed accounts) or whether one of the statements is inaccurate.","section":"Section 1.1 / Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important topic, and the public ad archive plus the detailed methodology are genuine contributions. The central income-gradient claim is currently vulnerable to the timing confound and to missing-data bias, but both are addressable with additional analysis; hence I recommend major revision rather than rejection. I would also ask the authors to reconcile the inconsistent numbers and figure captions, as those affect the paper's credibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the audit itself: first systematic measurement of ad delivery inside an LLM chatbot, with a public archive of over 3,600 ads, full prompt/response pairs, and a reproducible sock-puppet pipeline. That is a real contribution, and the descriptive findings—retail and software dominate, topic-level targeting, near-zero ads on health/politics—look solid. The race null is appropriately hedged given small cell sizes. I'd cite this as the baseline for any future LLM ad audit.\n\nThe weak spot is the income gradient. The logistic regression of ad exposure on ZIP income has no control for account creation date or days active in the March 8–31 window, and accounts were created in batches from late January to mid-March. Ads didn't exist before March 8, so an account created in February had far more opportunity to see an ad than one created in mid-March. If creation order correlates with income—and nothing in the paper rules that out—the odds ratio of 0.98 per $1k (p=0.0438) could be pure exposure opportunity. The robustness checks (quadratic term, Cook's distance deletion) test functional form and leverage, not this confound. The 14-day 'time to first ad' finding is also almost certainly an artifact: the range is 13–16 days, which is exactly what you'd expect if all first ads happened shortly after March 8 for accounts that had been active long enough. It says nothing about a platform rule. The incomplete-run problem (65% of prompts overall, below 35% on some days) is another unaddressed route to the same bias if missingness correlates with income.\n\nNone of this kills the paper. The descriptive content analysis and the archive are worth peer review on their own. But the abstract's claim that lower-income accounts are more likely to receive ads is not supported by the analysis as presented. The authors need to re-run with creation-date controls, report missingness by income, and probably soften the claim to a preliminary association. I'd send it to review rather than desk-reject: it's a first measurement of a billion-user platform's new ad channel, and the method is reusable. Just don't let the income sentence through as-is.","headline":"First useful baseline of ChatGPT ads, but the headline income gradient doesn't survive contact with the study's own timing data.","tokens_in":14254,"tokens_out":2599,"would_cite":true,"duration_ms":28025,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChatGPT’s first ad rollout concentrates ads on lower-income accounts.","keywords":["LLM advertising","sock puppet audit","algorithm auditing","ad targeting","income disparity","ChatGPT","demographic stratification","online advertising"],"falsifier":"Using the paper’s released data, compare account creation dates across income terciles and re-estimate the ad-exposure regression with days active before March 8 as a control; if the income coefficient drops to zero, the gradient is a timing artifact rather than income-based targeting.","tokens_in":13361,"feed_emoji":"📢","tokens_out":6800,"duration_ms":69324,"temperature":0.7,"pith_summary":"This paper reports the first audit of advertisements served inside ChatGPT, using 91 simulated accounts that signal three racial and three income groups through ZIP-code-targeted residential proxies. The central finding is that lower-income accounts were significantly more likely to receive ads, with each $1,000 increase in ZIP-level median income reducing the odds of ad exposure by about 2 percent; race showed no detectable association. The paper also establishes an early baseline for the new channel: ads first appeared roughly 14 days after account creation, were clearly labeled and separated from model responses, and were dominated by retail and information-sector advertisers. If correct, the result means the internet’s ad-supported business model is arriving in LLM chatbots with a distributional pattern that puts more advertising on users with less income.","feed_headline":"ChatGPT’s first ads concentrate on lower-income users","feed_subtitle":"Audit of 91 simulated accounts finds income, not race, drives who sees ads in the new LLM channel.","key_machinery":"The mechanism that carries the argument is a demographically stratified sock puppet audit. The researchers created 91 synthetic ChatGPT accounts, each bound to a U.S. ZIP code selected from census data to represent one of nine cells formed by crossing three racial/ethnic groups (Black, Hispanic, White) with three income terciles (low, medium, high). All traffic for an account was routed through a residential proxy matched to that ZIP code, and each chat session began with location-signaling prompts. Every day each account received the same random 30-prompt subset drawn from a 335-prompt corpus, and the full HTML of every response was captured so that ads could be identified by the “Sponsored” label. This pipeline converts demographic signals into observable differences in ad delivery and is designed to be replicable on other platforms or later time periods.","core_discovery":"The paper’s central empirical discovery is that in the first weeks of ChatGPT’s ad rollout, ad exposure was concentrated on lower-income accounts. A logistic regression of ad exposure on median household income of the assigned ZIP code yielded an odds ratio of 0.98 per $1,000 increase (95% CI [0.96, 1.00], p = 0.0438), and the association strengthened when three high-leverage accounts were removed (OR = 0.975, p = 0.0053). No significant association between race and ad delivery was found, though the authors caution that with 16–19 exposed accounts per racial group the study is underpowered to detect small effects. The paper also documents that ads began appearing on March 8, a median of 14 days after account activation, and that among exposed accounts ads appeared in a median of 24% of prompts after the first ad. In this early phase, ads were labeled “Sponsored”, visually separated from the model’s text, and concentrated in retail and information sectors, with Target, Top10.com, Preply, and Advance Auto Parts among the most frequent advertisers.","pith_inferences":["If the income gradient persists as ad targeting matures, ChatGPT advertising could create a regressive pattern in which lower-income users bear more commercial messages—a dynamic the paper’s three-week window cannot yet confirm.","The consistent 14-day gap between account creation and first ad suggests a warm-up or anti-abuse mechanism; future audits of LLM ads should explicitly model this lag rather than treating it as noise.","The released ad archive makes a natural longitudinal follow-up possible: re-running the same audit after the platform’s May 2026 international pilot would show whether the income gradient and content exclusions hold across markets."],"forward_implications":["If ad delivery concentrates on lower-income accounts, the new LLM ad channel begins with a distributional pattern that places a higher advertising burden on users with less income.","OpenAI’s stated exclusions for health, mental health, and political content appear to be working in this early window, since those prompt categories show near-zero ad rates.","The current ad format is clearly labeled and separate from model responses, giving researchers a stable baseline for tracking how disclosure and integration evolve.","Among exposed accounts, ad delivery is persistent rather than sporadic, with a median of 24% of prompts producing an ad once delivery begins.","The methodology of ZIP-stratified sock puppets can be reused to audit future LLM ad systems and other closed platforms."],"supporting_citations":[{"why":"Announces the ChatGPT ad rollout and the stated content exclusions that the audit tests.","marker":"OpenAI 2026"},{"why":"Provides the OpenAI usage report from which the prompt corpus’s common formulations and categories are drawn.","marker":"Chatterji et al. 2025"},{"why":"Supplies the ZIP-level median income and racial composition data used to build the nine demographic cells.","marker":"U.S. Census Bureau 2023"},{"why":"The foundational audit showing racial disparities in online ad delivery, which this study extends to LLM interfaces.","marker":"Sweeney 2013"},{"why":"Shows that ad delivery algorithms can produce demographically skewed outcomes despite neutral targeting; the study’s comparative case for measuring delivery disparities.","marker":"Ali et al. 2019"},{"why":"Defines the sock puppet audit method that the study adapts to a closed chatbot platform.","marker":"Sandvig et al. 2014"}],"fun_headline_variants":["ChatGPT ads skew toward lower-income users","Sock puppet audit: income, not race, drives ChatGPT ads","Lower-income accounts see more ChatGPT ads, audit shows","ChatGPT's ad rollout is income-skewed, new audit finds","First phase of ChatGPT ads hits low-income users hardest"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The income gradient in ad exposure is only meaningful if it reflects income-based targeting rather than the study’s own timing—specifically, accounts were created in batches from late January to mid-March, ads began on March 8, and the analysis does not adjust for account creation date or days active during the ad window.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT ads skew toward lower-income users","Sock puppet audit: income, not race, drives ChatGPT ads","Lower-income accounts see more ChatGPT ads, audit shows","ChatGPT's ad rollout is income-skewed, new audit finds","First phase of ChatGPT ads hits low-income users hardest"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000346,"raw_usage":{"total_tokens":1933,"prompt_tokens":1019,"completion_tokens":914,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":833}},"tokens_in":635,"tokens_out":914,"duration_ms":10670,"temperature":1.0,"reasoning_tokens":833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:38:19.964621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Using the paper’s released data, compare account creation dates across income terciles and re-estimate the ad-exposure regression with days active before March 8 as a control; if the income coefficient drops to zero, the gradient is a timing artifact rather than income-based targeting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the OpenAI usage report from which the prompt corpus’s common formulations and categories are drawn."}],"review_version":1}