Generative AI and Sales Productivity: Field Experiments in Online Retail

Dante Donati; Kaifu Zhang; Lu Fang; Miklos Sarvary; Zhe Yuan

REVIEW 3 major objections 4 minor 1 cited by

Large-scale randomized retail experiments show GenAI raises sales in most workflows — up to 16.3% — worth roughly $5 per consumer per year.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

GenAI deployed in seven retail workflows raised sales by up to 16.3% in randomized experiments, with an annualized incremental value near $5 per consumer, driven mainly by higher conversion rates.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Individual experiments are solid, but mistrust the $5 per consumer headline until annualization and abstract overreach are fixed. the 3 major comments →

arxiv 2510.12049 v6 pith:CKNXNLSG submitted 2025-10-14 econ.GN cs.AIq-fin.EC

Generative AI and Sales Productivity: Field Experiments in Online Retail

Lu Fang , Zhe Yuan , Kaifu Zhang , Dante Donati , Miklos Sarvary This is my paper

classification econ.GN cs.AIq-fin.EC

keywords generative AIfield experimentssales productivitytotal factor productivityonline retailconversion ratefriction reductiontreatment effect heterogeneity

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish, with direct causal evidence, that firm-level adoption of generative AI raises revenue-based productivity in real online retail. It reports randomized field experiments across seven consumer-facing workflows at one large cross-border platform, involving millions of consumers and products. The central result: GenAI increased sales in most workflows, from no detectable effect to 16.3%, and the four workflows with positive effects imply an annual incremental value of roughly $4.6–5 per consumer. Gains show up as higher conversion rates rather than larger baskets, which the authors read as evidence that GenAI reduces search, information, communication, and personalization frictions. This matters because it is large-scale causal evidence on whether GenAI investment pays off in actual purchase behavior, not just task-level efficiency.

Core claim

GenAI adoption increases sales in most of the seven workflows tested, with estimated treatment effects ranging from no detectable impact to a 16.3% sales lift for a pre-sale service chatbot; the four positive workflows together imply roughly $4.6–5 of annual incremental value per consumer, about 5.5–6% of global per-user e-commerce revenue growth in 2023–2024. Because prices and labor and capital inputs were held constant across experimental arms, the paper maps these output gains directly into total factor productivity growth. The mechanism operates on the extensive margin: conversion rates rose by 1–22% across workflows while average cart values did not change, and product return rates and

What carries the argument

The load-bearing design is a set of seven parallel randomized field experiments, each comparing a GenAI-integrated workflow against the platform's standard practice with prices and labor and capital inputs identical across arms; randomization at the consumer level (product level in one workflow) makes each sales difference an unbiased average treatment effect. The paper interprets results through standard Solow growth accounting: with capital and labor fixed, any measured output increase is attributed to total factor productivity. The mechanism probe is the conversion rate measured alongside sales — splitting the extensive margin (more consumers buying) from the intensive margin (cart value

Load-bearing premise

The $4.6–$5 per-consumer annual value rests on Section 4.3's assumptions that each short experiment's sales lift repeats every time a consumer meets that workflow all year and that the four workflows add up without overlap; if novelty fades or workflows cannibalize each other, the headline annual figure fails even though each individual experiment's effect can still be valid.

What would settle it

Run the same workflows for a full year and compare treatment effects in the first versus later months: a decay toward zero falsifies the constancy assumption. Or replace the assumed time multipliers (6, 40.6, 52.1, 365) with the platform's actual average workflow encounters per consumer per year — if real encounter rates are far lower, the annualized $5 value collapses. A third check measures overlap of the treated consumer populations across the four positive workflows; substantial overlap would violate linear additivity.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

If the effects are linearly additive as the paper assumes, every additional GenAI-deployed workflow with positive lift adds its own per-consumer value; the platform's expansion from 7 to over 60 workflows by 2025 implies aggregate gains well beyond the $5 estimate.
The conversion-not-basket result means GenAI expands the market at the extensive margin — converting hesitant, marginal consumers — rather than extracting more spending from existing buyers.
Disproportionate gains for small sellers and novice consumers imply GenAI adoption shifts some platform surplus toward the long tail and less experienced participants, narrowing outcome gaps.
The null and negative advertising results imply domain-specific fine-tuning is a prerequisite for positive returns; untuned foundation models can be value-neutral or value-destroying.
Because measured gains arise with constant inputs and unchanged prices, the paper treats the revenue lift as a conservative floor on GenAI returns that omits any future labor-cost savings.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If friction reduction is the general mechanism, the same experimental design should find larger GenAI lifts where baseline frictions are worst — no service, missing descriptions, underserved languages — and the paper's untested post-purchase workflows (returns, logistics, payments) are the natural next place to look.
The $5 annual figure is a point estimate on a wide band: it likely overstates steady-state value if novelty or learning effects decay, and understates it if habit formation or cross-workflow synergies compound; the paper's own annualization assumptions make the long-run number genuinely open.
In competitive equilibrium, if rival platforms deploy the same tools, this platform's relative advantage should erode and the durable surplus would shift toward consumers in the form of better matches or lower prices — so the welfare reading of the $5 differs from the revenue reading.
The heterogeneity result yields a sharp testable prediction: the same 'less experienced gain more' pattern should appear in other marketplaces and consumer settings, and average effect sizes should shrink as the user population becomes more familiar with AI-assisted shopping.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Individual experiments are solid, but mistrust the $5 per consumer headline until annualization and abstract overreach are fixed.

read the letter

Here's my take: the individual experiments are mostly well-run and the multi-workflow setup is genuinely valuable, but you should not trust the $5-per-consumer headline until the authors fix the annualization and clear up a direct contradiction between the abstract and the main text.

The real contribution is the size and scope: seven randomized evaluations in one large cross-border retailer, millions of consumers, and actual downstream sales and conversion rather than worker-task proxies. The individual ATEs look credible — covariate balance, cohort fixed effects, large samples. The chatbot experiment is the most careful: it includes comparisons against no-service, human-only, and hybrid conditions, and the hybrid-vs-human 11.5% effect is a plausible lower bound. I also give them credit for reporting a clear null for Google ad titles, and for the heterogeneity story: consistent gains for smaller sellers and less experienced buyers. That pattern has practical value.

The aggregate claim is the weak part. Table 6 multiplies each per-experiment ATE by 12/duration, and the footnote assumption — experiment duration equals the typical interval between treatment opportunities — is untested and likely wrong. The search experiment ran nine days; the multiplier is 40.6. If a consumer searches in those languages ten times a year instead of 40, that workflow's contribution drops from $2.63 to $0.65. The chatbot multiplier assumes six contacts per year; once a year would cut its contribution to a quarter. So the $4.96 total is about as solid as those multipliers. They also include the non-significant Marketing Push sales effect and exclude the negative Google Ad effect, with no confidence interval on the total. The right move is to present the aggregate as a scenario or bounding exercise, not as a point estimate.

There's also an internal inconsistency: the abstract says return rates and customer ratings do not deteriorate, but Section 6 says the authors lack data on product returns and retention. That has to be fixed before publication. Smaller issues: two workflows rely on the platform's internal metrics, which the authors couldn't verify; that's fine as secondary evidence, but they are treated as comparable to the randomized results. And the TFP mapping is an interpretation of the constant-input assumption, not something separately measured. None of this kills the core result — the individual ATEs are the contribution — but the headline needs to be rewritten.

This paper is for economists and applied researchers working on AI and productivity, plus retail practitioners. It deserves a serious referee. I'd send it to review, but I'd expect major revisions on the aggregate claims and the abstract.

Referee Report

3 major / 4 minor

Summary. The paper reports seven randomized field experiments conducted on a large cross-border e-commerce platform between September 2023 and June 2024, in which GenAI was introduced into consumer-facing workflows: pre-sale service chatbot, search query refinement, product description generation, marketing push messages, Google advertising title optimization, chargeback defense, and live chat translation. For five workflows the authors have granular transaction data and estimate OLS treatment effects with cohort fixed effects. They find a large positive effect for the pre-sale chatbot (16.3% sales increase), smaller effects for search query refinement (2.93%) and product descriptions (2.05%), a statistically insignificant positive effect for marketing push, and a negative insignificant effect for Google ad titles. The authors interpret the sales gains, with prices and inputs held constant, as TFP improvements, and they aggregate the four positive workflows into an annualized incremental value of approximately $4.6–$5 per consumer. Heterogeneity analyses show larger effects for less experienced consumers and smaller sellers.

Significance. If the individual ATEs are taken at face value, this is one of the largest collections of randomized evidence on GenAI in a live retail environment, covering millions of consumers and products across multiple workflows. The explicit focus on revenue-based outcomes, conversion margins, and demand-side frictions is a valuable complement to the worker-productivity literature. The main quantitative headline, however, is the annualized value, and it rests on assumptions about exposure frequencies, effect persistence, and linear additivity that are not supported by data in the manuscript. The underlying experimental estimates are likely sound; the aggregate claim needs substantial additional justification or careful reframing.

major comments (3)

[§4.3, Table 6] The annualized headline is directly proportional to the time multipliers (6.0, 40.6, 52.1, 365.0) in Table 6, and the only justification is the footnote that experiment duration equals the typical interval between treatment opportunities. No frequency data support this. The Search Query experiment covered Arabic/Japanese/Polish searches over nine days; multiplying by 40.6 assumes a representative consumer searches in one of these languages every nine days year-round. If the true frequency is 10 times per year, that workflow's contribution drops from $2.63 to ~$0.65 and the total from $4.96 to ~$2.98. Similarly, if pre-sale chatbot contact occurs once per year, its $1.64 contribution falls to $0.27. The calculation also assumes constant effects and linear additivity. Please provide platform exposure frequencies, or present the annualized numbers only as an illustrative sensitivity analysi
[§4.3, Table 6 vs Table 4] Table 6 includes Marketing Push Message with annualized contribution $0.15, but the underlying sales ATE (0.000402, SE 0.000812 in Table 4) is not statistically significant. The abstract's 'four GenAI applications with positive sales effects' therefore overstates the evidence: only three sales effects are statistically distinguishable from zero among the detailed-data workflows. Excluding the push contribution changes the aggregate to $4.81/$4.48, which is still close to '$5,' but the inclusion should be disclosed and the headline should not be stated as if based on four significant effects. The Google Advertising Title null/negative effect is excluded; since it is insignificant this is defensible, but the exclusion rule should be stated as 'statistically significant positive effects' or 'point estimates' consistently.
[§3.4, §4.1, Appendix C] For Chargeback Defense and Live Chat Translation, the paper relies on platform-internal estimates ('15% defense success rate increase', '5.2% consumer satisfaction increase') because raw data were not provided. The text acknowledges the chargeback estimate 'couldn't be verified.' These are not randomized experimental estimates in the same sense as the other five workflows, yet the abstract and introduction count them among 'seven workflows' and describe the paper as providing causal evidence across workflows. Please either obtain and analyze the underlying data, or clearly label these as platform-reported descriptive metrics and restrict causal claims to the five workflows with granular data.

minor comments (4)

[§3.2, Abstract] The phrase 'total factor productivity improvements' overstates the mapping. Holding inputs and prices fixed, a demand-side sales increase raises revenue per input, but this is not necessarily a technical efficiency change; 'revenue-based productivity' is the more precise term used elsewhere. Consider softening the TFP language.
[§4.3] The comparison of $4.6–$5 to '5.5–6% of per-user revenue growth' cites Statista but does not define the comparison universe or period precisely; add details.
[Table 4] Table 4 footnote says standard errors are in brackets but displays parentheses; also Table C5 reports SE 0.000816 for the Marketing Push sales coefficient while Table 4 reports 0.000812. Harmonize.
[Throughout] Typos: 'No Serice' in Table C1 note; 'cross-boarder' in Table 1; 'The it was conducted' in §5.2; 'Live Chat T ranslation' in a section heading. Also define the 'lower-bound' value in Table 6; it is from Table C1 but not identified in the table.

Circularity Check

0 steps flagged

No significant circularity: randomized treatment effects are self-contained; annualization and TFP labeling are explicit extrapolation/accounting assumptions, not fitted predictions.

full rationale

The paper's central estimates come from randomized field experiments comparing GenAI-enhanced workflows with control workflows. The treatment effects in Table 4 are estimated from within-experiment random assignment (Equation 2), so they do not reduce to the inputs of the model. The $5 per consumer aggregate in Table 6 is an explicit back-of-the-envelope calculation: each estimated ATE is multiplied by an annualization factor, and the footnote states the assumption that 'each experiment's duration reflects the typical interval between treatment opportunities for a representative consumer.' This multiplier is an untested extrapolation assumption, not a parameter fitted to the outcome and then relabeled as a prediction; it could be wrong, but it is not circular. Similarly, the TFP interpretation follows from the stated Solow accounting identity d ln Y = d ln A when K and L are assumed constant (Section 3.2). This is an interpretive assumption, not a derivation in which the conclusion is built into the definition of the treatment effect. The paper's self-citations (e.g., Fang et al. 2024; Sun et al. 2024; Zhou et al. 2025) appear in literature-review and heterogeneity-interpretation passages, but they are not used to justify the core ATEs or the aggregate calculation. The randomized design makes the main empirical content self-contained. Therefore no load-bearing circular step is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central quantitative results rest on measured treatment effects, not on a theoretical derivation. The main assumptions are economic-modeling choices and an aggressive annualization procedure with chosen multipliers.

free parameters (4)

Annualization time multiplier, Pre-sale Service Chatbot = 6.0
Table 6 annualizes a two-month experiment by 6; chosen from experiment duration, not fitted, but directly scales the $1.64 annual value.
Annualization time multiplier, Search Query Refinement = 40.6
Table 6 annualizes nine-day sub-experiments by 365/9 ≈ 40.6; assumes a representative consumer receives this treatment about every nine days all year.
Annualization time multiplier, Product Description = 52.1
Table 6 annualizes a one-week experiment by 365/7 ≈ 52.1; assumes weekly exposure throughout the year.
Annualization time multiplier, Marketing Push Message = 365.0
Table 6 annualizes a one-day experiment by 365; assumes daily push-message exposure and constant effect for every day of the year.

axioms (5)

domain assumption Cobb-Douglas production function with constant factor shares and TFP residual
Section 3.2, Eq. (1) and Assumption 4; used to interpret sales gains as TFP growth. This is a model choice, not derived from data.
domain assumption Labor, capital, prices, factor shares and utilization are held constant across GenAI and control conditions
Section 3.2, Assumptions 1-5; Section 3.3 asserts no labor displacement (except 3-5 outsourced workers in Chargeback Defense) and negligible compute costs, but no direct measurement of utilization or factor shares is provided.
ad hoc to paper Treatment effects are constant over time and linearly additive across workflows
Section 4.3 explicitly states: 'This calculation assumes that the treatment effects observed during the experimental period remain constant over time' and 'assumes that effects across workflows are linearly additive.' This is load-bearing for the headline $5-per-consumer total.
domain assumption SUTVA/no interference across consumers and workflows
Section 3.3 randomizes at the consumer/product level with less than 1% overlap; treatment of one unit is assumed not to affect outcomes of another. This is standard but not directly tested.
ad hoc to paper Platform internal estimates for Chargeback Defense and Live Chat Translation are accurate
Section 3.4 and Appendix C: for these workflows the authors report 'findings estimated by the platform’s internal data science team' and state 'we couldn’t verify.' Claims for these workflows rest entirely on unverified internal analysis.

reviewed 2026-08-04 · how reviews work

0 comments

Cite this review

Pith. "Pith review of Generative AI and Sales Productivity: Field Experiments in Online Retail." pith.science (2026). https://pith.science/paper/CKNXNLSG

@misc{pith2026251012049,
  author       = {Pith},
  title        = {Pith review of: Generative AI and Sales Productivity: Field Experiments in Online Retail},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKNXNLSG}},
  note         = {Machine review of arXiv:2510.12049}
}

Share X Bluesky LinkedIn Reddit HN

abstract

We quantify the short-term impact of Generative Artificial Intelligence (GenAI) on sales performance through a series of large-scale randomized field experiments involving millions of users and products at a leading cross-border online retail platform. Over 2023-2024, the platform integrated GenAI into seven business workflows spanning customer service, consumer-product matching, advertising, and seller services. We find that GenAI adoption increases sales in most workflows, with effects ranging from no detectable impact to $16.3\%$, depending on GenAI's marginal contribution relative to baseline firm practices. Across the four GenAI applications with positive sales effects, the implied annual incremental value is roughly $\$5$ per consumer$-$an economically meaningful impact given the retailer's scale and the early stage of GenAI adoption. The gains operate primarily through higher conversion rates rather than larger cart values, consistent with GenAI improving the shopping experience by reducing search, information, communication, and personalization frictions. Importantly, these effects are not associated with worse post-purchase outcomes, as product return rates and customer ratings do not deteriorate. Finally, we document substantial demand-side heterogeneity, with larger gains for less experienced consumers. Our findings provide novel, large-scale causal evidence on how GenAI shapes sales productivity in online retail, highlighting both its immediate value and broader potential.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Intelligence Impact Quotient (IIQ): A Framework for Measuring Organizational AI Impact
cs.AI 2026-05 unverdicted novelty 6.0

IIQ is a new 0-1000 normalized index that measures organizational AI impact via a novelty-weighted, time-decayed token stock plus usage frequency, leverage, complexity, and autonomy factors.

Reference graph

Works this paper leans on

4 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[1]

The simple macroeconomics of AI

Acemoglu, Daron (2025). “The simple macroeconomics of AI”. In: Economic Policy 40.121, pp. 13–

2025
[58]

Low-skill and high-skill automation

Acemoglu, Daron and Pascual Restrepo (2018). “Low-skill and high-skill automation”. In: Journal of Human Capital 12.2, pp. 204–232. Autor, David H., Frank Levy, and Richard J. Murnane (2003). “The skill content of recent tech- nological change: An empirical exploration”. In: The Quarterly Journal of Economics 118.4, pp. 1279–1333. Bai, Jie, Maggie X Chen,...

Pith/arXiv arXiv 2018
[2017]

Modeling consumer footprints on search engines: An interplay with social media

Ghose, A., P. G. Ipeirotis, and B. Li (2019). “Modeling consumer footprints on search engines: An interplay with social media”. In: Management Science 65.3, pp. 1363–1385. Ghose, Anindya, Avi Goldfarb, and Sang Pil Han (2014). “How Is the Mobile Internet Different? Search Costs and Local Activities”. In: Information Systems Research 24.3, pp. 613–631. Gol...

2019
[2455]

Generative AI in Real-World Workplaces: The Second Microsoft Report on AI and Productivity Research

Microsoft (2024). “Generative AI in Real-World Workplaces: The Second Microsoft Report on AI and Productivity Research”. In. Milgrom, Paul and Steven Tadelis (2018). “How Artificial Intelligence and Machine Learning Can Impact Market Design”. In: NBER Working Paper No. 24282 . Nguyen, Nhan and Sarah Nadi (2022). “An empirical evaluation of GitHub copilot’...

Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.

[1] [1]

The simple macroeconomics of AI

Acemoglu, Daron (2025). “The simple macroeconomics of AI”. In: Economic Policy 40.121, pp. 13–

2025

[2] [58]

Low-skill and high-skill automation

Acemoglu, Daron and Pascual Restrepo (2018). “Low-skill and high-skill automation”. In: Journal of Human Capital 12.2, pp. 204–232. Autor, David H., Frank Levy, and Richard J. Murnane (2003). “The skill content of recent tech- nological change: An empirical exploration”. In: The Quarterly Journal of Economics 118.4, pp. 1279–1333. Bai, Jie, Maggie X Chen,...

Pith/arXiv arXiv 2018

[3] [2017]

Modeling consumer footprints on search engines: An interplay with social media

Ghose, A., P. G. Ipeirotis, and B. Li (2019). “Modeling consumer footprints on search engines: An interplay with social media”. In: Management Science 65.3, pp. 1363–1385. Ghose, Anindya, Avi Goldfarb, and Sang Pil Han (2014). “How Is the Mobile Internet Different? Search Costs and Local Activities”. In: Information Systems Research 24.3, pp. 613–631. Gol...

2019

[4] [2455]

Generative AI in Real-World Workplaces: The Second Microsoft Report on AI and Productivity Research

Microsoft (2024). “Generative AI in Real-World Workplaces: The Second Microsoft Report on AI and Productivity Research”. In. Milgrom, Paul and Steven Tadelis (2018). “How Artificial Intelligence and Machine Learning Can Impact Market Design”. In: NBER Working Paper No. 24282 . Nguyen, Nhan and Sarah Nadi (2022). “An empirical evaluation of GitHub copilot’...

Pith/arXiv arXiv 2024

Generative AI and Sales Productivity: Field Experiments in Online Retail

The reading

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it