This site is undergoing review. Some annotations were human-generated, some AI-generated — all are being verified.

The Uneven Impact of Generative AI on Entrepreneurial Performance

Otis, Clarke, Delecourt, Holtz, Koning

2023Harvard Business School Working Paper141 citations

Experimental evidenceCausal

LLM / Generative AIDeveloping economiesHuman-AI collaborationDecision-making

View Repository DOI: 10.31219/osf.io/hdjpk

Abstract

Scalable and low-cost AI assistance has the potential to improve firm decision-making and economic performance. However, running a business involves a myriad of open-ended problems, making it difficult to know whether recent AI advances can help business owners make better decisions in real-world markets. In a field experiment with Kenyan entrepreneurs, we assessed the impact of AI advice on small business revenues and profits by randomizing access to a GPT-4-powered AI business assistant via WhatsApp. While we are unable to reject the null hypothesis that there is no average treatment effect, we find the treatment effect for entrepreneurs who were high performing at baseline to be 0.27 standard deviations greater than for low performers. Sub-sample analyses show high performers benefited by just over 15% from the AI assistant, whereas low performers did about 8% worse. This increase in performance inequality does not stem from differences in the questions posed to or advice received from the AI, but from how entrepreneurs selected from and implemented the AI advice they received. More broadly, our findings demonstrate that generative AI is already capable of impacting—though in uneven and unexpected ways—real, open-ended, and unstructured business decisions.

Summary

Otis, Clarke, Delecourt, Holtz, and Koning conduct a field experiment randomizing access to a GPT-4-powered AI business assistant among 640 Kenyan small business entrepreneurs to study the impact on firm revenues and profits over five months.

Main Finding

Access to a GPT-4-powered AI business assistant had no average effect on firm revenues and profits for Kenyan entrepreneurs, but masked substantial heterogeneity: high performers experienced approximately 15-18% revenue increases while low performers saw approximately 9-10% revenue decreases. This differential impact stemmed not from differences in questions asked or advice received, but from which AI suggestions entrepreneurs selected and implemented—high performers adopted tailored, specific advice while low performers implemented generic strategies like price cuts and advertising.

Primary Datasets

Self-reported business performance surveys (7 waves: 3 pre-treatment, 4 post-treatment); WhatsApp conversation logs between entrepreneurs and AI assistant; Post-treatment survey responses on business changes

Secondary Datasets

Meta advertisement platform data for recruitment

Key Methods: Field experiment with stratified randomization by pre-treatment performance quartiles and gender. OLS regression controlling for pre-treatment performance, time fixed effects, stratum fixed effects, and double-LASSO selected covariates. Heterogeneous treatment effects analysis by baseline performance. Text analysis using embeddings, random forests, and causal identification via Pearl's backdoor criterion.
Sample Period: 2023
Geographic Coverage: Kenya (44 of 47 counties represented)
Sample Size: 640 Kenyan SMB entrepreneurs; 4,434 performance observations across 7 survey waves; 4,810 WhatsApp messages sent to AI (1,392 substantive business messages); 4,207 AI-generated suggestions
Level of Analysis: Individual, Firm
Occupation Classification: None
Industry Classification: None

Notes

Harvard Business School Working Paper 24-042 [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes). [Claude classification]: Field experiment in Kenya with 640 SMB entrepreneurs. Control group received ILO business training guides (interpreted as placebo). Text analysis using GPT-4 embeddings (3,072 dimensions) and random forests to analyze questions, AI responses, and business changes. Novel application of Pearl's backdoor criterion to identify causal impact of AI advice content on business changes. Study period: May-November 2023. Cost per participant: few dollars (vs hundreds for typical training programs). Powered by GPT-4 via WhatsApp with custom system prompt. Both treated and control groups received brief online training (5-10 minutes).