This site is undergoing review. Some annotations were human-generated, some AI-generated — all are being verified.

The Heterogeneous Productivity Effects of Generative AI

Kreitmeir, Raschky

2024arXiv pre-print6 citations

Observational labor marketCausal

LLM / Generative AISoftware / codingJunior / entry-levelSenior / older workersHuman-AI collaborationAugmentation vs. substitution

View Repository DOI: 10.2139/ssrn.4745624

Summary

Kreitmeir and Raschky use difference-in-differences with daily GitHub activity data from over 36,000 developers in Italy and control European countries to study how Italy's sudden ChatGPT ban affected coding productivity, finding heterogeneous effects by experience level

Main Finding

The ChatGPT ban increased output quantity and quality by approximately 10% for less experienced developers in the two days following the ban, while experienced developers showed no systematic productivity change except small negative effects on routine tasks (issue resolution), suggesting ChatGPT may hinder rather than help less experienced workers on complex coding tasks

Primary Datasets

GitHub Archive (public event data); GitHub GraphQL API (user metadata)

Secondary Datasets

Google Trends

Other

Google Trends (VPN searches); TOR Metrics (encrypted routing usage); GitHub package repositories (PyPI, CRAN, JuliaRegistries, Awesome Lists)

Key Methods: Difference-in-differences exploiting Italy's ChatGPT ban as natural experiment; event-study specification; user and repository fixed effects; analysis at user-day and repository-user-day levels
Sample Period: 2023
Geographic Coverage: Italy, Austria, France, Spain (European countries)
Sample Size: 36,358 unique GitHub users observed over 8 work days (290,864 user-day observations); repository-level analysis: 11,938 users × 4,627 repositories
Level of Analysis: Individual
Occupation Classification: None
Industry Classification: None

Notes

arXiv:2403.01964 [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness [Claude classification]: ArXiv:2403.01964; Uses GitHub Archive and TOR/Google Trends data; Focus is on software developers (coding); Paper uses synthetic control method for circumvention analysis (VPN/TOR usage); Sample restricted to work days around Easter 2023; Multiple hypothesis testing corrections applied (Romano-Wolf); Repository-level analysis with repository×user fixed effects as robustness