🔍 Read the full analysis: How Much Data Science Skill Remained Without ChatGPT? on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A randomized trial at Boston Consulting Group found that consultants using ChatGPT outperformed a control group on coding, statistics and prediction tasks. After ChatGPT was removed, researchers found no evidence that the users performed better on technical questions, suggesting the observed boost did not establish lasting skill gains.
A randomized trial at Boston Consulting Group found that consultants given ChatGPT performed better than a control group on coding, statistics and prediction tasks, but researchers found no evidence of an advantage when participants later answered technical questions without the chatbot. Published in Nature Human Behaviour on October 26, 2026, the study bears on whether workplace AI assistance builds employees’ independent skills or mainly helps them perform while the tool is available.
The experiment involved consultants working on data-science tasks described by the researchers as outside many participants’ skill sets. The ChatGPT group received 15 to 20 minutes of prompting instruction and used GPT-4. The control group received guidance on using Google and resources including Stack Overflow and Khan Academy. Participants completed two of three 90-minute tasks: Python data cleaning, statistical analysis or a prediction exercise involving international football matches.
Among the 487 consultants who submitted work on both assigned tasks, the ChatGPT group scored 49 percentage points higher than the control group on coding, 20 points higher on statistics and 17 points higher on prediction. The researchers compared scores with results from 44 BCG data scientists who completed the tasks without AI. The ChatGPT users still scored below that expert benchmark on average across all three tasks.
After the work exercises, participants answered technical questions on related subjects without ChatGPT. The authors report that the ChatGPT group did not perform better than the control group on this test. They describe the task gains as temporary upskilling. The study also reported greater confidence among ChatGPT users that the tool could solve problems it could not; the authors characterize that result as exploratory.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Task Performance Is Not Skill Transfer
The findings distinguish between doing a task with AI and acquiring the knowledge to do it independently. For employers, that difference affects how they evaluate training, staffing and the reliability of work produced with AI assistance. Higher output during tool use may show that workers can complete more demanding assignments with support; it does not, by itself, show that they have learned the underlying technical skills.
The trial does not establish that AI cannot teach people. It tested a particular set of tasks, a short training session and a test administered after the tool was removed. Still, its results caution against treating immediate productivity gains as proof of durable learning. Organizations using AI in technical work may need separate ways to assess independent competence, alongside checks on the quality of AI-assisted work.
Popular right now
Vitamix VX1 Professional-Grade Blender, 64 oz, Black
Philips 2200 Series Fully Automatic Espresso Machine with Classic Milk Frother
Dyson Ball Animal 3 Upright Corded Vacuum
As an affiliate, we earn on qualifying purchases.
How the BCG Trial Was Designed
The study was preregistered on March 13, 2024, before data collection, and the experiment took place at BCG in late March and early April of that year. Of 986 consultants allocated to the study, 573 began the survey and 487 submitted work on both tasks, forming the main analysis sample. All had at least a bachelor’s degree; the authors say most reported no coding experience.
Researchers designed the exercises so ChatGPT could offer useful guidance but could not correctly answer every question on its own. BCG data scientists developed the tasks with OpenAI’s Economic Impacts Research team. The design included both the expert comparison group and the later no-AI questions, allowing the researchers to examine task performance with the tool and performance after its removal.
The study’s title, “Generative AI and the temporary upskilling of knowledge workers,” reflects the authors’ interpretation. They compare AI assistance to an exoskeleton: it can extend what a worker can do while in use, without necessarily changing the worker’s underlying capability.
““find no evidence” that the ChatGPT group performed better on the technical questions without the tool.”
— The study authors
Limits of the Skill Test
The post-task assessment covered five technical questions about probabilities, machine learning and coding. The reported result is no evidence of a difference between groups on that assessment; it does not prove that no participant learned anything or that no learning could persist under different conditions. The supplied study summary does not provide enough detail to assess how well this short test captures longer-term skill development.
The results also come from one consulting firm, specific tasks, GPT-4 and a brief training session. They may not generalize to other occupations, AI systems or longer instruction. Although the paper reports that ChatGPT users became more confident in the tool’s ability to solve problems it could not, the authors call that analysis exploratory, so it should not be read as a definitive conclusion about users’ confidence more broadly.
Testing Whether Skills Persist
The study summary does not identify a scheduled follow-up test or a future milestone. A key next step for research would be to test participants after a longer interval, using broader measures of independent performance, and compare different kinds of AI training. That could help determine whether changes in instruction or repeated practice produce skill gains that the short post-task assessment did not detect.
For employers, the immediate practical question is how to assess work both with AI and without it. The trial’s results support treating those as separate measures: one records what employees can accomplish with assistance, while the other tests what they can do independently. Whether AI-based workplace training produces durable technical expertise remains unsettled.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. After ChatGPT was removed, researchers found no evidence of better performance by that group on technical questions.
How much higher were the task scores?
The ChatGPT group scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction than the control group. Those are differences between groups, not claims that the users mastered the subjects.
Did ChatGPT users reach the data-science experts’ level?
No. The group scored below the average of 44 BCG data scientists on all three tasks. The study summary notes that some participants with coding experience came close to the experts on coding and statistics.
Does the study show that ChatGPT cannot teach skills?
No. It tested a short prompting session, specific work tasks and a brief assessment after tool use. The findings show no demonstrated advantage on that follow-up test, not that learning is impossible under all forms of AI training.
Source: ThorstenMeyerAI.com
