🔍 Read the full analysis: Consultants Worked Better At Data Science With ChatGPT on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
In a randomized trial at Boston Consulting Group, consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. The advantage did not show up in a later technical test without ChatGPT, and the researchers describe the skill gains as temporary.
A randomized trial involving consultants at Boston Consulting Group (BCG) found that those given ChatGPT and brief training performed better than a control group on three data science tasks, but showed no detectable advantage on a later test without the chatbot. Published in Nature Human Behaviour on October 26, 2026, the study suggests AI assistance can expand what workers complete while they have access to it, without necessarily teaching them to perform the same work independently.
The trial assigned 986 consultants to either a ChatGPT group or a control group that could use Google and sites including Stack Overflow and Khan Academy. The ChatGPT group received 15 to 20 minutes of prompting instruction and used GPT-4. The main analysis covered 487 participants who submitted work on both assigned tasks; 573 people began the survey. Most participants, according to the authors, reported no coding experience.
Participants completed two of three 90-minute assignments: a Python task involving data cleaning and merging, a statistics exercise, or a prediction task about international football matches. The researchers said the tasks were designed so ChatGPT could offer useful guidance but could not correctly answer the questions on its own. A separate group of 44 BCG data scientists completed the assignments without AI as an expert benchmark.
Compared with the control group, consultants using ChatGPT scored 49 percentage points higher in coding, 20 points higher in statistics and 17 points higher in prediction. The ChatGPT group also finished coding and prediction faster on average, by about nine and 14 minutes respectively; the difference in time for statistics was not significant. Across the full group, however, participants using ChatGPT remained below the data scientists’ average on all three tasks.
The trial in motion · BCG consultants · 2024
With ChatGPT: close to the experts. Without it: no measurable gain.
A randomized trial with a control group that used Google and Stack Overflow instead of ChatGPT.
- 1 · Training15–20 minutes on prompting ChatGPT
- 2 · TasksTwo 90-minute data science tasks, with ChatGPT
- 3 · ScoresGap to 44 data scientists closes, not fully
- 4 · Tool removedNo ChatGPT from here on
- 5 · TestFive technical questions
Coding task score
0 = data scientists’ average
ChatGPT on+49 pp on codingStill below the data scientists on average
Knowledge test without ChatGPT
ChatGPT group vs. control, five questions
ChatGPT off−7.7 to +4.8 ppNone significant · joint test p = 0.531
Source: Wiles, Krayer et al., “Generative AI and the temporary upskilling of knowledge workers”, Nature Human Behaviour (2026); authors’ manuscript, Sections 2.1 and 2.3, Tables B7 and B9. Bars scaled to the five published differences.
Graphic: Powered by Thorsten Meyer AI
Work Improved, Skills Did Not Persist
The findings distinguish AI-assisted performance from independent expertise. A worker may be able to complete more demanding work while using a chatbot, but that result does not by itself show that the worker has learned the underlying skill. In this trial, the performance gains were measured while participants had access to ChatGPT; the subsequent no-tool test did not show a corresponding knowledge advantage.
That distinction matters for employers making decisions about training, staffing and review. If companies treat work completed with AI as proof that employees can perform it unaided, they may overestimate their workforce’s skills or miss the need for human quality checks. The study does not establish how every workplace or AI system will perform, but it offers evidence that tool-enabled output and retained capability should be measured separately.
The study also reported a possible downside in participants’ judgments: the ChatGPT group became more confident that the system could solve problems it could not. The authors describe that analysis as exploratory, so it should not be treated as a settled finding about all users. Still, it points to a practical concern: users may need to evaluate AI answers, not simply rely on a tool’s apparent fluency.
Products Worth Considering
Amazon Kindle Paperwhite (12th Generation, 2024), 16GB, 7-Inch Display, Raspberry
Apple iPad 11-inch (A16), 128GB, Wi-Fi, Silver
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, IPX4, Black
As an affiliate, we earn on qualifying purchases.
Inside the BCG Experiment
The trial was conducted at BCG in late March and early April 2024 and preregistered on March 13, 2024, before data collection. The study was published more than two years later in Nature Human Behaviour. Its authors compared performance against data scientists who completed the same tasks without AI, giving the experiment an expert reference point in addition to its control group.
After completing their assignments, participants answered five technical questions about topics including probability, machine learning and coding. Neither group could use ChatGPT for this follow-up. The researchers report finding no evidence that the ChatGPT group performed better than the control group on those questions. They characterize the upskilling observed during the tasks as temporary.
Results varied by task and experience. Consultants the paper classified as competent coders who used ChatGPT came within five percentage points of the data scientists on coding and statistics; the authors found no evidence of a difference from the experts on those two tasks for this subgroup. On prediction, the subgroup remained 20 points away. Those subgroup results do not change the broader result that the ChatGPT group as a whole scored below the expert benchmark.
“The authors find no evidence that the ChatGPT group did better on the technical questions answered without the tool.”
— The study authors, as described in the paper
Limits of the Findings
The experiment involved consultants at one company, a particular version of ChatGPT and a short period of training. It does not establish whether similar effects would occur in other occupations, organizations, tasks or with newer AI systems. Nor does the design show whether longer training or repeated practice with feedback could produce skills that persist after AI access ends.
The follow-up consisted of five technical questions, not a longer-term assessment of workplace performance or retained knowledge. The available results also do not establish how often AI-assisted work would contain consequential errors in ordinary company settings. The confidence finding is explicitly exploratory. The study therefore supports a measured conclusion about this trial, rather than a universal claim that AI can never teach skills.
Questions for Workplace Training
The study’s immediate implication for employers is to distinguish output produced with AI from skills employees can demonstrate without it. Companies using chatbots for technical work can assess both, while setting review procedures appropriate to the task. The paper does not prescribe a particular training or oversight model.
Further research would be needed to test whether longer instruction, hands-on practice or structured feedback changes the results. The authors’ findings also leave open how well the performance gains translate beyond this BCG trial. No additional study or follow-up milestone is identified in the source material.
Key Questions
What did the BCG trial find?
Consultants using ChatGPT scored 49 percentage points higher in coding, 20 points higher in statistics and 17 points higher in prediction than a control group using web resources. They remained below the average of 44 BCG data scientists across the tasks.
Did consultants retain the skills after ChatGPT was removed?
The study reports no evidence of an advantage for the ChatGPT group on five technical questions answered without the tool. The authors describe the observed upskilling as temporary; the test does not settle whether other forms of longer-term training might have different results.
How was the study conducted?
Consultants were assigned to ChatGPT or a control group using Google and sites such as Stack Overflow. They completed two of three 90-minute data science tasks, and their results were compared with those of 44 BCG data scientists who worked without AI.
Does the study show that ChatGPT cannot teach skills?
No. It found no detectable advantage on a short follow-up test without ChatGPT after this trial’s brief training and task experience. The results do not determine what might happen with longer instruction, repeated practice or other settings.
Source: ThorstenMeyerAI.com
