Anthropic’s randomized study raises an uncomfortable operational question for AI-assisted engineering teams: what happens when the tool that removes friction also removes the repetitions through which engineers learn to diagnose, understand, and verify systems? For teams doing GEO work, that question matters because the job is not merely producing implementation quickly; it is being able to inspect whether a generated change, retrieval path, structured output, or integration is actually doing what it should.
The study does not establish a universal rule that AI makes developers worse. It does offer a more useful and narrower warning: when developers were learning an unfamiliar Python library, AI assistance was associated with lower immediate mastery, while its modest completion-time advantage was not statistically significant. The practical response is not an anti-AI policy. It is to preserve deliberate practice where the team is acquiring a skill it will later need to exercise judgment over.
The result is about learning, not routine throughput
Anthropic recruited 52 mostly junior software engineers who had used Python at least weekly for more than a year, were somewhat familiar with AI coding assistance, and were unfamiliar with the Trio Python library. Participants completed a self-guided coding task involving two Trio features, then took a quiz. The AI assistant could access their code and produce correct code when asked. Anthropic
The outcome is stark enough to deserve attention. The AI group averaged 50% on the quiz, versus 67% for participants who coded by hand: a 17% difference, described by Anthropic as nearly two letter grades. The AI group finished about two minutes faster on average, but that time difference was not statistically significant. Anthropic
This is not evidence that teams should measure productivity by quiz results. It is evidence that “the task was completed” and “the engineer now understands the system” are different outcomes. That distinction is especially consequential when a team is introducing a framework, library, platform capability, or technical pattern that will remain in production after the first implementation ships.
Anthropic’s own framing makes the boundary clear. Its prior observational research found that AI could reduce the time for some work tasks by 80%, but those observations concerned tasks where participants already had relevant skills. The newer experiment looked at what happened during acquisition of a new skill. Anthropic says it is possible for AI to accelerate work on well-developed skills while hindering acquisition of new ones, although more research is needed. Anthropic
That is the useful operating distinction: mature work versus learning work. Teams should not treat them as the same category simply because both arrive as tickets in the same backlog.
Debugging is the warning signal
The assessment covered debugging, code reading, code writing, and conceptual understanding. Anthropic weighted debugging, reading, and conceptual questions most heavily because it considered them central to oversight of increasingly AI-generated code. The largest performance gap between groups appeared on debugging questions. Anthropic
For GEO practitioners, that result is more important than the headline score difference. A team may use AI to draft code, transform content, create integrations, or produce tests. But its practical assurance comes from someone being able to spot an invalid assumption, trace an unexpected output, recognize a mismatch between a system’s behavior and its intended design, and determine why a change failed.
Those are debugging and comprehension activities, not generation activities. If an engineer can obtain a plausible implementation without encountering its failure modes, the team may gain speed on the immediate task while losing exposure to the reasoning needed to assess the next output. Anthropic hypothesizes that participants in the no-AI group may have improved debugging skills by independently resolving more errors, including Trio-concept errors that mapped directly to evaluated topics. Anthropic
That hypothesis is not proof that every error is educational or that teams should manufacture friction. It is a reminder that error resolution is often not waste. On unfamiliar technical terrain, it can be the work through which an engineer gains the mental model that later makes fast, correct judgment possible.
How people used AI mattered
The study’s qualitative analysis is the most actionable part, but it must be read carefully. Anthropic manually annotated screen recordings and identified interaction patterns. It explicitly says this analysis does not establish a causal relationship between a pattern and learning outcomes; it identifies behaviors associated with different outcomes. Anthropic
Low-scoring patterns included full AI delegation, progressively increasing reliance on AI, and iterative AI debugging in which participants asked the assistant to solve or verify problems instead of clarifying their own understanding. Average quiz scores in these low-scoring patterns were below 40%. Anthropic
Higher-scoring patterns looked different. Some participants generated code and then asked follow-up questions to improve their understanding. Others requested generated code alongside explanations. A third group asked conceptual questions and used the resulting understanding to complete the task, independently resolving errors along the way. Anthropic classified these as high-scoring patterns with average quiz scores of at least 65%. Anthropic
The implication is not that teams should ban generated code. It is that generated code should not be the end of the interaction when the work is educational. The behavior to cultivate is a loop: make or inspect the implementation, ask what mechanism explains it, test the explanation against the system, and diagnose the result independently before asking the model to repair it.
A practical policy for AI-assisted GEO engineering
I would separate work by the capability the team needs to retain. On familiar, repetitive work, optimize for throughput. On work that introduces a new technical capability or creates a verification burden the team will own, optimize for comprehension as well as delivery.
- Label learning-critical tasks. When an engineer is working with an unfamiliar library, system, or pattern, state that understanding is part of the deliverable, not an optional personal benefit.
- Require an explanation before acceptance. The engineer should be able to explain the relevant behavior, assumptions, failure modes, and why the implementation fits the task in their own terms.
- Keep one diagnostic step human-led. Before using AI to fix an error, ask the engineer to form a hypothesis, inspect the relevant evidence, and describe the next test.
- Review reasoning as well as diffs. For learning-critical changes, a review should test whether the author can account for the code, not only whether the code appears acceptable.
- Use AI as a tutor where possible. Prompt for conceptual explanations, trade-offs, and questions that check understanding rather than requesting a completed solution alone.
These are operating choices, not conclusions directly tested by Anthropic. The study tested a short learning task, not a team policy. But they follow the study’s core distinction between AI used to replace thinking and AI used to support comprehension.
Do not overstate what this study proves
The evidence is meaningful but preliminary. The sample was relatively small, participants were mostly junior engineers, and comprehension was measured shortly after the task. Anthropic says the study does not resolve whether immediate quiz performance predicts longer-term skill development, whether effects change as engineers gain fluency, how AI compares with human assistance during learning, or whether the finding applies beyond coding. Anthropic
It also did not test agentic coding products. Anthropic notes that the setup differed from products such as Claude Code and says it expects the effects of such programs on skill development may be more pronounced. That is an expectation, not a measured result in this experiment. Anthropic
Stephen Turner reaches a compatible practical interpretation: foundational competencies may need to come before broad AI-tool use, so practitioners can critically evaluate tool output. But Turner also says he is not sure exactly how educators should design for that balance. blog.stephenturner.us The uncertainty is worth retaining. There is no validated playbook here.
Still, the management implication is clear enough. If GEO teams want people capable of meaningful oversight, they cannot define successful AI adoption as maximum delegation. They need to decide where speed is the goal and where the goal is building engineers who can tell when the generated answer is wrong. Anthropic’s study suggests those goals can diverge precisely when a person is learning something new. Anthropic