- GitHub ran a two-phase randomized controlled trial: 243 developers with 5+ years of Python recruited, 202 valid submissions (104 with Copilot, 98 without), graded by 10 unit tests and a blind quality rubric.
- Developers with Copilot access were 53.2% more likely to pass all 10 unit tests (p<0.01), and 25 blind reviewers produced 1,293 anonymized reviews scoring readability, reliability, maintainability, and conciseness.
- Quality gains were real but small: readability +3.62% (p=0.003), conciseness +4.16% (p=0.002), reliability +2.94% (p=0.01), maintainability +2.47% (p=0.041), with 13.6% fewer readability errors per line.
Overview
Most claims about AI coding tools come from productivity surveys or vendor telemetry. This study is different: GitHub designed a controlled experiment to ask whether code written with Copilot is measurably better, not just faster to produce. The answer it gives is yes, with caveats that matter for anyone running an evaluation program.
The design is what makes it worth reading. Random assignment, anonymized review, and a fixed task let the researchers isolate Copilot’s effect from confounds like skill or motivation. That puts it a notch above the self-reported numbers that dominate this conversation.
How the trial was designed
GitHub recruited 243 developers, each with at least five years of Python experience, and randomly split them into two groups. One group used Copilot; the other wrote code without any AI assistance. After dropping incomplete or invalid work, 202 submissions remained — 104 from the Copilot group and 98 from the control group.
Every participant built the same thing: a web server for fictional restaurant reviews. Holding the task constant matters. When everyone solves the same problem, differences in the output trace back to the variable being tested rather than to who happened to draw the harder assignment.
Two layers of grading
Functionality came first. Each submission ran against 10 unit tests that checked whether the server actually behaved as specified. This is a binary, execution-based signal — the code either passes a test or it does not, and no reviewer opinion enters into it.
Quality came second, and here the design gets careful. In phase two, 25 developers who had passed all the unit tests themselves were assigned anonymized submissions to review. Each piece of code was seen by at least 10 reviewers, producing 1,293 reviews in total. Crucially, reviewers did not know which submissions used Copilot. That blinding is what keeps the quality scores honest — nobody could grade up or down based on a tool preference.
What the data showed
The headline result is functional: developers with Copilot access were 53.2% more likely to pass all 10 unit tests, at p<0.01. That is a large effect on the metric that matters most, because passing every test is the closest thing here to “the code works.”
The quality rubric tells a quieter story. Reviewers scored four dimensions, and Copilot code won on all of them, but by single-digit margins:
- Readability: 3.62% higher (p=0.003)
- Reliability: 2.94% higher (p=0.01)
- Maintainability: 2.47% higher (p=0.041)
- Conciseness: 4.16% higher (p=0.002)
Two more results round out the picture. Reviewers flagged 13.6% fewer readability errors per line of code in the Copilot group (p=0.002), and they were 5% more likely to approve Copilot submissions outright (p=0.014). Every one of these crosses conventional significance thresholds, so the direction is trustworthy. The size is the thing to keep in perspective: a 3% bump in maintainability is real, not transformative.
Why it matters for code-generation QA
The structure of this study is a template worth copying. It pairs an objective, execution-based gate (unit tests) with a subjective, blinded human review (the rubric), and it reports both rather than collapsing them into one number. Teams evaluating AI-generated code tend to lean on one or the other. Tests alone miss whether code is readable or maintainable; reviews alone miss whether it runs. GitHub measured both, and the gap between a 53.2% jump in test-passing and a ~3% rise in maintainability is itself the lesson — functional correctness and code quality are different axes that can move at very different rates.
The caveats are worth stating plainly. This is one language, one task, and one tool, with experienced developers. A restaurant-review web server is a constrained problem; it tells you little about how Copilot performs on a legacy codebase, an unfamiliar domain, or a junior engineer’s workflow. The quality rubric is also subjective by construction — blinding protects against bias, but a 2.47% maintainability delta sits close to the noise floor of human scoring, even at p=0.041. And the study measures the code that was submitted, not the long-term cost of maintaining AI-assisted code in production, which is where a lot of the real debate lives.
Read it as evidence, not proof. The functional result is strong and well-isolated; the quality results are directionally positive but modest. For your own evaluation work, the takeaway is methodological as much as it is about Copilot: hold the task constant, blind your reviewers, and report execution metrics and quality metrics separately so you can see when they diverge.
Read the original: Does GitHub Copilot improve code quality? Here’s what the data says — GitHub (Microsoft), 2024-11-18.