.svg)
Developer productivity is a critical factor in the success of any software development project. The continuous evolution of software development practices has led to the emergence of innovative tools aimed at streamlining the coding process. GitHub Copilot, introduced by GitHub in collaboration with OpenAI, is one such tool that utilizes advanced AI models to assist developers in generating code snippets, suggesting contextually relevant code, and providing coding insights. To scale developer efficiency, one of our customers adopted GitHub Copilot, leading to increased collaboration and shortened development cycles, as demonstrated by Harness SEI's comprehensive analysis.
Before implementing GitHub Copilot, developer teams grappled with challenges primarily centered around pull requests (PRs) activity and cycle time in their software development processes. The existing workflow exhibited limited PR activity, leading to isolated development efforts and sluggish code review cycles. This hindered collaboration among developers and extended the time taken to integrate changes. Additionally, the cycle time from task initiation to deployment was longer than desired, resulting in delayed feature releases and impacting the product’s ability to swiftly respond to market demands.
Manual code reviews were time-consuming and inconsistent, exacerbating the efficiency challenges.
These issues collectively created bottlenecks in collaboration, resource allocation, and timely delivery of software solutions.
In this study we tried to investigate the impact of GitHub Copilot on developer productivity, with a focus on the number of pull requests (PRs) and cycle time, within the context of a comparative analysis conducted using Harness SEI. The study was guided by the expertise of the Harness Software Engineering Insights SEI and involved a sample of 50 developers from a customer. The study took place over multiple months. In the first 2 months, the developers worked without GitHub Copilot's assistance. In the last few months, they used GitHub Copilot as an integrated tool in their coding workflow. Throughout the study, various performance metrics were collected and analyzed to gauge the impact of Copilot.
The study measured the impact of GitHub Copilot on two important metrics:
The average number of PRs is a critical indicator of development activity and collaboration. The analysis revealed a significant increase of 10.6% in the average number of PRs during the month when developers utilized GitHub Copilot compared to the month when Copilot was disabled. This increase suggests that GitHub Copilot can help to improve collaboration, as developers using Copilot can potentially iterate more rapidly, leading to increased code review and integration.
Cycle time is defined as the time taken to complete a development cycle from the initiation of a task to its deployment. It is a fundamental measure of development efficiency. The study demonstrated a reduction in cycle time by an average of 3.5 hours during the month when developers leveraged GitHub Copilot, representing a 2.4% improvement compared to the month when Copilot was not used. This reduction suggests that GitHub Copilot's assistance in generating code snippets and offering coding suggestions contributes to quicker task completion and ultimately shorter development cycles.
GitHub Copilot has demonstrated the product's potential to transform software development. The increase in pull requests (PRs) and the reduction in cycle time are two key metrics that demonstrate the positive impact of GitHub Copilot on developer productivity.
Harness SEI was used to facilitate this study. To summarize, the study proves the capability of GitHub Copilot to significantly improve developer productivity. However, there is still more to uncover. We are conducting more experiments and a more thorough analysis of the experiment data we already collected, looking into heterogeneous effects, or potential effects on the quality of code. We plan to share our findings in further case studies.
To understand developer productivity and unlock such actionable metrics and insights, please schedule a demo of the Harness Software Engineering Insights module here https://www.harness.io/demo/software-engineering-insights.
Sign up for our free plan, start building and deploying with Harness, take your software delivery to the next level.
Learn intelligent software delivery at your own pace. Step-by-step tutorials, videos, and reference docs to help you deliver customer happiness.
.svg){kind=small}
Learn intelligent software delivery at your own pace. Step-by-step tutorials, videos, and reference docs to help you deliver customer happiness.
Enjoyed reading this blog post or have questions or feedback?
Share your thoughts by creating a new topic in the Harness community forum.
