.svg)
When I started Drone over 10 years ago as an open source project, I was frustrated by how much time it took to ship code with existing CI tools. It made for a bad developer experience, and I wanted to solve that problem. My vision was to create a CI tool that was fast, simple, open, and secure. I’m excited to have realized that goal now with Harness CI.
On the heels of announcing several new speed enhancements for the Harness Continuous Integration (CI) module, I’m excited to share our test data showing that Harness CI builds up to four times faster than other leading CI solutions. These fast build times shown in our test results validate the impact of three important new feature enhancements: Cache Intelligence, Test Intelligence™ (a technology exclusive to Harness), and Hosted Builds – and, more importantly, they help to create a great developer experience.
At Harness, we understand that developers are frustrated with slow, flaky builds. Our focus is to remove friction points for developers, so they can focus on writing code and shipping it quickly, efficiently, reliably, and securely. Harness CI’s latest speed-enhancing features are a testament to this commitment.
You can read more about these new feature enhancements in our blog announcement, or dive deeper into the technical details on our CI product features page. In this blog, we’ll take a closer look at this impressive new data, as well as how Harness CI can save businesses up to 300% in infrastructure costs as a result of these recent innovations.
As we looked at the initial test results from our latest CI feature enhancements, we knew we were seeing unprecedentedly fast build times. We also knew immediately that we had to show developers just how much faster we could help them build.
We designed our tests to run the same builds with Harness CI, GitHub Actions, and another popular CI tool across the Apache Kafka, Apache RocketMQ, and Apache Zookeeper repositories. The chart below shows those results in minutes. (Note, the Harness “best case” results were achieved where the code changes were minimal and there was no test case run (e.g. this was an actual code change in the open source repo where very few test cases were identified to run).
The median test results show Harness CI is two to five times faster than the nearest competitor and three to four times faster than GitHub Actions. In some cases, Harness was 19 to 75 times faster than GitHub Actions.
Apache Kafka is a popular distributed event streaming platform with over 23,500 stars on GitHub. It’s also used by 80% of the Fortune 100 companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Below, you can see the build time results we recorded with the Apache Kafka repository.
Over the course of 50+ pipeline runs:
Since the Apache Kafka repo is a very large project, we also used it to demonstrate the substantial time savings gained from our Test Intelligence feature, a unique offering in the market. Using machine learning (ML), Test Intelligence determines which tests are necessary based on the code changes. In our demonstration, Test Intelligence effectively reduced the number of unit tests from 16,000 to 700. But that’s not all – Test Intelligence split the necessary tests and ran them in parallel.
We’ll take a closer look at how Test Intelligence works and those results after we detail the other tests and the methodology we used.
Apache RocketMQ is a distributed messaging and streaming platform with over 18,000 stars on GitHub that makes it easier to build event-driven applications.
The chart below shows the build time test results from the Apache RocketMQ repository tests.
Over the course of 50+ pipeline runs:
Apache Zookeeper has over 10,000 stars on GitHub and aims to make maintaining and coordinating distributed systems easier.
You can see our build time test results from the Apache Zookeeper repository in the chart below.
Over the course of 50+ pipeline runs:
We chose to test these three projects because of their popularity among the open source community and the high rate of adoption by development teams at enterprise organizations, including Goldman Sachs, Target, Intuit, Cisco, and more. Since we used open source projects, we also had the ability to replicate the same test scenario for each build.
We ran builds from these projects using the free offerings from GitHub Actions, another popular CI vendor, and Harness. The performance numbers were calculated by building the over 50 pull requests from each repository. This allowed us to test using actual code commits to the open source repositories in order to simulate a real use case that developers encounter every day. We used the out-of-the-box default configuration for each build run rather than fine-tuning the queries, because we know that developers have a better experience when they’re not required to make extra modifications.
You can see a screenshot of the actual pull request from the tests run in the Kafka repository below:
The speed results that Harness CI demonstrated in these tests were achieved by our latest feature enhancements. Let’s take a closer look at how each of these differentiating features impacted the test results.
Software development is inherently complex with various dependencies across build tools, package managers, and more. All this complexity increases build times. Developers don’t always remember to manually turn on caching or specify all of the necessary paths to store the right data in for each directory. By automatically caching well-known directories for Java and Node.js, Cache Intelligence helps developers spend less time waiting on builds to complete, which means more time coding.
Harness CI’s exclusive, ML-powered Test Intelligence™ delivers even more time savings now with the addition of test concurrency. Test Intelligence™ determines which subset of tests is necessary based on code changes while providing visibility into which tests are selected and why. After the required tests are identified, these tests run concurrently to further accelerate build time without compromising quality.
The chart below details the performance enhancement with Test Intelligence™ when 10 files were changed in the same Kafka pull request. Test Intelligence™ split and ran tests for just those 10 files in parallel, rather than running them sequentially. As a result, Harness ran approximately 700 tests that were related to impacts from the code change, whereas GitHub Actions and the other CI tool ran the entire suite of approximately 16,000 tests.
You can see how Test Intelligence™ saves a significant amount of time by only running the necessary tests and running them in parallel. With Test Intelligence™, Harness CI employs a unique approach to accelerating build times without sacrificing quality.
Gone are the days of “it just works” being good enough for CI in a market that demands that companies go faster to remain competitive.
We’ve talked about the recent improvements that enhance speed, but there are also cost savings when it comes to building faster. With Hosted Builds, companies don’t just alleviate the management burden of hosting infrastructure, they can also do more for less. Faster builds mean less build minutes consumed and a smaller footprint for resource utilization.
Combining Cache Intelligence and Test Intelligence™ with Hosted Builds ultimately saves businesses 200-300% annually on infrastructure costs. This savings is shown in the chart below using a Harness customer’s real usage data calculated to indicate how much time and money they would spend with Harness and each of the other CI vendors tested.
Here is the methodology we used to determine up to 300% cost savings based on real customer data.
Our customer has 90 software engineers who build 19,524 times a month using Harness CI. In keeping that pace, they build 234,288 times a year. Currently, the average time it takes to build is 15 minutes. We estimated that the average number of builds per developer per week was 11. All estimates assume builds are run on Linux. Based on this customer, this comes to a total of 292,860 build minutes a month and 3,514,320 build minutes a year.
Next, we compared these figures against two CI vendors. GitHub Actions costs 0.008 per build minute, and CI Vendor 2 costs 0.0120 per build minute. Harness CI is competitively priced relative to GitHub Actions and priced lower than CI Vendor 2.
With a very conservative estimate of two times faster than both vendors, Harness CI would cost $1,171 a month and $14,057 a year. Respectively, CI Vendor 2 cost $2,343 a month and $28,115 a year. GitHub Actions cost $3,514 a month and $42,172 a year. That yielded 200% annual savings over CI Vendor 2 and 300% savings over GitHub Actions.
On a personal note, I am extremely excited to share these updates. The last few years have left much to be desired with CI innovation, but today, with the ability to run builds up to four times faster, the game changes completely. It was my dream to make a CI system that would be simple, open, fast, secure and now this dream has become a reality. I invite you to try this sample to reproduce the results and see for yourself.
Ready to start building with the fastest CI platform on the planet? Get started for free with Harness CI.
Bradley Rydzewski is Senior Director of Product at Harness and founder of Drone.
Sign up for our free plan, start building and deploying with Harness, take your software delivery to the next level.
Learn intelligent software delivery at your own pace. Step-by-step tutorials, videos, and reference docs to help you deliver customer happiness.
.svg){kind=small}
Learn intelligent software delivery at your own pace. Step-by-step tutorials, videos, and reference docs to help you deliver customer happiness.
Enjoyed reading this blog post or have questions or feedback?
Share your thoughts by creating a new topic in the Harness community forum.
.png)
