Xactly’s Journey from Shadow IT to a Single Pane of Glass CI

Harness Impact

• Developer experience went from “impossible to debug” to full traceability for deployments through CI pipelines.
• Removed an EC2-based Jenkins farm amounting to $575000/year

About Xactly

Xactly is the company behind the popular revenue operations products such as Xactly Incent, Xactly Forecasting, and Xactly Operational Sales Management - part of its Intelligent Revenue Platform. Its users benefit from the platform’s precise compensation plans, better incentives, and data-informed insights to give them more confidence in their pipeline. It’s the kind of product most sales organizations love to have for many reasons, not the least of them being that it makes sense of compensation plans. 

Xactly’s Mission to Blend Modern Software and Data Science

To meet the expectations of excellence, the software Xactly delivers this value through has to meet the highest standards. Revenue is not something anyone wants to mess with. It not only has to be precise, but also go a step beyond it to provide a bigger picture to all incumbents. This experience takes the shape of Xactly’s Intelligent Revenue Platform, and Alby Graham, director of engineering, is one of the leaders behind its success. 

Graham oversees two fundamental pieces of Xactly software delivery efforts: on one hand, his team manages the platform that is designed for rapid application delivery; this same framework, as they like to call it, also manages a series of reference applications that feed data into the Incentive Compensation Management (ICM) solution. This last piece of the puzzle is the one responsible for providing AI and ML capabilities to the platform that, in turn, provide insights to Xactly’s customers – basically, a combination of pure software development and advanced data science in one single software delivery effort. For these operations to run smoothly, Alby has around 60 engineers and data scientists reporting to him. 

Xactly’s CTO doesn’t want to waste resources. Xactly’s technology leadership knows developing software will require strong investment, but they also know that if run efficiently, it can have a bigger impact on the business. Shadow IT, wild use of public cloud, and other consequences of perverse incentives that come from anarchic software delivery pipelines can be cut down without hindering the developer experience and software quality. In fact, some constraints will not only make the CTO happy, but the end user as well. Let’s dive into Xactly's journey to modern CI.

Monolithic Builds Becomes A Tale of Continuous Integration and Fishing

Alby tells the following anecdote about one of his best performing engineers: a production release was programmed to be delivered. However, that developer ran the build and Jenkins crashed. The developer let Alby know that he was going to go fishing since he knew that Jenkins wouldn’t be back up until the next day. Alby was completely ok with this, because it’s true. Jenkins wasn’t going to be up and running until the very next day. 

This is the story of many Jenkins shops. Jenkins is adopted and run, let’s say in the public cloud. The company’s software development teams grow and this initial CI implementation keeps working. The growing teams start committing more and more code to both the platform and the ML model repos. The Jenkins instance’s performance starts lagging, the build queue starts growing, and builds pile up. 

But let’s get back to Xactly’s particular Jenkins instance. It crashed or halted at least two to three times a day, rendering that pipeline idle. While the pipeline was running, however, queues would be around six or seven builds long at any given time during working hours. Effectively, someone would make a commit and only hours later would hit the target environment because of Jenkins. 

It was also not infrequent that the build “died in the middle.” At one point, there were 35 builds in the queue because a build died in the middle, and Alby’s team had to wait an hour to restart it. 

Fixing Build Speed for the Win

The initial solution was to split that Jenkins bottleneck into other Jenkins instances. That’s when the second variable of the Jenkins problem kicked in: the Jenkins sprawl. The event that triggered the Jenkins build process was commit to a central repo. The problem is that up to four or five Jenkins instances picked the same commit each time, generating multiple builds and deploys in parallel. There were several problems to this new approach to CI. One of the most important ones was traceability or auditability: it was effectively impossible to track back, let’s call it build 123 or release JE 12, to a specific Jenkins instance and its build pipeline. A second variable to this problem comes from answering the question: how much can you rely on tests if you don’t know where they are running? What’s the test coverage? Which build pipeline ran which tests? All those questions eventually became unanswerable - until Graham onboarded Harness Continuous Delivery (CD).

Once this scenario was fixed and only one Jenkins machine would pick up one commit each time only, it seemed that the situation was under control. Xactly already had Harness CD in use, which was listening for new Docker image tags that the Jenkins build system was updating after each build that produced a new artifact, and it would update the image and its tag pushed to Docker Hub. But again, the image tag is not the only parameter that a software delivery process should rely on. One cannot make the whole production environment rely only on an image tag whose source is fuzzy, at best. 

Developer Productivity and Single Pane of Glass: No Trade-offs

Naturally, managing such a scenario became a burden. Costs were growing and most of them came from truly unproductive tasks that were, in effect, hindering development to the business critical applications: the platform and the intelligence within it. A single Jenkins instance running in an AWS EC2 instance can cost $36,000 or more per year. Imagine 16 Jenkins instances running wild on EC2 compute layers, the cost becomes astronomical just to run CI only. 

Financial costs aside, wasting 40 developers’ precious time because a build system crashed and no one can get the most recent code for hours is a big deal. Developers lose trust in the delivery platform and are incentivized not to commit so frequently because they anticipate that problems will arise. That hinders developer confidence, which is one of the main drivers behind the company’s success. To get rid of this repeating scenario, a plan was put in place to progressively sunset all Jenkins instances and provide a better solution. 

Pay when it’s due: fast and easy like Xactly

Xactly’s approach to software development was that if a tool is purchased to support the delivery pipeline, it has to solve the problem with a deep feature and capability set. Spinnaker was, for this very reason, evaluated but not considered: a broader but somehow shallower feature set would not solve the problem with CI. Since Xactly was already a Harness CD customer and was familiar with what Harness is able to do, exploring Harness CI Enterprise was a natural step in the same direction. “Do one thing and do it well,” Alby said. 

Stability and time from commit to deployment are the two main areas in which Xactly has seen an improvement. For one, Harness CIE has not crashed a single time since it was implemented, and that’s a tremendous gain compared to the previous state with Jenkins. For another, removing an EC2-based Jenkins farm resulted in savings of $575,000 per year.

Go Fishing With No Worries in the Back of Your Mind 

Ultimately, every build from commit to completion takes the same amount of time. There is no queueing or build failures. This single consistent performance capability has helped with productivity immensely. Developers can see the impact of their changes in their alpha environment in a matter of minutes, compared to the multiple hours it would take with Jenkins. This incentivizes developers to contribute more and to try new things. Speed in CI means that more tests, even manual tests, can be run without hindering the process. Having CI be part of a bigger platform provides developers with shorter feedback loops, which in turn sparks creativity and promptness among them.

Looping back to Alby’s anecdote about one of his best developers going fishing rather than waiting for the Jenkins instance to get back up, this is what he had to say after Harness CI ran for only a couple of months in Xactly’s pipeline: “I don’t hear people complain anymore about builds. I can’t tell you how much better that makes me feel. Those things are hard to quantify, but so much more impactful than the dollars.”