Quality Metrics to Consider for Your SaaS Product

Post-deployment, one of the most common metrics for software quality is how many support issues a release generated. This however tells only one part of the story. The adjacent question we need to be asking is how fast are we able to resolve these support issues.

A higher time for resolution could be indicative of a few things such as 

1. Poor debuggability
2. Lack of engineering resources to work on issues
3. Overwhelmed Teams
4. Higher than expected support issues for the release
5. No expected SLA adherence

By setting different SLAs for different projects/priorities and issue types for Jira, you are not only setting expectations, but you also can track issues that have missed the SLA and measure the median response time for different groupings — priority, component, assignees, etc. Missed SLA analysis is a great way to look at failure points in the process and potential bottlenecks.

Bottlenecks

Measuring Resolution SLAs and the issues that miss SLAs tells us there is a bottleneck somewhere, so the immediate question would be “Where is the bottleneck?”. Understanding and easing bottlenecks are super critical to running a quality organization that is interacting with multiple dev and support teams.

By measuring how much time a ticket is spending in different stages, we can determine what the bottleneck is for that organization or service. For example, tickets spending a bulk of their resolution time in the backlog could be indicative of a resourcing issue. Too much time in Triage could point to the need for better debuggability and logging.

Testing Effectiveness

Over time, you come to realize that sometimes, more tests are just that — more tests. They do not necessarily enhance the quality of the release. Being able to measure the effectiveness of testing and investing in the right kind of testing is a great way to control the quality of a product.

How do we measure the efficacy of a test? One popular method is to track the bugs uncovered by the test. I am going to leave sanity tests out of this tally — they are needed even if they seldom find issues. Another is to measure the bugs found in-house vs the customer support issues and see where the gaps are. The difficulty here is that you have 3 different data sources — your test repository, Jira, and/or a customer success platform like Zendesk or Salesforce. But the trends give you a clear indication of effort and impact. For example, If you are running 100 additional tests but not increasing either the internal bug count or reducing the customer issues, then clearly the 100 tests do not have the required impact and need to be revisited.

Driving Visibility to Increase Quality

As we built and scaled teams, and implemented Quality processes, one of the biggest stumbling blocks was driving continuous visibility. This meant showcasing the most important issues on hand to the right set of people and also making sure nothing was falling through the cracks. We all know of that one bug that moved into Blocked state and never recovered.

The process of visibility has been mostly a manual one — maintaining and sharing lists with the stakeholders and personally following up on pending items. This is both laborious and error-prone. What we need is automation to remind team members of upcoming due dates, to alert on “stuck” issues so that they get the needed traction, and to escalate to the right set of people when needed.

Enter Harness Software Engineering Insights playbooks. With its low code / no code model, driving contextual visibility via proper development metrics is an easy click and drag 3 steps. Keeping track of bottlenecks and SLA and escalating to the right audience can all be easily automated.

Interested in learning more about how Harness Software Engineering Insights can help you get the right metrics? Request a demo.