VPs of Engineering often make decisions on various vectors such as business alignment, engineering execution, customer satisfaction, developer happiness, and outsourcing projects to vendors. Highly regarded VPs of Engineering seldom make these decisions based on gut and intuition. They must rely on accurate data and metrics.
Key questions that leaders need to ask include how often to measure, what are the right inferences to draw then, when to respond to changes in data, and when to take action. These are no simple answers but a good starting point for these is to have a well thought-out strategy on the metrics to measure. Equally important is the measurement frequency and knowing exactly when to take action in response to the metrics.
It’s important to differentiate between metrics that have a natural frequency and ones that don't. Metrics such as sprint metrics and release metrics have a natural frequency. For example, if your metrics relate to sprints, then its natural frequency is the sprint length (2, 3, or 4 weeks depending on the organization or product). Likewise, a release may have a natural frequency of 1, 2, or X months depending on the organization and product.
However, an engineering leader must also monitor several other metrics that do not have a natural frequency. How often should one measure developer productivity or developer happiness? Is it appropriate to measure it weekly, monthly, or quarterly? In such situations, choosing the measurement frequency is extremely critical because based on how frequently a metric is consumed and acted upon could lead to varying conclusions, actions, and results.
Two obvious pitfalls in choosing the frequency of measurement are either the metric is measured and acted upon too soon, or we let too much time pass before we measure and act upon the metric. Both can be problematic.
Measuring and acting too soon
Too much time between measurements and subsequent action
Ironically, the answer is it does not matter what metric you choose, there isn’t one ideal frequency. There are actually TWO frequencies for most metrics.
We suggest that you need both a short-term measurement and a medium-term measurement. This might seem counterintuitive or a letdown but there are very sound reasons and value in using both a short- and medium-term measurement for a metric. Here are the benefits of each.
Benefits of Short-Term Measurement
Benefits of Medium-Term Measurement
Then what is short term and medium term? Giventhat we need to measure in short term and medium term, the definition of short and medium term varies based on the metric being measured. This means we need great flexibility in how we slice data, view, aggregate, and respond to insights. Engineering leaders need a system that satisfies these criteria.
To put it bluntly, solutions or products that provide pre-canned metrics based on fixed time frames and with minimal context are actually misleading you into decisions that are sub-optimal, perhaps even detrimental.
Utilizing a platform such as the Harness Software Engineering Insights can provide support for the diverse DevOps toolsets along with the flexibility to adapt, combine and formulate analytics appropriate to each given audience. Harness offers out-of-the-box dashboards for standardized software metrics such as DORA, and it also provides the flexibility and adaptability to not only visualize the metrics required in your organizations but also provide predictive analytics and orchestration to take action upon those analytics.
Check out a live demo of how we can help you transform your engineering organization!
Enjoyed reading this blog post or have questions or feedback?
Share your thoughts by creating a new topic in the Harness community forum.