.png)
I have alluded to measurable success criteria in previous posts, but given how crucial metrics are in defining, shaping and, well, measuring our impact and therefore our success, I wanted to expand on the subject.
Metrics are hard and good metrics are even harder. You won’t get it right the first time, and possibly not even the second time - but as you get more experienced at wrangling these data points it will get easier to pick the right ones, those that truly provide a picture of how you’re doing and whether you’re on the right track.
What are we trying to measure?
Your metrics are supposed to provide guiding lights in your quest to achieve your outcome. The path there may not necessarily be obvious from the beginning - you may have a few ideas, hypotheses or experiments you want to conduct that you believe will contribute to the impact you desire. But unless you measure the impact, you won’t know whether you were right. So instead of blindly delivering “features” or “projects” we want to establish whether the work we’re doing is having the desired effect.
As you define your outcome statement, think about how you could measure whether your statement is true or false. What behaviours do we expect to observe when we have achieved our outcome? What impact on the bottom line do we expect to see when we succeed? What will be different from today and how could we measure this difference?
Sometimes you can’t exactly measure what you’re after. You may have to contend with close proxy metrics, something that resembles what you’re after but may be easier to observe and gather than the actual thing. That is completely fine - after all, the metrics are supposed to work for you, not the other way round. Even something with only 80% accuracy will provide sufficient insight to ensure you’re roughly heading in the right direction. Or more importantly, tell you when you’re not.
As you deliberate what metrics to include, here are a few considerations to bear in mind during your selection process:
Is the collection of metrics sufficient to predict a successful outcome?
In other words, if you reach all of the targets embodied in your metrics, will you necessarily have achieved your outcome? If the answer is no, look for the missing dimensions that would make your collection of metrics a complete set.
Do your metrics contain both leading and lagging indicators?
Leading indicators are measurements you can observe right now, without delay; they allow you to course-correct in the moment. Lagging indicators by definition lag behind the work - they are not immediately observable, they may take time to materialise. They will tell you whether or not something you did several weeks ago has had the desired effect.
Let’s look at an example: you hypothesise that taking up running at least three times a week will result in your body weight dropping and your fitness level increasing. A leading indicator would be the number of runs you complete a week, the number of miles you covered during those runs, the length of time you spent running. All of these measurements are in the moment, your actions create immediate results in these numbers, thereby allowing you to course-correct whenever needed. You’ve been working late, drinking too much, waking up hungover and skipping the morning run as a result? Your metrics will quickly tell you it’s time to up your game.
An obvious lagging indicator would be your actual body weight - it’s unlikely to shift immediately and may need a few weeks of dedicated effort before results are noticeable. Your body composition - i.e. your body fat percentage, muscle mass, etc - would be another lagging indicator. It’s another metric that won’t immediately respond after your first 5k run; it’s going to take prolonged effort to materialise the impact.
Are your metrics concerned with both the action and the impact?
This may sound rather obvious, but as you choose your metrics, ensure that you capture both the work you’re doing (and that you’re doing the right work!) as well as the impact you’re predicting.
In our fitness example we want to capture both the frequency, duration and intensity of our exercise (the action) and the metrics that monitor the expected results - body weight, body composition, etc. If you focus too much on the impact metrics, you risk losing sight of the work going in (or not going in!). You won’t know whether to just be patient and wait for the impact to materialise, or whether your actions are insufficient or even wrong.
If you focus too much on measuring the work you’re doing but don’t capture the impact this is supposed to generate, you can’t close the feedback loop. This deprives you of the data points you need to assess whether you’re doing the right work for your desired outcome. How do you know that all your effort is worthwhile?
Both types of metrics have to be in balance to paint a full picture and to allow you to assess whether the hypothesis in your outcome statement is indeed correct - or whether 3 runs a week cannot possibly outweigh the large slice of cake you treat yourself to at the end of each run.
Absolute targets or percentages?
Jeff Gothelf shared some thoughts on how to pick good target values in a recent newsletter. Once you’ve decided what you’d like to measure, you may wonder how to best express your targets - should you aim for a specific number (i.e. “3 runs a week”) or a percentage improvement (“drop 10% of body fat”)? The answer is, as always, it depends.
Jeff makes a good case that a percentage change target needs a baseline - you need to know where you are today in order to be able to assess a relative change. I usually recommend that teams with no experience in gathering and tracking metrics start off with no target at all - just get into the habit of collecting the data, looking at it and making decisions on the back of it. The fortuitous side effect of this approach is that you will have established a baseline by the end of your first quarter, which you can then leverage to set a first relative target in subsequent iterations.
Overall, a percentage change target is probably more meaningful in most circumstances than a fixed number as you’re likely to be looking to measure a change in behaviour. But then again, you may have a fixed fundraising target, or a customer acquisition target that will make you profitable or another fixed number that is relevant to your business. As long as there is context that everyone understands, both an absolute value and a relative change can provide meaningful insight into your journey.
Can the metrics be manipulated?
If you wanted to, how easy would it be to manipulate the metrics you have chosen without them actually representing a meaningful improvement? Liz Keogh talks about putting on your “Evil Hat” to experience how metrics could be falsely interpreted and open to manipulation. The team leader that is measured by the number of obstacles they remove for their team may just create some more obstacles to remove in order to inflate their metrics, yet the team will be no better off as a result.
What about everything else?
It’s easy to become focused on a goal, assess your progress by carefully watching your metrics and remain completely oblivious to unintended consequences. The team that regularly releases much celebrated features and enjoys high levels of customer engagement may just be crumbling under the superhero pressure. They may be working excessive hours in order to produce the highly anticipated outcomes, they may be cutting back on sleep, exercise, quality time with the family - until eventually they reach the point where enough is enough, take off the super hero cape and leave.
Don’t be that team. Keep an eye on health metrics. Speed over quality is not sustainable. High profits that lead to systemic burnout among the team aren’t sustainable. Consider what guardrails you may want to put in place to ensure that there are no unintended consequences at play. You want to increase your marketing outreach to ensure your customers know about all the great new products you’re launching? Great. Just make sure your content isn’t considered spam that is either automatically filtered by mail providers or leads to a spike in unsubscribe requests from annoyed customers.
In summary, your metrics want to be:
Complete,
A good mixture of leading and lagging indicators,
A good mixture of action and impact indicators,
With appropriate targets that are not easily manipulated, and
Include health metrics where a deterioration would demand immediate, rectifying action
I hope this is useful and provides you with a few pointers as to how to go about picking the metrics that are right for you and your team. As always, if you’re not sure where to start or need help along the way, drop me a line!
If you want to learn more about designing your organisation for agility, focus and delivery, this introduction to value stream teams is a good place to start.
Comments