Start Measuring Outcomes
Even with Feature Roadmaps you should be creating an Outcome Mindset
Thanks for reading the ProductFix newsletter. This issue is intended to address the challenge that some product leaders have in moving from Feature Roadmaps to Outcome Roadmaps. The approach is simple. Start by encouraging an Outcome Mindset by measuring your outcomes. This will drive organizational support in moving toward Outcome Roadmaps.
As always, if you find value in reading this please give me some love by liking and sharing with others.
Contents
Article: Measuring OutcomesRecommendationsWatercoolerArticle
Measuring Outcomes
Outcome Roadmaps are the best practice for communicating about and executing on a plan to deliver your product strategy. In anecdotal discussions with product managers, I have found that the business value of this approach is clear. Rather than focusing on a timeline of pre-conceived output commitments, as feature-focused roadmaps do, product teams are empowered to discover the best possible solutions to problems that are tightly aligned to the business and customer metrics that matter.
Despite understanding this value, I conducted a recent (non-scientific) online poll of product managers and found that only 20% of companies have adopted this practice. Unfortunately, these results align with the discussions I have had with many organizations.
There are many reasons why this may be the case, but I generally believe it is about institutional inertia and a lack of understanding of the benefits of a better approach. Shifting to thinking planning around outcomes impacts management, customer communications, internal team alignment, and the basic functioning of product teams.
Do you measure the outcomes after you ship your features?
Making the case for Outcome Roadmaps is complicated by the lack of understanding of the value being achieved with the current dominant Feature Roadmap model. How can you show that Outcome Roadmaps will lead to greater business value if you have no idea what the baseline is?
Are you Measuring your Outcomes?
The case for Outcome Roadmaps comes down to showing they deliver greater value to your customers and your business. If you are not currently measuring the outcomes derived from developing based on Feature Roadmaps it will be hard to convince skeptics of the value of shifting approaches.
I have found that organizations that use Feature Roadmaps tend to not measure actual delivered results for most or all of their enhancements. Organizations using Outcome Roadmaps have a greater natural inclination to measure for the actual results of their investments.
Predict Feature Outcomes
For the majority of organizations that are working with Feature Roadmaps today, most do have some sense of why each feature is on the roadmap. Frequently this is done formally with a product management tool that allows you to trace features back to some business objective. This business objective, in turn, can be translated into outcomes that the business plans to achieve. Whether using a purpose-built tool like Aha.io, hacking Jira+Confluence, or elaborate spreadsheets in Excel - this data often exists.
In cases where it is not explicitly captured and mapped, you should still be able to perform some analysis to tie everything you are planning to do to your strategic plan and objectives. If you don’t have a clearly articulated product strategy, you should be able to figure out what the anticipated business value is of everything on your roadmap.
Once you have the predicted value of features tracked you need to figure out how to measure for the actual outcome.
Measure Actual Outcomes
Measuring outcomes is part art and part science. While easy in some situations, it can be very difficult for others. In most cases, we need to look at two different challenges to measurement.
How do we collect data necessary to assess if the predicted value was achieved?
How quickly can we make this assessment?
Example:
The feature you are planning is specific data entry validation improvements to the date, name, and mobile phone number fields on the data entry form. The primary desired outcome this is mapped to is to “Reduce data entry errors by 10%”.
First, you need to instrument the application to allow collection of data to show the current rate of data entry errors. If you can’t instrument it directly, you need to find a means to analyze logs or stored data for errors that are a by-product of the existing data entry errors.
Second, develop and implement your enhancements.
Third, compare the rate of data entry errors before after the enhancement.
The above example is a simple scenario for a team developing a SaaS application to measure and learn from their work. Without measurement, you will never know if you hit 5% or 40% improvement. If you missed your target you could have learned something to further reduce data entry errors. On the opportunity cost side, you will miss out on the ability to market to your customers this awesome data quality improvement.
While simple, organizations that start by actually measuring their results in small ways will begin to develop richer product analytics and other approaches to measuring outcomes that will lead to better outcomes.
For on-premises software or cloud-based applications that have limited ability to be updated on a frequent basis, the timing of instrumenting and measuring the results becomes a challenge. This is where teams need to get more creative.
Continuing the example, measuring baseline performance could be done with a script against the on-premises data repository that best approximates the current error rate. Seek a few customers willing to help gather this anonymized data that are likely to take your update as soon as it’s available. Even delayed learning in this environment is better than never knowing.
These approaches take more up-front effort than just building a feature and shipping it. However, until you begin to measure the results, you will never know if you are successfully driving the value that matters.
Transition to Outcome Roadmaps
Over time the product team and broader organizations should be shifting to an outcome mindset. You will learn that some features achieve predicted outcomes better than others.
These key learnings and mindset shift will lead to thinking about ways to improve your practices to success rates. This can be the momentum needed to experiment with a shift toward Outcome Roadmaps.
If you have more than a few product teams, I recommend piloting with one first. They will need to operate in this mode for 2 quarters or 2 releases, whichever is longer. This will provide the necessary discovery and development cycles to see evidence of improvements the team can enjoy.
My hypothesis is that you will drive better outcomes when you focus on them from strategy through post-delivery. If it turns out that your teams get better results with Feature Roadmaps, then stick with them. You will at least have proven the practice that works best for your current organization.
Conclusion
Stop delivering features without measuring the actual value they provide. Delivering according to a Feature Roadmap will produce sub-optimal results for your customers and business - but until you measure you will never know.
Measuring features for outcomes will create a new Outcome Mindset. Leverage this to move toward an Outcome Roadmap and better practices.
Recommendations
Book: The Lean Startup by Eric Reis. The focus of measuring and learning from outcomes is closely related to the Build-Measure-Learn loop described in this must-read book for any involved in Product Management and Startups.
Person: Steve Johnson (@sjohnson717). Product Management thought-leader, coach, consultant, and spent 15 years instructing at Pragmatic Institute. He’s written books, articles, and is prolific on Quora.
Design Website: UserOnboard. Detailed critical teardowns of lots of software onboarding experiences. Wow. Even if you are not designing an onboarding experience as a Product Manager or Designer, there is a lot to learn here about good software design. This site is a treasure.
Watercooler
Sage cartoon advice from UserOnBoard.com. I loved this image immediately upon seeing it. I heard a passing reference to it in a recent talk by Bruce McCarthy and did a quick search.
It reminds me of the famous statement from Peter Drucker in The Practice of Management.
There is only one valid definition of business purpose: to create a customer.
Don’t be shy. Please hit the heart and comment to share your feedback. Beyond that, I coach product leaders to take advantage of best practices and challenge their own perspectives. If that sounds useful, take action now and schedule an appointment.