“The past cannot be changed, forgotten, edited or erased. It can only be accepted.” – Unknown
Let’s Define some metrics!
I had a conversation with a relatively new scrum master recently who was working to help establish some base level team metrics for their organization. They had made the journey to scrum from the role of a project manager and seemed to really transition the mindset of driving to create an environment of trust and transparency within a team and shunning the ideas of controlling the time/budget/scope and “resources” (word I hate when it comes to talking about people) viewpoint of traditional project management.
So my initial question was “who are these metrics being designed for”? This is something we often do not consider. There is a different need for reporting transparency and one to help teams improve. If oversight is the driving goal then the assistance to the team may become minimized. If the goal is to create metrics that are learning opportunities for the team or generate insight for a conversation then the construction of these may be much different but may provide less insight into what management may be seeking.
All metrics are not created equal …
Except, when it came to metrics it seemed. Most of the metrics they were developing were centered around trailing level metrics (lagging metrics for those who are KPI inclined) for the team. A lot of the focus was on the past as a predictor. So I raised the question; “why are you so concerned with the past” in terms of metrics? They explained to me that this is how he could help the team improve by knowing where they failed or made bad decisions beforehand. So I asked again “given the dynamic nature of iterative development, how can you ensure that the same cause will generate the same effect”?
We discussed in detail that they were extremely proud of was the tracking of “estimated” versus “actual” hours on tasks. They worked really hard to convince me that learning the difference between the two would help them more accurately estimate the work to be done. I let them go for a while and then I had to call bull%^#$ on this.
In my opinion, the only way an “actual” at a task level will be a predictive metric is if the the same circumstances occur exactly the same way the next time. Estimation is just that. It is not predictive in nature, it is a best guess based on the knowledge known at the time. And before anyone says it, you can make a better estimate by reducing the unknowns but you can never eliminate the unknowns.
If I have a headache, I had road rage on the drive in, I had more meetings than normal, I volunteered to help someone, I was tired … A whole lot of things could impact the differences between my estimation and delivery. So in my “estimation”, this metric is a “narcotic metric”, it tends to make you feel good but it actually could be very damaging when using it.
And what if we do know this information? The key is to be able to take information learned and make it actionable. How do we do that? Send an email and tell the team to estimate better? Begin to question their estimates? Make them sign off? I cannot see how you “use” this information to help them actionably improve.
One type of metric I always recommended for consideration is one I consider a refinement metric. This type of metric is designed to allow the team to perform self-reflection on a recent event given the context of more knowledge and apply that knowledge to a past decision. Instead of comparing the past to the outcome, it asks “given the things we learned, do we think we made an accurate assessment of the work”? Sounds similar but with a couple of differences. 1) It’s never at a task level. 2) It’s a team level rather than a individual developer level as the goal is that the team learns to become better as a team, not as just an individual member.
So what I suggested is something we would often do as part of a retrospective with a team. We would examine the stories for the sprint just delivered and look at the assigned story points. Then we would have a discussion and asked if given the knowledge today, would we apply the same story points. This typically leads to a discussion of dependencies, issues, etc and the team centers pretty quickly around a “stand/raise/lower”. This does not help them get better in defining the individual number itself but helps them reflect on this type of work and the hidden complexities that can be present to determine questions that they may ask themselves or the product owner to create a better consensus around an item in the future to the potential complexity of the work.
Just in Time Metrics
One metric that I have read about after working as a scrum master that I liked was a daily vote of confidence. If you have ever been in a U.S. hospital and had surgery there is a common tool used by nursing staff to determine the level of pain you are in to report and administer treatment. It is called the “pain assessment tool” often and looks like this:
This allows the patient to quickly indicate the level of pain that they are perceiving and reflect this to the nurse, doctor, etc. The blog post I had read (which I wish I could recall the link) suggested using a similar scale for each team member to allow them to forecast their confidence level to meet the current sprint commitment. This allows each team member to express their confidence on a similar scale from “Yep, we rock!” to “Hey guys, I am really, really worried where we are right now” so conversations can be had to determine what’s going on for the purposes of communication at the first responsible moment or to allow the team to swarm around and issue, etc. A simplified version of this has been used by teams that involved “Roman voting” (thumb up affirmation, thumb down condemnation) as well. This type of metric gives you perception of the current state of work in a very real form and let’s you make it actionable. This is often easier to use than seeing potential patterns within a sprint burndown (which is another post).
So just as I asked this scrum master, I ask all of you. Do you have metrics? Who are they created for? Are they actionable or are they just data? How do you use them? Are they actually helping you and even more important are they bringing value back to your team(s)? How do you know?
I leave you with a quote about the importance of knowing why you are actually measuring something …
“Remember, what gets measured; gets managed” – Peter Drucker