Tuesday, September 1, 2015

So What Do We Measure Then? –Creating Leading Indicators

A few weeks ago we wrote a post about the problems with measuring safety only by its absence through the use of incident rates. Then we wrote a post about how to think about creating new indicators of safety. As we discussed, you really need to make sure your indicators are specific to how your organization creates safety. Now, we want to spend some time talking about actually creating those indicators, including how to avoid some of the traps organizations fall into.

First things first, we need to get rid of this idea that “if you can’t measure it then you can’t manage it”. That slogan sounds really good, but it’s simply not true. There are plenty of things that we manage everyday without measuring. Some of the most important things in life are immeasurable. Take for instance love or happiness or hatred or consciousness. All of these are abstract concepts that are vastly important to the human experience, but we have no way to measure them directly.

Now, don’t get us wrong, you can measure the effects of these things. For example, people who are happy often report being happy and display behaviors that we associate with happiness (e.g., smiling). We can measure those things, but those are not measures of happiness directly. People who are unhappy can do the exact same things in a convincingly enough way, but will have low levels of happiness (at least we think so). So these things that we can measure are merely indicators of what we want to measure (but can’t). Yet somehow we still manage our happiness and attempt to manage the happiness of others. Certainly things that you can measure may be easier to manage, but that doesn’t mean things you can’t measure are unmanageable.

In the same way, safety is immeasurable. You cannot measure how much safety an organization has or does because safety is an abstract concept. The systems folks would say that safety is an emergent property of a complex system. We can’t identify it or measure it directly, but we can feel its effects. Yet, we still have to manage it.

If that’s true, then how do we know we’ve achieved anything resembling success? We need to take the focus off of measuring safety and look for indicators, things that would indicate the presence of the capability to create safety.

This leads to our second point, indicators often create new incentives that drive behavior. So as you look for those things that indicate the ability to create safety, you need to understand that everything you are doing is going to change what you are measuring. This is one of the reasons incident rates are so bad – they create an incentive to reduce the number, which is achievable in two ways. You can either have less incidents, or you can report less incidents, which obviously is very bad for us.

But this feature of creating unintended consequences is not unique to incident rates, it is a feature of measurement in complex social systems. For example, one commonly used indicator is time to close safety corrective actions. The goal is to get people to close corrective actions in a timely manner. And you can achieve this in two ways – complete the corrective action in a timely manner, or report that you’ve completed the corrective action in a timely manner. Something similar happened at the BP Texas City Refinery where work orders to fix safety critical equipment were closed as completed, but the work was not performed.

One way to avoid this is to create a countering indicator. If you have an indicator that is quantitative (say % of required training completed), couple it with a more qualitative indicator (e.g., employee reported confidence in completing the task, employee reports of training quality, observation of employee skill levels). This counter indicator may provide a balance to avoid some unintended consequences.

This leads us to our third point, which was brought to us by a colleague of ours, David Bond. The indicators you choose are less important than the conversations that they start about safety in your organization. You want indicators that don’t drive rote compliance or unthinking behavior. Things that drive out thinking are often nefarious enemies of safety. Instead, we want indicators that make people think and ask questions. We want indicators that drive reflection. For example, one organization allowed their managers to develop their own leading indicators for their business units and one of the indicators they came up with was the ratio of reactive to preventative maintenance. This indicator drives people to think about why that is an indicator of safety. Questions will be asked about when reactive maintenance is bad and why preventative maintenance is better, but not in all cases. And if reactive maintenance is not ideal, how can the organization manage risk in those times they have to have reactive maintenance. The indicator started people talking. Safety ensued.

Finally, our last recommendation is encompassed above – get employees involved. Ask your employees at all levels how they know they are safe. What are the things they do each day related to safety? And we don’t mean the stuff to be compliant, but the stuff they do to be safe. Sure you might need to educate them a bit about how to do it, but if you can give them the tools you will likely find that there is no one wiser in your organization. The people who are often best equipped to tell us about how we know on a moment by moment basis how safe we are, are often the people doing the work, not necessarily the safety people.


That’s it! Those are the basics. What do you think? What did we miss? This is a difficult concept, so lets keep this conversation going!

No comments:

Post a Comment