Over the last few years, I have frequently written about the problem with using metrics as a measure of value and performance. As long as we continue to be told that use of quantitative measures are important, I am gonna keep pushing back and reminding you it ain’t the be all and end all of evaluation.
Carter Gillies is actually more adamant and eloquent on this topic than I am so when I saw a piece on Aeon that started out sounding almost verbatim like Carter, I did a double take to check who the author was.
The author, Jerry Z. Muller, points out that performance metrics often incentive a gaming of the system in a manner which often runs counter to the purpose of the organization.
Or take the case of surgeons. When the metrics of success and failure are made public – affecting their reputation and income – some surgeons will improve their metric scores by refusing to operate on patients with more complex problems, whose surgical outcomes are more likely to be negative. Who suffers? The patients who don’t get operated upon.
One of the other issues is all too familiar to non-profit organizations come grant report time:
To the debit side of the ledger must also be added the transactional costs of metrics: the expenditure of employee time by those tasked with compiling and processing the metrics in the first place – not to mention the time required to actually read them. As the heterodox management consultants Yves Morieux and Peter Tollman note in Six Simple Rules (2014), employees end up working longer and harder at activities that add little to the real productiveness of their organisation, while sapping their enthusiasm.
Non-profit organizations are well acquainted with implications of metrics. Organizations are often restricted to what government entities, foundations and donors are willing to fund. It can be difficult to innovate or address needs if your funding source has different priorities or restricts how funding can be used. I have discussed before that there can be a tendency to report that everything you did met or exceeded the plans laid out in your grant proposal. The fear of losing funding for not being successful enough disincentivizes being honest about challenges the organization faced.
While there have been plenty of embezzlement scandals at non-profits to leave funders concerned about whether their money is being used responsibly, metrics provide faulty assurances because they are so easily falsified.
So what should be used instead of performance metrics? Well, Muller really doesn’t say. Doing a good job and having good outcomes might be a start. You’ll want to examine numbers to assist in the process of reducing needless waste. But trying to squeeze an extra percentage out so you can improve your efficiency score over last year when you squeezed an extra percentage over the previous year is not constructive.
Ultimately, the truth is that evaluation is hard. Even if we were to urge funders to invest more time in investigating outcomes directly rather than relying on numbers, the tendency to have positive associations for feel good stories will benefit some organizations over those that do unsexy, but impactful work. Then we will be back to rallying removing the emotional element by employing cold, hard numbers for evaluation.