Joi Ito who serves on the boards of both the Knight Foundation and MacArthur Foundation wrote a piece for Wired on the importance of finding the right metrics for measuring non-profit effectiveness.
He notes that if you use circulation as a measure, public libraries have been failing for years given that circulation has been continually falling.
But if you only looked at that figure, you’d miss the fascinating transformation public libraries have undergone in recent years. They’ve taken advantage of grants to become makerspaces, classrooms, research labs for kids, and trusted public spaces in every way possible…If we had focused our funding to increase just the number of books people were borrowing, we would have missed the opportunity to fund and witness these positive changes.
As I have quoted/paraphrased Carter Gillies many times, including just last week, just because you can measure it doesn’t mean the result is relevant or useful to you.
Ito writes that identifying relevant metrics is difficult and there is a tendency to default to what is easiest to measure.
The problem is that one pretty much never deals with an issue that is not part of a complex, complicated system. Indications of that problem being addressed successfully is not an indication that everything is running well.
He uses the example of iron levels as a measure of health. While iron is important as a measure of anemia, it can’t tell you about the health of a body by itself. All the medical tests you can conduct can’t tell you about the happiness of the person. (I daresay being subjected to all the tests will be detrimental to the happiness of the person.)
Ito goes on, (my emphasis)
…simple metrics often aren’t enough when it comes to quantifying success. They typically are easier to measure, and they’re not unimportant.
[…]
Similarly, while I believe rigor and best practices are important and support the innovation and thinking going into these metrics when it comes to all types of philanthropy, I think we risk oversimplifying problems and thus having the false sense of clarity that quantitative metrics tend to create.
One of the reasons philanthropists sometimes fail to measure what really matters is that the global political economy primarily seeks what is efficient and scalable. Unfortunately, efficiency and scalability are not the same as a healthy system.
As an example of the breadth and long term vision and planning that is perhaps necessary to employ, Ito cites the 1300 Ise Shrine in Japan which is completely rebuilt by craftsman every 20 years, supported by a supply chain management plan operating on a scope of 200 years. The measure of success of the shrine is entirely opposite the expectations of growth and scalability placed on most non-profit entities today.
The lumber mostly comes from the shrine’s forest managed in 200 year time scales as part of a national afforestation plan dating back centuries. The number of people working at Ise Shrine isn’t growing, the shrine isn’t trying to expand its business, and its workers are happy and healthy—the shrine is flourishing. Their primary concern is the resilience of the forest, rivers, and natural environment around the shrine. How would we measure their success and what can we learn from their flourishing as we try to manage our society and our planet?