Beyond Compliance: Measuring to Learn, Improve, and Create Positive Change

“There’s lots of analysis, but little influence on how we use resources…people have to use the data to actually change how they work.” –Dan Corry, NPC

“The strength of your evidence is an indicator of the development of your organization.” – Joe Ludlow, Nesta

“Evidence does not yet determine [organizational] survival or expansion.” –Clive Martin, Clinks

“Half of the organizations trying to measure impact end up developing their own tools from scratch. It’s either a waste of resources or a reflection on what’s available.”–Tim Crabbe, Substance

It was a lively conversation, and a great step towards understanding the “pain points” around impact measurement. In the same vein, our new joint report with Wharton Social Impact Initiative (Beyond Compliance: Measuring to Learn, Improve, and Create Positive Change) presents our findings on how nonprofits and donors are tackling measurement questions. We want to know how measurement can be done better—and what we can accomplish by getting it right. While you’ll have to check out the report for the full story, there are a few key takeaways that I want to highlight here.

There’s a failure to distinguish between goals and indicators of organizational performance management vs. sector or community-level impact assessment.

This failure results in an all-too-familiar tension: The nonprofit leaders we interviewed wanted to know about what they could conceivably control because that helps them make their daily decisions better, while funders want to know about far-reaching or long-term impacts. For instance, a food bank wants to know how many hungry people it served, and whether the food they provide meets the nutritional needs of their clients. They might even look at the overall levels of hunger in the communities they serve. On the other hand, funders sometimes ask organizations to go beyond those measures and demonstrate a longer-term or broader impact—in our food bank example, that might be something like the overall health of the community—and they don’t always fund the organization to answer those questions.

How can funders and nonprofits bridge the gap? Focus on what you need to know to do your work better. That might not be what you can easily assess, and it might not be everything you wish you could know. Think about what you will do with what you learn. No chance it’ll change your actions? Then it’s probably not worth measuring.

Everyone has a responsibility to keep track of what they’re doing, but measurement approaches have to be “right-sized.”

Measurement strategies have to match an organization’s capacity and the state of current evidence. Every organization needs to measure their direct outputs (things like the number of meals served) in order to operate competently. But not every organization has the capacity to measure and report beyond that, and not every organization needs to do so. When there is already strong evidence supporting an approach, further measurement may be duplicating efforts for little gain. But if there’s little evidence or it’s hard to collect the data, an organization still needs to do its best to figure out what’s happening, even through something as simple and “non-scientific” as a focus group. And if a funder wants an organization to go beyond their “right-sized” approach, they may need to provide more support.

The (over)abundance of measurement and management resources sometimes just leads to confusion.

There are many tools to track data or guide a strategy, but most are targeted towards either funders or nonprofits, rather than both. This separation makes it harder for donors and nonprofits to get on the same page. In addition, most tools focus on a few specific measurement questions without guiding a user through the whole process of developing a strategy, collecting data, and making decisions based on what you’ve learned. The result: siloed conversations where people are touching different parts of the proverbial elephant in the room.

At the conference, in our research, and in our conversations with donors and nonprofits, it’s been clear that impact measurement generates both excitement and confusion. We’ve tackled impact definition in our recent blog series; with this report, we hope to bring similar clarity to the measurement side by providing concrete steps towards a more effective measurement strategy. Why does it matter? I’ll leave you with the words of one more conference participant:

“If you can’t figure out what you’re doing, don’t do it.” –Fran Perrin, Indigo Trust

To learn more about our findings, as well as what both nonprofits and donors can do to improve, download the free report here.