Adapted from our presentation at the Jewish Federation General Assembly 2014.
There is a growing chorus insisting that donors and nonprofits ‘measure to results’ in order to ‘achieve impact’. Laudable goals. But how do you actually do that? How can your measurement practices provide real insight and confidence of progress, as opposed to simply turning into a bureaucratic compliance exercise? That was the question our team sought to answer in Beyond Compliance, our recent collaborative effort with Wharton Social Impact Initiative. In last week’s blog, we suggested 3 tips to start your social impact measurement efforts right. This week, we highlight 3 distinctions on the path to impact that can help avoid the confusion and frustration we heard among nonprofit leaders and the funders who support their work. In addition, we offer sample questions to consider when assessing each stage on the path to impact.
1. Organizational performance management focuses on the relationship among a nonprofit’s inputs, activities, outputs, and sometimes outcomes. Since these are the indicators that a nonprofit manager can control and count, understandably, these indicators are the ones that the nonprofit managers we spoke with found most immediately useful to informing their work. However, for both donors and nonprofit managers, it is important to remember that organizational performance management (e.g., efficiency) doesn’t always translate to high impact results (e.g., effectiveness). For example, a Scared Straight program could manage inputs, activities, and outputs so well that a growing number of children would be participating in the program (outputs). Yet, multiple randomized controlled trial studies have found that Scared Straight can increase the likelihood of juvenile delinquency. In other words, it can create negative impact despite high levels of organizational performance. Organizational performance metrics are useful and necessary to managing the day-to-day operations of a nonprofit. However, their usefulness in measuring progress towards social impact is only as good as the strength of the link between outputs and the outcomes sought.
Sample question to ask: What gives us confidence that the organizational performance metrics we are measuring are linked to the outcomes and impact we seek?
2. Organizational impact assessment considers not only a nonprofit’s performance, but also the relationship between all five components of building the evidence base for impact – inputs, activities, outputs, outcomes, impacts. Organizational leaders and funders often obtain information on all five components through a combination of in-house analyses, performance management data, and existing sector-level evidence. Since funders are often looking for indicators of longer-term change, many focus on these more distal indicators, which the nonprofits may lack the capacity (skills, systems, labor) to track well. Donors may want to consider support for impact assessment (e.g., effectiveness) above and beyond the performance management activities that are standard practice to ensure sound implementation.
Sample question to ask: Given the impact I seek, what outcomes can the nonprofit reasonably measure and attribute to its own efforts? Would the nonprofit find additional capacity for an impact assessment helpful?
3. Impact assessment at the sector level, often conducted by intermediaries outside the organization such as researchers and independent evaluators, focuses on the relationship between outputs, outcomes and impacts. For example, researchers have already found that the appropriate use of insecticide-treated bed nets, especially among pregnant women and children, can save lives at-risk of malaria. Similarly, we know from emerging brain research that having a positive relationship with at least one, caring adult has a protective effect on young children that can help build a healthy foundation for the rest of their lives. Such sector-level assessment of the link between outputs (e.g., number bed nets used appropriately, number of children with at least one effective caregiver) to ultimate outcomes and impact (e.g., lives saved, capacity to develop cognitively and psychologically despite deprivation or exposure to violence) can be used to inform all nonprofits working in a particular sector.
Sample question to ask: What sector-level knowledge should inform our activities and outputs? Do we have knowledge about the link between our activities, outputs and outcomes that could be useful to others working in our sector?
For more on Improving Your Social Impact Measurement Practice see Beyond Compliance: Measuring to Learn, Improve, and Create Positive Change and our blog series Five Myths and a Question About Impact.