Everyone seems to talk about impact, but who really gets to define impact and for whom? A collaborative effort between our team at the Center for High Impact Philanthropy and Women Moving Millions had us reflecting on this question as well as five myths that seem to surround discussion about impact.
- Myth 1: When folks talk about impact, they mean the same thing.
- Myth 2: There’s no impact unless you caused it.
- Myth 3: Impact is always positive.
- Myth 4: Some impacts can’t be measured.
- Myth 5: If you don’t see change, you’ve had no impact.
Myth 1: When folks talk about impact, they mean the same thing.
Everyone seems to be talking about it. Most say they want it. But who really gets to define impact and for whom? The definition of impact differs depending on who is speaking.
Technical definitions: The program evaluator perspective
There was broad consensus among program evaluation experts and some foundation leaders that impact is characterized by a tight causal link between action and change. In addition, that change – or impact – must be measurable, and generally must include consideration of what might have happened in the absence of the action.
Read More
Colloquial definition: The donor (including some foundations), nonprofit and impact investor perspective
In contrast to the relative consensus found among program evaluation experts, these groups demonstrated considerable diversity regarding the use and implied definitions of impact. Definitions ran along a continuum; some hewed closely to the technical (as described above) while others tended toward a more colloquial idea of impact as ‘something good’, not necessarily a measurable change directly attributable to a gift or grant. Individual donors were also more apt to describe impact as having two aspects: the impact of their philanthropy on others, but also the impact that their philanthropic giving and involvement had on the donors themselves.
Why do these different definitions matter? A clear definition of impact drives strategy, decisions, and dollars. Different – but unvoiced – assumptions about the definition of impact can create communication difficulties between donors, grantees, and beneficiaries; these miscommunications can get in the way of the very change sought.
Missing definition: The beneficiary perspective
Many social programs operate with a top-down approach: socioeconomic data indicate a need in a certain population, or a donor has a particular connection to a topic. Funders provide resources to nonprofits to implement programs to address the original need, collecting data to understand that program’s impact. But what if the target population would actually identify their need differently?
The market dynamics of philanthropy encourage this top-down approach, making donors the ‘customers’ for the services of nonprofits, even though they are not the ultimate consumers. For donors focused on improving the lives of others, two questions can be clarifying in the effort to define impact:
- What is the difference you want to make?
- Is that difference meaningful to the people you hope to help?
While the answers to those two questions may evolve over time, addressing them is the best first step for any donor seeking to make a positive difference in the lives of others. In an upcoming post, we’ll suggest additional ways to include beneficiary perspectives in both impact definition and program design.
Myth 2: There is no impact unless I caused it
In Risky Business, a piece on uncertainty in social interventions, Paul Brest asks us to imagine a group of villagers pushing a boulder up a hill, trying to get it out of a dangerously unstable spot. They’re moving slowly, and they ask for your help. You join in, and eventually you all get the boulder to the top. And that’s where the trouble starts – because the goal was reached, but who gets the credit? Was your contribution essential? Was it even helpful?
It’s a good analogy for high impact philanthropy, where donors and the organizations they support work to make progress against challenges that have been resistant to easy solutions. Success is the result of collective action.
Read More
Consider an organization working to improve the economic and social status of women. Let’s imagine that while women are heavily involved in agriculture in a particular country, they do not have the right to own land, which is a constraint on their earnings. This organization, along with others, mounts an advocacy campaign around women’s land rights. Even if there is a clear, visible success—passage of a land rights bill, for instance– it will be very difficult for any one of the many groups involved to say “This is our impact. We were the ones that made this happen.”
Attribution – “Owning the Impact”
As we discuss in our recent paper and in an earlier posting, for program evaluators, “impact” usually means the change that happened as a result of an action. That tight causal link between action and change– and the consideration of what would have happened without the action– is central to the evaluator’s technical definition of impact. Once that causal link is established, the impact of a particular action can be assessed.
For donors and the organizations they support, this is incredibly important: establishing the link between their interventions and the social change they seek gives them confidence that they are on the right path and that resources were put to good use. Considering attribution is crucial for anyone involved in working for a change. But focusing too narrowly on attribution can bring its own set of problems—because sometimes real impact can occur in situations where: a.) multiple factors or actors contributed to the observable change, and/or b.) it’s very difficult or prohibitively costly to accurately pinpoint which of many factors is most responsible for the observed change.
Contribution – “Part of the Winning Team”
The fact that the women’s rights organization cannot “own” the ultimate success of the advocacy campaign does not, however, mean that there was no impact or that the organization is off the hook for thinking about its own impact. While the precision of a randomized control trial is clearly off the table (there is no possibility of a control with a national advocacy campaign, and besides, they were one of many groups involved), there are other methods and metrics that can be used to understand the importance of an organization’s contribution (e.g. interviews with beneficiaries, other actors and lawmakers about the organization’s strategy and influence).
Returning to our boulder, one thing is certain: the boulder started out at the bottom of the hill, and now it’s at the top. That’s measurable, observable impact. And one other thing is certain: it got to the top because you and that group of people pushed it. Did you (or any one of you) cause anything that wouldn’t have happened otherwise? Perhaps you’ll never know with perfect certainty. But the boulder is where it needs to be.
Myth 3: Impact is always positive.
When most donors or non-profits talk about social impact, they mean a positive change in someone’s life or situation: “something good.” This positive change is what donors and non-profits are seeking to bring about. Sometimes, however, the impact you seek is not the same as the impact you create. Impact can be unintended, for better or for worse. It can be negative, or a combination of positive and negative change—and even negative change can sometimes be a sign of progress! Confused? Let’s take a look at some examples.
Read More
Negative Impact as Evidence of Ineffectiveness
Scared Straight is a once popular program meant to lower levels of youth crime and incarceration by exposing at-risk youth to the reality of prison. It was implemented in many locations before undergoing a rigorous assessment of its results—and when those evaluations were finally performed, investigators found that the program did not lower criminal behavior. Worse, it actually increased the risk of criminal behavior and incarceration among youth. Why? It turned out that exposure to prison made the consequences of criminal action less scary for these youth, rather than more.
Negative Impact as Unintended Consequence of Positive Impact
Negative impact can also be an unintended consequence of an apparently successful intervention. For example, some evaluations of micro-credit programs in India have found that the more successfully a program raises women’s income levels, the more likely it is for male earners in the household to shift responsibility for the household’s economic security onto women—while also taking control of women’s income. Worse, women also reported increased violence as a result of tensions around their new economic status.
While this is a frustrating result, the good news is that it was discovered at all. That discovery was only possible because the evaluators considered the full range of the program’s potential impact. If they had only looked for evidence of positive impact—the impact they were seeking and that they achieved—they would have missed all of the other ways that this intervention was affecting the lives of the women and girls they were trying to help. The program would have looked like an unqualified success, and its organizers would have missed the opportunity to improve their services and get closer to the true positive change they sought.
Occasionally, Negative Impact as Evidence of Effectiveness
More counter-intuitively, negative impact or change can also sometimes signal that an intervention is working. In efforts to challenge traditional power structures, one step forward can trigger a backlash that sends you two steps back.
When the suffragettes in the United States were advocating for women’s right to vote, ridicule from the media intensified after the high-profile Seneca Falls Convention. If you were an evaluator at the time, and you were looking at popular opinion to judge their progress, you might have thought that the overall impact of the conference was negative. But, as we now know, that effort was successful overall, and many credit that convention as the birth of the U.S. women’s rights movement. In that case and in some others, negative impact doesn’t simply mean lack of progress or effectiveness– it means that the external context has changed in response. This dynamic is particularly relevant for women and girls work, which is so often in opposition to established power structures.
Do No Harm: The Importance of Evaluation
All of these examples point to the same thing: impact itself is value-neutral. It doesn’t just mean the good that you do. It can be the good that you do, the harm that you create, or even both. In seeking positive impact or change, donors would do well to remember the Hippocratic oath: “Do no harm.”
In that spirit, evaluation is a critical tool to seeing clearly. When you take off the rose-colored glasses, you can begin to understand the whole range of your impact, and that knowledge can help you move closer and closer to the change you do seek.
Myth 4: Some impacts can't be measured.
When people say an impact can’t be measured, they often mean that they can’t practically measure that impact to the degree of accuracy they want and within the timeframe and budget they have. In other words, it’s not that the impact is technically impossible to measure, it’s that people believe it’s impractical to measure given known, existing tools and limited resources. It’s true that certain changes (e.g., an increase in income) may be much easier and cheaper to measure reliably than something like a change in social status of an oppressed segment of society (e.g., the Dalit population in India). However, saying something is impossible to measure makes it dangerously easy for donors and nonprofits to be let off the hook. What’s more, it can lead to a focus on what is easy to measure as opposed to what matters.
Read More
The Perfect Can Be the Enemy of the Good
There are genuine limitations on what most organizations can feasibly measure at any point in time. But even if an impact can’t be measured precisely, measurement is still possible. Perhaps there’s a proxy indicator—something easier to measure that is related to the goal you’re working towards. In the Dalit social status example, maybe publicly available data can give you a picture of changing employment opportunities available to this group. Or perhaps a limited survey of local perceptions of this group repeated over a period of time could give you a sense of whether attitudes toward Dalits are less negative, a proxy indicator for improvement in social status. And maybe you won’t be able to determine exactly what impact you had, but you can still get a sense of how much progress has been made overall in the areas you care about. The right approach will vary case by case, but nobody is off the hook when it comes to trying to measure their progress. Without some system of measuring, donors and nonprofits are flying blind.
Moving forward: Know your goals, know your trade-offs
Anything can be measured with enough time and money, and nobody can ignore their responsibility to measure their impact. But how do you combine those concepts to make decisions about your own measurement approach?
The key is in your goals. Good measurement clarifies your path to action. Put dollars towards the measurements that can help you or your grantees do their jobs better. You may not need to know your impact precisely, as long as you have reasonable evidence that you’re moving in the right direction. Or maybe you’re trying something brand new, and you need to measure many things very carefully in order to ensure that you’re doing more good than harm.
Measurement costs money, and there will be tradeoffs between the perfect, the good, and even the adequate. But as long as you’re making those tradeoffs with open eyes, you’ll be moving closer to achieving the impact you seek.
Myth 5: If I don’t see change, I’ve had no impact
Just as impact isn’t always positive, it’s not always visible as progress towards the change you seek. But if you don’t see change, you might still be having an impact…if you know how to look.
There are many situations where simply holding the line is a victory. Take the example of an organization working to improve learning levels for disadvantaged students. Year after year, the students in the program continue to read below grade level. Disappointing, right?
Except it isn’t. Learning is cumulative and in the absence of this organization, the learning gap would have widened. Dramatically. Students would have fallen farther and farther behind. The organization may not be getting kids all the way to the level desired, but it is definitely having an impact. The key to seeing this impact is comparison – what would have happened in the absence of the program.
Read More
Looking beyond “Before and After” to “Compared to What”
Often, before and after is a starting point for measuring impact, but it can be far from the full story. For example, suppose you’re considering the impact of a group that advocates for women’s rights. You fund this organization’s work to change women’s legal status in the region. If you see restrictive laws—perhaps women aren’t legally allowed to drive, or to travel without a male family member—and those restrictions haven’t loosened in the time that group has been active, you might conclude that the advocacy group isn’t very effective.
Where’s the flaw in that? In this case, before your funding and after your funding, the restrictive laws are still on the books. you didn’t see a change for the better, but you also didn’t see a change for the worse. Perhaps the advocacy group blocked the passage of even more restrictive laws, protecting existing rights. No new rights were added and no change is visible, but the group had impact nonetheless.
Slow and Steady Might Win the Race…Eventually
With big, ambitious goals, impact can’t always be captured within the time frame of a typical evaluation. The suffragette movement in the United States, for instance, was incredibly ambitious and audacious at the time. Progress was slow, and came in fits and starts with many setbacks. If there had been an evaluator trying to assess the suffragettes’ impact early on, they probably wouldn’t have found much– a three- or five- or even ten-year evaluation at the beginning of the movement might not have shown significant change. During that time, however, activists were laying the groundwork necessary for the movement’s eventual, more visible success.
Knowing where and how to look
Sometimes when you don’t see change, that is indeed an indication that you are not achieving your desired impact, and it is time to try a different approach. But sometimes there is impact: it is just hard to see. Before shifting gears, ask yourself: what would happen without this intervention? Are there other indicators that we should be looking at? Is the time frame reasonable for the impact we expect? Answering these questions should help determine whether it makes sense to stay the course, course correct . . . .or try cutting down a different tree.
Originally published in 2013 as part of a blog series about What Are We Talking About When We Talk About Impact?