photo credit, Michael Lokner, via Flickr
A missing piece in the discussion of advocacy evaluation has been the voices of advocates themselves. Too busy changing the world to be included in the discussion about how we measure those change efforts, the conversation has been happening almost behind their/our backs, and I was really glad to see this report, spearheaded by the Atlantic Philanthropies and Annie E. Casey Foundation, two of the leading philanthropic voices on social change, and, it turns out, evaluation of the same.
The purpose of this report is to provide nonprofit advocates with a platform to discuss their experiences with advocacy evaluation and to open communication with evaluators and donors about how to improve the enterprise. It opens, though, with the results of a survey of more than 200 advocacy grantees of some of the leading foundations in advocacy, and those results are themselves instructive for forming a portrait of the status of nonprofit advocacy.
Not surprisingly, only 25% of respondents have ever evaluated their advocacy. Even fewer of those have had the assistance of an external evaluator (which is significant given the limited experience of many nonprofit types in doing systematic evaluation of any kind)–only 17% of the total sample. Of course, I also question how useful the exchange with the external evaluators has been for advocates; anyone who has participated in an independent evaluation knows that evaluators vary in their willingness to actively engage program leaders in the process and shape a product that will meet the agency’s needs.
Sixty percent of nonprofit advocates are working within organizations with budgets less than $1 million annually; fully half have budgets less than $500,000/year. More than half of respondents, furthermore, dedicate fewer than half of their resources to advocacy, with smaller organizations more likely to be ‘purely’ advocacy. Human services are the most common advocacy priority of the respondents, at 40%. Advocates are mostly engaged in state, local, and regional work; only 21% are substantially working on national advocacy. That’s interesting, I think, not surprising, given the logistical and political challenges of impacting Congress, but rather discouraging given the rich possibilities of effective congressional advocacy.
Advocates are overwhelmingly focused on legislative advocacy (56%). This appears to include a strong grassroots lobbying component, though, with 47% citing participation in community organizing also. Only 12% are working on judicial strategies and only 5% on administrative/regulatory advocacy. That echoes what I often hear from nonprofit leaders when we talk about advocacy; they tend to think legislative work first and foremost and are often surprised and even confused when I talk about other types of engagement as ‘advocacy’. One of the findings that most resonated with me was that, despite the preference for legislative advocacy, only 22% of advocates judged legislative work as the most effective strategy!
Important for me as I continue exploring my consulting work with nonprofit organizations was the statement that research and communications assistance are the capacities that advocates view as most lacking. That surprised me, because I would think that those tools would be easiest to find from other sources, and it has caused me to rethink somewhat what I need to be discussing with nonprofit leaders.
As far as actual advocacy evaluation, those advocates that have done it note that it has helped them to refine their strategies, make the case for more funding, and pursue staffing changes. They point to lack of resources for evaluation, obviously, as a barrier, but also the need for better interim goals and an attitude that sees evaluation as a capacity-building tool rather than a punitive audit.
As the report states, the field of advocacy evaluation was virtually nonexistent not even 10 years ago and is now developing rather dramatically. The authors conclude by calling for advocacy evaluation to help advocates better change the world. In the race towards justice, they say, we need to know when to sprint and when to save our strength, and good advocacy evaluation can help us reach the finish line.
But HOW do we measure advocacy success?
As part of my continuing fascination with this question of how to evaluate ‘success’ in advocacy practice, I’ve been reviewing pretty much any documents I can find that purport to provide help for advocates and donors seeking to answer it.
The last one I read is this Annie E. Casey Foundation-funded guide to measuring advocacy and policy efforts. The guide has an appendix that includes examples for different evaluation protocols: measuring core outcome areas, evaluating strategic progress, assessing short-term incremental objectives, gauging organizational capacity, and documenting case studies.
The report makes some good points that must be addressed to advance this field:
But all of this discussion really got me thinking about the fundamental premise of this exercise in evaluating advocacy: that we need to determine what are the steps along the theory of change and measure how an effort is advancing along them. In this particular report, those indicators are defined as shifts in social norms, increases in organizational capacity, increases in alliances, strengthened base of support, improved policies, and, finally, changes in impact. And that’s when I started thinking: really, as an advocate for social justice (or, pretending for a moment, as a donor for the same), do I care if a particular group has improved capacity or more friends or stronger constituents or even a more sympathetic environment in which to advocate? I mean, I do, I guess, but only to the extent to which it yields what I really care about–the social change. If it doesn’t, for whatever reason, I’m going to be hard pressed to say that I think it was a real success.
So, then, in pursuit of some way to legitimize advocacy and policy work, to bring it more in line with social service programming and its allegiance (rather belatedly) to evidence-based funding, are we reducing social change work to a caricature of its ideal self? Are we pretending that we know the formula for what makes a social movement happen and giving groups/organizations too much credit for going through those paces, regardless of whether, at the end of the day, they deliver the win? In so doing, are we emphasizing things that can be quite fleeting (a bigger coalition or better poll numbers, for example), or tangential (more staff or funding for the organization) and are unlikely to radically transform the lives of those impacted by the social problem in question? Should the only ‘interim’ step that counts be transformation in the lives of those affected, so that we can at least claim that as a victory if we fall short of the final goal?
And if we abandon this quest for evaluation because we conclude that it doesn’t capture the radical nature of the work we want to support, then where are we left in terms of informing our strategies, convincing our donors, and elevating those with the greatest chance of success? Is the answer to do the same kind of evaluation but with a different articulated purpose? Or do we need entirely different kinds of evaluation, more retrospective in nature, that can then try to inform future efforts? And, particularly vexing, as we transform our work to make it more, well, transformational, by relinquishing the appearance of control and giving more power to the crowd, we make it that much harder to track our impact–how can we know exactly what difference we made, when we didn’t control (or sometimes even know about) everything that “we” did?
As usual, I’m left with more questions than conclusions. What do you think? Have you evaluated your advocacy efforts? If so, what was worthwhile and what was frustrating about the experience? If not, what barriers kept you from that work? What questions do you still have, and what do you want to see explored as next steps in this exploration of advocacy evaluation?
Share this:
Like this:
→ 2 Comments
Posted in Analysis and Commentary
Tagged advocacy, evaluation