As part of my continuing fascination with this question of how to evaluate ‘success’ in advocacy practice, I’ve been reviewing pretty much any documents I can find that purport to provide help for advocates and donors seeking to answer it.
The last one I read is this Annie E. Casey Foundation-funded guide to measuring advocacy and policy efforts. The guide has an appendix that includes examples for different evaluation protocols: measuring core outcome areas, evaluating strategic progress, assessing short-term incremental objectives, gauging organizational capacity, and documenting case studies.
The report makes some good points that must be addressed to advance this field:
But all of this discussion really got me thinking about the fundamental premise of this exercise in evaluating advocacy: that we need to determine what are the steps along the theory of change and measure how an effort is advancing along them. In this particular report, those indicators are defined as shifts in social norms, increases in organizational capacity, increases in alliances, strengthened base of support, improved policies, and, finally, changes in impact. And that’s when I started thinking: really, as an advocate for social justice (or, pretending for a moment, as a donor for the same), do I care if a particular group has improved capacity or more friends or stronger constituents or even a more sympathetic environment in which to advocate? I mean, I do, I guess, but only to the extent to which it yields what I really care about–the social change. If it doesn’t, for whatever reason, I’m going to be hard pressed to say that I think it was a real success.
So, then, in pursuit of some way to legitimize advocacy and policy work, to bring it more in line with social service programming and its allegiance (rather belatedly) to evidence-based funding, are we reducing social change work to a caricature of its ideal self? Are we pretending that we know the formula for what makes a social movement happen and giving groups/organizations too much credit for going through those paces, regardless of whether, at the end of the day, they deliver the win? In so doing, are we emphasizing things that can be quite fleeting (a bigger coalition or better poll numbers, for example), or tangential (more staff or funding for the organization) and are unlikely to radically transform the lives of those impacted by the social problem in question? Should the only ‘interim’ step that counts be transformation in the lives of those affected, so that we can at least claim that as a victory if we fall short of the final goal?
And if we abandon this quest for evaluation because we conclude that it doesn’t capture the radical nature of the work we want to support, then where are we left in terms of informing our strategies, convincing our donors, and elevating those with the greatest chance of success? Is the answer to do the same kind of evaluation but with a different articulated purpose? Or do we need entirely different kinds of evaluation, more retrospective in nature, that can then try to inform future efforts? And, particularly vexing, as we transform our work to make it more, well, transformational, by relinquishing the appearance of control and giving more power to the crowd, we make it that much harder to track our impact–how can we know exactly what difference we made, when we didn’t control (or sometimes even know about) everything that “we” did?
As usual, I’m left with more questions than conclusions. What do you think? Have you evaluated your advocacy efforts? If so, what was worthwhile and what was frustrating about the experience? If not, what barriers kept you from that work? What questions do you still have, and what do you want to see explored as next steps in this exploration of advocacy evaluation?