Tag Archives: evaluation

Execution Matters: Evaluation and Getting Advocacy Right

Posted on December 8, 2011 by | Leave a comment

In this final post on The Future of Nonprofits, I want to focus on a key point from early in the book, about how many (perhaps most) nonprofit organizations and their leaders are far better at coming up with creative approaches to solving problems (and accomplishing their core missions) than executing those ideas consistently and effectively.

It’s not meant to be a total bash on nonprofits and their employees; really, given how under-resourced most nonprofits are (related to our pathological aversion to investing in “overhead”–those important functions that, in fact, enable our programs and services to succeed) and how much of our working hours we spend trying to sort of keep our heads above water and look effective (whether or not we really know what “effective” would look like in our specific context), it’s not surprising that we seldom have the chance to step back and think about the kinds of processes and structures within our organizations (and our own workdays) that would raise our execution ability to a standard of excellence.

Instead, we’re always trying to do more with less (except when we’re admittedly doing less with our less, and busy making excuses for that). We stop doing some of what we should be doing, and close ourselves off to the possibilities of what we could be doing, in ways that mean, somewhat paradoxically, that we have to keep coming up with new inventions to increase our creativity, in order to compensate for how poorly we’re managing to pull off what it is that we do.

Exhausted yet?

In the advocacy context specifically, I see this when we develop campaigns that drift towards a new gimmick, or rely excessively on a particular technology, as though those are the tricks that will deliver the outcome we seek. We’re continually trying to one-up ourselves in terms of a slogan or a media event or a high-profile endorser, when what we should really focus on is hiring really good organizers, or investing in our relationships with our constituents, or personally connecting with every target policymaker (or all of the above). Or, we jump from issue to issue, diluting our potency and confusing our targets, lulling ourselves with the truth that “there are so many important causes out there.”

The Future of Nonprofits has a chart that shows how we can increase our aggregate impact either by raising our execution ability, even rather modestly, or by dramatically expanding our pool of creative ideas. There’s arguably a need for both. Given limited resources, though, it’s much more efficient to focus on marginally improving our delivery, especially because it can ripple into other areas of our organizational functioning, in terms of relationships built and skills enhanced.

And, so, what would it take to improve our execution in the advocacy arena?

First, we have to rigorously evaluate what it is that we’re doing: what isn’t working, and what is, and what really tips the balance. We have to identify our organizational and individual advocacy capacities, build up the areas where we are weakest, and develop benchmarks for what we should be delivering. We need to fully investigate where our own efforts have fallen short before assuming that our advocacy failures are to be blamed on adverse political or economic conditions. We need transparency and accountability for what our campaigns set out to do, just as we do in the fundraising and direct services arenas.

And we need organizational cultures committed not just to innovation, and not just to advocacy, but to excellence, and to intellectual honesty about how well we’re executing our most core programmatic functions, too.

A few weeks ago, I was reading an article about advocacy evaluation when a Board member for one of the organizations for which I do consulting (we were volunteering at the same event, and I’m never without reading material) looked over my shoulder. She shook her head at the article’s premise that the field of advocac evaluation is far behind that of traditional programmatic assessment, and I think that her critique is largely valid: too often, our obvious good works, in the nonprofit sector, excuse the fact that they’re not always done well.

In advocacy as in the rest of our endeavors, that’s an oversight we cannot afford.

In the new year, we may find that we don’t need to continually come up with as many new strategies or “innovative” approaches, if we are consistently doing what we do very, very well. And implementing evaluation systems that allow us–no, require us–to know when that is the case.

We won’t have to take as many shots, in other words, if we can hit them when we need to.

→ Leave a comment

Posted in Analysis and Commentary

Tagged , , ,

Evaluating Advocacy, de nuevo

Posted on September 22, 2011 by | Leave a comment

It’s “update” week at Classroom to Capitol.

As I read through previous posts for my summer maternity break hiatus, I found a few that I really wanted to revisit, rather than repost. This is the last of the three that I have chosen for this week, with new thoughts and, of course, new questions.

One of my academic interests over the past couple of years has related to questions of how we evaluate advocacy efforts: How do we know advocacy “success”, short of absolute policy change, so that we can build on it? How can we assess organizational capacity for advocacy (to have a better sense of who will succeed, and also to know where to invest)? What kinds of interim goals should form part of an advocacy strategy, and what kinds of benchmark measures should mark our progress?

Over the past year, I’ve had the chance to apply my study and training in this area to practice through work with the Sunflower Foundation and its advocacy initiatives. It’s tremendously rewarding to be able to not only help individual advocates and nonprofit organizations seeking to develop an advocacy voice figure out how they’ll gauge their work, but also to be part of this evolving field and to work alongside a funder investing so much energy in contributing to good practice around these questions, too.

I love it.

More recently, my work with the Sunflower Foundation has allowed me to contribute to some of the Alliance for Justice’s conversations about how they evaluate advocacy, both on the front end (in terms of organizational capacity) and as advocates and their donors seek to determine the relative impact of different advocacy strategies. I’m very excited about AFJ’s revised advocacy capacity tool, which will be available online soon, and particularly about their approach to this work, which is aimed at getting as many organizations as possible to evaluate their own capacity (in a variety of areas; it’s a pretty thorough look at the inputs that we believe position an organization to succeed in advocacy) in order to build the field of knowledge about what makes a difference in ultimate advocacy success.

In Kansas, our hope is to eventually be able to help a given nonprofit organization know where it sits, on some of these capacity measures, compared to an aggregate of its peers, and also to develop strategies that are at least likely to lead to enhanced capacity in those same areas, so that we can build a strong cadre of advocate organizations across the geography and in different fields.

Refining these measures, and these tools, is important not just because we want to know what works in advocacy (so that we can get better and better and win more and more often), but also because being able to demonstrate how our theory of change is leading to tangible results should push more funders to feel comfortable supporting advocacy (or, at least, to expose that their real fears are taking a stand on controversial issues, and we need to know that, too!). We’ve come quite far in the past few years, such that advocates are no longer left to flounder to come up with benchmarks, and no longer grasping for what might make sense for measurement. It’s tremendously exciting, for the academic side of me, but especially for the promise that these tools hold in making our advocacy more robust, more acclaimed, and, ultimately, more integrated into what nonprofit organizations do all day.

And it’s great to be part of it.

If your organization is interested in advocacy evaluation and/or assessing your organizational capacity for advocacy, we should talk! I’d love to connect you to resources and (full disclosure!) include you in some of our field-building efforts, too. Because once we know what works, we just have to gather the courage to go after the money to do it.

And, then, we’re unstoppable.

→ Leave a comment

Posted in My New Favorite Thing

Tagged , ,

Evaluating Advocacy: Of jumping hoops and learning loops

Posted on December 9, 2010 by | Leave a comment

photo credit, 2007 Powwow, Smithsonian Institution via Flickr Creative Commons

If you haven’t commented yet this week, this post is your last chance! (Except, of course, that you can go back to post on one of the other two!). Tomorrow, I’ll announce the winner of the free copy of The Networked Nonprofit!

There is a lot of content in the book about how organizations can, and should, approach social media as a sort of experiment, building in mechanisms that will help them to learn quickly, and well, from what they’re trying, so that they can modify it as needed. They stress a real intentionality in this approach, an emphasis, from the very beginning, on defining what it is that we hope to accomplish, and the measures that we’ll use to help us get there. They also create space, though, for different organizations (or even different campaigns within the same organization) to define “success” differently, and they caution against reducing social media to a mere numbers game.

As I wrap up a contract evaluating an advocacy initiative for a foundation here in Kansas, and continue my reading, speaking, and contemplating about how to evaluate advocacy, and why such evaluation is so important, there is a lot from the evaluation discussion in The Networked Nonprofit that I believe applies to this endeavor of advocacy evaluation, too.

Foremost is the idea that evaluation should be actionable, that is, evaluation should give practitioners real information that they can really use, and be imminently valuable to them as a real-time check on what they’re trying. Having such information not only improves practitioners’ ability to change what’s not working, but also serves to increase organizations’ willingness to take risks (like trying advocacy or social media), because there’s comfort in knowing that we’ll be able to tell what’s working and what’s not.

They call this “learning loops”, and the way that they talk about it will sound very appealing, I believe, to anyone who has participated in the “other” kind of evaluation–that which is designed by a third party to meet a donor’s, not the constituents’ or the practitioners’, needs for information, that which produces a bound report years after anyone stopped caring (or even remembering) what is being evaluated, and that which uses criteria that don’t remotely resemble ‘success’ according to the perspectives of those really doing the work.

The details on learning loops, below, come from Kanter’s work, but this is my conceptualization of how the idea applies to advocacy evaluation, and how it differs from “traditional” evaluation.

  • Learning loops emphasize planning for evaluation from the beginning, involving stakeholders in defining success and choosing measures, rather than tacking an evaluation study on at the end.
  • Learning loops provide real-time information, so that it can be applied to change course mid-stream. Organizations take a few hours every month to ask themselves questions about what’s working and what’s not, and they adjust workplans and even strategic goals to account for what they’re learning.
  • Practitioners collect the data that feed the learning loops, and they help to interpret them. They measure engagement (who’s connecting with our work, and what are they saying about that connection?), return on investment (the traction that they’re getting from specific tactics, and which ones deserve more attention), and social change (what is actually getting better about the problems that concern us).
  • Participants engage in a process of reflection as a part of the learning loop; the priority is on really learning something from the evaluation endeavor, and there’s a recognition that we learn best when we have a chance to process with others.
  • Learning loops use low-cost, relatively low-risk experiments, to test assumptions and begin the process of organizational change, as a prelude to lasting social change, rather than waiting until the end of an expensive and lengthy activity to see if it worked.

    There is still a lot that’s hard about evaluating advocacy, and there are still a lot of variables that impinge on our ability to measure precisely the impact of our interventions.

    Still, this kind of advocacy evaluation, woven seamlessly into the practice of advocacy itself, holds tremendous promise for overcoming our collective resistant to the idea and, therefore, beginning to build a body of knowledge that will help us get better at doing advocacy evaluation.

    And it starts with changing how we think about evaluation, not as a hoop through which some funder says we must jump, but instead as a part of the process of social change, and one that gives us another tool through which to improve our work.

    If you’ve been a participant in either approach to evaluation, especially evaluating advocacy or social media efforts, how were those experiences? How might you implement learning loops in your organization, specifically in your advocacy? How does this change how you think about evaluation?

    • → Leave a comment

    Posted in Analysis and Commentary

    Tagged , ,

    What does civic engagement look like, really?

    Posted on October 21, 2010 by | Leave a comment

    photo credit, Library of Congress, via Flickr Commons


    Social workers, especially us “macro” types, use a lot of pretty fuzzy language sometimes. What does “empowerment” really mean after all? How do we know effective advocacy when we see it?

    And what, really, is “civic engagement”, and how in the world do we measure that?

    Answering this question is important not just because it’s never a good idea to spend energy talking about something without really having any idea what we’re actually talking about, but also because defining and measuring and evaluating our civic engagement work is about accountability and integrity, which, after all, are some of the goals towards which our civic engagement work is focused in the first place.

    We know that civic engagement is far more than getting people registered to vote, or even than getting them to the polls. I remember a course that I took from Ernesto Cortes, of the Industrial Areas Foundation, in graduate school, and how he talked about how reducing civic engagement, and the exercise of our citizenship, to voting alone, really makes it essentially another aspect of consumerism–choosing between this or that preformulated option, which, of course, isn’t very engaging at all.

    But the other stuff, beyond voting, is even harder to measure and truly conceptualize: what does it look like to be authentically involved in the governance of one’s own community, or one’s own life, and how do we begin to track and evaluate that engagement on a broad scale?

    The folks at the Building Movement Project (I know, I know–I’m a bit obsessed) have a new paper, Evidence of Change, which discusses evaluating civic engagement efforts and, I believe, offers, if not a roadmap, at least some sparks of guidance for organizations trying to be clear about their goals in this client empowerment work and, ultimately, demonstrate its tangible value.

    I’ve been thinking a lot about this, because I really believe that there are particular opportunities for advancement of advocacy and civic engagement as legitimate activities, and, really, core strategies, of social service nonprofit organizations, but we’ll never solidify a place for them if we can’t figure out how to assess and communicate about what we’re doing, and why it matters.

    Some of the new insights for me from this, most recent, discussion:

  • We can’t measure civic engagement by looking only at the individuals (for example, our clients) involved; truly meaningful civic engagement should be transformative not just for those people, but also for the “host” organization’s capacity for social change, and for the society and institutional structures their engagement is aimed at changing.
  • Rigorously evaluating civic engagement work requires, for many nonprofit social service organizations, TWO significant culture shifts–first towards this kind of empowerment work as a core part of the agency’s operations, and second towards seeing formal evaluation as integral to the organization’s mission. No wonder it’s so hard, and so rare.
  • Just as we’re still in the process of developing new models of social service organizations that integrate advocacy and civic engagement in their direct service work, so, too, do we need to develop new models of evaluation, able to meet the demands of these kinds of nonlinear change processes. And we need the space, within academia and especially philanthropy, for these new evaluation methods to gain legitimacy.

    So getting out the vote among our clients and allies is obviously important. And being able to quantify the electoral impact of our work, and how it changes conversations about the issues we care about, is important in garnering the resources we’ll need to support its continuation. Absolutely.

    But we want more for those we have the honor to serve than a choice between candidate A and candidate B. We want them to be more than consumers–we see them and know them as stakeholders, capable of helping to build the kind of society we want for all of us.

    And that takes the kind of civic engagement that moves mountains.

    So we’d better be ready to measure how far they’ve come.

    • → Leave a comment

    Posted in Questions you're embarrassed to ask

    Tagged , , ,

    Confessions of a Nonprofit Consultant

    Posted on September 27, 2010 by | Leave a comment

    I’m just about one year after having commenced my nonprofit consulting work in earnest, and I can sincerely say that I feel…ambivalent about it.

    Especially in this economy, I’m asked pretty frequently by students about my consulting work, and the kinds of opportunities it provides. I tell them pretty unequivocably that it’s not where I’d recommend starting a career; both because the connections that I have make it feasible for me to build a job out of it, and because I really needed the legitimacy and structure that an organizational home offers, at one point.

    But, for me, at this point in my life, it makes a tremendous amount of sense: I get to work with organizations I care about, contribute to research and policy on a variety of causes, and have the flexibility to be a more hands-on parent during the day.

    But, to me, my consulting work, and whether or not it’s “working” has to be about a lot more than me, and my scheduling preferences. And, so, I guess as an exercise for me as much as anything, I’ve been thinking about what I like about it, in terms of my interactions with organizations, and what prompts some of those more complex emotions.

    First, the completely excellent:

  • 1. I’m relatively unbound by resource limitations and pragmatic organizational constraints. I get to make the suggestions and offer the critiques that I find most compelling, without having to always worry first whether they’re totally feasible.
  • 2. I build organizational capacity. My favorite work is doing training, providing materials, answering questions–building up the staff, Board members, and volunteers who work directly with clients and the social problems that plague us, and increasing their ability to make a difference in their communities.
  • 3. I have space in my life and legal and political latitude to be overtly political, when it’s necessary–I represent only myself, on a daily basis, and not always a 501(c)3 organization.
  • 4. I can tailor my work to an organization’s actual needs, in every way from how I communicate with staff to how much I charge to how much power and control they retain over the process.

    And, what I’m still grappling with:

  • 1. I seldom get to see things through to any sort of real implementation or follow up. There’s obviously never ‘completion’ in any social change work, but a lot of my consulting involves drafting recommendations, putting new processes in place, creating templates…without much knowledge of the degree to which those tools will really be used to effect change at the organization.
  • 2. That same freedom, to be divorced from resource and practical constraints, can mean that my recommendations seem, well, divorced from reality. A best practice is only a best practice, after all, if it can really be practiced, and I know that I sometimes set standards that don’t make sense in the daily lives of agency personnel.
  • 3. I have some real ambivalence about how my work promotes the idea of unmooring workers from an organizational context; given how critical I am of the increasing shift of risk to employees, and away from institutions, I’m very aware of how I’m contributing to this very trend. In some cases, this is less problematic, because I know that the work I’m performing wouldn’t be done at all if not for me, but, in others, I wonder if I’m not making it possible for organizations to avoid hiring regular, full-time staff for some of these functions.
  • 4. And, finally, I’m very cognizant of needing to always keep my work focused on building up the organization, rather than ever seeming like someone who swoops in, provides “expert” advice, and then leaves. That’s critical not just because of how we know change becomes institutionalized within an organization, but also because of my fundamental belief in empowerment.

    I’d love to hear from other nonprofit consultants (and it’s certainly a growing industry!) about how you balance these tensions, and what you see as the most rewarding and most challenging parts of this work. And, nonprofit leaders who have worked with consultants, what do you look for in a consultant? What contributions do consultants make to your organization that truly enrich your work, and how do they fall short of this goal?

    • → Leave a comment

    Posted in Analysis and Commentary

    Tagged , , ,

    “Creative destruction” and nonprofit consolidation

    Posted on June 1, 2010 by | 4 Comments

    photo credit, dhnieman, via flickr

    Let’s start summer right, folks!

    This week, I’m doing three posts related to concepts in Robert Egger’s book, Begging for Change. He has become one of my very favorite writers, speakers, and thinkers on topics related to nonprofit organizations and social change work, and I find myself continually challenged by his perspectives, going through an entire pack of sticky notes to mark pages I want to remember. If you haven’t read the book, you should, but, first, do me a favor and read the posts this week and share your thoughts about how I’ve connected his ideas (from 2004) to today’s not-for-profit landscape. And, you know, you can enjoy the sunshine, too.

    Almost every semester, I am struck by the inclusion in at least several students’ career goals of something related to “start my own nonprofit organization”. This semester, I had students express interest in starting adoption agencies and drug treatment programs and mentoring projects for at-risk youth. I also receive relatively regular inquiries from former students exploring starting their own nonprofits. Myself, I honestly don’t have an entrepreneurial bone in my body; I’d never want to start my own organization, and I have even turned down a few promotions because I’d have to spend more time paying attention to payroll and less time generally agitating.

    But it would certainly appear that the desire to be one’s own boss, combined with passion about the social problems we face, leads at least many within the social services sphere to dream of setting up their own shop. Now, you know that I don’t believe that duplication of effort is, necessarily, an evil. All things being equal, two excellent organizations working on the same social challenge should mean that, together (or apart, but headed towards the same goal), they reach victory more quickly than one would alone. And I like to win.

    But Egger’s book, and my conversations with my students, and some discussion in the blogosphere (see, in particular, Lucy Bernholz’s awesome post on peer-reviewed nonprofits) and the traditional media have me thinking about what it would take to really change the game in terms of the ‘marketplace’ for nonprofit organizations. I mean, why does it often seem easier to start a nonprofit than a for-profit business, when, most of the time, our goals are much more ambitious (making success, therefore, seem more elusive)? Why are there nonprofit organizations still in business long after they ceased to really meet a compelling social need? Why do our current organizations often fail to capture the imagination of bright and talented graduates, pushing them to envision charting their own path instead (especially when we have a near-crisis in executive leadership in the sector)? Why is the recent uptick in nonprofit mergers seen as a sign of doom, when for-profit mergers are often hailed?

    Egger calls for “creative destruction”, hence the title of this post–the consolidation or collapse of the most ineffective and wasteful organizations. He acknowledges that such a recommendation creates more questions than it answers: where would one draw this line? How do we define success? Without good benchmarks or a good roadmap to outcomes, how can we measure waste? How can we promote social enterprises that bring more social value to the for-profit sector, and are there places and ways in which such an approach is inappropriate? Questions that he doesn’t ask, but which must be addressed, include how to balance between consolidation and the sustenance of ‘niche’ organizations that effectively serve small, particular constituencies; and how long organizations should have to try innovative (but failing) approaches to entrenched problems. Should all nonprofit organizations be held to the same standards, once we can figure out what these standards should be? Or should they be more locally defined, taking into account differences in contexts and inputs? What are the responsibilities of donors, who give for all kinds of ‘illogical’ reasons, to stop supporting organizations that are failing in their missions?

    I read Egger’s discussion of creative destruction with a different lens, I expect, than that through which it was written: that was 2004 and this is 2010, and some even excellent nonprofit organizations are collapsing because of the dramatic dropoff in foundation and corporate giving, in particular. But, also, perhaps because I just finished my class on macro systems, I thought back to a lecture early in the semester when we talked about the role of stress in social systems, and how stress can provide the impetus for real transformation, as systems struggle to adapt to new and harsher realities.

    And that’s where I conclude this post, with questions, a little bit of despair, and some hope thrown in. I’d greatly appreciate others’ thoughts as I muddle through this–where do we go from here? Since we don’t have a ‘market’, per se, how do we make these decisions in a way that respects our shared values as a social service sector? How do we understand and communicate the stakes involved in perpetuating the status quo? And how do we use the current economic climate as the “crisis” that could best prod us in the right direction?

    → 4 Comments

    Posted in Analysis and Commentary

    Tagged , ,

    Evaluating Advocacy–the Organizational Capacity Question

    Posted on April 14, 2010 by | Leave a comment

    I really, really wish that I would have read this guide about 8 years ago (in my defense, it was published in January of this year). It’s interesting–while it’s really designed for “advocacy organizations”– those that primarily (or exclusively) conduct advocacy campaigns–I really think it’s even more valuable for nonprofit organizations that are layering advocacy onto social service work, as I did in my work with El Centro, Inc.

    The premise of this guide is that, while the new attention to evaluating advocacy campaigns is absolutely critical, a neglected area of study relates to organizations and their organizational capacity to support social change work. Too often, we nonprofit advocates, if we’re thinking capacity-building at all (rather than just “must get 500 people to this rally”, “must get Representative XYZ to vote no”, “must prepare leader ABC for press conference”), we think programmatic skill development: I want more media relations skills or I need help with Internet grassroots mobilization strategies. Research shows, however, that we need to pay attention to organizational structure and systems and the ways in which they support successful advocacy, or don’t.

    And I know from my own practice that this is key.

    I found myself nodding at the computer screen several times while reading this. The authors focus a lot on leadership (one of four areas of organizational capacity they identify: leadership, adaptive, management, and technical), and it’s undeniably the most critical link. Working for years in an organization with a primary emphasis on services and this tangential, although mostly sincere, interest in advocacy, I can testify to the importance of committed Board leadership, unity among stakeholders, organizational culture of advocacy, long-term goal orientation, and stakeholder engagement.

    These are the pieces that, I think, most trip up our nonprofit social services as they seek to integrate advocacy. When the leadership hasn’t really bought in, when they expect social change on a grant timeline, when they’re hiding the potential for controversy from Board members, when the organizational culture discourages risk, when there’s great distance between leadership and constituents…then advocacy will be avoided or, if pursued, siloed and marginalized and sporadically forgotten.

    This suggests, and my own practice wisdom confirms, the importance of really assessing our organization’s leadership and its “appetite” for advocacy before embarking on a social change agenda within our nonprofit organizations. And that’s the most valuable piece of this document for me. It has a “readiness to implement advocacy” checklist that I think all nonprofit organizations should look at, and it asks really important questions (as yet unanswered) about what “high-performing” advocacy organizations look like.

    The final two pieces that prompted a lot of thinking from me about my advocacy journeys within nonprofit organizations were the importance of building time for reflections and the primacy of human resource concerns. As far as the former, I always built reflection time with my grassroots leaders into our work, but I rarely had a chance to reflect with the leaders of our organization about where our advocacy was going and how we should adjust. And I think El Centro, Inc. really did pretty well in terms of organizational capacity for advocacy–it had a diversified resource base, donor flexibility, a strategic approach to alliances, many of the elements emphasized here–which makes this almost total lack of shared reflection notable.

    And for the latter, well, the authors make the point that, because advocacy depends on relationships, keeping the people who have these relationships on board and committed to the advocacy work is key to long-term success. Which is, I know, where I come in. I’m not honestly sure that there was anything that El Centro, Inc. could have done to keep me there, in full-time advocacy. The pull of my kids is very, very strong. But I know that if the organization had implemented some of the redundancies advocated here, in order to shift human resources to meet campaign AND personal needs–if, for example, I had had a real maternity leave, or been able to reduce my hours, or had more staff support–then I might have found a way to make it work. I’d be lying if I said that I don’t still think, just about every day, of what I could (should?) be doing in the movement, and reading like this makes me ponder those questions more profoundly.

    I believe completely that our nonprofit social service organizations hold much of the future of progressive advocacy. You know that by now, right? But this report helped me to see the importance of not putting that proverbial cart before the horse and of, instead, making sure that we have the pieces in place, that our organizations are ramped up and ready for advocacy success, and that we’re building the kind of capacity that yields a real force for change.

    → Leave a comment

    Posted in Background Reading

    Tagged , ,

    But HOW do we measure advocacy success?

    Posted on March 24, 2010 by | 2 Comments

    photo credit: Plastic Girl, via Flickr

    As part of my continuing fascination with this question of how to evaluate ‘success’ in advocacy practice, I’ve been reviewing pretty much any documents I can find that purport to provide help for advocates and donors seeking to answer it.

    The last one I read is this Annie E. Casey Foundation-funded guide to measuring advocacy and policy efforts. The guide has an appendix that includes examples for different evaluation protocols: measuring core outcome areas, evaluating strategic progress, assessing short-term incremental objectives, gauging organizational capacity, and documenting case studies.

    The report makes some good points that must be addressed to advance this field:

  • Some ‘progress’ means holding the line against regressive proposals: how do we measure maintaining the status quo?
  • “Long-term” has different meanings for different stakeholders, so we need a common language of timeline expectations.
  • We shouldn’t oversell all advocacy as long-term societal change, when some efforts are really smaller-scale and more programmatic than revolutionary.
  • If we’re going to really change society, we need to frame all of our advocacy or policy change efforts as steps towards broader social change, but advocacy grantmaking is seldom seen as an investment in social transformation (so, then, it’s not surprising that it rarely yields that).

    But all of this discussion really got me thinking about the fundamental premise of this exercise in evaluating advocacy: that we need to determine what are the steps along the theory of change and measure how an effort is advancing along them. In this particular report, those indicators are defined as shifts in social norms, increases in organizational capacity, increases in alliances, strengthened base of support, improved policies, and, finally, changes in impact. And that’s when I started thinking: really, as an advocate for social justice (or, pretending for a moment, as a donor for the same), do I care if a particular group has improved capacity or more friends or stronger constituents or even a more sympathetic environment in which to advocate? I mean, I do, I guess, but only to the extent to which it yields what I really care about–the social change. If it doesn’t, for whatever reason, I’m going to be hard pressed to say that I think it was a real success.

    So, then, in pursuit of some way to legitimize advocacy and policy work, to bring it more in line with social service programming and its allegiance (rather belatedly) to evidence-based funding, are we reducing social change work to a caricature of its ideal self? Are we pretending that we know the formula for what makes a social movement happen and giving groups/organizations too much credit for going through those paces, regardless of whether, at the end of the day, they deliver the win? In so doing, are we emphasizing things that can be quite fleeting (a bigger coalition or better poll numbers, for example), or tangential (more staff or funding for the organization) and are unlikely to radically transform the lives of those impacted by the social problem in question? Should the only ‘interim’ step that counts be transformation in the lives of those affected, so that we can at least claim that as a victory if we fall short of the final goal?

    And if we abandon this quest for evaluation because we conclude that it doesn’t capture the radical nature of the work we want to support, then where are we left in terms of informing our strategies, convincing our donors, and elevating those with the greatest chance of success? Is the answer to do the same kind of evaluation but with a different articulated purpose? Or do we need entirely different kinds of evaluation, more retrospective in nature, that can then try to inform future efforts? And, particularly vexing, as we transform our work to make it more, well, transformational, by relinquishing the appearance of control and giving more power to the crowd, we make it that much harder to track our impact–how can we know exactly what difference we made, when we didn’t control (or sometimes even know about) everything that “we” did?

    As usual, I’m left with more questions than conclusions. What do you think? Have you evaluated your advocacy efforts? If so, what was worthwhile and what was frustrating about the experience? If not, what barriers kept you from that work? What questions do you still have, and what do you want to see explored as next steps in this exploration of advocacy evaluation?

    • → 2 Comments

    Posted in Analysis and Commentary

    Tagged ,

    Advocates speak out on advocacy evaluation

    Posted on March 11, 2010 by | Leave a comment

    photo credit, Michael Lokner, via Flickr

    A missing piece in the discussion of advocacy evaluation has been the voices of advocates themselves. Too busy changing the world to be included in the discussion about how we measure those change efforts, the conversation has been happening almost behind their/our backs, and I was really glad to see this report, spearheaded by the Atlantic Philanthropies and Annie E. Casey Foundation, two of the leading philanthropic voices on social change, and, it turns out, evaluation of the same.

    The purpose of this report is to provide nonprofit advocates with a platform to discuss their experiences with advocacy evaluation and to open communication with evaluators and donors about how to improve the enterprise. It opens, though, with the results of a survey of more than 200 advocacy grantees of some of the leading foundations in advocacy, and those results are themselves instructive for forming a portrait of the status of nonprofit advocacy.

    Not surprisingly, only 25% of respondents have ever evaluated their advocacy. Even fewer of those have had the assistance of an external evaluator (which is significant given the limited experience of many nonprofit types in doing systematic evaluation of any kind)–only 17% of the total sample. Of course, I also question how useful the exchange with the external evaluators has been for advocates; anyone who has participated in an independent evaluation knows that evaluators vary in their willingness to actively engage program leaders in the process and shape a product that will meet the agency’s needs.

    Sixty percent of nonprofit advocates are working within organizations with budgets less than $1 million annually; fully half have budgets less than $500,000/year. More than half of respondents, furthermore, dedicate fewer than half of their resources to advocacy, with smaller organizations more likely to be ‘purely’ advocacy. Human services are the most common advocacy priority of the respondents, at 40%. Advocates are mostly engaged in state, local, and regional work; only 21% are substantially working on national advocacy. That’s interesting, I think, not surprising, given the logistical and political challenges of impacting Congress, but rather discouraging given the rich possibilities of effective congressional advocacy.

    Advocates are overwhelmingly focused on legislative advocacy (56%). This appears to include a strong grassroots lobbying component, though, with 47% citing participation in community organizing also. Only 12% are working on judicial strategies and only 5% on administrative/regulatory advocacy. That echoes what I often hear from nonprofit leaders when we talk about advocacy; they tend to think legislative work first and foremost and are often surprised and even confused when I talk about other types of engagement as ‘advocacy’. One of the findings that most resonated with me was that, despite the preference for legislative advocacy, only 22% of advocates judged legislative work as the most effective strategy!

    Important for me as I continue exploring my consulting work with nonprofit organizations was the statement that research and communications assistance are the capacities that advocates view as most lacking. That surprised me, because I would think that those tools would be easiest to find from other sources, and it has caused me to rethink somewhat what I need to be discussing with nonprofit leaders.

    As far as actual advocacy evaluation, those advocates that have done it note that it has helped them to refine their strategies, make the case for more funding, and pursue staffing changes. They point to lack of resources for evaluation, obviously, as a barrier, but also the need for better interim goals and an attitude that sees evaluation as a capacity-building tool rather than a punitive audit.

    As the report states, the field of advocacy evaluation was virtually nonexistent not even 10 years ago and is now developing rather dramatically. The authors conclude by calling for advocacy evaluation to help advocates better change the world. In the race towards justice, they say, we need to know when to sprint and when to save our strength, and good advocacy evaluation can help us reach the finish line.

    → Leave a comment

    Posted in Analysis and Commentary

    Tagged , , ,