How Not to Evaluate a Grant

Someone sends an e-mail message.  They received a grant that needs an evaluator.  They wrote an evaluation plan, and the plan requires an annual report.  They need someone to write the report.  Sometimes, they’ll call just as the funded project is underway; other times, they call when the report is due in a few months.  Sometimes, it’s a private, foundation grant; however, more often, it’s a federal grant.  Regardless of the source and the timing, they’re calling me far too late in the process.  I explain as professionally as I can that what they’re most often seeking is a compliance report rather than evaluation.  If they’re willing, I’ll help them do more than write a compliance report.  If they’re not, I wish them well. 

 

Any funded organization should show compliance to the funder’s requirements and progress in meeting the outcomes they established in their funding request.  However, the summative analysis within an annual report is a small part of what an evaluation could and should provide to any project.  Because I, and every other person who provides external evaluations, have these discussions often, I decided to write some recommendations for organizations that are writing funding requests so that they can understand what they miss by having a cursory, summary report of their project as their only attempt at evaluation. 

 

Recommendation 1:  Bring in an evaluator as you are designing your project – and it usually should not be the grant writer, especially if you’re using an external grant writer.  There’s an industry of grant writers who develop ancillary businesses by writing themselves into a grant as an external evaluator once the project is funded.  Typically, grant writers understand how to write a successful proposal.  That typically makes them qualified to write evaluations that ensure compliance to the outcomes written into the grant.  In contrast, an evaluator needs to be someone with a strong set of analytical tools, who can write objectives that are clear, yet provide for ongoing analyses of a project as it unfolds; who understands both qualitative and quantitative measures and when to apply them; who understands organizational development and systems; and who can recommend methods and processes that will assist the project to understand a project’s ongoing progress. 

 

An evaluator shouldn’t drive the aims of the grant, but that person should work alongside you to help you refine those aims to the betterment of your project.  That’s much more involved than writing a compliance report of simplistic objectives that you’ll meet each year or at the end of funding.  Grant writers usually don’t have the skills required to do all that.  If you find one who does have the skills of an evaluator, it might make sense to include that person as the external evaluator.  However, in my experience, folks who are busy writing grants sometimes find that they have to back out of the evaluation work as it competes with their more lucrative grant writing (that’s sometimes the reason that I get that e-mail request so late in the process).  

 

Recommendation 2:  As you’re writing your grant proposal, work with your evaluator to write outcomes and performance measures that will have meaning to you, not just to the funder.  Unlike compliance outcomes, the ones that a trained evaluator will help you generate will be more consequential for you as an organization.  Those will help you develop a plan that measures how and why the project is succeeding or not succeeding while it’s underway.  A skilled evaluator will help you design outcomes that are measurable, but, more importantly, outcomes that are useful to your organization as measures of progress – and measures of causes of that progress.  When done well, those measures will evaluate both the project and the organization in which the project exists.  Knowing that you met or didn’t meet a goal at the end of a funding year, or the end of the project, is not helpful if you don’t understand how or why that happened – or what those outcomes mean to you organizationally.  An evaluator will ask key questions that will help you to measure formatively so that you’re not waiting to the end to see what impacts your project is having.  That typically won’t happen if the evaluator isn’t helping to design the evaluation. 

 

Recommendation 3:  Make certain that your evaluation includes those formative analyses to determine what’s happening in the project as the project is underway.  The grants I’ve seen without this have outcomes that are simple numeric indicators that are easy to report.  To restate and emphasize this point, that happens because grant writers generally don’t know more about assessment and evaluation beyond identifying summative measures that will fulfill the funder’s needs.  But that misses a critical opportunity to learn and evolve from the grant.  More than one project I evaluated over the years began with a premise that a specific intervention was the solution that would help meet a challenge.  For these projects, as my team and I conducted the initial, probing analyses that an experienced evaluator will build into all projects, we sometimes discovered that the funded intervention is a small component of the possible solution.  With all of those projects, the organizations adapted their project to think more holistically and, as a result, had a more significant impact than the project originally intended.  That would not have happened if projects had not built sophisticated, formative analyses into their evaluation plans. 

 

What organizations often don’t consider when they’re writing a funding proposal is this opportunity to learn and to grow from the project.  Yes, of course, you want to change something about your work.  That’s what grants help you to do.  But you miss a critical opportunity to help your organization learn from the experience if you don’t build a formative learning model into your project.  All the organizations I work with don’t have the time or resources to do the kind of exploration that I’m suggesting in their normal operations.  Having an evaluator look deeply at your project can also be a way to leverage the work on the grant into your larger organizational needs.  The interviews, observations, surveys, analyses of documents, etc. that an evaluator will use to analyze your project formatively will give you a picture of how the project operates within your larger systems.  During one project that I was evaluating, my team’s analysis discovered that the organization’s IT structure and systems were hindering the implementation of the project.  We conducted a separate analysis of the IT department and how people perceived and interacted with it.  We were able to provide an additional report of our findings about the IT department, and that report had implications well beyond the project.  If we hadn’t done that work, the project might have looked elsewhere for the answers to questions about the project’s efficacy. 

 

Recommendation 4:  See an evaluation as an integral component of the grant’s work.  You’re seeking funding that will allow you to do work that’s beyond your organization’s base funding.  Take advantage of that and allow the evaluation to inform your organization’s work generally.  Use the evaluator’s expertise to help you operate.  In grant-funded projects that I evaluate, I typically will either design or implement measures that allow us to review documents, interview, survey, or observe activities to examine what is happening during the project.  That provides information about the project; however, it also provides a picture of the overall organization.  A grant-funded evaluator cannot help but to give that larger picture, if that work is seen as more than compliance checking.  If you’re an executive who has a grant-funded project and you’re not taking advantage of this important byproduct of every externally funded project, you’re shortchanging your organization.

 

Recommendation 5:  View the evaluator as a critical member of the project management.  All the above recommendations are of no use if the evaluator doesn’t participate in the design and implementation of the model.   In my consulting practice, I learned this the hard way and stopped serving as an external evaluator unless I am considered part of the project management team.  I don’t see my role as driving the direction of a project; however, as decisions are made by the executive team, I want to be there to offer advice on how to measure the impacts of decisions.  I also want to be there as program decisions are made so that I understand what it means for the work I need to provide the client.  At times, I’ve even provided support with strategic planning so that the organization could refine its intent and where the grant should fit that intent. 

 

Early in my consulting career, I would find myself working on an evaluation measure and in casual discussions with the project manager, the person would offer something like, “But we’re not doing that part of the project anymore.  We’ve decided to try something else.”  I understand that projects evolve over time.  They should.  By being part of the project management team, I can be aware of those changes as they happen so I can adjust my work to fit what’s happening.  That becomes especially critical when an annual report is due.  I can reference changed plans to explain the rationale for those changes if I have my notes from a meeting where the changes occurred.  That’s even more critical if those changes involve an activity that’s tied to a key outcome for the project.

 

Closing Thoughts:

 

In almost 30 years of evaluation work that’s looked at everything from small, private foundation grants to federally funded projects that involve multiple organizations and locations, I’ve learned that organizations that take advantage of the opportunities that evaluation offers are often organizations that grow and thrive.  Rather than chasing funding, they tend to look at grants and projects strategically to help them in their evolution.  What I’m advocating for, here, is that organizations be more like that in the ways in which they select and work with a critical role that their funding provides for them. 

 

Does it cost more to do this?  Yes and no.  In the short-term, yes, and in the long-term, no.  You’ll have to budget more for an evaluator in a grant proposal than you would pay for an annual compliance review.  However, the institutional support you’ll get for this far outweighs what you’d spend to hire an internal employee to provide the kind or institutional analyses that an evaluator offers.  Besides, in my recent experiences, private funders especially are now looking for more thoughtful analyses than compliance reports. 

 

There are resources that can help you find the kind of person I describe above.  The American Evaluation Association maintains a site where you can find members to interview about your needs:  https://my.eval.org/find-an-evaluator.  Like contracting for any work, you should interview a few people and get references to make a selection.  There are local affiliates of AEA that can be useful in finding people, too.  Check out:  https://www.eval.org/Community/Local-Affiliates.  As you’re interviewing, give them this essay and ask how they’d provide the services that I describe.  But, as noted above, you should be seeking an evaluator as you’re writing your proposal.  Do yourself a favor and don’t do that after you’ve written the proposal.  Once you see what a trained evaluator can do for you in designing and implementing a project, you’ll never do anything else.

 

I’ve waited to write this until I am no longer accepting clients so that no one accuses me of making this statement to advocate for a model that benefits me.  Throughout my academic career, I accepted evaluation work to keep current in the field and to have ways to engage my graduate students.  These days, I have other priorities.  For the past five years, I’ve only accepted a few client projects; and, at this stage, I’m not taking any more.  So please see this as coming from a colleague who also managed funding from granting agencies – in addition to having been an external evaluator.  Please see this as an opportunity to ask yourself questions about the next grant, the next funding source, or the next project where you’ll seek to better your organization.

 

I wish you all the best.