Open Philanthropy: Questions We Ask Ourselves Before Making a Grant

By Aaron Gertler 🔸 @ 2020-05-04T04:52 (+20)

This is a linkpost to https://www.openphilanthropy.org/blog/questions-we-ask-ourselves-making-grant

This article isn't especially new, but I'm cross-posting it because it's the clearest guide I've found to the processes of the largest grantmaker in EA.

One question that stood out to me when reading: "Who else could fund [this project], and why aren't they?" It seems really useful for grantmakers (especially those much smaller than Open Phil) to examine and learn from the decisions of other grantmakers in their areas.

Also notable:

Our best model is that many grantees are constantly trying to guess what they can get funded, won’t ask for as much money as they should ask for, and, in some cases, will not even consider what they would do with some large amount because they haven’t seriously considered the possibility that they might be able to raise it. We’ve had multiple experiences where a grantee asks for X, and we say “What would you do with 2X?” They usually say, “Never thought about it — let me get back to you,” and we often end up with a much better grant in the end.

 

The article

Although we have typically emphasized the importance for effective philanthropy of long-term commitment to causes and getting the right people in place, the most obvious day-to-day decision funders face is whether to support specific potential giving opportunities. As part of our internal guidance for program officers, we’ve collected a series of questions that we like to ask ourselves about potential funding opportunities, including:

This post, which I adapted from the internal guidance for program officers, reviews the value we get from these questions and some of our approaches to answering them.

This is a list of questions we’ve provided to staff to help them think about how to structure their internal grant writeups, and that grant reviewers tend to ask ourselves as we review grants. We don’t have these questions in our standard template or ask that they be itemized for each grant (there are many individual cases where particular questions are inapplicable or unimportant).

Evaluating a grant’s place in the ecosystem

As we try to determine whether a grant is likely to have a positive impact, we ask ourselves key questions including: What does this grant do for the overall ecosystem of organizations working on the cause in question? What key need is it addressing and how? What other ways might there be to fill/maintain/expand this need, and does this grant seem like the best way? If not, is it compatible with pursuing other approaches to the same need simultaneously?

Examples of ecosystem “needs” might be: intellectually solid analysis of what policies should be (What should sentencing laws be? How can the government improve existing programs to strengthen pandemic preparedness?); “insider” advocacy building support for the right policies (e.g., making the case to legislators, businesses, and other influential leaders); “grassroots” advocacy to build support for the right policies, or building the power of aligned constituencies.

That said, we recognize there are stark limits to our own knowledge of “what the field needs,” especially when we’re supporting organizations that primarily serve other organizations in the field (e.g. by providing training or support). In these cases, we try to think about how to create dynamics where a service-providing grantee will be accountable to the organizations that consume its services (e.g. the people who attend the trainings). One specific way we do this is by asking people in our fields whether they’d rather have grants for their own organizations or for the “public good” service provider we’re considering providing. We often ask, “who is the right conceptual ‘buyer’ for these services and do they actually think it would be worth paying this much for?”

One thing we generally try to avoid in assessing an ecosystem is creating an elaborate “theory of change” that requires our grantees to work together in highly specific ways as more than the sum of their parts. We tend to think that unless we’re effectively the only funder in a field, our grants are each relatively marginal and can be fairly accurately assessed on their own terms. Sometimes we see opportunities to help grantees coordinate around a particular strategy, but that is not our default approach. And, like other funders, we’re open to convening grantees (and non-grantees) to try to help develop shared strategies or goals; we’re just typically more skeptical of the idea that we need to fund all parts of a strategy or ecosystem for the whole to be effective.

Evaluating a grantee’s leadership

When deciding whether to make a grant and how to structure it, we ask ourselves: Who is being empowered by this grant? Are they an effective leader who we’re happy to bet on? Are they underfunded? What are their strengths and weaknesses? Are we empowering them to do what they already want to do, or are we asking them to do something different than they might have chosen on their own? If the latter, why is that a good idea, and have we considered deferring to their judgment instead?

Since our grantees tend to know their work much better than we do, we believe we’re usually better off finding people who share our key goals and letting them work out the details. Many of our favorite grants are in the model of “Find a person who is fantastic at doing something the field needs, and give them no-strings-attached support to help them do more, faster.” Some of the funders we think are most impressive focus much more on supporting the right people than on the details of those people’s plans. (For example, see our post on the Sandler Foundation.)

Because we are often trying to support the work of particular people housed within larger organizations, we might place restrictions on funding to make it conditional to those people’s involvement in, and control of, the work. However, once the right people are empowered, we try to be skeptical of our own impulses to narrowly direct how the funding is used. If we find ourselves asking a person to spend more time and energy on a specific project we prefer, we ask ourselves, “Are we sure we’re right, or is it possible that they know what the field needs better than we do, and we should be funding them to do their first-choice work?”

We are sometimes wary of potential grants where a well-funded organization offers to do a project that’s particularly appealing to us. In many cases, we believe they’re proposing a project they would do with or without our support, and our funding is effectively paying for other things they want to do. We often refer to this concept as “fungibility.” We try to avoid these cases by asking ourselves, “Why can’t this person do the project they’re pitching with money they already have?” If the answer is that they don’t have much money, it’s possible we should be giving unrestricted support. If the answer is that they don’t actually value the project as much as other things they could do, we ask whether there’s a good reason they value it less, and whether we can expect them to do good work on the project if we fund it.

Evaluating comparative advantage

As we assess whether a specific grantee is the right partner, we believe it’s helpful to ask ourselves whether it is the best organization to do this work, and if its work is the most valuable use of this organization’s additional resources. When we’re considering funding a research organization that wants to do public advocacy around its research, we consider looking for an advocacy organization that could potentially do the same work more effectively. When we’re considering funding an organization that will provide services to other organizations, we tend to ask if those “client” organizations are adequately funded, since they are often better positioned than we are to decide whether new services are worth paying for.

The question of comparative advantage applies to us as well. We often ask ourselves: Are we the best-placed organization to evaluate and fund this? Who else could fund it, and why aren’t they?

If we find ourselves considering a lot of similar grants, especially small ones, we tend to look for opportunities make a bigger grant and have someone else take the time to strategically regrant. For instance, rather than five small grants to five individuals doing similar activities in five different states, we may be better off finding (or creating) an organization that is well-placed to evaluate those activities, and giving them one big grant. This thinking informed our decision to support The Humane League’s Open Wing Alliance, a new coalition of promising farm animal welfare groups that are trained in corporate campaigning so they can achieve cage-free wins in new countries.

For us, determining whether other funders are better positioned to support a project isn’t just about saving money. It’s helpful for checking the basic case for the grant, identifying considerations we might have missed, and continuing to refine our theory of our own comparative advantage as a funder.

If we can think of someone who seems like they logically ought to be willing to fund a potential grant, we try to ask them what they think. Sometimes it will turn out that they know more about this type of grant than we do, and they will raise new considerations. Sometimes it will turn out that the grant is not actually a fit for what they do, and we’ll learn more about them and improve our model of them as a funder. Accurate models of other funders help us assess our own comparative advantage, and potentially help the other funders by allowing us to refer potential grantees who seem like a good fit for their goals.

Evaluating a grant’s size and duration

When we review grantees’ budgets and determine how much funding to provide, we ask ourselves: What are we paying for, above and beyond what’s already funded by others? Could we get the same work done with a lot less money? Are we sure this grant shouldn’t be bigger? What is our “willingness to pay” for success here?

While sometimes people will ask for much more funding than we think appropriate, we are also often concerned about the times when people ask for less than they should. After conversations with many funders and many nonprofits, some of whom are our grantees and some of whom are not, our best model is that many grantees are constantly trying to guess what they can get funded, won’t ask for as much money as they should ask for, and, in some cases, will not even consider what they would do with some large amount because they haven’t seriously considered the possibility that they might be able to raise it. We’ve had multiple experiences where a grantee asks for X, and we say “What would you do with 2X?” They usually say, “Never thought about it — let me get back to you,” and we often end up with a much better grant in the end. This was the case in our grant to support research at the University of Washington. Through ongoing conversations, the original grant proposal focusing on the development of a universal flu vaccine evolved into an expanded grant incorporating work on a computational protein design system that we believe could have much broader utility if it makes it possible to rapidly design new vaccines or antiviral drugs. We sometimes ask grantees what activities would occur at several different funding levels and consider these scenarios and tradeoffs as part of the decision-making process.

When we are considering higher salaries or more ambitious proposals, we need to ask ourselves (and our grantees) how many years we should commit to. Long-term commitments often help grantees plan, so if it seems like we’re funding something that’s going to take several years to play out, we generally aim to commit to more than one year up front. In the case of the new Center for Security and Emerging Technology, we think it will take some time to develop expertise on key questions relevant to policymakers and want to give CSET the commitment necessary to recruit key people, so we provided a five-year grant. That said, for small organizations, a few years can be a very long time in terms of learning new things, changing organizational direction and finding new funders, so a two- to three-year commitment will often (in our view) be sufficient to achieve the goal of helping with planning.

Evaluating a grant’s cost-effectiveness

When deciding whether to make a grant or hold the money in reserve to distribute elsewhere, we think about cost-effectiveness. Do we get good bang-for-the-buck?

Grant investigators sometimes include a “back-of-the-envelope-calculation” (“BOTEC”) to roughly estimate the expected cost-effectiveness of the potential grant in their writeup.

We recently shared an update, including some example BOTECs, on our thinking about how to evaluate giving aimed at helping people alive today, including a mix of direct aid, policy work, and scientific research. While we previously used unconditional cash transfers to people living in extreme poverty as “the bar” for being willing to make a grant, GiveWell has continued to find more and larger opportunities over time, which has the implication that we may raise “the bar” to something closer to the current estimated cost-effectiveness of GiveWell’s unfunded top charities.

“Longtermist” BOTECs focus more on how much a given grant might reduce the chances of a global catastrophic risk.

Noting and considering reservations

We generally try to ask (and our investigation encourages asking): If we had a smart, thoughtful friend who thought this grant was going to end up having no impact, what would they tell us and how would we respond?

Grant investigators include for decision-makers’ consideration the devil’s advocate argument against a grant’s potential impact, and explain why they don’t believe that argument to be decisive. We hope that grant investigators explicitly considering and vocalizing these objections helps reduce the chances that they follow a faulty line of reasoning too far, or that a crucial objection to a grant is never brought up.

Evaluating predictions about how a grant will play out

We hope to learn from our past grantmaking. Looking back on a grant, how might we determine whether it has gone well or badly?

By asking grant investigators to consider this question before recommending a grant, and by encouraging them to make quantified and objectively evaluable predictions about how a grant will play out over time, we try to make our future evaluation of a grant’s performance easier.

While we no longer publish these predictions in our public grant writeups, we continue to track them internally and use them in renewal decisions to update our expectations in light of facts on the ground.

(More on this practice here.)