Projects, People and Processes

By Holden Karnofsky @ 2017-06-26T13:27 (+3)

This is a linkpost to https://www.openphilanthropy.org/blog/projects-people-and-processes

One of the challenges of large-scale philanthropy is: how can a small number of decision-maker(s) (e.g., donors) find a large number of giving opportunities that they understand well enough to feel good about funding?

Most of the organizations I’ve seen seem to use some combination of project-based, people-based, and process-based approaches to delegation. To illustrate these, I’ll use the hypothetical example of a grant to fund research into new malaria treatments. I use the term “Program Officers” to refer to the staff primarily responsible for making recommendations to decision-makers.

• Project-based approaches: the decision-makers hire Program Officers to look for projects; decision-makers ultimately evaluate the projects themselves. Thus, decision-makers delegate the process of searching for potential grants, but don’t delegate judgment and decision-making. For example, a Program Officer might learn about proposed research on new malaria treatments and then make a presentation to a donor or foundation Board, explaining how the project will work, and trying to convince the donor or Board that it is likely to succeed.
• People-based approaches: decision-makers delegate essentially everything to trusted individuals. They look for staff they trust, and then defer heavily to them. For example, a Program Officer might become convinced of the merits of research on new malaria treatments and propose a grant, with the funder deferring to their judgment despite not knowing the details of the proposed research.
• Process-based approaches: the decision-makers establish consistent, systematic criteria for grants, and processes that aim to meet these criteria. Decisions are often made by aggregating opinions from multiple grant reviewers. For example, a donor might solicit proposals for research on new malaria treatments, assemble a technical review board, ask each reviewer to rate each proposal on several criteria, and use a pre-determined aggregation system to make the final decisions about which grants are funded. Government funders such as the National Institutes of Health often use such approaches. These approaches often seek to minimize the need for individual judgment, effectively delegating to a process.

These different classifications can also be useful in thinking about how Program Officers relate to grantees. Program Officers can recommend grants based on being personally convinced of a particular project; recommend grants based primarily on the people involved, deferring heavily on the details of those people’s plans; or recommend grants based on processes that they set up to capture certain criteria.

This post discusses how I currently see the pros and cons of each, and what our current approach is. In large part, we find the people-based approach ideal for the kind of hits-based giving we’re focused on. But we use elements of project-based evaluation (and to a much lesser degree, process-based evaluation) as well - largely in order to help us better evaluate people over time.

Projects, processes, and people

In some ways, project-based approaches are the most intuitive, and probably the easiest method for a funder to feel confident in by default. Organizations propose specific activities to Program Officers, who try to identify the ones that will appeal to decision-makers and make the case for them. Thus, decision-makers delegate the process of searching for potential grants, but don’t delegate judgment and decision-making.

I think the fundamental problem with project-based approaches is that the decision-maker generally has much less knowledge and context than the Program Officer (who in turn has much less than the organization they’re evaluating). This is a general problem with any kind of “top-down” decision-making process, but I think it is a particularly severe problem for philanthropy, because:

• There’s often a very small number of people (a wealthy individual or family) who are trying to give away a large amount of money.
• They often do not personally have extensive background knowledge for the causes they’re working in, and do not work full-time on any particular cause (and in many cases do not work full-time on philanthropy generally).
• Decisions usually can’t be subject to any straightforward or quick performance measurement. It usually takes a good deal of subjective judgment to decide whether a grant is going well.

From what I’ve seen, the result can often be that Program Officers recommend the giving opportunities they think they can easily justify, rather than the giving opportunities they personally think are best. This risks wasting much of the expertise and deep context Program Officers bring to the job. I think this is a major problem when trying to do hits-based giving, for reasons outlined previously.

People-based approaches are the opposite in some sense. Grants are made based on trust in the people recommending them, rather than based on agreement with the specific activities proposed. This is “bottom-up” where project-based giving is “top-down”; it delegates judgments to individuals, where project-based giving doesn’t delegate judgment at all. I think the advantages here are fairly clear: the people with the most expertise and context are the ones who lead the decision-making. People-based approaches seem likely, to me, to achieve the best results when done well.

However, I also see major challenges to people-based approaches:

• Everything comes down to picking the right people. I generally consider it very hard to evaluate people, and don’t know of any reliable and reasonably quick way to do so. Our experiences recruiting have generally left us feeling that the only good way to evaluate someone is to work with them for an extended period of time, and we’ve heard similar sentiments when seeking advice from other organizations.
• The more one defers to people, the more one is leaving the possibility open that they might make decisions arbitrarily, based on e.g. relationships rather than merit.
• I also think there is at least some tension between taking people-based approaches - at least pure people-based approaches that focus on “general impressiveness” of people - and pursuing neglected causes. If one simply finds the most impressive people (in a general sense) and defers to them, one is likely to end up supporting people who have already had success and achieved influence, and one is likely to end up supporting fields that are already popular. I believe that we are bringing a fairly novel angle to philanthropy - focusing on important, neglected, tractable causes. This angle is novel enough that we don’t think we should be restricting ourselves to working with people who fully share it (this would greatly reduce the pool of people to choose from). But if we instead evaluate people only for general impressiveness, we risk losing much of this angle, and risk doing status-quo-biased giving.

Process-based approaches are another approach to scaling understanding. Rather than try to evaluate each project, or defer to individuals’ judgment, the funder sets up processes that try to capture high-level criteria. For example, many National Institutes of Health (NIH) grants seek to optimize on criteria such as the significance of the work, the experience/training/track records of the investigator(s), the degree to which a project is innovative, etc. Process-based approaches often (as with the NIH) involve systematic aggregation of a large number of individual judgments, reducing reliance on any one individual’s judgment.

I think the appeal of process-based approaches is that they can integrate expertise and deep context into decisions more reliably than project-based approaches (which rely on the judgment of the decision-maker(s)), while also avoiding the disadvantages listed for people-based approaches: difficulty of choosing people, risks of arbitrariness and conflicts of interest, difficulty maintaining fidelity to unusual angles on giving that the decision-maker(s) might have. However, I believe that process-based approaches bring their own problems:

• Well-defined criteria and processes make it possible for potential grantees to “game the system” - coming up with grant proposals that are designed to get through the process rather than to propose and communicate the best work possible. And once gaming the system becomes possible, it may quickly become necessary in a competitive environment.
• Process-based approaches tend (I believe) to be fairly slow, unpredictable and inconvenient from a grantee’s perspective. They make it hard to have honest, informative conversations with potential grantees about their likelihood of getting funded. They also tend to be rigid: unable to support work that’s very different from (but perhaps better than) what the process designers anticipated.
• Process-based approaches tend to minimize the role of individual judgment, often by aggregating many judgments. But the more they do this, the less room they leave for the kind of high-risk, high-reward, creative, unusual decisionmaking we associate with hits-based giving.

My current view is that process-based approaches can be excellent for funders seeking to minimize risk (of being perceived as unfair, of supporting low-quality work, of supporting work for the wrong reasons). Government funders often fit this description. But process-based approaches seem much less appealing for hits-based giving, unless they are very carefully designed by people with strong expertise and context with the specific goal of pursuing “hits.”

Note that the above classifications are fairly simplified, and many funders’ decision-making processes have elements of more than one.

Our current approach

Our current approach is based on the idea that people-based giving is ideal for the kind of hits-based work we’re trying to do. Specifically, our ideal is to find people who make decisions as we would, if we had more expertise, context and time for decision-making. (Here “we” refers to myself and Cari Tuna, currently the people who sign off on Open Philanthropy Project grants.) We also encourage our Program Officers to take this attitude when seeking potential grantees, though they can ultimately choose whatever mix they want of project-, people- and process-based approaches for making recommendations to us.

Our ideal can be pursued at different levels of breadth. We can set particular focus areas, then try to find Program Officers who make decisions as we would for each focus area. Program Officers can then try to find people who make decisions as they would for particular sub-areas of the focus area they work in; for example, after identifying corporate cage-free reforms as a promising sub-area of farm animal welfare, we sought to support a set of people working on cage-free reforms.

The major challenge of this approach is determining which people we want to trust, and in what domains. We might find a particular person to be a great representative of our values in one area, but a poor representative in another. This is where a more project-based mentality comes in.

• We try to understand a focus area well enough to have fairly detailed discussions of strategy during our hiring process (for more on what we look for, see this post).
• Our grantmaking process involves Program Officers’ writing up the case for each grant and answering a number of key questions. We try to have enough general knowledge about the area they’re working in to evaluate the case they’re making at a high level. We ask critical questions and try to find ways in which we can learn from each other.

A fairly common dynamic with new Program Officers has been that we know far less about their field than they do, and we often learn about the field when we question the parts of a grant writeup we find counterintuitive; at the same time, we often (at first) have better-developed views on philanthropy-specific topics, such as assessing room for more funding. Our goal over time is to reach increasing common understanding with Program Officers and increasingly defer to their judgment.

One principle we’ve been experimenting with is a “50/40/10” rule:

• We want to have fairly good understanding of, and high excitement about, at least 50% of a Program Officer’s portfolio.
• We want another 40% of the portfolio to fit the description, “We can see how this might appear very exciting if we had more context, though we don’t feel personally convinced.”
• We’re willing to defer entirely to the Program Officer on the remaining 10% of the portfolio, even if we can’t see the case for it at all (as long as the downside risks are manageable).

The idea here is to try to stay synced up with Program Officers on enough of their work - using a “project-based” approach - to continually justify our confidence in them as decision-makers, while allowing a lot of leeway for them to use their own judgment and recommend grants that require deep expertise and context to appreciate.

Similar principles apply to how we support and evaluate grantees. Even when the main reason we’re supporting an organization is as a bet on the people involved, we still find it helpful to have an outline of the projects they plan on. This helps us evaluate whether these people are aligned with our goals and whether our funds will help us do much more than they could have otherwise. But once we’ve determined that the proposed activities seem promising and sensible, we tend to provide support with no strings attached, in case plans change.

Ultimately, I think that our work tends to look very “project-based” in a sense: we put a lot of effort into learning about our focus areas and we ask Program Officers a lot of questions about their recommended grants, and Program Officers in turn tend to ask potential grantees a lot of questions about the specifics of their plans. But the intent of this is more to spot-check our alignment and understanding than to comprehensively understand the grants. When a particular question is hard to resolve, we tend to defer to the people with the most expertise and context. We know we’ll never have the whole picture, and our goal is to understand enough of it to extrapolate the rest - by trusting the right people for the right purposes.