Five Years of Rethink Priorities: What We've Learned

By Peter Wildeford, Marcus_A_Davis @ 2023-06-21T16:20 (+144)

The post contains a reflection on our journey as co-founders of Rethink Priorities. We are Peter Wildeford and Marcus A. Davis.
 

In 2017, we were at a crossroads.

We had been working on creating new global health and development interventions, co-founding an organization that used text message reminders to encourage new parents in India to get their children vaccinated.

However, we felt there was potentially more value in creating an organization that would help tackle important questions within cause and intervention prioritization. We were convinced that farmed and wild animal welfare were very important, but we didn’t know which approaches to helping those animals would be impactful. Hits-based giving seemed like an important idea, but we were unsure how to empirically compare that type of approach to the mostly higher-certainty outcomes available from funding GiveWell’s top charities.

So, we chose to create a research organization. Our aim was to take the large amount of evidence base and strong approaches used to understand global health interventions and apply them to other neglected cause areas, such as animal welfare and reducing risks posed by unprecedented new technologies like AI. We wanted to identify neglected interventions and do the research needed to make them happen.

~~~

Five years later, Rethink Priorities is now a research and implementation group that works with foundations and impact-focused non-profits to identify pressing opportunities to make the world better, figures out strategies for working on those problems, and does that work.

Reflecting on everything the organization has accomplished and everything we want to happen in the next five years, we’re proud of a lot of the work our team has done.

For example, we went from being unsure if invertebrates were capable of suffering to researching the issue and establishing invertebrate welfare as a proposition worth taking seriously. Following through, we helped create some of the first groups in the effective animal advocacy space working on interventions targeting invertebrates. Our team did the deep philosophical work and the practical research needed to establish specific interventions, and we incubated groups to implement them.

Building on this work, our ambitious Moral Weight Project improved our understanding of both capacity for welfare and intensity of valenced experiences across species, and the moral implications of those possible differences. By doing so, the Moral Weight Project laid the foundation for cross-animal species cost-effectiveness analyses that inform important decisions regarding how many resources grantmakers and organizations should tentatively allocate towards helping each of these species.

We have also produced dozens of in-depth research pieces. Our global health and development team, alone, has produced 23 reports commissioned by Open Philanthropy that increased the scope of impactful interventions considered in their global health and development portfolio. This work has influenced decisions directing millions of dollars towards the most effective interventions.

Our survey and data analysis team also worked closely with more than a dozen groups in EA including the Centre for Effective Altruism, Open Philanthropy, 80,000 Hours, and 1 Day Sooner to help them fine-tune their messaging, improve their advertising, and have better data analysis for their impact tracking.

RP has provided 23 fellowships to aspiring researchers, building a robust talent pipeline. Many talented people have remained in successful careers at Rethink Priorities. RP staff have gone on to work at Open Philanthropy, Founders Pledge, the Centre for Effective Altruism, 80,000 Hours, and even U.S. state-level elected government.

Via our Special Projects program, we also started fiscal sponsorship and operations consulting to help several new promising organizations—such as Epoch, Apollo Research, and Condor Camp—get off the ground much faster than they otherwise would.

And now we’re really getting started on, among other things, working to understand and tackle the big policy changes needed to address the unprecedented challenges and risks posed by increasingly capable artificial intelligence. While this is a relatively new area of research for us that started in early 2022, we've already built out a strong team and serious research agenda to try to tackle the complicated problem of ensuring future challenges with AI go well.
 

Changes, Challenges, and Learnings

RP started as a two-person team five years ago. But we have aspired to grow to meet the scale needed to address monumental challenges.

Now, we have 70 core staff (with five more joining soon), 30 contractors, and another 25 staff of fiscally sponsored projects that we support operationally. The expertise and efficiency of our expanded team have been key to our ability to produce the outcomes we have achieved.

SP = Special Projects, Exec = Executive Team, Ops = Operations, GHD = Global Health and Development, Longtermism: includes the AI Governance and Strategy Team, as well the Existential Security Team (formerly General Longtermism Team)

Did we do everything perfectly? Definitely not.

We learned a lot over the past five years. There are a number of things we would change if we could about how we approached some of our prior work.

Reflecting back on our initial principles

Five years ago, we wrote a post introducing Rethink Priorities to the world. A lot has changed since then, and we think it is worth revisiting that post to give an update to how we’re thinking about the principles for research we outlined in that announcement.
 

Here are the principles we outlined:

• Generating new insight and knowledge in addition to synthesizing existing research. We’re focused more on producing cutting-edge content rather than summarizing existing content.
• Publishing shorter and more digestible information more frequently, rather than publishing sprawling research less frequently. By taking the same amount of information and breaking it down into “minimal publishing units,” we make it easier for ourselves and others to understand and build upon, and get quicker feedback loops.
• Emphasizing quick feedback loops trying to go from initial research to an initial publication in about a month.
• Seeking tractability in research by looking for questions that we may be able to make meaningful progress on. We’ll hack away on the edges, so to speak.
• Keeping all research in touch with the bigger picture, building upon individual publications to try to answer key questions.
• Finding new opportunities over creating more ranked lists of existing opportunities. We predict improving on existing uncertain ranking methods is less effective than finding directions that have been previously neglected in EA. We’re unlikely to convince people of the number one cause, but we might find something others have not thought about.
• Tracking the impact of our research to help us make key decisions. We have plans to do follow-up reviews on all our research, including surveys to determine the influence of the research.
• Shutting the project down if it doesn’t generate impact to avoid wasting our time and effort. We are pre-committing to an initial six month test phase and pivoting as necessary. We will completely shut down if there’s no discernable impact within the first year. If our work isn’t helping others make better decisions, we should try something else.

— Rethink Priorities Launch Post

Principles for Speed

When we launched, we emphasized speed with principles like considering Publishing shorter and more digestible information more frequently and Emphasizing quick feedback loops. While we did follow this principle in 2018 and 2019, we’ve moved away from it. There are many reasons for this change.

One reason is that every research project we do has a tradeoff between time to complete the project versus investing more time in the certainty and accuracy of the piece. As people have begun taking our work more seriously, this tradeoff has become more important.

When we launched, other organizations didn’t have expectations of or weren’t as reliant on our work. It’s more costly to be wrong when millions of dollars of decisions depend on your answer. We’re pleased with the kinds of signposting you see in our recent global health and development work (as in our shallow review of weather forecasting and the WHO Essential Medicines List), which states how confident researchers are in their claims and how much time they spent investigating. However, that type of research isn’t viable or desirable for all topics and questions.
 

Principles for Project Type

To turn to the types of projects we do: Generating new insight and knowledge, and Seeking tractability in research are still critical principles we follow. But, our thinking on, Keeping all research in touch with the bigger picture, and Finding new opportunities over creating more ranked lists of existing opportunities has changed significantly.
 

Seeking tractability in research

We are still very much focused on trying to do applied work that gives practical advice to donors and foundations, which means keeping a close eye on the tractability of research.

Keeping Research in Touch with the Bigger Picture

The question of how things fit into the bigger picture is more complicated. In one sense, some of our work clearly builds on prior work (i.e. in many ways, our Moral Weight Project built on our invertebrate sentience research). In another sense, we aren’t necessarily committed to creating years-long sequences of work across domains. Ultimately, for us, the bigger picture we are hoping to keep our work in line with is “How can we do the most good?”. Keeping on track with that large goal means acknowledging there are research dead ends or when other projects simply become more important than completing huge projects. That said, we still think starting out by creating smaller projects that can be built upon makes sense when trialing whether you can produce meaningful work.

Finding New Opportunities

On the margin, we still think we are much more likely to convince someone that a cause area or intervention they hadn’t considered is worthwhile than convince them that something they’ve already considered and dismissed, or deprioritized, is worthwhile.

When we started, we had a strong inclination for prioritized projects where we thought “someone else really should have done this by now.”  This approach pushed us to spaces where others had not previously done projects, but where work seemed tractable. This idea was never a complete guide to determining what to do, but we still think this approach has a lot of merit. Obviously, figuring out where it is viable to push into these types of areas is the real trick.

But pushing at the margins isn't the only path to impact. There may be other interventions that have been studied before, but haven’t become popular for some reason other than impact. When we started, we had less expertise on staff so it made more sense to push into boundary spaces. As our staff has grown it’s become more viable to make meaningful contributions in spaces that are somewhat less neglected.
 

Principles for Accountability

Finally, let’s address the remaining principles, those related to holding ourselves accountable: Tracking the impact of our research and Shutting the project down if it doesn’t generate impact.
 

Impact Tracking and Transparency

While we do some public write-ups of our impact tracking, and we have done more internal research on this topic, generally, we haven’t done as much public tracking of our impact as we think is ideally desirable. This type of transparency takes lots of effort and time, and as a result has a significant opportunity cost.

We’ve often prioritized other things over making our actions and beliefs apparent to the outside world or even internally. Mostly, this is a tradeoff between (a) doing more work that we think will produce value and that funders are interested in us doing, and (b) explaining to other parties why we think those actions were right. Typically (but not always), the latter has more limited support among donors. Relatedly, public write-ups of our impact tracking often require heavy time input from the two of us (Peter and Marcus), making this work less viable even if we have the money and desire to be more transparent.

All that said, we expect and hope to do more public write-ups going forward. At the very least, we expect to produce more regular updates about what we plan to work on and high level overviews of what drives the work we do.
 

Shutting Down the Project

As for shutting down the project, we obviously did not shut down after six months but the same principle still holds, though we’re large enough now that it’s plausible we could be having impact in some areas and less or little in other areas. For this reason, while we generally think completely shutting down is very unlikely, it’s plausible we think we should rearrange resources within and across departments or significantly shift our focus at some point in the future, even if we don’t currently believe this is particularly likely for any given area.
 

Incomplete and/or Unsuccessful Projects

We’ve done a large number of projects over the past five years, some of which were undoubtedly failures. Below are three examples that come to mind:

1. PriorityWiki, started in 2018, was likely predictably unsuccessful because executing on the plan would have involved a large volunteer coordination effort we weren’t well-placed to execute, and it is unclear how valuable it would have been even if successful. This is a case of poorly considering the probability of success of a project, the resources required for continued success, and not rigorously analyzing the potential value of the project before completing it.
    
2. Considering pure research projects, we never completed the health economics series we started in 2020. This was a case of creating a project that was ambitious but nearly entirely dependent on the expertise of a particular staff member who left RP during the series, and we thus weren’t in a position to complete the project. In some sense, it’s always true that project leads are important, but this is a case of a project being completely dependent on a single person, something that we have always tried to avoid but failed in this instance to do.
    
3. In 2019, we did a project in collaboration with Rethink Charity (who was still fiscally sponsoring us at the time). The project, called Rethink Grants, entailed directly publicly assessing the case for funding Donational. We believe the rigor of the quantitative portion of this project was well done (perhaps even overdone given the possible return), but there was little to no demand for this type of work. Without a clear target audience for who would donate to support Donational given our analysis, the theory of change was underbaked and is thus an example of one of our bigger mistakes.
    

Issues with Follow-through and Impact

Our biggest early mistake was not building a credible plan from each project’s conception about which decision makers would benefit from our work and how to influence their decisions.

As a research organization, even the most accurate and rigorous projects are not useful unless someone acts on them.

A somewhat related challenge has been measurement and evaluation. Being a research organization influencing others often means being two or three steps away from the actual impact in the world. As such, it’s vital that we rigorously analyze our impact. We don’t believe we’ve put enough resources into this area, and it’s something we’re working to improve. Last year, we hired a Chief Strategy Analyst, Kieran Greig, who, among other things, is working on better understanding our impact and paths to impact. We’ve also generally put more time this year into analyzing the impact of our work on an internal model and into our OKRs.

Some Internal Issues

As we’ve grown, the value of having clear and unified project management across the organization has increased. While we anticipated and altered our project management in preparation for team growth in 2022, we would have liked to dedicate more resources to establishing project management prior to scaling. We’ve done a significant amount to refine our approach this year and believe we’re on the right track.

Relatedly, while we think our research has often been impactful, it’s often been very difficult to predict when a given project will be completed. In general, as a research organization, defining the point at which you should stop working on a piece of research is difficult. Returns can decline initially and seem to be on pace to steadily decline only for an additional piece of information or perspective to change the direction of a piece. Various departments operate under different systems with regard to the hardness of deadlines, with some having clear external deadlines, and others tending more toward setting their own deadlines based on how much value they think remains in a piece.
 

The Next Five Years

When continuing to thoughtfully grow RP, we’ve kept in mind eight different key constraints for our work:
 

1. Valuable work – We need a sufficient amount of important, neglected, and tractable work for us to want to hire people to complete.
    
2. Strategic clarity – We need sufficient skill at prioritization, strategic clarity, and vision such that we're tackling the valuable potential work (see #1 above) in a good way.
    
3. Talent – We need a sufficient number of talented employees to apply to our roles such that we are able to hire all the people needed to do #1 and #2 well.
    
4. Management – We need a sufficient number of people managers and sufficiently strong people and project management systems so that the people hired as a part of  #3 are aligned with the goals from #1 and #2.
    
5. Operations – We need a sufficient amount of operations employees and operations processes to be able to run the organization in a way that facilitates all of the above.
    
6. Money – People are the key to doing the above and you can’t employ people without money. We need to keep spending in order to keep RP going and that involves more fundraising.
    
7. Throughput – Even if we have 10 people we want to hire and have the management and operations capacity to have them do good work, it will still take time for people to join, get onboarded, become productive, etc.
    
8. Culture – We need to maintain strong organization culture and happiness so that existing staff feel comfortable with scaling and that the growth can be productive in a healthy way.
    

Any of the above constraints could end up being a bottleneck for our work. To give some sense of the likelihood of each, we share brief notes on our current assessments of each below.

So far, we continue to have much larger research agendas than we could tackle even with 200 people, let alone 70. And many questions we tackle just generate more questions. We have many more important questions than we can answer and expect this to continue for a long while.

Our strategic clarity seems sufficient, though always with room for improvement as we continuously learn. We do have approaches to our problems that we feel confident in executing, and we’re excited to get to work on them.

Available talent is still clearly there. We are unsure if and when we will encounter limits on this, but so far there continue to remain far more talented people worth hiring than we currently can hire given the other constraints.

Management capacity has historically been a key constraint. While it varies across departments, surprisingly enough, we often currently find ourselves having a lot of staff interested in managing, who seem like they might be good at it, and we don’t have enough people for them to manage. We’ve also historically done well at hiring research managers from external hiring rounds. Thus, we don’t see management capacity as a key bottleneck right now. We think we could relatively easily absorb a dozen new staff with our current management capacity. The main constraint here would instead be that our new management would be fairly junior and this could put additional stress on our overall people and project management systems as we try to orchestrate the entire organization.

Operations capacity is a key constraint right now. We are working quickly on solving this, including by bringing on two new staff members and streamlining and upgrading some of our systems. We expect this to no longer be as much of a constraint by as soon as September this year.

It’s still unclear how much of a concern and constraint our organizational culture and happiness will be—this is something we continue to monitor. We run anonymous surveys twice a year and are in interpersonal contact with our staff to continue to monitor the organization’s health and are taking steps to proactively assess and address. It appears that organizational health and happiness is still high absolutely speaking but that it has gone down somewhat following our large growth.

Money is our biggest bottleneck for right now. Across everything we want to do, the core reason we can’t do more of it is not having enough money. We will be aiming to make these funding gaps better known and fundraise a lot, starting immediately, to continue our ability to do good work.

Going forward, RP will publish a post about our impact, our future plans and funding needs in the second half of July. You can help us build the future we are envisioning.
 

This post is a project of Rethink Priorities— a research and implementation group that partners with foundations and impact-focused non-profits to identify pressing opportunities to make the world better, figures out strategies for working on those problems, and does that work. This post was written by Marcus A. Davis and Peter Wildeford with editing from Rachel Norman and additional input from Sarina Wong, Kieran Greig, Abraham Rowe, and Janique Behman.

If you are interested in RP’s work, please visit our research database and subscribe to our newsletter.