Announcing the Change Our Mind Contest for critiques of our cost-effectiveness analyses

By GiveWell @ 2022-09-06T18:10 (+142)

This is a linkpost to https://blog.givewell.org/2022/09/06/announcing-change-our-mind-contest/

Author: Isabel Arjmand, GiveWell Special Projects Officer

We’re extremely excited to be announcing the Change Our Mind Contest to encourage critiques of our cost-effectiveness analyses that could lead to substantial improvements of our overall allocation of funds. For all the details, see this page.

Cost-effectiveness is the single most important input in our decisions about what programs to recommend, and we believe it’s possible that we’re missing important considerations or making mistakes that lead us to allocate funding suboptimally. We’ve been excited to see people engaging with our cost-effectiveness analyses, and we’d like to inspire more of that engagement.

With that in mind, we’re inviting you to identify potentially important mistakes or weaknesses in our existing cost-effectiveness analyses and tell us about them!

The first-place winning entry will receive $20,000, the runner-up will receive $10,000, and the honorable mention will receive $5,000. We may offer multiple runner-up and honorable mention prizes if the quality of submissions is particularly high. All other entries that meet our criteria will receive a participation prize of $500, capped at a total of 50 participation prizes for the first 50 submissions.

In addition to the monetary prizes, excellent entries may lead to changes in how we allocate millions of dollars of funding, leading to more lives saved or improved.

Entries must be received by October 31, 2022, and the requirements are described in detail on the contest page. We will announce winners by December 15, 2022, and will publish the winning entries online. Work that was submitted to the EA Criticism and Red Teaming contest is eligible as long as it meets the requirements stated on our contest page; authors are also welcome to submit an edited version of work they submitted to that contest, and might want to do so in order to better tailor their work to the prompt on this page. If you're posting your submission here on the EA Forum, please tag it with GiveWell Change Our Mind Contest so Forum readers can easily find all entries published here.

If you have any questions, please leave a comment on this post or email change-our-mind@givewell.org.

We’re running this contest because the recommendation decisions we make are extremely important, and we want to incentivize feedback that will improve our work, thereby enabling us to do more good. We hope you’ll consider participating!

MichaelPlant @ 2022-09-08T18:12 (+34)

I'm really pleased to see GiveWell is doing this, and particularly that you singled out HLI's critique of GiveWell's deworming CEA as an example of what you'd like to see. 

I am, however, disappointed that the scope of the competition is so narrow and a bit confused by its name.  The contest page says you do want people to re-analyse your existing interventions but that you don't want them to suggest different interventions or make  'purely subjective arguments' - I'm not sure what the latter bit means, but I guess it rules out any fundamental discussions about ethical worldviews or questions of how best to measure ‘good’. On this basis, it seems like you're asking people not to try to change your mind, but rather to check your working.

This strikes me as a lost opportunity. After all, rethinking what matters and what the top interventions are could be where we find the biggest gains.

At the risk of being a noisy, broken record, I, and the team at HLI, have long-advocated measuring impact using self-reports and argued that this could really shake up the priorities (spot the differences between these 2016, 2018 and 2022 posts). Our meta-analyses recently found that treating depression via therapy is about 9x more cost-effective than cash transfers (2021 analysis; 2022 update), we'd previously explored how to compare life-improving to life-saving interventions using the same method and pointed out how various philosophical considerations might really change the picture (2020).

I'm still not really sure what GiveWell thinks of any of this. There's been no public response except that, 9 months ago, GiveWell said they were working on their own reports on group therapy and subjective wellbeing and expected to publish those in 3-6 months. It looks like all this work would fall outside this competition but, if GiveWell were open to changing their mind, this would be one good place to look.

GiveWell @ 2022-09-12T18:58 (+20)

Hi Michael—this is Isabel Arjmand, Special Projects Officer at GiveWell. Thank you for the feedback and for HLI's critique of deworming, which played a role in inspiring this contest!

We designed this contest to incentivize critiques that are relatively straightforward for us to evaluate and particularly likely to change our mind about upcoming allocation decisions. This is our first time running a contest like this, so we wanted to keep the scope manageable. We may run future contests with different or broader prompts.

A bit more color on why we're keeping the contest's scope to our existing cost-effectiveness analyses:

  1. We believe critiques of our existing cost-effectiveness analyses will be relatively straightforward for us to review, as opposed to broader critiques, such as those that suggest we take an entirely different approach to recommending giving opportunities. We anticipate that having this well-defined scope will make it easier for us to compare and give due consideration to all entries with our current research capacity (which we are hoping to expand!).
  2. We're getting ready to make some large decisions about how to allocate funding across the programs we currently support at the end of the year. This contest is designed to solicit the feedback that we think has the greatest potential to improve those upcoming decisions; excellent entries could meaningfully change how we allocate funds, leading to more lives saved or improved. Broader critiques or proposals for wholly new approaches would be unlikely to influence this year’s decisions, given how much vetting we put into our allocations and how little time remains before they are finalized.

Outside of this contest, we welcome feedback on all aspects of our work, and we're glad to receive those at any time via email, as blog comments on our open threads, or here on the EA Forum. 

We appreciate your continued engagement on subjective well-being, particularly the useful feedback you provided on our draft reports explaining why we're not as optimistic about subjective well-being measures and Interpersonal Psychotherapy Groups as you are. We're still planning to publish those reports, but we're behind the timeline we originally laid out. Thanks for your patience with this! 

Karthik Tadepalli @ 2022-09-07T01:14 (+15)

Exciting contest! I'd encourage the creation of a tag for this contest, to help people collect and read through entries that are posted on the Forum.

GiveWell @ 2022-09-07T20:15 (+4)

Thanks, Karthik - good idea, and looks like someone has already done this on our behalf. We'll edit the post to ask people to use the tag.

david_reinstein @ 2023-01-19T23:17 (+6)

Update: I overlooked some substantial discussion here

I'm looking forward to further followup on ...

I guess it's always good when you have a suggestion contest to explain your process for responding to the suggestions.

GiveWell @ 2023-01-24T17:44 (+2)

Hey, David - yes, the section of the post you linked to contains our thinking on possible changes that could result from COM Contest critiques. We are still in the process of figuring out what specific changes we might make to our analyses based on these critiques.

Best, Miranda (GiveWell communications associate)