Cause-specific Effectiveness Prize (Project Plan)

By Gidi Kadosh @ 2020-05-02T16:25 (+31)

Context

This post is a project plan for a Cause-specific Effectiveness Prize, intended to be launched locally in Israel, as a part of EA Israel’s activities.

The project is currently in the stage of market research, and we wanted to post this plan in order to receive your feedback before we continue.

Definition

This project’s aim is to establish a competition/prize for cost-effectiveness, between charities that work on the same cause.

Motivation

Our assumptions for potential value in this project:

Encourage a conversation of impact effectiveness within local charity culture.
Drive innovation inside specific causes. Both in terms of creating/researching new interventions, and in terms of publicizing those interventions. For that reason, this project can actively put a spotlight on interventions rather than on charities.
Spark interest in EA within charities that aren’t naturally involved in EA. As of today, when a charity that doesn’t involve with EA’s top causes encounters EA, the message that charity receives is mainly “you better switch a cause or change everything you’re doing”; a message which doesn’t take into account personal attachments to that charity’s cause, and the emotional difficulty of changing assumptions. Instead, this charity could receive a message of (and potentially even guidance for) “here is how you can apply evidence and reason in order to improve your impact”.
In general, we want to test the potential of this route as a lead for new EA causes, and as a community-building approach (either by reaching out to new potential members, or encouraging individuals to move into EA-related causes from non-EA-related causes).
An important consideration regarding the cost-effectiveness of this project, is the potential of this project to provide resources we otherwise would have to work much harder for. Many donors have strong preferences to donate locally and are reluctant to donate to our top charities for that reason, but could find this project attractive. New volunteers had also shown much interest in this project.
Regarding the previous argument, this project can also spark the interest in EA by donors that aren’t naturally interested.

If successful, this project could be launched:

Locally, in other countries.
Globally, calling out for charities all over the world that are involved in a specific cause area.
Could evolve (after it is somewhat known within local charities) into more complex forms such as Impact Prizes, which seems very complicated to implement or pitch well without having an existing strong status.

Method

The general outline of this project consists of three main steps; Gathering resources for the project (especially funding), publishing the contest, and evaluating results. To simplify the evaluation process, we will pick a couple of different cause areas for the prize.

Below is a scale for the actions that can be undertaken by prize candidates. These actions are ordered by the amount of effort required by the charity.

Level 1: Figure out what to measure, then employ measuring techniques (a required action for applying).
Level 2: Make modifications in current interventions. Such modifications could be minor, for instance, increasing social influence by text modifications (example. Video about such modifications).
Level 3: Duplicate an intervention that is carried out somewhere else. We hypothesize that a majority of charities in Israel haven’t done that global-level research.
Level 4: Innovate an intervention based on subject-matter experience, theory or adjacent causes.

The prize requirements should be adjusted so charities can undertake only level 1 actions, and still have a chance of winning (in order to minimize the effort required by charities for competing). Nevertheless, we should provide guidance (at least through online material) for all levels.

Open Questions

Do we provide counseling to charities? Personal counseling, or online comprehensive guides?
RCTs are the golden standard of measuring impact, but are expensive and complicated. What easier alternatives do we allow? What’s the level of evidence and reliability we accept?
Evaluation process: Are charities required to file documents, do we pay for third-party analysts? Should we have a committee? How transparent can/should we be about the results?
How much money do we need?
There might be extremely important distinctions between the definition of the project: Is this a grant, prize, contest?

Possible Risks

Not enough charities will care or want to put in the effort we require.
Such contests might eventually discourage knowledge sharing between charities.
Legal risk - let’s not get sued.
Is EA Israel ready for such exposure, or for handling this amount of money? Could this create a misleading image of EA as mainly focused on cause-specificity (given that it’s our only project with this approach).
Miscalculating how complex our criteria for evaluation are, or leaving out opportunities for disruption of results.

Plan

Immediate Steps

Consulting

With other prize-focused projects in EA community.
With prize-focused organizations in Israel, such as Midot.

Market research (with charities)

Amount of money that will create interest.
Intent of participating. What would make them apply?
What information do they have today, or they can make available, for analysis?

Evaluate resources needed for this project: What personal do we need, evaluate costs.

Additional Steps

Funding - starting with our current funding leads.
Writing down precise definitions and requirements for the prize.

Definitions (and resolving the open questions).
Legal consulting.
Creating online materials.

Advertisement and reaching out to charities.
Follow-up on competing charities.
Evaluations of charity filings.
Optional: Final event for announcing the prize winner. Only if we can be certain enough about holding a big and fancy event. The main value of this is EA Israel’s status within Israeli charities, for other projects and future partnerships.
Retrospective analysis of the project (conclusions for another round of this project in Israel, publicizing results in EA community, organizing project’s resources for future replication).

Ramiro @ 2020-05-05T19:54 (+12)

Thanks fo the post, I really enjoyed this idea; please, let us know about its progress. I'd like to see something analogous for academic research and published papers.

As for your open questions:

Personal counseling, or online comprehensive guides?

I guess comprehensive guidelines are better, but they’re not exclusive alternatives, right?

RCTs are the golden standard of measuring impact, but are expensive and complicated. What easier alternatives do we allow? What’s the level of evidence and reliability we accept?

Maybe you should let your contestants and committee have this discussion; of course you can't RCT everything - e.g., no one would do it for parachutes. Actually, it might even be an easier problem than finding a common metric to compare different interventions - e.g., take a look at GiveWell's blog.

Evaluation process: Are charities required to file documents, do we pay for third-party analysts? Should we have a committee? How transparent can/should we be about the results?

Yes, yes, a lot. Suggestion: evaluate contestants in a two, perhaps three-level process:

a) approval voting in an online platform: you can either have an open website where people can vote for free (after identifying themselves), or you can charge voters a fee (two pros: voters will have some skin in the game, and partially fund your costs);

b) individual reviewers rate / select the n most voted contestants;

c) your committee decides the winners.

Risks: I guess reputational risk might be your real issue. Some sort of worst-case scenario: you lose your money and resources to this awful winner... but the real problem is that it goes viral, and everyone starts associating "Effective Altruism" with something like "Canadian Satanists chanting religious poetry in rich schools". Even my illustrative example sucks. Is there any way of letting EA get the benefits of the exposure, but not its risks?

BTW, do you plan to do it just once, or do you intend people to expect it to become a periodic contest? Signalling it could be repeated could influence future projects (in case you end up being successful)

MaxRa @ 2020-05-05T08:04 (+5)

Hey guys! I think it's a cool idea, and I think it's great form to share such a concise summary for feedback. Just some uncertain comments off the cuff, because nobody started the conversation yet:

1) It seems like a resource intense project, e.g. working out reasonable metrics, evaluating them, reaching out, get enough funding. I'd be worried that a bad execution might lead to bad press. For example, somewhere in the back of my mind I remember a discussion of charity representatives in Germany that were very dismissive of the idea that their impact could be measured.

2) "Contest" vs. "Prize": Maybe their is less risk at bad press when it is framed as a prize. Just a feeling that nobody ever forces you to compete for a prize, but it's sometimes mandatory to take part in a contest.

3) Maybe you could try to connect interested researchers with charities and let them work out a way to measure their impact. Then prizes go out to the best reports/papers (the money should probably go to the charity, to incentivize them). I think there is already an existing research field around impact measurement, so you could worry less about counseling the charities and let the researchers work this out with them.

Aaron Gertler @ 2020-05-12T09:27 (+3)

What are examples of times that it might be "mandatory to take part in a contest"? I'm curious what you're thinking of here, and whether my inability to think of examples might be based on a language barrier or something like that.

MaxRa @ 2020-05-12T15:48 (+1)

I think I should have emphasized that this was just me non-native speakers gut reaction. Maybe a competition is more what I meant? I think about competitions in school where you had no choice but to take part (e.g. maths or reading or sports), or the competition from the movie Hunger Games. Somehow a prize sounds more like something I get for a bonus, while competing against others is to me more associated with "There is not enough for everyone, this is zero-sum, who is this evil person that would make charities compete against each other for their financial survival?".

Aaron Gertler @ 2020-05-12T09:37 (+2)

Here's where I imagine most of the value coming from for a project like this:

EA Israel members learn about impact evaluation from working with charities.
Other national groups learn from your project and figure out whether they should implement something similar themselves (including in countries like India, where I'd guess that more local charities have a reasonable chance of being high-impact choices on the level of a GiveWell-recommended charity).

Both of these could be very valuable, and I'm curious to see how the project goes!

A few caveats, though:

I would guess that you won't learn about any charities that are really exciting in the sense of being competitive with GiveWell's recommendations.
If you get local publicity, you might be able to shift some donations toward higher-impact options within Israel, but I'd expect the overall effect to be pretty small.
1. Most people don't care that much about charity effectiveness and local giving at the same time, and I'm not sure how much impact differs between great and average Israeli charities (compared to the ten- or hundredfold increase in impact you can get by shifting people toward GiveWell-recommended orgs).

When you say that holding a nice event for the winner could help EA Israel's "status within Israeli charities, for other projects and future partnerships," are there any specific projects/partnerships you're thinking of? I can't immediately think of any examples of EA groups interfacing with non-EA-aligned charities for productive projects (though this doesn't mean it's never happened, or that it couldn't happen).