We Should Run More EA Contests

By Akash @ 2022-01-12T23:45 (+35)

TLDR

I think we should have more EA contests. EA contests can generate direct value, help people develop/test their aptitudes, incentivize people to develop inside views, and help organizations find potential hires. 

Also, I recently ran a small contest at a retreat (skip to the "Minimum Viable Contest" section to see how that went).

What is an EA Contest?

An “EA Contest” is essentially a call for proposals that includes a monetary reward to winners. In other words, a contest includes a contest prompt (e.g., work on X problem) and a prize (e.g., the winners get $Y).

Examples include the Creative Writing Contest, the Philosophy Contest, and the Eliciting Latent Knowledge Contest.

Why Are EA Contests Valuable?

First, I think contests can provide direct value. For example, it seems plausible that winning entries to the Creative Writing Contest could be used to generate EAs or increase the motivation of existing EAs. 

Second, I think contests help people test their aptitudes. I have met many EAs who think “ah, I have no idea if I’m a good fit for X, and I wish there were easy ways to test it.” Contests can give these people a straightforward, low-friction way to test out their fit for a particular aptitude.

For example, the Eliciting Latent Knowledge Contest provides a clear way to test your fit for theoretical alignment research. This reduces friction considerably. To spell it out:

Ron: How can I test my fit for theoretical alignment research?

Hermione: Well, theoretical alignment is a pretty messy and confusing area at the moment. I suggest you read a bunch of posts in the area and try to find holes in the arguments. But also be aware that many of the posts are unhelpful and inaccurate, so part of your job is to figure out which posts actually make compelling arguments.

Ron: Oh, uh, okay… *filled with confusion*

Harry: Actually, Ron, another option is to try the Eliciting Latent Knowledge Contest! There’s a defined problem that you can work on, all of the content is in one Google Doc, and it seems like you could get a “taste” for this work by investing about 30 hours.

Ron: Oh, wow. That seems less nebulous– I’ll give it a shot!

Third, I think contests improve the critical thinking of the EA community. Contests often incentivize people to engage with primary sources, form their own ideas/opinions, and refine their thoughts on a topic. Contests can also be used to help people develop inside views on important topics.

For example, let’s imagine a hypothetical contest that involves red-teaming Open Philanthropy’s EA/LT survey. There could be direct benefits to this kind of contest (e.g., helping us better understand how to interpret the results; finding ways that OP can improve the survey design). 

But I think the other benefit– which might be even more important– is that this contest would get more people to read the Open Phil survey (in detail) & generate their own interpretations. This illustrates a broader point: contests can incentivize people to defer less and form inside views more.

Fourth, I think contests help EA orgs identify untapped talent & potential hires. The Creative Writing Contest, for example, cast a spotlight on several talented EA writers. If an EA org is looking for creative writers, there’s now a pool to draw from. The Eliciting Latent Knowledge contest may also be used to help the Alignment Research Center find potential hires (indeed, this might be the primary goal of the contest!).

What could go wrong?

First, this could waste EA time. This is especially concerning if there’s a contest that looks appealing but actually produces little value. This is a waste of money and time (I’m especially concerned about the latter. A “bad contest” could suck up hundreds of hours of EA time, and EA time is a valuable/limited resource.)

Second, this could just flop. Maybe people are busy and contests just don’t generate a lot of interest. In this world, we still waste some EA time (e.g., the time of the contest organizers and grantmakers).

Third, maybe contests provide a bad incentive structure. People might optimize for “winning” rather than “discovering true and valuable ideas.” Contests may also discourage teamwork and sharing ideas.

Fourth, contests could generate a lot of publicity. Poor branding, marketing, or execution could harm the reputation of the movement.

Minimum Viable Contest

I recently ran a “Contest Contest” at a CEA retreat. I offered a $500 prize to whoever could come up with the best contest idea.

The contest lasted 24 hours, and I received 11 submissions (there were about 25 people at the retreat).

The winner (as judged by me, somewhat arbitrarily) was Ashley Lin, who came up with the following contest idea:

Design-A-Gap-Year Contest: "Prompt: imagine you received $20,000 to fund a gap year to do the most good. What would you do? Competition: present a gap year plan to judges that is impactful and feasible. Maybe the winner even gets funded!"

I was also really excited about another idea, so I offered a “surprise” 2nd place prize. This idea came from Kaivu Hariharan:

Community Building Reflection Contest: "Money for the best write up of detailed reflection on campus community building over a semester (eventually posted on the EA forum)  - what the incoming hypotheses were, what worked, what didn’t, what was tried, what was the landscape like, funding, etc."

Conclusion

Overall, I think the minimum viable product went well, and I’m excited to see more EA contests. I think well-run contests could a) generate direct impact, b) help people test relevant aptitudes, c) improve epistemics by encouraging people to form their own opinions/models, and d) help organizations identify talent & potential hires.

How You Can Help

I may apply for funding to run at least one contest in the upcoming months. Please reach out if you:

finm @ 2022-01-13T11:09 (+22)

Broadly agreed with this, but I'm a bit worried that contests with large prizes can have distortionary effects. That is, they might pull EAs towards using their time in ways which are not altruistically/impartially best. This would happen when an EA switches her marginal time to some contest with a big prize, where she otherwise would have been doing something expected to be more impactful (e.g. because she's a better fit for it), but which doesn't stand to win her as much money or acclaim.

For instance, I think the creative writing prize was a really great idea, but my own experience was one of feeling like I really ought to at least 'buy a raffle ticket', sinking a few hours into trying to write something halfway decent, then missing the deadling and feeling a bit deflated. I don't mean this in an arrogant way, but in retrospect I think it's likely I could have done something better with my time, impact-wise.

One fix could be leaning towards more and narrower contests, which require specialised skills, rather than fewer, highly general contests with large prizes. That way, the prize doesn't have such a big 'gravity well' that it sucks in a lot of 'might as well try' folks who were already doing useful stuff.

Another fix, which I like most, is more retroactive prizes/funding. The Forum Prize was a good start, although it's been retired with a promise of finding better (e.g. more democratic) alternatives. Note that some people have recently been writing about and trialling retroactive funding. There is likely more than a post's worth to discuss here, but I think one key idea is that there are potential EA projects where (i) only a couple people would really be suited to doing, such that (ii) a competitive prize wouldn't make sense; (iii) it's really hard to track all those ideas and promise rewards for them in advance; but (iv) it would be great if someone did them and were rewarded. I'd be interested to see more discussion of what scalable retroactive funding could look like in EA. In general, retroactive funding avoids this distortionary worry — as long as you trust the evaluators to judge what was most impactful, you can't beat just trying to do actually impactful things.

Anyway, bottom line is to notice that incentives can go wrong a bit more easily in an altruistic context, and you need to consider the kind of work that a well-meaning contest is getting people to replace. In some circumstances, the impact of a contest with a very large prize but questionable impact might be therefore be unclear, even imagining that money is free.

casebash @ 2022-01-15T08:16 (+3)

I think the forum prize should have focused on EAs not at orgs b/c those EAs are already sufficiently incentivised to do good work and when the prizes are dominated by people already at orgs this dilutes the ability of the forum prizes to highlight and encourage new talent.

casebash @ 2022-01-13T01:00 (+4)

Have you seen this post?

Emerson Spartz recently ran a similar contest where he was paying people $1000 to come up with bounties.

Here's a few ideas I shared in this thread.

I really like the second idea. I mean the first one isn't bad either, but a bit of a let down if you don't actually have the $20,000.