Personal Review of Participation in the Open Philanthropy Cause Exploration Prize

By Gavriel Kleinwaks @ 2022-11-02T06:05 (+30)

This is a linkpost to https://strainhardening.substack.com/p/personal-review-of-participation

Summary: The Open Phil Cause Exploration Prize was extremely top-heavy, which means that it might be more worth entrants’ time to work less hard and produce relatively lower-quality entries. Similar competitions should consider using less top-heavy structures. (Crossposted)

My colleagues and I recently submitted an entry to Open Philanthropy’s Cause Exploration Prize, and received an honorable mention, which came with a prize of $500. Open Philanthropy also awarded each good-faith entry with $200, awarded the first-place entry $25,000, and the three second-place entries with $15,000. 

Our entry was the culmination of a few months of work, started for a different project, so the OP contest was not particularly out of our way. However, in order to do extra research and write and polish the draft, another colleague and I dedicated essentially all of our work hours to the CEP draft alone, and the third member of our team dedicated another significant percentage of his time. Counting only the two of us who worked on it full-time, I would estimate that we spent about two full work weeks on it—assuming a 50-60 hour workweek (call it 55), this means that we dedicated about 220 hours on this project alone^[1], not even counting our third author’s efforts or the research we had done for the project prior to deciding to work on the CEP. This, of course, a terrible value for money if the prize is $500, especially since the honorable mention is relatively close in quality to the other entries. (The first-place award is 50 times the honorable mentions awards, but the report isn’t 50 times as good as any random honorable mention I read. I’d be hard-pressed to compare the quality nebulously, but intuitively I’d call the first-place entry perhaps 2-3 times as good as a random honorable mention entry, depending on what the judges’ criteria were.) 

That’s not to say I wasn’t excited to receive the award! I felt great about the recognition, but realistically, the main reward was that people liked the report and found it valuable. It definitely wouldn’t have been worth it to enter the contest if I thought writing the report would distract from my work. 

The balance of considerations becomes more extreme when you consider the award for a good-faith entry. I was proud of the report we turned in, because the deadline was extended by a week, so we were able to do extra research and the report was far more thorough than it would have been otherwise. On the other hand, considered from a purely financial perspective, this was a terrible choice—assuming we didn’t have the extra week, and could only spend 110 hours on the draft, we still would have received the good-faith entry prize. But we also would have received the good-faith entry prize if had done zero extra work and exclusively committed to writing up our existing research at the time, without adding any extra research—let’s say that would have taken ~20 hours (and, to be clear, would have been a wildly low-quality report). We would have spent ~10% of the time to receive 40% of the prize. If it weren’t for the fact that this was already part of my job, this would actually be better time use!

So, okay, given that this was part of my job, was there really any issue? Was there a better way to spend our work hours, or did this stage of the project justify at least two full work weeks across two authors? I think yes, it was justified. We’re in the process of revising the report significantly to more closely fit our original project goal, but it was incredibly helpful to us to have a forcing deadline; we had yet to give ourselves a deadline to produce a polished draft of anything. We’re reusing a lot of the writing; I think that forgoing the OP contest would probably have saved about 20% of the time spent, or ~44 hours, not quite a full work week for one person, and not a bad time outlay given the benefits. We’ve since sent the report around to some collaborators to provide background on our project. Also, on a personal level, I figure it’s generally good for me to produce public writing; maybe the other authors would say the same. The work tradeoff mostly came in the form of anxious false starts, time spent just on polishing writing, and letting other work fall by the wayside; aside from ditching another project to work on this one, I was fairly burned out in the week following the sprint to submit (although that’s a risk for any project).^[2] 

A prize competition structured like this one is weighted very heavily toward the ends—there’s a strong reason to put in tons of effort if you think you have a shot at winning first or second place; there’s also a decent reason to submit any good-faith entry regardless of quality.^[3] But given that most people don’t get first or second place, you should probably assume you won’t, and this might mean that you shouldn’t put much effort into your submission unless writing it is actually just your day job. Given that structure, probably lots of people enter reports related to their existing projects, which is good, but also maybe the competition organizers will have to spend a lot of time reading much lower-quality reports than might otherwise be produced. I can also say that outside of work, I would be generally more likely to enter a competition with a smoother prize gradient.

It would be a simple fix to restructure the prizes to be less top-heavy. For example: twenty honorable mentions were awarded, for a total of $10,000. If each were $1,000 instead of $500, they’d be five times the size of the good-faith entry awards. To avoid adding the $10,000 to the grantor’s cost, the top-tier prizes could each be reduced by $2,500 each, leaving even the second-place prizes at over 12 times the honorable mentions. I expect that in this and similar competitions, many of the honorable mentions were close to the winners in quality and usefulness, so that type of restructuring seems reasonable.  

I wanted the main thrust of this post to be the specifics of the OP Cause Exploration Prize, but it’d be incomplete if I didn’t mention that my thinking about this specific prize has made me more concretely skeptical of essay/report prizes in general as great uses of time for either the judges or participants. Consider that if I’d spent 20 hours writing up our existing research into a report that would net the good-faith entry award, I’d receive $200, or $10/hr. It’s not totally fair to compare prize money to normal wages; prize organizers definitely get more participation than they could afford at normal research contractor rates. However, prize structures potentially discourage people with high opportunity costs from participating, and it seems easy for judges to get swamped with low-quality entries. I said above that I’d be more likely to enter a competition with a smoother prize gradient—this is true, but actually I’m pretty unlikely to use free time to enter a prize competition at all. This situation makes me skeptical of prizes as incentives for problem-solving, rather than as rewards for something already achieved for other reasons (like Nobels).

That said—whether a specific essay competition is a good use of time is, of course, up to the organizers to decide. I can easily imagine the organizers having priorities that favor the top-heavy structure because they only care that they receive a few great entries, and care less about the quality of everything below the top tier. And anyway, I’m all for organizations experimenting with new ways to get input and information.

Many thanks to Austin Chen for the idea about how to restructure prizes and also for listening to my description of the problem in the first place and saying, “Hey it sounds like you should write an EA Forum post about that.” Also, seriously, thanks to the Open Phil judges for awarding us an honorable mention—I was legitimately delighted.

1. ^{^}

   Those 220 hours don’t literally mean that all of those hours were spent on the report—my estimate of a 55-hour workweek includes various workday distractions. However, I feel very comfortable labeling it as 220 hours dedicated to the report, since all productive hours were spent on the report and unproductive/distracted time wouldn’t have been used for any other work anyway.

2. ^{^}

   One possible outside-of-work tradeoff is that prize money would make it a slightly bigger headache than normal to do my taxes, but my colleague and I agreed to donate the prize through OP’s prize claim portal. Since I was pretty sure we could agree on a charity we’d both be happy to donate to anyway, I specifically suggested the donation to avoid the 1099. Incentives!

3. ^{^}

   Open Phil did provide requirements for considering something to be a “good-faith” entry, the most onerous of which were that the entries needed to engage with relevant academic literature and should be at least 3,000 words. That’s certainly not nothing, but it can be done pretty quickly if you have research experience, and yet will not necessarily result in seriously informative work.