How should we run the EA Forum Prize?

Regarding professional vs amatur researchers, I think that a technically easier distinction would be of posts that were written as a part of some organisation's work or independently. (So that professional researchers in EA Orgs would still compete for the prize like anyone else). This makes sense, as they got paid for that work and probably would have done it anyway regardless of the forum prize. This would apply to people working on funded independent projects too.

I think that other methods mentioned in the post would be better suited for leveling the playing field.

Personally I find the prize disproportionately motivating, in that it increases my desire to write EA forum content to a level beyond what I think I'd endorse if I reflected for longer.

Sorry if this is not very helpful; I imagine it's also not very representative.

General comments

In general, I am in favour of having a forum prize of some kind. I personally find it motivating and I think it is a useful way for people to gain credibility within the EA movement. I think it also helps lend the forum an air of seriousness, which is sometimes good and sometimes bad but does probably raise the quality of the content.

From the "giving individuals credibility" perspective, having a few large prizes is good (a $750 prize is large enough to put on a CV for many people, a $50 one is clearly not). From the "motivating good content" perspective, having a moderately larger number of moderately smaller prizes would probably be better.

Perhaps I'm wrong about this, but I've sometimes felt like comment prizes are used to make up the difference on post prizes ("your post was very good but not top-3 this month, so we found a decent comment you made on that post and gave it to that!"). If so I think that is good evidence that we want more smaller prizes.

Comments on specific proposals

Themed prizes (e.g. setting aside one prize for “the best post on topic X”)

This seems like something you might want to do less frequently (annually?) in addition to more regular general prizes.

Giving more prizes, even if they end up being smaller on average

On the whole I think this would be a good idea (see above).

Selecting at least one judge to represent each major EA cause area

I don't feel like any major cause area has been getting shafted here; I can readily recall winners on global health, animal welfare, long-term future, and meta topics. In the absence of a specific problem this seems like not a good idea; I feel like it would not provide much benefit and might encourage tribalism/factionalism.

The above comments assume topic-general prizes like the current system; I think themed prizes probably would want some specialist judges involved.

Including a community vote (not just upvotes, but a separate voting process). This would likely supplement judges' votes, not replace them.

A lot of award ceremonies have some kind of "people's choice" award. If you do this I again think it should probably be less frequent than monthly; perhaps quarterly or annually.

Having a special “first post” and/or “first comment” prize for people who make a really good first contribution from an account

Meh. This sounds cute but I don't actually think it would be very valuable, and could actually be harmful on net. I think it's good to set a high quality bar and incentivise people to work up to it, rather than agonising over making their first post perfect even more than they probably already do.

Having separate prizes for orgs/professional researchers and people who contribute to the Forum on more of an “amateur” basis

So I suppose the argument for this is that some people basically get paid to write on the Forum, and it's not surprising that those people (coughSauliuscough) win lots of prizes, which crowds out good content from amateurs. I'm not sure to what extent this is true, and if true, I'm unconvinced that it's bad.

Firstly, a lot of that content is just very good and I want there to be strong incentives to get that stuff on the forum. Secondly, a lot of prize-winning posts are from people who work for EA Orgs but are writing in their private capacity; indeed, you'd expect people who are especially committed to thinking about EA topics to both be more likely to win Forum prizes and be more likely to work for an EA Org. This seems like a good thing we want more of, no less so than contributions from non-Org-employees, so I don't think they should all be put in a special category and forced to compete with each other.

There's a similar argument which is something like "separate out content people write for work vs for pleasure". I think this is more defensible but still probably wrong. If prizes incentivise orgs to post more of their work on the EA Forum, this seems like a good thing that benefits everybody. But I wouldn't be extremely surprised to be argued out of this.

Having more flexible prize amounts (e.g. maybe one post should win all the money in some months if it’s especially good, or maybe money should be distributed according to vote ratios rather than just first/second/third place)

This sounds bad to me, but for fairly vague reasons. I feel like it gives the judges too many degrees of freedom, and that it's probably good practice to have prize amounts be fairly predictable. But I'm not sure about this.

Having judges who are somewhat removed the community (or finding some other way to reduce the extent to which the Prize may reflect the biases of the community or of central orgs within the community)

This sounds like a great way to remove most of the value of the prize in return for highly dubious gains.

If I look at the winners of the EA Forum Prize, I expect to see exemplars of great EA content. The best people to judge great EA content are EAs, i.e. community members. We could bring in external expects from adjacent fields (global health, AI, etc), but why?

• Firstly, those people are likely to be far more limited in the range of posts they can judge, which basically forces us into the "subject-specific prizes" model above; while I think those could be good to have occasionally I wouldn't want them to completely take over the prize.
• Secondly, those people are almost certain to be less well-aligned with the distinctive values of the EA community, which makes their opinions on which posts should get prizes much less valuable in this particular context.
• Thirdly, if we do want domain experts to judge a domain-specific EA prize I'm pretty sure we can find them within the EA community, rather than bending over backwards to make them external.
• And finally, all those fields already have their own fora, their own prizes, and their own way of sharing and evaluating information. It's fine – indeed, very good – for the particular nexus known as EA to have their own systems and prizes too. It only becomes a problem if we think EA Forum prizes are the sole arbiters of truth and quality.

Getting rid of the Prize entirely without replacing it (one survey respondent, who has written many excellent posts and won at least one Prize, believes it to be “distracting and unnecessarily divisive”)

Has there been any evidence of the prize being divisive, i.e. actually causing conflict? Perhaps there is, but I'm not aware of it. And calling it distracting is confusing to me; is the claim that it incentivises people to write content based on a Keynesian beauty contest, rather than what they actually think it would be best for them to write?

Anyway, I don't have much sympathy with that claim as stated, or for abolishing the prize entirely on that basis, but there might be an alternative interpretation that I'd be more sympathetic to.

Why don't you conduct an experiment? E.g. you could award prizes only for posts/comments written by users whose usernames start with letters A-L (and whose accounts were created prior to the announcement) and see if you notice any significant difference in the quality of those users' submissions.

I'll offer a data point: I'm not particularly motivated to post on the Forum by a monetary prize. It hasn't led me to post on the Forum more than I ordinarily would. I am somewhat interested in social rewards, but the karma feature seems to do that better than the Forum Prize.

Also, as someone who doesn't read every single post on the Forum, I also find the Prize useful for highlighting what content is actually worth reading, but again, I think highlighting posts based on karma instead (with or without a monetary prize) would work just as well.

If the Forum Prize does continue, I do think there should be separate categories for professional researchers and "amateurs."

So for me the prize fulfills some very important purposes. Perhaps the most important two are:

• Curating the best content
• Rewarding content creators for producing content

----------------------------------

Curating the best content

I regularly use the prize posts as a "summary of the best of the month" which I greatly appreciate. It helps me focus my attention on the best articles of the month. It is also a great experience for the authors, who just publish content as usual and without any additional overhead sometimes they get selected for the prize. This is how I wish more academic areas worked - everything published openly in a preprint archive and then the journals acting as "curators", selecting the best work. I really wish something like this will remain in the forum (a "best of the month" selection).

This is a very useful function also to help analyze a posteriori the impact of the best pieces of the forum, as for example with this  post.

-----------------------------------------------

Rewarding our content creators for producing content

I would think some people who are specially competitive are motivated by the prize to write more. But I don't know how large of a share of the community is like that.

Instead, I think the most important reward is making people feel proud and recognized for their work. When somebody I knew won the forum or comment prize, they were showered with praise and felt happy and appreciated.

This goal is a bit at tension with the goal of curating the best content. People are sometimes disheartened by the posts being consistently won by professional researchers who have the time and experience to write very good posts.

I think I would like this goals to be somehow separated - though I admit I am a bit confused about how one would go about doing that.

One specific point about the "first post" / "first comment" idea:

If someone asked me for advice on how to get good at writing EA Forum posts, the most succinct advice I could give is this:

write!

You get better by doing something by practicing, so I would encourage someone to produce more content.

Which seems to suggest that a first post/comment prize is bad because it holds people back from taking the first step, and therefore discourages people from getting better.

I appreciate that there's a counterargument to this (we don't want to flood the forum with bad content), but I don't feel like we have that a problem with this now, so I'm less worried about the counter.

In my case at least, I can say that both the money and the reputation for winning was extremely valuable. Thanks for that!

Maybe more but smaller prizes? I think karma works as a better signal to internalize since the feedback is more consistent and granular, but if more prizes were given out, this gap would be smaller. Maybe reduce the post prizes to $150 to $300 and give out more of them and/or more comment prizes?

If someone is working on a post, I don't think $200 vs $500 makes much difference in motivation beyond the karma on the post and recognition with the prize, but 0 vs $200 could (although I don't know that either would make much difference for me beyond the karma).

As discussed in a bit more detail in this post, I'd love to see themed prizes focusing specifically on critical engagement with effective altruism. This could be very broad (e.g., "Best critique of the effective altruism movement") or more narrow (e.g., something like "Best critique of a specific assumption that is widely made in the community" or "Best writeup on how applied longtermism could go wrong"). 

To the next content specialist on the Forum: I'd be happy to discuss further!

Almost all content useful to EAs is not written on the forum, and almost all authors who could write such content will not write it on the forum. So it would be a lot more valuable to reward good content whether or not it is on the forum. It is harder to evaluate all content, but one can consider nominated content. If this is outside one's job description, then can one change the job description?

Possibly, prizes can be directed to specific proposals. I can imagine a platform (which could be the comment section of a post) where people write what they intend to write about, or what they'd want someone else to write about, and people can comment and upvote on what seems to them important (and why). Out of these, several such proposals can be chosen and offered a bounty on them - which would be the maximal bounty if a thorough and well-written post would be written on them. Then, quality and how much the original question was solved would be assessed and the prize given would be the bounty times the precentage of the question solved times some quality factor.

A different approach. It seems that the prize is designed to improve incentives for writing high-quality posts. This is arguably not as important as other routes for better information flow, communication, coordination and epistemics. Perhaps people are already motivated enough to write posts, and are motivated enough to write them well. Perhaps they are even motivated for either of these too strongly (as opposed to other means of impacts and fleshing out less baked thoughts).

So several alternatives might be:

1. prize for best distillation, explanation or collection  
2. prize for best (unfunded) volunteer community project
3. prize for best public conversation/debate/anti-debate. 
4. prize for most important errors in arguments.
5. prize for general supportive and welcoming comments by a user over different occasions.