EA organizations should pay experts to peer review their paper drafts and research proposals

By Yarrow Bouchard 🔸 @ 2026-05-26T21:06 (+22)

A simple, inexpensive, relatively easy step that EA organizations could take to improve their research quality is to pay external experts to do peer review.

Another thought: it could potentially be disheartening to get this kind of feedback at the end of a research project, when the paper is almost ready to post online. So, maybe it would be even better to get input from experts in relevant fields at the earliest stage. Experts could review your research proposal and offer input, potentially saving you tons of time and heartache if you were about to make an avoidable error.

For example, if METR had gotten a research proposal for its AI time horizons work reviewed by some external experts, there are some avoidable errors in that work that potentially could have been averted. More discouragingly, but still important, if METR had submitted the draft of its paper on the time horizons work for peer review by external experts prior to posting it on its website, the paper could have better disclosed some of its errors and limitations. 

Peer review of both research proposals and paper drafts would be useful for two major reasons. First, it would be intrinsically useful because it would lead to better research. If the point of research is to tell us the truth, and we want to know the truth, well, then, better research will tell us the truth better.

Second, it would be instrumentally useful. An important goal for many EA organizations is to persuade a broader community of people about something — experts, policymakers, regulators, the general public, potential recruits to the EA movement. Higher-quality research is more persuasive. It’s also a good way to earn credibility and trust. Low-quality research is unpersuasive, and can even persuade people in the opposite direction. (“If that’s the best you could come up with, surely your conclusions must be wrong!”) Publishing low-quality or fatally erroneous research also damages credibility and trust. 

Two potential subcultural stumbling blocks:

  1. There is a strong undercurrent in the EA community of opposition to mainstream institutions — mainstream journalism, mainstream academia, and mainstream, institutional science. Even to mainstream society and culture.
  2. Not unrelatedly, there is a strong desire in the EA community to treat the community as an enclave (or conclave!), rather than a part of the wider world. For EA to rely only on itself for ideas, for input, for intellectual evaluation. 

I probably can’t convince anyone that these attitudes are wrong for intrinsic epistemic reasons. But maybe I can convince them that, in order to have a strong and durable influence on the wider world, it will be necessary for EA organizations to “play ball” and engage with the rest of the world on its terms. 

The EA community has certain beliefs, particularly about how close the world is to creating AGI, that most experts, forecasters, policymakers, and members of the general public disagree with. Some EA organizations just want to do technical research and don’t need to worry about what anyone else thinks. But other organizations want to persuade the world of the danger. 

Maybe some people feel cynical and don’t dare hope that the world could actually be persuaded on the basis of high-quality scientific evidence. Although scientific thinking and Enlightenment values are embattled, and there is a lot of misinformation out there, I still think scientific evidence matters a lot to a lot of people, including experts, policymakers, and the general public. The world is open to being persuaded. But you have to “play ball”.  

(Related post here.)

Note: I got tricked by deceptive SEO into thinking that a paid peer review service that used an academic publisher's name a lot was run by that academic publisher. Google Gemini Pro also lied to me during my search for such services and told me the academic publisher and the paid peer review service were one and the same. But it's totally my fault for not catching this. I apologize for the error. Clara Torres Latorre caught this mistake in this comment and I updated this post with this correction at 22:04 UTC on Wednesday, May 27, 2026.

Update (17:47 UTC on 2026-06-03): For a more detailed and rigorous argument about why EA organizations should engage with the peer review process, please read the philosopher David Thorstad's Reflective Altruism post on the subject.

Clara Torres Latorre 🔸 @ 2026-05-27T07:19 (+27)

I agree with the general point of the post.

But I disagree with specifically using Wiley's services to obtain peer review [1].

I would be excited for people in EA sending their "research" pieces to mainstream academic journals more often.

[1] I'm a math researcher in academia. When I peer review an article, I'm usually granted 60 to 90 (sometimes more) days, depending on length. Adding admin, sometimes multiple times of revision, the time that editors need to find a suitable reviewer, missed deadlines... the whole process usually amounts to at least 6 months, and sometimes it takes years.

I don't think 10 days is a reasonable timeline for peer review, and I find it even short to find a relevant and willing reviewer.

Yarrow Bouchard 🔸 @ 2026-05-27T15:47 (+2)

Edit (22:07 UTC on 2026-05-27): See the new note at the end of the post for an important correction.

So, your skepticism comes from the 10-day turnaround time? If it were 60 days or 90 days instead, you wouldn’t feel skeptical?

I wonder how/why they are able to offer such fast turnarounds and whether it’s by sacrificing quality. Do you think if you got paid, say, $150-250 per review you’d make time to do them faster? Or would it just be impossible regardless?

There are a number of other services similar to Wiley’s. I don’t know if any of them are any good.

Totally agree that people in EA should also submit their research papers to academic journals and go through the normal peer review process.

Clara Torres Latorre 🔸 @ 2026-05-27T16:45 (+4)

Take this with a huge grain of salt, because thing vary enormously field to field.

10 days turnaround sounds too good to be true to me. If it was 6 months I would maybe give it a try for value of information.

But there's more:

Price seems cheap: Last paper I reviewed took me 10-15 hours, so 300-500 + tax would be a reasonable price range and 150-250 sounds meagre. And you need to factor in admin, infra and costs that are not just paying the reviewer.

In my field, peer review is "pro bono" but done during the "working hours" of people with public salaries mostly. And there's the understanding that since we publish and someone else reviews, we also should review some. However, that means that the availability to review varies a lot depending on teaching, research, admin, etc, and we usually fit it in with low priority.

That means if you want someone to review a paper, even if they only need 2 days, the people that have the expertise usually have work to do and might not be willing to give up on next weekend.

About "what if I got paid", it's complicated, bc it depends on if I really have the time (meaning, no plans on the only weekend in the 10 day window, and willing to work), but probably not.

And there's also an ugh factor about getting paid for something that we usually do for free / as part of a salaried job. I'm not decided on it.

dan.pandori @ 2026-05-27T19:51 (+3)

I think you're selling yourself short at 300-500 USD. Gemini estimates 1600-4200 USD (for 3 reviewers total), Opus 400-1000 USD (for a single reviewer spending only 4-6 hours). I endorse those estimates.

Prompt for those curious: If academic peer reviews were compensated at market rate (ie, relative to industry pay for someone with the relevant expertise), how much would it cost to have a typical academic paper reviewed?

Clara Torres Latorre 🔸 @ 2026-05-27T20:04 (+2)

I computed the time that it takes me * my salary, approx.

Ofc if I did this as a freelancer I would charge more.

Yarrow Bouchard 🔸 @ 2026-05-27T17:03 (+2)

Hm, interesting! Thanks for weighing in!

My wild guess about the turnaround time is that they just have so many reviewers “on call” that even if most are unavailable within the 10-day window, at least some people will be available.

The price does seem kind of low. I wonder if the actual average price ends up being more than the list price? E.g., if drafts are above 5,000 words?

I do wonder if the price and turnaround time is too good to be true.

Clara Torres Latorre 🔸 @ 2026-05-27T20:10 (+16)

By the way, I clicked the link (finally), and:

I wouldn't even give it a chance, this aren't a couple red flags, we talking November 1917 situation here.

Which makes me sad, because I really like the broader point of engaging with mainstream academia and playing ball, and I've been nerdsniped by a discussion about peer review and feel that I'm derailing the comment section.

Yarrow Bouchard 🔸 @ 2026-05-27T21:57 (+2)

Hold on, you're right! They say "Wiley" a lot, but they aren't actually affiliated with Wiley! I think the "Wiley" thing was just an SEO trick! Okay, well now this company definitely seems sketchy, and I wouldn't trust them!

I was looking at this at the same time I was looking at Springer Nature's scientific editing service — which is affiliated with Springer Nature, but it's just editing, not peer review — and ended up thinking it was a similar service. (Google Gemini Pro lied to me/fell for Meritpeer's SEO and told me Metritpeer was Wiley, but it's totally my fault for not fact checking this better when I clicked through to Meritpeer's website.) I'm going to edit my post.

By the way, you're not derailing at all, this is an extremely important and helpful contribution!

The general idea of paying for external expert review or peer review still makes sense, but it would require more doing on the part of EA organizations to make it happen if it's not an off-the-shelf service. Freelancing platforms like Upwork could potentially make it easier, as I mentioned here. I say potentially because I don't know if you could reliably find good peer reviewers on Upwork.

NickLaing @ 2026-05-27T08:45 (+14)

I think it would be an interesting exercise to send a handful of articles to these services and see the quality of the feedback. I would also doubt it would be very good quality but would reserve judgement before trying!

Yarrow Bouchard 🔸 @ 2026-05-27T16:09 (+2)

Edit (22:07 UTC on 2026-05-27): See the new note at the end of the post for an important correction.

Where does your doubt come from? Do you doubt that peer review in general is good quality? Or does this service seem too cheap or too fast to be any good?

There’s also the EA organization called The Unjournal, which commissions reviews of EA research from external experts. But I don’t know if this is a better option than Wiley’s service.

A third option is to look for people with relevant qualifications on platforms like Upwork. Here’s a recent freelance job posted on Upwork:

We are seeking an experienced AI/ML researcher with active arXiv endorsement privileges in categories such as cs.AI, cs.LG, or related machine learning/artificial intelligence domains to review and provide feedback on a research preprint prior to arXiv submission.

Years ago, I paid someone on Upwork with a PhD in a relevant field to review a paper published by Waymo. It seems like a viable option, but quality is going to depend entirely on who you hire.

And of course option #4 is to submit papers to peer-reviewed journals.

David Thorstad @ 2026-05-28T22:04 (+9)

One thing that is worth thinking about is what the response would be if the peer reviews are negative. Open Philanthropy used to commission quite a number of academic reviews for some of their better-known reports (perhaps they still do this?). Not all of the reviews were enthusiastic, and there wasn't always action taken on those reviews. 

Yarrow Bouchard 🔸 @ 2026-05-29T00:11 (+2)

I didn't know that about Open Philanthropy!

If EA organizations commission academic reviews and ignore them, then, yeah, it's pointless. I guess there has to be some underlying belief that academic feedback is epistemically valuable. Or at least an underlying commitment to move ideas out of the EA echo chamber into wider acceptance by doing research that is persuasive to people outside of EA.

I see two discouraging signs. One, an anti-academic prejudice in EA. (Often along with a belief that EA is intellectually or epistemically superior to academia, and possibly the rest of the world, too.) Two, low patience for attempts to persuade people outside of EA about ideas that are popular within EA but unpopular outside it (e.g., a 50%+ chance of AGI within the next decade).

If people in EA want to switch gears from operating EA as an elite enclave (or conclave) to a movement that can influence the world at a large scale, including the policies of large liberal democracies like the United States, this change will be painful. People will have to learn how to go from having the majority opinion (in EA) to the minority opinion (in the world). From having the power to decide which opinions can and can't be expressed (in EA) to fighting to be heard in contexts where others have that power (in the world). This is as much about emotional regulation as it is about intellectual discipline.

Thanks for the comment, David.

dan.pandori @ 2026-05-27T03:58 (+6)

I would have found this much more persuasive if you'd tried these services yourself and found them valuable. Without that, my median expectation is that they will do a worse job than Claude Opus 4.7.