Should The Unjournal evaluate 1+ research outputs per relevant EA org?

By david_reinstein @ 2024-12-03T23:37 (+21)

Basic question 

Should The Unjournal commit to evaluate at least one piece of research for each ~EA-funded research org in our domain, with or without their permission?

We're looking for your feedback and  considerations. Would this be useful to the community and to funders? Do you see pitfalls?
 

Context

 The Unjournal (unjournal.org) finds and solicits and prioritizes open-access research for impact and commissions it to be publicly evaluated. We focus on economics, quantitative social science, and impact assessment. See our output at unjournal.pubpub.org

For prominent/influential work, from senior authors, we ask for engagement, not permission (see our 'direct evaluation stream'). If the authors are ~junior/not prominent/less influential we ask and require their permission before we evaluate this work. 

We've mostly evaluated academic-oriented work, but we also have an Applied and Policy stream.  

We'd like to do more in the latter stream, engage with more EA-aligned research orgs, and add more value to the EA community.  

We're considering the following approach: 

1. Identify organizations linked/adjacent to EA &/or focusing on impact, that have been granted over ~$300k to do research (especially from Open Phil, EA Funds, SFF, etc. )

2. Ask each organization to identify one piece of their research they would like to be evaluated (probably in our ~applied stream). For larger multi-division orgs, we might ask for one piece of research per major division (~per $300k)

We would otherwise evaluate the research that seems to be most heavily promoted/influential/driving funding.’ We could engage with the funding orgs/donors and this community to identify the best candidate research We'll allow some exceptions especially if the funders suggest this (e.g., for info hazards)

3. Commission these for public evaluations, give the authors/orgs a chance to respond, post evaluations prominently (on EA forum etc.) We will be careful about COI with members of our management team, evaluators, etc., as always.

Justification

The idea: ‘if you are getting funding to do impactful research you “should” be having some of it publicly evaluated ... unless there are strong reasons not to, such as info hazards’.

Why?  To help funders and practitioners know if and how to use the research, to provide building blocks for further research, to open productive communication, and for due diligence/quality control, so funders and donors can judge the value of the work.