Science policy as a possible EA cause area: problems and solutions

By PabloAMC 🔸 @ 2022-01-23T00:32 (+26)

Summary: Creating the right incentive structures in science could make science more fluid, efficient, and painless. Indeed, there seem to be multiple reasons why people complain about how science is done now. In this post, I analyze some of the problems science has, and give some hints at ideas that could improve the situation. The aim of this post is to suggest science policy as a possible research area for EAs where it might be possible to do progress that results in better science.

Introduction

Science is one of the key enablers of progress in our world. Yet it seems to me that there are many ways it could be improved. It seems to me from anecdotal evidence that most scientists are in fact not happy about the incentive structure, procedures, evaluation, or career advancement (https://fivethirtyeight.com/features/science-isnt-broken). This is a complex system and problem, so it seems unlikely that a simple solution exists for all of these problems. Yet, being such an important engine of our society, some effort by EAs to understand it better would be a great use of resources. In the following, I give an overview of the main problems I see in science, and in some cases some ideas of how we could find a solution.

Problems and solutions

Publishing

One of the main problems scientists complain about a lot is the extremely high charges most journals apply to publications (see a discussion in https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science). The key reason why this is possible, is because science is a market of prestige. And it is not possible to grow the total amount of prestige, since the more prestigious other people are, the more diluted your prestige becomes. Editorials channel this natural scarcity via well-known journals that have become some sort of oligopoly, creating extremely profitable businesses.

Due to this high-recognition / high-price scheme, it seems unlikely that one could break the monopoly via the demand side (eg, from article authors). However, I also think that editorials may have committed an unforced error by relying on unpaid and volunteer editors and referees. While being an editor might give you some recognition, becoming a referee is often not taken much into consideration for career promotion purposes. In consequence, a possible solution is some kind of coordinated action by scientists (or universities) to decline being referees for high-fee journals. The reason why I think this has some chance of succeeding is that most scientists find being referees as a boring task you have to get over relatively quickly, as you barely benefit. Indeed, from my experience trying to publish, in quite a few cases the problem with the quality of referee reports appear due to their complete lack of interest in being referees in the first place. To solve this problem, universities could try to enforce by contract with their researchers not to volunteer them to high-fee journals, forcing those journals to lower their fees to access referees. Don’t take this too seriously though, it will probably need a lot of thought before a solution ends up working, if it ever does.

Specialization

One thing that Eliezer Yudkowski requests in his book “Inadequate equilibria” is that scientists have more specialization. In particular, there might be a few different tasks that commonly scientists have to do: a) Research, that is, coming up with good ideas, writing papers… b) Teaching, especially at universities. c) Research evaluation (being a referee). d) Managerial tasks, such as asking for grants. I agree with him that we need to split up work. Some people like, enjoy, and are better at teaching. Others, at doing research. I really don’t think one should be requested to do everything. In addition, dedicated science evaluators might help a lot with replication problems, referee quality, and speed…

One funny aspect is that if the previous section suggestion were to force editors to hire researchers to be editors and referees, they might become more specialized and better at doing it. We might even be able to figure out ways to evaluate the evaluators, something that now is impossible due to the created scarcity of unpaid referees. Professional evaluators also have the advantage of being more capable to enforce norms in sensitive fields such as bioengineering or AI.

Research evaluation

Evaluating research is perhaps one of the most difficult topics. It requires significant expertise, is very unpredictable, and seems to be hard to find reliable systematic solutions. In this topic, I think the AI (or perhaps Computer Science) research community is doing a great job (much better than in other areas) at innovating a lot on different peer review systems. I also think I remember that the rationalist community was thinking of some ways of using prediction markets to assess research quality.

It is really not clear to me what is the best way to do research evaluation, but perhaps someone interested in this problem could start by producing a meta-analysis of the different experiments (peer review systems) carried out and their conclusions, and further experiments that should be done. It seems to me that at the very least we can treat this as a social science problem at a macroscopic level, where the intervention is the publication method and the result is the accuracy of the predicted scientific impact.

Career advancement

It is also a common source of complaint in the scientific community that the research career is very unstable, relatively poorly paid, and extremely competitive. To be honest, this is probably a consequence of too many people enjoying doing science with respect to the number of available research jobs. Indeed, the probabilities of landing a tenured research position in academia seem to have been shrinking for quite some time (https://forum.effectivealtruism.org/posts/3TQTec6FKcMSRBT2T/estimation-of-probabilities-to-get-tenure-track-in-academia). I don’t know what should be done about it, but sometimes it can hurt the quality of the research being published. While I don’t have strong opinions on how the situation could be systematically improved, it is worth analyzing more in-depth.

Conclusion

In summary, I think “fixing science” is a problem from multiple angles. However, to the best of my knowledge, there seems to be a relatively little systematic study on how to improve its situation. For that reason, I tend to think this might be a cause that should receive some attention from the EA community.