Lotteries for everything?
By FJehn @ 2020-11-27T09:40 (+22)
From time to time, I come across the idea of using lotteries for all kinds of purposes, because they are an easy way to avoid bias. It seems that often the skills to get into a position are not the same as the skills needed to do well in that position and humans are bad at noticing this difference. For example, political campaigning probably needs very different skills than holding a political office (populist parties come to mind). In addition, it can easily increase diversity, as random chance cannot discriminate against people. Cases where lotteries seem to work well include:
- Democracy in Practice: A NGO in Bolivia that introduced a lottery for student government. Every student that would want to run for office is added into the pool of candidates. From that pool, all members of the student government are drawn randomly. This led to a higher diversity in the student governments and they tended to tackle projects that are more ambitious. Many students that would be really good at working in a student government might be too shy to do campaigning. Using lotteries gives them the chance to participate.
- Research grants: More and more research agencies realize that the peer review of research grants is flawed. While peer review seems to work reasonably well to detect bad research, it has problems in ranking good research and tends to err on the side of caution. This makes it hard for new ideas to be funded, even when they later turn out be brilliant (e.g., later Nobel Prize winners getting their grant rejected). To tackle this problem a modified lottery seems to work well. This means that there is a stage of peer review that rejects all proposals that have obvious flaws and all remaining proposals are added to the lottery.
- Citizens’ Assembly in Ireland: To enable a better citizen participation this assembly was established in 2016. It consists of 100 people and 66 of those were randomly selected citizens. Its purpose is to give recommendations on matters specified by the government. The assembly is provided with expert presentation and then discusses the topic in depth over several sessions and creates a report. Overall, the recommendations by the assembly were well received and it is seen as a good example to allow more direct participations of citizens without the dangers of populism.
My questions regarding this are:
- Does anyone know of some good objections to those approaches and to the use of lotteries overall?
- Are there EA organizations that use lotteries for research grants or even for things like applications? If so, what are their experiences with this approach?
- If there are no EA organizations that use lotteries: Why are lotteries not used more? Especially when it comes to grants, it seems like an approach that is easy to test, reduces work and might even give better results.
Atarust @ 2020-11-27T15:45 (+7)
If I owe someone a small amount of money, I sometimes suggest tossing a coin: With a 50% chance we just both forget about the debt, otherwise, we double the debt. The expected money stays the same, the variance is insignificant as the amount is small and with a 50% chance, we do not need to make the hassle of exchanging coins. However, it seems that a lot of people are unwilling to do money betting. Even with amounts less than 10 Euro.
Another possible advantage of lotteries: You simply need to put in less work in decision-making. For example in the donor lottery: https://app.effectivealtruism.org/lotteries
Max_Daniel @ 2020-12-05T00:53 (+6)
I looked into using lotteries for funding a few years ago. I didn't really come to a strong conclusion either way, but some of the background considerations described in the post might still be interesting.
FJehn @ 2020-12-06T21:37 (+3)
That's a great resource. Thanks!
gavintaylor @ 2020-11-27T15:35 (+5)
I'm generally in favour of experimenting with different granting models and am glad to hear that funders are starting to experiment with random allocation. However, I'd be a little bit cautious about moving to a system based solely on random grant assignment. Depending on the actual grant success rate per round (currently often <20%), it seems likely that one would get awarded grants quite infrequently, which would interrupt the continuity of research. For instance, if somebody gets a random grant and makes an interesting discovery, it seems silly to then expect to wait several years for another random grant assignment to follow up on it. So I feel that random assignment is probably better used for assigning funding for early-career researchers or pilot projects.
With respect to quality control, the Nature news article linked above notes:
assessment panels spend most of their time sorting out the specific order in which to place mid-ranking ideas. Low- and high-quality applications are easy to rank, she says. “But most applications are in the midfield, which is very big.”
The current modified lottery systems just remove the low-ranking applications, but if it's easy to pick high-ranking applications, surely they should be given funding priority?
FJehn @ 2020-11-27T18:03 (+2)
Its true that this is probably most suited to a funding scheme aimed at early researchers due to the limitations mentioned by you. However, I might think that the grant success might go up if you use a model were you sort out all bad research first, because your 20 % is probably relate to the overall number of applications. Or maybe you could give people more tickets in the lottery if they have proven they can produce good research. However, this might introduce new biases.
In addition, it might still be a good approach for intermediate researchers because the overall time for the whole grant process gets reduced dramatically if you can cut out most of the peer review, which might lead to more calls for research proposals.
Concerning the Nature article and the modified lottery system: I read conflicting opinions on this. While the Nature article states that very good research can be identified easily, there are also others that state that researchers can only reliably identify bad research, but have a hard time to sort good research in any reproducible way.
peter_janicki @ 2020-12-03T18:26 (+2)
this page/link below is in german, but we know, there are some german-speaking ea´s. it´s an article in an economic newspaper and it´s about the benefit of using lotteries for choosing supervisors/superiors and there seem to be quiet some benefits in some cases. (google translate does a sufficient job here). https://www.wiwo.de/erfolg/management/aleatorische-verfahren-befoerderung-per-zufall-wir-wuerfeln-einen-chef/26621916.html
my guess is: whether lotteries are great depends on the sample. if it´s tricky to make a good decision -> lotteries may be great/ better (than a complex decision scheme). if it´s easy to make a good decision: lotteries sure are a bad thing.
example: as long as high skilled workers compete for a promotion: lotteries may be a good way to decide whom to promote. but after some years/ decades of lotteries workers may realize that there is no need to excel at ones job, because promotion does not depend on it. or even low-skilled workers may aplly for a promotion. same might be true for apllying for research grants, etc.
Thanks for writing this!
FCCC @ 2020-12-04T06:11 (+1)
When designing a system, you give it certain goals to satisfy. A good example of this done well is voting theory. People come up with apparently desirable properties, such the Smith criterion, and then demonstrate mathematically that certain voting methods succeed or fail the criterion. Some desirable goals cannot be achieved simultaneously (an example of this is Arrow's impossiblility theorem).
Lotteries give every ticket has an equal chance. And if each person has one ticket, this implies each person has an equal chance. But this goal is in conflict with more important goals. I would guess that lotteries are almost never the best mechanism. Where they improve situations is for already bad mechanisms. But in that case, I'd look further for even better systems.
FJehn @ 2020-12-04T16:33 (+1)
I think the problem that it is really, really hard to come up with better systems. As mentioned above research grants have quite a few problems. Those problems are founded in human bias and a lack of knowledge. I cannot really evaluate the value of a grant if I have not seen all the other grants and I might be influenced by my biases so give it to a scientist I like or trust. In addition, if there would be an easy and obvious system people would probably already have implemented it.
So, lotteries solve this problem. There might be better approaches, but many of them probably need an allknowing and unbiased arbiter and I have the impression that we lack those.
Basically it boils down to the question: Am I better in evaluting this than chance? And I think people often are not due to their unconcious bias and knowledge gaps.
FCCC @ 2020-12-06T23:41 (+1)
There are mechanisms that aggregate distributed knowledge, such as free-market pricing.
I cannot really evaluate the value of a grant if I have not seen all the other grants.
Not with 100 percent accuracy, but that's not the right question. We want to know whether it can be done better than chance. Someone can lack knowledge and be biased and still reliably do better than random (try playing chess against a computer that plays uniformly random moves).
In addition, if there would be an easy and obvious system people would probably already have implemented it.
Wouldn't the "efficient-policy hypothesis" imply that lotteries are worse than the existing systems? I don't think you really believe this. Are our systems better than most hypothetical systems? Usually, but this doesn't mean there's no low-hanging fruit. There's plenty of good policy ideas that are well-known and haven't been implemented, such as 100 percent land-value taxes.
Let's take a subset of the research funding problem: How can we decide what to fund for research about prisoner rehabilitation? I've suggested a mechanism that would do this.