Polis (Cluster) Poll: What are our opinions on EA, governance and FTX? (Now with visualisation)

By Nathan Young @ 2022-11-19T11:52 (+71)

Tl;dr

Link here: https://pol.is/6rk9hcav3w

Report here (this now works): https://pol.is/report/r7yrwpymn6yadbzwm3iyi

Longer intro

It is easy to have a false sense of what people think. But we don't have to - we have tools that allow us to hear what people actually think. One is representative polling. This isn't that. Here we can do cluster analysis (not a technical term) and look at what opinions are held in common. Do those who think Longtermisms is true tend to think that certain solutions are better than others? If so, why.

It shouldn't be used to say  X% of EAs think this. That's not what this tool is for. Instead, it should suggest interesting questions to ask in future.

It would be very easy to use the loudest voices to decided what EAs really think. But that is unwise. Instead, if we want community input we should try and understand what groups form the community as a whole and understand what underlies the desires of each group.

Example question

Report

https://pol.is/report/r7yrwpymn6yadbzwm3iyi

Clusters and visualisation (Coming when Polis decides to work)

There are three clusters currently:

The majority views:


Dancer @ 2022-11-19T12:10 (+18)

In case the results end up getting used to argue "EA thinks X", I want to add a couple of caveats regarding "our opinions":

1. This note from Linch on another post:

Keep in mind that the people who are commenting, writing call-out posts, etc., online are both a) only a small fraction of EAs, and b) a particularly skewed sample in the current moment.

Some of us either chose or were asked to be quiet for now, or are still reeling from our losses and not sure what to say.

2. I think it will be much more useful to focus on cold takes, when people are ready to share them over the coming weeks/months.

Nathan Young @ 2022-11-19T12:25 (+10)

Polis is a better tool for understanding opinions than most other discussions we've had here.

Polis isn't about size of opinions, but breadth and correlation. You don't need respondents to be representative to get a sense of what different parts of the community thinks here.

I agree with the cold takes point, there is a comment like that in the poll.

Dancer @ 2022-11-19T12:27 (+1)

Oh interesting, nice. Thanks for adding the 'Longer intro' section.

Nathan Young @ 2022-11-19T12:32 (+2)

I added that while you were commenting, but have updated it since your comments. Great minds think alike etc etc.

PeterSlattery @ 2022-11-19T18:54 (+3)

This is a good point. I am interested to hear what you think of the idea of having an EA research participant panel that we can recruit from to understand consensus within community debates?

Dancer @ 2022-11-19T19:52 (+1)

Thanks. I'm not entirely sure what you're proposing but my guess is that we have better things to do with our time. Sorry for the unsophisticated answer!

RobertJones @ 2022-11-20T09:50 (+14)

On the object level, there now seem to be two roughly equal clusters.  Cluster A are long-termist, consider EA epistemically humble,[1] don't think we should blame ourselves for the FTX farrago and are unenthusiastic about democratisation.  Cluster B take the opposite positions.

On the meta level:

  1. ^

    The most relevant statement here reads, "I think EA is epistemically humble but external people don't seem to think so."  It seems likely to me, based on other responses, that the disagreement stems from the first part, but of course a person who believed EA to be universally acknowledged as epistemically humble would also disagree.  In retrospect, it would have been better to have "EA is epistemically humble" and "External people don't consider EA to be epistemically humble" as separate statements.

  2. ^

    For some reason "EA should not aim to be a community" is also appearing here although only 55% of respondents disagree.

  3. ^

    Of course cluster B does have at least one concrete proposal, namely a community-elected board.  It's just that cluster A thinks that would be bad.

Guy Raveh @ 2022-11-20T11:11 (+4)

It looks to me like the clustering is basically ok, but the presentation breaks down when there are so many statements. Better to just look at the full list.

Joel Becker @ 2022-11-19T15:58 (+10)

Would like to see more of this!

AllAmericanBreakfast @ 2022-11-19T14:48 (+10)

I’m not convinced that the opinions that will be aggregated by this poll will constitute a representative sample of the EA movement.

Nathan Young @ 2022-11-19T15:19 (+8)

It shouldn't be used to say  X% of EAs think this. That's not what this tool is for. Instead, it should suggest interesting questions to ask in future.

I am confident they won't, nor is that what this is. 

AllAmericanBreakfast @ 2022-11-19T18:41 (+5)

Fair enough :) I still think it's worth amplifying this point.

Nathan Young @ 2022-11-19T21:36 (+2)

Thanks for saying it. 

Guy Raveh @ 2022-11-19T18:15 (+9)

There are many statements there not particularly related to FTX (some of which I added myself). Even if the sample is not representative, this grouping does feel like a real division that I also see in EA circles in real life.

[Edit: when this was written there were two groups, now there are three. Still, I maintain that there is some meaningful division here.]

Sanjay @ 2022-11-19T12:32 (+9)

I ended up answering "pass/unsure" to almost everything.

Lots of the time, it was because I thought the statement was ambiguous.

There was a space at the bottom where I could add a comment, but I wasn't clear on whether the comment I would add would be attached to the statement I was voting on, or whether it would create a new standalone statement

Nathan Young @ 2022-11-19T12:56 (+2)

Also expressing uncertainty is good. That’s a feature here.

Jason @ 2022-11-19T14:35 (+7)

It felt frustrating to me (and will likely complicate the findings) that "pass" had about five meanings (e.g., no opinion, too early to commit to an opinion, not applicable to me, too ambiguous, feelings too mixed for a response) and my pass response could be misinterpreted as having a meaning I didn't intend. That's just an inherent downside to the tool design, though.

Nathan Young @ 2022-11-19T12:33 (+2)

It would create a new standalone statement.

Nathan Young @ 2022-11-19T21:39 (+8)

This is the largest polis poll by response I've ever done. I think I might start deleting boring answers to make it easier to navigate:

David Varga @ 2022-11-19T22:29 (+3)

Please don't delete answers. This is a fascinating snapshot of what people are thinking and I think there is merit for the poll to remain open and be reviewed in detail. 

I wonder if there is any narrative coming out of the clusters/most agreed/disagreed/abstained questions/statements? 

Nathan Young @ 2022-11-19T22:40 (+2)

Well there are just too many answers now. 

Guy Raveh @ 2022-11-19T23:21 (+4)

When you say "delete answers", do you mean statements or votes?

David Varga @ 2022-11-19T22:35 (+2)

When I sort all statements by ranking, %agree and %disagree seem to rank as expedted - by % descending, but %passed seems to jump all over the place. Is there a reason for this or is this a bug?

Nathan Young @ 2022-11-19T22:39 (+2)

I don't know. I don't really have a gears level understanding of polis.

David Varga @ 2022-11-19T14:52 (+6)

Thanks Nathan, this is an excellent initiative. 

Guy Raveh @ 2022-11-19T23:19 (+3)

There are already 165 statements, more and more of them meta statements about the poll. Nathan, you created a monster.

(Please don't take this as serious criticism)

Jason @ 2022-11-19T18:50 (+3)

I added some statements intended to identify, in a general manner, the respondent's role and level of engagement. I think it would be helpful to know if certain clusters were associated with focus on meta level work vs. being in a role like AI research vs. those of us who work in non-EA roles.

PeterSlattery @ 2022-11-19T18:47 (+3)

(on phone) Thanks so much for doing this!I was going to post to request a survey today so really glad to this post. Polis seems cool and I have enjoyed using it.

I'll take this as an opportunity to mention that I think we should normalise using consensus assessments (like something like polis or a more conventional survey) for nearly all community disagreements.

At the moment it's usually impossible to determine what the EA community thinks on many issues. The lack of synthesis and consensus across complex debates makes it hard to update accurately.

I do also see some downside risks for example with the media so we should be careful when we do this, with the questions asked, and with how results are publicised.

PeterSlattery @ 2022-11-19T18:51 (+2)

Relatedly, I would like EA to have a participant panel of EA community members who can be hired for a representative sample.we would survey this group when we want to know what the community thinks and not just rely on assessing those who have the time and confidence to visit/comment/vote on the forum etc. I'd love to see that happen asap because it would be very useful at a time like now.

Jason @ 2022-11-20T15:24 (+2)

Thanks for running this, Nathan!

A few general comments for future runs:

People might benefit from more clarity around what makes a good statement to submit: e.g., respondents are expected to have different opinions/responses and those differences might be expected to cluster with existing statements in ways that would be helpful to understand.

Although I know Polis isnt intended as a crowdsourced poll per se, I wonder if there is value to running an "open" polis every quarter for that purpose. I got the sense that some community members saw a chance to anonymously get a sense of community sentiment on unrelated topics that were of concern to them (e.g., flirting at EAG). Obviously, there would have to be some norms and limits around this.

I think having a quick way for people to anonymously raise issues and get a measure of possible community sentiment is a good thing. Even understanding the sample bias problems, I think getting shallow feedback still often has enough value to justify five to seven minutes of the community's collective time. The result could be helpful in informing someone that the topic is worth devoting several hours to a detailed forum post, or inclusion in a more formal / widespread poll.

Hadrian @ 2022-11-21T04:54 (+1)

Thanks for this interesting tool.  I'd suggest that you suggest to submitters not to send compound statements in the section "What makes a good statement."  E.g., "It's really bad that EAs don't feel comfortable questioning longtermism. This points to deeper groupthink inside EA." If it's a premise-conclusion statement, you could also make clear that voters should pass unless they accept the premise.

mako yass @ 2022-11-21T04:34 (+1)

I'm excited by the prospect of Polis, but it's frustratingly limited. The system has no notion of whether people are agreeing with a statement because it's convincing or bridging the gap, or because it's banal.

In this case... I don't think we're really undergoing any factionalization about this? In that case, should we not just try talking more... that usually works pretty well with us.

Achim @ 2022-11-21T20:15 (+1)

Talking is a great idea in general, but it seems there are some opinions in this survey suggesting that there are barriers to talking openly?

mako yass @ 2022-11-21T20:49 (+1)

I believe the forum allows commenting anonymously, though I wouldn't know how to access that feature.

Psuedonyms would be a bit better, but it'll do.

Lorenzo Buonanno @ 2022-11-21T21:36 (+4)

As far as I know, the supported way to comment anonymously is to make an anonymous account