Revisiting the karma system

I don't fully understand what you're saying, but my guess is that you're suggesting we should take a user's total karma and divide that by votes. I don't understand what it means to "give weight to this"—does the resulting calculation become their strong vote power?  I am not being arch, I literally fully don't understand, like I'm dumb.

I know someone who has some data and studied the forum voting realizations and weak/strong upvotes. They are are totally not a nerd, I swear!

Thoughts:

A proximate issue with the idea I think you are proposing is that currently, voting patterns and concentration of voting or strong upvotes differs in a systematic way by the "class of post/comment":

• There is a class of "workaday" comments/posts that no one finds problematic and gets just regular upvotes on.
• Then there is a class of "I'm fighting the War in Heaven for the Lisan al Gaib" comments/posts that gets a large amount of strong upvotes. In my opinion, the karma gains here aren't by merit or content.

I had an idea to filter for this (that is sort of exactly the opposite of yours) to downweight the karma of these comments by their environment, to get a "true measure" of the content. Also, the War in Heaven posts have a sort of frenzy to them. It's not impossible that giving everyone a 2x - 10x multiplier on their karma might contribute to this frenzy, so moderating this algorithmically seems good.

A deeper issue is that people will be much more risk averse in a system like this that awards them for their average, not total karma. 

In my opinion, people are already too risk averse, in a way that prevents confrontation, but at least that leads to a lot of generally positive comments/posts, which is a good thing. 

Now, this sort of gives a new incentive, to aim for zingers or home runs. This seems pretty bad. I actually don't think it is that dangerous/powerful in terms of actual karma gain, because as you mentioned, this can be moderated algorithmically. I think it's more a problem that this can lead to changes in perception or culture shifts.

Maybe this underweights many people appreciating a given comment or post and regular upvoting it, but few of them appreciating it enough to strong upvote it, relative to a post or comment getting both a few strong upvotes and at most a few regular votes?

I agree that a multi-dimensional voting system would have some advantages.

I think that a fairly large fraction of posts is of a generalist nature. Also, my guess is that people with a large voting power usually don't vote on topics they don't know (though no doubt there are exceptions).

I'd welcome topic-specific karma in principle, but I'm unsure how hard it is to implement/how much of a priority it is. And whether karma is topic-specific or not, I think that large differences in voting power increase accuracy and quality.

I'm not saying that there's a perfect correlation between levels of karma and quality of opinion. The second paragraph of my comment points to quantity of comments/posts being a distorting factor; and there are no doubt others.

Ok, this comment is ideological (but really interesting). 

This comment is pushing on a lot of things, please don't take any of this personally .

(or take it personally if you're one of the top people listed here and fighting me!)

So below is the actual list of top karma users. 

BIG CAVEATS:

• I am happy to ruthlessly, mercilessly attack, the top ranked person there, on substantial issues like on their vision of the forum. Like, karma means nothing in the real world (but for onlookers, my choice is wildly not career optimal^[1]).
• Some people listed are sort of dumb-dumbs.
• This list is missing the vast majority (95%) of the talented EAs and people just contributing in a public way, much less those who don't post or blog.
• The ranking and composition could be drastically improved

But, yes, contra you, I would say that this list of people do have better and reasonable views of reality than the average person and probably average EA. 

More generally, this is positively correlated with karma.

Secondly, importantly, we don't need EAs on the forum to have the best "view of reality". 

We need EAs to have the best views of generating impact, such as creating good meta systems, avoiding traps, driving good culture, allocating funding correctly, attracting real outside talent, and appointing EAs and others to be impactful in reality. 

1. ^{^}

   Another issue is that there isn't really a good way to resolve "beef" between EAs right now.

   Funders and other senior EAs, in the most prosocial, prudent way, are wary of this bad behavior and consequent effects (lock-in). So it's really not career optimal to just randomly fight and be disagreeable.

I'm not quite sure I follow your reasoning. I explicitly say in the second paragraph of my comment that "right now sheer quantity is awarded more than ideal".

Ya, I agree. I think the only things I strong downvote are things worth reporting as norm-breaking, like spam or hostile/abusive comments.

We could also just weaken (strong) downvotes relative to upvotes.

Ah, I wasn't aware of the All-Posts page. That's helpful, but I'd wonder if it's also being used as much on the EA Forum. I'm probably relatively active on the EA Forum, and I only check Pinned Posts and Frontpage Posts, and I sometimes hit "Load more" for the latter.

Note also that one strong downvote might not put a post into negative karma, but just low positive karma, if the submitter has enough karma themself or others have upvoted the post.

It's not clear people would even know it's an abuse of the system to strong downvote a post early on, rather than wait.

I don't have access to data on this, but a few related anecdotes, although not directly instances of it.

This post itself was starting to pick up upvotes and then I think either received a bunch of downvotes or a strong downvote (or people removed upvotes), so I switched my regular upvote into a strong one. I think it went from 20-25 karma to 11-16. I'd guess this wouldn't be enough to hide it from the front page, though, but if it had been hidden early on, I might not have made my comment starting this thread.

I've seen a few suffering-focused ethics-related comments get strong downvoted by users with substantial karma, in particular a few of my own comments on my shortform (some that I had shared on the 80,000 Hours website, and another in an SFE/NU FB group) and a comment about utilitarianism.net omitting discussion of negative utilitarianism. I'm not aware of this happening to posts, but I wouldn't be surprised if such users would do the same to posts, including early on. These might have been 2 years ago by now, though.

I think there are some posts that should be made invisible; and that it's good if strong downvotes make them so. Thus, I would like empirical evidence that such a reform would do more good than harm. My hunch is that it wouldn't.

Or, instead of normalizing, just show the top N or p% of posts of each few months?

Relative to average of ~the month. It's not based on standard deviation, just karma/(avg karma).

I would imagine the forum has a record of the total voting power of all users, which is increasing over time, and the karma can be downscaled by this total.

It's just the posted-at date of the post, and makes no attempt to adjust for when the karma was given. So if something has stayed a classic for much longer than its peers, it will rank highly on this metric.

It collapses at -5, not just negative -1.

I didn’t downvote your comments and I think for most people, it’s bad when they go negative, like it’s a bad experience.

To be fair, the specific comment (which I didn’t downvote) says you think it’s a personal norm to downvote ideas you think are harmful (which is a subjective and complex judgement)…and they downvoted it.

This feels reasonable to me. Personally, I very rarely strong-upvote (except the default strong-upvote for my own posts), and almost never strong-downvote unless it's clear spam. If there's a clearer "use it or lose it" policy, I think I'd be more inclined to ration out more strong-upvotes and strong-downvotes for favorite/least favorite (in terms of usefulness) post or comment that week.

a few high-karma users are much more intense about it and don't hesitate to throw their weight around. These users can then have a wildly disproportionate effect on discourse because if their vote is worth, say, 7 points, their opinion on one piece of content vs. another can be and often is worth a full 14 points.

It would be interesting to get peoples guesses/hypothesis of this specific behavior,  then see how often it actually occurs.

Personally, my guess is that EA forum accounts with large karma don't often do this behavior "negatively" (they rarely strong downvote and and negatively comment). When this does happen, I expect to find this to be positive and reasonable.

In case anyone is interested, my activity referred to above, begun almost half a year ago, provides data to identify instances of the behavior mentioned by IanDavidMoss, and other potentially interesting temporal patterns of voting and commenting. There are several other interesting things that can examined too.

I would be willing to share this data, as well as provide various kinds of technical assistance to people working on any principled technical project^[1] related to the forum or relevant data.

1. ^{^}

   I do not currently expect to personally work on an EA associated project related to the forum or accept EA funding to do so.

The post, as of 2:33 pm PST has:

• 8 big upvotes
• 7 small upvotes 
• 1 "cancel" (was a vote but cancelled)
• 1 small downvote
• 1 big downvote

The above might not be perfect, I am not going to go through the codebase to figure this out.

So can you tell me what conditions you think would be sufficient to judge something as groupthink before I try to satisfy you?

A system or pattern or general belief that leads to a defect or plausible potential defect (even if there is some benefit to it), and even if this defect is abstract or somewhat disagreed upon.

The most clear defect would be something like “We are funding personal projects of the first people who joined EA and these haven’t gotten a review because all his friends shout down criticism on the forum and the forum self selects for devotees. Last week the director has been posting pictures of Bentleys on Instagram with his charity’s logo”.

The most marginal defects would be “meta” and whose consequences are abstract. A pretty tenuous but still acceptable one (I think?) is “we only are getting very intelligent people with high conscientiousness and this isn’t adequate. ”.

Because if that's true then unless you think the probability of EA groupthink is 0, this is going to be an expected downside of the karma system - so the argument should be whether the upsides outweigh the downsides, not whether the downsides exist.

Right, you say this…but seem a little shy to list the downsides.

Also, it seems like you are close to implicating literally any belief?

As we both know, the probably of groupthink isn’t zero. I mentioned I can think of up to 15 instances, and gave one example.

I would expect you and most other forum users to consider them not groupthink - because things we believe are true don't qualify.

My current read is that this seems a little ideological to me and relies on sharp views of the world.

I’m worried what you will end up saying is not only that EAs must examine themselves with useful and sharp criticism that covers a wide range of issues, but all mechanical ways where prior beliefs are maintained must be removed, even without any specific or likely issue?

One pragmatic and key issue is that you might have highly divergent and low valuations of the benefits of these systems. For example, there is a general sentiment worrying about a kind of EA “Eternal September” and your vision of karma reforms are exactly the opposite of most solutions to this (and well, have no real chance of taking place).

Another issue are systemic effects. Karma and voting is unlikely to be the root issue of any defects in EA (and IMO not even close). However, we might think it affects “systems of discussion” in pernicious ways as you mention. Yet, since it’s not central or the root reason, deleting the current karma system without a clear reason or tight theory might lead to a reaction that is the opposite of what you intend (I think this is likely), so it’s blind and counterproductive.

The deepest and biggest issue of all is that many ideological views that involve disruption are hollow and themselves just expressions of futile dogma, e.g cultural revolution with red books, or a tech startup with a narrative of disrupting the world but simply deploying historically large amounts of investor money.

Wrote this on my phone, there might be bad spelling or grammar issues but if the comment has the dumbs it’s on me.

Arepo @ 2022-05-30T18:48 (+11)

Fwiw I didn't downvote this comment, though I would guess the downvotes were based on the somewhat personal remarks/rhetoric. I'm also finding it hard to parse some of what you say. 

A system or pattern or general belief that leads to a defect or plausible potential defect (even if there is some benefit to it), and even if this defect is abstract or somewhat disagreed upon.

This still leaves a lot of room for subjective interpretation, but in the interests of good faith, I'll give what I believe is a fairly clear example from my own recent investigations: it seems that somewhere between 20-80% of the EA community believe that the orthogonality thesis shows that AI is extremely likely to wipe us all out. This is based on a drastic misreading of an often-cited 10-year old paper, which is available publicly for any EA to check.

Another odd belief, albeit one which seems more muddled than mistaken is the role of neglectedness in 'ITN' reasoning. What we ultimately care about is the amount of good done per resource unit, ie, roughly, <importance>*<tractability>. Neglectedness is just a heuristic for estimating tractability absent more precise methods. Perhaps it's a heuristic with interesting mathematical properties, but it's not a separate factor, as it's often presented. For example, in 80k's new climate change profile, they cite 'not neglected' as one of the two main arguments against working on it. I find this quite disappointing - all it gives us is a weak a priori probabilistic inference which is totally insensitive to the type of things the money has been spent on and the scale of the problem, which seems much less than we could learn about tractability by looking directly at the best opportunities to contribute to the field, as Founders Pledge did.

Also, it seems like you are close to implicating literally any belief?

I don't know why you conclude this. I specified 'belief shared widely among EAs and not among intelligent people in general'. That is a very small subset of beliefs, albeit a fairly large subset of EA ones. And I do think we should be very cautious about a karma system that biases towards promoting those views.

Charles He @ 2022-05-30T19:33 (+2)

I don't know why you conclude this. I specified 'belief shared widely among EAs and not among intelligent people in general'. That is a very small subset of beliefs, albeit a fairly large subset of EA ones. And I do think we should be very cautious about a karma system that biases towards promoting those views.

You are right. My mindset writing this comment was bad, but I remember thinking the reply seemed not specific and general, and I reacted harshly, this was unnecessary and wrong. 

This still leaves a lot of room for subjective interpretation, but in the interests of good faith, I'll give what I believe is a fairly clear example from my own recent investigations: it seems that somewhere between 20-80% of the EA community believe that the orthogonality thesis shows that AI is extremely likely to wipe us all out. This is based on a drastic misreading of an often-cited 10-year old paper, which is available publicly for any EA to check.

I do not know the details of the orthogonality thesis and can't speak to this very specific claim (but this is not at all refuting you, I am just literally clueless and can't comment on something I don't understand).

 

To both say the truth and be agreeable, it's clear that the beliefs in AI safety are from EAs following the opinions of a group of experts. This just comes from people's outright statements. 

In reality, those experts are not at the majority of AI people and it's unclear exactly how EA would update or change its mind.

 

Furthermore, I see things like the below, that, without further context, could be wild violations of "epistemic norms", or just common sense. 

For background, I believe this person is interviewing or speaking to researchers in AI, some of whom are world experts. Below is how they seem to represent their processes and mindset when communicating with these experts.

One of my models about community-building in general is that there’s many types of people, some who will be markedly more sympathetic to AI safety arguments than others, and saying the same things that would convince an EA to someone whose values don’t align will not be fruitful. A second model is that older people who are established in their careers will have more formalized world models and will be more resistance to change. This means that changing one’s mind requires much more of a dialogue and integration of ideas into a world model than with younger people. The thing I want to say overall: I think changing minds takes more careful, individual-focused or individual-type-focused effort than would be expected initially.

I think one’s attitude as an interviewer matters a lot for outcomes. Like in therapy, which is also about changing beliefs and behaviors, I think the relationship between the two people substantially influences openness to discussion, separate from the persuasiveness of the arguments. I also suspect interviewers might have to be decently “in-group” to have these conversations with interviewees. However, I expect that that in-group-ness could take many forms: college students working under a professor in their school (I hear this works for the AltProtein space), graduate students (faculty frequently do report their research being guided by their graduate students) or colleagues. In any case, I think the following probably helped my case as an interviewer: I typically come across as noticeably friendly (also AFAB), decently-versed in AI and safety arguments, and with status markers. (Though this was not a university-associated project, I’m a postdoc at Stanford who did some AI work at UC Berkeley).

The person who wrote the above is concerned about image, PR and things like initial conditions, and this is entirely justified, reasonable and prudent for any EA intervention or belief. Also, the person who wrote the above is conscientious, intellectually modest, and highly thoughtful, altruistic and principled.

 

However, at the same time, at least from their writing above, their entire attitude seems to be based on conversion—yet, their conversations is not with students or laypeople like important public figures, but the actual experts in AI. 

So if you're speaking with the experts in AI and adopting this attitude that they are preconverts and you are focused on working around their beliefs, it seems like, in some reads of this, would be that you are cutting off criticism and outside thought. In this ungenerous view, it's a further red flag that you have to be so careful—that's an issue in itself. 

 

For context, in any intervention, getting the opinion or updating from experts is sort of the whole game (maybe once you're at "GiveWell levels" and working with dozens of experts it's different, but even then I'm not sure—EA has updated heavily on cultured meat from almost a single expert). 

Pablo @ 2022-05-30T19:06 (+4)

My apologies: I now see that earlier today I accidentally strongly downvoted your comment. What happened was that I followed the wrong Vimium link: I wanted to select the permalink  ("k") but selected the downvote arrow instead ("o"), and then my attempt to undo this mistake resulted in an unintended strong downvote. Not sure why I didn't notice this originally.

---

Charles He @ 2022-05-30T19:07 (+1)

Ha.

Honestly, I was being an ass in the comment and I was updating based on your downvote. 

Now I'm not sure anymore! 

Bayes is hard.

 

(Note that I cancelled the upvote on this comment so it doesn't show up in the "newsfeed")