Getting a feel for changes of karma and controversy in the EA Forum over time

Thanks for this, pretty interesting analysis.

Every time I come across an old post in the EA forum I wonder if the karma score is low because people did not get any value from it or if people really liked it and it only got a lower score because fewer people were around to upvote it at that time.

The other thing going on here is that the karma system got an overhaul when forum 2.0 launched in late 2018, giving some users 2x voting power and also introducing strong upvotes. Before that, one vote was one karma. I don't remember exactly when the new system came in, but I'd guess this is the cause of the sharp rise on your graph around December 2018. AFAIK, old votes were never re-weighted, which is why if you go back through comments on old posts you'll see a lot of things with e.g. +13 karma and 13 total votes, a pattern I don't recall ever seeing since. 

Partly as a result, most of the karma old posts have will have been from people going back and upvoting them later once the new system was impemented, e.g. from memory my post from your list was around +10 for most of its life, and has drifted to its current +59 over the past couple of years.

This jumps out to me because I'm pretty sure that post was not a particularly high-engagement post even at the time it was written, but it's the second-highest 2015 post on your list. I think this is because it's been linked back to a fair amount and so can partially benefit from the karma inflation.

(None of which is meant to take away from the work you've done here, just providing some possibly-helpful context.)

For what it's worth, I generally downvote a post only when I think "This post should not have been written in the first place", and relatedly I will often upvote posts I disagree with.

If that's typical, then the "controversial" posts you found may be "the most meta-level controversial" rather than "the most object-level controversial", if you know what I mean.

That's still interesting though.

This is really cool, thank you! :-)

One thought: it is much easier now than it used to be to look at highly upvoted posts. In the old forum old popular posts simply fell by the wayside, now you can sort by them. We also now have the favourites section which encourages people to read highly upvoted posts they haven't read yet.

So I think highly popular posts now look more popular than they really are compared to the past even according to your metric.

Also, I'm proud to say I guessed the most well received forum post according to your metric correctly!

I would be interested in how to circumvent this for future analysis.

You can query by year, and then aggregate the years. From a past project, in nodejs:

/* Imports */
import fs from "fs"
import axios from "axios"

/* Utilities */
let print = console.log;
let sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms))

/* Support function */
let graphQLendpoint = 'https://www.forum.effectivealtruism.org/graphql/'
async function fetchEAForumPosts(start, end){
  let response  = await axios(graphQLendpoint, ({
    method: 'POST',
    headers: ({ 'Content-Type': 'application/json' }),
    data: JSON.stringify(({ query: `
       {
        posts(input: {
          terms: {
          after: "${start}"
          before: "${end}"
          }
          enableTotal: true
        }) {
          totalCount
          results{
            pageUrl
            user {
              slug
              karma
            }
            
          }
        }
      }`
})),
  }))
  .then(res => res ? res.data ? res.data.data ? res.data.data.posts ? res.data.data.posts.results : null : null : null : null)
  return response
}

/* Body */
let years = [];
for (var i = 2005; i <= 2021; i++) {
   years.push(i);
}

// Example, getting only 1 year.
let main0 = async () => {
  let data = await fetchEAForumPosts("2005-01-01","2006-01-01")
  console.log(JSON.stringify(data,null,2))
}
//main0()

// Actual body
let main = async () => {
  let results = []
  for(let year of years){
    print(year)
    let firstDayOfYear = `${year}-01-01`
    let firstDayOfNextYear = `${year+1}-01-01`
    let data = await fetchEAForumPosts(firstDayOfYear, firstDayOfNextYear)
    //console.log(JSON.stringify(data,null,2))
    //console.log(data.slice(0,5))
    results.push(...data)
    await sleep(5000)
  }
  print(results)
  fs.writeFileSync("eaforumposts.json", JSON.stringify(results, 0, 2))
}
main()

Thanks for this.

Without having the data, it seems the controversy graph could be driven substantially by posts which get exactly zero downvotes.

Almost all posts get at least one vote (magnitude >= 1), and balance>=0, so magnitude^balance >=1. Since the controversy graph goes below 1, I assume you are including the handling which sets controversy to zero if there are zero downvotes, per the Reddit code you linked to.

e.g. if a post has 50 upvotes:
0 downvotes --> controversy 0 (not 1.00)
1 downvote --> controversy 1.08
2 downvotes --> controversy 1.17
10 downvotes --> controversy 2.27
so a lot of the action is in whether a post gets 0 downvotes or at least 1, and we know a lot of posts get 0 downvotes because the graph is often below 1.

If this is a major contributor, the spikes would look different if you run the same calculation without the handling (or, equivalently, with the override being to 1 instead of 0). This discontinuity also makes me suspect that Reddit uses this calculation for ordering only, not as a cardinal measure -- or that zero downvotes is an edge case on Reddit!

FJehn @ 2021-04-08T12:32 (+1)

That's a valid point. Here's the controversy graph if you exclude all posts that don't have any downvotes:

Overall trend seems to be similar though. And it makes me even more interested what happened in 2018 that sparked so much controversy^^

First of all, thanks for posting this, I think it's interesting to see some analysis on this topic I actually just yesterday thought about when I looked at an old post. I don't know if you already tried looking at this and/or whether it is even possible to do this, but I think an interesting metric would be something like "number of people who upvoted (or downvoted?) divided by number of unique people who have viewed the article". I doubt that would perfectly fix the "old posts' votes are underrepresented" (if, for example, there are any kind of chronological snobbery or "old news = boring news" biases).
Is it possible to see how many unique users have viewed an article?

Thanks for writing this. The link to the Github repository returns a 404 error. Maybe it's not set to public?

Not sure how much it matters, but if you weight vote balances by forum activity during month of publication, you aren't controlling for votes outside month of publication. This means that older posts that have received a second wind of upvotes will be ranked higher.

We can use this mean monthly karma to scale all our posts by simply dividing the karma every post has by the mean karma of the month it was published. This allows us to determine which posts received the most praise in relation to how active the forum was at the time it was published.

I expect that the variance has fluctuated over time. If this were true, something like standard deviation away from the mean would be significantly more informative than simple adjustment on the basis of the mean.

FWIW, I also expect https://forum.effectivealtruism.org/posts/ngdvhPQ2NaLHCpG9T/getting-a-feel-for-changes-of-karma-and-controversy-in-the?commentId=fcEudfKQDzgZsnhhW that the point made in this comment is valid, and therefore it may be ~impossible (unless the timing of votes is preserved) to understand reception at-the-time.