How we think about the Forum

By JP Addison, Centre for Effective Altruism @ 2019-10-15T16:24 (+24)

This is the second post out of two in a series about how the Forum gets made. In the last post, we introduced ourselves. In this post, we’ll try to lay out the models that guide our work and the plans we have for the future.

Models

The Forum is not a program executing a single plan; it’s a distributed ecosystem of many people trying to achieve many goals. The impacts are as numerous as the aims of every post and comment. Still, we can classify them using a few broad categories:

• Sharing of existing ideas
• Including making common knowledge
• Development and refinement of new ideas
• Talent discovery
• Public accountability
• Spreading of norms
• Encouraging coordination

(Credit to Oliver Habryka for an initial list upon which we built to produce the above.)

To achieve those goals (or really, a much broader class of goals), we think the Forum needs both of the following intermediate goals:

• People reading good content
• Good content for people to read (this includes both posts and comments)

We’ve developed a metric we use to figure out whether this is happening (and thus, whether we’re doing a good job). We evaluate our plans largely based on how we think they’ll impact the number of good posts [1] and the amount of views those posts receive. [2]

Don’t read too much confidence into this section. We view it as a better-than-nothing operating framework. In an ideal world, we’d have much more empirical data backing up our model. We’ve also had an actual, updating graph of the metric for all of a few weeks. Finally, we also track other metrics (from “time on site” to “mentions on popular blogs”) to get a better picture of the Forum’s impact.

Plans

I know the title of the section is “Plans,” but first, here are some things we’ve already done that we think of as aiming towards the “views on good posts” metric:

• Built the timeframe feature into the All Posts page to enable easier post discovery
• Since that page was hard to discover, we ported (and refactored) an improved sidebar from LW
• Ported and modified the Community Favorites section and added it to the homepage
• If you wanted, you could think of the Forum Prize as assisting with this metric, although it predates the theory
• Posted an open offer for Forum editing from Aaron, which has substantially increased the number of people seeking feedback on posts, and hopefully the counterfactual quality of those posts (we had done this on an ad-hoc basis before the posting)

Quarterly goals for Q4

A lot of our goals aren’t explicit features. They’re things like “merge in LW with maximum 4 weeks delay” or “meet these goals every two-week sprint”. Our main relevant feature-goals are:

• Creating a “Best posts since you last visited” section
• Would allow users who visit infrequently to see the top-rated posts they haven’t been around for.
• Porting the sequences feature
• We’d build this feature to support the work on the EA Handbook 3.0. We probably won’t finish the Handbook this quarter, so this will remain hidden for a while.

There are other goals that aren’t large enough to quite be “quarterly goals” in the OKR system we use, but will likely be a sprint goal, such as fixing our (currently LessWrong-green) emails, adding an option to hide karma, and improving our error reporting.

The content side (aka Aaron), also has some goals pertaining to the Forum:

• Encourage more people to do AMAs (like this one)
• Get more blogs and orgs to crosspost their content
• Write a few posts himself, and continue to encourage others to write up their ideas (e.g. from this thread)
• Write / curate the first sections of a new version of the EA Handbook (though these may not appear on the Forum until they’ve been subject to feedback from a small group of advisors)

Later goals

We might do these, but not this quarter.

• Technical SEO
• We’re currently probably underperforming in Google search results relative to where we could be if we put effort into optimizing the site to match the whims of the great judge Google.
• Category tagging
• We definitely want to be cautious here, but we’re interested in investigating ways where people who are interested in a particular sub-field of EA can find the content they’re interested in more easily, and it’s possible that we could develop something around category tags to do this.
• Recommendation algorithm
• Currently the new Community Favorites section give you recommendations for posts to read that a) are popular, and b) you haven’t read. That’s not a very complicated algorithm. We could make it much smarter; think of the Netflix recommendation algorithm. I’d guess it could really increase the amount of good posts people read, but it would be a lot of work.

Final Thoughts

We hope, after reading this, you have a good sense of the actual dynamics that shape this Forum. Don’t be shy to ask questions! We’re especially interested in reactions to the proposed features.

Footnotes

[1] What’s a “good post”? Currently, we mark certain posts as representing the kind of content we most want on the Forum (this corresponds to roughly 30% of posts). We also include posts that sparked lively/productive conversations, even if the posts themselves were very brief/just asking a question. This doesn’t mean we don’t value lots of other content — we do! A lot! — but tracking views to all posts seems less informative than watching views on the posts we think offer the most value to those who read them. (This evaluation is cause-impartial and does not affect the visibility of posts in any way.)

[2] Currently, we’re focusing on views from logged-in users. This has several good properties:

• We’re less tempted to Goodhart the metric by sharing our favorite posts as widely as possible, even if we don’t think it will be all that helpful to a broad audience
• We can use our own database to measure views, rather than relying on Google Analytics (which works well, but doesn’t see users who have an ad-blocker turned on)
• We get a better sense of how well posts are inspiring people to vote and comment, because every view we track is (in theory) someone who could do those things