Thoughts on EA Epistemics Projects

For a while, I've been thinking about the following problem: as you get better models of the world/ability to get better models of the world, you start noticing things that are inconvenient for others. Some of those inconvenient truths can break coordination games people are playing, and leave them with worse alternatives.

Some examples:

• The reason why organization X is doing weird things is because their director is weirdly incompetent
• XYZ is explained by people jockeying for influence inside some organization
• Y is the case but would be super inconvenient to the ideology du jour
• Z is the case, but our major funder is committed to believing that this is not the case
  • E.g., AI is important, but a big funder thinks that it will be important in a different way and there is no bandwidth to communicate.
• Jobs in some area are very well paid, which creates incentives for people to justify that area
• Someone builds their identity on a linchpin which is ultimately false ("my pet area is important")
• "If I stopped believing in [my religion] my wife would leave me"—true story.
• Such and such a cluster of people systematically overestimate how altruistic they are, which has a bunch of bad effects for themselves and others when they interact with organizations focused on effectiveness

Poetically, if you stare into the abyss, the abyss then later stares at others through your eyes, and people don't like that.

I don't really have many conclusions here. So far when I notice a situation like the above I tend to just leave, but this doesn't seem like a great solution, or like a solution at all sometimes. I'm wondering whether you've thought about this, about whether and how some parts of what EA does are premised on things that are false.

Perhaps relatedly or perhaps as a non-sequitur, I'm also curious about what changed since your post a year ago talking about how EA doesn't bring out the best in you.

1.  Thanks for creating this post! 
2. I think it could be worth clarifying how you operationalize EA epistemics. In this comment, I mostly focus on epistemics at EA-related organizations and focus on "improving decision-making at organizations" as a concrete outcome of good epistemics. 
3. I think I can potentially provide value by adding anecdotal data points from my work on improving epistemics of EA related organizations. For context, I work at cFactual, supporting high-impact organizations and individuals during pivotal times. So far we have done 20+ projects partnering with 10+ EA adjacent organizations. 
   1. Note that there might be several sampling biases and selection effects, e.g., organizations who work with us are likely not representing all high-impact organizations.
   2. So please read it what it is: Mixed confidence thoughts based on weak anecdotal data which are based on doing projects on important decisions for almost 2 years. 
4. Overall, I agree with you that epistemics at EA orgs tend to be better than what I have seen while doing for-profit-driven consulting in the private, public and social sectors. 
   1. For example, following a simple decision document structure including: Epistemic status, RAPID, current best guess, main alternatives considered, best arguments for the best guess, best arguments against the best guess, key uncertainties and cruxes, most likely failure mode and things we would do if we had more time, is something I have never seen in the non-EA world. 
5. The services we list under "Regular management and leadership support for boards and executives" are gaps we see that often ultimately improve organizational decision-making.
   1. Note that clients pay us, hence, we are not listing things, which could be useful but don't have a business model (like writing a report on improving risk management by considering base rates and how risks link and compound). 
   2. I think many of the gaps, we are seeing, are more about getting the basics right in the first place and don't require sophisticated decision-making methods, e.g.,
      1. spending more time developing goals incl. OKRs, plans, theories of change, impact measurement and risk management 
         1. Quite often it is hard for leaders to spend time on the important things instead of the urgent and important things, e.g., more sophisticated risk management still seems neglected at some organizations even after the FTX fall-out
      2. improving executive- and organization-wide reflection, prioritization and planning rhythms 
      3. asking the right questions and doing the right, time-effective analysis at a level which is decision-relevant
      4. getting an outside view on important decisions and CEO performance from well-run boards, advisors and coaches
      5. improving the executive team structure and hiring the right people to spend more time on the topics above
   3. Overall, I think the highest variance on whether an organization has good epistemics can be explained by hiring the right people and the right people simply spending more time on the prioritized, important topics. I think there are various tweaks on culture (e.g., rewarding if someone changes someone's mind, introducing an obligation to dissent and Watch team backup), processes (e.g., having a structured and regular retro and prioritization session, making forecasts when launching a new project) as well as targeted upskilling (e.g., there are great existing calibration tools which could be included in the onboarding process) but the main thing seems to be something simple like having the right people, in the right roles spending their time on the things that matter most. 
      1. I think simply creating a structured menu of things organizations currently do to improve epistemics (aka a google doc) could be a cost-effective MVP for improving epistemics at organizations
6. To provide more concrete, anecdotal data on improving epistemics of key organizational decisions, the comments I leave most often when redteaming google docs of high impact orgs are roughly the following:
   1. What are the goals?
   2. Did you consider all alternatives? Are there some shades of grey between Option A and B? Did you also consider postponing the decisions?
   3. What is the most likely failure mode? 
   4. What are the main cruxes and uncertainties which would influence the outcome of the decision and how can we get data on this quickly? What would you do if you had more time?
   5. Part X doesn't seem consistent with part Y
7. To be very clear, 
   1. I also think that I am making many of these prioritization and reasoning mistakes myself! Once a month, I imagine providing advice to cFactual as an outsider and every time I shake my head due to obvious mistakes I am making. 
   2. I also think there is room to use more sophisticated methods like forecasting for strategy, impact measurement and risk management or other tools mentioned here and here

Three Potential Project Ideas:

Alternate Perspectives Fellowship:

A bunch of EA's explore different non-EA perspectives for X number of weeks then each participant writes up a blog post based on something that they learned or one place where they updated. There could also be a cohort post that contains a paragraph or two from each participant about their biggest updates.

It probably wouldn't make sense for this to scale massively. Instead, you'd want to try to recruit skilled communicators or decision-makers such that this course could be impactful even with only a small number of participants.

Rethinking EA Fellowship

Get a bunch of smart young EA's from different backgrounds into the same room for two weeks. In the first week, ask them to rethink EA/the EA community from the ground up taking into account how the situation has shifted since EA was founded.

During the second week, bring in a bunch of experienced EA's who can share their thoughts on why things are the way that they are. The young EA's can then revise their suggestions taking their feedback into account.

Alternate Positive Vision Competition

We had a criticism competition before. Unfortunately, only a small proportion of the criticism seemed valuable (no offense to anyone!) and too much criticism creates a negative social environment. So what I'd love to propose as an alternative would be a competition to craft an alternate positive vision.

I am happy to see this. Have you messaged people on the EA and epistemics slack?

Here are some epistemics projects I am excited about:

• Polymarket and Nate Silver - It looks to me that forecasting was 1 - 5% of the Democrats dropping Biden from their ticket. Being able to rapidly see the drop in his % chance of winning during the debate, holding focus on this poor performance over the following weeks and seeing momentum increase for other candidates all seemed powerful^[1].
• X Community Notes - that one of the largest social media platforms in the world has a truth seeking process with good incentives is great. For all Musk's faults, he has pushed this and it is to his credit. I think someone should run a think tank to lobby X and other orgs into even better truth seeking
• The Swift Centre - large conflict of interest, since I forecast for them, but as a forecasting consultancy that is managing to stand largely (entirely?) without grant funding, just getting standard business gigs, if I were gonna suggest epistemics consulting, I'd probably recommend us. The Swift Centre is a professional or that has worked with DeepMind and the Open Nuclear Network.
• Discourse mapping - Many discussions happen often and we don't move forward. Personally I'm really excited about trying to find consensus positions to allow focus to be freed for more important stuff. Here is the site my team mocked up for Control AI, but I think we could have similar discourse mapping for SB 1047, different approaches to AI safety
• The Forum's AI Welfare Week - I enjoyed a week of focus on a single topic. I reckon if we did about 10 of these we might really start to get somewhere. Perhaps with clustering on different groups based on their positions on initial spectra. 
• Sage's Fatebook.io - a tool for quickly making and tracking forecasts. The only tool I've found that I show to non-forecasting business people that they say "oh what's that, can I use that". I think Sage should charge for this and try and push it as a standard SaaS product.^[2]

And a quick note:

An example of a potential project here: A consultancy which provides organisations support in improving their epistemics. 

I think the obvious question here should be "how would you know such a consultancy has good epistemics".

As a personal note, I've been building epistemic tools for years, eg estimaker.app or casting around for forecasting questions to write on. The FTXFF was pretty supportive of this stuff, but since it's fall I've not felt like big EA finds my work particularly interesting or worthy of support. Many of the people I see doing interesting tinkering work like this end up moving to AI Safety. 

1. ^{^}

   Not that powerful and positively impactful aren't the same thing, but here people who said Biden was too old should be glad he is gone, by their own lights

2. ^{^}

   Though maybe we let Adam finish his honeymoon first. Congratulations to the happy couple!

I think that "awareness of important simple facts" is a surprisingly big problem.

Over the years, I've had many experiences of "wow, I would have expected person X to know about important fact Y, but they didn't".

The issue came to mind again last week:

My sense is that many people, including very influential folks, could systematically—and efficiently—improve their awareness of "simple important facts".

There may be quick wins here. For example, there are existing tools that aren't widely used (e.g. Twitter lists; Tweetdeck). There are good email newsletters that aren't reliably read. Just encouraging people to make this an explicit priority and treat it seriously (e.g. have a plan) could go a long way.

I may explore this challenge further sometime soon.

I'd like to get a better sense of things like:

a. What particular things would particular influential figures in AI safety ideally do?
b. How can I make those things happen?

As a very small step, I encouraged Peter Wildeford to re-share his AI tech and AI policy Twitter lists yesterday. Recommended.

Happy to hear from anyone with thoughts on this stuff (p@pjh.is). I'm especially interested to speak with people working on AI safety who'd like to improve their own awareness of "important simple facts".

I think this is a really important area, and it is great someone is thinking whether EAIF could expand more into this sort of area. 

To provide some thoughts of my own having explored working with a few EA entities through our consultancy (https://www.daymark-di.com/) that works to help organisations improve their decision making processes, along with discussions I've had with others on similar endeavours:

1. Funding is a key bottleneck, which isn't surpising. I think there is naturally an aversion to consultancy-type support in EA organisations, mostly driven by lack of funds to pay for it and partly because I think they are concerned how it'll look in progress reports if they spend money on consultants. 
   1. EAIF funding could make this easier, as it'll remove the entire (or a large part) of the cost above.

2. There does appear a fairly frequent assumption that EA organisations suffer less from poor epistemics and decision making practices, which my experience suggests is somewhat true but unfortunately not entirely. I want to repeat what Jona from cFactual commented below that there are lots of actions EA organisations take that are very positive, such as BOTECs, likely failure models, and using decision matrices. This should be praised as many organisations don't even do these. However, the simple existence of these is too often assumed to mean good analysis and judgment will naturally occur and the systems/processes that are needed to make them useful are often lacking. To be more concrete, as an example few BOTECs incorporate second order probability/confidence correctly (or it is conflated with first order probability) and they subsequently fail to properly account for the uncertainty of the calculation and the accurate comparisons that can be made between options. 

   It has been surprising to observe the difference between some EA organisations and non-EA institutions when it comes to interest in improving their epistemics/decision making. With large institutions (including governmental) being more receptive and proactive in trying to improve - often those institutions are mostly being constrained by their slow procurement processes as opposed appetite.

When it comes to future projects, my recommendations of those with highest value add would be:

1. Projects that help organisations improve their accountability, incentive, and prioritisation mechanisms. In particular helping to identify and implement internal processes that properly link workstreams/actions/decisions/judgments to the goals of the organisation and that role's objectives. This is something which is most useful to larger organisations (and especially those that make funding/granting decisions or recommendations), but can also be something which smaller organisations would benefit from.
2. Projects that help organisations improve their processes to more accurately and efficiently reason under uncertainty, namely assisting with defining the prediction/diagnostic they are trying to make a judgment on, identify the causal chain and predictors (and their relative importance) that underpins their judgment, and provide them with a framework to communicate their uncertainty and reasoning in a transparent and consistent way.
3. Projects that provide external scrutiny and advanced decision modelling capabilities. I think at a low level there are some relatively easy wins that can be gained from having an entity/service which red teams and provides external assessment of big decisions that organisations are looking at making (e.g. new requests for new funding). At a greater level there should be more advanced modelling (including using tools such as Bayesian models) which can provide transparent and updatable analysis which can expose conditional relationships that we can't properly account for in our minds.
   1. I think funding entities like EA Funds could utilise such an entity/entities to inform their grant making decisions (e.g. if such analysis was instrumental in the decision EA Funds made on a grant request, they'd pay half of the cost of the analysis).