Interesting vs. Important Work - A Place EA is Prioritizing Poorly

I agree in principle with this argument but ....

Here are some of my concrete candidates for most interesting work: infinite ethics, theoretical AI safety, rationality techniques, and writing high-level critiques of EA[1].

I really don't think there are many people at all putting substantial resources into any of these areas.

• Theoretical AIS work seems really important and depending on your definition of 'theoretical' there are probably 20-100 FTE working on this per year. I would happily have at least 10-100x this amount of work if the level of quality can be at least maintained.
• Who is actually working on infinite ethics?? I don't think this has pulled a lot of EA talent, I'd guess fewer than 5 FTE.
• Rat techniques - CFAR basically doesn't exist and has fewer than 5 FTE, I'd guess most of this work is done on the side now but it certainly does not seem to be over-done by EAs.
• Criticisms - I think this is more reasonable, I don't think lots of time is spent here but a lot of attention is and I generally haven't found criticism posts to be particularly insight generating (although this one is more interesting).

My suspicion is that this "people are generally overprioritising interesting things" claim sounds nice but won't hold up to empirical investigation (at least on my world view).

He more recently mentioned that he noticed “people continuously vanishing higher into the tower,” that is, focusing on more abstract and harder to evaluate issues, and that very few people have done the opposite. One commenter, Ben Weinstein-Raun, suggested several reasons, among them that longer-loop work is more visible, and higher status. 

I disagree that longer-loop work is more visible and higher status, I think the opposite is true. In AI, agent foundations researchers are less visible and lower status than prosaic AI alignment researchers, who are less visible and lower status than capabilities researchers. In my own life, I got a huge boost of status  & visibility when I did less agent foundationsy stuff and more forecasting stuff (timelines, takeoff speeds, predicting ML benchmarks, etc.).

I think that we as a community need to put status in better places.

My sense is that it's relatively hard (though not completely infeasible) to change the status of particular jobs by just trying to agree to value them more.

An alternative strategy is to pay more for impactful roles that you can't otherwise fill. That would directly incentivise people to take them, and probably increase their status as well.

I'm generally skeptical of arguments of the form "we probably have a bias in favour of X, so we should do less X" without having an underlying model that lets you understand why you should deprioritise it. It's like when you walk with a limp in one leg and decide to start limping with your other leg to balance it out, instead of figuring out how to heal your leg in the first place (ht. Eliezer for this metaphor). Moves like that are shortsighted, and doesn't take us to a greater theoretical understanding of how to walk faster.

If the reason you're biased in favour of X (interesting) is because you don't intuitively care about Y (impactfwl) enough, then the solution is to figure out how to intuitively care more about Y. This can involve trying therapy, or it can involve growing a community that helps shape your intuitions to be more in line with what you reflectively care about.

"longer-loop work is more visible, and higher status"

Well, depends. I think legible, technical, concrete, results-producing work is sometimes overweighted because it often looks impressive and more like "doing actual work". Whereas I think backchaining is incredibly important, and working on nodes far away from our current technical frontier is almost always going to look wishy-washy and fail to produce concrete results. Unfortunately, what you call "longer-loop" work is often hard to verify, so there will be unaligned people just having fun for fun's sake, but that's not an indictment of the activity itself, or of work that just looks superficially similar.

As a deconfusion researcher, and as part of a notoriously un-paradigmatic field lacking a clear formalization, I feel like that regularly. From where I come from, math just looks more like research and actually doing work, instead of simply talking about stuff. And there is definitely a lot of value in formalization and its use to unveil confusing parts of what we’re investigating.
    --  Adam Shimi 

In conclusion, we have a lot of biases in all kinds of directions, and we should be wary of Goodharting on them. But the way to do that is to learn to see and understand those biases so we can optimise more purely. The solution is not to artificially add a constant weight in the opposite direction of whatever biases we happen to notice exists sometimes.

For what it's worth, I agree with the theoretical argument that people might be biased to overweight interestingness over importance. 

But I think I disagree fairly strong with the following (implied?) points:

• The incentive gradient within EA  (always, or usually) points people towards work with longer feedback loops than shorter feedback loops.
• The incentive gradient within EA rewards you for academia than for more direct work in EA.
• The incentive gradient within EA rewards you for more theoretical work than more applied work

I think I agree with a limited version of:

• People seem drawn to "bigger picture", more abstract work over narrow, concrete, specific problems.
• People may naturally be overly inclined to do theoretical work over applied work
  • But I think this is in spite of EA incentive gradients, not because of them.
• Perhaps in some specific circumstances, EA social incentives need to be stronger to point people away from interestingness and towards importance.

Some collection of personal experiences:

• On the forum: I feel like I get many more points (in the sense of recognition, but also literally EAF karma) per hour for commenting than for top-level posts. The former is faster and has locally better feedback loops.
• On my day job (research manager): I feel like the incentive gradient (especially locally) points me to doing local activities like day to day management, giving reviews/feedback on people's research, and away from forming deeper clarity/high-level strategic thinking.
  • This despite everyone involved generally being supportive and thinking the latter work is very important.
• Grantmaking vs. conceptual research: I think per hour, I gain more recognition, encouragement, etc., from grantmaking than for forming conceptual views and research. The former also has much faster feedback loops.
  • (This is ameliorated by me personally finding grantmaking much less interesting/fun than my other projects).
• Friends in ML academia vs alignment labs: I feel like my friends in academia usually get less points when they want to stay in academia, as opposed to joining Anthropic/Redwood.
  • And more locally, they seem to get more prestige points per hour for stuff like hosting podcasts, grantmaking, LW posts, etc, than from writing papers and deepening their understanding of ML.
    • (This isn't a clear misallocation. Naively you'd expect people staying in academia to get more prestige points from academia/the outside world, so EA giving them relatively less points seem like an appropriate redistribution).
• How I got into EA research: I got into EA research in 2020 via one of the EAish projects with the fastest feedback loops ever (coronavirus forecasting). I suspect I'd be a lot slower to either a) find internal traction or b) external engagement/recognition if I were to try to start with, e.g., AI strategy or some of the other research projects I was excited about in early 2020 (e.g. heavy-tailedness of talent).

More broadly, I think the general vibe I get from the world is that you get locally rewarded more for things that have faster feedback loops and are more attention grabbing (related notes: gamification, contrast with Deep Work). So it should be surprising if EA is a stark contrast. 

That said, I do think something about people finding "bigger picture" problems more interesting  and doing them anyway, despite incentives pointing in the other discussion, is a relevant piece of the puzzle.

I think I've been in multiple positions where I strongly advised very junior researchers to focus on narrower, more concrete research questions, as it would be better for their career and (by my personal estimation) their personal learning to start small and concrete. Usually, they ended up tackling the "big picture" nebulous problems anyway. See also Nuno's notes.

(I've also been on the receiving end of such advice) 

__

[epistemic status: idle, probably mean, speculation]

Still, there's something that needs to be explained via both your and Duncan's beliefs that people keep going up the longer feedback loop ladder and the abstraction ladder. If I were to hazard a guess for why people were to believe this, I'd probably go to:

Another important issue is that “it's easier to pass yourself off as a long-looper when you're really doing nothing,” but that’s a different discussion.

I think you treat that as a sidenote, but to me maybe this is the whole story. Speaking in a overly general way, we can imagine that EA prestige rewards people in roughly the following ways:

Theoretical wins (e.g. successfully champion a new cause area)> Concrete wins^[1](e.g. founding a  company doing great work, distributing many bednets) >> Theoretical failure >> Concrete failure.

In this story, it's easier/much more legible to see somebody fail when they do very concrete work. People are scared of openly failing, so they if they think they can't get concrete wins (or they don't believe it has sufficiently high probability), they're drawn to doing theoretical work that is harder to pinpoint exactly if, or when, they've failed. This despite concrete work having perhaps higher EV (both altruistically and selfishly). 

So if we celebrate legible failures more,  and have less of an act/omission distinction, then perhaps the incentive gradient will point people who fail at conceptual/theoretical work to be more excited to take a shot at doing more concrete work.  

Perhaps not, too. I'm far from confident that I pinpointed the exact problem or solution.

___ 

I also want to quickly note that your post conflates "theoretical/conceptual work" and "slow feedback loops." I think this is only sort of true.

1. ^{^}

   An important caveat here is that I think the rest of the world will privilege a concrete win over a theoretical win. So for example, most core EAs would consider "being one of the three most important people instrumental in causing EAs to work on AI alignment as a major cause area" to be more impressive than e.g. founding Wave. But I expect almost anybody else in the world to consider founding Wave a bigger deal.

I suggest that [pointing out a bias, like "interesting", that leads more people to some direction] and [suggesting that we should put slightly less focus on that direction because of that bias] isn't such good decision making

Maybe a helpful reframe that avoids some of the complications of "interesting vs important" by being a bit more concrete is "pushing the knowledge frontier vs applied work"?

Many of us get into EA because we're excited about crucial considerations type things and too many get stuck there because you can currently think about it ~forever but it practically contributes 0 to securing posterity. Most problems I see beyond AGI safety aren't bottlenecked by new intellectual insights (though sometimes those can still help). And even AGI safety might turn out in practice to come down to a leadership and governance problem.

"Academic work should get less status than concrete projects."

This seems like a pretty strong claim and I'm not sure I agree with it. While yeah, sure, I'm guessing people working on theory sometimes end up doing too much of what's interesting and not enough of what's the most useful, I'm guessing projects are also mostly things that people find interesting but might not be useful . 

Theory = not useful, projects = useful doesn't seem to exactly hit what I think the problem is. I'm guessing theory researchers specifically gravitate towards the bits of theory that seem the most interesting, projects people gravitate towards the most flashy/interesting projects, and these are the problems. 

I think the key thing here is that the criterion by which EA intellectuals decide whether something is interesting is significantly related to it being useful.  Firstly, because a lot of EA's are intellectually interested in things that are at least somewhat relevant to EA, lots of these fields seem useful at least at a high level; moral philosophy, rationality, and AI alignment are all clearly important things for EA. Moreover, many people actually don't find these topics interesting at all, and they are thus actually highly neglected.  This is compounded by the fact that they are very hard, and thus probably only quite smart people with good epistemics can make lots of progress on them.  These two features in turn contribute to the work being more suspiciously theoretical than it would be if the broad domains in question (formal ethics, applied rationality, alignment) were less neglected, as fields become increasingly applied as they become better theorized.   In other words, it seems prima facie plausible that highly talented people should work in domains that are initially selected partially for relevance to EA and that are highly neglected due to being quite difficult and also not as interesting to people who aren’t interested in topics related to EA, and thus more theoretical than they would be if more people worked on them.

I'm considering working at arxiv and think that would be really interesting. Any pushbacks?