Future of Humanity Institute 2005-2024: Final Report

By Pablo @ 2024-04-17T14:25 (+224)

This is a linkpost to https://static1.squarespace.com/static/660e95991cf0293c2463bcc8/t/661a3fc3cecceb2b8ffce80d/1712996303164/FHI+Final+Report.pdf

Anders Sandberg has written a “final report” released simultaneously with the announcement of FHI’s closure. The abstract and an excerpt follow.

Normally manifestos are written first, and then hopefully stimulate actors to implement their vision. This document is the reverse: an epitaph summarizing what the Future of Humanity Institute was, what we did and why, what we learned, and what we think comes next. It can be seen as an oral history of FHI from some of its members. It will not be unbiased, nor complete, but hopefully a useful historical source. I have received input from other people who worked at FHI, but it is my perspective and others would no doubt place somewhat different emphasis on the various strands of FHI work.

What we did well

One of the most important insights from the successes of FHI is to have a long-term perspective on one’s research. While working on currently fashionable and fundable topics may provide success in academia, aiming for building up fields that are needed, writing papers about topics before they become cool, and staying in the game allows for creating a solid body of work that is likely to have actual meaning and real-world effect.

The challenge is obviously to create enough stability to allow such long-term research. This suggests that long-term funding and less topically restricted funding is more valuable than big funding.

Many academic organizations are turned towards other academic organizations and recognized research topics. However, pre-paradigmatic topics are often valuable, and relevant research can occur in non-university organizations or even in emerging networks that only later become organized. Having the courage to defy academic fashion and “investing” wisely in such pre-paradigmatic or neglected domains (and networks) can reap good rewards.

Having a diverse team, both in terms of backgrounds but also in disciplines, proved valuable. But this was not always easy to achieve within the rigid administrative structure that we operated in. Especially senior hires with a home discipline in a faculty other than philosophy were nearly impossible to arrange. Conversely, by making it impossible to hire anyone not from a conventional academic background (i.e., elite university postdocs) adversely affects minorities, and resulted in instances where FHI was practically blocked from hiring individuals from under-represented groups. Hence, try to avoid credentialist constraints.

In order to do interdisciplinary work, it is necessary to also be curious about what other disciplines are doing and why, as well as to be open to working on topics one never considered before. It also opens the surface to the rest of the world. Unusually for a research group based in a philosophy department, FHI members found themselves giving tech support to the pharmacology department; participating in demography workshops, insurance conferences, VC investor events, geopolitics gatherings, hosting artists and civil servant delegations studying how to set up high-performing research institutions in their own home country, etc. - often with interesting results.

It is not enough to have great operations people; they need to understand what the overall aim is even as the mission grows more complex. We were lucky to have had many amazing and mission-oriented people make the Institute function. Often there was an overlap between being operations and a researcher: most of the really successful ops people participated in our discussions and paper-writing. Try to hire people who are curious.

Where we failed

Any organization embedded in a larger organization or community needs to invest to a certain degree in establishing the right kind of social relationships to maintain this embeddedness. Incentives must be aligned, and both parties must also recognize this alignment. We did not invest enough in university politics and sociality to form a long-term stable relationship with our faculty.

There also needs to be an understanding of how to communicate across organizational communities. When epistemic and communicative practices diverge too much, misunderstandings proliferate. Several times we made serious missteps in our communications with other parts of the university because we misunderstood how the message would be received. Finding friendly local translators and bridgebuilders is important.

Another important lesson (which is well known in business and management everywhere outside academia) is that as an organization scales up it needs to organize itself differently. The early informal structure cannot be maintained beyond a certain size, and must be gradually replaced with an internal structure. Doing this gracefully, without causing administrative sclerosis or lack of delegation, is tricky and in my opinion we somewhat failed.

So, you want to start another FHI?

Did FHI become humanity's best effort at understanding and evaluating its own long-term prospects? We leave that to the future to evaluate properly, but we certainly think we did unexpectedly well for a “three-year project”.

FHI is ending and we are sad to see it go. We think it could have achieved far more than it did, but circumstances made it impossible to continue. On the plus side, we know FHI did not live past its time. There is a real risk that organizations lose their mission and become self-perpetuating users of resources that could better be used for other things, preventing the flowering of the new.

In fact, as mentioned above, FHI has seeded a number of new organizations, fields and topics. In a biological or memetic sense, it would count as having had great fitness in propagating successors, although many are not FHI-like or have different goals.

What would it take to replicate FHI, and would it be a good idea? Here are some considerations for why it became what it was:

A comment from a member:

“I think there’s no cookie-cutter template for replicating FHI because it depends critically on having the right (rare) people and a particular intellectual culture. The secret sauce was not an organizational structure or some kind of management process. But with the right people and culture, then shielding from other constraints can become enabling.

“To the extent that it can be replicated, I think it is because (a) it was an existence proof of organization, template ideas and research results, and legitimization, and (b) the intellectual culture has spread (e.g. in the wider rationalist and EA networks). But it could probably not be replicated by having some random administrator or manager trying to reproduce the same organizational structure - that would be like the cargo cult.”

So, the conclusion may be that while the above considerations give a recipe to aim for, the key question for any replication should be: “What are the important topics this organization should aim at?” Pursuing those topics must always be at the center of what is being done (both in research and administration), even when new knowledge and developments change them and their priorities.

finm @ 2024-04-17T18:47 (+199)

I think it is worth appreciating the number and depth of insights that FHI can claim significant credit for. In no particular order:

Note especially how much of the literal terminology was coined on (one imagines) a whiteboard in FHI. “Existential risk” isn't a neologism, but I understand it was Nick who first suggested it be used in a principled way to point to the “loss of potential” thing. “Existential hope”, “vulnerable world”, “unilateralist's curse”, “information hazard”, all (as far as I know) tracing back to an FHI publication.

It's also worth remarking on the areas of study that FHI effectively incubated, and which are now full-blown fields of research:

FHI was staggeringly influential; more than many realise.

Edit: I wrote some longer reflections on FHI here.

MathiasKB @ 2024-04-18T07:41 (+17)

I'm awestruck, that is an incredible track record. Thanks for taking the time to write this out.

These are concepts and ideas I regularly use throughout my week and which have significantly shaped my thinking. A deep thanks to everyone who has contributed to FHI, your work certainly had an influence on me.

Linch @ 2024-04-22T03:09 (+2)

What a champ. if institutions can be heroes, FHI is surely one. 

carrickflynn @ 2024-04-18T20:21 (+196)

I want to take this opportunity to thank the people who kept FHI alive for so many years against such hurricane-force headwinds. But I also want to express some concerns, warnings, and--honestly--mixed feelings about what that entailed. 

Today, a huge amount of FHI's work is being carried forward by dozens of excellent organizations and literally thousands of brilliant individuals. FHI's mission has replicated and spread and diversified. It is safe now. However, there was a time when FHI was mostly alone and the ember might have died from the shockingly harsh winds of Oxford before it could light these thousands of other fires. 

I have mixed feelings about encouraging the veneration of FHI ops people because they made sacrifices that later had terrible consequences for their physical and mental health, family lives, and sometimes careers--and I want to discourage others from making these trade-offs in the future. At the same time, their willingness to sacrifice so much, quietly and in the background, because of their sincere belief in FHI's mission--and this sacrifice paying off with keeping FHI alive long enough for its work to spread--is something for which I am incredibly grateful. 

A small selection from the report:

Bostrom has stated: “I wish it were possible to convey the heroic efforts of our core administrative team that were required to keep the FHI organizational apparatus semi-performant and dynamic for all those years until its final demise! It is an important part of the story. And the discrepancy between the caliber of our people and the typical university administrators - like Andrew carpet bombing his intray with pomodoros over the weekends... or Tanya putting in literal 21 or 22 hour workdays (!) for weeks at an end. Probably not even our own researchers fully appreciate what went on behind the scenes.”  

21 and 22 hour workdays sounds like hyperbole, but I was there and it isn't. No one should work this hard. And it was not free. Yet, if you ever meet Tanya Singh, please know you are meeting a (foolishly self-sacrificing?) hero. 

And while Andrew Snyder-Beattie is widely and accurately known as a productivity robot, transforming into a robot--leaving aside the fairytales of the cult of productivity--requires inflicting an enormous amount of deprivation on your human needs.

But why did this even happen? An example from the report:

One of our administrators developed a joke measurement unit, “the Oxford”. 1 Oxford is the amount of work it takes to read and write 308 emails. This is the actual administrative effort it took for FHI to have a small grant disbursed into its account within the Philosophy Faculty so that we could start using it - after both the funder and the University had already approved the grant.) 

This again sounds like hyperbole. It again is not. This was me. After a small grant was awarded and accepted by the university, it took me 308 emails to get this "completed" grant into our account.

FHI died because Oxford killed it. But it was not a quick death. It was a years-long struggle with incredible heroism and terrible casualties. But what a legacy. Thank you sincerely to all of the ops people who made it possible.

Sean_o_h @ 2024-04-19T09:00 (+74)

Strong +1 re: 'hero' work culture. especially for ops staff. This was one of the things that bothered me while there and contributed to my moving on - an (admittedly very nice) attitude of praising (especially admin/management) people who were working stupidly hard/long, rather than actually investing in fixing a clearly dysfunctional situation. And while it might not have been possible to fix later on due to embedded animosity/frustration on both sides => hiring freeze etc, it certainly was early on when I was there.

The admin load issue was not just about the faculty. And the breakdown of relationship with the faculty was really was not one-sided, at least when I was there (and I think I succeeded in semi-rescuing some of the key relationships (oxford martin school, faculty of philosophy) while I was there, at least temporarily).

Richard Y Chappell @ 2024-04-17T16:08 (+89)

This is really sad news. I hope everyone working there has alternative employment opportunities (far from a given in academia!).

I was shocked to hear that the philosophy department imposed a freeze on fundraising in 2020. That sounds extremely unusual, and I hope we eventually learn more about the reasons behind this extraordinary institutional hostility. (Did the university shoot itself in the financial foot for reasons of "academic politics"?)

A minor note on the forward-looking advice: "short-term renewable contracts" can have their place, especially for trying out untested junior researchers. But you should be aware that it also filters out mid-career academics (especially those with family obligations) who could potentially bring a lot to a research institution, but would never leave a tenured position for short-term one. Not everyone who is unwilling to gamble away their academic career is thereby a "careerist" in the derogatory sense.

AnotherAnonymousFTXAccount @ 2024-04-17T20:22 (+25)

On your second point, FHI had at least ~£10m sitting in the bank in 2020 (see below, from the report). So the fundraising freeze, while unusual, wasn't terminal. A rephrasing of your question is "What adminstrative and organisational problems at FHI could possibly have prompted the Faculty to take the unusual step of a hiring and fundraising freeze in 2020, and why could it not be resolved over the next two to three years?"

"Open Philanthropy became FHI’s most important funder, making two major grants: £1.6m in 2017, and £13.3m in 2018. Indeed, the donation behind this second grant was at the time the largest in the Faculty of Philosophy’s history (although, owing to limited faculty administrative capacity for hiring and the subsequent hiring freezes it imposed, a large part of this grant would remain unspent)"

Pablo @ 2024-04-17T14:37 (+53)

There is now a dedicated FHI website with lists of selected publications and resources about the institute. (Thanks to @Stefan_Schubert for bringing this to my attention.)

Pablo @ 2024-04-20T14:04 (+41)

See also Anders’s more personal reflections:

I have reached the age when I have seen a few lifecycles of organizations and movements I have followed. One lesson is that they don’t last: even successful movements have their moment and then become something else, sclerotize into something stable but useless, or peter out. This is fine. Not in some fatalistic “death is natural” sense, but in the sense that social organizations are dynamic, ideas evolve, and there is an ecological succession of things. 1990s transhumanism begat rationalism that begat effective altruism, and to a large degree the later movements suck up many people who would otherwise have been recruited by the transhumanists.

FHI did close before its time, but it is nice to know it did not become something pointlessly self-perpetuating. As we noted when summing up, 19 years is not bad for a 3-year project. Indeed, a friend remarked that maybe all organisations should have a 20-year time limit. After that, they need to be closed down and recreated if they are still useful, shedding some of the accumulated dross.

The ecological succession of organizations and movements is not all driven by good forces. A fresh structure driven by interested and motivated people is often gradually invaded by poseurs, parasites and imitators, gradually pushing away the original people (or worse, they mutate into curators, gatekeepers and administrators). Many ideas develop, flourish, become explored and then forgotten once a hype peak is passed – even if they still have merit. People burn out, lose interest, form families and have to change priorities, or the surrounding context make the movement change in nature. Dwindling activist movements may suffer “core collapse” as moderate members drift off while the hard core get more radical and pursue ever more extreme activism in order to impress each other rather than the world outside.

FHI did not do any of that. If we had a memetic failure, it was likely more along the lines of developing a shared model of the world and future that may have been in need for more constant challenge. That is one reason why I hope there will be more organizations like FHI but not thinking alike – places like CSER, Mimir, FLI, SERI, GCRI, and many others. We need the focus of a strongly connected organization to build thoughts and systems of substance but separate organizations to get mutual critique and diversity in approaches. Plus, hopefully, metapopulation resilience against individual organizational failures.

Cullen @ 2024-04-18T16:03 (+24)

I made a perma.cc copy of the Final Report here: https://perma.cc/3KP9-ZSFB

niplav @ 2024-04-17T15:55 (+21)

I'd be curious about a list of topics they would like others to investigate/continue investigating, or a list of the most important open questions.

Pablo @ 2024-04-17T16:21 (+17)

There is a list by Sandberg here. (The other items in that post may also be of interest.)

Arepo @ 2024-04-17T21:24 (+20)

That's sad. For anyone interested in why they shut down (I'd thought they had an indefinitely sustainable endowment!), the archived version of their website gives some info:

Over time FHI faced increasing administrative headwinds within the Faculty of Philosophy (the Institute’s organizational home).  Starting in 2020, the Faculty imposed a freeze on fundraising and hiring.  In late 2023, the Faculty of Philosophy decided that the contracts of the remaining FHI staff would not be renewed.  On 16 April 2024, the Institute was closed down.

Milena Canzler @ 2024-04-19T07:04 (+33)

I still don't understand why the University of Oxford was not cooperative with the institute, and why later it decided to freeze it completely. What was that about?

Hamish McDoodles @ 2024-04-20T03:24 (+14)

I'm also confused by this. Did Oxford think it was a reputation risk? Were the other philosophers jealous of the attention and funding FHI got? Was a beaurocratic parasitic egregore putting up roadblocks to siphon off money to itself? Garden variety incompetence?

Sean_o_h @ 2024-04-20T08:34 (+54)

Having worked there and interfaced with the Faculty for 4 years, yes, I would expect garden variety incompetence on Bostrom's part in terms of managing the relationship was a big part; I would predict the single biggest contributer to the eventual outcome.

Hamish McDoodles @ 2024-04-20T12:31 (+18)

Why was relationship management even necessary? Wasn't FHI bringing prestige and funding to the university? Aren't the incentives pretty well aligned?

Hamish McDoodles @ 2024-04-21T22:50 (+10)

Why are people pressing the "disagree" button? Do they disagree with the idea that FHI brought prestige? Do they disagree with the framing? Is it because I have a silly username?

Clearly there's some politics going on here, but I have no idea who the factions are or why.

Someone help me out?

Rebecca @ 2024-04-21T23:33 (+14)

I’m very confused why you think that FHI brought prestige to Oxford University rather than the other way around

Linch @ 2024-04-22T03:27 (+6)

The vast majority of academic philosophy at prestigious universities will be relegated to the dustbins of history, FHI's work is quite plausibly an exception. 

To be clear, this is not a knock on philosophy; I'd guess that total funding for academic philosophy in the world is on the order of 1B. Most things that are 0.001% of the world economy won't be remembered much 100 years from now. I'd guess philosophy in general punches well above its weight here, but base rates are brutal.

Rebecca @ 2024-04-22T09:03 (+13)

You’re answering a somewhat different question to the one I’m bringing up

Hamish McDoodles @ 2024-04-22T00:26 (+6)

My thinking was that Because they were doing influential research and brought in funding? FHI's work seems significantly better than most academic philosophy, even by prestigious university standards.

But on reflection, yes, obviously Oxford University will bring more prestige to anything it touches.

Linch @ 2024-04-22T03:41 (+5)

Erm, looking at the accomplishments of FHI, I'd be genuinely surprised if random philosophers from Oxford will be nearly as influential going forwards. "It's the man that honors the medal."

Rebecca @ 2024-04-22T13:06 (+16)

Influence =/= prestige

Linch @ 2024-04-22T20:37 (+4)

I might not be tracking all the exact nuances, but I'd have thought that prestige is ~just legible influence aged a bit, in the same way that old money is just new money aged a bit. I model institutions like Oxford as trying to play the "long game" here.

Rebecca @ 2024-04-22T22:17 (+13)

The point I’m trying to make is that there are many ways you can be influential (including towards people that matter) and only some of them increase prestige. People can talk about your ideas without ever mentioning or knowing your name, you can be a polarising figure who a lot of influential people like but who it’s taboo to mention, and so on.

I also do think you originally meant (or conveyed) a broader meaning of influential - as you mention economic output and the dustbins of history, which I would consider to be about broad influence.

Rebecca @ 2024-04-22T21:03 (+11)

Andrew Tate is very influential, but entirely lacking in prestige.

Arepo @ 2024-04-30T22:33 (+8)

This seems like quite an in-group perspective. From the perspective of a generic philosophy faculty, that looks like a very small list of papers for a department that was running for nearly two decades. Without knowing their impact factor (which I'd guess was higher than average, but not extreme) it's hard to say whether this was reasonable from a prestige perspective.

Linch @ 2024-04-30T22:37 (+4)

I don't think it's just an in-group perspective! Bostrom literally gives and receives feedback from kings; other members of FHI have gone on to influential positions in multi-billion dollar companies. 

Are you really saying that if you ask the general public (or members of the intellectual elite), typical philosophy faculty at prestigious universities will be recognized to be as or more impressive or influential in comparison? 

Arepo @ 2024-04-30T22:55 (+9)

When did he get feedback from Kings? Googling it, the only thing I can see is that he was invited to an event which the Swedish king was also at.

Also, most of Bostrom's extra-academic prestige is based on a small handful of the papers listed. That might justify making him something like a public communicator of philosophy, but it doesn't obviously merit sponsoring an entire academic department indefinitely.

To be clear, I have no strong view on whether the university acted reasonably a) in the abstract or b) according to incentives in the unique prestige ecosystem which universities inhabit. But I don't think listing a handful of papers our subgroup approves of is a good rationale for claiming that it did neither.

Linch @ 2024-04-30T23:00 (+2)

I'm at work and don't have the book with me, but you can look at the "Acknowledgements" section of Superintelligence. 

I agree that it's not clear whether the Department of Philosophy acted reasonably in the unique prestige ecosystem which universities inhabit, whether in the abstract or after adjusting for FHI quite possibly being unusually difficult/annoying to work with. I do think history will vindicate my position in the abstract and "normal people" with a smattering of facts about the situation (though perhaps not the degree of granularity where you understand the details of specific academic squabbles) will agree with me.

Ben Millwood @ 2024-04-22T12:21 (+6)

This sounds like it's disagreeing with the parent comment but I'm not sure if it is?

Linch @ 2024-04-22T20:40 (+2)

I claim that on net FHI would've brought more prestige to Oxford than the other way around, especially in the counterfactual world where it thrived/was allowed to thrive (which might be impractical for other reasons). 

Owen Cotton-Barratt @ 2024-04-22T22:06 (+15)

I might think of FHI as having borrowed prestige from Oxford. I think it benefited significantly from that prestige. But in the longer run it gets paid back (with interest!).

That metaphor doesn't really work, because it's not that FHI loses prestige when it pays it back -- but I think the basic dynamic of it being a trade of prestige at different points in time is roughly accurate.

𝕮𝖎𝖓𝖊𝖗𝖆 @ 2024-04-22T18:56 (+15)

Sad to hear this happened, but it seems the situation was irrecoverable, and the organisation was already dead for a bit before it officially shuttered.

Glad for this post and all the comments.

DominikPeters @ 2024-04-17T21:51 (+13)

From Bostrom's website, an updated "My Work" section reads:

... That’s why I founded the Future of Humanity Institute at Oxford University in 2005. FHI brought together an interdisciplinary bunch of brilliant (and eccentric!) minds, and sought to shield them as much as possible from the pressures of regular career academia; and thus were laid the foundations for exciting new fields of study.

Those were heady years. FHI was a unique place - extremely intellectually alive and creative - and remarkable progress was made. FHI was also quite fertile, spawning a number of academic offshoots, nonprofits, and foundations. It helped incubate the AI safety research field, the existential risk and rationalist communities, and the effective altruism movement. Ideas and concepts born within this small research center have since spread far and wide, and many of its alumni have gone on to important positions in other institutions.

Today, there is a much broader base of support for the kind of work that FHI was set up to enable, and it has basically served out its purpose. (The local faculty administrative bureaucracy has also become increasingly stifling.) I think those who were there during its heyday will remember it fondly. I feel privileged to have been a part of it and to have worked with the many remarkable individuals who flocked around it. [Update: FHI officially closed on 16 April 2024]

ClimateDoc @ 2024-04-20T12:58 (+12)

A very interesting summary, thanks.

However I'd like to echo Richard Chappell's unease at the praising of the use of short-term contracts in the report. These likely cause a lot of mental health problems and will dissuade people who might have a lot to contribute but can't cope with worrying about whether they will need to find a new job or even career in a couple of years' time. It could be read as a way of avoiding dealing with university processes for firing people - but then the lesson for future organisations may be to set up outside a university structure, and have a sensible degree of job security.

Jelle Donders @ 2024-04-20T11:22 (+10)

FHI almost singlehandedly made salient so many obscure yet important research topics. To everyone that contributed over the years, thank you!

det @ 2024-04-17T19:57 (+10)

Nick Bostrom's website now lists him as "Principal Researcher, Macrostrategy Research Initiative."

Doesn't seem like they have a website yet.

AnotherAnonymousFTXAccount @ 2024-04-17T20:14 (+23)

The Macrostrategy Research Initiative was registered as a company in August 2023. Its Director is Toby Newberry ("Occupation: Chief of Staff"), who's been at FHI/GPI for a few years.

https://find-and-update.company-information.service.gov.uk/company/15054502

Deborah W.A. Foulkes @ 2024-04-23T05:19 (+1)

Not to be confused with The Macrostrategy Partnership: https://www.macrostrategy.co.uk/

Chris Leong @ 2024-04-18T00:57 (+11)

For anyone wondering about the definition of macrostrategy, the EA forum defines it as follows:

Macrostrategy is the study of how present-day actions may influence the long-term future of humanity.[1]

Macrostrategy as a field of research was pioneered by Nick Bostrom, and it is a core focus area of the Future of Humanity Institute.[2] Some authors distinguish between "foundational" and "applied" global priorities research.[3] On this distinction, macrostrategy may be regarded as closely related to the former. It is concerned with the assessment of general hypotheses such as the hinge of history hypothesis, the vulnerable world hypothesis and the technological completion conjecture; the development of conceptual tools such as the concepts of existential risk, of a crucial consideration and of differential progress; and the analysis of the impacts and capabilities of future technologies such as artificial general intelligence, whole brain emulation and atomically precise manufacturing, but considered at a higher level of abstraction than is generally the case in cause prioritization research.

SummaryBot @ 2024-04-18T12:43 (+2)

Executive summary: The Future of Humanity Institute (FHI) achieved notable successes in its mission from 2005-2024 through long-term research perspectives, interdisciplinary work, and adaptable operations, though challenges included university politics, communication gaps, and scaling issues.

Key points:

  1. Long-term research perspectives and pre-paradigmatic topics were key to FHI's impact, enabled by stable funding.
  2. An interdisciplinary and diverse team was valuable for tackling neglected research areas.
  3. Operations staff needed to understand the mission as it grew in complexity.
  4. Failures included insufficient investment in university politics, communication gaps, and challenges scaling up gracefully.
  5. Replicating FHI would require the right people, intellectual culture, and shielding from constraints, not just copying its structure.
  6. The most important factor is pursuing the key topics and mission, even as knowledge and priorities evolve.

 

 

This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.