Ingredients for creating disruptive research teams

By stefan.torges @ 2019-05-16T16:23 (+159)

Introduction

This post tries to answer the question of what qualities make some research teams more effective than others. I was particularly interested in learning more about “disruptive” research teams, i.e. research teams that have an outsized impact on (1) the research landscape itself (e.g. by paving the way for new fields or establishing a new paradigm), and/or (2) society at large (e.g. by shaping technology or policy).[1] However, I expect the conclusions to be somewhat relevant for all research teams.

Research seems to have become increasingly important within the effective altruism community. In the past few years, GPI was founded, FHI started growing significantly, and Open Phil is expanding its research capacity. Will MacAskill even called effective altruism a “research program”. From this perspective, we should be both interested in creating new fields of research, or at least substantially influencing existing ones, as well as impacting society.

Acknowledgments:

I did some of the research presented here as part of my work at the Berlin-based Effective Altruism Foundation (EAF), a research group and grantmaker dedicated to preventing suffering in the long-term future. Thanks to Jonas Vollmer, Jan Dirk Capelle, Max Daniel, and Alfredo Parra for valuable comments on an early draft of this post.

Summary

I looked at the two most comprehensive and rigorous academic studies on productive research teams I could find after a shallow review of the available literature (one literature review, Bland & Ruffin (1992), and one meta-analysis, Hülseger, Anderson & Salgado (2009)). Unfortunately, I could not find similarly comprehensive studies of disruptive research teams in particular. I complemented this with seven case studies of research teams I picked based on my own non-systematic judgment that they have been particularly disruptive. These are the RAND Corporation, the Sante Fe Institute, the Palo Alto Research Center (PARC), Bell Labs, Skunk Works, the Los Alamos Laboratory, and the partnership of Kahneman & Tversky.

The following are my key findings based on this research:

After discussing the key findings, I go on to list learnings for my own organization and review the considered evidence in more detail.

Methodology

When talking about “teams”, I’m referring to a somewhat independent functional unit with distinct leadership, vision, and research direction. So such a team might well exist within a larger (research) organization. For instance, a university would not count as a research team whereas a particular lab would.

Academic literature on disruptive research teams

After some search on google scholar, I found there to be little academic research on disruptive research teams, let alone comprehensive reviews. There is a recent paper by Wu, Wang & Evans which presented solid evidence that disruptive teams tend to be smaller. This is in line with my impression of similar previous research, mainly in the field of disruptive innovation. However, they don’t cite other work on disruptive research teams.

Faced with this lack of research on disruptive research teams, I broadened the scope to high-performance research groups in general. This is either operationalized with subjective ratings or metrics like paper and patent count, which is sometimes adjusted for impact. So it might still capture some of the disruptive nature I was interested in. I set out to find the most relevant meta-studies in the field to save time and focus on the most robust results. I first looked at the studies Max Dalton covers in his literature view and expanded from there by looking at the references of these papers. I also performed a few searches on google scholar to make sure I hadn’t missed anything. Ultimately, I settled on focusing on two studies:

While BR is fairly old, Bill Dunn from the Oxford Learning Institute claims that their findings have held up since then. HAS is more recent, but they looked at innovation for teams in general (in work contexts) instead of dedicated research teams. This makes the study less applicable for the purpose of this investigation. I still included it because it’s very comprehensive and I expect the lessons to be transferable to a significant extent.

Even after doing this work, I still don’t feel very well versed in this field of study and might well have missed something. I have the general sense that studying groups is hard. Many constructs seem fuzzy to me and it’s difficult to run experiments in this context. So this field does not seem particularly reliable to me.

Case studies of disruptive research teams

I included case studies for several reasons. The academic literature I could find was not concerned with disruptive teams in particular. With case studies, I could pick out teams whose research specifically either pioneered fields or profoundly changed society through their work. There might be differences between the very best groups and merely good groups. Meta-analyses, in particular, are at risk of overemphasizing aspects which are easy to quantify.

There are downsides to relying on case studies which the reader should be aware of:

My selection was not very systematic and mostly involved asking people which research groups they thought have been influential, looking into how new research fields had been pioneered, and reading books which seemed to focus on this topic. Organizing Genius by Bennis and Biedermann, in particular, was a helpful starting point. This approach likely selected for teams with particularly great (and visible) societal effects. This struck me as acceptable, given that I was looking for teams with a profound influence on society. It likely excludes those with less traceable effects.

Ultimately, I ended up looking into the following seven groups: the RAND Corporation, the Santa Fe Institute, Xerox PARC, Bell Labs, Lockheed’s Skunk Works, the Los Alamos Laboratory, and the collaboration of Kahneman & Tversky.

This selection is to some extent idiosyncratic, but I think there is a good case for the inclusion of each group in terms of accomplishments. It’s more likely that I overlooked groups which deserve to be included. Other groups I considered but didn’t include are the Cowles Foundation, the Institute for Advanced Study, the MIT Lincoln Laboratory, the Cavendish Laboratory, the Institut des hautes études scientifiques[2], and the MIT Media Lab. The main reason for their exclusion was the lack of available sources on the inner workings of these institutions. It’s entirely possible that there are great primary sources, interviews, or similar distributed resources that one might be able to use to expand on this post.

Integration

I didn’t integrate these two strands very systematically. I drew out the relevant lessons from each part and tried to synthesize them. I indicate what resources my conclusions are based on.

Key findings

In this section, I list the ingredients for disruptive research teams that I believe matter based on the research I’ve done. I don’t discuss in detail factors which strike me as irrelevant. While there might be additional ingredients which I haven’t identified, I do think I cover the most important levers. Instead, I believe that vague categories are the most likely flaw of my analysis since they might allow each reader to simply fill them with their own preconceived ideas of what they mean. I have attempted to make each item as concrete as possible, but this is very hard when aggregating evidence from diverse sources, case studies in particular. I considered adding subjective credences for each section, but I concluded that doing so would have led readers to assume more precision than is actually there, given the fuzzy nature of many of the factors.

I do not make the strong claim that all particularly disruptive research teams combine all of the ingredients I list to a very large extent. Rather, I make the weaker claim that the best disruptive research teams likely combine many of these ingredients to a significant degree. I also discuss their relative importance where I think it matters. Further, I don’t claim that a research team with these ingredients will be among the best disruptive teams. I think it’s likely that outside factors like luck and timing play a significant role.

This list is compiled from the perspective of trying to create disruptive research teams as opposed to trying to identify disruptive research teams. I believe that these tasks are subtly different, and require slightly different instruments. So whenever possible I tried to identify the underlying factors, as opposed to the surface level appearances. For instance, BR list “distinctive culture” as an important factor and all teams I studied did seem to share a special atmosphere, reminiscent of early-stage start-ups and similar to the description HAS provide for “task orientation”: They were all fairly exclusive and isolated groups, sometimes even secret, which bred the sense of being on a special and important mission. Often, they developed their own idiosyncratic terms and rituals. They all shared extremely high standards for their work, a commitment to having the best idea prevail, a joint sense of ownership for the entire project, and an openness to counterintuitive and weird ideas. The teams also seemed to have enjoyed themselves immensely despite the extremely hard work. However, I think that’s largely an epiphenomenon of all the other factors being in place. So I decided not to list it.

The factors I do list almost certainly interact with each other. Where I have a particular reason to believe that they do so in some specific way, I have tried to point this out throughout the text.

You can compare my list of ingredients to the list drawn up by BR (see relevant section for more details) and the list distilled by Bennis and Biederman in Organizing Genius (see appendix). There are significant similarities.

Excellent researchers

Particularly disruptive research teams always seem to contain a significant number of excellent researchers and even those who are not brilliant are still extraordinarily capable. Teams seem to benefit from cognitive diversity but the evidence for demographic diversity is limited.

Individual qualities

In the case study accounts, the individual capability of the researchers was emphasized again and again. In the words of Bob Taylor (PARC): “Never hire ‘good’ people because ten good people together can’t do what a single great one can do.” Leaders of these groups sought to recruit the best people in their field and were often themselves very capable individuals. At a time when air travel was very expensive, they often flew around the country to persuade particular individuals to join their team. RAND, the Santa Fe Institute, Bell Labs, PARC cultivated relationships with the best departments in the country and organized events to scout talent. This emphasis on hiring the best is also mentioned in BR (as part of “Concentration on recruitment and selection”). HAS did not investigate this factor, so can’t tell us much here.[3] I’m aware that there is some scholarly debate on whether group intelligence is mainly determined by factors governing the interaction of team members or individual intelligence. On my very shallow reading of the evidence, I could go either way.[4] What stood out most to me is that the experimental tasks used in these studies (footnote 4) seem fairly easy and not representative of the challenges faced by disruptive research teams. So I think they do not give a lot of reason to update my prior judgment that individual intelligence matters a lot once tasks become sufficiently hard. Given this and the evidence from the case studies, I’m very confident that even perfect group interaction cannot make up for individual capability below a certain level. It’s not clear to me where this level is exactly, but I have a hard time believing it’s below the 80th percentile of Ph.D. holders in a field like physics; presumably much higher for less rigorous fields.

I have not looked in detail at the evidence on what exact capabilities make for an excellent member of disruptive research teams. Based on my impressions from reading about these groups, high general mental ability seems to matter a lot. Other qualities might be extraordinary curiosity and willingness to collaborate. Bennis claims they tend to be “deep generalists”[5] as opposed to specialists. This fits my impression from the case studies. Often, they seem to be young and optimistic.

Diversity

There is some evidence that cognitive diversity across team members matters for disruptive teams. HAS find a modest correlation of job-relevant diversity on team-level innovation (ρ=.240, 95% CI=[.044, .436])[6]. BR seem to agree that this is a beneficial factor. I also had this impression from the Santa Fe Institute in particular. I’m uncertain with regard to the other teams though. So overall I’m not very confident in this conclusion. Demographic diversity does not seem to be very important. Neither HAS nor BR find a positive relationship. The case studies support the opposite conclusion if anything. However, one has to bear in mind the structural disadvantages faced by women and minorities at the time, in particular in the professions and environments that these groups operated in. So I'm not drawing any conclusions from this fact.

Shared direction

Disruptive research teams seem to benefit from a clear vision that describes the kind of change they want to affect in the world. My best guess is that it’s not enough to put bright people in a room together; they need a joint purpose, and likely they ended up in that same room because of that.

At the highest level, a shared purposeful vision gives direction to the group. Here are a few example visions from the case studies:

Since visions tend to be very abstract, they can be complemented by more concrete goals. However, I suspect that this is likely more difficult for disruptive teams focused on making new discoveries.

Purposeful vision

The evidence is clear that a clear vision catalyzes innovation. Both BR and HAS find that vision is important for productive research teams. HAS find the strongest correlation of all factors they looked at (ρ=.493, 95% CI=[.355, .631]), with the main effect on the level of the team, not the individual. Every research team I looked at also had a strong vision. Almost all of them had anecdotes about the effects of the vision on individual team members, both before and after joining the team. It also fits my intuition about effective teams in general. The Institute for Advanced Study (IAS) might be a counterexample. They literally just put the brightest minds of the generation into the same building, and they still seem to have a lot of insights. I vaguely remember the lack of shared vision being put forward as a criticism of the IAS, but I could not find the source for this.

A vision seems to serve multiple functions. First, it seems to coordinate the efforts of everybody on the team. While each member works very autonomously, the vision should serve as the yardstick for deciding which questions or projects are worthwhile. Second, it instills a sense of purpose in each team member, i.e. it should make clear to everybody why the work and “sacrifice” is worth it.[7] Third, and relatedly, it serves as a recruitment tool, attracting those who want to make that vision a reality.

I’m uncertain what exactly makes for a compelling vision. My impression is that prosocial is better than not prosocial, concrete is better than vague, and urgency helps. I’d expect the extensive literature on setting a vision and mission from a nonprofit or business perspective to be at least somewhat helpful for answering this question.

Concrete goals

When it comes to concrete organization-wide goals, the evidence is mixed. HAS find a modest correlation for goal interdependence, the degree to which team members have to rely on each other for achieving their goals (ρ=.276, 95% CI=[.118, .434]). BR mention this when discussing the factor “clear goals that serve a coordinating function”. The case studies give a mixed impression: Skunk Works and Los Alamos Laboratory clearly would have scored very high on this dimension. PARC, RAND, and the Santa Fe Institute might have done so for some of its projects. I find this hard to answer for Bell Labs and Kahneman & Tversky. Overall, the evidence seems to suggest to me that this type of goal-setting is beneficial when feasible but not as important as a vision and it might simply not be possible for blue-skies research. Engineering challenges seem to be probably particularly well-suited for this. Perhaps group projects with very concrete output goals (and deadlines), e.g., drafting a comprehensive research agenda, can still capture some of these benefits.

Leadership

Leadership likely has an outsized impact on how productive and disruptive a research group is. BR list leadership as an important factor, pointing out that leaders have a large influence on many other group factors, including many listed here. So to the extent that these factors matter and are a function of leadership, leadership matters. This makes a lot of intuitive sense and is echoed in the case studies. The leaders received a lot of attention and were credited to a significant extent with the founding and success of the group. I should note, however, that individual stories obviously lend themselves particularly well to the narrative style of the case study accounts. HAS did not investigate this factor, so doesn’t provide any evidence either way. My conclusion that leaders to exert significant influence is driven to a large extent by the intuitive plausibility with the evidence supporting this view.

Leaders need to be able to shape the factors listed here in a positive way. According to BR, the best leaders have a research background, even if they don’t do a lot of research themselves in their leadership role. This seems to have several advantages: earning them the necessary respect from the team; allowing them to give substantive support to their research staff; utilizing their research network for recruitment and collaborations; articulating a compelling vision. This fits with the case studies I looked at. Bob Taylor at PARC was the only leader who did not have any research experience. While he still seems to have been steeped in the research, I read repeatedly that his lack of research experience did hold him back.

I’m very uncertain about other beneficial qualities or skills. Some of the case studies stressed the importance of leadership when it comes to mediating conflict. Bob Taylor at PARC apparently was very good at this. He would often encourage team members to move from what he called “Type 1 disagreements” to “Type 2 disagreements”. In the latter, each side passes the Ideological Turing Test of the other side. At the Los Alamos Laboratory, Robert Oppenheimer successfully mediated many interpersonal conflicts which were jeopardizing the success of the mission. My impression is that such conflicts might be more common in disruptive research teams because they tend to attract individuals who are disagreeable and the groups themselves tend to be high-intensity environments. The importance of this seems to depend on the specific team in question to a large extent.

For groups which are part of some larger organization, in particular one which is not obviously aligned with the goals of the group, a second administrative leader can apparently be important. Their role is to continuously defend the relative independence of the research team from the larger organization, secure resources, and fend off the creeping bureaucracy. They build connections necessary for disseminating the research. This is solely based on some of the case studies. Examples are Leslie Groves at the Manhattan Project, Jerry Elkind at PARC, and Frank Collbohm (together with Curtis LeMay) at the RAND Corporation. Overall, this strikes me as a useful division of responsibility in some cases.

No inconveniences

It seems very, very plausible that a research team is more likely to realize its full disruptive potential if the researchers do not have to do anything but research. So it’s important that the resources necessary for any research projects are abundant and easily accessible. That includes technical and operational support staff as well as learning opportunities. Researchers should be freed from trivial or bureaucratic tasks as much as possible. That being said, all of this probably won’t transform a mediocre group into an excellent one.

This makes a lot of prima facie sense and is backed up by the evidence as well as my impression of academics complaining about having to spend time on bureaucracy, teaching, writing grant applications and reports. BR list “accessible resources” and “assertive participative governance” as important factors, the latter referring to researchers having significant say over the rules that govern them. HAS find a moderate to strong correlation for “support for innovation” which includes both material and immaterial support (ρ=.470, 95% CI=[.407, .533]). It was also a theme of the case studies I looked at: while the environment that the groups worked in was often not very comfortable, they still had everything they needed or sought out ingenious ways of getting what they needed. RAND decided to open their office 24/7 to accommodate different working schedules. The team at PARC built their own PDP-10 when Xerox prevented them from buying one. Leslie Groves worked tirelessly to get all the equipment that the team in Los Alamos needed, including expensive early computers. It’s perhaps worth noting that Feynman has criticized this kind of freedom in the context of the Institute for Advanced Study (IAS). However, I don’t give this much weight without having looked at the IAS in more detail. There is also the general notion that constraints stimulate creativity but the overall evidence seems to point in the other direction.

Autonomy & self-organization

Within the constraints of the vision (and organizational goals), individual researchers in disruptive groups seem to thrive when given a large degree of autonomy. That means there is little formal organizational structure and they’re allowed to work and collaborate in a largely self-directed manner. Instead of having metrics or incentives, it seems to work best to encourage researchers to set their own goals and to give them considerable freedom to work outside of usual incentive structures like “publish or perish” or quick commercialization.

Organizational structure

Collaboration seems to happen organically as opposed to being imposed from the top. So researchers are usually not restricted to work on particular projects, but given the freedom to flock to the most interesting and important work. Correspondingly, the hierarchy of the team is often flat, with all researchers reporting to the leader of the team. This seems to be most easily implemented by sufficiently small teams (see section on small team size).

BR list “decentralized organization” and “assertive participative governance” as important factors. The former refers to flat hierarchies and self-organization through peer interaction. The latter refers to researchers being involved in organizational decision-making. “Decentralized organization” in particular seems to support my characterization. Out of the concepts investigated by HAS “task orientation” comes closest to what I have in mind. Among other things, it is supposed to measure “task reflexivity[,] which refers to the process in which the team reflects upon the team’s objectives, strategies, and procedures”. However, this is only tangentially related and task orientation includes a few other constructs, so I only consider this very, very weak evidence.

At first glance, the case studies paint an ambiguous picture. The Santa Fe Institute and PARC were non-hierarchical and self-organized in the way I described above. I’m uncertain about RAND. Researchers seemed to have had a lot of freedom to accept and reject assignments and could work on things they generally thought interesting. There was some department structure from what I can tell. Overall, my impression is that they had a large degree of autonomy. Bell Labs, Skunk Works, and Los Alamos Laboratory had fixed teams and hierarchies. However, Bell Labs and Los Alamos Laboratory both had over 3,000 employees, making some kind of hierarchy inevitable. Still, Bell Labs famously gave lots of freedom to its basic science teams to pursue research as they saw fit. They also encouraged cross-project collaboration from what I can tell. So viewed at the right level, Bell Labs still exemplified this attribute to some extent. Los Alamos Laboratory and Skunk Works were both engineering projects with very tangible goals that required a lot of coordination to be achieved. My best guess is that a hierarchical structure makes more sense in such cases. Flat hierarchies also seem to prevent status conflicts between members, something that Los Alamos Laboratory and Bells Labs often struggled with. One account of the Santa Fe Institute claims that one downside of this approach is that responsibility is sometimes diffused which can lead to delays or suboptimal processes. Overall, my assessment is that disruptive research teams tend to do better with flat hierarchies and a lot of autonomy if they are sufficiently small.

Metrics, goal-setting, and incentives

I did not find much evidence on the merit of metrics and individual goal-setting. According to BR, goals are useful and best set by researchers themselves in line with the vision and in collaboration with leadership and peers. However, they also point out that excessive autonomy can be detrimental, for junior researchers in particular. They describe this as a state in which the organization and its leadership provide no organizational goals or research direction. No case study account mentioned metrics or said anything on goal-setting. It’s noteworthy, however, that PARC leadership demanded exemption from commercialization pressures for at least five years from Xerox. They argued that short-term incentives would prevent work on more disruptive innovations. Instead, they wanted the lab to be evaluated after ten years. My impression is that academic tenure serves a similar function. It exempts academics from the pressure to conform and deliver results quickly (“publish or perish”) such that they are free to work on ideas that are controversial or take a lot of time to develop. After briefly looking for more evidence, the only source on this question I could find seems to suggest that metrics or incentives are not suitable for basic research. In some cases, individual goals or milestones might still make sense though. This also summarizes my all-things-considered view fairly well.

Spaces for interaction

Despite the importance of autonomy, there is good evidence that internal communication is beneficial for productive research teams. BR list access to human resources and “frequent communication”, which includes internal and external communication, as important factors. HAS also find a moderate correlation for “internal communication” (ρ=.358, 95% CI=[.228, .488]). I got a similar impression from the case studies. Interactions seem to have been important for improving ideas, if not for having them (also see section on small team size). The teams spent a lot of time together talking about their research, also in informal settings. It’s hard to tell how helpful this really was, but it seems plausible.

I’m not sure what the right kinds of interactions are. Since collaboration and exchange are not imposed (see section on autonomy & self-organization), the right interactions are supposed to emerge organically anyway from what I can tell. So groups need to create spaces that allow for these exchanges to occur. The most basic ones are a shared physical space and shared “psychological spaces”. I’m less convinced of policies designed to force interactions but they might be helpful in some cases.

Shared physical space

It seems to be important that researchers share the same physical space. Many of the groups I looked at put emphasis on having their group in the same location, even designing the office space such that researchers were more likely to interact with one another (e.g. RAND, Bell Labs, Santa Fe Institute). BR also find that physical proximity likely plays a role. I want to note that all this evidence comes from a time when electronic communication was far less advanced. However, I do think that physical proximity has benefits that are extremely hard to recreate electronically, serendipitous meetings and exchanges for instance.

Shared “psychological spaces”

In addition to physical proximity, there are also several hints that point toward the benefits of shared “psychological spaces” for lack of a better term. BR’s emphasize interactions with “conceptually close” peers, i.e., people who inhabit a similar intellectual space. HAS find that goal interdependence, which refers to the degree that team members rely on each other for achieving their goals, also correlates moderately with innovation (ρ=.276, 95% CI=[.118, .434]). They hypothesize that this interdependence leads to engagement, collaboration, and mutual feedback, which in turn stimulates idea generation. Overall, I find it plausible that interactions are more likely if researchers are intellectually or psychologically entangled in some way. However, I’d expect this to follow naturally to a large extent simply from having a shared vision.

Forced interaction

There are also examples of concrete policies for facilitating interactions: Bell Labs had an “open door” policy, i.e., it was expected of even the most senior researchers to engage with others if they had questions or required their expertise. Gertner points to several instances where this led to ideas that otherwise would not have happened. PARC had their famous “Dealer Meeting” each Tuesday morning. Bob Taylor would start with 15 minutes of housekeeping, followed by 45 minutes led by either an internal researcher or a guest. They could use this time however they saw fit, hence the name. Usually, somebody would present an idea and invite feedback. This was the only mandatory meeting at PARC and the accounts I read claim that this provided space for fruitful idea exchange and feedback. Overall, the evidence remains anecdotal. Such policies are probably useful in some cases and, therefore, worth experimenting with.

Psychological safety

Psychological safety describes the feeling that voicing controversial ideas or dissent will not cause abandonment or loss of status. This seems to be an important factor for making interactions between researchers particularly fruitful. Unfortunately, it’s not clear to me how exactly it is created.

There are several findings that gesture in this direction. BR seem to include this in their construct of “positive group climate” which they list as an important factor. HAS investigated “participative safety”, which includes “intragroup safety” (very closely related to psychological safety) as one of two subcomponents, and find a small but significant correlation, which does not seem to generalize well across contexts though (ρ = .148, 80% Credibility Interval = [–.113, .410], 95% Confidence Interval = [.080, .216]). They don’t have data on intragroup safety specifically. HAS also investigated cohesion which they describe as “interpersonal attraction, task commitment, and group pride”. They hypothesize that this attachment to the group also contributes to the psychological safety necessary for innovation. For this construct, they find a moderate correlation with innovation (ρ=.307, 95% CI=[.179, .435]). So the academic evidence seems to support psychological safety as an important factor. This is echoed in the case studies which often described the groups as very tight-knit and marked by deep feelings of mutual respect (e.g. RAND, PARC, Kahneman & Tversky, Los Alamos Laboratory).

This open and safe space can also lead to conflict though. Some of the groups I looked at seem to have had very harsh intellectual atmospheres. Ideas would be publicly eviscerated and subjected to grueling criticism (e.g., at PARC and RAND).[8] However, HAS seems to suggest that neither task conflict (ρ=.067, 95% CI=[–.134, .268]) nor relationship conflict (ρ=–.092, 95% CI=[–.252, .068]) correlate with innovation. So it’s not clear to me what role this kind of conflict plays and how to reconcile it with psychological safety since it seems to point in the opposite direction. One explanation could be that such conflicts are helpful in teams which exhibit deep mutual respect amongst their members and destructive in all other teams.[9] Even if psychological safety and conflict cannot be reconciled, I find the evidence for psychological safety more compelling.

However, it’s not clear to me how this shared psychological state is created. I should note that I have not looked at independent evidence on this question. Based on the above evidence, group pride and public knowledge of everybody’s commitment to the shared vision might matter. I also find it intuitively plausible that psychological safety requires the self-esteem of the members not to be bound up with group membership. Otherwise, conflict might trigger insecurity and prevent members from speaking up. The case studies might also be instructive: At PARC, everybody was involved in the hiring process. Applicants had to give a talk in front of the entire team and hiring decisions were made near-unanimously. This might have created a filter that only people which had the respect of everybody could pass. There is probably also a role for leadership: At Los Alamos Laboratory, Oppenheimer is credited with making everybody feel like their work was vital to the success of the joint endeavor. At Skunk Works, Johnson, their leader, gave everybody who joined the sense that they had been handpicked and the best person for the job, instilling in them a sense of excellence.

Small team size

I’m conflicted about team size. Different factors seem to push in opposite directions. I weakly believe that the ideal team size for disruptive research teams is such that team members still know each other sufficiently well for them to feel comfortable voicing controversial ideas and dissent. I’d suspect this to be less than 15 people, but would not be very surprised if this was number was around 100 after all.

The evidence on this seems to be somewhat conflicting. BR find that research productivity tends to increase with the size of the group. HAS find that team size correlates positively with team innovation (ρ=.259, 95% CI=[.157, .360]) and weakly negatively with individual innovation (ρ=–.101, 95% CI=[–.253, .051]), the overall correlation being weakly positive (ρ=.172, 95% CI=[.078, .266]). However, neither looked at disruptive teams in particular. As I mentioned earlier, Wu, Wang & Evans present solid evidence[10] that disruptive teams tend to be smaller.[11] They find this at every level, i.e., teams of nine tend to be more disruptive than teams of ten, teams of two more so than teams of three, and individual researchers more so than teams of two. However, their notion of “team” refers to the number of co-authors on an article or on a patent, which is different from how I have conceptualized it so far. So they don’t capture the influence that a team, more broadly considered, might have on individual researchers. The other factors I have looked at in this post seem to suggest that there are benefits to interaction and exchange when done right.

I’m uncertain what conclusions to draw from the case studies. Members of PARC, RAND (in the early days), and the Santa Fe Institute (in the early days) seem to have cherished the intimate atmosphere of being part of a small team. The small size seems to allow for a higher level of intimacy which could be important for fruitful interactions. This was certainly also the case for Kahneman & Tversky. Skunk Works also tried to keep their team really small, in part to reduce bureaucratic overhead. Bell Labs and the Los Alamos Laboratory were much larger (>3,000 employees), and both managed to be incredibly innovative at the same time. However, as I mentioned before, it’s not clear that one should count these large organizations as single teams, Bell Labs in particular. Their units were much smaller, two to five people from what I could tell based on the sources that I read.

Overall, I would tentatively conclude that smaller teams tend to be more disruptive than larger teams. This is mainly driven by the Wu et al. study which I would weigh more heavily than BR and HAS, given that they’re more focused on disruption. However, I do not think that this evidence warrants the conclusion that researchers should work on their own without a team or that teams of two are better than teams of three. The other factors I have looked at seem to suggest that there are benefits to being part of the right team. However, I suspect these returns probably decrease at some point before turning negative when the team gets too big as indicated by the case studies. My rough model of this is the following: researchers are more likely to have disruptive ideas on their own, but benefit from the exchange with others up to the point where team size prevents the level of intimacy required for fruitful exchange and cumulative skepticism toward disruptive ideas prevents people from voicing or pursuing them. I don’t know where this point lies. Dunbar’s number might give an informed upper bound but I’d intuitively expect it to be much lower. My best guess is around 15, but that’s really just a wild guess.

Impactful theory of change

For research to have an impact on the world, it has to influence people outside of the team. Since this does not happen automatically, it’s important to develop a theory for how this will come about, a theory of change, and to execute it well. As Steve Jobs put it: “Real artists ship.” Having an innovative idea or product is not sufficient if you are unable to bring it to market, i.e., shipping it. I find it plausible that the theory of change acts as a force multiplier for the research, i.e., it may reduce its impact to 0, amplify its impact hundredfold (compared to other potential theories of change), or even turn its impact negative in some cases. Following this model, one should try to find the theory of change with the highest expected value.

For research groups, a theory of change determines what change they want to bring about: changing the paradigm in their field, improving policy, helping develop a technology? It means figuring out who they need to influence to affect that change: other academics in their field, policy-makers, companies? It means figuring out how their research will have the best chance of reaching and influencing this group: publishing academic papers in journals, giving talks at conferences, networking, sending accessible summaries to the relevant individuals with an offer to meet? Gesturing vaguely toward the marketplace of ideas is not a theory of change (see below for some examples for how the groups from the case studies succeeded or failed to bring about change or read this article by Aaron Schwartz).

For some situations there exist theory of change templates. For instance, for an academic group trying to influence their field, the most prestigious journals and conferences of the field are probably their best bet. This does not require elaborate planning. I should note, however, that for disruptive research the case might not be as straightforward even in this case since the old paradigm usually does not go quietly. For many situations, it is not as clear-cut to begin with and it probably makes sense to develop a clear theory of change. For research teams dedicated to effective altruism, this is probably even more important since they presumably care more about changing the world than other research groups.

I have a strong intuition that this matters a lot. It seems obvious to me that how disruptive a particular insight is, depends a lot on who learns about this insight in what way. This can differ greatly between different theories of change and their execution. The evidence is mixed. Neither BR nor HAS mention this as an important factor: HAS did not investigate it in the first place, and since BR did not start out with any hypotheses, it’s unclear why they didn’t include it. The case studies, however, paint a very clear picture. From the start, RAND wrote their reports as contractors for the US government, the military in particular. Their insights reached the right people and carried significant weight. The Santa Fe Institute purposefully set itself up as a visiting institution so that visiting scholars would spread the ideas of the SFI at their home institutions upon their return. Xerox famously failed to exploit the breakthrough technologies developed by PARC and were ultimately scooped by Apple and Microsoft. Bell Labs had an established development pipeline from basic science to development to manufacturing which translated their scientific insights into mass-produced components of the telecommunications network. Skunk Works and the Los Alamos Laboratory had clear products they were working toward on behalf of the US government. Kahneman & Tversky seem to have thought strategically about the journals in which to publish, how to frame their discoveries, and how to reach people outside their narrow field. So overall, I remain convinced that this matters.

External input and feedback

High-quality communication with external stakeholders seems to matter. Based on the case studies, high-intensity in-person exchange with people working on similar problems seems to be most valuable.

BR list communication with external stakeholders as an important factor.[12] HAS find a moderate to strong correlation (ρ=.475, 95% CI=[.380, .570]). Unfortunately, neither make very clear what kind of communication at which frequency is desirable. The closest we get is BR favoring communication with peers at other schools or institutions (as opposed to colleagues from other departments at the same institution). The case studies provide more detail: PARC, for instance, regularly invited guests to host Dealer Meetings (see section on spaces for interaction). Despite being very secretive, RAND and the Los Alamos Laboratory had a roster of prominent advisors and consultants. They seem to have visited regularly to help with specific challenges. The Santa Fe Institute is a full-blown visiting institution with rotating scholars. Bell Labs’ open door policy facilitated personal communication between departments. Most of these seem to point toward short high-intensity in-person interactions with people working on similar problems. However, it’s plausible that other forms are simply not reported because they’re less noteworthy. Other forms of communication (e.g., video chat, voice memos, real-time chat) were probably also less accessible during the relevant times. In contrast to the other teams, Skunk Works and Kahneman & Tversky seem to have been very insular. I’m very uncertain about the latter though.

The evidence from the case studies broadly fits my model of why and how much communication with external stakeholders is valuable but I’m still not confident that this is the right view. According to this model, there is a balance to be struck between maintaining an atmosphere for counterintuitive ideas to thrive and receiving outside input and feedback from the outside (also see section on small team size). Put differently, excessive outside perspectives can smother new ideas while too little exposure to outside perspectives might stifle creativity or leave existing ideas unconstrained or unrefined. My intuition is that regular high-intensity exchange balances this trade-off best. I also suspect that the value of outside perspectives becomes less important to the extent that there is fast, unambiguous, real-world feedback: regardless of what other people thought, the planes developed by Skunk Works either took off or they didn’t; the atomic bombs either exploded or they didn’t; the transistors either worked or they didn’t.

There are also non-epistemic factors for why interaction with outside stakeholders might matter. Since they likely improve the network of the team, they may help increase the influence of one’s ideas (see section on impactful theory of change) and aid recruitment and collaboration.

Immaterial rewards

Salary does not seem to matter much beyond market rate. Instead, immaterial rewards like praise and perceiving one’s work as impactful seem to matter more. BR support this view. This also fits with my impression from the case studies and my prior judgment. People who signed up for these teams were rarely interested in making a lot of money but strongly motivated to make a difference as part of a great group. They often describe this time as the most fulfilling of their entire lives, hinting at what HAS call “cohesion”. I’d expect non-monetary rewards to be a strong correlate of a purposeful vision (and culture), and, therefore, a dependent factor one cannot easily “manufacture”.

Learnings for the Effective Altruism Foundation

Based on the findings above, these are the most important takeaways for our research team at the Foundational Research Institute (FRI) as I see them:

In-depth look at the evidence

Academic Literature

Bland & Ruffin (1992): Characteristics of a Productive Research Environment. Literature Review

Methodology

Key findings

They identify 12 group-level characteristics of high-performance research groups. Below I provide a summary[13]:

1. Clear goals that serve a coordinating function

2. Research emphasis

3. Distinctive culture

4. Positive group climate

5. Assertive participative governance

6. Decentralized organization

7. Frequent communication

8. Accessible resources, particularly human

9. Sufficient size, age, and diversity of the research group

10. Appropriate rewards

11. Concentration on recruitment and selection

12. Leadership with research expertise and skill in both initiating appropriate organizational structure and using participatory management practices

Strategies for establishing in maintaining these characteristics

They suggest several policies/actions for:

Hülseger, Anderson & Salgado (2009): Team-Level Predictors of Innovation at Work. A Comprehensive Meta-Analysis Spanning Three Decades of Research

Methodology

Results

Vision

External communication

Support for innovation

Task orientation

Internal communication

Cohesion

Goal interdependence

Team size

Job-relevant diversity

Participative safety

Task conflict

Task interdependence

Team longevity

Relationship conflict

Background diversity

Case Studies

RAND Corporation

Main source: Abella: Soldiers of Reason

Brief summary: RAND was founded in 1948 as a contract research organization for the US Air Force. Since then they have developed very close ties to the entire US national security infrastructure. Their analyses and reports have significantly shaped US defense policy during the Cold War and the Vietnam War. By now their research portfolio also includes areas unrelated to national security and they provide services for non-government organizations. Most information is from the first three decades of RAND.

What they achieved: RAND had a profound influence on the early nuclear strategy of the US and made large academic contributions as a byproduct of their work. However, the ultimate value of this work is contested. Abella argues that they promoted a callous attitude toward civilian casualties, had a misguided view of the Soviet Union, and their analyses were ignorant of human factors. I don’t have the expertise to evaluate the ultimate merit of Abella’s claims. Their inclusion is warranted on the basis of their large influence.

What I learned about disruptive research teams:

Santa Fe Institute

Main sources:

Brief summary: The Santa Fe Institute (SFI) is an independent, nonprofit theoretical research institute dedicated to the multidisciplinary study of complex adaptive systems, including physical, computational, biological, and social systems. It was founded in 1984 by George Cowan and a number of scientists also affiliated with the Los Alamos National Laboratory (which had grown out of the Manhattan Project). All of them shared skepticism about the reductionism present in many scientific disciplines at the time. It has very few permanent research staff but hosts a great number of visiting scholars, workshops, and summer schools. I looked mainly at the first few years of work at SFI.

What they achieved: The SFI pioneered the study of complex adaptive systems which has since evolved into a major scientific field. The two accounts I read agree that this would have happened regardless, but that the SFI sped up this development significantly. Individual achievements of the SFI are harder to isolate and attribute since it relies so heavily on visiting scholars, and its main contribution is arguably fusing similar ideas from different individuals into a field of inquiry.

What I learned about disruptive research teams:

Palo Alto Research Center (PARC) at Xerox

Main sources:

Brief summary: The Palo Alto Research Center (PARC) is a research group founded by Xerox in 1970 to bring the company into the emerging computing market. Its main contributions to the field were made during the 1970s, after which many of the most scientists moved on to other companies. During that time it had three labs, the two prominent being the Computer Science Lab (de facto led mainly by Bob Taylor) and the System Science Lab (where Alan Kay provided the main vision). This is also the time I focused on.

What they achieved: They fully developed personal distributed computing as we know it today, mainly in form of the Xerox Alto (including computer-generated bitmap graphics and the WYSIWYG text editor Bravo), many aspects of the modern graphical user interface (GUI) and the Ethernet. They also pioneered laser printing and popularized object-oriented programming via Smalltalk. It’s worth noting that a number of relevant ideas and technologies already existed, but PARC managed to fuse them together into a coherent whole. In the end, Xerox and PARC failed to bring a lot of these innovations to market (with the exception of the laser printer).

What I learned about disruptive research teams (Dominic Cummings also has a post about this and ARPA-style funding):

Bells Labs

Main sources: Gertner: The Idea Factory. Bell Labs and the Great Age of American Innovation

Brief summary: Bell Labs served as the research lab of AT&T during their monopoly years as the sole telephone network provider, mainly between 1925 and 1982. They were charged with developing new technologies to improve the (tele)communications network operated by AT&T. From the beginning, they had a strong commitment to basic research.

What they achieved: Research conducted at Bell Labs can be rightfully credited with ushering in the information age as we know it today. Similar developments would probably have been made in any case, but my impression is that Bell Labs did speed up these innovations considerably. They did pioneering work in solid-state physics, culminating in the development of the transistor, a crucial technology for digital computing. Shannon developed information theory while at Bell Labs and did groundbreaking work on cryptography. They also developed radio astronomy, the laser, and the charge-coupled device (CCD), contributed to fiber-optic technology, and built the first photovoltaic cells. They contributed to early computing: designing the Unix operating system and the programming languages C, C++, and S. They are responsible for sending the first telecommunications satellite into orbit and developing cellular telephony. Nine Nobel Prizes have been awarded for work completed at Bell Laboratories. The Turing Award (also known as the Nobel Prize of computing) has been won three times by Bell Labs researchers.

What I learned about disruptive research teams:

Skunk Works (Lockheed)

Main sources:

Brief summary: Skunk Works is the informal name of the special project division of Lockheed Martin. It was founded in 1943 by Kelly Johnson, who later handed over the reins to Ben Rich. It operates as a contractor to the US Air Force and the CIA.

What they achieved: Over many decades they developed extremely advanced and original military aircraft that provided the US with crucial strategic advantages during the Cold War, primarily via improved observation capabilities. They’re famous for extremely lean and fast development and production. In 1943, they built America’s first jet-propelled military aircraft, the P-80 Shooting Star. In 1955, they built the U-2 spy plane for the CIA, intended to conduct observation missions over the Soviet Union. It was able to fly at altitudes which made it impossible to reach by Soviet interception fighter planes. Its overflights led to the discovery of missiles on Cuba and were responsible for dispelling both the “bomber gap” myth and the “missile gap” myth. After U-2 missions became impossible, the Skunk Works designed the SR-71 Blackbird during the 60s, a Mach-3+ aircraft that flew at such high speeds and altitudes that it was again impossible to intercept. The engineering challenges for this plane were considerable. In 1976/77 they developed the first stealth aircraft which seems to have provided a considerable strategic advantage during the first Gulf War.

What I learned about disruptive research teams:

Los Alamos Laboratory (Project Y)

Main sources:

Brief summary: The Los Alamos Laboratory, also known as Project Y, was a secret laboratory located in Los Alamos, New Mexico. It was established by the Manhattan Project during World War II to design and build the first atomic bombs. I focused on the time of Robert Oppenheimer’s tenure as its first director, from 1943 to December 1945.

What they achieved: They built the first atomic bombs within just 30 months. During this time, they developed two functional and distinct bomb designs: (1) “Little Boy” was a gun-type fission weapon using enriched uranium. (2) “Fat Man” was an implosion-type fission weapon using plutonium. Concurrently, with far fewer resources, they did early research on the so-called “Super”, a fusion bomb. After the war, the continuation of this research would lead to the development of hydrogen bombs.

What I learned about disruptive research teams:

Kahneman & Tversky

Main sources: Lewis: The Undoing Project. A Friendship That Changed Our Minds

Brief summary: Kahneman and Tversky are two Israeli psychologists who successfully collaborated over several decades.

What they achieved: They successfully challenged the rational choice model of human behavior in psychology and economics through their work on heuristics, biases, and prospect theory. They kickstarted behavioral economics through their influence on Richard Thaler and are responsible for similar developments in other fields such as medicine, sports, and business. Kahneman received the Nobel Prize in economics for their work.

What I learned about disruptive research teams:

Appendix

Bennis: Characteristics of “Great Groups”

In Organizing Genius, Bennis distills fifteen characteristics of what he calls “Great Groups”. He looked at the Disney Company, PARC, the Bill Clinton election campaign of ‘92, Skunk Works (Lockheed), Black Mountain College, and Los Alamos Laboratory (the “Manhattan Project”). This does not only include research teams but is still somewhat informative. These are my notes from the last chapter in which he summarizes the lessons he draws from them.

  1. Greatness starts with superb people.
  1. Great Groups and great leaders create each other.
  1. Every Great Group has a strong leader.
  1. The leaders of Great Groups love talent and know where to find it.
  1. Great Groups are full of talented people who can work together.
  1. Great Groups think they are on a mission from God.
  1. Every Great Group is an island—but an island with a bridge to the mainland.
  1. Great groups see themselves as winning underdogs.
  1. Great Groups always have an enemy.
  1. People in Great Groups have blinders on.
  1. Great Groups are optimistic, not realistic.
  1. In Great Groups, the right person has the right job.
  1. The leaders of Great Groups given them what they need and free them from the rest.
  1. Great Groups ship.
  1. Great work is its own reward.

  1. Note that this is my loose definition. There is significant scholarly debate on what an appropriate and rigorous operationalization of the term should look like. ↩︎

  2. Thanks to Max Daniel for bringing this group to my attention. This report in the 1999 edition of the Notices Of The American Mathematical Society paints a picture very similar to the other groups I investigated. However, I did not find more in-depth accounts. ↩︎

  3. What they call “task orientation” might gesture in this direction, but the connection is thin. ↩︎

  4. Woolley et al. (2010) find that the average social sensitivity of group members, the equality in distribution of conversational turn-taking, and the proportion of females in the group are most predictive for group intelligence. In a replication attempt, Bates & Gupta (2017) find that individual intelligence of the team members account for most of the variance in group intelligence (80%). Woolley et al. have almost twice the number of participants, but the replication is probably less likely to be the result of publication bias. ↩︎

  5. To my knowledge Bennis coined this term to refer to people who have acquired considerable expertise in multiple domains and can comfortably apply the tools of these disciplines whenever appropriate. He argues that this allows them to be open to new findings and approaches because they’re less constrained by the received wisdom of their disciplines”, while still being able to apply the best tools they offer to solve the novel problems they face. ↩︎

  6. In this section, I will only list the mean correlation as well as the 95% confidence interval. If you’re interested in further information, please refer to the relevant parts of the next section. ↩︎

  7. There is a case to be made that tribal psychology can be harnessed for this by casting an outside group as an enemy to be defeated (see Appendix). While this might work in some instances, I do not think this is a wise strategy in the long term. ↩︎

  8. This impression is echoed in this article on the harsh cultures of innovative companies. ↩︎

  9. The idea of different intellectual cultures might be relevant here. ↩︎

  10. They analyzed more than 65 million papers, patents, and software products from 1954 to 2014, looking at the relationship between the number of authors and disruptiveness. They don’t provide a correlation coefficient but the findings seem unambiguous, given their notion of disruptiveness. They conclude: “In summary, we report a universal and previously undocumented pattern that systematically differentiates the contributions of small and large teams in the creation of scientific papers, technology patents and software products. Small teams disrupt science and technology by exploring and amplifying promising ideas from older and less-popular work. Large teams develop recent successes, by solving acknowledged problems and refining common designs.” ↩︎

  11. A similar relationship seems to hold for the number of founders of disruptive companies, i.e. fewer founders tend to be better. However, I haven’t looked at this in detail at all. ↩︎

  12. Their factor “frequent communication” includes communication with external stakeholders. ↩︎

  13. Dr. Bill Dunn from the Oxford Learning Institute has also summarized the key findings of this study independently. ↩︎

  14. From the paper: “Credibility intervals indicate whether the corrected correlation can be generalized or whether it is situation specific (i.e., whether it varies between different organizational settings). Thus, this interval conveys information on the variability of individual correlations. From the width of the credibility interval, it can be inferred whether moderators are operating. In the case of both positive and negative mean corrected correlations (ρ), generalizability can be inferred if the credibility interval does not include zero.” ↩︎

  15. I omitted most references for readabillity. ↩︎

John_Maxwell_IV @ 2019-05-30T05:18 (+33)

I asked my father, who has spent the past 40 years at Xerox PARC and worked with Bob Taylor, what he thought of this post. He wrote:

That all seems reasonable to me. My guess is that the most important factors are great people and a great leader. One of my co-workers, who was involved with starting a research center in France said “A people hire A people. B people hire C people”. So, the first few people that you hire are really important.

I think that the main job of the leader is to keep people happy and focused. Most of my managers have been really good leaders.

I also think that being co-located is very important. When I am out of touch with my co-workers, I tend to lose motivation.
...
BTW, one of the reasons that the best leaders usually have a technical background is that it is hard to identify the very best people without it. That is why non-technical companies have trouble hiring good programmers, and conversely why the best tech companies were founded by people with a technical background.

Another thing I remember him once mentioning to me is that PARC bought its researchers very expensive, cutting-edge equipment to do research with, on the assumption that Moore's Law would eventually drive down the price of such equipment to the point where it was affordable to the mainstream.

He's willing to answer questions.

SiebeRozendal @ 2019-06-04T09:52 (+19)
Research teams seem more likely to realize their full disruptive potential if the researchers do not have to do anything but research and have easy access to all the resources they need.

Note that Richard Hamming disagreed with this. He makes the point that restricted resources force creativity, and the best breakthroughs come through this creativity. I think he has got a point, but I'm not sure what they all-things-considered conclusion should be. I think the 'constraints breed creativity' applies more to the tools people work with, and other constraints like teaching, administrative tasks, and grant applications mostly waste time.

Here's an excerpt form his famous talk on research (for individual researchers, not teams):

What most people think are the best working conditions, are not. Very clearly they are not because people are often most productive when working conditions are bad. One of the better times of the Cambridge Physical Laboratories was when they had practically shacks - they did some of the best physics ever.
I give you a story from my own private life. Early on it became evident to me that Bell Laboratories was not going to give me the conventional acre of programming people to program computing machines in absolute binary. It was clear they weren't going to. But that was the way everybody did it. I could go to the West Coast and get a job with the airplane companies without any trouble, but the exciting people were at Bell Labs and the fellows out there in the airplane companies were not. I thought for a long while about, ``Did I want to go or not?'' and I wondered how I could get the best of two possible worlds. I finally said to myself, ``Hamming, you think the machines can do practically everything. Why can't you make them write programs?'' What appeared at first to me as a defect forced me into automatic programming very early. What appears to be a fault, often, by a change of viewpoint, turns out to be one of the greatest assets you can have. But you are not likely to think that when you first look the thing and say, ``Gee, I'm never going to get enough programmers, so how can I ever do any great programming?''
And there are many other stories of the same kind; Grace Hopper has similar ones. I think that if you look carefully you will see that often the great scientists, by turning the problem around a bit, changed a defect to an asset. For example, many scientists when they found they couldn't do a problem finally began to study why not. They then turned it around the other way and said, ``But of course, this is what it is'' and got an important result. So ideal working conditions are very strange. The ones you want aren't always the best ones for you.

In the same talk he also criticizes the Institute for Advanced Study, claiming that they only produced good output because they recruited the very best, but that counterfactually these researchers would have produced better output.

stefan.torges @ 2019-06-04T17:55 (+6)

This issue is something I am still somewhat confused about. Feynman makes a similar point about the IAS. I also know about a few more anecdotes in line with the "constraints breed creativity" point.

I think the 'constraints breed creativity' applies more to the tools people work with, and other constraints like teaching, administrative tasks, and grant applications mostly waste time.

There might be something to this, but I distinctly recall reading somewhere that having state of the art tools is also crucial for being able to work at the frontier. Without an electron microscope, some research is simply unavailable. (It might also create an incentive to develop an alternative and this is the kind of disruption we're actually looking for.) More powerful computers also seem like a good thing in general. So I'm not sure how to resolve this.

Edit: Also consider the anecdote mentioned by John Maxwell about PARC of course.

Another thing I remember him once mentioning to me is that PARC bought its researchers very expensive, cutting-edge equipment to do research with, on the assumption that Moore's Law would eventually drive down the price of such equipment to the point where it was affordable to the mainstream.

gwern @ 2019-07-21T21:08 (+5)

On the other hand, in that same talk, Hamming pointed out the importance of abundant computing resources:

One lesson was sufficient to educate my boss as to why I didn't want to do big jobs that displaced exploratory research and why I was justified in not doing crash jobs which absorb all the research computing facilities. I wanted instead to use the facilities to compute a large number of small problems. Again, in the early days, I was limited in computing capacity and it was clear, in my area, that a "mathematician had no use for machines." But I needed more machine capacity. Every time I had to tell some scientist in some other area, "No I can't; I haven't the machine capacity," he complained. I said "Go tell your Vice President that Hamming needs more computing capacity." After a while I could see what was happening up there at the top; many people said to my Vice President, "Your man needs more computing capacity." I got it!
Ben_West @ 2019-05-22T22:18 (+18)

Thanks so much for writing this! It looks like a really thorough investigation, and you found more concrete suggestions than I would've expected.

Regarding psychological safety: Google also found that psychological safety was the strongest predictor of success on their teams, and has created some resources to help foster it which you might be interested in.

Milan_Griffes @ 2019-05-22T23:15 (+15)

+1 to the importance of psychological safety.

My (weakly held) view is that the median EA org is weak on psychological safety & also underweights its importance (when reflecting on what to prioritize, org-development-wise).

jandrewrogers @ 2019-07-22T16:37 (+14)

I’ve ended up in the position of explicitly designing these organizations in many contexts (government vs private, secrecy, etc). I’ve also had the opportunity to test some ideas to understand the underlying dynamics in practice. The above collection of observations, while directionally correct in my experience, overlooks or undersells some critical relationships when making effective disruptive research organizations a repeatable engineered result. It would be easy to overfit to the identified characteristics and end up with a mediocre research organization.

Many organizations fail to understand what the role of a visionary leader actually is and the characteristics of an effective one in this context. First, these people are always executing their own vision, not the organization’s, and you fill these roles by finding a leader whose personal vision is already aligned with where the organization wants to go. The organization provides a platform for that person’s vision, it doesn’t impose it. The common practice of having a charismatic person execute someone else’s vision leads to poor navigation of the ambiguity inherent in these visions. Second, they must be viewed as true technical peers by the excellent people they are leading or there will be a lack of trust in that vision. Ideally, you want everyone in the research organization to harbor the suspicion that the visionary leader could effectively do their job if required. Too many organizations try to substitute this with someone that is senior, pedigreed, and credentialed but lacks technical gravitas. Finding appropriate leaders is the most difficult part of building these organizations.

Every person on the team should be selected for a unique, critical expertise that does not overlap with any other person. This naturally creates clear ownership of something important, encourages the team to self-organize organically around their individual strengths, and tends to engender mutual recognition of the value other members of the team. This simple practice mitigates a large amount of adverse social dynamics and politicking that destroys the productivity of teams in practice. Excellent researchers bring their own vision of their expertise and having two people responsible for the same thing leads to conflicts of vision (“ought”) that hinder execution and add little value. Concomitant with that, all members of the team (including leaders) have a requirement to be able to clearly articulate and defend ideas and decisions to anyone in the org that asks. There is an element of psychological safety that comes naturally from being the person best equipped to address questions and criticism.

As a general observation, there also needs to be a forcing function toward achieving a concrete objective in a resource constrained way. Without this, there is little incentive to prune the search space.

Aaron_Scher @ 2023-09-23T00:49 (+3)

I'm curious why you don't include intellectually aggressive culture in the summary? It seems like this was a notable part of a few of the case studies. Did the others just not mention this, or is there information indicating they didn't have this culture? I'm curious how widespread this feature is. e.g., 

The intellectual atmosphere seems to have been fairly aggressive. For instance, it was common (and accepted) that some researchers would shout “bullshit” and lecture the speaker on why they were wrong.

Misha_Yagudin @ 2022-11-23T02:49 (+2)

Interesting thread on early RAND culture: https://twitter.com/jordanschnyc/status/1593294746725756929

Misha_Yagudin @ 2022-10-02T18:22 (+2)

Terri Griffith [thinks](https://econjwatch.org/File+download/1236/UnderappreciatedWorksSept2022.pdf?mimetype=pdf Research Team Design and Management for Centralized R&D is their most neglected paper. They summarize it as follows:

It is a field study of 39 research teams within a global Fortune 100 science/technology company. As we write in the abstract, we demonstrate that “teams containing breadth of both research and business unit experience are more effective in their innovation efforts under two conditions: 1) there must be a knowledge-sharing climate in the team (arguably allowing the team to have access to the knowledge developed through the members’ breadth of experience) and 2) the team leader also has a breadth of research and business experience allowing for the member breadth to be knowledgably managed.” With 13 years perspective, I still find these results valuable and often share them in my innovation management courses.

Peter Wildeford @ 2022-02-25T17:43 (+2)

Is there any information on how large the total staff were at RAND, Santa Fe, PARC, Bell Labs, etc.? And how the staff were distributed into teams, how teams interacted, what roles the staff had, etc.?

Max_Daniel @ 2020-07-29T11:31 (+2)

[ETA: Retracted because you actually cite that paper later.]

Quantitative evidence (publications, citations, etc.) could provide a complementary perspective to these case studies. E.g., this paper seems to confirm your conclusion that small teams are conducive to disruptive research:

Teams dominate the production of high-impact science and technology. Analyzing teamwork from more than 50 million papers, patents, and software products, 1954-2014, we demonstrate across this period that larger teams developed recent, popular ideas, while small teams disrupted the system by drawing on older and less prevalent ideas.

I found this via Fortunato et al. (2018), a review of the quantitative study of science. Another good review is Clauset et al. (2017).

Aaron Gertler @ 2020-05-22T03:22 (+2)

This post was awarded an EA Forum Prize; see the prize announcement for more details.

My notes on what I liked about the post, from the announcement:

Stefan Torges’ “Ingredients for creating disruptive research teams” is a thorough, well-done literature review — one that wouldn’t be out of place in an academic journal, save for an additional section on what Torges’ organization (the Effective Altruism Foundation) took away from the research. While I thought the entire review was excellent, the takeaway section was the part which excited me most; it gives readers who work at research-focused organizations a sense for how they might begin to apply the lessons themselves.

Related: In November, Torges won a Forum Prize for “Takeaways from EAF’s Hiring Round,” which also took readers inside of his organization’s operations. It’s rare that EA organizations offer such a close look at their internal processes, but the more often it happens, the easier it becomes for established organizations to learn from each other, and for newly-founded orgs to get off to a strong start.

nonzerosum @ 2019-07-22T18:00 (+1)

The book "Loonshots" also has useful lessons for anyone running/starting a research team or research lab.

gavintaylor @ 2019-06-07T18:00 (+1)

Very interesting! In terms of requiring a shared interaction space, Shannon Labs was recently trying to set up something similar to Bell Labs in a remote context. I don't think that project even got started, but it would be interesting to know if there were any best practices that can be used to create interactions between remote team members? There are quite a few successful remote only tech companies that have done well, so they might provide some inspiration for cases when having a the whole group onsite isn't feasible.

Jamie_Harris @ 2019-05-27T21:23 (+1)

This was very interesting. There were several aspects I found surprising, such as the apparent importance of collaboration and shared physical space. Thanks for writing and sharing this.

I'm interested in the methodology, since I've also been working on both 1) case studies, which I hope to able to compare between at a later point and 2) a literature review.

How long do you estimate that you spent looking at each of the case studies?

It seems that most are based on a small number of sources. Did you find that reading additional sources changed your views about a particular research team compared to the first source or two that you read? Do you expect steeply diminishing returns from investing more time into digging further into particular case studies?

stefan.torges @ 2019-06-04T17:47 (+2)

How long do you estimate that you spent looking at each of the case studies?

Good question. I'd say on average about 10 hours; some more, some less.

It seems that most are based on a small number of sources. Did you find that reading additional sources changed your views about a particular research team compared to the first source or two that you read? Do you expect steeply diminishing returns from investing more time into digging further into particular case studies?

In my experience, most of the material went back to one or two authoritative accounts of these teams. So there appeared to be little value beyond finding and reading these. I'm not sure how well this generalizes to other case studies though.