Hmm my guess is that you're underrating the dangers of making more easily accessible information that is already theoretically out "in the wild." My guess is that most terrorists are not particularly competent, conscientious, or creative.^[1] It seems plausible and even likely to me that better collations of publicly available information in some domains can substantially increase the risk and scale of harmful activities.

Take your sarin gas example.

sarin gas via instructions that were easily findable in this 1995 article: 

"How easy is it to make sarin, the nerve gas that Japanese authorities believe was used to kill eight and injure thousands in the Tokyo subways during the Monday-morning rush hour?

"Wait a minute, I'll look it up," University of Toronto chemistry professor Ronald Kluger said over the phone. This was followed by the sound of pages flipping as he skimmed through the Merck Index, the bible of chemical preparations.Five seconds later, Kluger announced, "Here it is," and proceeded to read not only the chemical formula but also the references that describe the step-by-step preparation of sarin, a gas that cripples the nervous system and can kill in minutes.

"This stuff is so trivial and so open," he said of both the theory and the procedure required to make a substance so potent that less than a milligram can kill you."

I think it is clearly not the case that terrorists in 1995, with the resources and capabilities of Aum Shinrikyo, can trivially make and spread sarin gas so potent that less than a milligram can kill you, and that the only thing stopping them is lack of willingness to kill many people. I believe this because in 1995, Aum Shinirikyo had the resources, capabilities, and motivations of Aum Shinrikyo, and they were not able to trivially make highly potent and concentrated sarin gas.

Aum intended to kill thousands of people with sarin gas, and produced enough to do so. But they a) were not able to get the gas to a sufficiently high level of purity, and b) had issues with dispersal. In the 1995 Tokyo subway attack, they ended up killing 13 people, far less than the thousands that they intended.

Aum also had bioweapons and nuclear weapons programs. In the 1990s, they were unable to be "successful" with either^[2], despite considerable resources.

1. ^{^}

   No offense intended to any members of the terror community reading this comment.

2. ^{^}

   My favorite anecdote is that they attempted to cultivate a botulism batch. Unfortunately, Aum lab security protocols were so lax that a technician fell into the fermenting tank. The man almost drowned, but was otherwise unharmed.

Thanks! This is helpful because it clarifies a few areas where we disagree. 

If a bioterrorist is already capable of understanding and actually carrying out the detailed instructions in an article like this, then I'm not sure that an LLM would add that much to his capacities. 

I think future LLMs will likely still be very helpful for such people since there are more steps to being an effective bioterrorist than just understanding, eg existing reverse genetics protocols. I don't want to say much more on that point. That said, I'm personally less concerned about LLMs enhancing the capabilities of people who are already experts in some of these domains versus enhancing the ability of non-experts. 

Conversely, handing a detailed set of instructions like that to the average person poses virtually no risk, because they wouldn't have the knowledge or abilty to actually do anything with it.

I disagree. I think future LLMs will enhance the ability of average people to do something with biology. I expect LLMs will get much better at generating protocols, recommending upskilling strategies, providing lab tutorials, interpreting experimental results, etc etc. And it will do all of those things in a much more accessible manner. Also, keep in mind Fig 1 in our paper shows that there is more than one path to obtain 1918 virus. 

I also think there is an underappreciated point here about LLMs making it more likely for people to attempt bioterrorism in the first place. If a malicious actor looking to cause mass harm spends a couple of hours in conversation with an uncensored LLM, and learns that biology is a feasible path towards doing that... then I expect more people to try – even if it takes significant time and money.

There are much easier and simpler ways that are already widely discoverable: 1) Make chlorine gas by mixing bleach and ammonia (or vinegar); 2) Make sarin gas via instructions that were easily findable in this 1995 article:  

These examples indeed constitute nasty ways to cause harm to people and sound significantly easier. However, the scale of harm you can cause with infectious or otherwise exponential biology is significantly beyond that of targeted CW attacks. The potential harm is such that the statement "hardly anyone wants to carry out such attacks" doesn't seem a sufficient reason not to be concerned. 

Just saw this piece, which is strongly worded but seems defensible:  https://1a3orn.com/sub/essays-propaganda-or-science.html 

For one answer to this question, see https://www.lesswrong.com/posts/ytGsHbG7r3W3nJxPT/will-releasing-the-weights-of-large-language-models-grant?commentId=FCTuxs43vtqLMmG2n 

For lots more discussion, see the other LessWrong comments at: https://www.lesswrong.com/posts/ytGsHbG7r3W3nJxPT/will-releasing-the-weights-of-large-language-models-grant 

And also check out my rather unpopular question here: https://www.lesswrong.com/posts/dL3qxebM29WjwtSAv/would-it-make-sense-to-bring-a-civil-lawsuit-against-meta 

I am genuinely interested in gathering valid critiques on my work so that I can do better in the future.

Yes, the information is available on Google. The question is, in our eyes, more about whether a future model could successfully walk an unskilled person through the process without the person needing to understand it at all.

The paper is an attempt to walk a careful line of warning the world that the same information in more capable models could be quite dangerous, but not actually increasing the likelihood of someone using the current open source models (which it is too late to control!) for making biological weapons.

If there are specific questions you have, I'd be happy to answer.

My interpretation of the Gopal paper is that LLMs do meaningfully change the risks:

1. They'll allow you to make progress without understanding, say, the Luo paper or the technology involved.

2. They'll tell you what equipment you'd need, where to get it, how to get it, and how to operate it. Or they'll tell you how to pay someone else to do bits for you without arousing suspicion.

Perhaps model this as having access to a helpful amoral virologist?