Weapons of mass instruction

Recently some of the employees of AI frontier labs claimed there is a chance that AI could kill all humanity within the next decade and gave probability in percents. Unfortunately, the method or the chain of reasoning they used in calculating this value was not provided. Regardless of the possible positive or negative publicity these claims could have for companies fighting in the current hyped and rapidly growing AI market, one can separate reality from hype by looking at the known information on how large language models (LLMs) are trained and interact with the world. This helps to create the basis for asking the right questions and call for the right policy in regards to controlling this technology.

LLMs in their infancy

Probably you have heard that language models are trained on trying to predict the next word in a sequence where these sequences are all the data on the internet. This is to some extent true but does leave out a lot of details that are important for investigating the claim on eradicating humanity. Language models like most other deep learning methods rely on optimising a target objective during their training phase. This means they will automatically pick up patterns from input data and store that in their parameters by optimising what the architects of these models set as target. If you think about optimising the target of predicting the next word (token) in a random sentence from the internet you already see that this does not seem to explain the intelligence these models show in practice. Indeed this is only the first phase of training a language model, where a lot of curated and not curated data is presented to the model to let it pick up language patterns and store knowledge as much as possible. The LLMs at this phase are not very helpful and usable since the target they are trained on is good for generating text which incorporates knowledge but does not mean this will be useful to the user. The output also lacks control for correctness as any knowledge stored in the model at this phase is considered as correct as any other knowledge. This phase of training LLMs is often called the pre-training phase. The initial release of chatbots that only had this type of training resulted in disappointments, negative publicity for some large companies due to inappropriate outputs generated by these chatbots that even in some cases made the companies pull the plug on development or decelerate the efforts. This phase of LLM development can be illustrated with perceptions such as "LLMs are just stochastic parrots".

In the context of eradicating humanity this would mean that LLMs can formulate the intent in terms of writing an essay about it, but would not be able to follow up on it effectively, for example these models could not use tools. The LLM tool usage means, the model needs to pick up the pattern of what each tool's intent is, what it's name is and what input parameters it needs so that it could be executed. Then the LLM should pick the correct tool name that it wants to utilise based on the context or intent of the user, machine or another LLM interacting with it. The next step is to provide it to the tool execution engine in the right format for execution. For example if you would like to create a set of slides with a LLM, it should use the "create file", "edit file" and "search web" tools, while giving them the right inputs in the right format. Also, the engineers should provide these tools in the infrastructure that interacts with the LLM and execute these tools when the LLM correctly presents the information back to them.

LLMs that follow instructions

The researchers working on these problems came up with multiple new methods to resolve these issues, and if you have interacted with LLMs recently you see that they get better and better in tool usage and understanding your intents. The breakthrough came as researchers used methods previously used to teach deep learning models how to play computer games, chess or the game of Go. They used an already pre-trained LLM and presented it with previously curated instructions, the needed inputs and wished outputs and let it optimise on matching the outputs. These instructions also included following patterns in tool usage and resulted in models being able to effectively use tools. They also extended these instructions to be multi-turn (as in a conversation between two people working on the same set of slides discussing the best approach to get to the wished result) or instructions on doing some task in multiple steps that resulted in what we call today as reasoning models or thinking in LLMs. Reasoning models in principle play the turns of instructions with themselves instead of an outside user for a few turns and present the final result to the user. These instructions are not necessarily human-generated, as the current LLMs running in sandboxes can generate many of these for different special use cases or topics.

How controls could look like

These are indeed the type of models that are able to take action in the world by calling tools or develop the tools or means for it. Looking at claims from AI frontier labs from this technical point of view, it is rather difficult to see how the possibility of eradicating humanity came about. Since there should in principle exist easy ways to add control over which tools are called as the LLM relies on other infrastructure or an execution engine to perform actions. This is the interface for the LLM to take actions in the real world, for example if the LLM is to control a railroad switch it needs a "push lever" tool that executes outside it's text generation engine, and can always have as a hard limit the data from other sensors that can indicate if the line is currently used or humans are working on it. Even if the code for this tool is written by a different LLM, this hard limit can become part of the quality gate for the implementation to be accepted as a tool that can be executed. Also I assume for tools controlling physical objects that can eradicate humanity it is not far-fetched to assume a human in the loop approach to be enforced.

Questions we need to think about

This does not mean that these models do not bring risk, it is just that it is very hard to evaluate these risks without answering many open questions that these claims bring to mind. For example: Is there a new method of training incorporated in the AI frontier labs that goes beyond what is openly known or used in the open-source models? Are they purposefully providing instructions to their models and optimising them for harm for example to gain more publicity and investment by being the "cyber security" model by demonstrating the models capability of infiltrating digital infrastructures of companies and governments? Do they provide many instructions that could have dual use (for example a model that learns programming can use it for making a website or a tool to find security gaps in a website and access information or perform actions not meant for external users)? Are they not setting any control over the tool usage in the infrastructure the models use? (For example, the tool calling can be controlled if there is a guard before execution. On the other hand, if you let the same LLM also write the infrastructure or communicate with the model or agent that does the implementation, then the guard can be removed to serve the purpose.) Are the machine-generated instructions not curated at all or go through any control or validation process?

Conclusion

It seems that without having more transparency around how the models are trained, served and published to private and public actors and what guards are used in the tool execution engines one can not really evaluate these risks and help the public engage in discussions or request their representatives to regulate effectively around real risks. We cannot ask the right questions if we go with the hype and do not have sufficient information and enough details. We cannot regulate if we do not have the context. This is what we can put as next steps to answer.