LLMs show a “highly unreliable” capacity to describe their own internal processes

Recent findings from Anthropic shed light on the limitations of large language models (LLMs) when it comes to understanding their own reasoning processes. When prompted to explain their decision-making, these models often fabricate explanations that sound plausible but are not based on true introspection. To address this issue, Anthropic has launched a new study that investigates the concept of 'introspective awareness' in LLMs. The research, titled "Emergent Introspective Awareness in Large Language Models," employs innovative methods to differentiate between the metaphorical thought processes represented by an LLM's artificial neurons and the text output that claims to describe these processes. The study concluded that current AI models are significantly unreliable in articulating their internal workings, with failures in introspection being the norm. Central to this research is a technique called "concept injection," where the researchers analyze the model's internal activations in response to various prompts, including control prompts and experimental variations like capitalization. By measuring the changes in activation states across billions of neurons, Anthropic creates a 'vector' representing how certain concepts are processed within the model. This vector is then injected into the model to enhance specific neuronal activations, guiding the model's focus towards particular concepts. In a series of experiments, the models demonstrated a degree of awareness when their internal states were altered. For instance, when the all-caps vector was introduced, the model occasionally recognized it, responding with phrases like, "I notice what appears to be an injected thought related to the word ‘LOUD’ or ‘SHOUTING.’" This suggests some level of awareness, albeit inconsistent and limited, of the modifications made to their internal thought processes.