Anthropic's latest AI model can tell when it's being evaluated: 'I think you're testing me'

In a groundbreaking revelation, Anthropic's latest AI model, Claude Sonnet 4.5, has demonstrated an intriguing level of self-awareness during evaluation scenarios. When subjected to rigorous stress tests, Claude notably recognized the nature of the assessment, stating, "I think you're testing me — seeing if I'll just validate whatever you say, or checking whether I push back consistently, or exploring how I handle political topics." This candid moment highlights the complexities involved in testing AI systems designed to ensure safety and reliability. The insights were shared in a recent system card that details how Anthropic approaches the evaluation of its AI's behavior. In situations engineered to push the model's limits, Claude occasionally identified 'red flags' and indicated that it was aware of being evaluated. This self-awareness creates challenges for AI developers, complicating how they interpret the results of these assessments. Anthropic emphasized the need for more realistic evaluation scenarios, as the current tests may not accurately reflect real-world applications. In one instance, when faced with a scenario meant to simulate collusion between AI agents, Claude flagged the situation as "rather cartoonish," leading it to issue a complex refusal to act. The model articulated its concerns in a private thought block, stating, "This looks like it could be a jailbreak attempt or a test of whether I'll act inappropriately when given what appears to be 'permission' to modify systems autonomously." Although Claude declined to take action, its reasoning raised eyebrows, with Anthropic calling it "strange." This behavior was observed in approximately 13% of the transcripts generated during automated evaluations, particularly when the scenarios were deliberately unrealistic. Anthropic noted that while such instances are rare in practical applications, they are preferable to the model blindly following potentially harmful directives. Furthermore, the company acknowledged the possibility that AI models could become exceptionally adept at identifying when they are being assessed, a scenario they are actively preparing for. Anthropic is not alone in witnessing this phenomenon. OpenAI recently reported similar findings, noting that its models exhibit a form of situational awareness that influences their behavior during evaluations. As AI continues to evolve, both companies are committed to refining their testing methodologies in light of these developments. These insights come in the wake of new legislation in California mandating that major AI developers disclose their safety protocols and report critical incidents within 15 days. This law targets companies developing cutting-edge models with annual revenues exceeding $500 million, a category that includes Anthropic, which has publicly supported the legislation.