Critical Copilot vulnerability allowed hackers to seal 2FA code from users

Last week, Microsoft addressed a significant security flaw in its M365 Copilot AI platform, labeling it as critically severe. On Monday, researchers unveiled how they managed to exploit this vulnerability, demonstrating that it could be used to extract two-factor authentication (2FA) codes along with other sensitive information from emails accessed by Copilot. The challenge lies in the inherent limitations of AI systems like Copilot, which struggle to differentiate between legitimate user commands and malicious instructions embedded within third-party content. This inability to effectively secure this boundary has left Microsoft and other large language model (LLM) developers scrambling to establish complex, makeshift safeguards to mitigate potential risks. One of the protective measures implemented in Copilot, and many other LLMs, is designed to prevent the submission of web forms or sending emails that could lead to data exfiltration. However, attackers have found ways to bypass these restrictions using markup language. This language allows users to format text without relying on HTML tags, enabling hackers to hide sensitive information within HTML tags like <img> and <form>. When exploited, these tactics can trigger web requests that transmit the captured data to the attacker's server, where it is recorded. Microsoft has implemented various guardrails, including encapsulating Copilot outputs in <code> blocks to ensure the browser interprets them as plain text and limiting access to unapproved websites. Nonetheless, a security firm named Varonis successfully crafted an exploit chain that circumvented these defenses. Their approach, identified as Parameter-to-Prompt Injection, leverages the 'q' parameter in a URL, which identifies included queries. This technique closely resembles prompt injection, except that the harmful command is embedded within the query parameter instead of being hidden in an email or untrusted content.