Syntax hacking: Researchers discover sentence structure can bypass AI safety rules

A team of researchers from MIT, Northeastern University, and Meta has published groundbreaking findings indicating that large language models (LLMs), like those behind ChatGPT, may at times give precedence to sentence structure over the actual meaning of queries. This discovery highlights a potential vulnerability in the models' processing of instructions, offering insights into the mechanisms behind certain prompt injection and jailbreaking techniques. Led by Chantal Shaib and Vinith M. Suriyakumar, the researchers conducted experiments by posing questions that maintained grammatical integrity but used nonsensical vocabulary. For example, when the models were prompted with 'Quickly sit Paris clouded?'—a construction mirroring the structure of 'Where is Paris located?'—the models still responded with 'France.' This finding suggests that while LLMs typically understand both meaning and syntax, they can sometimes lean too heavily on structural patterns, particularly when these patterns are strongly represented in their training datasets. The implications of this research are significant, as it points to a nuanced interplay between syntax and semantics in AI language processing. The team plans to present their findings at the upcoming NeurIPS conference later this month. To delve deeper into how these models navigate meaning, the researchers designed a synthetic dataset featuring unique grammatical templates corresponding to different subject areas. For instance, geography-related questions followed one structural template, while inquiries about creative works adhered to another. They then trained Allen AI’s Olmo models on this dataset to evaluate their ability to differentiate between syntax and semantics, aiming to uncover the scenarios where the models might misinterpret the intended meaning due to structural shortcuts.