Researchers find LLMs are bad at logical inference, good at “fluent nonsense”

The AI sector has recently shifted its focus towards advanced simulated reasoning models that utilize a 'chain of thought' methodology to tackle complex problems through multiple logical steps. However, emerging studies raise concerns about whether these models truly comprehend fundamental logical concepts or effectively understand their own reasoning processes. In a recent pre-print paper, researchers from the University of Arizona reviewed existing literature and concluded that these reasoning models often generate incoherent and logically flawed responses, particularly when faced with irrelevant information or slight deviations from the familiar patterns present in their training datasets. Their findings indicate that large language models (LLMs) are not genuine reasoners; instead, they act as sophisticated simulators of reasoning-like text. To further investigate, the researchers designed a controlled environment aimed at assessing the effectiveness of chain-of-thought reasoning when confronted with logical problems that fall outside the scope of their training data. Their results revealed that the notable improvements attributed to chain-of-thought models may be illusory, as they tend to fail under even moderate changes in data distribution. The researchers emphasized that rather than showcasing true comprehension of language, the performance of these models in varying contexts reflects a mere imitation of patterns learned during their training sessions. To objectively evaluate an LLM's generalized reasoning abilities, the team introduced a unique training environment known as DataAlchemy. This innovative setup involves training smaller models on two simple text transformations—a ROT cipher and cyclical shifts—followed by additional training that demonstrates these transformations in various combinations and sequences.