The real danger of AI isn't that it's wrong — it's that it could make us stop thinking for ourselves, a professor says

Artificial intelligence is increasingly being seen as a powerful tool, capable of delivering impressive answers to complex questions. However, Lucy Gill-Simmen, associate dean for Education and Student Experience at Royal Holloway, University of London, warns that the most significant danger lies not in AI's inaccuracies but in its potential to diminish our ability to think critically. Gill-Simmen emphasizes that while AI can convincingly produce information, the real concern is that it may discourage individuals from questioning, verifying, and investigating the information they receive. "Hallucinations in AI—instances where the technology generates incorrect or false data—are noticeable problems. But the deeper issue is that people may stop asking how they know if an answer is accurate," she explained in a recent discussion with Business Insider. Traditionally, acquiring knowledge has demanded effort, such as comparing different sources, testing assumptions, and navigating uncertainty. AI, on the other hand, can deliver answers effortlessly, which could lead to a dangerous complacency. "My worry extends beyond mere misinformation; it is about a growing disinclination to verify, question, and explore information independently. This struggle often fosters real learning," she noted. Research from Wharton has introduced the concept of "cognitive surrender," which refers to the tendency of individuals to accept AI-generated responses without critical evaluation. Steven Shaw, a postdoctoral researcher at Wharton, highlighted this phenomenon in a report earlier this year, indicating that people might become "passive followers of unexamined thoughts" if they adopt AI-generated ideas uncritically. In a series of experiments involving over 1,300 participants, Shaw found that when individuals consulted an AI assistant, they accepted its recommendations approximately 92.7% of the time when it was correct, and 79.8% even when the AI was intentionally incorrect. This trend raises significant concerns, especially in educational contexts. Kimberley Hardcastle, a business and marketing professor at Northumbria University, echoed these worries, suggesting that reliance on AI could weaken people’s abilities to independently verify, challenge, and construct knowledge without algorithmic assistance. Gill-Simmen described this as "epistemic atrophy," where the skills necessary for knowledge acquisition progressively diminish. She further explained that when individuals solve problems independently, they develop mental frameworks, enhance their reasoning abilities, and build confidence in their own thought processes. "The problem isn't that AI could replace memory; it's that it risks becoming a substitute for genuine thinking," she cautioned. Gill-Simmen has observed this trend among her students, who often display an "illusion of understanding." While AI-generated explanations may seem credible, students frequently struggle to elaborate on the reasoning behind them or to apply concepts in new contexts. Ethan Mollick, a Wharton professor, advocates for a balanced approach where AI functions as a tool for collaborative intelligence. He stresses the necessity for students to critically evaluate AI outputs rather than accepting them passively, allowing AI to enhance their work rather than replace it. Gill-Simmen concludes by highlighting an ironic truth: as AI continues to advance, the importance of human critical thinking becomes even more pronounced. In a landscape dominated by fluent AI-generated information, success will hinge on our ability to wisely evaluate that information.