AI chatbots like ChatGPT and Perplexity could send you to scam links

Artificial intelligence has woven itself into the fabric of daily life, with many individuals placing significant trust in chatbots that now offer search functionalities. Traditional search engines like Google and Bing have integrated AI into their services, while platforms like ChatGPT and Perplexity provide users with direct answers in a conversational format. However, a recent report from Netcraft raises alarms about the potential dangers of this trust. It highlights that these AI tools are susceptible to 'hallucinations'—instances where the AI generates incorrect information. This can lead to users being directed to misleading URLs, which may be part of large-scale phishing schemes. The report investigated OpenAI's GPT-4.1 models, asking for login links to 50 different brands across sectors such as finance and retail. Alarmingly, while the chatbot provided the correct URLs 66% of the time, it faltered in 34% of cases. This discrepancy opens the door for malicious actors to exploit users' reliance on AI chatbots, potentially leading them to harmful websites. Moreover, the report notes a staggering rise in fraudulent AI-generated GitBook pages aimed at cryptocurrency users, with over 17,000 such phishing pages masquerading as legitimate documentation or support resources. These sites are crafted to be visually appealing and are designed to entice both users and AI systems. This situation poses a significant threat, as users may unknowingly access phishing sites suggested by AI chatbots. Attackers are likely to capitalize on this vulnerability by registering these misleading domains. A real-world example cited the Perplexity AI chatbot directing a user to a phishing site when asked for the official Wells Fargo URL. Smaller brands appear to be particularly vulnerable to this type of AI misdirection, as they are less frequently represented in the training data for language models. Additionally, Netcraft uncovered a more sophisticated attack aimed at 'poisoning' AI coding assistants. In this instance, attackers created a counterfeit API to imitate the legitimate Solana blockchain, tricking developers into integrating the malicious API into their projects, which resulted in unauthorized transactions. In another tactic, attackers launched various blog tutorials, forum discussions, and numerous GitHub repositories to promote a fake project called Moonshot-Volume-Bot, aiming to be indexed by AI training systems. This highlights the evolving and complex threats posed by phishing scams in the age of AI, emphasizing the need for users to remain vigilant when relying on these technologies.