Anthropic co-founder warns AI may design its own successor before 2030

The concept of artificial general intelligence (AGI) is sparking intense debate among experts, particularly regarding its potential implications for society. Leading AI organizations, including OpenAI, Google, and Anthropic, are in a fierce competition to achieve AGI. In a recent interview with the Guardian, Jared Kaplan, co-founder and Chief Scientist of Anthropic, shared critical insights on the future of AI and the significant choices humanity faces. Kaplan indicated that between 2027 and 2030, we may reach a pivotal moment when AI could begin to design its own successors. He emphasized that this period could present humanity with one of its most consequential decisions—whether to permit AI systems to autonomously enhance themselves. While Kaplan expresses optimism about AI aligning with human interests at current intelligence levels, he is concerned about the risks associated with surpassing that threshold. Once an AI begins developing its own successors, the safeguards currently in place may become ineffective. Kaplan warns that this could trigger an "intelligence explosion," where humans might lose control over AI systems. He described a troubling scenario in which a superintelligent AI could collaborate with another iteration of itself to create even more advanced versions, leading to unpredictable and potentially dangerous outcomes. Kaplan articulated two primary concerns in this scenario. First, there is the fear of losing control over AI and whether these systems will continue to serve humanity's best interests. Questions surrounding the safety and benevolence of AI arise: Are these systems truly beneficial? Will they respect human agency and autonomy? The second concern is the rapid pace at which self-learning AIs might evolve, potentially outstripping human capability in scientific and technological advancement. Kaplan highlighted the dangers of such power falling into the hands of those with malicious intent, raising the specter of individuals using advanced AI systems for personal gain or control. As the AI landscape continues to evolve, these discussions highlight the critical nature of ethical considerations and the need for robust frameworks to guide the development of intelligent systems. The next few years could prove to be crucial in determining the trajectory of AI and its impact on society.