An NYU professor who hates that students' work reads like McKinsey memos held AI oral exams to 'fight fire with fire'

In a bold move to improve educational assessments, a professor at NYU's Stern School of Business has introduced AI-driven oral exams as a countermeasure to the prevalent use of artificial intelligence in student assignments. Panos Ipeirotis, who specializes in data science, expressed his concerns in a recent blog post about the quality of student submissions that resembled polished corporate memos but lacked genuine comprehension. Ipeirotis noticed that when students were called upon to justify their written work, many faltered. "If you cannot defend your own work live, then the written artifact is not measuring what you think it is measuring," he stated. To address this issue, he decided to reintroduce oral exams, leveraging AI technology to administer them on a larger scale, effectively “fighting fire with fire.” He emphasized the need for assessments that promote understanding and real-time reasoning, saying, "Oral exams used to be standard until they could not scale. Now, AI is making them scalable again." In his blog, Ipeirotis detailed how he and a colleague created an AI examiner using ElevenLabs' conversational speech technology, allowing them to set up the oral exams in mere minutes. The structure of the oral exam included two parts: first, the AI agent posed questions regarding students’ capstone projects, exploring their decision-making processes. Following this, it challenged students with one of the cases discussed in class, pushing them to articulate their thoughts in real time. Over a span of nine days, the AI system evaluated 36 students, with each session lasting about 25 minutes and costing roughly $15 for the total assessments—significantly less than traditional human-led oral exams, which can run into the hundreds. Additionally, Ipeirotis implemented AI for grading, using three models—Claude, Gemini, and ChatGPT—to evaluate the transcripts. These models collaborated to revise their scores, with Claude serving as the synthesizer of their evaluations. Ipeirotis noted that this “council of LLMs” provided more consistent and fair grading than human assessors, yielding superior feedback that highlighted areas needing improvement in the curriculum. Reactions among students were mixed; while some appreciated the AI oral exams, many found them more stressful than conventional written tests, even as they acknowledged their effectiveness in measuring true understanding. Ipeirotis remarked that the oral exams illustrated the essence of learning: "The more you practice, the better you get." This innovation comes at a time when educational institutions are grappling with the challenges posed by AI in student assessments. A recent study published in "Assessment & Evaluation in Higher Education" labeled the issue as a "wicked problem," revealing that many educators feel overwhelmed by AI's impact on their workload and are uncertain about how to structure assessments that remain effective in this new landscape. As discussions around AI in education continue to evolve, figures like LinkedIn co-founder Reid Hoffman have suggested that traditional assessment formats may need a complete overhaul to ensure they accurately reflect student learning in an AI-enabled world.