From hallucinations to hardware: Lessons from a real-world computer vision project gone sideways

In the realm of computer vision, projects often take unexpected turns, and one recent endeavor exemplified this struggle. The goal was straightforward: create a model capable of analyzing photos of laptops to detect physical damage, such as cracked screens, missing keys, or broken hinges. Initially, this seemed like a perfect application for image models and large language models (LLMs), but the reality was much more complex. As the project progressed, the team encountered several challenges, including hallucinations, unreliable outputs, and even images that didn’t feature laptops at all. To address these obstacles, they adopted an innovative agentic framework in a novel manner—not for task automation but to enhance the model’s performance. The project began with a standard approach for multimodal models. The team utilized a single, extensive prompt to input images into an image-capable LLM, instructing it to identify visible damage. This straightforward prompting method worked reasonably well for clean, well-defined tasks. However, the unpredictability of real-world data quickly became apparent. Three primary issues emerged early in the process, highlighting the need for iteration. One significant discovery was the impact of image quality on the model’s performance. Users submitted a diverse range of images, from high-resolution and sharp to indistinct and blurry. This realization led the team to consult research on how image resolution affects deep learning models. They trained and tested their model using both high and low-resolution images, aiming to improve resilience against the varying quality encountered in practice. While this approach enhanced consistency, the underlying problems of hallucinations and handling irrelevant images remained. Inspired by recent experimental methods that combined image captioning with text-based LLMs, the team decided to explore this direction. Although theoretically sound, this technique introduced new complications that did not resolve their core issues. Recognizing a turning point, the team pondered whether breaking down the image interpretation task into smaller, specialized agents could yield better results. They constructed an agentic framework designed to execute more focused tasks, which ultimately led to more precise and explainable outcomes. As a result, hallucinations were significantly reduced, irrelevant images were effectively flagged, and each agent maintained a clear and manageable focus on its task. Despite these advancements, two main limitations emerged: the need to balance precision with coverage. To address these gaps, the team developed a hybrid system that combined the strengths of their agentic framework with the broad capabilities of monolithic prompting and the assurance provided by targeted fine-tuning. By the conclusion of the project, the team recognized that their initial concept of utilizing an LLM prompt for detecting physical damage in laptop images had evolved into a profound exploration of various AI techniques designed to confront real-world unpredictability. Throughout the journey, they discovered that some of the most effective tools were those not originally intended for such applications. By creatively repurposing agentic frameworks, they successfully constructed a system that not only achieved greater accuracy but also proved easier to comprehend and manage in practical scenarios. This project, spearheaded by Shruti Tiwari, an AI product manager, and Vadiraj Kulkarni, a data scientist at Dell Technologies, underscores the importance of adaptability and innovative thinking in the fast-evolving field of AI.