Startup Gimlet Labs is solving the AI inference bottleneck in a surprisingly elegant way

Zain Asgar, an adjunct professor at Stanford and a seasoned entrepreneur, has successfully raised $80 million in a Series A funding round for his startup, Gimlet Labs. This innovative company aims to tackle the persistent AI inference bottleneck with a groundbreaking approach. Menlo Ventures led the funding round. Gimlet Labs boasts the creation of what it describes as the first-ever 'multi-silicon inference cloud.' This unique software solution enables AI workloads to be executed simultaneously across a variety of hardware types, including traditional CPUs, AI-optimized GPUs, and high-memory systems. 'We essentially leverage all available hardware,' Asgar explained in a conversation with TechCrunch. The technology allows a single agent to chain multiple steps, each demanding different hardware capabilities. In a blog post discussing the funding, Tim Tully of Menlo Ventures emphasized that while there is no single chip that can handle all tasks efficiently, new hardware is continuously being developed. He noted, 'The multi-silicon fleet is ready — it just needs the right software layer to unlock its potential.' According to McKinsey, if the trend of increasing computational deployment persists, data center investments could reach nearly $7 trillion by 2030. Asgar pointed out that current applications are only utilizing existing hardware resources between 15 to 30 percent of the time. 'This means hundreds of billions of dollars are wasted due to idle resources,' he remarked, emphasizing their goal to enhance AI workload efficiency by up to tenfold. To achieve this, Asgar and his co-founders, Michelle Nguyen, Omid Azizi, and Natalie Serrino, developed orchestration software that optimally distributes workloads across diverse hardware platforms. Gimlet Labs claims it can accelerate AI inference by three to ten times while maintaining cost and power efficiency. The company even has the capability to partition underlying models to run on various architectures, utilizing the most suitable chip for each segment. Gimlet Labs has established partnerships with major chip manufacturers, including NVIDIA, AMD, Intel, ARM, Cerebras, and d-Matrix. Their product, available as software or through an API via Gimlet Cloud, is designed for significant AI model labs and large data centers rather than average AI application developers. Since its public launch in October, Gimlet Labs reported achieving eight-figure revenues from the start, amounting to at least $10 million. Asgar revealed that the company’s client base has more than doubled in just four months, now including a prominent model developer and a major cloud computing enterprise, although he kept their identities confidential. The founding team previously collaborated at Pixie, a startup that developed an open-source observability tool for Kubernetes, which was acquired by New Relic shortly after its launch in 2020. After a chance meeting with Tully last year and receiving angel investments from Stanford faculty, interest from venture capitalists surged, leading to an oversubscribed funding round. With the latest round, Gimlet Labs has amassed a total of $92 million in funding, including contributions from influential angels such as Bill Coughran of Sequoia, Stanford Professor Nick McKeown, former VMware CEO Raghu Raghuram, and Intel CEO Lip-Bu Tan. The startup currently employs a team of 30 professionals.