Nvidia CEO Jensen Huang can't stop talking about tokens. Here's what they are and how they're reshaping AI budgets.

At the recent GTC conference, Nvidia's CEO Jensen Huang emphasized a pivotal concept: AI tokens. During a dialogue with analysts, he painted a picture of future computing as akin to manufacturing equipment that generates tokens, suggesting these tokens will soon be essential entries in corporate financial plans, similar to expenses for laptops or software subscriptions. But what exactly are these tokens? They represent units of text—either whole words or segments of words—that serve as the foundation for measuring and pricing AI tasks. For instance, a short word may count as a single token, while longer entries could be divided into multiple tokens, with a general guideline suggesting one token averages about four characters. Major AI models, including OpenAI's ChatGPT and Anthropic's Claude, monitor the number of tokens used during interactions, charging businesses based on usage. As the volume of text processed increases, so does the number of tokens required for AI computations. In contrast to traditional software costs that are typically billed as fixed fees or subscriptions, AI expenses will hinge on actual usage. Huang forecasts a future where engineers might be allocated specific "token budgets" to enhance productivity. He even proposed an intriguing compensation strategy during his keynote, suggesting that Nvidia could offer engineers tokens equivalent to half their annual salary to attract top talent. Huang reiterated this notion, asserting the investment in token costs is justified, particularly for highly compensated engineers capable of generating significant productivity improvements through the use of autonomous applications, also known as agents. He argued that Nvidia's advancements in more robust and energy-efficient hardware would enable cheaper token generation over time, stating, "If I added to them $100 a day of inference cost — token cost — I'd be more than happy to do it," especially during crunch periods. This idea is gaining traction beyond Nvidia, as engineers are increasingly inquiring about computational resources during job interviews, and executives are beginning to factor these considerations into compensation packages. Huang noted that the emergence of agentic AI could lead to a substantial rise in token usage, as these systems will operate independently. He remarked, "Right now, as we speak, all of our laptops are kind of sitting idle. But in the future, the computer is going to be running 24/7 and creating tokens because your agents are off doing work."