Report: Nvidia is working on a top secret AI inference chip that could debut next month – News

Nvidia Corp. is reportedly working on a dedicated inference processor that will be used by OpenAI Group PBC and other artificial intelligence companies to develop faster and more efficient models, according to a report late Friday in the Wall Street Journal.

The new inference platform is expected to be launched at Nvidia’s annual GTC developer conference in San Jose later this month, and will integrate technology the company acquired from the chip startup Groq Inc. in December.

Inference, which refers to the process of running trained AI models in production, has emerged as a key area of focus in the AI industry. Nvidia rivals such as Google LLC and Amazon Web Services Inc. have both developed specialized inference chips that compete with its graphics processing units, and it also faces competition from dedicated inference chip startups such as Cerebras Systems Inc. and SambaNova Systems Inc.

The Journal said OpenAI has had early access to Nvidia’s new inference chip and will become one of its earliest adopters, in what amounts to a significant win for the chipmaker. Though OpenAI has been shopping for more efficient alternatives to Nvidia’s GPUs in order to diversify its computing stack, it received $30 billion in funding from the world’s top chipmaker last week in a deal that reaffirms its commitment to the company.

Nvidia is the world’s most dominant maker of GPUs, which are specialized processors that can perform billions of tasks simultaneously. But although the company continues to insist that they’re useful for both training and inference, its GPUs are no longer considered the most efficient option for powering AI applications. Many companies have found that Nvidia’s chips consume too much energy, making them extremely costly for applications such as AI agents, which carry out tasks autonomously on behalf of human users and require immense computing power.

That’s why OpenAI signed a multibillion-dollar contract with Cerebras last month to access its dinner plate-sized inference-focused chips. Cerebras claims that its silicon is much faster than Nvidia’s GPUs when it comes to inference tasks.

Nvidia’s inference chip is reportedly going to integrate technology developed by Groq. Nvidia paid $20 billion to license Groq’s technology on a nonexclusive basis in December, and as part of that deal it also hired its founding Chief Executive Officer Jonathan Ross and its President Sunny Madra. It was billed at the time as one of the largest-ever “acquihires” in Silicon Valley’s history.

Groq’s inference chips are known as “language processing units,” and they’re based on an entirely novel architecture that enables them to perform inference with much lower energy usage. However, Nvidia hasn’t said how it plans to use the startup’s technology.

OpenAI reportedly wants to use Nvidia’s new inference chip to power its Codex programming tool, which is a rival to Anthropic PBC’s Claude Code. Coding applications have emerged as one of the most powerful and profitable use cases for generative AI, and it’s an area where OpenAI is only second-best, for Claude Code is widely considered to be the market leader.

Nvidia is also pushing its central processing units as another alternative for running inference workloads. Traditionally, most companies pair its GPUs with CPUs, using the two chips in tandem to compensate for the inefficiencies of the other.

But Nvidia says some agentic AI workloads can actually run more efficiently on its most advanced Grace CPUs alone. Last month, Meta Platforms Inc. became the first company to commit to making the first sizable CPU-only deployment to support its ad-targeting agents in production.

Image: News/Microsoft Designer