AI inference chip startup Groq nabs $750M at $6.9B valuation – News

Groq Inc., a developer of artificial intelligence inference chips, today announced that it has raised $750 million in new funding.

Databricks Inc. backer Disruptive led the round. It was joined by Cisco Systems Inc., Samsung Electronics Co., Deutsche Telekom Capital Partners and several investment firms. Groq is now valued at $6.9 million, up from $2.8 billion last year.

Groq offers a processor called the Language Processing Unit, or LPU. The company claims that the chip can run some inference workloads with up to ten times higher energy efficiency than graphics cards. Groq says the LPU’s efficiency is the fruit of several optimizations not included in rival chips.

Coordinating the different processor components that are involved in running an AI model can consume a significant amount of computing power. According to Groq, its LPU reduces that overhead, which leaves more processing capacity for inference. The company says that its silicon can run models with 1 trillion parameters.

Groq’s LPU reduces the overhead associated with circuit coordination tasks using a custom compiler. The compiler calculates which circuit should perform what task before an inference workload launches, which removes the need to run the necessary calculations at runtime.

Groq’s compiler also optimizes AI models in other ways. Many chips use a technique called quantization to compress neural networks, which reduces their memory footprint at the expense of some output quality. Groq says that its compiler uses an improved version of the technique called RealScale. The technology only compresses the parts of a neural network that don’t experience a significant decrease in output quality when they’re quantized.

Another selling point of the LPU is that it has a so-called deterministic architecture. As a result, it’s possible to predict how much time each given computing operation will take with a granularity of individual clock cycles. According to Groq, the LPU’s predictability facilitates  performance optimizations that would otherwise be difficult to implement.

The company sells its chips as part of an appliance called the GroqRack. The system includes nine servers that each hold multiple LPUs. Grok says that the GroqRack requires less external networking hardware than competing appliances, which lowers costs, and can be installed in data centers without major facility upgrades.

The company also provides access to its chips via a cloud platform. GroqCloud, as the platform is called, hosts LPU-powered AI models that developers can integrate into their software via an application programming interface. Groq will use its new funding to expand the data center network that powers GroqCloud. 

The investment comes amid rumors that a rival provider of inference chips is seeking a new funding round of its own. Santa Clara, California-based Rivos is reportedly in talks to raise up to $500 billion at a $2 billion valuation. The company is developing a system-on-chip that combines a graphic card with central processing unit cores.

Image: Groq