Nvidia: Let’s get physical with AI – News

Nvidia Corp. recently held an industry analyst briefing on the topic of physical artificial intelligence, and Chief Executive Jensen Huang has been consistent in his talk track in every keynote he has done this year that physical AI is the next wave of AI. In fact, he has often stated that eventually anything that moves – from lawnmowers to forklifts to cars — will be autonomous giving rise to the physical AI era.

Though most people think of physical AI or the world of robots as the stuff of science fiction and a niche technology, the benefits will be widespread. I recently talked with a chief information officer from a healthcare organization in the Mid-Atlantic and he explained that autonomous wheelchairs would enable patients to be taken curbside without having to use a person enabling that clinician to spend more at a patient’s bedside. Retailers can use robots to scan shelves for better inventory control, and anyone who flies United Airlines has likely seen the robot that moves around the lounge for people to place used dishes in.

Presenter Rev Lebaredian, Nvidia’s vice president of Omniverse and simulation technology, dove deep into this fascinating — and fast-growing — segment of the AI boom.

The physical AI era has arrived

Making AI useful and productive for the real world is the realm of physical AI. But what defines the boundaries between the different types of AI?

“Generative AI involves models we’ve all been using, such as large language models and maybe some image models,” said Lebaredian. “Essentially, you give it some input, and output comes up. With LLMs, the input tokens are text, and the outputs are also text.” But with physical AI, the model is different. “We bring in input that would be the equivalent of what the sensors on a robot would experience,” he explained.

When it comes to AI, the terms “robot” and “physical AI” are general terms for a broader category. This includes humanoid robots, manipulator arms, self-driving cars or anything else that moves. However, physical AI also extends to things such as radio towers — anything that could sense the physical world and then go operate in it.

These robotic devices input sensor data, including the equivalent of what language models also input, such as text and other modes of input. “We can combine our understanding of abstract knowledge in the LLMs along with an understanding of the physics of the world to then output action tokens,” he said. “These are actions that end up controlling an embodiment of the robot. On a manipulator arm, that would be the torques and forces that are created by the motors to change the angles of the points on the robot arm. It could also be the steering, braking, and acceleration of a self-driving car. It could be anything that’s a control signal for the actual body. The application of this is endless.”

Real-world applications

Lebaredian said while there have been breakthroughs and steady progress in physical AI development and refinement, there’s plenty of work to be done in this segment of the industry. “The things we need to do in the real world are obviously extremely valuable. Once we crack this problem of physical AI, we can enhance everything from factories and warehouses to all of transportation, and humanoid and other robots can do the equivalent of human physical labor.”

Why is this important? Labor shortages present a real challenge to industries and companies that are seeking to grow. In many industries labor shortages are massive. Businesses are having a hard time hiring enough skills factory and warehouse workers and people to stock retail shelves. This article on meteorspace.com states that the US warehousing industry is facing a shortage of more than 35,000 workers, with companies such as Amazon reporting turnover rates of over 150%, so while hiring people is difficult, keeping them is even harder.

Looking ahead, many countries are facing an aging and declining population creating a situation where producing the same volume of goods year over year becomes increasingly difficult. There are also global supply chain issues due to geopolitics that are resulting in a great deal of manufacturing that was once farmed out to manufacturing hubs in Asia and elsewhere being reshored, especially in the U.S.

But is physical AI technology ready to take advantage of this opportunity?

The era of general-purpose robots

Are the mechanical and, more importantly, the software and AI technologies needed to build and operate sophisticated general-purpose robots ready for the job? Lebaredian believes so.

“For the first time, we have a line of sight to building the algorithms, to building the brains of a general-purpose, robust robot. The industry had the capability to build physical robots for quite a while. We’ve been introducing mechatronics and robotics into the industrial space for decades now, but we didn’t have the capability of making them intelligent enough so that those robots can see and act autonomously in a general way. We had to program them specifically to do one task repeatedly,” he said.

The invention and evolution of AI has accelerated robot and physical AI development. The massive amount of innovation from Nvidia and the rest of the AI industry has created the ability to build a “brain” Lebaredian referred to and to democratize it across all the domains and physical spaces. That wasn’t possible five years ago, but today it is.

Nvidia’s role in AI-driven robotics

Lebaredian made it clear that Nvidia isn’t in the business of building robots or other AI devices, such as autonomous vehicles. But the company plays a critical role in making those devices possible and capable of accomplishing vital activities.

The company has had tremendous success building reference architectures for systems across almost every industry it serves, and physical AI is no different. Nvidia comes up with the blueprints for physical AI and then enables others to leverage it. Nvidia has three computer platforms for physical AI: Omniverse & Cosmos on RTX PRO for simulation, the DGX and HGX for training, and the Jetson Thor AGX for deployment and operation. The three computers run on Nvidia’s popular Blackwell GPUs.

The Jetson Thor AGX provides the brains for the automated vehicles that Nvidia is helping to develop. “That’s a very important computer,” said Lebaredian. “It needs to be power-efficient while being extremely powerful. It’s a specialized kind of computer. It has to be able to deal with lots of sensor data and execute advanced AI models with a lot of compute but doing that efficiently in a specific power envelope requires lots of specific software to run it.

“All three computers, because they’re built on the same architecture and run the same algorithms, and all of your software is portable between all three, they’re all backwards and future compatible as well, and future proof as well, with all the NVIDIA architectures,” Lebaredian said.

The rise of AI factories

Lebaredian says AI factories powered by Nvidia’s DGX and HGX unified AI development platforms play a crucial role in the development of AI-driven robotics. The AI factories take the raw data from the physical world and the tasks that required to execute in the physical world, and they output models—effectively the brain of the robot—and then uploaded that information to Jetson.

But even with such cutting-edge development tools and platforms, building physical AI systems is a challenging task. “To create a brain, to create any AI, you need massive amounts of data, and you need massive amounts of the right data,” Lebaredian explained. “You need accurate data, and you need well-labeled data for the knowledge space, and that’s hard. It’s already hard getting all the data we scrape the internet; we find all this information that’s readily available, and it’s still not enough.”

He says the data just doesn’t exist anymore and collecting it by capturing it through sensors in the physical world “is just too expensive, too time-consuming, and in many cases, too dangerous or even impossible to get with the accuracy we need. You just cannot have enough of the right sensors in the physical world with the accuracy that we need.

“The only way to really generate all of the data we need to collect it is by first taking the rules of the physical world — physics — and replicating it inside a computing system, building a simulator of that physical world, and that simulator becomes a generator for the kind of data that we need to feed into the AI factory, which could then produce the AI algorithms we then deploy,” Lebaredian said.

But the work doesn’t end once the simulators are created. “We also need these simulations, not only as a data generator, but to test them before we deploy those AI brains that we train onto the real robot,” he said. “We need to test them for millions and millions of hours, drive our AI-driven AV vehicles for millions and millions of miles before unleashing them onto the world. And the best place to do that, and the fastest place, the least expensive place to do it, and the least dangerous place, is in simulation.”

The simulation computer fulfills two functions. It’s the data generator that feeds the AI factory, and it’s also where the testing and validation is done before deploying these physical AI systems in the real world. This is a critical step as without it, companies would have to build physical environment to tests robots. They can get damaged when they fall, tests can be incomplete, and it takes a long time to create new scenarios.

Success requires cooperation

Though Nvidia is rightfully proud of all the company has contributed to the growth of AI, it will require continued collaboration from all technology sectors to make the full potential of physical AI a reality.

“No one company can solve all of these problems,” Lebaredian said. “It’s just way too large and way too big. We’re building the core computational part of this. We’re building the three computers with all the operating systems for that, but we need many in the ecosystem to build the layers of software on top, to build the physical hardware, and every combination in between. The only way we are going to build physical AI that’s robust, that really addresses all of the needs of the industries I mentioned in that $100 trillion market, is by doing it together.”

Most of the media focus and hype around AI is focused on generative AI, but physical AI is right around the corner and will change all our lives. Very soon, vacuums, lawn mowers, golf carts and other consumer devices will be autonomous as well as industrial equipment.

Zeus Kerravala is a principal analyst at ZK Research, a division of Kerravala Consulting. He wrote this article for News.

Image: Nvidia