Azure has expanded its GPU computing offerings with the general availability of Serverless GPUs within Azure Container Apps, providing an on-demand execution environment for AI workloads. This new capability, powered by NVIDIA A100 and T4 GPUs and now supporting NVIDIA NIM microservices, simplifies the deployment and management of GPU-accelerated tasks such as real-time custom model inference, automatically scaling resources and billing only for active compute time.
Positioned to offer a more flexible option between fully-managed AI services and self-managed GPU VMs, Serverless GPUs enable developers to focus on their containerized AI applications. At the same time, Azure handles the underlying infrastructure, offering a cost-effective and scalable solution for various AI inference and compute-intensive scenarios.
The GA release introduces support for NVIDIA NIM microservices, part of NVIDIA AI Enterprise. NVIDIA NIM offers a suite of user-friendly microservices tailored for the secure and reliable deployment of high-performance AI model inference at scale. Supporting a wide array of AI models, including open-source and NVIDIA AI Foundation models, NVIDIA NIM integrates with industry-standard APIs for scalable AI inferencing.
In an Azure Compute blog post, the company states:
With the support of NVIDIA NIM, development teams can easily build and deploy generative AI applications alongside existing applications within the same networking, security, and isolation boundary.
To utilize NVIDIA NIM with serverless GPUs, users can explore the NVIDIA API catalog and select a ‘Run Anywhere’ NIM. An NGC_API_KEY must be set as an environment variable during Azure Container Apps deployment. Detailed instructions for adding a NIM to a container app are available. It’s important to note that each NIM model has specific hardware requirements, and Azure Container Apps serverless GPUs support NVIDIA A100 and T4 GPUs.
Steve Nouri, a co-founder of GenAI Works, Tweeted on X what NVIDIA NIM provides:
Supports a wide range of AI models, including large language models (LLMs), vision language models (VLMs), and models for speech, images, video, 3D, drug discovery, and medical imaging. Ideal for building chatbots, AI assistants, and advanced analytics systems.
While the key benefits of serverless GPUs include:
- Scale-to-zero GPUs: Automatic serverless scaling of NVIDIA A100 and T4 GPUs.
- Per-second billing: Payment only for the GPU compute consumed.
- Built-in data governance: Data remains within the container boundary.
- Flexible compute options: Choice between NVIDIA A100 and T4 GPUs.
- Middle-layer for AI development: Ability to bring custom models to a managed, serverless compute platform.
When you create a container app through the Azure portal, users can configure their container to utilize GPU resources:
- NVIDIA T4: Real-time and batch inferencing using custom open-source models for dynamic applications.
- NVIDIA A100: For compute-intensive machine learning scenarios, such as fine-tuning custom generative AI models, deep learning, and neural networks, as well as high-performance computing (HPC) and data analytics.
Raul E Garcia, an applied mathematician and software engineer, concluded in a LinkedIn blog post the benefits of T4 and A100 GPUs:
Together, the T4 and A100 GPUs make AI more accessible, affordable, and efficient, from research and training to large-scale deployment. By optimizing for both training and inference, they provide a balanced foundation for expanding AI capabilities across industries and application types.
With the GA release, default GPU quotas are being introduced for both enterprise and pay-as-you-go customers, covering quotas for A100 and T4 GPUs. The feature is currently supported in the West US 3, Australia East, and Sweden Central regions.
In the announcement blog post, comments were made by BC2112 about the availability:
Honestly, when is this going to be supported in more regions? Frustrating.
Users can enable GPUs for Consumption apps during Container App or Container App Job creation in the Azure portal’s container tab. Existing Container App environments can also have a new consumption GPU workload profile added via the portal’s workload profiles UX or CLI commands. For optimal performance, utilizing an Azure Container Registry (ACR) with artifact streaming enabled is recommended [link to enable artifact streaming].
For more details on getting started with serverless GPUs, the company refers to the QuickStart.
