Meta released details about its Generative Ads Model (GEM), a foundation model designed to improve ads recommendation across its platforms. The model addresses core challenges in recommendation systems (RecSys) by processing billions of daily user-ad interactions where meaningful signals such as clicks and conversions are very sparse. GEM tackles the complexity of learning from diverse ads data including advertiser goals, creative formats, measurement signals, and user behaviors across multiple delivery channels.
The company built the system using three approaches: model scaling with advanced architecture, post-training techniques for knowledge transfer, and enhanced training infrastructure that uses thousands of GPUs with advanced parallelism to support the computational demands of large-scale foundation model training.
Source: GEM Architecture
Meta re-engineered its training stack to support GEM at a scale comparable to modern large language models. The company employs multi-dimensional parallelism strategies tailored to different model components. Dense model parts use Hybrid Sharded Distributed Parallel (HSDP) to optimize memory usage and reduce communication costs across thousands of GPUs. Sparse components, primarily large embedding tables for user and item features, use a two-dimensional approach combining data parallelism and model parallelism.
Meta implemented several GPU-level optimizations to reduce training bottlenecks. These include a custom in-house GPU kernel designed for variable-length user sequences, graph-level compilation in PyTorch 2.0 that automates activation checkpointing and operator fusion, and memory compression techniques such as FP8 quantization for activations.
The team developed GPU communication collectives through NCCLX, Meta’s fork of NVIDIA’s NCCL, that operate without utilizing Streaming Multiprocessor resources. This eliminates contention between communication and compute workloads. Meta reduced job startup time by 5x through optimizations to trainer initialization, data reader setup, and checkpointing. PyTorch 2.0 compilation time decreased by 7x via caching strategies, improving the proportion of training time spent processing new data.
The system optimizes GPU efficiency across the model lifecycle. During exploration, lightweight model variants support over half of all experiments at lower cost compared to full-sized models. Meta performs continuous online training to refresh the foundation models and shares traffic between training and post-training knowledge generation to reduce computational demand.
Meta designed GEM to transfer knowledge to hundreds of user-facing vertical models that serve ads across its platforms. The company employs two transfer strategies to translate the foundation model’s capabilities into measurable gains.
Direct transfer enables GEM to pass knowledge to major vertical models within the same data spaces where GEM was trained. Hierarchical transfer distills knowledge from GEM into domain-specific foundation models, which then teach vertical models.
The approaches use knowledge distillation, representation learning, and parameter sharing to maximize transfer efficiency across Meta’s ad model ecosystem.
Swapnil Amin, former director at Tesla, commented that GEM
feels like the shift we all knew was coming — a model that actually learns creative, context, and user intent together instead of stitching pieces after the fact.
He highlighted the 23x effective FLOPs jump as
the part that changes the economics.
Sri.P, a senior product manager at Microsoft, sees potential applications for advertisers and stated.
This is a game changer for marketers/advertisers! I can see it potentially saving small businesses a lot of money since they won’t have to experiment with marketing strategies and can instead rely on intelligent models to make the most of their ad spend
Meta envisions that foundation models for ads recommendation systems will develop a deeper understanding of user preferences and intent, designed to make interactions feel more personal. For advertisers, the company positions this as an approach to enable one-to-one connections at scale.
