Training Strategy For LLM Video Generation | HackerNoon

4. LLM Pretraining for Generation

5. Experiments

For multi-task training, we use the Alternating Gradient Descent (AGD) method (Akbari et al., 2023) to train videos of varying lengths. We design the tasks in the AGD format resulting in a near 0% padding ratio, lower than that of the packing approach (Raffel et al., 2020). This is accomplished by grouping tasks by sequence length and alternately sampling one group at each iteration. Since sequence lengths are fixed and vary significantly across tasks, e.g., first frame and long video generation, we achieve efficient training with minimal padding.

We find that sampling from image and video datasets uniformly across time can lead to suboptimal results, as training on images can enhance the model’s understanding of objects but does not capture any motions that are represented in video data. Thus, we devise a two-stage pretraining strategy, where we augment our sampling weights to sample image data 90% of the time and video data 10% of the time for the first 25% iterations of training. We then switch to training on video 90% and image 10% for the remaining iterations.

We fine-tune our pretrained model for enhanced performance on specific tasks or for new task adaptation, such as text-to-video and image-to-video tasks, using a high-quality data subset. This results in improved generation quality, consistent with Zhou et al. (2023), and addresses decoding collapse issues, characterized by repetitive token predictions. Such fine-tuning not only diversifies outputs but also allows for a higher classifier-free guidance scale (Ho & Salimans, 2022), boosting overall quality.