Efficient Distributed MLLM Training with Cornstarch

Multimodal large language models (MLLMs) extend the capabilities of large language models (LLMs) by combining heterogeneous model architectures to handle diverse modalities like images and audio. However, this inherent heterogeneity in MLLM model structure and data types makes makeshift extensions to existing LLM training frameworks unsuitable for efficient MLLM training, especially in distributed training. In this paper, we present Cornstarch, an efficient distributed MLLM training framework that contemplates MLLM's unique characteristics in both model and data parallelization. Cornstarch introduces frozen-aware pipeline parallelism and workload-balanced context parallelism to improve MLLM training throughput. Our extensive evaluation shows that Cornstarch outperforms state-of-the-art solutions by $2.26\times$ on average in terms of MLLM training throughput. Cornstarch is an open-source project and available on [Github](https://github.com/cornstarch-org/Cornstarch).

Artificial intelligence models are evolving from text-only systems into "multimodal" systems that can understand images and audio and more types of data. However, because these systems mix different types of data and model structures, traditional AI training methods are highly inefficient when dividing the workload across multiple computers. To solve this, we developed Cornstarch, a software framework designed specifically to speed up the training of these multimodal systems. Cornstarch introduces two new methods that smarter distribute the computing workload and eliminate unnecessary idle time during training. Our testing shows that Cornstarch trains these advanced AI systems an average of $2.26\times$ faster than current industry-standard solutions.