• Identifies the three primary GPU memory consumers in large-scale model training: model parameters, model states, and model activations.
• Provides a systematic overview of memory optimization techniques, including parallelism, offloading, and activation checkpointing, tailored for limited GPU memory.
• Addresses the GPU memory wall problem, highlighting the gap between exponential parameter growth and linear memory capacity increase.
• Concludes with future research directions, advocating for continued innovation in memory-efficient training methods for large-scale language models.