Oct 27, 2023 A few things you need to know about Megatron-LM DistributedOptimizer Oct 19, 2023 Math for Megatron Mixture-of-Experts (MoE)