Tag: 分布式计算 (2 articles)

Ulysses Sequence Parallelism addresses the challenges of training large language models with long sequences, significantly enhancing the capability to process million-token contexts.

Mixture of Experts (MoEs) are becoming a new trend in Transformers by enhancing computational efficiency and optimizing parallel processing, driving the evolution of large language models.