• METP avoids duplicated tensors and uses send/recv communication instead of collective communication, improving memory efficiency.
• With double buffering, METP achieves effective overlap between computation and communication, with a theoretical condition for full overlap.
• METP provides O(1/p^3) memory overhead without FlashAttention and saves at least 41.7% memory compared to TP when using FlashAttention.
• On eight A100 GPUs, METP increases the maximum sequence length by 2.38–2.99 times versus other parallelism methods.