Minimizing transformer inference overhead using controlling element on Shenwei AI accelerator

• Comprehensive analysis decomposes transformer inference overhead to identify primary bottlenecks across operator, runtime, and control layers. • Three-tier scheduling framework on Shenwei AI accelerator MPE cuts host-device launches to approximately 1/10,000 of the original PyTorch-GPU setup. • Zero-copy memory management with segment-page fusion substantially reduces memory access latency and improves overall inference efficiency. • Fast model loading eliminates redundant verification and initialization computations, slashing large-model loading time from 22,128.31 ms to 1,041.72 ms.