Multi-talker audio–visual speech recognition towards diverse scenarios

• A novel end-to-end AVSR framework is proposed for realistic multi-talker scenarios, addressing both unknown speaker counts and modality misalignment. • The speaker-number-aware mixture-of-experts (SA-MoE) mechanism adaptively fuses audio and visual information based on the number of overlapping speakers, using speaker counting as an auxiliary task. • A cross-modal realignment (CMR) module robustly handles asynchronous audio-video inputs, overcoming temporal misalignment in real-world recordings. • The challenge-based curriculum learning (CBCL) strategy prioritizes difficult samples, improving training efficiency and overall performance on complex multi-talker speech.