Multi-talker audio–visual speech recognition towards diverse scenarios

Abstract

Recently, audio–visual speech recognition (AVSR) has attracted increasing attention. However, most existing works simplify the complex challenges in real-world applications and only focus on scenarios with two speakers and perfectly aligned audio-video clips. In this work, we study the effect of speaker number and modal misalignment in the AVSR task, and propose an end-to-end AVSR framework under a more realistic condition. Specifically, we propose a speaker-number-aware mixture-of-experts (SA-MoE) mechanism to explicitly model the characteristic difference in scenarios with different speaker numbers, and a cross-modal realignment (CMR) module for robust handling of asynchronous inputs. We also use the underlying difficulty difference and introduce a new training strategy named challenge-based curriculum learning (CBCL), which forces the model to focus on difficult, challenging data instead of simple data to improve efficiency.

1. Introduction

Speech recognition in multi-talker scenarios remains one of the most challenging tasks in the speech processing community. Although modern systems have achieved human-level performance on clean speech benchmarks and demonstrated remarkable robustness against background noise (Xiong et al., 2016; Nguyen et al., 2021), their performance degrades significantly in the presence of overlapping speech. This limitation poses substantial barriers to practical applications such as meeting transcription systems, where spontaneous multi-speaker interactions are common. In response, recent research has increasingly focused on multi-talker speech separation (Hershey et al., 2016) and recognition (Gulati et al., 2020; Guo et al., 2021), aiming to extend the applicability of speech technologies to broader real-world settings.

One promising research direction leverages the visual modality through audio–visual speech recognition (AVSR) (Gao and Grauman, 2021; Cheng XZ et al., 2023; Yang XD et al., 2024). Fueled by the availability of large-scale audio–visual datasets (Nagrani et al., 2017; Afouras et al., 2018a, 2018b), AVSR extends traditional automatic speech recognition (ASR) into a multi-modal framework by leveraging visual cues from lip movements. This approach offers two key advantages: It enables artificial intelligence (AI) systems to perceive speech more similarly to humans, and it naturally resolves the permutation ambiguity inherent to multi-talker ASR (Yu et al., 2017) by visually specifying the target speaker.

Although many existing AVSR studies focus on improving performance in clean audio scenarios (Ma et al., 2021; Haliassos et al., 2023), recent works have begun addressing speech noise challenges, primarily focusing on single intervening speaker cases (Shi et al., 2022b; Ma et al., 2023). However, real-world applications present more complex challenges that current research has yet to fully address. First, the number of overlapping speakers involved in a recording is often unknown. Although some audio–visual speech separation systems train on data with varying speaker counts (Cheng HY et al., 2023), they typically treat this as simple data augmentation without explicitly studying how different speaker numbers affect recognition performance. We argue that the impact of speaker counts should be systematically considered, as optimal modality utilization varies with scenario. In simpler scenarios with no intervening speakers, audio alone may suffice, whereas visual information becomes increasingly crucial as speaker overlap grows. To address this dynamic requirement, we propose a speaker-number-aware mixture-of-experts (SA-MoE) mechanism. This architecture explicitly assigns different scenarios to different experts while using speaker counting as an auxiliary task for automatic expert assignment. By incorporating low-rank decomposition (Hu EJ et al., 2022), SA-MoE achieves this adaptive capability with minimal additional parameters.