Dynamic prompting class distribution optimization for semi-supervised sound event detection

Abstract

Semi-supervised sound event detection (SSED) tasks typically leverage a large amount of unlabeled and synthetic data to facilitate model generalization during training, reducing overfitting on a limited set of labeled data. However, the generalization training process often encounters challenges from noisy interference introduced by pseudo-labels or domain knowledge gaps. To alleviate noisy interference in class distribution learning, we propose an efficient semi-supervised class distribution learning method through dynamic prompt tuning, named prompting class distribution optimization (PADO). Specifically, when modeling real labeled data, PADO dynamically incorporates independent learnable prompt tokens to explore prior knowledge about the true distribution. Then, the prior knowledge serves as prompt information, dynamically interacting with the posterior noisy-class distribution information. In this case, PADO achieves class distribution optimization while maintaining model generalization, leading to a significant improvement in the efficiency of class distribution learning. Compared with state-of-the-art methods on the SSED datasets from DCASE 2019, 2020, and 2021 challenges, PADO achieves significant performance improvements. Furthermore, it is readily extendable to other benchmark models.

1. Introduction

Sound event detection (SED) has gained significant attention due to its practical relevance in various real-world applications such as audio surveillance (Crocco et al., 2016; Park and Kim, 2020), acoustic scene understanding (Imoto et al., 2020), and human–machine interaction (Fu et al., 2019). SED tasks require recognizing the categories of events and marking the onset and offset time for each event in a mixed audio recording. This generally involves two separate sub-tasks: audio tagging and audio localization (Mesaros et al., 2021). Specifically, when given an input spectrogram, the SED model will output two-level predictions: clip-level probabilities for audio tagging and frame-level probabilities for event localization.

Traditionally, training a well-performing SED model requires an ample amount of manually labeled training data (Gao LJ et al., 2019, 2022, 2024; Li et al., 2020; Serizel et al., 2020). However, the fine-grained manual annotation at the frame level is extremely time-consuming and results in a severe shortage of annotated samples, presenting a major hurdle for SED research in the big data era. To address this issue, researchers have shifted their focus towards semi-supervised sound event detection (SSED) tasks, leveraging large-scale unlabeled and synthetic data for generalization learning to effectively mitigate overfitting on the limited labeled data. Recently, a teacher–student framework (Yan et al., 2020; Koh et al., 2021; Zheng et al., 2021b; Gao LJ et al., 2023) was established by the most popular methods in SSED commonly based on the consistency regularization assumption (Tarvainen and Valpola, 2017). In this framework, the teacher model provides pseudo-labels to guide the student model for generalization training on the unlabeled and synthetic data, while the student model exploits real labeled data concurrently for supervised training.

Despite the success of the teacher–student framework in generalization learning, noisy interference introduced by pseudo-labels or domain knowledge bias of synthetic data in class distribution learning is often under-considered, resulting in suboptimal performance in SED tasks. Therefore, one of the most challenging tasks in SSED is alleviating the noisy interference in class distribution learning. To this end, recent works have achieved performance gains through effective pseudo-labeling (PL) strategies (Chan and Chin, 2021; Koh et al., 2021), or try to transfer the domain knowledge from synthetic data domain to real domain (Zheng et al., 2021a). However, there is a trade-off between the quality and quantity of pseudo-labels, leading to a reduction in the utilization of unlabeled data. Additionally, effective...