Temporal fidelity enhancement for video action recognition

Abstract

Temporal attention mechanisms are essential for video action recognition, enabling models to focus on semantically informative moments. However, these models frequently exhibit temporal infidelity—misaligned attention weights caused by limited training diversity and the absence of fine-grained temporal supervision. While video-level labels provide coarse-grained action guidance, the lack of detailed constraints allows attention noise to persist, especially in complex scenarios with distracting spatial elements. To address this issue, we propose temporal fidelity enhancement (TFE), a competitive learning paradigm based on the disentangled information bottleneck (DisenIB) theory. TFE mitigates temporal infidelity by decoupling action-relevant semantics from spurious correlations through adversarial feature disentanglement. Using pre-trained representations for initialization, TFE establishes an adversarial process in which segments with elevated temporal attention compete against contexts with diminished action relevance. This mechanism ensures temporal consistency and enhances the fidelity of attention patterns without requiring explicit fine-grained supervision. Extensive studies on UCF101, HMDB-51, and Charades benchmarks validate the effectiveness of our method, with significant improvements in action recognition accuracy.

1. Introduction

Learning paradigms based on physical systems and neural mechanisms have advanced video-based action recognition by enhancing temporal dependency modeling and discriminative motion pattern extraction (Jiao LC et al., 2024, 2025). While traditional three-dimensional (3D) convolutional networks and two-stream architectures (Liu ZY et al., 2021; Liu Y et al., 2024) have established the foundation for temporal modeling, their reliance on predefined hierarchies limits the adaptability to long-term dynamics. Multi-scale temporal dependency methods (Yu et al., 2020; Zhou JM et al., 2021; Zhang et al., 2025) partially alleviate this, but remain constrained by fixed parameterization. This has driven the use of temporal attention mechanisms, which dynamically prioritize critical temporal segments and improve motion feature extraction (Wu CY et al., 2022; Jiao JY et al., 2023; Yamazaki et al., 2023; Gao et al., 2024; Zhou JM et al., 2024).

However, these attention-based models for video action recognition (Mondal et al., 2023; Wang H et al., 2024; Wu WH et al., 2024) primarily depend on video-level supervision, providing only coarse-grained supervision despite the inherent fine-grained temporal variations in action videos (e.g., eating speed differences or utensil usage patterns). This discrepancy causes temporal infidelity (Liang et al., 2020; Aghaeipoor et al., 2023)—a misalignment between attention distributions and action-relevant temporal segments. For instance, as shown in Fig. 1, for actions such as “eating a sandwich,” a model may mistakenly pay significant attention to adjacent segments that are irrelevant while overlooking key discriminative moments in unseen scenarios. This limitation arises from the training paradigm which depends solely on coarse video-level supervision, and fails to capture the fine-grained temporal dynamics intrinsic to action semantics.

To address this challenge, we propose a novel temporal fidelity enhancement (TFE) framework based on the disentangled information bottleneck (DisenIB) theory (Pan et al., 2021). TFE explicitly optimizes temporal attention fidelity by decoupling the latent video embedding into the action-relevant and semantically redundant components. Unlike conventional attention refinement methods, TFE resolves infidelity through adversarial disentanglement, which maximizes the sufficiency of critical temporal features while suppressing spurious correlations from noisy contexts. The framework first encodes video segments into temporally discriminative embeddings using pre-trained models, followed by an adversarial process to approximate the DisenIB objective. A disentangler separates embeddings into temporal fidelity-preserving and fidelity-deviating ones, while dual approximators compete to retain action-relevant semantics.