TimeJudge: empowering video-LLMs as zero-shot judges for temporal consistency in video captions

Abstract

Video large language models (video-LLMs) have demonstrated impressive capabilities in multimodal understanding, but their potential as zero-shot evaluators for temporal consistency in video captions remains underexplored. Existing methods notably underperform in detecting critical temporal errors, such as missing, hallucinated, or misordered actions. To address this gap, we introduce two key contributions. (1) TimeJudge: a novel zero-shot framework that recasts temporal error detection as answering calibrated binary question pairs. It incorporates modality-sensitive confidence calibration and uses consistency-weighted voting for robust prediction aggregation. (2) TEDBench: a rigorously constructed benchmark featuring videos across four distinct complexity levels, specifically designed with fine-grained temporal error annotations to evaluate video-LLM performance on this task. Through a comprehensive evaluation of multiple state-of-the-art video-LLMs on TEDBench, we demonstrate that TimeJudge consistently yields substantial gains in terms of recall and F1-score without requiring any task-specific fine-tuning. Our approach provides a generalizable, scalable, and training-free solution for enhancing the temporal error detection capabilities of video-LLMs.

1. Introduction

Video large language models (video-LLMs) are rapidly transitioning from research prototypes to real-world products, making rigorous and scalable evaluation indispensable. Although human assessment remains the gold standard, it is slow, costly, and subjective, prompting a shift toward automated, model-based protocols.

Building on the success of the "LLM-as-a-Judge" paradigm for text, the emerging "Video-LLM-as-a-Judge" framework proposes to apply powerful video-LLMs to score, rank, and filter the video captions generated by other video language models (VLMs) (Zheng et al., 2023; Liu and Zhang, 2025). Besides offering low-cost, high-throughput evaluation, this approach accelerates the evolution of video-LLMs from dialogue systems into general-purpose multimodal agents with broad applications in evaluation, alignment, retrieval, and reasoning (Bai YS et al., 2023; Lee et al., 2023; Li RS et al., 2023; Liang et al., 2023; Liao et al., 2024; Xu et al., 2024).

However, when applied to video-understanding benchmarks, existing video-LLM judges prove unreliable. As illustrated in Fig. 1, directly prompting them to verify the fidelity of machine-generated captions often yields incorrect verdicts, particularly regarding temporal coherence, missing events, hallucinated actions, or misordered sequences. We identify three underlying causes: (1) an over-reliance on static spatial cues (e.g., objects and scenes) at the expense of temporal perspicacity; (2) limited capacity for multi-hop, high-level reasoning in the temporal domain; (3) pronounced biases toward textual priors or visually dominant patterns rather than grounding decisions on video evidence. These shortcomings, rooted in both training data and model architecture, erode the reliability of video-LLM-based evaluation for temporally sensitive tasks such as event localization and action detection.