GMCoT: a graph-augmented multimodal chain-of-thought reasoning framework for multi-label zero-shot learning

Abstract

In recent years, multi-label zero-shot learning (ML-ZSL) has garnered increasing attention because of its wide range of potential applications, such as image annotation, text classification, and bioinformatics. The central challenge in ML-ZSL lies in predicting multiple labels for unseen classes without requiring any labeled training data, which contrasts with conventional supervised learning paradigms. However, existing methods face several significant challenges. These include the substantial semantic gap between different modalities, which impedes effective knowledge transfer, and the intricate and typically complex relationships among multiple labels, making it difficult to model them in a meaningful and accurate manner. To overcome these challenges, we propose a graph-augmented multimodal chain-of-thought (GMCoT) reasoning approach. The proposed method combines the strengths of multimodal large language models with graph-based structures, significantly enhancing the reasoning process involved in multi-label prediction. First, a novel multimodal chain-of-thought reasoning framework is presented which imitates human-like step-by-step reasoning to produce multi-label predictions. Second, a technique is presented for integrating label graphs into the reasoning process. This technique enables the capture of complex semantic relationships among labels, thereby improving the accuracy and consistency of multi-label generation. Comprehensive experiments on benchmark datasets demonstrate that the proposed GMCoT approach outperforms state-of-the-art methods in ML-ZSL.

1. Introduction

Real-world machine learning applications such as image annotation, music categorization, and medical diagnosis require assigning more than one class label to each input instance. For example, in image annotation, a model may need to assign several labels (e.g., sky, sea, and ship) to a single image. This fundamentally differs from conventional multiclass classification, which assumes that each instance is associated with exactly one label. Developing effective multi-label classification models typically involves additional challenges. In particular, it is necessary not only to accurately associate input instances with multiple relevant labels but also to model and leverage label correlations, which frequently arise due to the co-occurrence of certain labels in real-world data.

In general, binary relevance (Tsoumakas and Katakis, 2008) is the simplest solution to multi-label classification problems in which the original task is converted into multiple disjoint binary classification problems. However, it cannot model label co-occurrences and may not be preferable. Approaches such as that proposed by Read et al. (2011) take cross-label correlation by assuming label priors, whereas label embedding-based methods (Balasubramanian and Lebanon, 2012; Chen YN and Lin, 2012; Tai and Lin, 2012; Changpinyo et al., 2016, 2017) project both input images and their labels onto a latent space to exploit label correlation. Deep neural network-based methods have also been proposed. BP-MLL (Zhang ML and Zhou, 2006) was the first to propose a loss function for modeling the dependency across labels. Other works have proposed different loss functions (Nam et al., 2014) or architectures (Wei et al., 2014; Wang J et al., 2016; Yeh et al., 2017) to further improve performance.

Multi-label zero-shot learning (ML-ZSL) extends conventional multiclass classification paradigms by addressing a more challenging setting within the zero-shot learning (ZSL) framework. Unlike standard classification tasks, ML-ZSL requires models to simultaneously predict multiple semantic labels for each instance, including novel classes that are absent during training. This poses a complex inference problem that requires the design of advanced transfer mechanisms capable of exploiting semantic relationships between seen and unseen label spaces. The core innovation of ML-ZSL lies in its ability to generalize learned representations across disjoint label distributions while capturing label co-occurrence patterns.