Key Takeaways & Executive Findings
- •• Visual knowledge is a novel knowledge representation rooted in cognitive psychology, encapsulating visual concepts and relations in an interpretable form. • Large AI models (foundation models) extract implicit knowledge from data but suffer from opacity, data demands, and hallucination. • Integrating visual knowledge can potentially alleviate weaknesses of large models by providing interpretable reasoning and real-world understanding. • The paper reviews the origins and development of visual knowledge and outlines promising future directions for the big model era.
Abstract
Visual knowledge is a new form of knowledge representation that can encapsulate visual concepts and their relations in a succinct, comprehensive, and interpretable manner, with a deep root in cognitive psychology. As the knowledge about the visual world has been identified as an indispensable component of human cognition and intelligence, visual knowledge is poised to have a pivotal role in establishing machine intelligence. With the recent advance of Artificial Intelligence (AI) techniques, large AI models (or foundation models) have emerged as a potent tool capable of extracting versatile patterns from broad data as implicit knowledge, and abstracting them into an outrageous amount of numeric parameters. To pave the way for creating visual knowledge empowered AI machines in this coming wave, we present a timely review that investigates the origins and development of visual knowledge in the pre-big model era, and accentuates the opportunities and unique role of visual knowledge in the big model era.
1. Introduction
The concept of visual knowledge (Pan, 2019) was recently proposed as a form of knowledge representation that differs from the traditional ones adopted/learned by symbolic and sub-symbolic AI approaches (e.g., knowledge graph, handcrafted image descriptors, distributed visual representations). Drawing on cognitive studies (Anderson and Crawford, 1980) of human mental imagery, which enables us to manipulate visual entities in our mind, visual knowledge theory posits that next-generation AI needs to fully express visual concepts and their attributes (e.g., shape, structure, motion, affordance), as well as reason about their transformations, compositions, comparisons, predictions, and narrations, through a unified, abstract, and interpretable form of representation.
After the emergence of large language models like GPT-3 (Brown et al, 2020), the field of natural language processing has experienced remarkable advancements: traditional “narrow” language models that are trained to perform specific tasks in a single domain are giving way to highly sophisticated and versatile language models that are trained on a vast corpus of unlabeled textual data that can be used for different language tasks across domains. Like GPT for natural language processing, the recent work known as Segment Anything Model (SAM) (Wu et al, 2023) ushered the field of computer vision into the era of visual foundation models — by training on >1B segmentation masks in >11M natural images, SAM shows the promise of a broad applicability to various image segmentation tasks, without re-training or fine-tuning as previously needed. With incredible speed, large models are revolutionizing AI field and transforming the landscape of scientific research.
Albeit the unprecedented progress, it is becoming increasingly evident that large AI models still suffer several deficiencies that compromise their reliability and efficacy. Chief among these are their pronounced opacity, which poses great challenges for trust, accountability, and effective debugging, as well as their insatiable demand for data and computational resources, which raise both ethical and environmental concerns. These limitations are inherited from their rudimentary predecessors but exacerbated by their heightened sophistication and scale. Compounding these concerns, large AI models are susceptible to generating nonsensical or unfaithful content, known as “hallucination”, exposing their inherent biases, lack of real-world understanding, and weakness in generalizing or reasoning beyond their scope.
Loading authentic research manuscript (Pages 1–5)...
Wenguan WANG, Yi YANG, Yunhe PAN (2025). Visual Knowledge in the Big Model Era: Retrospect and Prospect. arXiv. https://doi.org/10.1631/FITEE_2400250
Research & Educational Purpose Only:The translations, structured abstracts, analytical annotations, and data reports provided by SinoTechIntel are intended exclusively for academic research, internal corporate R&D, and educational benchmarking. They do not constitute formal engineering, chemical safety, legal, or professional advice.
Copyright & Intellectual Property Notice: Original copyright of the underlying source articles and experimental data remains with the respective authors, institutions, and original publishing journals. SinoTechIntel claims intellectual property only over its proprietary translations, analytical syntheses, and AEO structured enhancements in accordance with international fair use and academic citation principles.
Frequently Asked Questions
What is visual knowledge?
Visual knowledge is a new form of knowledge representation that encapsulates visual concepts and their relations in a succinct, comprehensive, and interpretable manner. It is deeply rooted in cognitive psychology and is considered essential for machine intelligence.
How does visual knowledge relate to large AI models?
Large AI models (foundation models) extract implicit knowledge from broad data as numeric parameters. The paper argues that visual knowledge can complement these models by providing interpretable representation and reasoning, potentially alleviating weaknesses such as opacity and hallucination.
What are the main weaknesses of big AI models?
Big AI models suffer from pronounced opacity, high data and computational demands, and a tendency to generate nonsensical or unfaithful content (hallucination). These issues raise concerns about trust, accountability, ethics, and environmental impact.
How can visual knowledge help improve large models?
Visual knowledge offers expressive and interpretable representation, manipulation, and reasoning of visual concepts. Incorporating it into large models can enhance their real-world understanding, generalizability, and reasoning capabilities, thereby addressing some of their inherent limitations.
What does the paper review and prospect?
The paper provides a timely review of the origins and development of visual knowledge in the pre-big model era, and outlines promising directions for exploring visual knowledge in the big model era, emphasizing its unique role in creating visual knowledge empowered AI machines.
Related Technical Papers & Translations
Design and optimization of a high-efficiency distillation process for cellulosic fuel ethanol integrated with thermal coupling and molecular sieve adsorption
To address the challenges of high energy consumption and prominent costs in the traditional three-columns distillation process for cellulosic fuel ethanol, a distillation—molecular sieve coupling separation process is proposed. This process integrates a three-column (crude distillation column, first distillation column, second distillation column) system with a 3A molecular sieve adsorption deep dehydration unit. A thermal coupling network is constructed via differential pressure design (steam from medium/high-pressure columns as mutual heat sources, reboiler liquid waste heat for feed preheating), and molecular sieve adsorption conditions are optimized. The study first performs a thermodynamic consistency test on the ethanol—water system, determines optimal non-random two-liquid (NRTL) model binary interaction parameters via experimental data regression for Aspen Plus simulation. Aiming at minimum total annual cost (TAC), Aspen Plus is used to optimize process parameters (theoretical tray number, feed location, reflux ratio, side-draw position, etc.). Economic analysis shows this process reduces CO2 emission costs by 27.56%, TAC by 15.58% (to 5.123 × 106 USD·a-1), and increases ethanol purity to ＞99.6%, providing an effective solution for green, efficient separation.
A cohesion loss model for determining residual strength of deep bedded sandstone
Rock residual strength, as an important input parameter, plays an indispensable role in proposing the reasonable and scientific scheme about stope design, underground tunnel excavation and stability evaluation of deep chambers. Therefore, previous residual strength models of rocks established were reviewed. And corresponding related problems were stated. Subsequently, starting from the effects of bedding and whole life-cycle evolution process, series of triaxial mechanical tests of deep bedded s
Federated model with contrastive learning and adaptive control variates for human activity recognition
Recent attention to privacy issues demands a communication-safe method for training human activity recognition (HAR) models on client activity data. Federated learning (FL) has become a compelling technique to facilitate model training between the server and clients while preserving data privacy. However, classical FL methods often assume independent and identically distributed (IID) data among clients. This assumption does not hold true in practical scenarios. Human activity in real-world scena