A focused crawling strategy based on comprehensive priority evaluation of hyperlinks and improved Bayesian classifier

Abstract

Avoidance of topic drift and enabling crossing tunnels are two main difficulties in focused crawling. To overcome the problem of topic drift, we design a comprehensive priority evaluation (CPE) method based on the web text, anchor text, and context of hyperlinks, which improves the topic-relevance evaluation of unvisited hyperlinks. Subsequently, we propose an improved Bayesian classifier with weights (BCW), which adds label weights to the feature words of the Bayesian classifier to enhance the accuracy of webpage classification. To cross tunnels through which some topic-relevant webpages can be reached from low-relevance webpages, we construct a content block segmentation (CBS) technology for webpages based on the backtracking method, which segments a webpage into multiple blocks and then judges the relevance of every content block, extracting hyperlinks with high comprehensive relevance. Finally, a BCW-based focused crawling strategy combining the CPE and CBS strategies (BCW_CC) is proposed and experimentally evaluated for focused crawling in two domains: rainstorm disasters and sports. The results demonstrate the effectiveness of the developed BCW_CC method.

1. Introduction

Owing to its diversity, timeliness, and sharing ability, the Internet has become an important information source. The 53rd survey report of the China Internet Network Information Center (2024) recorded 3.88 million websites and 382 billion webpages in China as of December 2023. The number of webpages has increased by 6.5% since December 2022. The number of webpages on the global Internet is even vaster and very difficult to estimate. Faced with such huge resources, traditional search engines such as Google and Baidu cannot always match users’ personalized needs to topic-relevant webpages. Furthermore, some existing open-source crawler tools, such as WebCollector, Crawler4j, Scrapy, and Nutch, are generally limited by low recall and accuracy (Yu J and Liu, 2015; Hosseinkhani et al., 2021). Unlike general web crawlers, the focused crawler (FC) (Deng, 2020; Xiong and Yang, 2025) can filter webpages related to specific topics.

The FC comprehensively judges the topic relevance of webpages by setting a threshold or classifier based on various evaluation indicators of the webpages, according to the needs of users. The FC, which usually crawls the webpages of specific topics more accurately than traditional crawler tools, is widely applied in information filtering, precise information retrieval, data mining and analysis, and large models (for example, data crawling of pre-trained corpora). The FC can also collect public opinions and observe the emotional tendencies on specific topics on social media and news websites, helping enterprises and governments understand the public’s views on a certain topic. This paper investigates the FC on two topics: rainstorm disasters and sports events. Early warnings, preventive measures, and emergency response information are essential for reducing and avoiding losses caused by a rainstorm disaster and ensuring the safety of human life and property. In sports, the FC can not only recommend important sports events related to the user’s interests but also provide users with relevant comments at a deeper level. However, the information in specific fields is generally sparse and scattered throughout the vast Internet, possessing big data characteristics that challenge the accuracy rate (AC) of information retrieval by FC.

Most of the current FC methods are based on heuristic strategies, semantic analysis, or machine learning.