Performance of any search engine relies heavily on its Web crawler. Web crawlers are the programs that get webpages from the Web by following hyperlinks. These webpages are indexed by a search engine and can be retrieved by a user query. In the area of Web crawling, we still lack an exhaustive study that covers all crawling techniques. This study follows the guidelines of systematic literature review and applies it to the field of Web crawling. We used the standard procedure of carrying out a systematic literature review on 248 studies from a total of 1488 articles published in 12 leading journals and other premier conferences and workshops. Existing literature about the Web crawler is classified into different key subareas. Each subarea is further divided according to the techniques being used. We analyzed the distribution of various articles using multiple criteria and depicted conclusions. Various studies that use open source Web crawlers are also reported. We have highlighted future areas of research. We call for an increased awareness in various fields of the Web crawler and identify how techniques from other domains can be used for crawling the Web. Limitations and recommendations for future are also discussed. WIREs Data Mining Knowl Discov 2017, 7:e1218. doi: 10.1002/widm.1218

A survey of Web crawlers for information retrieval

– The purpose of this paper is to decrease the traffic created by search engines’ crawlers and solve the deep web problem using an innovative approach. , – A new algorithm was formulated based on best existing algorithms to optimize the existing traffic caused by web crawlers, which is approximately 40 percent of all networking traffic. The crux of this approach is that web servers monitor and log changes and communicate them as an XML file to search engines. The XML file includes the information necessary to generate refreshed pages from existing ones and reference new pages that need to be crawled. Furthermore, the XML file is compressed to decrease its size to the minimum required. , – The results of this study have shown that the traffic caused by search engines’ crawlers might be reduced on average by 84 percent when it comes to text content. However, binary content faces many challenges and new algorithms have to be developed to overcome these issues. The proposed approach will certainly mitigate the deep web issue. The XML files for each domain used by search engines might be used by web browsers to refresh their cache and therefore help reduce the traffic generated by normal users. This reduces users’ perceived latency and improves response time to http requests. , – The study sheds light on the deficiencies and weaknesses of the algorithms monitoring changes and generating binary files. However, a substantial decrease of traffic is achieved for text-based web content. , – The findings of this research can be adopted by web server software and browsers’ developers and search engine companies to reduce the internet traffic caused by crawlers and cut costs. , – The exponential growth of web content and other internet-based services such as cloud computing, and social networks has been causing contention on available bandwidth of the internet network. This research provides a much needed approach to keeping traffic in check.

Search engines crawling process optimization: a webserver approach

Centralized crawlers are not adequate to spider meaningful and relevant portions of the Web. A crawler with good scalability and load balancing can bring growth to performance. As the size of web is growing, in order to complete the downloading of pages in fewer amounts of time and increase the coverage of crawlers it is necessary to distribute the crawling process. In this paper, we present client server architecture based smart distributed crawler for crawling web. In this architecture load between the crawlers is managed by server and each time a crawler is loaded, load is distributed to others by dynamically distributing the URLs. Focused crawlers makes efficient usage of network bandwidth and storage capacity, when distributed can enhance the performance.

Smart distributed web crawler

The World Wide Web (WWW) is a diverse source of information. A large part of the Web is hidden behind search forms and is reachable only when a user types in a set of keywords or queries. This part of Web is known as hidden Web or deep Web. The webpages in the hidden Web are not accessible by following hyperlinks and hence are not indexed by the search engine. Various strategies are proposed in the literature for crawling the hidden Web that suggested form filling and getting response from the server. These request response based analysis techniques result in poor efficiency and network utilization. This paper contributes a novel approach for crawling the large size data in the hidden Web by sending the mobile crawler to the remote server. As mobile crawlers are scalable and do not require query response phenomenon, they can crawl the data in the hidden Web with minimum resources. The major advantage of using mobile crawler is moving code to data. As indexing and analysis is performed locally at the data server, it helps in reducing network load. Moreover, once the crawler is at server end the heterogeneous data can be accessed by issuing efficient queries.

Design of a mobile Web crawler for hidden Web

Integration and Analysis of Agricultural Market Information Based on Web Mining

Maintaining the search engine freshness using mobile agent

Article history: Received 28 Dec 2021 Revised 02 Feb 2022 Accepted 08 Feb 2022 Available online The revolutionary in cyber attacks, especially in smartphones are rising. The Android operating system is becoming one of the most leading operating systems. Therefore, Android malware is rising in terms of popularity. Malware makers are using novel techniques to develop malicious Android applications, drastically diminishing the capabilities of traditional malware detectors. In consequence, those Anti-malware detectors become unable to detect these unexplained malicious apps. Currently, machine learning techniques are extensively used to discover new unknown Android viruses by analyzing the functionality of static and dynamic app reviews. In this paper, we introduce an Android malware detection technique based on API and permissions. Our purpose is to evaluate and examine the incorporation of machine learning classifiers with featured Android features such as APIs and permissions. We investigated several classification methods in characterizing Android malware with respect to the used feature. We discovered varied performance when we analyses all Android malware detection classifiers that use machine learning, suggesting that machine learning algorithms are effectively utilized in this area of identifying Android malicious apps.

Ammar Mohamed

Papers

Maintaining the search engine freshness using mobile agent

Using Machine Learning to Identify Android Malware Relying on API calling sequences and Permissions