A Multi-Agent Simulation Framework for Spiders Traversing the Semantic Web

TLDR: This paper introduces BioSpider, an agent-based simulation framework for developing and testing autonomous, intelligent, semantically-focused Web spiders, and assumes a direct analogy of the problem at hand with a multi-variate ecosystem, where each member is self-maintaining.

Abstract: Although search engines traditionally use spiders for traversing and indexing the web, there has not yet been any methodological attempt to model, deploy and test learning spiders. The flourishing of the Semantic Web provides un- derstandable information that may improve the accuracy of search engines. In this paper, we introduce BioSpider, an agent-based simulation framework for developing and test- ing autonomous, intelligent, semantically-focused web spi- ders. BioSpider assumes a direct analogy of the problem at hand with a multi-variate ecosystem, where each mem- ber is self-maintaining. The population of the ecosystem comprises cooperative spiders incorporating communica- tion, mobility and learning skills, striving to improve effi- ciency. Genetic algorithms and classifier rules have been employed for spider adaptation and learning. A set of ex- periments has been performed in order to qualitatively test the efficacy and applicability of the proposed approach.