Complex Event Processing Framework for Big Data Applications
01 Jan 2016-pp 41-56
TL;DR: This chapter proposes CEP-based solution to continuously collect and analyze the data generated from multiple sources in real time to detect and produce timely reaction to the occurrence of real-world situations in the system environment.
Abstract: The fundamental requirement for modern IT systems is the ability to detect and produce timely reaction to the occurrence of real-world situations in the system environment. This applies to any of the Internet of Things (IoT) applications where number of sensors and other smart devices are deployed. These sensors and smart devices embedded in IoT networks continually produce huge amounts of data. These data streams from heterogeneous sources arrive at high rates and need to be processed in real time in order to detect more complex situations from the low-level information embedded in the data. Complex event processing (CEP) has emerged as an appropriate approach to tackle such scenarios. Complex event processing is the technology used to process one or more streams of data/events and identify patterns of interest from multiple streams of events to derive a meaningful conclusion. This chapter proposes CEP-based solution to continuously collect and analyze the data generated from multiple sources in real time. Two case studies on intrusion detection in a heterogeneous sensor network and automated healthcare monitoring of geriatric patient are also considered for experimenting and validating the proposed solutions.
Citations
More filters
TL;DR: In this paper, the authors provide a systematic and comprehensive literature review of recent papers to draw a framework of the past, and to shed light on potential directions for future research in Logistics Internet-of-Things (L-IoT).
70 citations
TL;DR: This paper proposes a geospatial extension for CEP that allows monitoring environmental requirements considering the geographic location of the processed data by extending an existing platform-independent, model-driven approach and specifying patterns using geographic operators.
Abstract: In a context of e-government, there are usually regulatory compliance requirements that support systems must monitor, control and enforce. These requirements may come from environmental laws and regulations that aim to protect the natural environment and mitigate the effects of pollution on human health and ecosystems. Monitoring compliance with these requirements involves processing a large volume of data from different sources, which is a major challenge. This volume is also increased with data coming from autonomous sensors (e.g. reporting carbon emission in protected areas) and from citizens providing information (e.g. illegal dumping) in a voluntary way. Complex Event Processing (CEP) technologies allow processing large amount of event data and detecting patterns from them. However, they do not provide native support for the geographic dimension of events which is essential for monitoring requirements which apply to specific geographic areas. This paper proposes a geospatial extension for CEP that allows monitoring environmental requirements considering the geographic location of the processed data. We extend an existing platform-independent, model-driven approach for CEP adding the geographic location to events and specifying patterns using geographic operators. The use and technical feasibility of the proposal is shown through the development of a case study and the implementation of a prototype.
3 citations
Cites background from "Complex Event Processing Framework ..."
...Indeed, CEP technologies are applicable to big data scenarios and we consider that CEP and big data technologies can benefit from each other [41, 42]....
[...]
14 Oct 2017
TL;DR: To unlock the potential of IoT, a system architecture, termed QuantCloud, is proposed for modern quantitative trading firms in the field, and an overview of challenges and opportunities presented by this new paradigm in the QF industry is provided.
Abstract: The Internet of Things (IoT) is a novel paradigm that communicates information among smart devices that are connected to the Internet. In this context, such devices would leverage our understanding and capabilities of big data, deep analysis and artificial intelligence to solve problems in real-time. The IoT paradigm has successfully benefited many applications in the social sciences and industries. However, in the rise of IoT, there is at least one question that has been left unanswered: Can Quantitative Finance (QF) benefit from IoT? The QF is a field that extends sophisticated mathematical models and utilizes advanced computer techniques to link with global finance markets. By taking market and social information as input, a QF model can derive profitable insights and control the risk to make trading decisions. Today, many Internet-based techniques are extensively employed in the field as: (a) market and social data is provided via Internet; (b) big data infrastructures are built in the Cloud; and (c) deep learning tools are accessible in Internet. Even trading models and strategies could be exerted through Internet. In this paper, we will provide an overview of challenges and opportunities presented by this new paradigm in the QF industry. To unlock the potential of IoT, a system architecture, termed QuantCloud, is proposed for modern quantitative trading firms in the field.
2 citations
Cites background from "Complex Event Processing Framework ..."
...The financial services industry is a pioneer in utilizing the complex event processing (CEP) technology [11] to organize data-driven events so that it could inform algorithmic trading behavior by timely identifying opportunities and/or risks....
[...]
...An ideal case is when the time-series databases are data providers for historical and real-time data; meanwhile, the CEP and AI methods are data consumers to derive hidden opportunities and assess portfolio risks [3, 6, 11]....
[...]
...Such CEP-based approach is also a popular IoT solution to process multiple streams of data/events to identify patterns of interest [11, 12]....
[...]
01 Jan 2023
TL;DR: In this paper , a methodology for verifying the status and evaluating the level of pesticides in regularly consumed veggies as well as fruits has been identified, named as the Hybrid Chronic Multi-Residual Framework (HCMF), in which the harmful level of used pesticide residues has been predicted for contamination in agro products using Q-Learning based Recurrent Neural Network and the predicted contamination levels have been analyzed using Complex Event Processing (CEP) by processing given spatial and sequential data.
Abstract: Pesticides have become more necessary in modern agricultural production. However, these pesticides have an unforeseeable long-term impact on people's wellbeing as well as the ecosystem. Due to a shortage of basic pesticide exposure awareness, farmers typically utilize pesticides extremely close to harvesting. Pesticide residues within foods, particularly fruits as well as veggies, are a significant issue among farmers, merchants, and particularly consumers. The residual concentrations were far lower than these maximal allowable limits, with only a few surpassing the restrictions for such pesticides in food. There is an obligation to provide a warning about this amount of pesticide use in farming. Previous technologies failed to forecast the large number of pesticides that were dangerous to people, necessitating the development of improved detection and early warning systems. A novel methodology for verifying the status and evaluating the level of pesticides in regularly consumed veggies as well as fruits has been identified, named as the Hybrid Chronic Multi-Residual Framework (HCMF), in which the harmful level of used pesticide residues has been predicted for contamination in agro products using Q-Learning based Recurrent Neural Network and the predicted contamination levels have been analyzed using Complex Event Processing (CEP) by processing given spatial and sequential data. The analysis results are used to minimize and effectively use pesticides in the agricultural field and also ensure the safety of farmers and consumers. Overall, the technique is carried out in a Python environment, with the results showing that the proposed model has a 98.57% accuracy and a training loss of 0.30.
References
More filters
TL;DR: This paper surveys context awareness from an IoT perspective and addresses a broad range of techniques, methods, models, functionalities, systems, applications, and middleware solutions related to context awareness and IoT.
Abstract: As we are moving towards the Internet of Things (IoT), the number of sensors deployed around the world is growing at a rapid pace. Market research has shown a significant growth of sensor deployments over the past decade and has predicted a significant increment of the growth rate in the future. These sensors continuously generate enormous amounts of data. However, in order to add value to raw sensor data we need to understand it. Collection, modelling, reasoning, and distribution of context in relation to sensor data plays critical role in this challenge. Context-aware computing has proven to be successful in understanding sensor data. In this paper, we survey context awareness from an IoT perspective. We present the necessary background by introducing the IoT paradigm and context-aware fundamentals at the beginning. Then we provide an in-depth analysis of context life cycle. We evaluate a subset of projects (50) which represent the majority of research and commercial solutions proposed in the field of context-aware computing conducted over the last decade (2001-2011) based on our own taxonomy. Finally, based on our evaluation, we highlight the lessons to be learnt from the past and some possible directions for future research. The survey addresses a broad range of techniques, methods, models, functionalities, systems, applications, and middleware solutions related to context awareness and IoT. Our goal is not only to analyse, compare and consolidate past research work but also to appreciate their findings and discuss their applicability towards the IoT.
2,542 citations
Book•
01 May 2002
TL;DR: Some possible long-term future roles of CEP in the Information Society are discussed along with the need to develop rule-based event hierarchies on a commercial basis to make those applications possible.
Abstract: Complex Event Processing (CEP) is a defined set of tools and techniques for analyzing and controlling the complex series of interrelated events that drive modern distributed information systems. This emerging technology helps IS and IT professionals understand what is happening within the system, quickly identify and solve problems, and more effectively utilize events for enhanced operation, performance, and security. CEP can be applied to a broad spectrum of information system challenges, including business process automation, schedule and control processes, network monitoring and performance prediction, and intrusion detection.
This talk is about the rise of CEP as we know it today, its historical roots and its current position in commercial markets. Some possible long-term future roles of CEP in the Information Society are discussed along with the need to develop rule-based event hierarchies on a commercial basis to make those applications possible. The talk gives empahsis to the point that "Rules are everywhere" and that mathematical formalisms cannot express all the forms that are in use in various event processing systems.
1,380 citations
TL;DR: A framework for the realization of smart cities through the Internet of Things (IoT), which encompasses the complete urban information system, from the sensory level and networking support structure through to data management and Cloud-based integration of respective systems and services, and forms a transformational part of the existing cyber-physical system.
Abstract: Increasing population density in urban centers demands adequate provision of services and infrastructure to meet the needs of city inhabitants, encompassing residents, workers, and visitors. The utilization of information and communications technologies to achieve this objective presents an opportunity for the development of smart cities, where city management and citizens are given access to a wealth of real-time information about the urban environment upon which to base decisions, actions, and future planning. This paper presents a framework for the realization of smart cities through the Internet of Things (IoT). The framework encompasses the complete urban information system, from the sensory level and networking support structure through to data management and Cloud-based integration of respective systems and services, and forms a transformational part of the existing cyber-physical system. This IoT vision for a smart city is applied to a noise mapping case study to illustrate a new method for existing operations that can be adapted for the enhancement and delivery of important city services.
1,178 citations
TL;DR: The design and implementation of a complete running system, called VigilNet, for energy-efficient surveillance, which allows a group of cooperating sensor devices to detect and track the positions of moving vehicles in an energy- efficient and stealthy manner is described.
Abstract: This article describes one of the major efforts in the sensor network community to build an integrated sensor network system for surveillance missions. The focus of this effort is to acquire and verify information about enemy capabilities and positions of hostile targets. Such missions often involve a high element of risk for human personnel and require a high degree of stealthiness. Hence, the ability to deploy unmanned surveillance missions, by using wireless sensor networks, is of great practical importance for the military. Because of the energy constraints of sensor devices, such systems necessitate an energy-aware design to ensure the longevity of surveillance missions. Solutions proposed recently for this type of system show promising results through simulations. However, the simplified assumptions they make about the system in the simulator often do not hold well in practice, and energy consumption is narrowly accounted for within a single protocol. In this article, we describe the design and implementation of a complete running system, called VigilNet, for energy-efficient surveillance. The VigilNet allows a group of cooperating sensor devices to detect and track the positions of moving vehicles in an energy-efficient and stealthy manner. We evaluate VigilNet middleware components and integrated system extensively on a network of 70 MICA2 motes. Our results show that our surveillance strategy is adaptable and achieves a significant extension of network lifetime. Finally, we share lessons learned in building such an integrated sensor system.
550 citations
01 Jan 2006
TL;DR: The correctness, robustness, and extensibility of the system architecture is shown through a scenario-based evaluation of the integrated ALARM-NET system, as well as performance data for individual software components.
Abstract: We describe ALARM-NET, a wireless sensor network for assisted-living and residential monitoring. It integrates environmental and physiological sensors in a scalable, heterogeneous architecture. A query protocol allows real-time collection and processing of sensor data by user interfaces and back-end analysis programs. One such program determines circadian activity rhythms of residents, feeding activity information back into the sensor network to aid context-aware power management, dynamic privacy policies, and data association. Communication is secured end-to-end to protect sensitive medical and operational information. The ALARM-NET system has been implemented as a network of MICAz sensors, stargate gateways, iPAQ PDAs, and PCs. Customized infrared motion and dust sensors, and integrated temperature, light, pulse, and blood oxygenation sensors are present. Software components include: TinyOS query processor and security modules for motes; AlarmGate, an embedded Java application for managing power, privacy, security, queries, and client connections; Java resident monitoring and sensor data querying applications for PDAs and PCs; and a circadian activity rhythm analysis program. We show the correctness, robustness, and extensibility of the system architecture through a scenario-based evaluation of the integrated ALARM-NET system, as well as performance data for individual software components.
371 citations