Gathika : dynamic query distribution mechanism for complex event processing systems

Abstract

Complex Event Processing (CEP) is heavily used in real time systems where people are interested in extracting valuable information from event streams. Scalability and fault tolerance are major requirements for such systems that do complex event processing. It is very hard to rely on a single machine to do the processing of all events. Therefore, requiring distributed systems for processing event streams is an obvious choice. Such a system should be able to cater to the requirement of processing a large number of events. Event queries are deployed in event processing nodes to extract useful information from event streams. In real time, event processing nodes get overloaded due to event bursts. In addition, there are situations where a large set of queries need to be deployed to extract useful information from the events. Due to all these conditions, the overall throughput of the whole system degrades. Distribution of queries is therefore essential in a complex event processing system. Distributing complex queries statically within the event processing nodes (at system initialization) is not a trivial task. Dynamic query distribution (during system operation time) is even harder due to factors such as fault tolerance, availability, scalability, predictable performance, and security requirements of the distributed CEP system. Network connectivity and the status of the processing nodes are some of the essential factors that need to be considered when doing query distribution. This research focuses on developing dynamic query distribution mechanisms for a distributed complex event processing system. A dynamic query distribution algorithm capable of deploying the queries dynamically across the nodes of the distributed CEP system is designed. Query distribution is done considering the resource utilization levels of the event processing nodes, the complexity of the query to be deployed, and the type of queries deployed in the processing nodes. Through our experiments, it was evident that the performance of the system is proportional to the number of processing nodes in the system. When dynamic query distribution is properly executed, the overall system performance can be improved by balancing the load among the processing nodes. Two important rules were defined to guarantee this proper execution: minimum time between two successive dynamic query distributions and minimum number of queries to trigger dynamic query distribution in the system. Having low latency when distributing queries dynamically and high throughput after dynamic query distribution are the key success of this dynamic query distribution mechanism. Therefore, it is beneficial to have a dynamic query distribution mechanism in CEP systems that experience frequent event bursts and query/node deployments.