STHITHIKA cost-based query distribution with query re-writing distributed complex event processing systems

Abstract

Complex event processing (CEP) is very useful in analyzing event streams and identifying useful patterns from them. Due to the distributed nature of existing applications, high volume of event generation and complex queries, using a single node CEP became problematic. One way to overcome this problem is to introduce multiple complex event processing nodes and distribute the queries among them for load balancing. However, due to the stateful nature of events, distributing queries among CEP nodes is not an easy task. Query distribution across CEP nodes is an NP hard problem. This research is focused on the problem of optimally processing a large number of different event streams using a large number of CEP queries in a distributed manner. Optimization of query processing and distribution is done in two aspects: optimizing the individual query by introducing query rewriting, and optimizing query distribution across multiple nodes by introducing new factors to the query distribution algorithm. Cost of individual queries, number of event streams common to queries, CPU and memory utilization of nodes that run CEP queries, type of queries, and the number of queries in each node are the factors considered for query distribution. Usability improvement is done in two ways: adding standard communication by introducing JSON messages for communication, and integrating firebase messaging service to standardize the event source. Experiments show that with these optimizations, compared to existing systems, STHITHIKA is capable of providing a higher system throughput, without making an adverse impact on event duplication or process load variance across processing nodes. It has the ability to handle higher number of queries compared to existing system. It is also more robust to event bursts. Due to the changes in query distribution and re-writing, time taken for initial query distribution has increased. Usability improvement enabled the easy integration with other technology and decoupling event source from the system.