Sthithika: distributed complex event processing with query rewriting

Abstract

This paper presents the STHITHIKA distributed Complex Event Processing (CEP) system, which is focused on the problem of optimally processing a large number of different event streams using a large number of CEP queries in a distributed manner. Optimization is done in two levels: individual query optimization using query rewriting, and optimization of query distribution across multiple nodes. Cost of individual queries, number of events streams common to queries, CPU and memory utilization of nodes that run CEP queries, type of queries, and the number of queries in each node are the factors considered in the query distribution algorithm. Experiments show that with these optimizations, compared to existing systems, STHITHIKA is capable of providing a higher system throughput, without making an adverse impact on event duplication or load variance across processing nodes. It is also more robust to event bursts.