A Big data analytic framework over federated data centers for intelligent travel recommenders

Abstract

Big Data is a series of enormous and complex data sets that are nearly impossible to store and process using traditional data storing and processing methods. The emergence of heterogeneous data in different domains causes significant challenges in data manipulation and decision making. In recent years, the requirement for analysis of heterogeneous data on distributed data storages has been increased and has gained a lot of researchers’ attention. Distributed data storage systems and parallel data processing techniques are typically used for data-intensive computing today. Due to the rapid growth of data, a single-cluster environment is inadequate to process that much data. At the same time, there are heterogeneous data sources on different platforms, which need to inter-connect to derive meaningful analysis. The MapReduce software paradigm has surfaced to fill the gap, and it has been successfully operating on systems. However, only single cluster environments are supported by the current implementation of MapReduce and cannot be applied to federated heterogeneous data centers. Hence, it does not have enough capabilities to process heterogeneous data sources. This research presents a big data analytic framework that supports the integration of heterogeneous data sources on distributed computing models across different data centers. The architecture of this framework is based on a master/slave distributed computing model and Map - Reduce - Merge - GlobalReduce is presented as the programming model. Besides, the performance of the novel approach is measured under different cluster configurations, and experimental evaluations had shown promising results for the proposed framework compared to a single cluster environment.