Analysis of performance, integration, and scalability limitations of data virtualization layers for big data processing in emerging use cases

Abstract

In today’s data-driven world, data virtualization has emerged as a transformative technology that addresses these needs by providing a unified, abstract view of data from heterogeneous sources. Unlike traditional data integration methods, Data virtualization eliminates the need for physical data movement and replication, enabling seamless, real-time data access while reducing complexity and enhancing agility. These features make Data virtualization a cornerstone of modern data management systems, particularly in applications like business intelligence, data science, big data analytics and cloud computing [1]. However, despite its potential, data virtualization faces significant performance, scalability, and integration challenges when applied to complex, different varieties of workloads and application scenarios. These limitations hinder its adoption in scenarios involving high volumes of data and federated queries across diverse systems. This research investigates these critical challenges to bridge the gap between Data virtualization’s potential and its practical implementation. Additionally, it explores how Data virtualization can be optimized for emerging low-resource applications, such as personal IoT data management or small-scale analytics, extending its utility beyond enterprise environments.