Authors: Shah Md. Tanzimul Kabir, Zahid Hassan Ome
Abstract: This paper provides a comprehensive analysis of scalable database systems, specifically designed to support big data analytics, and examines their evolution, challenges, and emerging technologies in the exascale data processing era. By examining recent research studies from 2021 to 2026, the current paper seeks to investigate how distributed database architectures, including NewSQL, cloud-native, and data lakehouse, address the fundamental scalability challenge known as the "scalability trilemma" consisting of consistency, availability, and partition tolerance. The current research introduces the Adaptive Scalability Evaluation Framework (ASEF), which integrates horizontal scaling, elastic resources, query optimization, and storage efficiency. The analysis shows that recent scalable database architectures are based on disaggregated storage and compute architectures, enabling near-linear scaling to thousands of nodes with query latencies under 100ms for petabyte-scale data sets. Cloud-native database architectures are shown to be highly elastic, with variations in query latency at the 95th percentile below 15% during scaling events. Newly emerging architectures for lakehouses, which bring the flexibility of data lakes and the performance of data warehouses, provide query performance that is 3 to 5 times better than traditional data lakes and reduce the total cost of ownership by 30 to 50 percent. Evaluation in five dimensions for analytical workloads, such as scaling behavior, consistency model, query performance, storage efficiency, and operational complexity, shows that systems with workload awareness and adaptivity perform much better than static configurations. Continuous optimization provides an improvement in throughput performance that is between 2 to 4 times.
DOI: https://doi.org/10.5281/zenodo.20270564
