Authors: Khaleel Khan Mohammed
Abstract: Data engineering is now an essential subject for handling, processing, and analysing big data as the amount of data collected is increasing exponentially. This paper gives a future-focused overview of data engineering. The creation, building, upkeep, and optimization of data architecture, infrastructure, and pipelines are all essential components of data engineering, a field within data science. This paper presents a systematic study of data engineering pipelines with a focus on leakage-safe data splitting, preprocessing order, evaluation protocols, and reproducibility practices. We outline a canonical preprocessing workflow that enforces strict separation between training and evaluation data while ensuring that all data-dependent transformations are learned exclusively from training partitions. The paper further discusses suitable validation strategies for both static and time-dependent data, emphasizes the role of nested and repeated cross-validation, and highlights the importance of ablation and stability analysis in assessing model robustness. Finally, we examine provenance-aware logging and experiment tracking as essential components for reproducible and auditable machine learning systems. The proposed guidelines aim to support the development of trustworthy, scalable, and reproducible ML pipelines across data-intensive domains.
DOI: https://doi.org/10.5281/zenodo.18677279
