Authors: Dr. Daniel Foster, Dr. Olivia Bennett, Ethan Clarke, Dr. Hannah Mitchell, Andrew Richard
Abstract: The rapid growth of deep learning has enabled state-of-the-art performance across vision, speech, and natural language processing tasks, driving widespread adoption in both academic research and industrial applications. However, this progress has been accompanied by a steady increase in model depth, parameter count, and computational complexity, which poses significant challenges for deployment in resource-constrained environments such as mobile devices, embedded systems, and edge computing platforms with limited memory, power, and latency budgets. To address these constraints, this article presents a comprehensive review of model compression and knowledge distillation techniques developed between 2000 and 2021, synthesizing foundational methods including network pruning, low-precision quantization, and entropy-based coding, as well as teacher–student learning paradigms that transfer representational and decision-level knowledge from large, overparameterized models to compact alternatives. Using representative architectural and training diagrams, we illustrate how these approaches systematically reduce memory footprint and computational cost while preserving, and in some cases improving, predictive accuracy. Finally, we examine key empirical findings across vision, speech, and language domains, identify persistent limitations related to generalization, hardware efficiency, and evaluation methodology, and outline future research directions toward scalable, energy-efficient, and deployable AI systems.
DOI: https://doi.org/10.5281/zenodo.20052481
