Transformers: A Review and Use in Text Analytics, Topic Modelling and Summarization

Abstract-Automatic text summarization and zero-shot classification are crucial tasks in natural language processing (NLP), aiding in information retrieval, content compression, and text classification. Recent advances in deep learning and transformers have significantly improved the accuracy and efficiency of these tasks. This study evaluates multiple state-of-the-art transformer-based models for text summarization, including Google’s T5, PEGASUS, Facebook’s BART, and Longformer Encoder-Decoder (LED). We assess their performance using the ROUGE and BERTScore metrics to determine their effectiveness in generating concise and contextually accurate summaries. The T5 model, pre-trained on C4, achieves state-of-the-art results on many NLP benchmarks while being flexible enough to be fine-tuned to a variety of important downstream tasks [1]. Also, zero-shot classification with the facebook/bart-large-mnli model is considered in this work, with no training labels beforehand for classification of text into predefined categories. Classification accuracy for a variety of domains, including Politics, Sport, Technology, Entertainment, and Business, is considered in analysis. To classify even more precisely, a corpus with labels is fine-tuned with the BART model and improvement in prediction accuracy and loss over a range of training runs measured. Zero-shot classification, useful for general categories, is seen to have improvement room in specific domains for classification. Classification with fine-tuning of the BART model reduces evaluation loss but comes with hyperparameter search and a larger corpus for even heightened accuracy. Traditionally, zero-shot learning (ZSL) most often referred to a fairly specific type of task: learn a classifier on one set of labels and then evaluate on a different set of labels that the classifier has never seen before [2].