Enhancing BERT for Question Answering with Token Transformation Networks
Authors:-Yoga Harshitha Duddukuri, Dr. Yugandhar Garapati
Abstract-This paper proposes an enhanced architecture to improve the accuracy of BERT models fine-tuned on the Stanford Question Answering Dataset (SQuAD). The presented approach introduces a Token Transformation Model designed to refine embedding, making them more effective for question answering tasks. Initially, the question and context inputs are tokenized using a BERT-large tokenizer. These tokens are then processed through the Token Transformation Model, which enhances the quality and relevance of the embedding. The refined embedding are subsequently utilized by a TinyBERT model that has been fine-tuned on SQuAD with knowledge distillation (KD) techniques. The proposed method aims to leverage the strengths of large-scale tokenization and advanced embedding transformations to achieve higher accuracy in question answering scenarios, offering a more precise and efficient solution. Experimental results demonstrate the effectiveness of this architecture in improving the performance of BERT models on SQuAD.
