Authors: Ajay Singh, Aditya Marathe, Aniket Gaikwad, Om ahire, Jay modiya, Utkarsh musale
Abstract: Artificial Intelligence (AI) continues to play a transformative role in healthcare, particularly through advancements in large language models (LLMs) and computer vision (CV). These technologies are now being increasingly applied in predictive healthcare systems to improve diagnosis, reduce human error, and enhance patient engagement. However, general-purpose pre-trained models often underperform in specialized medical contexts where accuracy, domain-specific knowledge, and multimodal understanding are essential. This research proposes a hybrid AI framework that combines natural language processing (NLP) and computer vision to support predictive and interactive healthcare use cases. In the NLP component of the system, we perform a comparative evaluation of six leading open-source LLMs—Mistral, FLAN-T5, GPT-Neo, DialoGPT, LLaMA, and Ollama—analyzing their adaptability to domain-specific tasks such as symptom triage, patient education, and medical question answering. These models were fine-tuned using full parameter updates and reinforcement learning from human feedback (RLHF), which allowed the models to better align their outputs with the nuanced communication styles and ethical expectations in clinical settings. In parallel, the CV module addresses a critical real-world challenge: automated prescription handwriting recognition, which is essential for minimizing misinterpretation of medication names and dosages. To tackle the variability and complexity of handwritten medical prescriptions, we utilize convolutional neural networks—specifically VGG16 and EfficientNet—for image-based classification and text recognition. A custom dataset of handwritten prescription images was created and annotated using domain knowledge, and the models were trained to map image inputs to structured medicine names. Our experiments reveal that EfficientNet, with its compound scaling and optimized architecture, outperforms VGG16 in both accuracy and training efficiency, particularly under noisy or low-resolution input conditions. By integrating these two components, we build a multimodal chatbot capable of receiving an image of a handwritten prescription, recognizing the medication using a CNN model, and generating an informative or advisory response using an LLM fine-tuned for medical NLP. This enables seamless user interaction, allowing patients or practitioners to interact with the system using both text and image inputs. Such a system has practical applications in telemedicine, hospital kiosks, pharmacy automation, and rural health outreach, where both human expertise and infrastructure may be limited. Our results demonstrate the effectiveness of combining LLM fine-tuning and CNN-based vision models for predictive healthcare. While larger LLMs like LLaMA and FLAN-T5 achieve higher accuracy in clinical language tasks, lighter models like DialoGPT and Mistral offer faster, more cost-effective deployment options. On the CV side, EfficientNet offers superior generalization with fewer parameters compared to legacy architectures. This research provides a comprehensive performance analysis and design framework for AI systems in healthcare, offering actionable insights into how different model configurations, training strategies, and hardware choices affect outcome quality and deployment feasibility.
DOI: http://doi.org/10.61137/ijsret.vol.11.issue3.119
