Citation

Deshpande, Sudhindra B. and Desai, Priyanka and Goh, Kah Ong Michael and Ajawan, Pratijnya and Mathad, Kadayya and Deshpande, Uttam U. (2026) A comprehensive comparative analysis of transformer based large language models across natural language processing benchmarks. Discover Artificial Intelligence, 6 (1). ISSN 2731-0809

Text s44163-026-01700-7.pdf - Published Version Restricted to Repository staff only Download (6MB)

Abstract

This work presents a comparative experimental study of selected transformer-based language model architectures: GPT (Generative Pre-trained Transformer), BERT (Bidirectional Encoder Representations from Transformers), and BLOOM (BigScience Large Open-science Open-access Multilingual Language Model). The evaluation spans execution time, accuracy, computational efficiency, cost-effectiveness, and task-specific performance across six key Natural Language Processing (NLP) benchmarks: Stanford Question Answering Dataset (SQuAD), General Language Understanding Evaluation (GLUE), Super General Language Understanding Evaluation (SuperGLUE), Cross-lingual Natural Language Inference (XNLI), AG News Text Classification Dataset (AG News), and the Massive Multitask Language Understanding benchmark (MMLU). BERT-Large demonstrates strong performance in classification-oriented tasks, achieving 94.1% accuracy on AG News and an 89.2% F1 score on SQuAD. GPT-3 excels in high-quality text generation but carries the highest training cost ($4.6M) and inference cost ($0.060 per 1K tokens). BLOOM-7B provides relatively stronger multilingual capabilities with a 91.2% accuracy on XNLI while maintaining moderate computational requirements. GPT-2 offers the most cost-efficient option for resource-constrained applications with a training cost of only $40K. Model selection should therefore be guided by the intended task requirements, computational budget, and deployment constraints. This analysis provides experimentally grounded insights under a controlled yet task-aligned evaluation setting for researchers, practitioners, and organizations aiming to determine the most suitable LLM architecture for their specific use cases,while acknowledging practical differences in evaluation methodology across model types rather than a fully uniform benchmarking pipeline.

Item Type:	Article
Uncontrolled Keywords:	Large language models, Transformer architectures, GPT, BERT, BLOOM, NLP benchmarks, Multilingual models, Computational efficiency, Model evaluation, Natural language understanding
Subjects:	P Language and Literature > P Philology. Linguistics
Divisions:	Faculty of Information Science and Technology (FIST)
Depositing User:	Ms Suzilawati Abu Samah
Date Deposited:	28 Oct 2026 08:05
Last Modified:	28 Oct 2026 08:05
URII:	http://shdl.mmu.edu.my/id/eprint/16955

Downloads

Edit (login required)