Citation

Nisha, Salma Jahan (2025) Sign language chatbot using CNN and LLM prompting for application in E-Commerce. Masters thesis, Multimedia University.

Full text not available from this repository.

Abstract

This research proposes a real-time, hybrid Sign Language (SL) conversational framework to enhance e-commerce accessibility for Deaf and Hard-of-Hearing (DHH) users. While conventional chatbots rely on text-based input, this creates a significant linguistic barrier for native signers, as written English often functions as a second language with a distinct syntax compared to SL. This research addresses this modality gap by providing a native-syntax interface tailored for e-commerce transactions. The main idea was to create a low-latency, easy-to-use framework to understand users’ signs and provide responses. The methodology utilizes SSD MobileNetV2 to ensure low-latency detection on consumer-grade hardware, coupled with a custom temporal filtering layer specifically designed to stabilize noisy real-time streams which is a known limitation in current State-of-the-Art (SOTA) systems. A Large Language Model (LLM) semantic interpretation layer is then employed to map these stabilized gloss sequences into fluent natural language sentences. The system addresses gaps in current SOTA systems regarding compounding noise in real-time streams, focusing on 35 e-commerce-specific phrases in American Sign Language (ASL). Among the contributions of the proposed idea are a feasible framework for a hybrid model, solutions for recognized issues, and a goal for promoting digital inclusion. The proposed framework was empirically benchmarked against a no-filtering baseline and SOTA architectures, including YOLOv8n and Faster R-CNN, to validate its effectiveness in noise reduction and recognition stability. Empirical evaluation shows the hybrid framework achieved an overall system-level accuracy of 92.3% and a Mean Average Precision (mAP) of 91.5%. This research contributes a validated architectural blueprint for bridging the gap between Sign Language and conversational AI, demonstrating that hybrid LLM-CV systems can significantly improve digital inclusion for the DHH community without requiring users to conform to text-based communication norms.

Item Type:	Thesis (Masters)
Additional Information:	Call No.: QA76.9.H85 S25 2025
Uncontrolled Keywords:	Human-computer interaction
Subjects:	Q Science > QA Mathematics > QA71-90 Instruments and machines > QA75.5-76.95 Electronic computers. Computer science
Divisions:	Faculty of Computing and Informatics (FCI)
Depositing User:	Ms Nurul Iqtiani Ahmad
Date Deposited:	16 Jul 2026 04:13
Last Modified:	16 Jul 2026 04:13
URII:	http://shdl.mmu.edu.my/id/eprint/16374

Downloads

Edit (login required)