Citation

Haw, Su Cheng and Athirah, Raja Roza and Soon, Lay Ki (2019) Exploring Edit Distance for Normalising Out-of-Vocabulary Malay Words on Social Media. MATEC Web of Conferences, 255. ISSN 2261-236X

Text 134.pdf - Published Version Restricted to Repository staff only Download (2MB)

Abstract

Users interact using short-formed words and abbreviations and this results in a message full of noisy words that are not recognized by the system’s knowledge. The aim of this research is to overcome the limitations that still bar the progression of normalizing Malay noisy words from social media platforms. The testing data gathered is 25,000; 15,000 Tweets from Twitter and 10,000 comments from Facebook respectively. Pre-processing steps were carried out to clean the entire dataset which consists of unique 179,786 words. 36,587 out-of-vocabulary (OOV) Malay terms were then extracted and checked against an invocabulary (IV) Malay corpus using the Levenshtein edit distance formula and character manipulation rules. The resultant output is 3,964 unique IV Malay words. Based on the results, the usage of edit distance and rules can be further improved to elevate the normalisation of the ever changing colloquial terms of the Malay language

Item Type:	Article
Uncontrolled Keywords:	Social Media
Subjects:	H Social Sciences > HM Sociology > HM711-806 Groups and organizations
Divisions:	Faculty of Computing and Informatics (FCI)
Depositing User:	Ms Suzilawati Abu Samah
Date Deposited:	11 Jan 2022 03:47
Last Modified:	11 Jan 2022 03:47
URII:	http://shdl.mmu.edu.my/id/eprint/8962

Downloads

Edit (login required)