Citation
Rahman, Md. Armanur and Hossen, Md. Jakir and Sultana, Aziza and Al Mamun, Abdullah Sarwar and Ab Aziz, Nor Azlina (2021) A smart method for spark using neural network for big data. International Journal of Electrical and Computer Engineering (IJECE), 11 (3). ISSN 2088-8708
Text
A smart method for spark using neural network for big data.pdf Restricted to Repository staff only Download (817kB) |
Abstract
Apache spark, famously known for big data handling ability, is a distributed open-source framework that utilizes the idea of distributed memory to process big data. As the performance of the spark is mostly being affected by the spark predominant configuration parameters, it is challenging to achieve the optimal result from spark. The current practice of tuning the parameters is ineffective, as it is performed manually. Manual tuning is challenging for large space of parameters and complex interactions with and among the parameters. This paper proposes a more effective, self-tuning approach subject to a neural network called Smart method for spark using neural network for big data (SSNNB) to avoid the disadvantages of manual tuning of the parameters. The paper has selected five predominant parameters with five different sizes of data to test the approach. The proposed approach has increased the speed of around 30% compared with the default parameter configuration.
Item Type: | Article |
---|---|
Uncontrolled Keywords: | Big data |
Subjects: | Q Science > QA Mathematics > QA71-90 Instruments and machines > QA75.5-76.95 Electronic computers. Computer science |
Divisions: | Faculty of Engineering and Technology (FET) |
Depositing User: | Ms Nurul Iqtiani Ahmad |
Date Deposited: | 21 Apr 2021 15:12 |
Last Modified: | 21 Apr 2021 15:12 |
URII: | http://shdl.mmu.edu.my/id/eprint/8607 |
Downloads
Downloads per month over past year
Edit (login required) |