Weather Data Analysis Using Hadoop: Applications and Challenges

Joint Conference on Green Engineering Technology & Applied Computing 2019

工业技术（总论）;计算机科学

Adam Ibrahim Fakherldin, Mohammed^3 ; Adam, Khalid^2 ; Akma Abu Bakar, Noor^1 ; Abdul Majid, Mazlina^1

Faculty of Computer System and Software Engineering, Universiti Malaysia Pahang, Malaysia^1

Faculty of Electric and Electronic Engineering, Universiti Malaysia Pahang, Malaysia^2

Faculty of Computer Science and Information Systems, Jazan University, Malaysia^3

关键词: Distributed environments;    Distributed processing technologies;    Fault-tolerant;    Geographical locations;    Historical data;    Historical weather datum;    Weather parameters;    Weather prediction;   

Others  :  https://iopscience.iop.org/article/10.1088/1757-899X/551/1/012044/pdf
DOI  :  10.1088/1757-899X/551/1/012044

来源: IOP

【 摘 要 】

Weather data is very crucial in every aspect of human daily life. It plays an important role in many sectors such as agriculture, tourism, government planning, industry and so on. Weather has a variety of parameters like temperature, pressure, humidity and wind speed. The meteorological department deployed sensors for each weather parameter at different geographical locations to collect data. This data is stored mostly in the unstructured format. Thus, a big amount of data has been collected and archived. Therefore, storage and processing of this big data for accurate weather prediction is a huge challenge. Hadoop an apache product it used to support big data sets in a distributed environment. Hadoop has greatest advantages over scalable and fault-tolerant distributed processing technologies. This paper explains a system that uses the historical weather data of a region and apply the MapReduce and Hadoop techniques to analysis these historical data.