From big data to smart data: a sample gradient descent approach for machine learning

Journal of Big Data

Research

Aadil Gani Ganie¹  Samad Dadvandipour¹ 

[1] University of Miskolc, 3515, Miskolc, Hungary;

关键词: Big data;    Gradient decent;    Machine learning;    PCA;    Loss function;   

DOI  :  10.1186/s40537-023-00839-9

 received in 2023-06-15, accepted in 2023-10-03,  发布年份 2023

来源: Springer

【 摘 要 】

This research paper presents an innovative approach to gradient descent known as ‘‘Sample Gradient Descent’’. This method is a modification of the conventional batch gradient descent algorithm, which is often associated with space and time complexity issues. The proposed approach involves the selection of a representative sample of data, which is subsequently subjected to batch gradient descent. The selection of this sample is a crucial task, as it must accurately represent the entire dataset. To achieve this, the study employs the use of Principle Component Analysis (PCA), which is applied to the training data, with a condition that only those rows and columns of data that explain 90% of the overall variance are retained. This approach results in a convex loss function, where a global minimum can be readily attained. Our results indicate that the proposed method offers faster convergence rates, with reduced computation times, when compared to the conventional batch gradient descent algorithm. These findings demonstrate the potential utility of the ‘‘Sample Gradient Descent’’ technique in various domains, ranging from machine learning to optimization problems. In our experiments, both approaches were run for 30 epochs, with each epoch taking approximately 3.41 s. Notably, our ‘‘Sample Gradient Descent’’ approach exhibited remarkable performance, converging in just 8 epochs, while the conventional batch gradient descent algorithm required 20 epochs to achieve convergence. This substantial difference in convergence rates, along with reduced computation times, highlights the superior efficiency of our proposed method. These findings underscore the potential utility of the ‘‘Sample Gradient Descent’’ technique across diverse domains, ranging from machine learning to optimization problems. The significant improvements in convergence rates and computation times make our algorithm particularly appealing to practitioners and researchers seeking enhanced efficiency in gradient descent optimization.

【 授权许可】

CC BY   
© Springer Nature Switzerland AG 2023

【 图 表 】

12864_2015_2198_Article_IEq1.gif

12936_2017_1866_Article_IEq9.gif

Fig. 2

Fig. 5

Fig. 3

12951_2015_155_Article_IEq70.gif

Fig. 5