Reduced-set models for improving the training and execution speed of kernel methods

Machine learning;Kernel methods;Reproducing kernel Hilbert spaces;Adaptive control;Manifold learning

Kingravi, Hassan ; Vela, Patricio A. Electrical and Computer Engineering Verriest, Erik Davenport, Mark A. Madisetti, Vijay K. Chowdhary, Girish ; Vela, Patricio A.

University:Georgia Institute of Technology

Department:Electrical and Computer Engineering

关键词: Machine learning;    Kernel methods;    Reproducing kernel Hilbert spaces;    Adaptive control;    Manifold learning;   

Others  :  https://smartech.gatech.edu/bitstream/1853/51799/1/KINGRAVI-DISSERTATION-2014.pdf

美国|英语

来源: SMARTech Repository

【 摘 要 】

This thesis aims to contribute to the area of kernel methods, which are a class of machine learning methods known for their wide applicability and state-of-the-art performance, but which suffer from high training and evaluation complexity. The work in this thesis utilizes the notion of reduced-set models to alleviate thetraining and testing complexities of these methods in a unified manner. In the first part of the thesis, we use recent results in kernel smoothing and integral-operator learning to design a generic strategy to speed up various kernel methods. In Chapter 3, we present a method to speed up kernel PCA (KPCA), which is one of the fundamental kernel methods for manifold learning, by using reduced-set density estimates (RSDE) of the data. The proposed method induces an integral operator that is an approximation of the ideal integral operator associated to KPCA. It is shown that the error between the ideal and approximate integral operators is related to the error between the ideal and approximate kernel density estimates of the data. In Chapter 4, we derive similar approximation algorithms for Gaussian process regression, diffusion maps, and kernel embeddings of conditional distributions. In the second part of the thesis, we use reduced-set models for kernel methods to tackle online learning in model-reference adaptive control (MRAC). In Chapter 5, we relate the properties of the feature spaces induced by Mercer kernels to make a connection between persistency-of-excitation and the budgeted placement of kernels to minimize tracking and modeling error. In Chapter 6, we use a Gaussian process (GP) formulation of the modeling error to accommodate a larger class of errors, and design a reduced-set algorithm to learn a GP model of the modeling error. Proofs of stability for all the algorithms are presented, and simulation results on a challenging control problem validate the methods.