Function Approximation Based on a Network with Kernel Functions of Bounds and Locality : an Approach of Non-Parametric Estimation

ETRI Journal

关键词: Function Approximation Based on a Network with Kernel Functions of Bounds and Locality : an Approach of Non-Parametric Estimation;   

Others  :  1183899
DOI  :  10.4218/etrij.93.0193.0004

【 摘 要 】

This paper presents function approximation based on nonparametric estimation. As an estimation model of function approximation, a three layered network composed of input, hidden and output layers is considered. The input and output layers have linear activation units while the hidden layer has nonlinear activation units or kernel functions which have the characteristics of bounds and locality. Using this type of network, a many-to-one function is synthesized over the domain of the input space by a number of kernel functions. In this network, we have to estimate the necessary number of kernel functions as well as the parameters associated with kernel functions. For this purpose, a new method of parameter estimation in which linear learning rule is applied between hidden and output layers while nonlinear (piecewise-linear) learning rule is applied between input and hidden layers, is considered. The linear learning rule updates the output weights between hidden and output layers based on the Linear Minimization of Mean Square Error (LMMSE) sense in the space of kernel functions while the nonlinear learning rule updates the parameters of kernel functions based on the gradient of the actual output of network with respect to the parameters (especially, the shape) of kernel functions. This approach of parameter adaptation provides near optimal values of the parameters associated with kernel functions in the sense of minimizing mean square error. As a result, the suggested nonparametric estimation provides an efficient way of function approximation from the view point of the number of kernel functions as well as learning speed.

   

【 参考文献 】

• [1]G. A. Carpenter and S. Grossberg, "Art2: Stable self-organization of pattern recognition codes for analog input patterns," Applied Optics, 26:4919-4930, 1987.
• [2]D. L. Reilly, L. N. Cooper, and C. Elbaum, "A neural model for category learning," Biological Cybernetics, 45:35-41, 1982.
• [3]R. Hecht-Nielsen, "Kolmogorov mapping neural network existence theorem," IEEE International Conference on Neural Networks, 3:11-43, 1987.
• [4]M. A. Aizerman, E. M. Braverman, and L. I. Rozonoer, "Theoretical foundations of the potential function method in pattern recognition learning," Avtomatika i Telemekhanika, 25:917-936, 1964.
• [5]S. Lee and R. M. Kil. "Multilayer feed forward potential function network," IEEE International Conference on Neural Networks, 1:161-171, 1988.
• [6]S. Lee and R. M. Kil. "A gaussian potential function network with hierarchically self organizing learning," Neural Networks, 4(2):207-224, 1991.
• [7]M. Niranjan and F. Fallside, Neural networks and radial basis functions in classifying static speech patterns. Technical Report CUED/FINFENG/TR22, Cambridge University, 1988.
• [8]J. Moody and C. J. Darken, "Fast learning in networks of locally-tuned processing units," Neural Computation, 1:281-294, 1989.
• [9]D. E. Rumelhart, G. E. Hinton, and R, J. Williams, Parallel Distributed Processing, volume 1, pages 318-362. MIT Press/Bradford Books, 1986.
• [10]M. Minsky and S. Papert, Perceptions. The MIT Press, 1969.
• [11]E. Parzen, "On the estimation of a probability density function and mode," Annals of Mathematical Statistics, 33:1065-1076, 1962.
• [12]K. Funahashi, "On the approximate realization of continuous mappings by neural networks," Pleural Networks, 2(3): 183-192, 1989.
• [13]K. Hornik, M. Stinchcombe, and H. White, "Multilayer feed forward networks are universal approximators," Neural Networks, 2:359-366, 1989.
• [14]C. E. Shannon, "Communication in the presence of noise," Proceedings of IRE, 37:10, 1945.
• [15]S. Lee and R. M. Ku, "Nonlinear system control based on Gaussian potential function network," IEEE International Symposium on Intelligent Control, pp. 423-429, 1991.
• [16]A. S. Lapedes and R. Farber, Nonlinear signal processing using neural networks: Prediction and system modeling, Technical Report LAUR-87-2662, Los Alamos National Laboratory, 1987.
• [17]A. S. Lapedes and R. Farber, "How neural nets work," Neural Information Processing Systems, pages 442-456, 1988.