Natural Way to Overcome Catastrophic Forgetting in Neural Networks

Современные информационные технологии и IT-образование

Alexey Kutalev¹ 

[1] JSC InfoWatch;

关键词: neural network;    catastrophic forgetting;    elastic weight consolidation;    back propagation;    total absolute signal;   

DOI  :  10.25559/SITITO.16.202002.331-337

来源: DOAJ

【 摘 要 】

The problem of catastrophic forgetting manifested itself in models of neural networks based on the connectionist approach, which have been actively studied since the second half of the 20th century. Numerous attempts have been made and various ways to solve this problem have been proposed, but until very recently substantial successes have not been achieved. In 2016, a significant breakthrough occurred – a group of scientists from DeepMind proposed the method of elastic weight consolidation (EWC), which allows us to successfully overcome the problem of catastrophic forgetting. Unfortunately, although we were aware about the cases of using this method in real tasks, it has not yet obtained widespread distribution. In this paper, we want to propose alternative approaches for overcoming catastrophic forgetting, based on the total absolute signal passed through the connection. These approaches demonstrate similar efficiency as EWC and, at the same time, have less computational complexity. These approaches have a simpler implementation and seem to us to be essentially closer to the processes occurring in the brain of animals to preserve previously learned skills during subsequent training. We hope that the ease of implementation of these methods will serve their wider application.

【 授权许可】

Unknown