Attention Mechanisms and Their Applications to Complex Systems

Entropy

JoséM. Amigó¹  Adrián Hernández¹ 

[1] Centro de Investigación Operativa, Universidad Miguel Hernández, Av. de la Universidad s/n, 03202 Elche, Spain;

关键词: attention;    deep learning;    complex and dynamical systems;    self-attention;    neural networks;    sequential reasoning;   

DOI  :  10.3390/e23030283

来源: DOAJ

【 摘 要 】

Deep learning models and graphics processing units have completely transformed the field of machine learning. Recurrent neural networks and long short-term memories have been successfully used to model and predict complex systems. However, these classic models do not perform sequential reasoning, a process that guides a task based on perception and memory. In recent years, attention mechanisms have emerged as a promising solution to these problems. In this review, we describe the key aspects of attention mechanisms and some relevant attention techniques and point out why they are a remarkable advance in machine learning. Then, we illustrate some important applications of these techniques in the modeling of complex systems.

【 授权许可】

Unknown