Minimising the Kullback–Leibler Divergence for Model Selection in Distributed Nonlinear Systems

Entropy

Oliver M. Cliff¹  Robert Fitch¹  Mikhail Prokopenko² 

[1] Australian Centre for Field Robotics, The University of Sydney, Sydney NSW 2006, Australia;Complex Systems Research Group, The University of Sydney, Sydney NSW 2006, Australia;

关键词: Kullback–Leibler divergence;    model selection;    information theory;    transfer entropy;    stochastic interaction;    nonlinear systems;    complex networks;    state space reconstruction;   

DOI  :  10.3390/e20020051

来源: DOAJ

【 摘 要 】

The Kullback–Leibler (KL) divergence is a fundamental measure of information geometry that is used in a variety of contexts in artificial intelligence. We show that, when system dynamics are given by distributed nonlinear systems, this measure can be decomposed as a function of two information-theoretic measures, transfer entropy and stochastic interaction. More specifically, these measures are applicable when selecting a candidate model for a distributed system, where individual subsystems are coupled via latent variables and observed through a filter. We represent this model as a directed acyclic graph (DAG) that characterises the unidirectional coupling between subsystems. Standard approaches to structure learning are not applicable in this framework due to the hidden variables; however, we can exploit the properties of certain dynamical systems to formulate exact methods based on differential topology. We approach the problem by using reconstruction theorems to derive an analytical expression for the KL divergence of a candidate DAG from the observed dataset. Using this result, we present a scoring function based on transfer entropy to be used as a subroutine in a structure learning algorithm. We then demonstrate its use in recovering the structure of coupled Lorenz and Rössler systems.

【 授权许可】

Unknown