Probability Mass Exclusions and the Directed Components of Mutual Information

Entropy

JosephT. Lizier¹  Conor Finn² 

[1] IT, The University of Sydney, NSW 2006, Australia;;Complex Systems Research Group and Centre for Complex Systems, Faculty of Engineering &

关键词: entropy;    mutual information;    pointwise;    directed;    information decomposition;   

DOI  :  10.3390/e20110826

来源: DOAJ

【 摘 要 】

Information is often described as a reduction of uncertainty associated with a restriction of possible choices. Despite appearing in Hartley’s foundational work on information theory, there is a surprising lack of a formal treatment of this interpretation in terms of exclusions. This paper addresses the gap by providing an explicit characterisation of information in terms of probability mass exclusions. It then demonstrates that different exclusions can yield the same amount of information and discusses the insight this provides about how information is shared amongst random variables—lack of progress in this area is a key barrier preventing us from understanding how information is distributed in complex systems. The paper closes by deriving a decomposition of the mutual information which can distinguish between differing exclusions; this provides surprising insight into the nature of directed information.

【 授权许可】

Unknown