Minimum Mutual Information and Non-Gaussianity through the Maximum Entropy Method: Estimation from Finite Samples

Entropy

Carlos A. L. Pires¹ 

[1] Instituto Dom Luiz (IDL), University of Lisbon (UL), Lisbon, P-1749-016, Portugal

关键词: mutual information;    non-Gaussianity;    maximum entropy distributions;    Entropy bias;    mutual information distribution;    morphism;   

DOI  :  10.3390/e15030721

来源: mdpi

【 摘 要 】

The Minimum Mutual Information (MinMI) Principle provides the least committed, maximum-joint-entropy (ME) inferential law that is compatible with prescribed marginal distributions and empirical cross constraints. Here, we estimate MI bounds (the MinMI values) generated by constraining sets T_cr comprehended by m_cr linear and/or nonlinear joint expectations, computed from samples of N iid outcomes. Marginals (and their entropy) are imposed by single morphisms of the original random variables. N-asymptotic formulas are given both for the distribution of cross expectation’s estimation errors, the MinMI estimation bias, its variance and distribution. A growing T_cr leads to an increasing MinMI, converging eventually to the total MI. Under N-sized samples, the MinMI increment relative to two encapsulated sets T_cr1 ⊂ T_cr2 (with numbers of constraints m_cr1 < m_cr2) is the test-differencewhose upper quantiles determine if constraints in T_cr2/T_cr1 explain significant extra MI. As an example, we have set marginals to being normally distributed (Gaussian) and have built a sequence of MI bounds, associated to successive non-linear correlations due to joint non-Gaussianity. Noting that in real-world situations available sample sizes can be rather low, the relationship between MinMI bias, probability density over-fitting and outliers is put in evidence for under-sampled data.

【 授权许可】

CC BY   
© 2013 by the authors; licensee MDPI, Basel, Switzerland.