In this paper, we propose new information-theoretic methods to stabilize feature detection. We have introduced information-theoretic methods to realize competitive learning. It turned out that mutual information maximization corresponds to a process of competition among neurons. This means that mutual information can be effective in describing competitive processes. Thus, by using this mutual information, we have introduced information loss to interpret internal representations. By relaxing competitive units by some components such as units and connection weights, a neural network’s information is decreased. If the information loss is sufficiently large, the components play important roles. However, with the information loss, there have been some problems, such as the instability of final representations. This means that final outputs are significantly dependent upon chosen parameters. To stabilize final representations, we introduce two computational methods, that is, <em>relative relaxation</em> and <em>weighted information loss</em>. The relative relaxation is introduced because mutual information is dependent upon the Gaussian width. Thus, we can relax competitive units or softly delete some components, relative only to a predetermined base state. In addition, we introduce weighted information loss to take into account information on related components. We applied the methods to the well-known Iris problem and a problem regarding the extinction of animals and plants. In the Iris problem, experimental results confirmed that final representations were significantly stable if we appropriately chose the parameter for the base state. On the other hand, in the extinction problem, weighted information losses showed better performance, where final outputs were significantly more stable than those by the other methods.
Constrained versus Unconstrained Rational Inattention
