Reducing Entropy Overestimation in Soft Actor Critic Using Dual Policy Network

In reinforcement learning (RL), an agent learns an environment through hit and trail. This behavior allows the agent to learn in complex and difficult environments. In RL, the agent normally learns the given environment by exploring or exploiting. Most of the algorithms suffer from under exploration in the latter stage of the episodes. Recently, an off-policy algorithm called soft actor critic (SAC) is proposed that overcomes this problem by maximizing entropy as it learns the environment. In it, the agent tries to maximize entropy along with the expected discounted rewards. In SAC, the agent tries to be as random as possible while moving towards the maximum reward. This randomness allows the agent to explore the environment and stops it from getting stuck into local optima. We believe that maximizing the entropy causes the overestimation of entropy term which results in slow policy learning. This is because of the drastic change in action distribution whenever agent revisits the similar states. To overcome this problem, we propose a dual policy optimization framework, in which two independent policies are trained. Both the policies try to maximize entropy by choosing actions against the minimum entropy to reduce the overestimation. The use of two policies result in better and faster convergence. We demonstrate our approach on different well known continuous control simulated environments. Results show that our proposed technique achieves better results against state of the art SAC algorithm and learns better policies.

Thermodynamics Studies for 6-(4-chlorophenyl)-1,2,3,4-tetrahydro-4-oxo-2-thioxopyrimidine-5-carbonitrile in 60% Dimethyl Sulphoxide at Different Temperatures

Viscometric studies for ternary mixture of 6-(4-chlorophenyl)-1,2,3,4-tetrahydro-4-oxo-2-thioxopyrimidine-5-carbonitrile in 60% dimethyl sulphoxide was carried out at different temperatures ranging between 298 and 313 K over different concentration. From viscometric data ΔG, ΔS and ΔH were evaluated. The result shows that the magnitude of ΔG and ΔH are negative while ΔS is positive showing spontaneity of reaction according to thermodynamics pointing structure breaking takes place between solute and solvent. As ΔH is negative and ΔS positive, the reaction is spontaneous at low temperatures (decreasing the magnitude of the entropy term). The thermodynamic study reveals that as concentration increases spontaneity of reaction decreases.

Path Planning for Multi-Arm Manipulators Using Deep Reinforcement Learning: Soft Actor–Critic with Hindsight Experience Replay

Since path planning for multi-arm manipulators is a complicated high-dimensional problem, effective and fast path generation is not easy for the arbitrarily given start and goal locations of the end effector. Especially, when it comes to deep reinforcement learning-based path planning, high-dimensionality makes it difficult for existing reinforcement learning-based methods to have efficient exploration which is crucial for successful training. The recently proposed soft actor–critic (SAC) is well known to have good exploration ability due to the use of the entropy term in the objective function. Motivated by this, in this paper, a SAC-based path planning algorithm is proposed. The hindsight experience replay (HER) is also employed for sample efficiency and configuration space augmentation is used in order to deal with complicated configuration space of the multi-arms. To show the effectiveness of the proposed algorithm, both simulation and experiment results are given. By comparing with existing results, it is demonstrated that the proposed method outperforms the existing results.

Shannon Entropy of Epithelium and Five-Fold Morphology: A Fundamental Model to Explain Geometrical Organizations as a Source of Information in Biological Systems

Based on a measuring system to determine levels of spatial organization in 2D polygons (homogeneous or heterogeneous partition of defined areas) lying on principles of regularity, we propose the entropy term linked to the concept of “information”, from the “information theory field”, in order to obtain an information measurement regarding a quantity of or amount of information in the architecture of complex 2D biological organizations. The term “quantity” does not refer to the amount of data (size), but to the probability of a geometrical basic pattern within a set of possible statistical configurations regarding levels of homogeneity and heterogeneity. It is this notion of information that is important in information theory, and measures of information in units of bits, what we propose to use for measuring quantities of organization in the architecture of complex geometrical systems. Two complex systems are tested, biological and non biological in order to obtain experimental results, which are verified with the evaluation criteria “entropy”. Experimental results show that the lowest levels of information and entropy, in addition with low rates of heterogeneity and high rates of homogeneity are particular features of geometrical organizations in biological systems.

Unexpected Resistance to Base-Catalyzed Hydrolysis of Nitrogen Pyramidal Amides Based on the 7-Azabicyclic[2.2.1]heptane Scaffold

Non-planar amides are usually transitional structures, that are involved in amide bond rotation and inversion of the nitrogen atom, but some ground-minimum non-planar amides have been reported. Non-planar amides are generally sensitive to water or other nucleophiles, so that the amide bond is readily cleaved. In this article, we examine the reactivity profile of the base-catalyzed hydrolysis of 7-azabicyclo[2.2.1]heptane amides, which show pyramidalization of the amide nitrogen atom, and we compare the kinetics of the base-catalyzed hydrolysis of the benzamides of 7-azabicyclo[2.2.1]heptane and related monocyclic compounds. Unexpectedly, non-planar amides based on the 7-azabicyclo[2.2.1]heptane scaffold were found to be resistant to base-catalyzed hydrolysis. The calculated Gibbs free energies were consistent with this experimental finding. The contribution of thermal corrections (entropy term, –TΔS‡) was large; the entropy term (ΔS‡) took a large negative value, indicating significant order in the transition structure, which includes solvating water molecules.

Two-component molecular crystals: relationship between the entropy term and the molecular volume of co-crystal formation

There are very few articles that investigate the thermodynamic formation of two-component molecular crystals.

Entropy Learning in Neural Network

In this paper, entropy term is used in the learning phase of a neural network. As learning progresses, more hidden nodes get into saturation. The early creation of such hidden nodes may impair generalisation. Hence entropy approach is proposed to dampen the early creation of such nodes. The entropy learning also helps to increase the importance of relevant nodes while dampening the less important nodes. At the end of learning, the less important nodes can then be eliminated to reduce the memory requirements of the neural network.

Modification of Bikerman model with specific ion sizes

Classical Poisson-Boltzman and Poisson-Nernst-Planck models can only work when ion concentrations are very dilute, which often mismatches experiments. Researchers have been working on the modification to include finite-size effect of ions, which is non-negelible when ion concentrations are not dilute. One of modified models with steric effect is Bikerman model, which is rather popular nowadays. It is based on the consideration of ion size by putting additional entropy term for solvent in free energy. However, ion size is non-specific in original Bikerman model, which did not consider specific ion sizes. Many researchers have worked on the extension of Bikerman model to have specific ion sizes. A direct extension of original Bikerman model by simply replacing the non-specific ion size to specific ones seems natural and has been acceptable to many researchers in this field.Herewe prove this straight forward extension, in some limiting situations, fails to uphold the basic requirement that ion occupation sites must be identical. This requirement is necessary when computing entropy via particle distribution on occupation sites.We derived a new modified Bikerman model for using specific ion sizes by fixing this problem, and obtained its modified Poisson-Boltzmann and Poisson-Nernst-Planck equations.

Temperature and Stress Dependence of Mobility of Screw Dislocation in BCC Iron

The Peierls stress and barrier of a screw dislocation in body-centered cubic iron at finite temperature is investigated by using the free energy gradient method. The Peierls barrier is shown to decrease from 12 to 5 meV per unit length of the Burgers vector with increasing temperature from 0 to 400 K. The entropy term of the Peierls barrier is estimated to be 0.2kB. The Peierls stress also decreases from 900 to 400 MPa with increasing temperature from 0 to 300 K. The change in the Peierls stress due to the entropic effect is larger than that of the Peierls barrier because of thermal softening.