scholarly journals Navigation of micro-swimmers in steady flow: the importance of symmetries

2021 ◽  
Vol 932 ◽  
Author(s):  
Jingran Qiu ◽  
Navid Mousavi ◽  
Kristian Gustavsson ◽  
Chunxiao Xu ◽  
Bernhard Mehlig ◽  
...  
Keyword(s):  
Reinforcement Learning ◽  
Optimal Strategies ◽  
Laboratory Frame ◽  
Nonlinear Flow ◽  
Survival Strategies ◽  
Environmental Cues ◽  
Global Information ◽  
Local Flow ◽  
Flow Vorticity ◽  
Micro Organisms

Marine micro-organisms must cope with complex flow patterns and even turbulence as they navigate the ocean. To survive they must avoid predation and find efficient energy sources. A major difficulty in analysing possible survival strategies is that the time series of environmental cues in nonlinear flow is complex and that it depends on the decisions taken by the organism. One way of determining and evaluating optimal strategies is reinforcement learning. In a proof-of-principle study, Colabrese et al. (Phys. Rev. Lett., vol. 118, 2017, 158004) used this method to find out how a micro-swimmer in a vortex flow can navigate towards the surface as quickly as possible, given a fixed swimming speed. The swimmer measured its instantaneous swimming direction and the local flow vorticity in the laboratory frame, and reacted to these cues by swimming either left, right, up or down. However, usually a motile micro-organism measures the local flow rather than global information, and it can only react in relation to the local flow because, in general, it cannot access global information (such as up or down in the laboratory frame). Here we analyse optimal strategies with local signals and actions that do not refer to the laboratory frame. We demonstrate that symmetry breaking is required to find such strategies. Using reinforcement learning, we analyse the emerging strategies for different sets of environmental cues that micro-organisms are known to measure.

Off-policy integral reinforcement learning algorithm in dealing with nonzero sum game for nonlinear distributed parameter systems

2020 ◽  
Vol 42 (15) ◽  
pp. 2919-2928
Author(s):  
He Ren ◽  
Jing Dai ◽  
Huaguang Zhang ◽  
Kun Zhang
Keyword(s):  
Differential Equation ◽  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Control Strategies ◽  
Iteration Process ◽  
Optimal Strategies ◽  
Distributed Parameter Systems ◽  
Adaptive Dynamic Programming ◽  
Distributed Parameter ◽  
Value Functions

Benefitting from the technology of integral reinforcement learning, the nonzero sum (NZS) game for distributed parameter systems is effectively solved in this paper when the information of system dynamics are unavailable. The Karhunen-Loève decomposition (KLD) is employed to convert the partial differential equation (PDE) systems into high-order ordinary differential equation (ODE) systems. Moreover, the off-policy IRL technology is introduced to design the optimal strategies for the NZS game. To confirm that the presented algorithm will converge to the optimal value functions, the traditional adaptive dynamic programming (ADP) method is first discussed. Then, the equivalence between the traditional ADP method and the presented off-policy method is proved. For implementing the presented off-policy IRL method, actor and critic neural networks are utilized to approach the value functions and control strategies in the iteration process, individually. Finally, a numerical simulation is shown to illustrate the effectiveness of the proposal off-policy algorithm.

scholarly journals From active stresses and forces to self-propulsion of droplets

2017 ◽  
Vol 821 ◽  
pp. 595-623 ◽  
Author(s):  
R. Kree ◽  
P. S. Burada ◽  
A. Zippelius
Keyword(s):  
Mass Velocity ◽  
Body Force ◽  
Internal Flow ◽  
Fixed Time ◽  
Laboratory Frame ◽  
Velocity Fields ◽  
Centre Of Mass ◽  
Hydrodynamic Models ◽  
Micro Organisms ◽  
Spherical Droplets

We study the self-propulsion of spherical droplets as simplified hydrodynamic models of swimming micro-organisms or artificial micro-swimmers. In contrast to approaches that start from active velocity fields produced by the system, we consider active interface tractions, body force densities and active stresses as the origin of autonomous swimming. For negligible Reynolds number and given activity, we compute the external and internal flow fields as well as the centre of mass velocity and angular velocity of the droplet at fixed time. To construct trajectories from single time snapshots, the evolution of active forces or stresses must be determined in the laboratory frame. Here, we consider the case of active matter, which is carried by a continuously distributed rigid but sparse (cyto)-skeleton that is immersed in the droplet interior. We calculate examples of trajectories of a droplet and its skeleton from force densities or stresses, which may be explicitly time-dependent in a frame fixed within the skeleton.

Development of a Virtual Coil Model for Blood Flow Simulation in Coil-Embolized Aneurysms

2013 ◽  
Author(s):  
Tomohiro Otani ◽  
Satoshi Ii ◽  
Toshiyuki Fujinaka ◽  
Masayuki Hirata ◽  
Junko Kuroda ◽  
...  
Keyword(s):  
Blood Flow ◽  
Porous Media ◽  
Packing Density ◽  
Coil Embolization ◽  
Flow Analysis ◽  
Local Flow ◽  
Flow Vorticity ◽  
Porous Media Model ◽  
Significant Difference ◽  
Blood Flow Analysis

Hemodynamics is considered to be one of the indices to evaluate the effects of the treatment by coil embolization for cerebral aneurysms. For the sake of detailed analysis of hemodynamics in coil-embolized aneurysms, we develop a virtual coil model based on the mechanical theory that the coil deforms toward minimizing the elastic energy, and represent a realistic configuration of the embolized coils in the aneurysm by the insertion simulation. Then, the blood flow analysis is done by solving the N.S. and continuity equations numerically with the finite volume method using polyhedral mesh. The coil insertion simulation demonstrated that almost uniform distribution of the coil in the aneurysm was achieved at over 10% packing density of the coil. The blood flow analysis using the virtual coil model showed that the flow momentum inside the aneurysm was reduced to less than 10% by coil embolization with a packing density over 20%. In comparison to the simulation results using a porous media model for the embolized coil, there was no significant difference in the reduction ratio of the flow momentum in the aneurysm by coil embolization. However, local flow dynamics evaluated by the flow vorticity was different in the virtual coil model and the porous media model, in particular at the neck region of the aneurysm.

Encountering the Survival Strategies Using Various Nano Assemblages

2021 ◽  
pp. 159-187
Author(s):  
Lakshmipathy Muthukrishnan
Keyword(s):  
Antimicrobial Resistance ◽  
Chemical Properties ◽  
Fine Tuning ◽  
Survival Strategies ◽  
Natural Phenomenon ◽  
Physico Chemical ◽  
Life Threatening ◽  
Medicinal Properties ◽  
Micro Organisms ◽  
Microbial Systems

The technological advancements have not only made humans more civilized but have also caused the micro-organisms to develop several survival strategies via antimicrobial resistance to keep pace. Such highly developed microbial systems have been classified as superbugs, exhibiting Trojan-horse mechanism. This uncertain behaviour in microbes has challenged humans to scour around novel moiety to shield themselves from the detrimental effects. One such natural phenomenon that has drawn the attention of researchers is the metal-microbe interaction where microbes were found to be controlled during their interaction with metals. Fine tuning could bestow them with enhanced physico-chemical properties capable of controlling life-threatening micro-organisms. Nano forms of metals (nanoparticles, quantum dots, polymeric nanostructures) exhibiting medicinal properties have been implied toward biomedical theranostics. This chapter highlights the mechanistic antimicrobial resistance and the containment strategy using various nano assemblage highlighting its fabrication and bio-molecular interaction.

The spatial distribution of gyrotactic swimming micro-organisms in laminar flow fields

2011 ◽  
Vol 680 ◽  
pp. 602-635 ◽  
Author(s):  
R. N. BEARON ◽  
A. L. HAZEL ◽  
G. J. THORN
Keyword(s):  
Random Walk ◽  
Diffusion Model ◽  
Diffusion Models ◽  
Random Walk Model ◽  
Two Dimensional ◽  
Local Flow ◽  
Spatial And Temporal Scales ◽  
Advection Diffusion ◽  
Micro Organisms ◽  
Good Agreement

We compare the results of two-dimensional, biased random walk models of individual swimming micro-organisms with advection–diffusion models for the whole population. In particular, we consider the influence of the local flow environment (gyrotaxis) on the resulting motion. In unidirectional flows, the results of the individual and population models are generally in good agreement, even in flows in which the cells can experience a range of shear environments, and both models successfully predict the phenomena of gravitactic focusing. Numerical results are also compared with asymptotic expressions for weak and strong shear. Discrepancies between the models arise in two cases: (i) when reflective boundary conditions change the orientation distribution in the random walk model from that predicted by the long-term asymptotics used to derive the advection–diffusion model; (ii) when the spatial and temporal scales are not large enough for the advection–diffusion model to apply. We also use a simple two-dimensional flow containing a variety of flow regimes to explore what happens when there are localized regions in which the generalized Taylor dispersion theory used in the derivation of the population model does not apply. For spherical cells, we find good agreement between the models outside the ‘break-down’ regions, but comparison of the results within these regions is complicated by the presence of nearby boundaries and their influence on the random walk model. In contrast, for rod-shaped cells which are reorientated by both vorticity and strain, we see qualitatively different spatial patterns between individual and advection–diffusion models even in the absence of gyrotaxis, because cells are advected between regions of differing rates of strain.

scholarly journals Can Meta-Interpretive Learning outperform Deep Reinforcement Learning of Evaluable Game strategies?

2019 ◽  
Author(s):  
Céline Hocquette
Keyword(s):  
Reinforcement Learning ◽  
Optimal Strategies ◽  
Learning Systems ◽  
Logical System ◽  
Inductive Bias ◽  
Player Game ◽  
World Class ◽  
Logical Representation ◽  
Data Efficiency ◽  
Human Player

World-class human players have been outperformed in a number of complex two person games such as Go by Deep Reinforcement Learning systems GO. However, several drawbacks can be identified for these systems: 1) The data efficiency is unclear given they appear to require far more training games to achieve such performance than any human player might experience in a lifetime. 2) These systems are not easily interpretable as they provide limited explanation about how decisions are made. 3) These systems do not provide transferability of the learned strategies to other games. We study in this work how an explicit logical representation can overcome these limitations and introduce a new logical system called MIGO designed for learning two player game optimal strategies. It benefits from a strong inductive bias which provides the capability to learn efficiently from a few examples of games played. Additionally, MIGO's learned rules are relatively easy to comprehend, and are demonstrated to achieve significant transfer learning.

scholarly journals An autonomous operational trajectory searching system for an economic and environmental membrane bioreactor plant using deep reinforcement learning

2020 ◽  
Vol 81 (8) ◽  
pp. 1578-1587 ◽  
Author(s):  
KiJeon Nam ◽  
SungKu Heo ◽  
Jorge Loy-Benitez ◽  
Pouya Ifaei ◽  
ChangKyoo Yoo
Keyword(s):  
Reinforcement Learning ◽  
Membrane Bioreactor ◽  
Optimal Operation ◽  
Treatment Efficiency ◽  
Optimal Operating ◽  
Operational Costs ◽  
Aeration System ◽  
Effluent Discharge ◽  
Aeration Process ◽  
Micro Organisms

Abstract Optimal operation of membrane bioreactor (MBR) plants is crucial to save operational costs while satisfying legal effluent discharge requirements. The aeration process of MBR plants tends to use excessive energy for supplying air to micro-organisms. In the present study, a novel optimal aeration system is proposed for dynamic and robust optimization. Accordingly, a deep reinforcement learning (DRL)-based optimal operating system is proposed, so as to meet stringent discharge qualities while maximizing the system's energy efficiency. Additionally, it is compared with the manual system and conventional reinforcement learning (RL)-based systems. A deep Q-network (DQN) algorithm automatically learns how to operate the plant efficiently by finding an optimal trajectory to reduce the aeration energy without degrading the treated water quality. A full-scale MBR plant with the DQN-based autonomous aeration system can decrease the MBR's aeration energy consumption by 34% compared to other aeration systems while maintaining the treatment efficiency within effluent discharge limits.

Rotations and accumulation of ellipsoidal microswimmers in isotropic turbulence

2018 ◽  
Vol 838 ◽  
pp. 356-368 ◽  
Author(s):  
N. Pujara ◽  
M. A. R. Koehl ◽  
E. A. Variano
Keyword(s):  
Isotropic Turbulence ◽  
Swimming Velocity ◽  
Triaxial Ellipsoid ◽  
Local Flow ◽  
Equal Distribution ◽  
Ellipsoidal Particles ◽  
Velocity Gradients ◽  
Lower Fluid ◽  
Micro Organisms ◽  
Particle Shapes

Aquatic micro-organisms and artificial microswimmers locomoting in turbulent flow encounter velocity gradients that rotate them, thereby changing their swimming direction and possibly providing cues about the local flow environment. Using numerical simulations of ellipsoidal particles in isotropic turbulence, we investigate the effects of body shape and swimming velocity on particle motion. Four particle shapes (sphere, rod, disc and triaxial ellipsoid) are investigated at five different swimming velocities in the range $0\leqslant V_{s}\leqslant 5u_{\unicode[STIX]{x1D702}}$, where $V_{s}$ is the swimming velocity and $u_{\unicode[STIX]{x1D702}}$ is the Kolmogorov velocity scale. We find that anisotropic, swimming particles preferentially sample regions of lower fluid vorticity than passive particles do, and hence they accumulate in these regions. While this effect is monotonic with swimming velocity, the particle enstrophy (variance of particle angular velocity) varies non-monotonically with swimming velocity. In contrast to passive particles, the particle enstrophy is a function of shape for swimming particles. The particle enstrophy is largest for triaxial ellipsoids swimming at a velocity smaller than $u_{\unicode[STIX]{x1D702}}$. We also observe that the average alignment of particles with the directions of the velocity gradient tensor are altered by swimming leading to a more equal distribution of rotation about different particle axes.

scholarly journals Contrasting temporal difference and opportunity cost reinforcement learning in an empirical money-emergence paradigm

2018 ◽  
Vol 115 (49) ◽  
pp. E11446-E11454 ◽  
Author(s):  
Germain Lefebvre ◽  
Aurélien Nioche ◽  
Sacha Bourgeois-Gironde ◽  
Stefano Palminteri
Keyword(s):  
Reinforcement Learning ◽  
Human Subjects ◽  
Optimal Strategies ◽  
Theoretic Approach ◽  
Monetary Search ◽  
Commodity Money ◽  
Theoretical Predictions ◽  
Monetary Equilibrium ◽  
Reinforcement Learning Models ◽  
Classical Economic

Money is a fundamental and ubiquitous institution in modern economies. However, the question of its emergence remains a central one for economists. The monetary search-theoretic approach studies the conditions under which commodity money emerges as a solution to override frictions inherent to interindividual exchanges in a decentralized economy. Although among these conditions, agents’ rationality is classically essential and a prerequisite to any theoretical monetary equilibrium, human subjects often fail to adopt optimal strategies in tasks implementing a search-theoretic paradigm when these strategies are speculative, i.e., involve the use of a costly medium of exchange to increase the probability of subsequent and successful trades. In the present work, we hypothesize that implementing such speculative behaviors relies on reinforcement learning instead of lifetime utility calculations, as supposed by classical economic theory. To test this hypothesis, we operationalized the Kiyotaki and Wright paradigm of money emergence in a multistep exchange task and fitted behavioral data regarding human subjects performing this task with two reinforcement learning models. Each of them implements a distinct cognitive hypothesis regarding the weight of future or counterfactual rewards in current decisions. We found that both models outperformed theoretical predictions about subjects’ behaviors regarding the implementation of speculative strategies and that the latter relies on the degree of the opportunity costs consideration in the learning process. Speculating about the marketability advantage of money thus seems to depend on mental simulations of counterfactual events that agents are performing in exchange situations.

scholarly journals Cortical mechanisms for reinforcement learning in competitive games

2008 ◽  
Vol 363 (1511) ◽  
pp. 3845-3857 ◽  
Author(s):  
Hyojung Seo ◽  
Daeyeol Lee
Keyword(s):  
Game Theory ◽  
Reinforcement Learning ◽  
Learning Algorithm ◽  
Optimal Strategies ◽  
Simple Zero ◽  
Anterior Cingulate ◽  
Outcome Monitoring ◽  
Dorsolateral Prefrontal ◽  
Competitive Games ◽  
Zero Sum

Game theory analyses optimal strategies for multiple decision makers interacting in a social group. However, the behaviours of individual humans and animals often deviate systematically from the optimal strategies described by game theory. The behaviours of rhesus monkeys ( Macaca mulatta ) in simple zero-sum games showed similar patterns, but their departures from the optimal strategies were well accounted for by a simple reinforcement-learning algorithm. During a computer-simulated zero-sum game, neurons in the dorsolateral prefrontal cortex often encoded the previous choices of the animal and its opponent as well as the animal's reward history. By contrast, the neurons in the anterior cingulate cortex predominantly encoded the animal's reward history. Using simple competitive games, therefore, we have demonstrated functional specialization between different areas of the primate frontal cortex involved in outcome monitoring and action selection. Temporally extended signals related to the animal's previous choices might facilitate the association between choices and their delayed outcomes, whereas information about the choices of the opponent might be used to estimate the reward expected from a particular action. Finally, signals related to the reward history might be used to monitor the overall success of the animal's current decision-making strategy.

Sign in / Sign up

Export Citation Format

Share Document