Multiphase flow widely exists in the nature and engineering. The two-phase flow is the highlight of the studies about the flow in the vessel and steam explosion in nuclear severe accidents. The Moving Particle Semi-implicit (MPS) method is a fully-Lagrangian particle method without grid mesh which focuses on tracking the single particle and concerns with its movement. It has advantages in tracking complex multiphase flows compared with gird methods, and thus shows great potential in predicting multiphase flows. The objective of this thesis is to develop a general multiphase particle method based on the original MPS method and thus this work is of great significance for improving the numerical method for simulating the instability in reactor severe accident and two-phase flows in vessel. This research is intended to provide a study of the instability based on the MPS method. Latest achievements of mesh-free particle methods in instability are researched and a new multiphase MPS method, which is based on the original one, for simulating instability has been developed and validated. Based on referring to other researchers’ papers, the Pressure Poisson Equation (PPE), the viscosity term, the free surface particle determination part and the surface tension model are optimized or added. The numerical simulation on stratification behavior of two immiscible flows is carried out and results are analyzed after data processing. It is proved that the improved MPS method is more accurate than the original method in analysis of multiphase flows. In this paper, the main purposes are simulating and discussing Rayleigh-Taylor (R-T) instability and Kelvin-Helmholtz (K-H) instability. R-T and K-H instability play an important role in the mixing process of many layered flows. R-T instability occurs when a lower density fluid is supported by another density higher fluid or higher density fluid is accelerated by lower density fluid, and the resulting small perturbation increases and eventually forms turbulence. K-H instability is a small disturbance for two different densities, such as waves, at the interface of the two-phase fluid after giving a fixed acceleration in the fluid. Turbulence generated by R-T instability and K-H instability has an important effect in applications such as astrophysics, geophysics, and nuclear science.