To remember or forget our acquaintances’ strategies can influence our decision-making significantly. In this paper, we explore the evolutionary prisoner’s dilemma (PD) game model with punishment and memory mechanism in the time-varying network. Our results show that a larger temptation gain [Formula: see text] or a larger the number of connected edges of activated individuals [Formula: see text] would result in the decrease of the final fraction of cooperators. However, with the increase of the maximum penalty cost, the maximum punishment intensity or the value of individual’s “memory factor”, players are more inclined to choose cooperative strategy. In addition, an effective way to promote the cooperation is to improve the social subsidy. Remarkably, only when the social subsidy is greater than the temptation gain, the density of cooperators could increase significantly. Interestingly, there is a linear relationship between the threshold of social subsidy and the temptation gain. The final results show that ones’ activity rates have no significant correlation with their strategies.