Memory demand is associated with increased mental workload. The objective of the present study was to examine the effects of visuospatial memory secondary tasks on driving performance. Memory tasks for the unknown word-figure pairs and recognition tasks for word-figure pairs at two-level difficulties were employed separately to represent working memory’s process and long-term memory’s process. A simulator study was conducted based on the simulation of the standard environment of Lane change test (LCT). The performance of lane keeping, lane change, and secondary tasks was measured by statistical methods. The comprehensive appraisal model was constructed to quantify total driving performance. The results showed that the mean path deviation, steering angle, and lane excursion times increased, and the proportion of correct lane change decreased, with the perceived workload increasing and the total driving performance decreasing in dual-task driving condition. Compared with the simple working memory group, as the difficulty of tasks increased in difficult working memory group, lane change performance degraded and the perceived workload increased. In contrast to difficult working memory group, the performance of lane keeping and lane change increased, while the perceived workload decreased and the total performance increased by about 50% in difficult recognition group. There were few differences between the simple working memory group and simple recognition group. The difficult working memory group had the lowest total driving performance. The results indicate that as the secondary task’s difficulty increases, driving performance will degrade. Performance improves significantly when the working memory process is converted to the recognition process. This trend is more obvious when the memory task assumes to be more difficult.