We discuss a confocal surface measurement system having improved accuracy for rough surfaces and speed. To enhance speed, we propose (1) a novel nonscanning confocal microscope, (2) a fast object surface reconstruction algorithm, and (3) an optical focused plane shift mechanism. The system we developed operates at speeds exceeding a million points per second. To improve accuracy for rough surfaces, we propose optical smoothing using confocal beam spot microscanning. Results of experiments confirmed that the system measures rough surfaces nearly as accurately as smooth surfaces.