Surface integrity of machined components is critical for product performance in service. Process dynamic parameters, such as cutting speed and the changing contact condition between the tool flank face and machined surface, have a significant influence on surface integrity of a machined surface. Due to the very small scale of surface integrity factors on a machined surface, nanoindentation can be used to determine the surface/subsurface mechanical properties. However, the test data may be significantly influenced by machining induced residual stresses, strain hardening, and microstructure changes. The fundamental relationships between residual stress, microstructure, and nanohardness in the machined surface are yet to be understood. Further, it is not clear how to determine residual stress, at least its nature of tensile or compressive, from the nanoindentation data with the presence of complex residual stress state, strain hardening, and microstructure changes. This study focuses on the effects of cutting speed and machining system damping or rigidity (through varying tool flank wear) on subsurface mechanical state and the basic relationships between residual stress, white layer, and nanohardness. A series of nanoindentation tests were conducted to machined samples with distinct surface integrity by hard turning, grinding, and honing. It was found that white layer increases nanohardness and dark layer decreases nanohardness in subsurface, while strain hardening only slightly increases subsurface hardness. The research results indicate that subsurface residual stress can be qualitatively characterized by the load-displacement curve pattern and its parameters such as slope at initial loading, total depth, residual depth, and the ratio of residual depth to total depth. Residual stress would affect a load-displacement curve shape only at onset of yielding. Microstructure changes would make a significant difference on the characteristics of a load-displacement curve, while strain hardening exerts slight influence on the curve characteristics. In addition, the mechanism of residual stress on indentation depth was explained using a Mohr’s circle.