The purpose of this study was to understand the ordered structure of starch in rice flour based on a physical modification with non-heating, milling, and water sorption through the structural evaluation of rice flour using small-angle X-ray scattering (SAXS) and infrared spectroscopy within the 4000–100 cm−1 region. The SAXS pattern of the samples with low moisture contents subjected to milling yield a band within the 0.4–0.9 nm−1 of the q range owing to a lamellar repeat of starch with an ordered structure in rice flour. We proposed an order parameter using the intensity of the SAXS band to quantify the order structure of starch in rice flour, and the true density was negatively correlated with the order parameter. Infrared band at 990 cm−1 in COH bending mode applied to the hydroxyl group of C6 shifted to a low wavenumber corresponding to the order parameter. A linear correlation was found between the order parameter and the 990 cm−1 and band at 861 cm−1 owing to COC symmetrical stretching of glycoside bond and CH2 deformation of the glucose unit of starch, 572, 472, and 436 cm−1, owing to the pyranose ring in the glucose unit of starch. The identified infrared bands are effective for quantifying the ordered structure of starch at the lamellar level. When subjected to water sorption, the band position at 990 cm−1 shifted to a higher wavenumber above a water activity of 0.7. This result revealed that the water-induced transition of glass to rubber of starch in rice flour can be clearly evaluated through infrared spectroscopy using the band at 990 cm−1. In addition, the band at 861 cm−1 also shifted to a higher wavenumber, whereas those at 572 and 436 cm−1 did not show a significant shift. These results indicate that water sorption slightly affects the internal structure and may mainly affect the surface of starch.