In this article, pure epoxy resin and silica–epoxy nanocomposite models were established to investigate the effects of hyperbranched polyester on microstructure and thermomechanical properties of epoxy resin through molecular dynamics simulation. Results revealed that the composite of silica can improve the thermomechanical properties of nanocomposites, including the glass transition temperature, thermal conductivity, and elastic modulus. Moreover, the thermomechanical properties were further enhanced through chemical modification on the silica surface, where the effectiveness was the best through grafting hyperbranched polyester on the silica surface. Compared with pure epoxy resin, the glass transition temperature of silica–epoxy composite modified by silica grafted with hyperbranched polyester increased by 38 K. The thermal conductivity increased with the increase of temperature and thermal conductivity at room temperature increased to 0.4171 W/(m·K)−1 with an increase ratio of 94.3%. Young’s modulus, volume modulus, and shear modulus all fluctuated as temperature rise with a down overall trend. They increased by 44.68%, 29.52%, and 36.65%, respectively, when compared with pure epoxy resin. At the same time, the thermomechanical properties were closely related to the microstructure such as fractional free volume (FFV), mean square displacement (MSD), and binding energy. Silica surface modification by grafting hyperbranched polyester reduced the FFV value and MSD value most and strengthened the combination of silica and epoxy resin matrix the best, resulting in the best thermomechanical properties.
