The aim of this study was to examine the effect of size on the mixed-mode fracture toughness of quasi-brittle nanocomposites with the help of modified maximum tangential stress criterion. The literature reveals that the effect of size on mixed-mode fracture behavior of brittle nanocomposites has not been well investigated previously using modified maximum tangential stress criterion. The studied nanocomposites were made of epoxy resin reinforced with 7 wt%, 20–30 nm nanosilica. The accuracy of the method was assessed by taking into account the high-order terms of Williams series expansion along with finite element over-deterministic method. To investigate the effect of size on fracture toughness, a number of three-point semi-circular bending tests with different radii and four angles of edge–crack orientation were conducted and subjected to mixed-mode loading. The size of fracture process zone and apparent fracture toughness ( Kc) were also evaluated as a function of sample size. Experimental results showed that the proposed approach can accurately predict the fracture behavior of studied nanocomposites.