To better understand the marginality of marginal granular materials (i.e., an aggregate that does not fully meet a premium quality specification), this study investigated the permanent strain behavior of two granular materials with different geological and mineralogical characteristics under repeated loading and different moisture conditions. The secant permanent strain rate and shakedown approach were used to analyze the permanent strain behavior of the materials. A one-dimensional swelling test was used to explain the mechanism of the permanent strain behavior of the soaked materials under repeated loading. The results show that compared with a New Zealand premium quality aggregate, the marginal aggregate exhibits a more significant increase in the cumulative permanent strain and the secant permanent strain rate after the 4-day soaking process. The results of the shakedown approach illustrate that the marginal aggregate performs reasonably well to resist permanent deformation in a dry condition, but the performance will considerably deteriorate when in contact with water under high stress. This change is a result of the presence of swelling clay minerals (e.g., smectite) in the marginal aggregates. The result of the one-dimensional swelling test indicates the high swelling potential of marginal aggregate, which provides an explanation for its poorer resistance to permanent strain. The marginal aggregate gains undesirable internal lubrication when surrounded by (moist) swelling clay mineral particles; thus, its shear strength reduces and permanent strain rate increases under repeated loading.