Objective Neutrophil-to-lymphocyte ratio (NLR) value has emerged as a cardiovascular prognostic marker. Although several recent studies suggested NLR was associated with arterial stiffness, it was still controversial. The aim of this study was to investigate the correlation between NLR and arterial stiffness by measuring of brachial-ankle pulse wave velocity (baPWV) in an apparently healthy population. Methods This retrospective study enrolled 5612 participants during the health examinations from 1 October 2007 to 30 September 2011. Arterial stiffness was measured by baPWV. NLR was calculated as the ratio of the absolute neutrophil count to the absolute lymphocyte count in peripheral blood. According to the quartiles of NLR, the patients were categorized into four groups in males and females, respectively. Associations between NLR and baPWV were evaluated using partial correlation and multivariate logistic regression analysis. Results Both female and male subjects with increased arterial stiffness (baPWV ≥ 1400 cm/s) were likely to be older (females: P < 0.001, males: P < 0.001) and have higher systolic blood pressure (females: P < 0.001, males: P < 0.001), diastolic blood pressure (females: P < 0.001, males: P < 0.001), fasting plasma glucose (females: P < 0.001, males: P < 0.001), serum total cholesterol (females: P < 0.001, males: P = 0.028), triglyceride (females: P < 0.001, males: P = 0.031), urea nitrogen (females: P < 0.001, males: P < 0.001) than those without increased arterial stiffness. In addition, compared to those without increased arterial stiffness, body mass index ( P < 0.001), waist circumference ( P < 0.001), low-density lipoproteins cholesterol ( P < 0.001), creatinine ( P < 0.001), uric acid ( P < 0.001) and lymphocytes ( P = 0.001) were higher in females with increased arterial stiffness. However, males with increased arterial stiffness had higher NLR value (2.0 ± 0.7 vs. 2.1 ± 0.9, P < 0.001) and neutrophils (4.3 ± 1.4 vs. 4.5 ± 1.5, P < 0.001) than those without increased arterial stiffness, while the difference was not found in females. ANCOVA showed that males with quartile 3 and quartile 4 of NLR had greater levels of baPWV. NLR was correlated to baPWV in males by partial correlation analysis (r = 0.110, P < 0.001), but not in females. In multiple logistic regression analysis, the quartile 4 of NLR was positively associated with increased arterial stiffness in males (OR = 1.43, 95% confidence intervals [CI]=1.12–1.82, P = 0.004), but there was no obvious correlation in females. Conclusions Our findings suggest that there is a gender difference in the relationship between arterial stiffness and NLR. After adjusting for other confounders, the risk of increased arterial stiffness in apparently healthy adult males (rather than females) is independently associated with the highest quartile of NLR.