Highly accurate atmospheric models, based on molecular resonance information contained within the HITRAN database, were used to simulate the propagation of high capacity single-carrier quadrature amplitude modulated signals through the atmosphere for various modulation orders. For high-bandwidth signals such as those considered in this work, group velocity dispersion caused by atmospheric gases distorts the modulated waveform, which may produce bit errors. This leads to stricter Signal-To-Noise Ratio requirements for error-free operation, and this effect is more pronounced in high-order modulation schemes. At the same time, high-order modulation schemes are more spectrally efficient, which reduces the bandwidth required to maintain a given data rate, and thus reduces the total group velocity dispersion in the link, resulting in less distortion and better performance. Our work with M-ary quadrature amplitude modulated signals shows that optimal selection of modulation order can minimize these conflicting effects, resulting in decreased error rate, and reducing the performance requirements placed on any equalizers, other dispersion-compensating technologies, or signal processing hardware.