AbstractIn this paper, performance of QAM sequence generator in coherent optical communication system is investigated. Results are obtained for different square constellation types such as 4-QAM, 16-QAM, 64-QAM and 256-QAM using Hyperbolic-secant pulse generator. For the distance of 120 km and bit rate of 10 Gbps, it is investigated that 4-QAM provides maximum Q factor of 33.93 and minimum BER of 6e-253, whereas 16-QAM, 64-QAM & 256-QAM provides Q factor of 31.47, 22.85 & 8 and BER of 5e-218, 3e-116 & 2e-16, respectively. Decrement in Quality factor and increase in BER are observed with increment in the M-value of QAM. It is further observed that Eye opening decreases with increment in levels of QAM.