AbstractHigher spectral efficiency and data rate per channel are the most cost-effective approaches to meet the exponential demand of data traffic in optical fiber network communication system. In this paper, diverse modulation formats are analyzed for Dense Wavelength Division Multiplexed system at 100 Gbps * 16=1600 Gbps data rates. The performance analysis of proffered system for Non-Return to Zero, Return to Zero, Carrier- Suppressed Return to Zero and Duo binary RZ with duty cycle 0.5 to 0.7 ranges like modulation formats are considered to find optimum modulation format for a 100 Gbps bit rate per channel optical fiber transmission network system. The simulations are analyzed for different values of input power, length of fiber, nonlinear refractive index, nonlinear dispersion and nonlinear effective area for all above mentioned modulation formats with spacing 100 to 250 GHz. to evaluate the effect of modulation format Fiber Bragg Gratting, optical fiber amplifier and Dispersion Compensation Fiber dispersion compensation techniques are enacted on this proposed optical network system.