An experimental study has been carried out to characterize and investigate the performance of a flat-field grating spectrograph (FFGS) for higher diffraction order in the wavelength range of 80–170 Å. An aberration-corrected, mechanically ruled spherical grating with variable line spacing was used as a dispersive element in the spectrograph. The study was carried out using high-order harmonic radiation generated by an interaction of Ti:sapphire laser pulses with inert gas-filled cells. It was observed that the fraction of photons diffracted in the second diffraction order to that of the first order is as high as ∼ 65% at ∼ 80 Å wavelength, which reduces to ∼ 15% at ∼ 150 Å, whereas for the third diffraction order, the fraction was substantially lower, ∼ 15% at ∼ 100 Å. The observed results match well with reflectivity calculated using REFLEC software and also with the measurement carried out using INDUS-1 reflectivity beamline at RRCAT, Indore. The use of a high-order harmonic source for the study is advantageous, as the higher diffraction order contribution can be estimated from a single spectral image, which can be recorded in a single laser shot. The study will be useful for estimation of conversion efficiency–photon flux of a source using a variable line spaced (VLS) grating based spectrograph/monochromator.