Boundary localization is one of the key issues for reliable iris recognition system. For non-ideal iris images, eyelashes or eyelids occlusions and low contrast between iris and sclera will lead to inaccurate boundary localization. Specifically, if the intensive transition from iris to sclera is too smooth, outer boundary localization will be very difficult. To stress the problem, in this paper the boundary localization method is proposed in which nonlinear gray transformation is innovated in outer boundary localization process. The experimental results depict that our algorithm have improved the localization accuracy for non-ideal iris compared to the classical algorithms.