This work is designed to artificially create test specimens with flaws that behave the same way as real-function flaws when observed by nondestructive testing (NDT) technologies. Thus, the understanding of the detection limitations of NDT methods is needed. In this study, real, realistic, and artificial flaws were compared by ultrasonic phased array technology. Fatigue flaws, which belong to the most common structural issues (Ruzicka, M., Hanke, M., and Rost, M., 1987, Dynamicka Pevnost a Zivotnost, CVUT, Prague, Czech Republic, p. 75), are investigated. Measurements have revealed significant differences in the amplitude of ultrasonic echo from fatigue cracks in distinct phases of crack propagation. Studied specimens with realistic flaws have demonstrated their quality for calibration, staff training, and NDT system qualification. More realistic test specimens will increase ultrasonic test result reliability.