Refactoring is a transformation that preserves the external behavior of a program and improves its internal quality. To help developers in this task, current IDEs, such as Eclipse and NetBeans, automate a number of refactorings. However, implementing refactorings is a complex task, and so, even mainstream IDEs contain critical bugs. We propose an automated approach for testing of Java refactoring engines. Its key components are: JDOLLY, a Java program generator, and SAFEREFACTOR, a program for checking behavioral changes. The technique uses JDOLLY to generate programs as test inputs. For each generated program, it applies the refactoring by using the engine under test, and uses oracles based on SAFEREFACTOR to evaluate the correctness of the transformation. In the end, it classifies the failures into distinct bugs. We have evaluated this technique by testing up to 10 refactorings from Eclipse, NetBeans and the JastAdd Refactoring Tools. Our technique tested 153,444 transformations, and identified more than 100 bugs, which were reported to engines’ developers. They accepted most of them, and already fixed 35 bugs.