This paper presents a new 4D chaotic system by extending a 3D system to investigate chaotic systems2 with hidden attractors further and explore their engineering applications in color image encryption. The system has hyperbolic shape equilibriums and the equilibriums are nonhyperbolic which is interesting and rarely mentioned before. The system has been investigated deeply by phase plot, time series, the largest Lyapunov exponent, bifurcation diagram, Poincaré map and 0–1 test. A coexisting hidden attractor has been found in this new system and the circuit simulation model of the system is built by Power Simulation (PSIM). Circuit simulation results are consistent with results of numerical simulations which verify its dynamic behavior further. Then, by combining the generalized Arnold transform with the new chaotic system, a new color image encryption algorithm has been proposed which has large key space and can encrypt either square images or nonsquare images with different size. Finally, the results of encryption experiment and security analysis prove the effectiveness of the algorithm.