Based on the classic quadratic map (CQM) with abundant bifurcations and periodic windows, a new 3D improved coupling quadratic map (3D-ICQM) is constructed, and its phase diagram, Lyapunov exponent (LE) and randomness testing by TestU01 demonstrated that it has better ergodicity, more complex nonlinear behavior, larger chaotic range and better randomness. To investigate its application in cryptography, a 3D-ICQM based color image encryption algorithm with round key expansion is designed. To further enhance the randomness of key stream, the hash value of intermediate cipher of the previous round is converted into new initial values and fed back into 3D-ICQM, to generate the key stream for the current round. The security and statistical analyses demonstrated that the encryption scheme has better security performance than several existing algorithms.