Redox flow batteries (RFBs) are considered an outstanding candidate for the integration of renewable energy sources into the existing power grids. A key property of RFBs is the open circuit voltage (OCV) corresponding to the currentless equilibrium state. In the literature, the Nernst equation describing this property is often simplified by neglecting the activity coefficients. In this work, using a thermodynamically rigorous approach, we show that activity coefficients have a significant influence on the OCV of the Iron-Cadmium and All-Vanadium RFBs. Moreover, this influence varies with the state of charge. Therefore, activity coefficients should not be neglected in the Nernst equation. We show that when doing so, the resulting offset in OCV is actually comparable to typical voltage losses occurring during operation. Hence, fitting kinetic parameters to measurement data of voltage losses can lead to ambiguous results if only the idealized OCV, obtained by neglecting the activity coefficients, is used in that evaluation. Therefore, the implementation of a thermodynamically rigorous model has the potential to significantly improve state-of-the-art models for RFBs.