In this chapter, a significant objection to contextualist theories of moral responsibility is examined, ‘the problem of unfairness’. After outlining the problem, reasons are offered to support the claim that moving to an invariant analysis of moral responsibility is unwarranted. It is argued that there is good reason, independent of contextualism, to divorce the concept of moral responsibility from that of blame, when the latter is construed as entailing harmful treatment of others. Even if this is denied, however, inter-agential judgements of moral responsibility can still be rendered fair. Consequently, a contextualist theory of moral responsibility can be retained whilst avoiding the problem of unfairness. This undercuts a significant motivation for endorsing an amelioration of our concept of moral responsibility, in either the direction of invariant compatibilism or that of invariant incompatibilism.