In this paper, we study the submodular load balancing problem with submodular penalties. The objective of this problem is to balance the load among sets, while some elements can be rejected by paying some penalties. Officially, given an element set V, we want to find a subset R of rejected elements, and assign other elements to one of m sets A1,A2,⋯,Am. The objective is to minimize the sum of the maximum load among A1,A2,⋯,Am and the rejection penalty of R, where the load and rejection penalty are determined by different submodular functions. We study the submodular load balancing problem with submodular penalties under two settings: heterogenous setting (load functions are not identical) and homogenous setting (load functions are identical). Moreover, we design a Lovász rounding algorithm achieving a worst-case guarantee of m+1 under the heterogenous setting and a min{m,⌈nm⌉+1}=O(n)-approximation combinatorial algorithm under the homogenous setting.