This paper considers a non-orthogonal multiple access (NOMA)-assisted ambient backscatter communication (AmBC) system. To maximize the achievable sum rate (ASR) of the AmBC system, a joint optimization problem over a backscatter device (BD) grouping strategy, reflection coefficients, and decoding order is formulated, where the BD grouping strategy contains the number of BD groups and the BD allocation strategy. The BD grouping strategy, the reflection coefficients, and the decoding order are all intertwined, and the global search is extremely complex. As a result, we propose a four-step optimization algorithm. First, we give the closed-form optimal solution of the BD decoding order and reflection coefficient for a given grouping strategy. Then, for a given number of BD groups, we propose a low-complexity BD allocation strategy based on the complexity–performance trade-off. Finally, the number of BD groups with the largest ASR is selected as the global optimal number of BD groups. The simulation results show that the proposed four-step optimization algorithm is better than the benchmark solution.