In some particular situations, participants need to recover different secrets both within a group (i.e., intragroup) and between two groups (i.e., intergroup). However, most of the existing multilevel secret sharing (MLSS) and multigroup secret sharing (MGSS) schemes mainly focus on how to protect a secret between one or more groups. In this paper, we propose a polynomial-based scheme to share multiple secret images both within a group and between groups. The random elements’ utilization model of integer linear programming is used to find polynomial coefficients that meet certain conditions so that each participant holds only one shadow image and some of them can recover secrets of both intergroup and intragroup. In addition, our scheme based on polynomials has the advantage of low computational complexity. Theoretical analysis and experiments show that the proposed scheme is feasible and effective.