We resolve the existence problem of $(96,20,4)$ difference sets in 211 of 231 groups of order $96$. If $G$ is a group of order $96$ with normal subgroups of orders $3$ and $4$ then by first computing $32$- and $24$-factor images of a hypothetical $(96,20,4)$ difference set in $G$ we are able to either construct a difference set or show a difference set does not exist. Of the 231 groups of order 96, 90 groups admit $(96,20,4)$ difference sets and $121$ do not. The ninety groups with difference sets provide many genuinely nonabelian difference sets. Seven of these groups have exponent 24. These difference sets provide at least $37$ nonisomorphic symmetric $(96,20,4)$ designs.
Almost difference sets in nonabelian groups