The Hidden Subgroup Problem (HSP) has been widely studied in the context of quantum computing and is known to be efficiently solvable for Abelian groups, yet appears to be difficult for many non-Abelian ones. An efficient algorithm for the HSP over a group \f G\ runs in time polynomial in \f{n\deq\log|G|.} For any subgroup \f H\ of \f G, let \f{N(H)} denote the normalizer of \f H. Let \MG\ denote the intersection of all normalizers in \f G (i.e., \f{\MG=\cap_{H\leq G}N(H)}). \MG\ is always a subgroup of \f G and the index \f{[G:\MG]} can be taken as a measure of ``how non-Abelian'' \f G is (\f{[G:\MG] = 1} for Abelian groups). This measure was considered by Grigni, Schulman, Vazirani and Vazirani, who showed that whenever \f{[G:\MG]\in\exp(O(\log^{1/2}n))} the corresponding HSP can be solved efficiently (under certain assumptions). We show that whenever \f{[G:\MG]\in\poly(n)} the corresponding HSP can be solved efficiently, under the same assumptions (actually, we solve a slightly more general case of the HSP and also show that some assumptions may be relaxed).