Genetic liability to substance use disorders can be parsed into loci conferring general and substance-specific addiction risk. We report a multivariate genome-wide association study that disaggregates general and substance-specific loci for problematic alcohol use, problematic tobacco use, and cannabis and opioid use disorders in a sample of 1,025,550 individuals of European and 92,630 individuals of African descent. Nineteen loci were genome-wide significant for the general addiction risk factor (addiction-rf), which showed high polygenicity. Across ancestries PDE4B was significant (among others), suggesting dopamine regulation as a cross-trait vulnerability. The addiction-rf polygenic risk score was associated with substance use disorders, psychopathologies, somatic conditions, and environments associated with the onset of addictions. Substance-specific loci (9 for alcohol, 32 for tobacco, 5 for cannabis, 1 for opioids) included metabolic and receptor genes. These findings provide insight into the genetic architecture of general and substance-specific use disorder risk that may be leveraged as treatment targets.