We present a recursive construction for difference sets which unifies the Hadamard, McFarland and Spence parameter families and deals with all abelian groups known to contain such difference sets. The construction yields a family of difference sets whose parameters were not previously known, and establishes a necessary and sufficient condition for the existence of a new family of McFarland difference sets. The results depend on a second recursive construction, for semi-regular relative difference sets with an elementary abelian forbidden subgroup. This second construction deals with all abelian groups known to contain such relative difference sets and significantly improves on previous results. Both of the recursive constructions generalise to nonabelian groups.