The Primordial Inflation Explorer (PIXIE) is an Explorer-class mission concept to measure the gravitational-wave signature of primordial inflation through its distinctive imprint on the linear polarization of thecosmic microwave background (CMB). Its optical system couples a polarizing Fourier transform spectrometerto the sky to measure the differential signal between orthogonal linear polarization states from two co-pointedbeams on the sky. The double differential nature of the four-port measurement mitigates beam-related systematic errors common to the two-port systems used in most CMB measurements. Systematic errors coupling unpolarized temperature gradients to a false polarized signal cancel to first order for any individual detector. Thiscommon-mode cancellation is performed optically, prior to detection, and does not depend on the instrumentcalibration. Systematic errors coupling temperature to polarization cancel to second order when comparing signals from independent detectors. We describe the polarized beam patterns for PIXIE and assess the systematicerror for measurements of CMB polarization.