Although recent work has seen a rise in methods which perform categorization on each pixel of an image, we suggest that this is not the correct level of granularity. Instead of using pixels, we advocate the use of superpixels as the fundamental component of a class segmentation or pixel localization scheme. This statement may seem obvious to visual recognition practitioners, and at the same time flat wrong to theoretical statisticians, since merging pixels entails intermediate decisions that cannot improve the overall classification (Rao & Blackwell). We will address the issue by measuring the performance improvement of our algorithm over existing pixel-based approaches. To this end, we construct a classifier on the histogram of local features found in each superpixel.We find that we can regularize this classifier by aggregating histograms in the neighborhood of the superpixel. Our proposed method exceeds the previously published state-of-the-art on two extremely challenging datasets: Graz-02 and the PASCAL VOC 2007 Segmentation Challenge.