Are you from North or South India? A hard face-classificationtask reveals systematic representational differences betweenhumans and machines

Journal of Vision

article

Harish Katti¹  S. P. Arun¹ 

[1] Centre for Neuroscience, Indian Institute of Science

关键词: face categorization;    ethnicity;    computational models;    deep convolutional networks;    feature representation;   

DOI  :  10.1167/19.7.1

来源: Association for Research in Vision and Ophthalmology

【 摘 要 】

We make a rich variety of judgments on faces, but theunderlying features are poorly understood. Here wedescribe a challenging geographical-origin classificationproblem that elucidates feature representations in bothhumans and machine algorithms. In Experiment 1, wecollected a diverse set of 1,647 faces from India labeledwith their fine-grained geographical origin (North vs.South India), characterized the categorizationperformance of 129 human subjects on these faces, andcompared this with the performance of machine visionalgorithms. Our main finding is that while many machinealgorithms achieved an overall performance comparableto that of humans (64%), their error patterns across faceswere qualitatively different despite training. To elucidatethe face parts used by humans for classification, wetrained linear classifiers on overcomplete sets of featuresderived from each face part. This revealed mouth shapeto be the most discriminative part compared to eyes,nose, or external contour. In Experiment 2, we confirmedthat humans relied the most on mouth shape forclassification using an additional experiment in whichsubjects classified faces with occluded parts. InExperiment 3, we compared human performance forbriefly viewed faces and for inverted faces. Interestingly,human performance on inverted faces was predictedbetter by computational models compared to uprightfaces, suggesting that humans use relatively moregeneric features on inverted faces. Taken together, ourresults show that studying hard classification tasks canlead to useful insights into both machine and humanvision.

【 授权许可】

CC BY|CC BY-NC-ND