Improvement and measurement of neural style transfer

style transfer;gram matrix;texture synthesis

Yeh, Mao-Chuang ; Forsyth ; David A.

关键词: style transfer;    gram matrix;    texture synthesis;   

Others  :  https://www.ideals.illinois.edu/bitstream/handle/2142/102527/YEH-THESIS-2018.pdf?sequence=1&isAllowed=y

美国|英语

来源: The Illinois Digital Environment for Access to Learning and Scholarship

【 摘 要 】

Style transfer methods produce a transferred image which is a rendering of a content image in the manner of a style image. There is a rich literature of variant methods.We seek to understand how to improve style transfer: in particular, there is some evidence that cross-layer losses are helpful, and some evidence that optimization problems might present difficulties. To do so requires quantitative evaluation procedures, but current evaluation is qualitative, mostly involving user studies.We describe a novel quantitative evaluation procedure. Our procedure relies on two statistics: the Effectiveness (E) statistic measures the extent that a given style has been transferred to the target, and the Coherence (C) statistic measures the extent to which the original image's content is preserved.Our statistics are calibrated to human preference: targets with larger values of E (resp C) will reliably be preferred by human subjects in comparisons of style (resp. content).We use these statistics to investigate relative performance of a number of recent style transfer methods, revealing a number of intriguing properties.{Our experiments pool multiple style transfers from many different styles to many different content images using many different style weights, allowing us to make general statements about what influences style transfer. }Admissible methods lie on a Pareto frontier (i.e. improving E reduces C, or vice versa). Three methods are admissible: Universal style transfer produces very good C but weak E; modifying the optimization used for Gatys' loss produces a method with strong E and strong C; and a modified cross-layer method has slightly better E at strong cost in C.While the histogram loss improves the E statistics of Gatys' method, it does not make the method admissible.Surprisingly, style weights have relatively little effect, and most variability in transfer is explained by the style itself (meaning experimenters can be misguided by selecting styles).