Wednesday, October 26, 2016
Posted by Vincent Dumoulin*, Jonathon Shlens and Manjunath Kudlur, Google Brain Team

Pastiche. A French word, it designates a work of art that imitates the style of another one (not to be confused with its more humorous Greek cousin, parody). Although it has been used for a long time in visual art, music and literature, pastiche has been getting mass attention lately with online forums dedicated to images that have been modified to be in the style of famous paintings. Using a technique known as style transfer, these images are generated by phone or web apps that allow a user to render their favorite picture in the style of a well known work of art.

Although users have already produced gorgeous pastiches using the current technology, we feel that it could be made even more engaging. Right now, each painting is its own island, so to speak: the user provides a content image, selects an artistic style and gets a pastiche back. But what if one could combine many different styles, exploring unique mixtures of well known artists to create an entirely unique pastiche?

Learning a representation for artistic style

In our recent paper titled “A Learned Representation for Artistic Style”, we introduce a simple method to allow a single deep convolutional style transfer network to learn multiple styles at the same time. The network, having learned multiple styles, is able to do style interpolation, where the pastiche varies smoothly from one style to another. Our method enables style interpolation in real-time as well, allowing this to be applied not only to static images, but also videos.

 
Here is a video from YouTube. 
 
Credit: awesome dog role played by Google Brain team office dog Picabo.

In the video above, multiple styles are combined in real-time and the resulting style is applied using a single style transfer network. The user is provided with a set of 13 different painting styles and adjusts their relative strengths in the final style via sliders. In this demonstration, the user is an active participant in producing the pastiche.

A Quick History of Style Transfer

While transferring the style of one image to another has existed for nearly 15 years [1] [2], leveraging neural networks to accomplish it is both very recent and very fascinating. In “A Neural Algorithm of Artistic Style” [3], researchers Gatys, Ecker & Bethge introduced a method that uses deep convolutional neural network (CNN) classifiers. The pastiche image is found via optimization: the algorithm looks for an image which elicits the same kind of activations in the CNN’s lower layers - which capture the overall rough aesthetic of the style input (broad brushstrokes, cubist patterns, etc.) - yet produces activations in the higher layers - which capture the things that make the subject recognizable - that are close to those produced by the content image. From some starting point (e.g. random noise, or the content image itself), the pastiche image is progressively refined until these requirements are met.

Content image: The Tübingen Neckarfront by Andreas Praefcke, Style painting: “Head of a Clown”, by Georges Rouault.

The pastiches produced via this algorithm look spectacular:

Figure adapted from L. Gatys et al. "A Neural Algorithm of Artistic Style" (2015). 

This work is considered a breakthrough in the field of deep learning research because it provided the first proof of concept for neural network-based style transfer. Unfortunately this method for stylizing an individual image is computationally demanding. For instance, in the first demos available on the web, one would upload a photo to a server, and then still have plenty of time to go grab a cup of coffee before a result was available.

This process was sped up significantly by subsequent research [4, 5] that recognized that this optimization problem may be recast as an image transformation problem, where one wishes to apply a single, fixed painting style to an arbitrary content image (e.g. a photograph). The problem can then be solved by teaching a feed-forward, deep convolutional neural network to alter a corpus of content images to match the style of a painting. The goal of the trained network is two-fold: maintain the content of the original image while matching the visual style of the painting.

The end result of this was that what once took a few minutes for a single static image, could now be run real time (e.g. applying style transfer to a live video). However, the increase in speed that allowed real-time style transfer came with a cost - a given style transfer network is tied to the style of a single painting, losing some flexibility of the original algorithm, which was not tied to any one style. This means that to build a style transfer system capable of modeling 100 paintings, one has to train and store 100 separate style transfer networks.

Our Contribution: Learning and Combining Multiple Styles

We started from the observation that many artists from the impressionist period employ similar brush stroke techniques and color palettes. Furthermore, painting by say, Monet, are even more visually similar.

Poppy Field (left) and Impression, Sunrise (right) by Claude Monet. Images from Wikipedia

We leveraged this observation in our training of a machine learning system. That is, we trained a single system that is able to capture and generalize across many Monet paintings or even a diverse array of artists across genres. The pastiches produced are qualitatively comparable to those produced in previous work, while originating from the same style transfer network.

Pastiches produced by our single network, trained on 32 varied styles. These pastiches are qualitatively equivalent to those created by single-style networks: Image Credit: (from top to bottom) content photographs by Andreas PraefckeRich Niewiroski Jr. and J.-H. Janßen, (from left to right) style paintings by William GlackensPaul SignacGeorges RouaultEdvard Munch and Vincent van Gogh.
 

The technique we developed is simple to implement and is not memory intensive. Furthermore, our network, trained on several artistic styles, permits arbitrary combining multiple painting styles in real-time, as shown in the video above. Here are four styles being combined in different proportions on a photograph of Tübingen:

Unlike previous approaches to fast style transfer, we feel that this method of modeling multiple styles at the same time opens the door to exciting new ways for users to interact with style transfer algorithms, not only allowing the freedom to create new styles based on the mixture of several others, but to do it in real-time. Stay tuned for a future post on the Magenta blog, in which we will describe the algorithm in more detail and release the TensorFlow source code to run this model and demo yourself. We also recommend that you check out Nat & Lo’s fantastic video explanationon the subject of style transfer.

References

[1] Efros, Alexei A., and William T. Freeman. Image quilting for texture synthesis and transfer (2001).

[2] Hertzmann, Aaron, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. Image analogies (2001).

[3] Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. A Neural Algorithm of Artistic Style(2015).

[4] Ulyanov, Dmitry, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky. Texture Networks: Feed-forward Synthesis of Textures and Stylized Images (2016).

[5] Johnson, Justin, Alexandre Alahi, and Li Fei-Fei. Perceptual Losses for Real-Time Style Transfer and Super-Resolution (2016).

 
 
 

(转) Supercharging Style Transfer的更多相关文章

  1. Image Style Transfer:多风格 TensorFlow 实现

    ·其实这是一个选修课的present,整理一下作为一篇博客,希望对你有用.讲解风格迁移的博客蛮多的,我就不过多的赘述了.讲一点几个关键的地方吧,当然最后的代码和ppt也希望对你有用. 1.引入: 风格 ...

  2. 项目总结四:神经风格迁移项目(Art generation with Neural Style Transfer)

    1.项目介绍 神经风格转换 (NST) 是深部学习中最有趣的技术之一.它合并两个图像, 即 内容图像 C(content image) 和 样式图像S(style image), 以生成图像 G(ge ...

  3. DeepLearning.ai-Week4-Deep Learning & Art: Neural Style Transfer

    1 - Task Implement the neural style transfer algorithm Generate novel artistic images using your alg ...

  4. 课程四(Convolutional Neural Networks),第四 周(Special applications: Face recognition & Neural style transfer) —— 2.Programming assignments:Art generation with Neural Style Transfer

    Deep Learning & Art: Neural Style Transfer Welcome to the second assignment of this week. In thi ...

  5. Art: Neural Style Transfer

    Andrew Ng deeplearning courese-4:Convolutional Neural Network Convolutional Neural Networks: Step by ...

  6. Perceptual Losses for Real-Time Style Transfer and Super-Resolution and Super-Resolution 论文笔记

    Perceptual Losses for Real-Time Style Transfer and Super-Resolution and Super-Resolution 论文笔记 ECCV 2 ...

  7. pytorch实现style transfer

    说是实现,其实并不是我自己实现的 亮出代码:https://github.com/yunjey/pytorch-tutorial/tree/master/tutorials/03-advanced/n ...

  8. fast neural style transfer图像风格迁移基于tensorflow实现

    引自:深度学习实践:使用Tensorflow实现快速风格迁移 一.风格迁移简介 风格迁移(Style Transfer)是深度学习众多应用中非常有趣的一种,如图,我们可以使用这种方法把一张图片的风格“ ...

  9. 《Perceptual Losses for Real-Time Style Transfer and Super-Resolution》论文笔记

    参考 http://blog.csdn.net/u011534057/article/details/55052304 代码 https://github.com/yusuketomoto/chain ...

随机推荐

  1. html5-浮动

    #div1{    background: rgba(255,0,0,0.5);    width: 250px;    height: 250px;    float: right;}#div2{  ...

  2. Problem(莫比乌斯反演)

    我不是传送门 题意 : 中文题目不解释 求gcd(x,y) = k (a<=x<=b, c<=y<=d); 根据gcd(ka,kb) = k*gcd(a,b), 可将问题转化为 ...

  3. D Tree Requests dfs+二分 D Pig and Palindromes -dp

    D time limit per test 2 seconds memory limit per test 256 megabytes input standard input output stan ...

  4. Spark学习之路 (十八)SparkSQL简单使用

    一.SparkSQL的进化之路 1.0以前: Shark 1.1.x开始: SparkSQL(只是测试性的)  SQL 1.3.x: SparkSQL(正式版本)+Dataframe 1.5.x: S ...

  5. linux常用命令:more 命令

    more命令,功能类似 cat ,cat命令是整个文件的内容从上到下显示在屏幕上. more会以一页一页的显示方便使用者逐页阅读,而最基本的指令就是按空白键(space)就往下一页显示,按 b 键就会 ...

  6. Selenium自动化测试,接口自动化测试开发,性能测试从入门到精通

    Selenium自动化测试,接口自动化测试开发,性能测试从入门到精通Selenium接口性能自动化测试基础部分:分层自动化思想Slenium介绍Selenium1.0/2.0/3.0Slenium R ...

  7. linux内核的双链表list_head、散列表hlist_head

    一.双链表list_head 1.基本概念 linux内核提供的标准链表可用于将任何类型的数据结构彼此链接起来. 不是数据内嵌到链表中,而是把链表内嵌到数据对象中. 即:加入链表的数据结构必须包含一个 ...

  8. vue:vue-resource

    vue-resource是一个非常轻量的用于处理HTTP请求的插件,它提供了两种方式来处理HTTP请求: 使用Vue.http或this.$http: 使用Vue.resource或this.$res ...

  9. [转载] c#中decimal ,double,float的区别

    一直很奇怪C#的预定义数据类型中为什么加了一个decimal,有float和double不就够了吗?今天来挖一挖. 浮点型 Name CTS Type De script ion Significan ...

  10. 前端框架VUE----面向对象

    JavaScript 语言中,生成实例对象的传统方法是通过构造函数. function Animal(name,age){ this.name = name; this.age = age; } An ...