Hello, this is the less formal but way more exciting part of my website where I post all of my tech experiments.

Search This Blog

Friday, April 30, 2021

Arbitrary Style Transfer in an Illustrative Workflow

Style transfer tends to be an awkward topic around my artist friends. There are two general perspectives that I see. First is fear that the world won't need their skills if AI starts illustrating. The more prevalent idea is that style transfer doesn't do anything artistically useful.

To figure out the truth about style transfer, I decided to dive into it myself. I started by downloading an arbitrary style transfer app called Pikazo onto my iPhone. (There are better methods with more control but, this the most artist-friendly interface that I have found for style transfer)


The marketing for Pikazo is that it takes a photo and a reference piece of art and then recreates the photo in the style of the original. let's do some tests.

I started with something I thought might be a bit simpler because glitch art elements are a little less context-sensitive than say a cartoon style in which case the AI needs to be able to tell where the eye is and where the mouth is etc...

Then I tried a more complicated one based on an illustrative portrait by Kevin D. Sezgin
As an artist, you might look at these two examples and be discouraged at the possibilities of this tool. Both results look like mush. There seems to be no rhyme or reason for how the style elements are applied to the image. In some ways, style transfer just needs to gain more conceptual processing capabilities but, I want to give you a few tips on how to use arbitrary style transfer in a useful way as it exists today.

1. Choose Textural images rather than illustrative images for style reference

2. Give the AI a jump start by starting a bit of the process before feeding the image to the AI

3. Use the result as an underpainting rather than the final result.

let's see these principles applied
I will start by brushing over my photo with the smudge tool.

Feed it through with a textured image
Modify the result to accentuate focus points

Here is another example:

By using these three techniques I can get much better results.

An AI developer may say that I didn't use the best examples in the first attempts by choosing illustrative examples instead of using an abstractly textured image like a Van Gogh painting in my earlier examples. To that, I would say, "That type of example uses ambiguity to hide the shortcomings of style transfer and is not how most artists would need this tool."

Here is another method I tested for creating an underpainting using style transfer.

Create mattes

Style transfer appropriate textures onto mattes.

Apply multiple style transfers to the whole image and apply selectively.

Paint over the result.

Final Result

You need to understand both how a tool works and how it is broken to feed it the right ingredients for it to return the desired result. If it isn't working massage the material. Don't assume that AI tools will work intuitively out of the box. Many of these tools are in the research and development stage and need the experimentation of artists to discover best practices.