Perspective Correction for XR using UV hacks (virtual-production-ish stuff)

Here is the end result and you can read more if you think its interesting.

Why I find this interesting

Firstly, because of its relevance in things like virtual-production-like techniques used in filming The Mandalorian and secondly I think that perspective correction offers a zero barrier opportunity for a single viewer to experience mixed reality.

An idea of application

In my head I am picturing a gallery filled with mostly traditional art but when you walk by one frame it has a screen with a 3D scene portrayed inside it. As you walk by, the screen morphs to display the scene so that it looks approximately correct from the angle you are looking at it as though the screen were a window to another world rather than a flat image.

Some limitations to this method are:

- you can only have one primary viewer at a time (fine with the current social distancing)

- the lack of stereoscopy can be disorienting (the perspective correction is averaged between the eyes)

How to UV hack it

Here is a basic overview of how to achieve this using touch designer. I will not be going over every button click in detail just the main concepts.

You will need:

    an Xbox Kinect and adapter for PC

    TouchDesigner

    a human head (could be yours)

    a general understanding of how UVs work

Step 1: Setting up the virtual scene

Add a box with the dimensions of your screen.

Separate into two objects one with just the 'screen object' itself and one with the 'box back'.

The 'box back' will eventually be replaced with whatever you want to be viewed through the screen but for now it will be a stand-in for us to test whether perspective correction is working

Step 2: Calibration

Set up the kinect with touch designer and isolate the head and a hand input. The hand will be used for calibration.

Create a place holder object and input the hand location to it.

The object should mirror your movements in virtual space.

Place your hand on the corner of the computer screen and match transforms of the 'box back' and 'screen object' to it.

Then do the same for another corner.

Step 3: UV hacking

Once the screen is properly calibrated in the virtual world add a camera and reference your head location to the transform of the camera. Then put the 'screen object' as the look at object for the camera.

only include the 'box back' in the renderable objects. Do not include the screen

Then project UVs onto the screen object from the camera perspective

You will need to subdivide your 'screen object' before you do this because UV's are linearly interpreted between vertices.

Then set up another camera that is pointed directly at the 'screen object'

Then output that render.

And that's it.

Training a ML Network to recognize Lucky Charms using Synthetic Data in Blender

**This exploration was completed with the help of Immersive Limit's Blender Synthetics Course**

First I gathered a small data set to test against by photographing 3 different kinds of lucky charms marshmallows with my phone against a white background.

     

I split them into folders of their class and created a photoshop action that reduced their resolution to 224 and saved the image.

Then, I opened a new file in Blender to create the synthetic data.

First, I created a base mesh for the Love marshmallow.

Then, I created a geometry nodes modifier to subdivide and randomize the shape a bit so that each render's shape would be slightly different. The geometry nodes distort the shape in 2 ways: one large distortion on the base mesh to vary the shape of the whole and a second displacement to vary the subdivided surface to create texture.

One benefit of using geometry nodes is that, similar to materials, a set of geometry nodes can be applied to any number of objects, and modification of that set will be universally applied.

Next, I created a material to approximate the surface of the Marshmallow.

While not photoreal, I was intentional about including variation in the surface color and displacement that could easily be controlled by an external script.

I then created 2 more base meshes for Luck and BlueMoon and linked them to the same set of geometry nodes for shape variation.

I duplicated the material from Love and made a variation for the other two.

I then created a simple environment with a white surface and an area light. I also created a simple camera rig using nulls so that I could vary the angle for each shot.

I also parented all of the marshmallows to a null so that I could easily control their location and rotation.

I created a script that would render images into folders, modifying the scene each time.

The object variations are crucial to creating ML that can generalize.

I rendered 300 training images 80 validation images and 1 test image just to create the folders that I would populate with my photographs of real marshmallows to test against.

I then used my data set to retrain an ImageNet ML network using TensorFlow in Jupyter Notebooks.

I retrained the network using 7 epochs.

Once trained, I ran the test photographs through the network with a 100% success rate although I would need a larger sample size with more variation to really test its limits. For my first ML network, I am satisfied with what I have learned.