In my last few days before starting at Facebook I worked on another AR project, this time involving a more complex animated 3D model. My goal was to make an AR experience for this poster on my wall:
Here's a demo video of the final product.
Project Outline
- Use Photoshop to remove the fingers from the original image, leaving behind black holes where each of the fingers poke out. Later, this altered image can be placed over the original image in the AR effect to make space for the 3D fingers to poke through
- Build a rigged 3D model of the hand in Blender
- Animate the hand so the fingers repeatedly bend and unbend
- Load the animated model into Spark AR and set up a target tracker for the poster
1. Remove fingers with Photoshop
I'm very familiar with Photoshop, so this step didn't take long. I continued the pattern of pink and black lines over the fingers and rounded out the holes that each finger came out of. The original image wasn't very high quality, so there's some mismatch in the resolution of the lines I drew compared to the original lines. It's not too noticeable unless you zoom in pretty far, which would mean moving the camera extremely close to the target image in AR-- not too likely to happen. Here's the final result:
2. Rigged hand model with Blender
I have almost no Blender experience, so I watched a lot of tutorials for this step. This tutorial was extremely helpful for creating a stylized hand model with a skeleton. I modified it slightly to add fingernails onto the hand, since the original poster includes some bright blue nails that I wanted to match. I was surprised how easy it was to build a basic hand shape! I got a lot of practice with the extrude tool, as well as navigating around Blender's UI. The rigging was also straightforward, and I deviated from the tutorial again to make the finger joints a bit more lifelike. I watched another tutorial on texture painting to create a material for the hand. This step helped create a 2D colored skin for the hand that can be mapped onto the 3D model. For once, I'm glad that I took Computer Graphics in college, because knowing about UV mapping came in handy.
3. Animation
This step took the most time by far. I positioned the hand's fingers so that they lined up with the black holes on the altered poster image (see image below). I had to tweak the model a bit to get the fingers to be proportional to the ones in the poster. After that initial setup, I was able to make some keyframes to define the hand's movement. Keyframes are a standard method of animation where you define the start and end configuration of an object, and you interpolate between the two to create a smooth series of frames that transition from the start pose to the end. I defined one keyframe with all the fingers unbent, and one with them all bent in a claw handshape.
I iterated on this step a few times to try to improve the results. First, I actually animated from within Spark AR after skipping ahead to Step 4 to setup the target tracker. I initially tried using a simple looped animation to repeat a single transition for each finger bone. This had some limitations, since I could only animate from one position to another with no pause in between to have the hand stay clenched for a few seconds. Next, I switched to using a keyframe patch to handle more complex animation sequences. This worked, but Spark AR's keyframe editor UI was frustrating to work with, requiring a separate keyframe editor for each bone. Positioning a single finger meant I had to switch back and forth between editors and manually type in degrees of rotation for each bone. Not fun.
I moved to creating the animations directly in Blender and baking them into the exported 3D model. The Blender UI was much easier to use, and I could capture a keyframe for the hand's current pose all at once, rather than going bone by bone as with Spark AR. I created an even more complex animation here, where the fingers curled into the poster one by one, flexed open and closed, and then curled out in reverse order. Unfortunately, the model I worked with for that animation had a bug with the texture that I couldn't figure out. I ended up sticking with the simpler animation I made in Spark AR because I had had more than enough practice with keyframes for one weekend.
4. Put it all together in Spark AR
I created a target tracker for the original poster image, imported the Blender model, and set the animation to play on loop. Something that I didn't anticipate was that the sides of the palm and wrist would be visible when looking at the target from the side! I created an occluder box flush with the poster plane to hide anything behind or next to the poster, so that only the fingertips extending out of the image would be visible.
Demo
To try this AR camera effect out for yourself, scan this QR code with your phone camera (requires an Instagram account). You probably want to open this page on another device, like a laptop, so you can point your phone camera at the original poster below.
If all goes well, you should see something like this, with the fingers occasionally moving:
