Last summer, I was so tired of my client works indeed. So I decided to go back to my hometown and devote the whole summer to the music video for my friend producer, imai. I’ve made the video for ‘Eye’ from his electronic duo group_inou, which used StreetView and other creative coding techniques. Also, in this video, I introduced openFrameworks and ViveTracker to realize the smooth and free camera movement in stop motion. I would like to write the behind-the-scene down shortly.

Japanese Confectionery  —  Cultural Context

Some people in non-Asian countries might think, “what are these objects dancing on a table?”. The answer is this emoji:🍡. These are various types of rice cakes, which are especially popular with old people in Japan. I personally love their taste, minimalistic appearance, and softness. These have completely nothing to do with the lyrics — maybe I unconsciously wanted a kind of therapy by moving them.

The title of the track ‘Fly’ is often misheard as ‘fried’ by Japanese. Scene three rice cakes avoiding fried shrimp and then deep-fried was a kind of wordplay. In contrast, they finally flew in the air in the last scene.

Also, the entire sequence had shot at my grandparents’ house that I have nostalgic memories.

Visualizing Camera Trajectory with ViveTracker

I was wondering how to make the video look something new while using a traditional stop-motion technique. My attempt in that video was to introduce a shaking camera movement. I built a special shooting system like the one below to realize it.

I should have done with PC not to make the system such tangled.

The basic idea is:

1. Keep tracking camera position and angle with ViveTracker.
2. Every time a frame is captured, record a transformation of the camera at the moment.
3. Visualize a camera trajectory in 3D space using openFrameworks. (CamGraph.app)
4. By looking at the visualization, move the tripod by hand.

Me shooting the video with the system

Although it’s hard to explain even to people in the video industry, it’s NOT a motion control system. I’ve felt most of cameras works using some sliders or rigs are too smooth compared to the animation by hand. The key point of this system is that it doesn’t move a camera automatically but displays ‘how the camera has moved. Looking at the information, an animator has to move the tripod by themselves in the same method as other objects in a frame.

This system worked successfully beyond my expectations. I think the smooth but bit wiggled camera shaking really matched the groove of the track. I open-sourced the very tangled codes just in case.

Github : baku89/imai-fly

For more information:

• Dragonframe 4 — the most popular stop-motion software in the world
• SteamVR Tracking without an HMD by Luke Beno

Syncing with Music

I’m basically a motion designer fighting with fucking AfterEffects every day, but not a good stop-motion animator. So I somehow tried to integrate my way of thinking about motion with this physical technique. I never wanted to make a pre-vis on my computer because it forces me to behave as if ‘human-powered renderer’ in the real world. I tried the approach in this video with Hu Yu and Laurent Gray two years ago. However, it was so painful, and we were about to fall into depression, haha

In this project, I just drew a kind of graphic notation on my iPad Pro. Then I used it only as a hint for timings of motion so that I could leave enough space to decide how to move them, just like the straight-ahead method.

Some professionals use X-sheet feature on Dragonframe to do this, but I’m used to a horizontal timeline, so displayed it on my custom app.

In this several years, one of my theme as a video director is how I can extend the video production technique, like a lot of visual artists such as Tony Hill, Rybczynski, and Michel Gondry has done.

The point is that they merely show their unique and up-to-date processes inside their videos. Even if we know nothing about the behind-the-scenes, their videos still feel something new, yet emotional.

When I look at the recent art & tech scene, I occasionally feel there are so many projects that ended up with just a demo of cutting-edge technology. Such a video tends to begin with sequences such as calibrating projectors’ grids, and someone wiring lots of stepper motors with distracting technical descriptions.

In this project, I also wanted to show my attitude toward video production — designing a new workflow to create something new with whatever I can use, but no need to show its process inside the video. I hope everyone just enjoys the 🍡s’ dance along with imai’s emotional track, and only some geeks like you would understand what made the video feel something new technically.

I don’t know whether it worked, though; I would be glad if someone felt empathy for my stance.