Last summer, I had been so tired of my client works indeed. So I decided to go back to my hometown and devote the whole summer to the music video for my friend producer, imai. I’ve made the video for ‘Eye’ from his electronic duo group_inou, which used StreetView and other lots of creative coding techniques. Also in this video, I introduced openFrameworks and ViveTracker to realize the smooth and free camera movement in a stop motion. I would like to write the behind-the-scene down shortly.
Japanese Confectionery — Cultural Context
Some people in non-Asian countries might think “what are these objects dancing on a table?”. The answer is this emoji:🍡. These are various types of rice cakes, which especially popular with old people in Japan. I personally love their taste, minimalistic appearance, and the softness. These have completely nothing to do with the lyrics — maybe I unconsciously wanted a kind of therapy by moving them haha
The title of the track ‘Fly’ is often misheard as ‘fried’ by Japanese. The scene three rice cakes avoiding from fried-shrimp and then deep-fried was a kind of wordplay. In contrast, they finally flew in the air in the last scene.
Also, the entire sequence had shot at my grandparents’ house that I have nostalgic memories.
Visualizing Camera Trajectory with ViveTracker
I was wondering how to make the video look something new while using traditional stop-motion technique. My attempt in that video was to introduce a shaking camera movement. I built a special shooting system like below to realize it.
The basic idea is:
- Keep tracking camera position and angle with ViveTracker.
- Every time a frame is captured, record a transformation of the camera at the moment.
- Visualize a camera trajectory in 3D space using openFrameworks. (CamGraph.app)
- By looking at the visualization, move the tripod by hand.
Although it’s hard to explain even to people in the video industry, it’s NOT a motion control system. I’ve felt most of cameras works using some sliders or rigs are too smooth compared to the animation by hand. The key point of this system is that it doesn’t move a camera automatically but just displays ‘how the camera has moved’. Looking at the information, an animator has to move the tripod by themselves as the same method as other objects in a frame.
This system worked successfully beyond my expectation. I think the smooth but bit wiggled camera shaking really matched with a groove of the track. I open-sourced the very tangled codes just in case.
For more information:
- Dragonframe 4 — the most popular stop-motion software in the world
- SteamVR Tracking without an HMD by Luke Beno
Syncing with Music
I’m basically a motion designer fighting with fucking AfterEffects every day, but not a good stop-motion animator. So I somehow tried to integrate my way of thinking for motion with this physical technique. I never wanted to make a pre-vis on my computer because it forces me to behave as if ‘human-powered renderer’ in a real world. I’ve tried the approach in this video with Hu Yu and Laurent Gray two years ago. However, it was so painful and we were about to fall depression haha
In this project, I just drew a kind of graphic notation on my iPad Pro. Then I used it only as a hint for timings of motion so that I can leave enough spaces to decide how to move them just like the straight-ahead method.
Some professionals use ‘X-sheet’ feature on Dragonframe to do this, but I’m used to a horizontal timeline so displayed it on my custom app.
In this several years, one of my theme as a video director is how I can extend the video production technique, like a lot of visual artists such as Tony Hill, Rybczynski, and Michel Gondry has done.
The point is that they merely show their unique and up-to-date processes inside their videos. Even if we know nothing about the behind-the-scenes, their videos still feel something new, yet emotional.
When I look at the recent art & tech scene, I occasionally feel there are so many projects ended up with just a demo of the cutting-edge technology. Such a video tend to begin with sequences such as calibrating projectors’ grid, someone wiring lots of stepper motors with distracting technical descriptions.
In this project, I also wanted to show my attitude for video production — designing new workflow to create something new with whatever I can use, but no need to show its process inside the video. I hope people to just enjoy the 🍡s’ dance along with imai’s emotional track, and only some geeks like you would understand what made the video feel something new technically.
I don’t know whether it worked, though, I would be grad if someone feels empathy for my stance.