#508 – AI + Animation = ?

Animation really is the forefather of cinema.

It was around before, it's around now and it will be around till the end.

In this Issue we'll get an inside look on where the next generation of creative tools are taking this century old art form, when AI meets animation, thanks to a collaboration between a team of Pixar alumni and Google DeepMind engineers.

Director Connie He spent over a year leading a team of 45 artists to create the animated short, Dear Upstairs Neighbours, which previewed at the Sundance Film Festival this year.

What's interesting to me, when you look past the hot-take tweets and I-only-read-the-headline-opinion-pieces on this film's production process, is how even with direct access to some of the industries leading AI engineers backed up by technically astute animators – AI still isn't able to one-shot it's way to the screen.

Even with the control provided by fine-tuning and video-to-video workflows, none of our final shots were created in a single “one-click” generation.

— Google DeepMind BTS

Dear Upstairs Neighours has a unique painterly look inspired by Connie He's Father's own paintings – a style that would have been impossible for the team to have created consistently by themselves so the tight collaboration with Google was essential to achieve the director's vision for the film.

Google DeepMind has shared a really insightful behind the scenes article, which is well worth a read, for a ton of details on the film's production process, including where all the work went into using AI in the process:

• Text-to-video with a reference image didn't allow for control over the motion of the characters, shot or objects so a video-to-video workflow was required.
• The team fed 3D animation into a custom Veo model fine-tuned on their concept art, character sketches and painterly references to re-skin it with the final look.
• This alone wasn't enough as every shot required several iterations including building custom tools to allow for changes to be made to specific areas in a shot without re-generating the whole thing.
• 4K upscaling was needed to prepare the film's final deliverables, which Google's engineers customised to the filmmakers tastes.

Guided by our artists' critique, our researchers carefully tuned the [4K upscaling] model's behavior to add rich detail that preserved every nuance of the artistic style. The Veo 4K upscaling model is available in Flow and coming to Google AI Studio and Vertex AI later this month to meet the real-world needs of filmmakers.

— Google DeepMind BTS

To me this is an encouraging example of where the technology really is a creative tool in the hands of the artists. And an encouragement that AI isn't ‘there yet’ when it comes to prompt-to-printing your way to artistically substantive shots.

But I wanted to know more, so in this Issue we'll get to hear from the film's editor, Sarah Affleck, on what it's like to edit a short film like Dear Upstairs Neighbours.

Take This Further

• #501 – Is the Future of Cinema in the hands of AI?
• #487 – AI Hype Vs Reality
• #447 – Workflow Tips for AI-Generated Video
• #418 - AI hasn't slowed down.
• #397 - AI Video Editing Workflows