Doug Meet Zoey – fxguide
Earlier today at the FMX conference in Germany, Digital Domain, introduced for the first time, “Zoey,” their most advanced autonomous human yet, and were joined by fxguide’s Mike Seymour for a Q&A.
Digital Domain are leaders in digital human technology and an Academy Award-winning VFX studio. Zoey builds on the technology first seen with Digital Doug, a virtual version of Doug Roble(see our story here from 2018. For decades now, Digital Domain has done pioneering work with digital humans that included groundbreaking VFX work and iconic film characters such as Benjamin Button and Thanos. Building on that work, Digital Domain’s Digital Human Group started work on producing photorealistic virtual beings, powered by AI. These virtual assistants started as puppetted avatars and then progressed to AI digital agents such a range of cutting-edge technology. While still a proof-of-concept, Zoey can hold conversations, answer questions, and more, all while running on a custom server via the Unreal Engine.
“With the potential of the metaverse and continued advancements in AI, the desire to interact with autonomous humans face-to-face is becoming more important and part of modern life,” said Daniel Seah, Global CEO of Digital Domain. “For decades, Digital Domain has been pioneering digital human technology, and Zoey takes the concepts of virtual assistants like Alexa and Siri several steps further, creating a digital helper you can truly interact with.”
At FMX Melissa Cell, Technology Supervisor, and Visual Effects Supervisor, Matthias Wittmann discussed their approach of modularizing different technologies to design characters that feel compelling in appearance, voice, expression, mannerisms, and behavior, all in order to ultimately breathe life into modern chatbots. Melissa Cell has worked at Digital Domain for 9 years as a Technical Director and Software Engineer. In 2018 she joined Digital Domain’s Digital Human Group to develop realistic digital humans for visual effects, video games, live events, and other real-time applications. Matthias Wittmann is a recognized digital human pioneer, having won a Visual Effects Society award for “Outstanding Animated Character in a Live Action Motion Picture” for “The Curious Case of Benjamin Button,” a Grand Clio for “Virtual 2Pac – Coachella,” and a VES nomination for the character Thistlewit in “Maleficent.” Since 2015 Matthias has specialized in developing interactive virtual humans.
Building on their first proof-of-concept autonomous human, “Douglas,” the new Zoey is the result of years of research and development, spearheaded by its internal Digital Humans Group. The physical appearance of Zoey is based on actress Zoey Moses, who worked with Digital Domain to create a comprehensive set of facial movements and mannerisms, as well as a full range of emotive expressions.
Using that data and its proprietary facial animation tool Charlatan. Digital Domain has been working with neural rendering and AI machine learning for some time having done projects such as David Beckham aging (see our fxguide story from 2020) and similar special projects. Free Guy (2021) represented the first time Digital Domain has used its Charlatan program in a feature film.
To give Zoey her personality, Digital Domain used multiple forms of machine learning to ensure she could understand most questions and formulate proper replies, and then access the internet (or stored data) for its answer. But rather than simply give a verbal or text response, As the team demonstrated during their talk, Zoey can fully emote. If a question is confusing, she will look perplexed; if she is told a joke, she’ll smile. Throughout a conversation, Zoey will also move and fidget in accordance with her answers, and even get annoyed when interrupted. She can engage with participants, asking them questions and opinions, and facial recognition software lets her recognize and remember people. Chatbots can also be added, giving Zoey the potential to quickly assume the role of an established virtual assistant for deployment in hospitality or the service industry.
To provide Zoey with a robust and flexible voice system capable of more than just reciting scripted answers, Digital Domain employed AI-powered text-to-speech technology from WellSaid Labs. Using WellSaid Labs’ tools, Zoey can access a vast vocabulary of words and phrases, ensuring that no answer is beyond its capacity to vocalize. There are also multiple options to control the tone of her replies, including different levels of enthusiasm to reflect the goal of the interactions. Zoey also has the ability to expand her speech capabilities by incorporating multiple language packs, making her fully bilingual.
Zoey uses speech to text, connecting with Google’s Chatbot Dialogflow. The Dialogflow software is a natural language understanding platform used to design and integrate a conversational user interface into mobile apps, web applications, etc and it is part of the design team’s approach of making a modular system that can use the best APIs and advances from not only Digital Domains own internal teams but from anywhere across the internet.
Once Zoey is trained or “educated” and ready to interact with the real world, she can then be added to multiple platforms, including custom systems designed by potential users, or game engines like Unity and Epic Games’ software. For today’s introductory presentation at FMX, Zoey can be seen running in real-time on Epic Games’ UE4.
“For the last 30 years, Digital Domain has been one of the world leaders behind some of the most sophisticated and memorable visual effects ever seen, so it was a natural progression for us to become a leader in digital and virtual humans and to create ‘Zoey,’ the world’s most advanced autonomous human,” said John Fragomeni, global president of Digital Domain. “We will continue to push the boundaries of visual effects in all fields and on any screen, as we look for ways to offer people a better experience, regardless of the medium or delivery platform.”
Zoey will be available to license from Digital Domain in the near future. The FMX talks such as this are available online until the end of May at FMX as video on demand.