Nexus/2AD Running on a custom metahuman character show casing generated lip-sync and animation

Unity based demo of Nexus/2AD showcasing how characters can be driven

About

Nexus/2AD is a generative animation engine. It takes user created animation and pieces it together to create almost infinite variations. It uses different user defined parameters to intelligently run and animate characters with context. Nexus/2AD works best with generative AI characters that can plug generated text into Nexus/2AD’s lip-sync and automatically create matching animation.

Project Info

Role: Programmer

Team Size: 20

Time Frame: 3 Years

Engine: Unity, Unreal Engine 4, Unreal Engine 5

Overview

I worked on both the design and implementation of the core engine of Nexus/2AD, as well as the implementation and toolsets for each engine including Unity, UE4, and UE5.

The core system of Nexus/2AD is based around generating comprehensive timelines that contain animation, audio, lip-sync and custom user actions that drive movement of a character inside the engine.

Using these timelines and building on top of the animation systems of each engine I worked to create plugins that would send data to the core system to create timelines for characters that then would run animation for said character. This gave me a lot of insight into the workings of Unity and Unreal’s animation systems.

I also built a series of tools that allowed user customization and ease of use with our plugin these included a custom timeline builder as well as a character importer which would take any model and skeleton and help set the character up as a Nexus/2AD character.

With these tools I also worked on a few projects that leveraged generative character AI that would create speech at runtime to plugin to our system, giving a character to the chatbot behind it.