Initial avatar design_substance painter

Final version

Cloth design_marvlous designer

blender_rigging with metahuman's skeleton

weight paint in Unreal

battlefiled scene demo

embedded ECV

Motion Captrure
‍
Our motion capture was conducted at NYU Tandon IDM R Lab. We used Optitrack to capture the motions and then imported the FBX files into Unreal Engine using IK Retargeting for Metahumans. To facilitate the retargeting process, we matched the motion capture files to Metahuman's A pose in Maya for precision. ( we used Python scripts and XML to quickly characterize Metahumans in Maya. )

Here is the link to the scripts: https://github.com/kromond/OptiSkelForUEScripts

OptiSkelForUEScripts

In motion capture, glitches are inevitable. To address this, we directly adjusted the movement of damaged bones in Unreal.

We use sequencer to manually adjust the keyframes of the bones for repair. Previously, we used motion builder for this purpose, but due to the minor glitches and for better visualizing the process, we chose to complete the process in Unreal.

Scene design
‍
We created storyboards and animatics for our film, which takes place on a battlefield where the avatar expresses the intensity of the conflict through dance. After the avatar dies on the battlefield, they enter the Six Realms of Reincarnation before achieving Nirvana and rebirth.
Based on these storyboards, we designed multiple scenes inspired by Buddhist art and Robert Wilson's stage art, utilizing abstract spaces and highly saturated color blocks. The lighting design for each scene draws from various stage lighting effects to create a dramatic atmosphere.

Stable diffusion

We utilized stable diffusion and compositing to transition the avatar from a volumetric form to a 3D mesh. This section was completed by Derek Chan.

We rendered the scenes and characters as EXR files in Unreal, and then utilized AI and After Effects for composition. (this scene is from Unreal)

VFX

The VFX in the film, such as the glowing magic symbol and volumetric clouds, were created in unreal. We sourced Niagara for blood splatter effects online. The color gradient effects in the scenes were achieved through the Sky Sphere's material. Additionally, we adjusted many of the post-processing settings to achieve the final scene effects.

symbol material

symbol niagara

sky shpere

before using post-processing

after using post-processing

Cinematography

Camera Operate & Sequnencer
To better illustrate our storyboards and animatics, we created camera movement prototypes based on existing assets. The incomplete assets were replaced by Moodboards. Here is a list of some of the camera movement prototypes:

Most of the camera work and editing for the film were done in Sequencer. To achieve a more realistic camera movement, we utilized virtual camera to capture certain scenes.

camera setting

virtual camera

What I leart from this whole process......

-Understanding the virtual production tools and workflows in Unreal Engine 5 and their capabilities and limitations.
-Learning the technical aspects of virtual production, such as motion capture, character rigging, animation, lighting, and compositing, and how they work together in a real-world production.
-Optimizing workflows and pipelines to improve efficiency, reduce production time, and minimize errors.
-Collaborating with a team of artists, dancer, and sound designer, and effectively communicating to achieve project goals.