Actions with deformable skinned meshes

In this tutorial we will demonstrate the complete pipeline on creating an Action with a deformable skinned mesh. From the 3D model to the unity importing then all the configurations and finally the Action script. We will implement a medical example, in particular the initial incision from the Total Knee Arthroplasty operation.

Tutorial overview:

1. Generate the animated 3D model

2. Import the 3D model to Unreal

3. Split the animation into smaller animation assets

4. Animator setup

5. Configure the CharacterController

6. Generate the Action Actors

7. Implement the Action Blueprint

Generate the animated 3D model

In this scenario we will use an animated leg, with various baked skinned deformations. The image below shows our 3D model in Maya.

As you can see, in Maya we design the full animation using standard joints and key frames. The animation process depends on the scenario you want to design. In this case, we rigged the right foot adding joints to cut the skin and the different muscle layers until we have a clear view of the knee.

The image below shows the joints we used for the skin animation along with their keyframes.

Now we have to export the 3D model. We use the FBX format to export our 3D models since it is the most reliable format to work with unreal.

From the Maya menu bar navigate to File/Export all. Now you need to configure some options first. Make sure the Animation option is checked to export the animation as well. Make sure the Files of type option is set to FBX export.

There are specific situations where you need to bake the animation instead of exporting it without the bake option. If the animation has complex animation behaviours or blends there is a chance that they will not import right in Unreal. If you face some issues on the importing and the animation does not appear to work right check the Bake Animation option at the export window.

Warning

If you bake the animation, Maya will generate key frames for all joints. After that, the modifying the animation would be really inconvenient. Remember to keep the Maya binary file to modify the animation when baking.

Import the 3D model to Unreal

To import the model into Unreal just drag and drop the FBX to your project. Depending on the import settings you choose different .uasset files will be created. These can be:

1. Skeleton

2. Mesh

3. Animations

4. Materials

This is the final result

Split the animation into clips

We will split the animation into smaller clips to feed our CharacterController, a class that controls our patient’s animations.

To split the animation into smaller clips, select the imported animation and right click it, then select duplicate.

Open the duplicated animation and then move the key frame cursor (the red bar) to the last frame of this animation segment. Right click and select remove key frames from :code:’XX frame’ to :code:’XX frame’ (the last frame of the whole animation). Remove in a similar way the starting redundant frames.

Finally, save your changes and rename the animation asset.

Repeat this process for all your animation segments.

Below you can see the clips we made for this operation. In this tutorial we will use Cut1, Cut2, Cut3, Cut4.

Character Controller Setup

To properly execute the animations and manage their transitions we have to setup an Animator for our model.

From the Unreal scene, navigate to your Actor (in this case the leg), then select Add Component and add a CharacterAnimationController. Here you need to configure your own Animation to match your needs.

The next step is to add the animations in our CharacterAnimationController. In the Animations array of the CharacterAnimationController component add all the animation assets you created in the previous step.

Below you can see our CharacterAnimationController.

Generate the Action actors

As we mentioned, the MAGES metaphor of the skin incision will be a UseAction. We use the UseAction in cases where we need to grab a certain interactable object (pliers, scalpel etc) to complete an objective by “touching” the tool on a predefined area.

For the UseAction we need to prepare the following prefabs

1. Use Colliders

2. The actual interactable (a scalpel)

Use Collider

To generate the use collider navigate to MAGES/Create Prefab/Use Collider and the use collider actor template will appear in the world outliner.

The use colliders are waiting for the use actor to trigger them. Once all of the colliders are triggered the Action will perform. In this scenario we will use four colliders since we have four cut animations. The image below shows our actor with the four colliders components.

Now we need to assign that only the scalpel will trigger these colliders. To do that, navigate to the UseColliderPrefabConstructor add a new Actor element at the Prefabs Used List, then select the scalpel (image below)

Finally, we need to add the animations that will be played with each box. These animations are added in the UseColliderPrefabConstructor component in the field, Animation Names.

Use Interactable Actor

The next step is to configure the Interactable actor that we will use for this scenario, the scalpel. We need to create a Tool Actor. A tool actor is the same as an interactable actor with an extra component, GestureHands.

This component allows us to enhance its interacting behaviour, you can read more about it here.

Implement the Action Blueprint

The final step is to write the Action script. Below you can see the basic ToolAction script that describes this ToolAction.

Lets explain the Action a bit. You can see that we call the SetToolActionPrefab method to instantiate our tool colliders and set the actor that we will use to trigger them. In addition two functions are called, the Bind Character Animation to Perform and :code:`Bind Character Animation on Undo which take the actor that has the CharacterAnimationController component (the patient) and an animation sequence as arguments. The first function will add an animation to be played once the user performs this action, while the latter will play an animation on undo. These functions are useful in case the user skips or undoes the action.

Note

To perform the ToolAction, the user needs to hit all the registered tool colliders.