GLSL Programming/Unity/Nonlinear Deformations

Illustration of deformations by corsets. These are examples of nonlinear deformations that cannot be modeled by a linear transformation.

This tutorial introduces vertex blending as an example of a nonlinear deformation. The main application is actually the rendering of skinned meshes.

While this tutorial is not based on any other specific tutorial, a good understanding of Section “Vertex Transformations” is very useful.

Blending between Two Model Transformations

Most deformations of meshes cannot be modeled by the affine transformations with 4×4 matrices that are discussed in Section “Vertex Transformations”. The deformation of bodies by tight corsets is just one example. A more important example in computer graphics is the deformation of meshes when joints are bent, e.g. elbows or knees.

This tutorial introduces vertex blending to implement some of these deformations. The basic idea is to apply multiple model transformations in the vertex shader (in this tutorial we use only two model transformations) and then blend the transformed vertices, i.e. compute a weighted average of them with weights that have to be specified for each vertex. For example, the deformation of the skin near a joint of a skeleton is mainly influenced by the position and orientation of the two (rigid) bones meeting in the joint. Thus, the positions and orientations of the two bones define two affine transformations. Different points on the skin are influenced differently by the two bones: points at the joint might be influenced equally by the two bones while points farther from the joint around one bone are more strongly influenced by that bone than the other. These different strengths of the influence of the two bones can be implemented by using different weights in the weighted average of the two transformations.

For the purpose of this tutorial, we use two uniform transformations `mat4 _Trafo0` and `mat4 _Trafo1`, which are specified by the user. To this end a small JavaScript (which should be attached to the mesh that should be deformed) allows us to specify two other game objects and copies their model transformations to the uniforms of the shader:

```@script ExecuteInEditMode()

public var bone0 : GameObject;
public var bone1 : GameObject;

function Update ()
{
if (null != bone0)
{
renderer.sharedMaterial.SetMatrix("_Trafo0",
bone0.renderer.localToWorldMatrix);
}
if (null != bone1)
{
renderer.sharedMaterial.SetMatrix("_Trafo1",
bone1.renderer.localToWorldMatrix);
}
if (null != bone0 && null != bone1)
{
transform.position = 0.5 * (bone0.transform.position
+ bone1.transform.position);
transform.rotation = bone0.transform.rotation;
}
}
```

The two other game objects could be anything — I like cubes with one of the built-in semitransparent shaders such that their position and orientation is visible but they don't occlude the deformed mesh.

In this tutorial, the weight for the blending with the transformation `_Trafo0` is set to `gl_Vertex.z + 0.5`:

`float weight0 = gl_Vertex.z + 0.5;`

and the other weight is `1.0 - weight0`. Thus, the part with positive `gl_Vertex.z` coordinates is influenced more by `_Trafo0` and the other part is influenced more by `_Trafo1`. In general, the weights are application dependent and the user should be allowed to specify weights for each vertex.

The application of the two transformations and the weighted average can be written this way:

`vec4 blendedVertex = weight0 * (_Trafo0 * gl_Vertex) + (1.0 - weight0) * (_Trafo1 * gl_Vertex);`

Then the blended vertex has to be multiplied with the view matrix and the projection matrix. The view transformation is not available directly but it can be computed by multiplying the model-view matrix (which is the product of the view matrix and the model matrix) with the inverse model matrix (which is available as `_World2Object` times `unity_Scale.w` except for the bottom-right element, which is 1):

```      mat4 modelMatrixInverse = _World2Object * unity_Scale.w;
modelMatrixInverse[3][3] = 1.0;
mat4 viewMatrix = gl_ModelViewMatrix * modelMatrixInverse;

gl_Position = gl_ProjectionMatrix * viewMatrix * blendedVertex;
```

In order to illustrate the different weights, we visualize `weight0` by the red component and `1.0 - weight0` by the green component of a color (which is set in the fragment shader):

`color = vec4(weight0, 1.0 - weight0, 0.0, 1.0);`

For an actual application, we could also transform the normal vector by the two corresponding transposed inverse model transformations and perform per-pixel lighting in the fragment shader.

All in all, the shader code looks like this:

```Shader "GLSL shader for vertex blending" {
Pass {
GLSLPROGRAM

// Uniforms set by a script
uniform mat4 _Trafo0; // model transformation of bone0
uniform mat4 _Trafo1; // model transformation of bone1

// The following built-in uniforms
// are also defined in "UnityCG.glslinc",
// i.e. one could #include "UnityCG.glslinc"
uniform mat4 _Object2World; // model matrix
uniform vec4 unity_Scale; // w = 1/scale
uniform mat4 _World2Object; // inverse model matrix
// all but the bottom-right element should be
// multiplied with unity_Scale.w

// Varyings
varying vec4 color;

#ifdef VERTEX

void main()
{
float weight0 = gl_Vertex.z + 0.5; // depends on the mesh

vec4 blendedVertex = weight0 * (_Trafo0 * gl_Vertex)
+ (1.0 - weight0) * (_Trafo1 * gl_Vertex);

mat4 modelMatrixInverse = _World2Object * unity_Scale.w;
modelMatrixInverse[3][3] = 1.0;
mat4 viewMatrix = gl_ModelViewMatrix * modelMatrixInverse;

gl_Position =
gl_ProjectionMatrix * viewMatrix * blendedVertex;

color = vec4(weight0, 1.0 - weight0, 0.0, 1.0);
// visualize weight0 as red and weight1 as green
}

#endif

#ifdef FRAGMENT

void main()
{
gl_FragColor = color;
}

#endif

ENDGLSL
}
}
}
```

This is, of course, only an illustration of the concept but it can already be used for some interesting nonlinear deformations such as twists around the $z$ axis.

For skinned meshes in skeletal animation, many more bones (i.e. model transformations) are necessary and each vertex has to specify which bone (using, for example, an index) contributes with which weight to the weighted average. However, Unity computes the blending of vertices in software; thus, this topic is less relevant for Unity programmers.

Summary

Congratulations, you have reached the end of another tutorial. We have seen:

• How to blend vertices that are transformed by two model matrices.
• How this technique can be used for nonlinear transformations and skinned meshes.