GLSL Programming/GLUT/Cutaways

This tutorial covers discarding fragments, determining whether the front face or back face is rendered, and front-face culling. This tutorial assumes that you are familiar with varying variables as discussed in the tutorial on a RGB cube.

The main theme of this tutorial is to cut away triangles or fragments even though they are part of a mesh that is being rendered. The main two reasons are: we want to look through a triangle or fragment (as in the case of the roof in the drawing to the left, which is only partly cut away) or we know that a triangle isn't visible anyways; thus, we can save some performance by not processing it. OpenGL supports these situations in several ways; we will discuss two of them.

Very Cheap Cutaways

The following shader is a very cheap way of cutting away parts of a mesh: all fragments are cut away that have a positive $y$ coordinate in object coordinates (i.e. in the coordinate system in which it was modeled; see “Vertex Transformations” for details about coordinate systems). Here is the vertex shader:

attribute vec4 v_coord;
uniform mat4 m, v, p;
varying vec4 position_in_object_coordinates;

void main()
{
  mat4 mvp = p*v*m;
  position_in_object_coordinates = v_coord;
  gl_Position = mvp * v_coord;
}

And here is the fragment shader:

varying vec4 position_in_object_coordinates;

void main()
{
  if (position_in_object_coordinates.y > 0.0) 
    {
      discard; // stop processing the fragment if y coordinate is positive
    }
  if (gl_FrontFacing) // are we looking at a front face?
    {
      gl_FragColor = vec4(0.0, 1.0, 0.0, 1.0); // yes: green
    }
  else
    {
      gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0); // no: red
    }
}

When you apply this shader to any of the default objects, the shader will cut away half of them. This is a very cheap way of producing hemispheres or open cylinders. If you enable back-face culling:

    glEnable(GL_CULL_FACE);

you will also notice that the “inside” of the object is invisible. We discuss back-face culling further below.

Discarding Fragments

Let's first focus on the discard instruction in the fragment shader. This instruction basically just discards the processed fragment. (This was called a fragment “kill” in earlier shading languages; I can understand that the fragments prefer the term “discard”.) Depending on the hardware, this can be a quite expensive technique in the sense that rendering might perform considerably worse as soon as there is one shader that includes a discard instruction (regardless of how many fragments are actually discarded, just the presence of the instruction may result in the deactivation of some important optimizations). Therefore, you should avoid this instruction whenever possible but in particular when you run into performance problems.

One more note: the condition for the fragment discard includes only an object coordinate. The consequence is that you can rotate and move the object in any way and the cutaway part will always rotate and move with the object. You might want to check what cutting in view or world space looks like: change the vertex and fragment shader such that the view coordinate $y$ or world coordinate $y$ is used in the condition for the fragment discard. Tip: see the tutorial on shading in view space for how to transform the vertex into world space.

Better Cutaways

You might try the following idea to improve the shader: change it such that fragments are discarded if the $y$ coordinate is greater than some threshold variable. Then introduce a uniform to allow the user to control this threshold. Tip: see the tutorial on shading in view space for a discussion of uniforms.

Distinguishing between Front and Back Faces

As demonstrated by the shader, a special boolean variable gl_FrontFacing is available in the fragment shader that specifies whether we are looking at the front face of a triangle. Usually, the front faces are facing the outside of a mesh and the back faces the inside. (Just as the surface normal vector usually points to the outside.) However, the actual way front and back faces are distinguished is the order of the vertices in a triangle: if the camera sees the vertices of a triangle in counter-clockwise order, it sees the front face. If it sees the vertices in clockwise order, it sees the back face.

Our fragment shader checks the variable gl_FrontFacing and assigns green to the output fragment color if gl_FrontFacing is true (i.e. the fragment is part of a front-facing triangle; i.e. it is facing the outside), and red if gl_FrontFacing is false (i.e. the fragment is part of a back-facing triangle; i.e. it is facing the inside). In fact, gl_FrontFacing allows you not only to render the two faces of a surfaces with different colors but with completely different styles.

Note that basing the definition of front and back faces on the order of vertices in a triangle can cause problems when vertices are mirrored, i.e. scaled with a negative factor. We can try to turn the inside out by multiplying one (or three) of the coordinates with -1, e.g. by assigning gl_Position this way in the vertex shader:

  gl_Position = mvp * vec4(-v_coord.x, v_coord.y, v_coord.z, 1.0);

This just multiplies the $x$ coordinate by -1. For a sphere, you might think that nothing happens, but it actually turns front faces into back faces and vice versa; thus, now the inside is green and the outside is red. (By the way, this problem also affects the surface normal vector.) Thus, be careful with mirrors!

Culling of Front or Back Faces

We saw that we can use glEnable(GL_CULL_FACE) to turn on any triangle culling. By default back faces are culled away as if the line glFrontFace(GL_CCW) was specified. You can also specify the culling of front faces with glFrontFace(GL_CW). You can enable culling of back-facing, because the inside of objects is usually invisible; thus, back-face culling can save quite some performance by avoiding to rasterize these triangles as explained next. Of course, we were able to see the inside with our shader because we have discarded some fragments; thus, we have to deactivate back-face culling in that case.

How does culling work? Triangles and vertices are processed as usual. However, after the viewport transformation of the vertices to screen coordinates (see “Vertex Transformations”) the graphics processor determines whether the vertices of a triangle appear in counter-clockwise order or in clockwise order on the screen. Based on this test, each triangle is considered a front-facing or a back-facing triangle. If it is front-facing and culling is activated for front-facing triangles, it will be discarded, i.e., the processing of it stops and it is not rasterized. Analogously, if it is back-facing and culling is activated for back-facing triangles. Otherwise, the triangle will be processed as usual.

Summary

Congratulations, you have worked through another tutorial. (If you have tried one of the assignments: good job! I didn't yet.) We have looked at:

How to discard fragments.
How to activate the culling of back faces.
How to render front-facing and back-facing triangles in different colors.