StencilGL 1.0

glStencilFunc

Sets the stencil test: which fragments may be drawn, by comparing a reference value with the stencil buffer.

Signature

void glStencilFunc(GLenum func, GLint ref, GLuint mask);

How it works

The stencil buffer stores a small integer per pixel (8 bits, 0 to 255). With GL_STENCIL_TEST enabled, every fragment runs this test before it is drawn:

(ref & mask) func (stored & mask)

If the comparison is true the fragment goes on to the depth test; if not, it is thrown away. ref is also the value that GL_REPLACE writes (see glStencilOp).

The typical pattern has two passes: first draw an object with GL_ALWAYS and GL_REPLACE to paint 1s into the stencil buffer (a mask), then draw something else with GL_EQUAL or GL_NOTEQUAL to keep it inside or outside that mask.

Every fragment meets the stencil test first, then the depth test. glStencilFunc sets the first test; glStencilOp chooses what is written into the stencil buffer at each of the three exits: sfail, dpfail and dppass. A discarded fragment draws nothing, but its stencil action still runs.

Parameters

funcGLenum

The comparison between ref and the stored value.

Common values

GL_ALWAYS: Always pass (default). Used while writing a mask.
GL_EQUAL: Pass where the stored value equals ref: draw only inside the mask.
GL_NOTEQUAL: Pass where it differs from ref: draw only outside the mask (outlines).
GL_LESS / GL_LEQUAL / GL_GREATER / GL_GEQUAL: Compare ref with the stored value, e.g. GL_LESS passes when ref < stored.
GL_NEVER: Never pass.

refGLint

The reference value, 0 to 255 with an 8-bit stencil buffer.

maskGLuint

Bits compared: both sides are ANDed with it first. 0xFF compares all 8 bits.

Tips & pitfalls

Nothing happens until glEnable(GL_STENCIL_TEST), and the framebuffer needs a stencil buffer (GLFW gives 8 bits by default; an FBO needs a GL_DEPTH24_STENCIL8 attachment).
This mask only affects the test. The bits that get written are chosen by glStencilMask.

Errors

When the call is invalid it does nothing and sets an error, which glGetError returns:

GL_INVALID_ENUM: func is not one of the eight comparisons.

Example

example.cpp

1// pass 1: paint the object's pixels with 1
2glStencilFunc(GL_ALWAYS, 1, 0xFF);
3glStencilMask(0xFF);
4drawObject();
5
6// pass 2: a bigger copy only where the stencil is NOT 1 -> outline
7glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
8glStencilMask(0x00);
9drawObjectScaledUp();

Official Khronos documentation