When drawing transparent surfaces, `rgl`

tries to sort
objects from back to front to get better rendering of transparency.
However, it doesn’t sort each pixel separately, so some pixels end up
drawn in the incorrect order. This note describes the consequences of
that error, and suggests remedies.

We’ll assume that the standard
`glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)`

blending
is used. That is, when drawing with transparency \(\alpha\), the new colour is mixed at
proportion \(\alpha\) with the colour
previously drawn (which gets weight \(1-\alpha\).)

This note is concerned with what happens when two transparent objects are drawn at the same location. We suppose the further one has transparency \(\alpha_1\), and the closer one has transparency \(\alpha_2\). If they are drawn in the correct order (further first), the three colours (background, further object, closer object) should end up mixed in proportions \(C = [(1-\alpha_1)(1-\alpha_2), \alpha_1(1-\alpha_2), \alpha_2]\) respectively.

If the objects are drawn in the wrong order, the actual proportions of each colour will be \(N = [(1-\alpha_1)(1-\alpha_2), \alpha_1, \alpha_2(1-\alpha_1)]\) if no masking occurs.

`rgl`

currently defaults to depth masking using
`glDepthMask(GL_TRUE)`

. This means that depths are saved when
the objects are drawn, and if an attempt is made to draw the further
object after the closer one (i.e. as here), the further object will be
culled, and the proportions will be \(M =
[(1-\alpha_2), 0, \alpha_2]\).

The question is: which is better, `glDepthMask(GL_TRUE)`

or `glDepthMask(GL_FALSE)`

? One way to measure this is to
measure the distance between \(C\) and
the incorrect proportions. (This is unlikely to match perceptual
distance, which will depend on the colours as well, but we need
something. Some qualitative comments below.)

So we have

\[|C-N|^2 = 2\alpha_1^2\alpha_2^2\],

and

\[|C-M|^2 = 2\alpha_1^2(1-\alpha_2)^2\].

Thus the error is larger with \(N\) when \(\alpha_2 > 1/2\), and larger with \(M\) when \(\alpha_2 < 1/2\). The value of \(\alpha_1\) doesn’t affect the preference, though small values of \(\alpha_1\) will be associated with smaller errors.

Depending on the colours of the background and the two objects, this recommendation could be modified. For example, if the two objects are the same colour (or very close), it doesn’t really matter how the 2nd and 3rd proportions are divided up, and \(N\) will be best because it gets the background proportion exactly right.

Typically in `rgl`

we don’t know which object will be
closer and which one will be further, so we can’t base our choice on a
single \(\alpha_i\). The recommendation
would be to use all small levels of `alpha`

and disable
masking, or use all large values of `alpha`

and retain
masking.

The classic example of an impossible to sort scene involves three triangles arranged cyclicly so each one is behind one and in front of one of the others (based on https://paroj.github.io/gltut/Positioning/Tut05%20Overlap%20and%20Depth%20Buffering.html).

```
<- 2*pi*c(0:2, 4:6, 8:10)/12
theta <- cos(theta)
x <- sin(theta)
y <- rep(c(0,0,1), 3)
z <- cbind(x, y, z)
xyz <- xyz[c(1,2,6, 4,5,9, 7,8,3),]
xyz open3d()
```

```
## null
## 17
```

```
par3d(userMatrix = M)
triangles3d(xyz, col = rep(c("red", "green", "blue"), each = 3))
```

To see the effect of the calculations above, consider the following four displays.

`open3d()`

```
## null
## 18
```

```
par3d(userMatrix = M)
layout3d(matrix(1:9, ncol = 3, byrow=TRUE),
widths = c(1,2,2), heights = c(1, 3,3),
sharedMouse = TRUE)
text3d(0,0,0, " ")
next3d()
text3d(0,0,0, "depth_mask = TRUE")
next3d()
text3d(0,0,0, "depth_mask = FALSE")
next3d()
text3d(0,0,0, "alpha = 0.7")
next3d()
triangles3d(xyz, col = rep(c("red", "green", "blue"), each = 3), alpha = 0.7, depth_mask = TRUE)
next3d()
triangles3d(xyz, col = rep(c("red", "green", "blue"), each = 3), alpha = 0.7, depth_mask = FALSE)
next3d()
text3d(0,0,0, "alpha = 0.3")
next3d()
triangles3d(xyz, col = rep(c("red", "green", "blue"), each = 3), alpha = 0.3, depth_mask = TRUE)
next3d()
triangles3d(xyz, col = rep(c("red", "green", "blue"), each = 3), alpha = 0.3, depth_mask = FALSE)
```

As you rotate the figures, you can see imperfections in rendering. On the right, the last drawn appears to be on top, while on the left, the first drawn appears more opaque than it should.

In the figure below, the three triangles each have different transparency, and each use the recommended setting:

`open3d()`

```
## null
## 19
```

```
par3d(userMatrix = M)
triangles3d(xyz[1:3,], col = "red", alpha = 0.3, depth_mask = FALSE)
triangles3d(xyz[4:6,], col = "green", alpha = 0.7, depth_mask = TRUE)
triangles3d(xyz[7:9,], col = "blue", depth_mask = TRUE)
```

In this figure, all three triangles are the same colour, only lighting affects the display:

`open3d()`

```
## null
## 20
```

```
par3d(userMatrix = M)
layout3d(matrix(1:9, ncol = 3, byrow=TRUE),
widths = c(1,2,2), heights = c(1, 3,3),
sharedMouse = TRUE)
text3d(0,0,0, " ")
next3d()
text3d(0,0,0, "depth_mask = TRUE")
next3d()
text3d(0,0,0, "depth_mask = FALSE")
next3d()
text3d(0,0,0, "alpha = 0.7")
next3d()
triangles3d(xyz, col = "red", alpha = 0.7, depth_mask = TRUE)
next3d()
triangles3d(xyz, col = "red", alpha = 0.7, depth_mask = FALSE)
next3d()
text3d(0,0,0, "alpha = 0.3")
next3d()
triangles3d(xyz, col = "red", alpha = 0.3, depth_mask = TRUE)
next3d()
triangles3d(xyz, col = "red", alpha = 0.3, depth_mask = FALSE)
```

Here `depth_mask = FALSE`

seems to be the right choice in
both cases.