Isometric outline rendering

17 Dec 2019

Move the red sorcerer around:

For this project I mostly wanted to play with outlines applied in the shader. Move the outline_mode slider to the left for “screen space” outlines and to the right for “world space” outlines. Two different approaches, with pros and cons for each.

For this proof of concept project, there are some things I haven’t implemented and some bugs I haven’t fixed. I wanted to get enough to see that a technique does or doesn’t work.

1 Motivation#

I like to periodically test out rendering ideas. Back in the Flash days I had experimented with isometric + field of view^[1] in Flash, but WebGL exposes more GPU features than Flash did, and I wanted to try doing more. For the Flash project did 2d outlines and visibility on the CPU. Here, I am doing both on the GPU. I also wanted to make fancier outlines. I didn’t want to outline every sprite; I wanted to outline only the wall boundaries:

Outlines around every sprite vs around walls

I also used this project to play with instanced rendering and billboard sprites. For this project I wanted to do everything on the GPU, even though there were many things that would’ve been better on the CPU. This was a way to practice my GPU/shader programming skills.

I implemented two different outline algorithms, screen-space and world-space. Rezoner on Twitter suggested I use both light and dark outlines^[2], so I implemented for the world-space outline mode. It’s not something that works nicely in screen-space outline mode.

2 Implementation#

2.1. Sprite rendering

I initially copied the sprites into a separate texture to add padding and outlines. I also ran into a difference between Safari and Chrome/Firefox. In Safari canvas, drawing a canvas onto another canvas with shadow will apply the shadow, then clip to the rect I want to draw. On Chrome/Firefox it clips first, then applies the shadow. I ended up having to make an extra copy to clip first, then apply the shadow in a separate draw.

But I threw all that away. I’m now using the unaltered spritesheet, and applying the effects in the shader.

First problem I ran into was that when I drew a sprite, it would leak pixels from adjacent sprites. I was using GL_NEAREST so I thought it would be fine, but noooo, it looks like I get texture coordinates outside the 0,0 – 1,1 range! Maybe for antialiasing? I don’t know.

In any case the fix was to clamp the coordinates to the sprite range and then do the texture lookup.

2.2. Cube rendering

I draw each face of the cube in a separate draw call. That lets me put the normal into a uniform, and then I can put four corners into instanced rendering. The wall positions are in the attribute once, and then I call draw five times on that same attribute buffer, but with different faces of the cube each time.

With this camera angle, I never need to draw the back or right side of each cube. I drew them anyway. Bug: the back and right side of the cube don’t render outlines correctly in world-space mode.

2.3. Outlines

2.3.1. Screen space

Set outline_mode to 0 in the demo.

I draw the walls and floors with a different color per surface id. In a post-process step I draw a black pixel whenever the surface id changes:

Each face is rendered in a separate pass

This approach works for the billboard sprites too. Each sprite gets its own surface id (see Omar Shehata’s article about this^[3]).

Unfortunately there are glitches. The post-process step doesn’t really understand what’s in front or back. Where the goose and wall meet, the outline pushes into the wall:

Glitches in rendering of wall-object boundary

It’s also a problem with walls alone:

Glitches in rendering of wall-wall boundary

Move the color slider to 0 to see these glitches in the live demo.

I made the lines thinner to hide these problems, but that doesn’t solve them. I thought a better fix would be to use the depth data (distance to camera) to decide which surface receives the outline. I experimented a little bit with that but it didn’t give great results either.

2.3.2. World space

Set outline_mode to 1 in the demo.

To avoid the problems I ran into with screen space outlines, I tried a completely different technique:

Draw the map data in world coordinates to a texture.
When drawing walls, look at the adjacent map tiles to decide whether to draw a line or not.
When drawing billboard sprites, draw a black pixel if the texture is transparent at this pixel, and one of the 8 bordering pixels is opaque.

The downsides of this approach:

Some outlines will be twice as thick as others, where two visible surfaces meet, because each surface will render its own outline, not realizing the other is also rendering one. I think I can compensate for this if I keep the camera angle fixed by drawing outlines on only some sides of each face. For the demo, I didn’t implement the fix, so if you look closely you’ll see the problem.
The outlines are constant thickness in world space, which is fine in orthographic view, but closer cubes will have a thicker outline in perspective view. I might be able to compensate for this by using shader derivatives. The demo is in orthographic view so you don’t see this problem.
The back-facing walls don’t receive an outline. To fix this, I could draw the upper outlines on the block tops instead of on the walls. I didn’t implement this for the demo.

2.3.3. Hybrid

The good thing about screen space is that the outlines have constant width, even when using perspective, where the closer objects are larger than farther objects. The good thing about world space is that because we know about the objects we can position the lines properly, and we can apply lighting. A hybrid approach would be to draw in world space, but to use OES_derivatives to vary the lines to be constant width on the screen. I haven’t tried this.

2.4. Field of view

I implemented field of view by drawing shadow polygons for each wall face:

Construct a shadow polygon from each wall line segment

I drew these out to a texture, then used that as a lookup when drawing the objects. Everything in the shadow area is darkened. This approach is explained elsewhere:

2.4.1. Billboards

The way I draw billboards doesn’t work with the light/shadow system, so I had to apply one matrix to orient the billboards for lighting purposes, and apply a different matrix to orient the billboards for rendering purposes. I think there must be a simpler way but I haven’t investigated.

2.4.2. Cube tops

The top of a wall block will always be in shadow!

Possible fix: sample from the closest edge of the square. This doesn’t work well because there’s not continuity between adjacent wall blocks.

Possible fix: dilate the lit areas in the shadow map so that they extend into walls. Haven’t tried this.

Possible fix: sample from both the horizontal and vertical edges of the square, and pick the max. This works better but interior corners never get lit up.

Possible fix: sample from both the horizontal and vertical edges and also the diagonal corner. This helps with interior corners but now non-corners get lit up. → this is what is implemented for the demo

Possible fix: sample from all edges and light up the block if any wall is lit. This is more expensive and looks wrong, as it’s not continuous with the floors.

Possible fix: distinguish interior corners by writing the wall vs floor data to the shadow map in another channel. I need this data for world-space outlines anyway.

{ this section would be better with screenshots, and I forgot to take them }

There are still glitches.

3 Assets#

I am using Oryx’s sprites from the Assemblee Competition^[8] (Creative Commons CC BY-NC 3.0 US). I load these into GPU textures, unmodified.