How to create a shader for casting shadows on an isometric tilemap ?

[system]

	Attention	Topic was automatically imported from the old Question2Answer platform.
	Asked By	CravateZeus

Hi, I’m trying to create a shader that will cast the shadows of isometric objects placed on a tilemap :

To achieve this, this article here Dynamic Lighting and Shadows In My 2D Game | Matt Greer suggests using a heightmap for each sprite.
So is there a way I give each tile in my tilemap a heightmap texture and use it in a shader ?

As a workaround, since my textures are rather small, I was thinking I could use transparency on the tile normalmap texture as heightmap data. But I still have no idea on how to access this normalmap from a shader to then draw a shadow outside of that specific tile.

I’m rather new to shader programming so any help is appreciated !

[system]

	Reply From:	exuin

Giving a tile a normal map is easy - just look for the property under “selected tile”.

I couldn’t find a way to access the normal map from inside a shader though. I’m not sure if you could write a shader like this with Godot right now since you can’t draw outside the UV coordinates of a tile. Maybe you can use a Light2D instead?

(I tried using a Light2D but I couldn’t get it to work so I probably did something wrong)

Edit:
I finally got something working - what you need to keep in mind is that Godot’s y-coordinates are flipped, so you will need to vertically flip your normal maps as well. For example, here is my cube.

And here is what it looks like with the lighting and occlusion shape added.

Thanks a lot for your answer !
I already know how to use Godot’s light2D but like we can see in your example, the shadow of your cube just continues indefinitly until it leaves the light2D texture so it’s not exactly what I’m trying to do. I added a response to my own post where I describe how I almost implemented what I was talking about if you’re interested (though I still have two major issues).

CravateZeus | 2021-03-30 08:12

[system]

	Reply From:	CravateZeus

Hi, in the meantime I actually came up with a solution that almost works properly :

First I have a simple shader for my sprite where I give it a heighmap. Depending on the color of each pixel in the heightmap, I use the HDR value of that pixel to pass the height information onto another full-screen shader.

I’m sorry if my code is not super clean, I pretty much got my implementation to work by trial and error.

shader_type canvas_item;
uniform sampler2D heightmap;

void fragment(){
	vec4 heightmap_tex = texture(heightmap, UV));
	if(heightmap_tex.a > 0.9){
		// I'am adding 1.0 * h to to that pixel's color, where h
        // is the height of that pixel on the heightmap
		COLOR = texture(TEXTURE, UV) + vec4(1.0, 1.0, 1.0, 0.0) * (heightmap_tex.r * 255.0 + 1.0);
	} else {
		COLOR = texture(TEXTURE, UV);
	}
}

Then my screen shader knows that if a pixels’color is greater than 1.0, it means it has some height.

shader_type canvas_item;
uniform float xyangle;
uniform float zangle;

void fragment(){
	bool isinshade = false;
	float dist;
	float height;
	float lookupheight;
	float traceheight;
	vec4 col;
	
	//first I calculate the height of the current pixel
	col = texture(SCREEN_TEXTURE, SCREEN_UV);
	height = col.r;
	if(height < 1.0){height = 0.0;}
	height = round(height);

	//then I look at some pixels in a given direction
	for(int i = 0; i < 100; i ++){
		//distance of the pixel I'm looking at
		dist = SCREEN_PIXEL_SIZE.x * float(i)*2000.0;
		//UV of that pixel on screen
		float lookupx =SCREEN_UV.x + (0.001) * (-4.0) * dist ;
		float lookupy =SCREEN_UV.y + (0.001) * 0.0 * dist ;
		//Height of that pixel
		lookupheight = texture(SCREEN_TEXTURE, vec2(lookupx,  lookupy)).r;
		
		if(lookupheight < 1.0){lookupheight = 0.0;}
		lookupheight = round(lookupheight);
		
		//If that pixel's height is greater than the height of my current pixel
		if(lookupheight > height){
			//I calculate the height that that pixel should be if it was to draw a shadow
			//on my current pixel
			traceheight = dist * tan(zangle) + height;
			if(traceheight < 1.0){traceheight = 0.0;}
			traceheight = round(traceheight);
			
			//If the height of that pixel and the height it needs to be are close enough
			//then my current pixel needs to be shaded
			if(abs(traceheight-lookupheight) < 4.0){
				isinshade = true;
			}
		}
	}
	
	//Now I'm juste resetting the pixel's color to it's real value
	if(col.r > 1.0 || col.g > 1.0 || col.b > 1.0){
			col.r = mod(col.r, 1.0);
			col.g = mod(col.g, 1.0);
			col.b = mod(col.b, 1.0);
			COLOR = col;
			isinshade = false;
		}

	if(isinshade){
		COLOR = col - vec4(0.2, 0.15, 0.15, 0.0);
	} else {
		COLOR = col;
	}
}

Ok so as I said, using those two shaders I get some decent results but I still have two major problems:

First you may have notice that I’m not using xyangle in my screen shader. It’s because whenever I use :
float lookupx =SCREEN_UV.x + (0.001) * cos(xyangle) * dist ;
float lookupy =SCREEN_UV.y + (0.001) * sin(xyangle) * dist ;
it causes ugly artifacts:

I’m pretty sure it has something to do with an imprecision caused by the floating values of the sin() and cos() but I’m not exactly sure.
So does anyone know how I could fix those ?

Second since I’m using SCREEN_TEXTURE to read the height value of each pixel, I can’t access the height of a pixel off screen. So whenever a sprite is partly outside the screen, its shadow gets partly cutted out. So is there a way I could access pixels slightly off screen in my second shader ?

Thanks a lot for reading me !

Unfortunately I don’t think there’s a way to access pixels off screen with a shader

exuin | 2021-03-30 14:59

Hmm, would it be possible to just take the original sprite’s sprite, rotate it and scale it, and then change the color?

exuin | 2021-04-02 02:18

Yes that could work for simple sprites like the one I showed, but for bigger, more complex ones it really doesn’t look great. I guess could just use hand drawn shadows on an other tilemap for example but then I would not have as much control on it.

Right now I’m trying to see if I can use a viewport to render a bigger area and then only display a portion on screen. This way, maybe I could access pixels outside the screen by using that viewport texture. But since all that is pretty far from what I initially asked, I’m probably going to open a new question for it.

Thanks for your help !

CravateZeus | 2021-04-03 09:32