How to apply antialiazing in a shader ?

:information_source: Attention Topic was automatically imported from the old Question2Answer platform.
:bust_in_silhouette: Asked By leo-pnt

Hi

I made a sort of ocean floor map with a noise texture. it looks like this :
Basically it’s a noise texture where the shader gives this sort of topographical map effect.
enter image description here

The original solution came from LuckSmith in this post Topographical map - #4 by solub - Coding Questions - Processing Foundation

So my shader is like this

shader_type canvas_item;

uniform sampler2D tex;
uniform float width;
uniform float height;
uniform float l;

void fragment() {
	vec2 north_coor = vec2(UV.x, UV.y -1.0/height);
	vec2 west_coor = vec2(UV.x -1.0/width, UV.y);
	
	int hN = int(round(texture(tex, north_coor).r * l)); //because it's black&white so take the r value  or anything else (blue, green), it doesn't matter
	int hW = int(round(texture(tex, west_coor).r * l));
	int h0 = int(round(texture(tex, UV).r * l));
	float h = texture(tex, UV).r;
	
    COLOR = vec4(0.0, 0.0, 0.0, 1.0); //background
	
	if(h0 != hW || h0 != hN) {
		COLOR = vec4(0.0, 5.0 * pow(h, 5.0), 0.0, 0.7); //here we want it green
	}
}

So as you can see it’s aliazed a lot. The shader applies on a noise sprite texture. It is a fixed background.
Moreover, I suspect this aliazing to give a flickering effect when the player moves at slow speed as the background is the only element to flicker compared to other element (displayed by func _draw():)

Filters are enable but it does nothing.
I tried COLOR = textureLod(SCREEN_TEXTURE, SCREEN_UV, blur_amount); at the end of fragment() but it result in a black image because the contrast (think of it as the slope) is actually what allows me to draw those lines.
So how to apply an antialiazing effect directly in the shader ?

Also, here is my terrain.gd class code if it can help to find an alternative

extends Sprite
class_name Terrain

var noise:OpenSimplexNoise
var noiseImage:Image

var terrainPixelSize:int


func _init():
	texture = NoiseTexture.new()
	texture.seamless = true
	texture.width = 1024
	texture.height = texture.width # because must be a SQUARE
	
	region_enabled = true
	region_rect = Rect2(0, 0, texture.width * 4, texture.height * 4)
	
	noise = OpenSimplexNoise.new()
	noise.seed = randi()
	
	#noise parameters
	noise.octaves = 4
	noise.period = 300
	noise.persistence = 0.5
	noise.lacunarity = 3
	
	texture.noise = noise
	
	noiseImage = noise.get_seamless_image(texture.width) # same must be square texture
	noiseImage.lock()
	
	
	var l = 60 # number of contour height plans
	
	
	material = ShaderMaterial.new()
	material.shader = load("res://shaders/terrain.shader")
	material.set_shader_param("width", texture.width)
	material.set_shader_param("height", texture.height)
	material.set_shader_param("tex", texture)
	material.set_shader_param("l", l)

Sorry for the long question and thanks for your time !

:bust_in_silhouette: Reply From: Calinou

So how to apply an antialiazing effect directly in the shader ?

You could apply FXAA in your shader (like this), but doing so in 2D is ill-advised as it will introduce a noticeable amount of blur.

Another solution you have is to use supersampling. That is, render the shader’s texture to a higher resolution then display it a lower scale. This is the best way of antialiasing you can get, but it’s also the most demanding one on the GPU.

FXAA would be great but apparently we can’t make two shader pass in 2D. the code that I have in my shader needs the noise texture as-is because of the way it’s made, so I can’t seem to be able to use any premade functions like that as it messes with “slope” detection.

I tried supersampling but my game actually uses the noise texture values as a virtual 2D depth

I also found a way to apply blur on top of it but it didnt resolve the flickering problem.

So apparently, my only way is to improve my shader in itself

Thanks for the suggestions anyway :wink:

leo-pnt | 2020-12-03 18:01