I think it might be both, I get this too, but I changed some settings and it improved some. if you have shadow3s, I turned this off. Maybe SSAO could take place of the shadows. Each time they update Godot, performance seems to get better, so maybe it will be not as bad when 3.2 or other future versions release.