No problem. Scaling is used for the overall scale of the node. If you want to affect only the font size you want to do it in the dynamic font settings.
It's also worth noting that the default font for Godot is low res. So scaling it up usually makes it look blurry. I would pick a font in google fonts that you like and use that instead.