While I didn't test it in code, I'm pretty sure your issue is caused by the offset moving with changed scale (so it's hopping back and forth all the time, crossing the "border" where scaling changes). With no offset set, you're scaling (0, 0), so nothing happens. But if you scale (1, 0), you'll make it start moving/hop around.
To solve this, I'd simply make the visible card a child node of an invisible/default Node2D. You reposition and check the parent node's position, but you actually scale (and never move) the child node showing the card.