Do FPS come back up if you interact with the window? Check your graphics card power management state, sometimes it might decrease due to "inactivity" which causes slowdowns like this. Check also memory and GPU memory (not just from Godot's debugger).
Another possibility is a bug in your game accumulating something, but not sure what it could be (it doesnt have to be objects, could be an array getting bigger and bigger, a draw call iterating more and more, or a GDNative library doing something wrong).
Otherwise, Godot bug, but it's even harder to tell what. Maybe you can try making a simple project to see if anyone else can reproduce it.
FYI, in Godot 3.2.1 (still using 3.0?), idle time correspond to this section of the main loop: https://github.com/godotengine/godot/blob/02ed72c3733aa2290dec9ffca52a7bbbc96382c3/main/main.cpp#L2026-L2049