As you guessed, the engine is waiting for something to happen.
For those large spikes, a lot of something has happened in that frame which the engine has to finish working on before it starts the next frame.
If you're trying to limit these spikes, you'll need to think carefully about what you strictly need to have in your scripts or what rendering tasks you could batch or improve (if any) and optimise from there. Check out the performance/optimisation guides in the docs for lots of more interesting stuff. There is also certain "best practices" for other programming languages that can apply to GDScript, such as basic math being extraordinarily cheap, and although it sounds weird you could also look at optimisation techniques used by developers in other game engines which may give you clues as to how you could improve your own code.
But the most basic thing will restrict performance in Godot: V-Sync is enabled by default in your Project Settings, which is the likely culprit of high idle times and far lower-than-expected framerates due to it literally "idling" between frames to try to sync the framerate with what it thinks is the monitor refresh rate.
Apart from V-Sync, there may be other settings activated in your default Project Settings which you may want to browse through, change and/or disable as necessary. Very few will have the same drastic effect that V-Sync does but it's important to check out and know what options you have available.
Keep in mind that Idle Time represents everything else besides physics, including rendering, print functions and even the profiler that you are looking at, as well as the game engine main loop which will happen regardless of your own code. So there will always be some idle time, but it certainly can be a lot lower.
As for the Physics Frame, it represents the Physics Servers attempt to maintain a constant frame time. This doesn't affect your rendering performance, but excessive physics simulation/code may cause your idle time to come up.