That was the exact solution, thanks a lot!
While experimenting with this, I found that the animations I had created for my character in first person were only affecting some of the bones inside the Viewport. No idea why, I tried debugging it, but I couldn't even reproduce it with a clean project.
I had to follow your suggestion exactly make it all work, so the only child of the viewport is the camera (which I did have to sync up).