GDNative is not different than just developing your own library, so all things you know about this development cycle will apply (dependencies, supported platforms etc). The fact Godot is able to load libraries with a GDNative API isn't magically changing anything.
The main things to know about is how GDNative lets you expose your functionalities to Godot. You can register classes as if they were scripts (a script = a class), and you can register methods on them.
You also have to register methods using types that are exclusively available within Godot. So for example, you won't be able to directly bind a function receiving a
std::string, because Godot doesn't use STL, so neither GDScript. So just like Godot does internally with its own third party libraries, you will need to create "wrapper objects" which are Godot-friendly, and then you access your library stuff in their private implementation.
Binding a linear algebra library is possible, however keep in mind you can only expose classes (aka reference types that are created using
new in GDScript).
Also, due to what I explained above, you will have to map Godot types to it. So if your library expects a 3-component vector as a
glmVector3 for example, you will need a wrapper method that accepts a
Vector3 from Godot and convert it to a
glmVector3 in your C++ code. You could also bind the said type as a full-fledged class but it would be awkward to do it given the fact Godot already has such a type.
You should not need to convert pointers to integer, it's not good to expose that to the script API. To call your functions or to call Godot functions, there is a bunch of
void* being passed, if you use GDNative C API directly. However, using a binding library (like the C++ one made by Karroffel) you should not have to worry about that.