0 votes

I am playing around with generating voxels in Godot. I've just been using GDScript so far to build a prototype. I was looking into ways to use less memory when dealing with lots and lots of voxels and came across PoolByteArray since GDScript does not seem to have a byte type. It looks promising but it's not working for me.

The problem I'm having is that any array with more than 65536 indexes that contains PoolByteArrays crashes the game. If the array contains normal arrays it works just fine. Example of what I mean below... Also please see my edit at the bottom.

The following code works fine:

var tstArr = []
var randNumber
func _ready():
    for i in 65536:
        randNumber = randi() % 23
        var NormalArr = [1, randNumber]
        tstArr.push_back(NormalArr)

    print(tstArr.size()) # prints "65536" as expected

This code however crashes, the debugger I don't think even gets to start, it doesn't report any errors anyway:

var tstArr = []
var randNumber
func _ready():
    for i in 65536:
        randNumber = randi() % 23
        var bytPool = PoolByteArray()
        bytPool.push_back(1)
        bytPool.push_back(randNumber)
        tstArr.push_back(bytPool)

    print(tstArr.size()) 

However if in the above code I change for i in 65536: to for i in 65535: then it runs fine.

I'm not sure if this is a bug or if I am doing something wrong. I don't have a great concept of how the PoolByteArray actually interacts with the computer's memory but I suspect that could have something to do with this.

Any info or tips you can provide are greatly appreciated! Thanks!

EDIT: I've also discovered that there seems to be a limit to the number of PoolByteArrays even when they are contained in separate arrays.

var testArrOne = []
var testArrTwo = []
var randNumber
var maxNum = 65535
func _ready():
    for i in maxNum:
        randNumber = randi() % 23
        var bytPool = PoolByteArray()
        bytPool.push_back(1)
        bytPool.push_back(randNumber)
        testArrOne.push_back(bytPool)
    for i in maxNum:
        randNumber = randi() % 23
        var bytPool = PoolByteArray()
        bytPool.push_back(1)
        bytPool.push_back(randNumber)
        testArrTwo.push_back(bytPool)

The above does not work but if you change it to var maxNum = 32767 (half of the limit) or less, it does work. This is starting to seem more like a bug.

Again, thanks for any insight you can provide!

in Engine by (121 points)
edited by

1 Answer

+1 vote

PoolByteArray is continuous in memory. If you are removing and adding elements often, you use something else.

http://docs.godotengine.org/en/latest/classes/class_poolbytearray.html?highlight=PoolByteArray

https://github.com/godotengine/godot/blob/master/core/dvector.h#L512

You should be calling resize as little as possible.

65536 this number is the max 16 bit. so it is kinda suspecious

btw, I tested your code above in a script with ~/godot/bin/godot -S

with the file extending Mainloop.

I couldnt reproduce the crash.

My advice to only resize it once. Memory reallocation sometimes failed if you redo it often too

var tstArr = PoolByteArray().new()
tstArray.resize(65536)
func _ready():
    for i in 65536:
        tstArray[i] = randi() % 23
 print(tstArr.size()) 
by (427 points)
edited by

Thanks Hungrymonkey,
With my game the way it is now I don't often add or remove elements from the byte array once it is loaded, I do modify the values of the bytes when voxels are changed/removed but that should be okay right? Would the whole array (tstArr) of PoolByteArrays be forced to be continuous or would it just be continuous for each one (bytPool) contained in tstArr? Without a byte type what is the best way to store a huge number of variables without consuming too much memory? I will give it a try with just resizing the array once like you show.

I agree that the number is suspicious!

Also were you in time to see my edit to my original post?

Thanks again!

PoolByteArray are continuous

List/Arrays are not necessary continuous.

Are you afraid of too much memory?

You do realize that reallocating and deallocating memory will lead to even worse performance than the amount of memory saved.

Honestly, do the math on how much space you need.

Yes, memory reduction is the goal and my reasoning for trying to use a PoolByteArray instead of just a few ints. I am using voxels but my worlds are finite and not that large; 96x64x96 voxels currently, though I would make them larger if I could reduce memory usage.

Once that world is loaded I wouldn't like to deallocate any chunks or voxels of it, since it is finite I would like to keep them all in memory.

Okay so I tried your example and while it does work, it does not actually fit my criteria.

var arrSize = 65536
var tstArr = PoolByteArray()
func _ready():
    tstArr.resize(arrSize)
    for i in arrSize:
        tstArr[i] = randi() % 23
 print(tstArr.size()) 

The problem is that this just creates one huge PoolByteArray. For my voxels in my actual game I have them divided up into a 3D array of chunks, each chunk then contains a 3D array of voxels. I was hoping to use one individual PoolByteArray for each voxel just to store the properties of the voxel (ie: type, color, etc...) and the normal 3D arrays to store it's position. I suppose I could rework it to use a flat array instead but I'd rather not if possible. I was hoping to implement run length encoding as well but this would be a bad idea on a huge continuous array as it would mean lots of reallocating and deallocating. I was really just using PoolByteArray as a way to use less memory by saving voxel data to bytes instead of ints, I was not trying to create a continuous array. If GDScript had a byte type I would have just used that.

I'm starting to think I should try use C# or something for handling my voxel data.

So I tried playing with it a bit more and came up with this instead, unfortunately it still has the same issue of crashing when over 65536.

var arrSize = 65536
var tstArr = []
var randNumber
func _ready():
    tstArr.resize(arrSize )
    for i in arrSize :
        randNumber = randi() % 23
        var bytPool = PoolByteArray()
        bytPool.resize(2)  # <- probably not necessary but tried it anyway
        bytPool[0] = 1
        bytPool[1] = randNumber
        tstArr[i] = bytPool
    print(tstArr.size())

Again thank you for your help!

Welcome to Godot Engine Q&A, where you can ask questions and receive answers from other members of the community.

Please make sure to read How to use this Q&A? before posting your first questions.
Social login is currently unavailable. If you've previously logged in with a Facebook or GitHub account, use the I forgot my password link in the login box to set a password for your account. If you still can't access your account, send an email to webmaster@godotengine.org with your username.

Categories