Large arrays (over 65536) containing PoolByteArrays crash the game

:information_source: Attention Topic was automatically imported from the old Question2Answer platform.
:bust_in_silhouette: Asked By path9263

I am playing around with generating voxels in Godot. I’ve just been using GDScript so far to build a prototype. I was looking into ways to use less memory when dealing with lots and lots of voxels and came across PoolByteArray since GDScript does not seem to have a byte type. It looks promising but it’s not working for me.

The problem I’m having is that any array with more than 65536 indexes that contains PoolByteArrays crashes the game. If the array contains normal arrays it works just fine. Example of what I mean below… Also please see my edit at the bottom.

The following code works fine:

var tstArr = []
var randNumber
func _ready():
	for i in 65536:
		randNumber = randi() % 23
		var NormalArr = [1, randNumber]
		tstArr.push_back(NormalArr)
		
	print(tstArr.size()) # prints "65536" as expected

This code however crashes, the debugger I don’t think even gets to start, it doesn’t report any errors anyway:

var tstArr = []
var randNumber
func _ready():
	for i in 65536:
		randNumber = randi() % 23
		var bytPool = PoolByteArray()
		bytPool.push_back(1)
		bytPool.push_back(randNumber)
		tstArr.push_back(bytPool)
		
	print(tstArr.size()) 

However if in the above code I change for i in 65536: to for i in 65535: then it runs fine.

I’m not sure if this is a bug or if I am doing something wrong. I don’t have a great concept of how the PoolByteArray actually interacts with the computer’s memory but I suspect that could have something to do with this.

Any info or tips you can provide are greatly appreciated! Thanks!

EDIT: I’ve also discovered that there seems to be a limit to the number of PoolByteArrays even when they are contained in separate arrays.

var testArrOne = []
var testArrTwo = []
var randNumber
var maxNum = 65535
func _ready():
	for i in maxNum:
		randNumber = randi() % 23
		var bytPool = PoolByteArray()
		bytPool.push_back(1)
		bytPool.push_back(randNumber)
		testArrOne.push_back(bytPool)
	for i in maxNum:
		randNumber = randi() % 23
		var bytPool = PoolByteArray()
		bytPool.push_back(1)
		bytPool.push_back(randNumber)
		testArrTwo.push_back(bytPool)

The above does not work but if you change it to var maxNum = 32767 (half of the limit) or less, it does work. This is starting to seem more like a bug.

Again, thanks for any insight you can provide!

:bust_in_silhouette: Reply From: hungrymonkey

PoolByteArray is continuous in memory. If you are removing and adding elements often, you use something else.

http://docs.godotengine.org/en/latest/classes/class_poolbytearray.html?highlight=PoolByteArray

https://github.com/godotengine/godot/blob/master/core/dvector.h#L512

You should be calling resize as little as possible.

65536 this number is the max 16 bit. so it is kinda suspecious

btw, I tested your code above in a script with ~/godot/bin/godot -S

with the file extending Mainloop.

I couldnt reproduce the crash.

My advice to only resize it once. Memory reallocation sometimes failed if you redo it often too

var tstArr = PoolByteArray().new()
tstArray.resize(65536)
func _ready():
    for i in 65536:
        tstArray[i] = randi() % 23
 print(tstArr.size()) 

Thanks Hungrymonkey,
With my game the way it is now I don’t often add or remove elements from the byte array once it is loaded, I do modify the values of the bytes when voxels are changed/removed but that should be okay right? Would the whole array (tstArr) of PoolByteArrays be forced to be continuous or would it just be continuous for each one (bytPool) contained in tstArr? Without a byte type what is the best way to store a huge number of variables without consuming too much memory? I will give it a try with just resizing the array once like you show.

I agree that the number is suspicious!

Also were you in time to see my edit to my original post?

Thanks again!

path9263 | 2018-04-05 21:21

PoolByteArray are continuous

List/Arrays are not necessary continuous.

Are you afraid of too much memory?

You do realize that reallocating and deallocating memory will lead to even worse performance than the amount of memory saved.

Honestly, do the math on how much space you need.

hungrymonkey | 2018-04-05 21:38

Yes, memory reduction is the goal and my reasoning for trying to use a PoolByteArray instead of just a few ints. I am using voxels but my worlds are finite and not that large; 96x64x96 voxels currently, though I would make them larger if I could reduce memory usage.

Once that world is loaded I wouldn’t like to deallocate any chunks or voxels of it, since it is finite I would like to keep them all in memory.

Okay so I tried your example and while it does work, it does not actually fit my criteria.

var arrSize = 65536
var tstArr = PoolByteArray()
func _ready():
	tstArr.resize(arrSize)
	for i in arrSize:
		tstArr[i] = randi() % 23
 print(tstArr.size()) 

The problem is that this just creates one huge PoolByteArray. For my voxels in my actual game I have them divided up into a 3D array of chunks, each chunk then contains a 3D array of voxels. I was hoping to use one individual PoolByteArray for each voxel just to store the properties of the voxel (ie: type, color, etc…) and the normal 3D arrays to store it’s position. I suppose I could rework it to use a flat array instead but I’d rather not if possible. I was hoping to implement run length encoding as well but this would be a bad idea on a huge continuous array as it would mean lots of reallocating and deallocating. I was really just using PoolByteArray as a way to use less memory by saving voxel data to bytes instead of ints, I was not trying to create a continuous array. If GDScript had a byte type I would have just used that.

I’m starting to think I should try use C# or something for handling my voxel data.

So I tried playing with it a bit more and came up with this instead, unfortunately it still has the same issue of crashing when over 65536.

var arrSize = 65536
var tstArr = []
var randNumber
func _ready():
	tstArr.resize(arrSize )
	for i in arrSize :
		randNumber = randi() % 23
		var bytPool = PoolByteArray()
		bytPool.resize(2)  # <- probably not necessary but tried it anyway
		bytPool[0] = 1
		bytPool[1] = randNumber
		tstArr[i] = bytPool
	print(tstArr.size())

Again thank you for your help!

path9263 | 2018-04-05 22:07

:bust_in_silhouette: Reply From: Zylann

A bit late, but the issue is that Godot 3 limits the amount of PoolVectors you can create (their size doesn’t matter). That is still the case in Godot 3.2.4: https://github.com/godotengine/godot/blob/21746ce7140286846a953a8e4b0c3f852b2a3489/core/pool_vector.h#L76

The issue is tracked here: MemoryPool size hardcoded? · Issue #37154 · godotengine/godot · GitHub

I think this is no longer limited in Godot 4.