File reading is super-slow?

[system]

	Attention	Topic was automatically imported from the old Question2Answer platform.
	Asked By	Farflame

New to Godot. On my learning project, I’m loading in a text file. It’s a list of surnames, about 10,000 names long. It’s reading them line-by-line and appending them to an array. It’s very slow, takes about 2 seconds. It’s instantaneous in other languages, this sort of thing isn’t really a problem nowadays.

I’ve tried loading it as a single string and then splitting it, but it still takes a long time. I’ve also tried just loading it and not adding it to an array, thinking that maybe the array was slow, but it’s still the same.

Is this a known issue or am I doing something wrong? My code is just a very simple…

var f = File.new()
f.open("res://Files//firstnames//english.txt", f.READ)
while not f.eof_reached():
	Firstnames.append(f.get_line())
f.close()

For now I’ve just reduced it from 10,000 names to 10 and it’s super-fast, so it does appear to be struggling to read the file.

Loading a file that long can take some time. With GDscript, there’s a tiny bit of overhead (part of the downsides with a dynamically typed language).

Do you require 10,000 names? Could you cut it down to 1,000, or even 100?

Ertain | 2018-10-29 23:36

For testing I’ve cut it down to 10 which is instant. I could probably manage with 1,000 eventually but I’d prefer the entire list. In Unity, GM and VS, this happens in the blink of an eye, seems strangely slow here.

Farflame | 2018-10-30 02:49

how about just using f.get_line() without appending it to the array?

volzhs | 2018-10-30 05:23

A suggestion would be to read the entire file in one go using content=File.get_as_text() instead of seeking for the next linebreak, and then split the text into an array using Firstnames=split("\n",content)
Depending on os it may be combinations of \n,\r,\n\r you need to split on - ideally you would split on the regEx /[\n\r]+/ but I can see a way to that using gdscript.

For more complex data I would suggest using parse_json and storing the data as JSON.
If you want to use gdscript to convert the data, do as you do now and then write the result of to_json(Firstnames) to a file - that should result i a noticeable speedup

usefull links;
@GDScript — Godot Engine (latest) documentation in English

@GDScript — Godot Engine (latest) documentation in English

guppy42 | 2018-10-30 08:46

I did try reading it as one line and then splitting it but it still took a long time. Based on the answer below, I’m thinking maybe that it’s something to do with my file, although it does read fast in other languages. It does at least look like there may be a solution to this, thanks for the replies.

Farflame | 2018-10-30 14:49

[system]

	Reply From:	Zylann

Testing this in Godot 3.1 alpha 1, GDScript actually does this fast for me:

I tried to generate a list of 10,000 names, it took 13 milliseconds.
Then I wrote code to read these names back, it took 9 milliseconds.
Same measurements in Godot 3.0.6.

I’m not sure what causes it to be slow on your side. Mind sharing that file?
Here is the test code I used:

extends Node


func _ready():
	#generate()
	var time_before = OS.get_ticks_usec()
	var names = read_names()
	var time_taken = OS.get_ticks_usec() - time_before
	print("Took ", time_taken, " microseconds")
	print("Got ", len(names), " names")


func generate():
	var f = File.new()
	f.open("names.txt", File.WRITE)
	for i in 10000:
		f.store_line(str("name_", i))
	f.close()


func read_names():
	var f = File.new()
	f.open("names.txt", File.READ)
	var names = []
	while not f.eof_reached():
		names.append(f.get_line())
	f.close()
	return names

Bonus note: with JSON it took 42 milliseconds, it’s actually slower.

Interesting because that’s basically the same as my code, so as you say, there must be something wrong with my file, although as I pointed out, it reads super-fast in other languages (GM, VS, Unity). How do I share the file? I do know that each line ends with a cr cr lf (that’s viewing it in Notepad++). I have a version with those characters trimmed off, but then I didn’t know how to split it because it wasn’t responding to readline (it was just reading it as a single line).

How do I share the file? I’m guessing that I need to change the formatting in it.

Farflame | 2018-10-30 14:52

I’ve just run your code through on my machine and it’s instantaneous on mine too. I even set it to 1 million reads and it took about 1 second. When I look at the generated file, it shows as name_1name_2name_3 etc, all stuck together, whereas mine in my file are all separated. I’m guessing that’s what’s slowing it down. I’ll see if I can find a way to remove all line endings but still have it know where to separate them.

Farflame | 2018-10-30 15:07

So according to Notepad++, my file was in Macintosh format (lines ending in CR). I also had a version in Windows format (CR LF). It appears that what I need for Godot is Unix format (just LF). Luckily Notepad++ has a simple conversion, so I just converting it over to Unix and it’s much faster now

Farflame | 2018-10-30 15:24

Seems like use csv is faster than the txt and json

yikescloud | 2021-02-08 13:48

Not surprising, I guess that’s because CSV is simple enough and the loop is done by the engine instead of GDScript

Zylann | 2021-02-08 14:29

Sorry for that, seems still is txt > csv > json
I think Zylann is write, the speed is base on the data structure. txt got nothing extra, then json has “” and csv got a comma.
So I think if you want to make a giant world and load real time, it will be a problem, maybe need some gdnative, I have no idea

yikescloud | 2021-02-08 15:06