How to extract amplitude, pitch, frequency from audio stream data without analyzer and playback

:information_source: Attention Topic was automatically imported from the old Question2Answer platform.
:bust_in_silhouette: Asked By Dumuz

I am very new to anything sound or music related so bear with me.

How would I be able to extract (at least 1 if not all of) the amplitude and/or frequency and/or pitch from the PoolByteArray of the data of an AudioStream?

I would like be able to retrieve this information without needing to playback the audio using the SpectrumAnalyzer in real time.

Appreciate your time.

I can’t answer your question but I can make a few comments.

The Fourier transform is the (mathematical) transformation from a sound to its frequency-amplitude pairs, say [[f1, A1], [f2, A2], … , [fn, An]]. There are good algorithms for doing the Fourier transform, but none that are available in GDscript that I know of.

One problem you might have with doing a Fourier transform is that it treats the whole sound as if it has one list of frequency-amplitude pairs. But for a sound that has different piano chords being played one after another, there should be a different list of frequency-amplitude pairs for each chord. So what you can do is break a sound up into shorter sounds, and then Fourier transform each shorter sound. That will give you multiple lists of frequency-amplitude pairs, one for each part of the sound. For example, if you have a sound that is 3 seconds long with a different note being played each second, then you would be better off splitting the sound into 3 shorter sounds and Fourier transforming each shorter sound to get three lists of frequency-amplitude pairs: [[f1, A1], [f2, A2], … , [fn, An]] for the first second, [[f1, A1], [f2, A2], … , [fn, An]] for the second second, and [[f1, A1], [f2, A2], … , [fn, An]] for the third second. Doing Fourier transforms on a bunch of short pieces of a sound is called a “short-time” Fourier transform.

The SpectrumAnalyzer Fourier transforms each short burst of sound as it’s being played in real time. At first glance the SpectrumAnalyzer only works by playing the sound, but maybe there is a way to get it to work without having to play the sound. If so, fingers crossed that someone who knows how will see your question and answer it. If not, there are other options.

If your sound is the same each time the game runs, you should get your sound outside Godot and do a short-time Fourier transform on it, then save the frequency-amplitude pairs that you care about into a text file and then use a File resource to load it when your game loads. You also might have to experiment with how short you make each piece of your sound. Start by just doing a FT on the whole sound and if that doesn’t give you good results when playing the game, then try 2 pieces, or 3 pieces, or 4 pieces, etc. If the sound is really long then probably try more pieces faster, like 10 pieces, 100 pieces, 1000 pieces, etc.

If your sound is different each time the game runs, or at different times of the game, then you have to do the FT during runtime. I don’t know of any FT algorithm in GDscript, nor do I know how to get access to one. So you might have to figure out how to import or write a Fourier transform algorithm. I would urge you to import, since debugging and testing numerical algorithms might be more trouble than it’s worth. As for what Fourier transform library to import, I don’t know but there must be some well known C++ libraries that have a fast Fourier transform algorithm. I’ve never imported a C++ library so I unfortunately can’t help you with that.

I hope this gives you some relevant insights into your situation. Feel free to ask additional questions.

haydenv | 2022-07-09 13:48

Haydenv,

I’m coming back to this concept and trying to find a real solution for this problem now. I’ve come back to your answer every now and then trying to think of the best way to achieve what I’m trying to do.

You seem to be very knowledgeable on the subject, as I have no idea about FFT’s until your answer, so thank you very much for that!

Is there any way I could get in touch with you via discord to ask questions about how to implement a possible solve using a GDNative/GDExtensions and a 3rd party c++ library to? The only script I know how to sort of use right now it GDScript so, C++ and visual studio is a little overwhelming.

Dumuz | 2023-01-16 19:21

Hi, yes I’d love to investigate how this could be done in a Godot project. My discord is noBrainur#6113. (I think that’s enough to find me on discord? not sure)

haydenv | 2023-01-16 20:55

Awesome! keep a look out for Sir.Mr.Dr.Jr.

Dumuz | 2023-01-16 21:10