Need an audio analysis library to create real-time feedback from an audio file?

Question

Real-time is not necessarily required, however I am creating a game for my last project of the year, and I want to use the power of sound to create dynamic levels based solely on the music track that is playing. I want to create this game for PS Vita using playstation mobile and C #, but if I want, I can switch to C ++ and PSP.

I can use the WAV file and, hopefully, extract the amplitude of the waveform, and also calculate other characteristics, such as average frequency and approximate BPM, from this data to create a level.

I have no problem trying to work with this raw data, I just want to know how I can GET this information in the first place. If I can extract samples and determine the different characteristics of these samples, I can save them and work out changes in volume, pitch, and much more to create notes, etc.

I use C #, but if at all possible, I can either use p / invoke or switch my project to another device using C ++ instead of C #.

I'm a little panicky here because I'm really a little puzzled.

Thanks so much guys.

Answer 1

The problem you are describing here is one of extracting music / audio, and there is a significant amount of academic work you can build on. Another useful art term to look for is to search for music information (MIR).

The list of “features” that researchers have tried to extract from the records is large and varied, from deterministic things such as step and key, through emotional characteristics such as “energy”.

Most of them turned out to be more complicated than you could imagine, and, as a rule, only 60-70% more accurate - although this is probably enough for your requirements.

A good entry point is to download Sonic Visualiser , for which there are a large number of modules for extracting functions, and they are open-source, you will at least feel that it is possible.

Update. Another useful art term is Onset detection - this is commonly used to describe beat detection algorithms.

Answer 2

Unfortunately, I don’t think you can use C # for this - AFAIK, there is no JIT compiler for it. I remember reading about something for Mono that would make it available for use with C #, but I'm not sure right now.

That says - I would go with C ++. If you go this way, you can use a huge number of sound analysis libraries such as CLAM (http://clam-project.org/).

Do not panic (submit large, friendly letters.) Provide the necessary details for the project step by step, solve it one by one, and you will do it as soon as possible. =)

Answer 3

Aubio is a C / C ++ library that, among other things, tracks step, start detection and bpm tracking.

Regarding the “extraction of waveform amplitude", the waveform is amplitude, i.e. you can simply select the sound sample with the largest absolute value for each n samples and use this value to perform the “amplitude” part of the visualization.

Here is the code that can help you start reading WAVE data in C #.

Here is some information on writing a C # wrapper for the FFTW library.