Does anyone have any tips for programming real-time sound synthesis?

Question

I am currently working on a personal project: I am creating a library for real-time sound synthesis in Flash. In short: tools for connecting wavegenarators, filters, mixers, etc. With eachother and supplying a sound card with raw (real-time) data. Something like max / msp or Reaktor.

I already have some working material, but I am wondering if the basic setup I wrote is correct. I do not want to run into problems later to force me to change the core of my application (although this can always happen).

Basically, what I'm doing now starts at the end of the chain, in the place where the (source) audio data goes out (to the sound card). To do this, I need to write ByteArrays for the object, and to get this piece, I ask which module is connected to my Sound module to give me my own piece. This module makes the same request to the module that is connected to its input, and this continues until the beginning of the chain is reached.

Is this the right approach? I can imagine problems with work if there is feedback, or if there is another module without output: if I had connected a spectrum analyzer somewhere, it would be a dead end in the chain (module without outputs, just an input). In my current installation, such a module does not work, because I am just starting to calculate the sound output from the module.

Does anyone have any experience programming something like this? I would be very interested to know about the right approach. (For clarity: I am not looking for specific Flash implementations, and therefore I did not put this question under flash memory or actioncript)

Answer 1

I did a similar thing a while ago, and I used the same approach as you - start with a virtual line and trace the signal to the top. I did this on the sample, but not on the buffer; if I were to write the same application today, I could choose one buffer instead, because I suspect that it will work better.

The spectrometer was designed as an insertion module, that is, it would work only if its input and its output were connected, and it did not transmit its input to the output unchanged.

To handle the feedback, I had a special auxiliary module that introduced a delay of 1 sample and would only extract its input once per cycle.

Also, I think that doing all your internal processing using float and thus float arrays as buffers would be much simpler than byte arrays and this will save you the extra effort of converting integers and floats all the time .

Answer 2

In later versions, you may have different rates in different parts of the network.

One example might be if you expand it to transfer data to or from disk. Another example would be that small data rate control variables, such as a single control echo delay, may later become part of your network. You probably do not want to process control variables at the same frequency as for processing audio packets, but they are still “real time” and part of the functional network. They may, for example, need smoothing to avoid sudden transitions.

As long as you call all of your functions at the same speed, and all the functions are mostly performed continuously, your data pull approach will work fine. There will be little to choose between pulling data and clicking. Pulling is somewhat more natural for reproducing sound, pressing is more natural for recording, but it either works or ends with the same call to the basic functions of sound processing.

• For a spectrometer , you have a multiple absorber problem for data, but that is not a problem. Enter a fictitious link to it from the real shell. A dummy link may cause a request for data that is not an honor. As long as the dummy link knows this dummy and does not care about the lack of data, everything will be OK. This is a standard technique for reducing multiple receivers or sources to one.

• In such a network, you do not want to do the same calculation twice in one full update. For example, if you mix a high-frequency and a low-profile version of a signal, you do not want to evaluate the original signal twice. You should do something like write a timer value with each buffer and stop spreading the excerpts when you see the current checkmark value. The same mechanism will also protect you from feedback loops in the evaluation.

So, these two issues that concern you are easily resolved within your current structure.

Matching speeds , where there are different packet rates in different parts of the network, problems with the current approach will occur. If you write audio to disc, then for efficiency you want to write large chunks infrequently. You do not want to block the maintenance of more frequent small input and output data processing packets during these write operations. A single pulling or pushing strategy will not be enough.

Just admit that at some point you may need a more sophisticated upgrade method than a single speed network. When this happens, you will need threads for different bets to be launched, or you will write your own simple scheduler, perhaps as simple as calling less frequently performed functions once in n, so that the bets match. You do not need to plan this in advance. Your audio functions will almost certainly already delegate responsibility for making their input buffers ready for other functions, and these will be only those other functions that need to be changed, and not the audio functions themselves.

The only thing I would like to advise at this stage is to be careful to centralize the sound buffer , noting that the buffers are like fence fences. They do not belong to the audio function, they lie between the audio functions. Centralizing the distribution of the buffer makes it easy to retrospectively modify the update strategy for different speeds in different parts of the network.