How to use CoreAudio AudioConverter to encode AAC in real time?

Question

The entire sample code I can find that uses AudioConverterRef focuses on use cases where I have all the data up (for example, converting a file to disk). They are usually called AudioConverterFillComplexBuffer when PCM is converted to inInputDataProcUserData and just populates it in a callback. (Is this really the way it is supposed to be used? Why would he need a callback?) In my use case, I am trying to transmit aac audio from a microphone, so I don't have a file and my PCM buffer is full in real time.

Since I don't have all the data up, I tried doing *ioNumberDataPackets = 0 in the callback when my input is missing, but it just puts the AudioConverter in a dead state, where it should be AudioConverterReset() ted, and I don't get any data from him.

One approach I've seen on the Internet is to return an error from the callback if the data I saved is too small, and just try again as soon as I have more data, but it seems such a waste of resources that I can’t force myself to even try it.

Do I really need to perform a “retry until my input buffer is large enough,” or is there a better way?

Answer 1

AudioConverterFillComplexBuffer doesn't really mean "populate the encoder with my input buffer , which I have." This means "fill this output buffer here with encoded data from the encoder." From this point of view, the callback suddenly makes sense - it is used to extract the source data to satisfy the request "fill this output buffer for me." This may be obvious to others, but it took me a long time to figure it out (and from the entire AudioConverter example code that I see, it floats around where people send input through inInputDataProcUserData , I assume I'm not the only one).

The AudioConverterFillComplexBuffer call is blocked and expects you to pass data to it synchronously from the callback. If you are coding in real time, you will need to call FillComplexBuffer in a separate thread that you configured yourself. In the callback, you can check the available input, and if it is not available, you need to lock the semaphore. Using NSCondition, the encoder stream will look something like this:

 - (void)startEncoder { OSStatus creationStatus = AudioConverterNew(&_fromFormat, &_toFormat, &_converter); _running = YES; _condition = [[NSCondition alloc] init]; [self performSelectorInBackground:@selector(_encoderThread) withObject:nil]; } - (void)_encoderThread { while(_running) { // Make quarter-second buffers. size_t bufferSize = (_outputBitrate/8) * 0.25; NSMutableData *outAudioBuffer = [NSMutableData dataWithLength:bufferSize]; AudioBufferList outAudioBufferList; outAudioBufferList.mNumberBuffers = 1; outAudioBufferList.mBuffers[0].mNumberChannels = _toFormat.mChannelsPerFrame; outAudioBufferList.mBuffers[0].mDataByteSize = (UInt32)bufferSize; outAudioBufferList.mBuffers[0].mData = [outAudioBuffer mutableBytes]; UInt32 ioOutputDataPacketSize = 1; _currentPresentationTime = kCMTimeInvalid; // you need to fill this in during FillComplexBuffer const OSStatus conversionResult = AudioConverterFillComplexBuffer(_converter, FillBufferTrampoline, (__bridge void*)self, &ioOutputDataPacketSize, &outAudioBufferList, NULL); // here I convert the AudioBufferList into a CMSampleBuffer, which I've omitted for brevity. // Ping me if you need it. [self.delegate encoder:self encodedSampleBuffer:outSampleBuffer]; } } 

And the callback might look like this: (note that I usually use this trampoline to immediately jump to the method of my instance (by redirecting my instance to inUserData , this step is omitted for brevity)):

 static OSStatus FillBufferTrampoline(AudioConverterRef inAudioConverter, UInt32* ioNumberDataPackets, AudioBufferList* ioData, AudioStreamPacketDescription** outDataPacketDescription, void* inUserData) { [_condition lock]; UInt32 countOfPacketsWritten = 0; while (true) { // If the condition fires and we have shut down the encoder, just pretend like we have written 0 bytes and are done. if(!_running) break; // Out of input data? Wait on the condition. if(_inputBuffer.length == 0) { [_condition wait]; continue; } // We have data! Fill ioData from your _inputBuffer here. // Also save the input buffer start presentationTime here. // Exit out of the loop, since we're done waiting for data break; } [_condition unlock]; // 2. Set ioNumberDataPackets to the amount of data remaining // if running is false, this will be 0, indicating EndOfStream *ioNumberDataPackets = countOfPacketsWritten; return noErr; } 

And for completeness, here is how you could serve this data encoder and how to disable it correctly:

 - (void)appendSampleBuffer:(CMSampleBufferRef)sampleBuffer { [_condition lock]; // Convert sampleBuffer and put it into _inputBuffer here [_condition broadcast]; [_condition unlock]; } - (void)stopEncoding { [_condition lock]; _running = NO; [_condition broadcast]; [_condition unlock]; } 

Answer 2

For future reference, there is an easier way.

CoreAudio Header Status:

If the callback returns an error, it should return null data packets. AudioConverterFillComplexBuffer will stop production and return all output has already been issued to its caller along with an error code. This mechanism can be used when the input proc has temporarily exhausted the data but has not yet reached the end of the stream.

So do just that. Instead of returning noErr with * ioNumberDataPackets = 0, return any error (just do one, I used -1) and the converted data will be returned while Audio Converter will stay alive and should not be reset.