Create mp4 files on Android using Jcodec

Question

I have some problems writing mp4 files on Android using MediaRecorder and Jcodec, here is my code

public class SequenceEncoder { private final static String CLASSTAG = SequenceEncoder.class.getSimpleName(); private SeekableByteChannel ch; private byte[] yuv = null; private ArrayList<ByteBuffer> spsList; private ArrayList<ByteBuffer> ppsList; private CompressedTrack outTrack; private int frameNo; private MP4Muxer muxer; ArrayList<ByteBuffer> spsListTmp = new ArrayList<ByteBuffer>(); ArrayList<ByteBuffer> ppsListTmp = new ArrayList<ByteBuffer>(); // Encoder private MediaCodec mediaCodec = null; public SequenceEncoder(File out) throws IOException { this.ch = NIOUtils.writableFileChannel(out); // Muxer that will store the encoded frames muxer = new MP4Muxer(ch, Brand.MP4); // Add video track to muxer outTrack = muxer.addTrackForCompressed(TrackType.VIDEO, 25); // Encoder extra data ( SPS, PPS ) to be stored in a special place of // MP4 spsList = new ArrayList<ByteBuffer>(); ppsList = new ArrayList<ByteBuffer>(); } @SuppressWarnings("unchecked") public void encodeImage(ByteBuffer buffer, int width, int height) throws IOException { if (yuv == null) { int bufferSize = width * height * 3 / 2; yuv = new byte[bufferSize]; int bitRate = bufferSize; int frameRate = 25; String mimeType = "video/avc"; // "video/avc" mediaCodec = MediaCodec.createEncoderByType(mimeType); MediaFormat mediaFormat = MediaFormat.createVideoFormat(mimeType, width, height); mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitRate); // 125000); mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, frameRate); mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar); mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5); mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); mediaCodec.start(); } byte[] rgba = buffer.array(); // Convert RGBA image to NV12 (YUV420SemiPlanar) Rgba2Yuv420.convert(rgba, yuv, width, height); synchronized (mediaCodec) { try { ByteBuffer[] inputBuffers = mediaCodec.getInputBuffers(); ByteBuffer[] outputBuffers = mediaCodec.getOutputBuffers(); int inputBufferIndex = mediaCodec.dequeueInputBuffer(-1); if (inputBufferIndex >= 0) { ByteBuffer inputBuffer = inputBuffers[inputBufferIndex]; inputBuffer.clear(); inputBuffer.put(yuv); mediaCodec.queueInputBuffer(inputBufferIndex, 0, yuv.length, 0, 0); } MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo(); int outputBufferIndex = mediaCodec.dequeueOutputBuffer( bufferInfo, 0); while (outputBufferIndex >= 0) { ByteBuffer outputBuffer = outputBuffers[outputBufferIndex]; byte[] outData = new byte[bufferInfo.size]; outputBuffer.get(outData); ByteBuffer frameBuffer = ByteBuffer.wrap(outData); spsListTmp.clear(); ppsListTmp.clear(); H264Utils.encodeMOVPacket(frameBuffer, spsListTmp, ppsListTmp); if (!spsListTmp.isEmpty()) spsList = (ArrayList<ByteBuffer>) spsListTmp.clone(); if (!ppsListTmp.isEmpty()) ppsList = (ArrayList<ByteBuffer>) ppsListTmp.clone(); outTrack.addFrame(new MP4Packet(frameBuffer, frameNo, 25, 1, frameNo, true, null, frameNo, 0)); frameNo++; mediaCodec.releaseOutputBuffer(outputBufferIndex, false); outputBufferIndex = mediaCodec.dequeueOutputBuffer( bufferInfo, 0); } if (outputBufferIndex < 0) switch (outputBufferIndex) { case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: outputBuffers = mediaCodec.getOutputBuffers(); break; case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: break; case MediaCodec.INFO_TRY_AGAIN_LATER: break; default: break; } } catch (Exception e) { } } } public void finish() throws IOException { if (!ch.isOpen()) return; if (mediaCodec != null) { mediaCodec.stop(); mediaCodec.release(); } outTrack.addSampleEntry(H264Utils.createMOVSampleEntry(spsList, ppsList)); // Write MP4 header and finalize recording muxer.writeHeader(); NIOUtils.closeQuietly(ch); ch.close(); } } 

As we can see, Android MediaCodec expects the YUV420SemiPlanar to be used as an input image, so I give it the correct one. As a result, I have a damaged mp4 file with invalid colors when I open this mp4 file from AVCon. I see that the color format in the output file is yuv420p, so maybe that is the problem? Please suggest how to fix this.

There is also another question: how to add a compressed audio stream to the multiplexer, we did not find examples.

Answer 1

Android 4.3 (API 18) has two new features that may be useful.

Firstly, the MediaCodec class accepts input from the surface, so anything you can decode to the surface or render using OpenGL ES can be written without having to paint with YUV color planes.

Secondly, the new MediaMuxer class provides a way to combine H.264 audio and video into a .mp4 file.

An example of the source code (mainly for video aspects) can be found here .

Answer 2

YUV420SemiPlanar 4x4 is a format like YYYYYYYYYYYYYYYYYUVUVUVUV, not YYYYYYYYYYYYYYYYYUUVVVV. I can get the mp4 file with the correct color using Jcodec and MediaCodec on Android after I transferred the image with the format.

I have no answer about audio.