How to get step alignment and Y values ​​for MediaCodec encoder

Question

There are several related questions and discussions on this topic:

• Mediacodec and camera, wrong color space
• Getting QualComm encoders to work through the MediaCodec API
• https://groups.google.com/d/msg/android-platform/awaNwgb6EbY/a-YiIOwaL0QJ
• https://code.google.com/p/android/issues/detail?id=37769

I NV21 camera preview frames ( NV21 converted to NV12 ) in the MediaCodec encoder ( NV12 aka COLOR_FormatYUV420SemiPlanar ). It seems that on some QualComm encoder QualComm that run under Android versions less than 4.3 , I need to do input frame processing to get the inverse frame with the correct color.

In Sony Xperia ZR running Android 4.2.2 I need to add Y plane alignment so that it works with almost all resolutions. The above code adds byte alignment of 1024 for widths that cannot be divided into 32 and 2048 byte alignment for other resolutions. It allows MediaCodec correctly encode frames for all resolutions that can be divided by 16 (except 176x144 , for which the UV plane looks inconsistent).

 int getYPadding() { if (mediaCodecInfo.getName().contains("OMX.qcom") && android.os.Build.VERSION.SDK_INT < 18) { if ((getWidth() % 32) != 0) { return (getWidth()*getHeight()) % 1024; } else { return (getWidth()*getHeight()) % 2048; } } return 0; } 

I tried to check this alignment on the LG G2 , which works with the same Android 4.2.2 and has a QualComm encoder, and it looks like it is working incorrectly. The UV plane is offset (green bar at the bottom of the frame). I was not able to calculate the gasket that will work for both phones.

I also have access to the Sony Xperia Z1 running Android 4.3 with the QualComm chipset, and it looks like it doesn't have such problems. The video at each resolution looks great, and the Y plane should not be aligned anyway.

I understand that this is hardware related and can be complicated, but since I have to support users running Android up to 4.3 , I have a question. Is it possible to programmatically determine the alignment of the Y plane and the vertical / horizontal pitch values ​​that the encoder expects for a given color format?

Answer 1

The problem in a nutshell: there were no CTS tests for encoding video up to Android 4.3 (API 18).

As a result, the behavior of MediaCodec on different devices was inconsistent, and several errors went unnoticed. The EncodeDecodeTest tests perform the functions you ask for, and as a result, you can reliably download YUV data to the device with 4.3+ (although you still need to work -detect whether it wants to be planar or half-flat).

For your specific question, the Y plane on older Qualcomm devices should be aligned at the 2K border, which is not exactly what your code does. For 720p video, this is natural (720 * 1280 == 450 * 2048), for 176x144 you must set it to 1280 to run the UV plane at 26624 instead of 25344. You need to set the absolute alignment in the buffer, and not a fixed amount of indentation - use uvoffset = (width*height + 2047) & ~2047 .

You will need to determine the software version of the codec and the software version for Android, and if it is Qualcomm on pre-4.3, you will need to complete this setup. If your requirements change and you can target API 18+, these problems will go away. (And you can use the Surface input for MediaCodec , which avoids the U / V exchange problem, although depending on your needs, that might not be useful to you.)