Android Asynctask for processing live video frames

Question

I am using OpenCV to attempt real-time video processing. Since the processing is quite heavy, it significantly slows down the output frames, which makes the real-time stream look choppy.

I would like to offload part of the processing in AsyncTask. I tried this and it actually makes the video a lot smoother. However, he finishes launching a large number of Tasks at the same time, and then they will slowly return with some results.

Is there a way to slow this down and wait for the result, either using the Synchronize statements, or in some other way?

On each frame of the camera, I run one of these tasks. DoImGProcessing performs long processing and returns the result of the string.

private class LongOperation extends AsyncTask<Mat, Void, String> { @Override protected String doInBackground(Mat... params) { Mat inputFrame = params[0]; cropToCenter(inputFrame); return doImgProcessing(inputFrame); } @Override protected void onPostExecute(String result) { Log.d(TAG, "on post execute: "+result); } @Override protected void onPreExecute() { Log.d(TAG, "on pre execute"); } } public Mat onCameraFrame(Mat inputFrame) { inputFrame.copyTo(mRgba);//this will be used for the live stream LongOperation op = new LongOperation(); op.execute(inputFrame); return mRgba; } 

Answer 1

I would do something like this:

 // Example value for a timeout. private static final long TIMEOUT = 1000L; private BlockingQueue<Mat> frames = new LinkedBlockingQueue<Mat>(); Thread worker = new Thread() { @Override public void run() { while (running) { Mat inputFrame = frames.poll(TIMEOUT, TimeUnit.MILLISECONDS); if (inputFrame == null) { // timeout. Also, with a try {} catch block poll can be interrupted via Thread.interrupt() so not to wait for the timeout. continue; } cropToCenter(inputFrame); String result = doImgProcessing(inputFrame); } } }; worker.start(); public Mat onCameraFrame(Mat inputFrame) { inputFrame.copyTo(mRgba);//this will be used for the live stream frames.put(inputFrame); return mRgba; } 

OnCameraFrame puts the frame in the queue, a polling poll from the queue.

This will decorate the reception and processing of the frame. You can track the growth of the queue using frames.size() .

This is a typical example of a consumer producer.

Answer 2

If you do this on every frame, it looks like you need a stream. AsyncTask is designed when you want to do a one-time activity on another thread. Here you want to do it again. Just create a thread, and when it finishes the frame, it will send a message to the handler to start the post-step in the user interface thread. He can wait for the semaphore at the top of his loop so that the next frame is ready.