Replacing the detected object in the frame with an image. (ImageProcessing)

Question

Overview:

I am working on a video project. The technology I use: imageMagick, php, ffmpeg.

Current status:

Currently, the project is able to create videos using images and texts and several basic transitions. The way I do this is to use imagemagick to create a gif using input images (with transition effects in them), and then convert all the gifs into videos and combine the videos together at the same time.

The following transition (My question):

Now I am set to go to the next level. So, I have a video (1920x1080) with some white frames (1280x720) that keep the offset in each frame. I want to replace these white frames appearing in some frames of the video with some images (1280x720) that I want to use. Please see the image here and you will get an idea: these are just two frames from my video. If you see that the images are shifted (space is not constant).

Expectation:

So, I want to fill these gaps with one of my own images. If the case were for only one frame, I could use ffmpeg to overlay the image at the exact width and height. But here the empty space is not fixed and continues to shift in all frames, and there are many frames. So, I'm looking for something like opencv or some other technology that can be used to detect an object in a video or in a frameset and replace the detected area with some other image.

I just need a punch. So, if someone has already worked on something like this, just suggest me which technology I can use. Thanks in advance.

Answer 1

It all depends on what you can assume:

If you can safely assume that your rectangle border is never closed (hidden) in any way, you can try to find the edges in your image (for example, the edge of the OpenCV Canny), and then look for a rectangular shape (corners forming a distorted rectangle, or very popular Hough lines).

If the rectangle you are looking for is always white, you can threshold image in a color space such as HSV to look for the maximum value (V in HSV brightness ~), and then look for the rectangular shape in the binary image.

If your corners are occluded, sometimes you will have to make some settings with your image, such as morphological operations (“grow and shrink” the binary threshold image), then Hough Lines can do the trick.

Please note that this answer assumes that as soon as you find out where the rectangle is, you are “finished” and you just need to rewrite the rectangle using special content. I also do not check for continuity: the video frame may skip depending on the appearance of the rectangle. You will need to add some information about previous positions.