Determining whether an object is part of a moving object from a sparse optical stream (KLT)

Question

I did retrieve the optical stream from a sequence of images using KLT. The camera is on a moving vehicle filming a transport scene. Now I can get the results of the comparison without inconsistent points.

I just use OpenCV functions (C ++) for function extraction and tracking. I have no programming problems.

cvGoodFeaturesToTrack (), cvFindCornerSubPix (), cvCalcOpticalFlowPyrLK ()

Can anyone suggest which topic / article should be searched to distinguish whether matching points that are part of a moving object or not correspond? I do not want to restore camera movement. Just to check if the point is from a moving object or not.

I do not have an external resource, such as a rangefinder or other sensors, but I know the speed of the car (mounted by the camera) and the angle at which the camera is mounted on the vehicle if it leans up or down the road. (I don't know yaw rate though)

Checking the direction and distance of the optical flow is insufficient. For example, the optical streams of vehicles moving in the opposite section are often similar to stationary elements or a moving vehicle with an equivalent speed as a camera.

What topic should I learn to handle this? If I need to choose between FOE or Kalman filter, otherwise? I wonder if RANSAC will help. (I read several scientific articles, but it seemed unlucky. I looked at the Kalman filter, but I'm not sure how this can help. I also found the resource on FOE quite limited, especially in the tutorial).

I appreciate all the answers. Thanks a lot (sincerely)

Answer 1

Not sure if this is not suitable for answering my own question. But, since no one answers this question, and I did some research work, not a success story, I think it's better to share some. This may be useful for those who have the same problem.

I have a sequence of images capturing the scene of urban transport. Images are captured using a smartphone on a moving car every 0.5 seconds.

For testing purposes, I use only a few pairs of images for testing instead of a whole sequence. I got some consistent points using KLT and performed a two-step deletion deletion. The corresponding results are good, not a single, very few inconsistencies.

To reject points on moving objects, I followed the work presented below:

Jung B. and Sukhatme, GS, 2004. "Detecting moving objects using a single camera on a mobile robot in an outdoor environment" (the revised version presented in the journal is called "real-time motion tracking from a mobile robot")

To the resume, in terms of their work, they reject outliers (moving objects) by calculating a transformation model between pairs of images. In the work, a bilinear model was used. The procedure is that they calculate the parameters of the transformation model T and reject the consistent ones if | x2 - T (x1) | <threshold. Here x2 and x1 mean a pair of corresponding points in the image at time t2 and t1.

I tried T as an affine model, a bilinear model, and a pseudo-perspective model. My experimental results show that if the number of moving objects is small, this procedure will always fail because they rely on consistent points. In my case, images are captured on a city highway in which there are a lot of moving objects. Therefore, I cannot refuse emissions because of this technique. Therefore, I believe that RANSAC will not help either. Therefore, many works involve a small number of moving objects. Among these three models, I found that affinity shows show the worst result, but cannot say which one is better among the other two.

Hope this helps.