Augmented Reality SDK with OpenCV

Question

I am developing an Augmented Reality SDK on OpenCV. I had some problems to find tutorials on this topic, what steps to follow, possible algorithms, fast and efficient coding for working in real time, etc.

So far, I have gathered the following information and useful links.

Install OpenCV

Download the latest version .

You can find installation guides here (platforms: linux, mac, windows, java, android, iOS).

Online documentation .

Augmented Reality

For starters, here is a simple augmented reality code in OpenCV. This is a good start.

For those who are looking for a well-designed modern SDK, I found several common steps that each augmented reality based on marker tracking should take into account, taking into account OpenCV features.

• The main program: creates all classes, initialization, captures frames from video.

• Class AR_Engine: controls parts of an augmented reality application. There must be two main states:

  • detection : tries to detect a marker in the scene
  • tracking : once it is detected, it uses lower computational methods to track the marker in future frames.

There should also be some algorithms for finding the position and orientation of the camera in each frame. This is achieved by detecting a homography transformation between the marker found in the scene and the two-dimensional image of the marker that we processed offline. An explanation of this method is here (p. 18). The main steps to assess posture:

• Download the built-in camera settings . Previously retrieved offline through calibration.

• Download the tracking template (marker): this is the image of the planar marker that we will track. It is necessary to extract functions and generate descriptors ( key points ) for this template, so that later we can compare with functions from the scene. Algorithms for this task:

  • SIFT
  • Fast
  • Surf

• For each frame update, run the function extraction detection algorithm from the scene and create descriptors. Again we have several options.

  • SIFT
  • QUICKLY
  • Surf
  • FREAK : the new method (2012) has become the fastest.
  • ORB

• Find a match between the template and the scene descriptors.

  • TIP FLANN

• Find the Homography matrix from these matches. RANSAC can be used earlier to find sheets / outliers in a set of matches.

• Extract camera poses from homography.

  • Sample code Pose from homography .
  • Sample code Homography from a pose .

Complete examples:

• aruco
• Mastering OpenCV Samples

Answer 1

Since AR applications often run on mobile devices, you can consider other functions: detector / descriptor:

• Freak
• ORB

Answer 2

In general, if you can select markers, you first detect a square target using an edge detector, and then either Hough or just outlines - then identify a specific marker from the interior design. Instead of using a common point connector.

Take a look at Aruco 's well-written sample code.