The internal matrix of a 3x3 camera is used to compare the coordinates in the image with the physical world coordinates. In the same way, the role of the main point in this matrix is the display of the “intersection of the optical axis with the image plane”, between the image coordinates and the physical world coordinates. Ideally, the principal point is in the center of the image for most cameras, but in practice this is not always the case. The principal point may be slightly disconnected from tangential distortion or imperfect centering of the lens components and other manufacturing defects. The internal matrix of a 3x3 camera is trying to correct this distortion.
I found this site to help me when I found out about camera calibration. Although it is in MATLAB, it is based on the same camera calibration as in OpenCV.
source share