I'm a little late for this party. But I ran into this problem when I was looking for the same problem. I spent the whole day on this and worked, although some excellent articles I found here are: http://www.antongerdelan.net/opengl/virtualcamera.html
I finished working with the projection matrix and worked back. I got the same formula that you mentioned in your post above. (where cot (x) = 1 / tan (x))
radius_pixels = (radius_worldspace / {tan(fovy/2) * D}) * (screen_height_pixels / 2)
(where D is the distance from the camera to the target boundary sphere)
I use this approach to determine the radius of an imaginary trackball that I use to rotate my object.
Btw Florian, you can extract fovy from the Projection matrix as follows:
If you take the Sy component from the projection matrix, as shown below:
Sx 0 0 0 0 Sy 0 0 0 0 Sz Pz 0 0 -1 0 where Sy = near / range and where range = tan(fovy/2) x near
(you can find these definitions on the page above)
if you replace the range in Sy eqn above, you will get:
Sy = 1 / tan(fovy/2) = cot(fovy/2)
permutations:
tan(fovy/2) = 1 / Sy
taking arctan (reverse tan) on both sides, we get:
fovy/2 = arctan(1/Sy)
So,
fovy = 2 x arctan(1/Sy)
Not sure if you care - it has been a while! - but maybe it will help someone else.
source share