What you ask for is impossible numerically. Rounding errors make such a test completely inconsequential.
However, you can check "if two triangles are on the same plane, within a certain tolerance." This is very difficult to do, and rounding errors here are also likely to spoil any method. Indeed, whenever triangles are thin, in the plane on which they live, great uncertainty arises.
I could point you to some letter if you really want (it would be best to look at the CGAL library and see if they have implemented something that is relevant to your problem). Everything, most likely, is connected with exact floating points of accuracy, smart reordering of operations and in any case will lead to fuzzy results.
Therefore, I highly recommend that you find a different approach to your real problem.
Rounding errors are a (huge) problem if you try to calculate the equation of a plane passing through three points and then checking the other three. There is one more solution.
You can calculate the inertial matrix of your six points, diagonalize it and see if its smallest eigenvalue is within the limits of some small value of the other two. This will mean that your six points actually lie on the same plane within the tolerance.
source share