Short answer:
The condition is that p3 must be on the right side of the plane defined by (p0, p1, p2) .
So, after calculating the normal for this plane, you need to determine whether the vector from (say) p0 to p3 points in the same normal direction or in the opposite direction, taking the product point dot(normal, p3-p0) .
More mathematically:
You need to find the determinant of a 4x4 matrix formed by the homogeneous coordinates of four points. The sign of the determinant determines whether the condition is met; the corresponding sign depends on the specific conventions used, but ideally it should be positive:
require: 0 < det(p0, p1, p2, p3) == det [ p0.x p0.y p0.z 1 ] [ p1.x p1.y p1.z 1 ] [ p2.x p2.y p2.z 1 ] [ p3.x p3.y p3.z 1 ]
If a certain ordered set of points has a negative determinant, you can fix it by replacing any two points (which negates the determinant):
eg, swapping p0 and p2: det(p0, p1, p2, p3) = - det(p2, p1, p0, p3) ^ ^ ^ ^
or, in a more general sense, switching between even and odd permutations of four vertices.
If the determinant is zero, the four points are coplanar and cannot be fixed in this way.
Finally, the code:
A relatively simple way to compute this determinant with 3rd vector math:
let: v1 = p1 - p0 v2 = p2 - p0 v3 = p3 - p0 norm12 = cross(v1, v2) -> determinant = dot(norm12, v3)
The final determinant is also known as the "triple product" v1, v2 and v3.
Please note that I did not dare to try to decode the exact agreement on the sign (i.e. whether you need the determinant to be positive or negative) from your question: the wording and diagram that you supply are more than confused.
Since you have the original library and its documentation, you are, however, in a better position to answer this question. In extreme cases, you can try the empirical method: try both signs and choose the one that does not explode ...