I'd like to use the determinant of a matrix to draw a plane in the fragment shader. But I don't understand how to come up with the matrix. Let's say that I have 3 points that lie on the plane: A, B, and C. I don't know how to combine them into the matrix I need.
// "point" represents the current pixel in the shader. I tried this:
mat4 matrix = mat4(vec4(A.x, A.y, A.z, 1.0),
vec4(B.x, B.y, B.z, 1.0),
vec4(C.x, C.y, C.z, 1.0),
vec4(point.x, point.y, point.z, 1.0));
float result = determinant(matrix);
if (result == 0.0) {
// Color the plane.
ALBEDO = vec3(.3, .3, .3);
}