Given the input vectors
Vector3 AB = Vector3(Bx, By, Bz) - Vector3(Ax, Ay, Az);
Vector3 Caxis = Vector3(Cx, Cy, Cz) - Vector3(Ax, Ay, Az); // This is the vector from A to C
The direction of the result vector C we seek is the same direction as (AB cross Caxis) cross AB
Vector3 Cdir = Vector3.Cross(Vector3.Cross(AB, Caxis), AB).Normalize();
Finding the magnitude is more involved. We know that starting at the point Cz and going in the direction Cdir must intersect along AB
Caxis - alpha * Cdir == beta * AB
and we need to find unknowns alpha and beta. We can find a vector perpendicular to Caxis by projecting AB onto the plane to which Caxis is normal
Vector3 Cperp = Vector3.ProjectOnPlane(Caxis, AB).Normalize();
so our equation with alpha and beta can be reduced to one unknown
- alpha * (Cdir . Cperp)/(AB . Cperp) == beta
So solving for alpha (which is in fact the magnitude of our final vector C)
Caxis == alpha * (Cdir - AB* (Cdir . Cperp)/(AB . Cperp) )
This is a vector equation, but we can solve for alpha by taking the dot product with Caxis on both sides. Writing code again this is
float num = Caxis.magnitude*Caxis.magnitude;
float denom1 = Vector3.Dot(Cdir, Caxis);
float denom2 = Vector3.Dot(AB, Caxis) * Vector3.Dot(Cdir, Cperp) / Vector3.Dot(AB, Cperp);
float Cmag = num/(denom1 - denom2);
C is then, using the sign convention from your diagram,
Vector3 C = Cmag*Cdir;
I'm leaving it without checks for cases where you get a divide by zero, such as when points B and C are identical.