Doing some shorthand and assuming point C is on the z axis without loss of generality,
Vector3 AB = Vector3(Bx, By, Bz) - Vector3(Ax, Ay, Az);
Vector3 Caxis = Vector3(0, 0, Cz); // This is the point C
The direction of the result vector C we seek is the same direction as AB cross (AB cross Caxis)
Vector3 Cdir = Vector3.Cross(AB, Vector3.Cross(AB, Caxis)).Normalize();
Finding the magnitude is more involved. We know that starting at the point Cz and going in the direction Cdir must intersect along AB
Caxis - alpha * Cdir == beta * AB
and we need to find unknowns alpha and beta. We can find a vector perpendicular to Caxis by projecting AB onto the plane to which Caxis is normal
Vector3 Cperp = Vector3.ProjectOnPlane(Caxis, AB).Normalize();
so our equation with alpha and beta can be reduced to one unknown
- alpha * (Cdir . Cperp)/(AB . Cperp) == beta
So solving for alpha (which is in fact the magnitude of our final vector C)
Caxis = alpha * (Cdir - AB* (Cdir . Cperp)/(AB . Cperp) )
This is a vector equation, but we can solve for alpha by taking the dot product with Caxis on both sides. Writing code again this is
float num = Caxis.magnitude*Caxis.magnitude;
float denom1 = Vector3.Dot(Cdir, Caxis);
float denom2 = Vector3.Dot(AB, Caxis) * Vector3.Dot(Cdir, Cperp) / Vector3.Dot(AB, Cperp);
float Cmag = num/(denom1 - denom2);
C is then, using the sign convention from your diagram,
Vector3 C = Caxis - Cmag*Cdir;