This is way too complicated.
Let's say the screen center is at (0,0), the asteroid coords are x,y, and the screen size is w,h. Then this should work:
float x2 = x / w;
float y2 = y / h;
float s = max(abs(x2), abs(y2));
float x3 = x2 / s * w / 2;
float y3 = y2 / s * h / 2;
x3,y3 are the desired coords.