This is way too complicated.

Let's say the screen center is at (0,0), the asteroid coords are x,y, and the screen size is w,h. Then this should work:

float x2 = x / w;
float y2 = y / h;
float s = max(x2, y2);
float x3 = x2 / s * w / 2;
float y3 = y2 / s * h / 2;

x3,y3 are the desired coords.

This is way too complicated.

Let's say the screen center is at (0,0), the asteroid coords are x,y, and the screen size is w,h. Then this should work:

float x2 = x / w;
float y2 = y / h;
float s = max(abs(x2), abs(y2));
float x3 = x2 / s * w / 2;
float y3 = y2 / s * h / 2;

x3,y3 are the desired coords.