Overflow of 32 bit variable

If the number is too big for a 32 bit variable, then you either use more bits (either by storing in a bigger variable, or using multiple variables) or you give up precision and store it in a float. Since Y can be MAX_INT, by definition you can't multiply it by a number greater than 1 and still have it fit in a 32 bit int.

I'd use loop, instead of multiplication, in this case. Something like this:

int newX = X;
int poweredB = ( 1 << B ); // 2^B
for( int i = 0; i < poweredB ; ++i )
{
    newX += Y; // or change X directly, if you will not need it later.
}
int result = ( 1 << A ) * newX;

But note : this will work only for some situations - only if you have the guarantee, that this result will not overflow. In your case, when Y is large positive and X is large negative number ("large" - argh, this is too subjective), this will definitely work. But if X is large positive and Y is large positive - there will be overflow again. And not only in this case, but with many others.

Based on the values for A and B in the assignment I suppose the expected solution would involve this:

• the following are best done unsigned so store the signs for X and Y and operate on their absolute value

• Store X and Y in two variables each, one holding the high 16 bits the other holding the low bits something like
  int hiY = Y & 0xFFFF0000;

  int loY = Y & 0x0000FFFF;

• Shift the high parts right so that all the variables have the high bits 0

• Y*(2^B) is actually a shift of Y to the left by B bits. It is equivalent to shifting the high and low parts by B bits and, since you've shifted the high part, both operations will fit inside their 32 bit integer

• Process X similarly in the high and low parts

• Keeping track of the signs of X and Y calculate X + Y*(2^B) for the high and low parts

• Again shift both the high and low results by A bits

• Join the high and low parts into the final result

If you can't use 64-bits because your local C does not support them rather than some other overriding reason, you might consider The GNU Multiple Precision Arithmetic Library at http://gmplib.org/