why does a type alias to engine specific header violate encapsulation?

It sounds like your teacher is referring to the fact that to use Sprite in this case (which is an Overlord::Engine::Texture) you must depend on the "Engine.h" header from Overlord, thus preventing whatever file has that code snippet you posted from completely encapsulating its dependency on Overlord. Any code that needs to depend on the header containing your posted code snippet must also depend on "Overlord/Engine.h."

I don't really see this as a problem for that specific code snippet, personally. You cannot hide every dependency, and if that code is in your game (that uses the Overlord engine) or in the Overlord engine itself, this is basically not worth worrying about.

This kind of header dependency bleed is usually more of an issue when you're hoisting dependencies up through many layers of encapsulation. For example, if that snippet of code was from the Overlord engine itself (say, maybe it's in "Overlord/Sprite.h") and instead looked like this

#include <d3d11.h>
using Sprite = ID3D11Texture*;

then this might be more of a concern. Now end-users of Overlord are forced to be aware of D3D11, forced to have it included in all their files depending on this Overlord header, and potentially forced to deal with linking the correct D3D11 libraries.

You can use information-hiding techniques like PIMPL or similar compile-time polymorphism tricks to keep #includes of lower-level dependencies like your specific graphics or physics APIs out of your public-facing headers. This can significantly simplify the use of your library, in some cases; in other cases it's simply a stylistic preference on the part of the library author.

By way of example, you could eliminate the D3D header in my example by doing:

struct Sprite {
private:
  struct Implementation; // Note: forward declaration
  Implementation* implementation;
};

In the .cpp file for Sprite, you'd write

#include <d3d11.h>

struct Sprite::Implementation {
  ID3D11Texture* texture;
};

Sprite::Sprite()
: implementation(new Implementation)) {
  implementation->texture = ...
}

and so on. This prevents the type information for ID3D11Texture from being needed in the public header, and thus prevents you from including D3D headers in your own public headers.

Based on the context I'm inferring from your question though, I don't think this is something you'd want to bother with for your specific scenario.