Launch Angular with ng serve with dotnet core Angular

I have created a new Angular project with the dotnet core CLI:

dotnet new angular -o my-app

I have set up my own deployment and build process outside of my project, so my project is essentially dumb when it comes to building and deploying. My project should never build any .js files and store them locally in the project. This is basically what I want:

1. I press F5/launch debugger
2. The dotnet core project starts and runs npm install && ng serve
3. Once done, or whenever it's needed, a browser will open and show the ng serve site (I think default is http://localhost:4200)

So far I'm close to getting this to work, but for some reason, the dotnet core project serves me static files that are built, which I don't want (because I want live reload from ng serve). Here's what I can do right now:

1. Press F5 to launch the debugger
2. A ng window appears with the debugging information, that shows info like npm install and ng serve were run
3. Opens http://localhost:5000 which serves me the static files, as if I built the application with ng build

If I can explain what I really want:

1. Press F5 to launch the debugger
2. Project starts and the ng window appears, showing I ran npm install && ng serve -port 4200
3. Browser launches http://localhost:4200

Right now it opens at port 5000 (or whatever I specify in the launchSettings.json file), but I can't get it to launch 4200 because technically I don't want my dotnet core project to start a new web server (since ng serve already creates one). Is this possible?

package.json

"scripts": {
    "start": "npm install && ng serve --port 4200 &REM"
}

Startup.cs

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
    }

    public void Configure(IApplicationBuilder app, IHostingEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }
        else
        {
            app.UseExceptionHandler("/Error");
        }

        app.UseSpa(spa =>
        {
            spa.Options.SourcePath = "component";

            if (env.IsDevelopment())
            {
                spa.UseAngularCliServer("start");
            }
        });
    }
}