Karma is deprecated

[joeyparrish]

I just found out that Karma was officially deprecated 4 months ago. https://dev.to/this-is-angular/angular-testing-in-2023-past-present-and-future-j5m

This isn't urgent, but we should decide what our test runner will look like in the future.

Things we need:

1. Real browser testing through Selenium grid so we can test video playback on real devices (not using jsdom or phantomjs)
2. Testing in-browser through a URL, instead of through WebDriver commands that manipulate the browser (this gets us the exotic devices in our lab through generic-webdriver-server)
3. Serve tests over HTTPS with a custom certificate (without which we lose EME coverage and encrypted playback tests in the lab)
4. A way for the in-browser tests to invoke the WebDriver screenshot command in the test runner and have the runner compare that screenshot to the reference screenshots (today we have a custom Karma middleware that the tests can call into)
5. Machine-readable test coverage reports

Things that would be make the transition easier:

1. Support for our current test framework, Jasmine (though we could rewrite our tests if necessary)
2. Test coverage output in the same JSON format we parse today in our workflows (though we could hypothetically convert to that or rewrite our workflows)
3. A way to run local browser tests over WebDriver, maybe with a plugin (this is how we get screenshot tests without using the lab today, via karma-local-wd-launcher)

Input from the community is very welcome. I've done very little research so far.

[joeyparrish]

I believe jasmine-browser-runner is the way forward. It will let us keep our existing Jasmine tests, it will connect to a remote Selenium grid, and we can reimplement our custom middlewares in express via configuration.

I'm not yet sure about:

1. TLS configuration for testing over HTTPS Resolved by my PR, Add TLS and hostname support jasmine/jasmine-browser-runner#49
2. Launching local WebDriver instances to support local screenshot testing (probably could do it by starting with something like this)
3. Coverage reporting

[joeyparrish]

jasmine-browser-runner also seems to only run a single browser. That will also need to be addressed.

[joeyparrish]

The maintainers of jasmine-browser-runner are resistant to adding support for multiple browsers running in parallel, even if I provide a PR. (And I had this working locally.) They just don't seem to want the complexity that they feel would come with that functionality. So we may have to go another direction.

I'm currently investigating Web Test Runner, which is built to launch real browsers, just like Karma was. It supports both Selenium and WebDriverIO as WebDriver clients to launch browsers.

Needs met by Web Test Runner:

• Real browser testing through Selenium grid so we can test video playback on real devices
• Testing in-browser through a URL, instead of through WebDriver commands that manipulate the browser
• Serve tests over HTTPS with a custom certificate * [ ] A way for the in-browser tests to invoke the WebDriver screenshot command in the test runner and have the runner compare that screenshot to the reference screenshots (today we have a custom Karma middleware that the tests can call into)

Needs, not sure how to get from Web Test Runner:

• Machine-readable test coverage reports
• Web server plugin to trigger WebDriver screenshots from test code

Wants:

• Support for our current test framework, Jasmine (not critical)
• Test coverage output in junit format, for compatibility with current tooling (not critical)
• A way to run local browser tests over WebDriver, for local screenshot tests (strong preference)