No matter what fancy syntax features you are using, this is always going to be an operation with an algorithmic complexity of \$O(n * m)\$ (with n and m being the sizes of the respective lists). withWith two unordered lists, there is no way around iterating every single element of the first list and comparing it with every single element of the second. So no matter if you use a lambdaRemoveAll, a LINQ function or write the algorithm on your own loops, you will achieve micro-optimizations at best.
However, if you change the type of the _list from a List to a SortedSet or HashSet, then you can do this:
_set.ExceptWith(_listToRemove);
The ExceptWith(IEnumerable<T> other) method of both SortedSet and HashSet "is an \$O(n)\$ operation, where n is the number of elements in the other parameter.".
If you are not sure which flavor of ISet is more appropriate for your use-case, check this question on Stackoverflow (there is also a lot of interesting debate in the comments on the answers).