#Your problem is this if-statement:

    if(selected){
       addTo();
    }

If the unit isn't selected, nothing happens. It doesn't matter that you do remove or clear inside addTo because addTo isn't being called unless this object is being selected.

    if(selected){
       addTo();
    }
    else{
       UnitController.SelectedUnits.Remove(gameObject);
    }

Your problem is this if-statement:

    if(selected){
       addTo();
    }

If the unit isn't selected, nothing happens. It doesn't matter that you do remove or clear inside addTo because addTo isn't being called unless this object is being selected.

    if(selected){
       addTo();
    }
    else{
       UnitController.SelectedUnits.Remove(gameObject);
    }