Dealing with Camera "bump" during floating origin reset

I am asking this question because my Google searches have yielded little results. My problem is basically this: I've implemented a very simple smooth follow on my camera to trail the player (who is free to move on all axes). The smooth follow method is as below.

        targetPosition = target.position + (target.rotation * offset);
        transform.position = Vector3.SmoothDamp(transform.position, targetPosition, ref velocity, smoothTime);
        transform.LookAt(target, target.up);
        transform.rotation *= rotationalOffset;

I also have a simple "floating origin" implementation that moves all objects back to the world origin if the player crosses a certain threshold (in this case [100, 100, 100] for testing purposes). The method for that is:

if (isCrossedXThreshold || isCrossedYThreshold || isCrossedZThreshold)
{
    for(int i = 0; i < SceneManager.sceneCount; i++)
    {
        foreach(GameObject rootObj in SceneManager.GetSceneAt(i).GetRootGameObjects())
        {
            rootObj.transform.position -= playerPosition;
        }
    }
}

By themselves, they work as they should. However, when working together, the camera will briefly 'bump' towards the player when it crosses the floating origin threshold, making for a not-so-smooth transition back to the origin. From my understanding, this is happening because smoothTime creates a delay in repositioning the camera.

How could I go about ensuring a more smooth transition back to the origin without any unwanted camera shakes or bumps?

PS: All player and camera movements are done within the LateUpdate method, if thats relevant.