I think you can compare Input.mousePosition (in pixels) directly:

static Vector3 mouseDelta;
//from (-Screen.width, -Screen.height, 0) to (Screen.width, Screen.height, 0)
mouseDelta = (mouseOrigin - Input.mousePosition);
//convert pixels to world-units
mouseDelta /= panSpeed;
Camera.main.transform.Translate(mouseDelta, Space.World);
mouseOrigin = Input.mousePosition;

I think you can compare Input.mousePosition (in pixels) directly:

static Vector3 mouseDelta;
//from (-Screen.width, -Screen.height, 0) to (Screen.width, Screen.height, 0)
mouseDelta = (mouseOrigin - Input.mousePosition);
//convert pixels to world-units
mouseDelta /= panSpeed;
Camera.main.transform.Translate(mouseDelta, Space.World);
mouseOrigin = Input.mousePosition;

Note: you should clamp the transform's position to valid coordinates.