Well, it actually does block execution. It is just called every frame (Or more then once actually), and when you press the button in one frame, it will return true that frame, all other frames it will return false.
For instance, this peace of code will completely fill up your console in no-time:
private void OnGUI()
{
Debug.Log($"Frame: {Time.frameCount}, current onGUI type: {Event.current.type}");
}
This is because it is executed once with the 'layout' event, and once with the 'repaint' event every frame. Then other events like 'mousedown' and 'mouseup' also invoke OnGUI an additional time in the frames that these events occur.
Now how it is actually implemented, you'd have to check unity's reference on github. This is the part of interest, although it gets pretty complicated the deeper you go ;)
But to get an idea of how it somewhat works. Sinse it is called every frame, it can check Input.GetMouseButtonDown every frame just like you can, it can also check if the mouse position is inside the buttons rect (which you either provide in GUI.Button() or let unity create in the layout pass of OnGUI when you use GuiLayout.Button()). It then uses GUIUtility.hotControl to handle whether a button is clicked (both mousedown and mouseup event happened on the same button)
As you also might see in the implementation btw. GUILayout.Button always returns false in the layout and repaint passes, it will only return true in the mouseup pass. Which makes sense, because the code you place inside that if should only be executed once when you press it, not once for every event that frame (layout, repaint, mouseup).
On a sidenote, this (And everything that uses the OnGUI callback) is what Unity calls IMGUI, which is currently not recommended by unity to use for runtime game UI. See the documentation so see in which sutuations IMGUI is recommended.