I'd argue that the physics of light isn't going to be a bottleneck: for now you can easily get 50ms extra end-to-end latency if you are not careful when picking local monitor+keyboard setup, and nobody besides the most hard-core FPS gamers is complaining.

50ms is forever in terms of "the physics": a round-trip between Shanghai and Tokyo is less than 30ms, with all the switches/routers/... overhead considered.