Systems and methods for arbitrary viewpoint robotic manipulation and robotic surgical assistance

Systems and methods for arbitrary viewpoint robotic manipulation and robotic surgical assistance are disclosed. According to an aspect, a system includes one or more controllers configured to receive an image dataset of an actual environment within which the robotic tool is positioned. The controller(s) are also configured to generate a virtual environment of the actual environment based on the image dataset. Further, the controller(s) can control display of the virtual environment including a virtual tool controllable by a user for use to control the robotic tool within the actual environment. The controller(s) can receive user input for ...