A rapid 3D reconstruction of bones and other structures during an operation is an important issue. However, most of existing technologies are not feasible to be implemented in an intraoperative environment. Normally, a 3D reconstruction has to be done by a CT or an MRI pre operation or post operation. Due to some physical constraints, it is not feasible to utilise such machine intraoperatively. A special type of MRI has been developed to overcome the problem. However, all normal surgical tools and instruments cannot be employed. This chapter discusses a possible method to use a small number, e.g. 5, of conventional 2D X-ray images to reconstruct 3D bone and other structures intraoperatively. A statistical shape model is used to fit a set of optimal landmarks vertices, which are automatically created from the 2D images, to reconstruct a full surface. The reconstructed surfaces can then be visualised and manipulated by surgeons or used by surgical robotic systems.