This paper proposes a novel, accurate, and efficient hybrid CPU/GPU-based 3-DOF haptic rendering algorithm for highly complex and large-scale virtual environments (VEs) that may simultaneously contain different types of object data representations. In a slower rendering process on the GPU, local geometry near the haptic interaction point (HIP) is obtained in the form of six directional depth maps from virtual cameras adaptively located around the object to be touched. In a faster rendering process on the CPU, collision detection and response computations are performed using the directional depth maps without the need for any complex data hierarchy of virtual objects, or data conversion of multiple data formats. To efficiently find an ideal HIP (IHIP), the proposed algorithm uses a new “abstract” local occupancy map instance (LOMI) and the nearest neighbor search algorithm, which does not require physical memory for storing voxel types during online voxelization and reduces the search time by a factor of about 10. Finally, in order to achieve accurate haptic interaction, sub-voxelization of a voxel in LOMI is proposed. The effectiveness of the proposed algorithm is subsequently demonstrated with several benchmark examples.