An autonomous excavator system for material loading tasks

Excavators are widely used for material handling applications in unstructured environments, including mining and construction. Operating excavators in a real-world environment can be challenging due to extreme conditions—such as rock sliding, ground collapse, or excessive dust—and can result in fatalities and injuries. Here, we present an autonomous excavator system (AES) for material loading tasks. Our system can handle different environments and uses an architecture that combines perception and planning. We fuse multimodal perception sensors, including LiDAR and cameras, along with advanced image enhancement, material and texture classification, and object detection algorithms. We also present hierarchical task and motion planning algorithms that combine learning-based techniques with optimization-based methods and are tightly integrated with the perception modules and the controller modules. We have evaluated AES performance on compact and standard excavators in many complex indoor and outdoor scenarios corresponding to material loading into dump trucks, waste material handling, rock capturing, pile removal, and trenching tasks. We demonstrate that our architecture improves the efficiency and autonomously handles different scenarios. AES has been deployed for real-world operations for long periods and can operate robustly in challenging scenarios. AES achieves 24 hours per intervention, i.e., the system can continuously operate for 24 hours without any human intervention. Moreover, the amount of material handled by AES per hour is closely equivalent to an experienced human operator.