A novel approach to generic (or generalized) robot programming and a novel simplified, block-based programming environment, called “Assembly”, are introduced. The approach leverages the newest graphical user interface automation tools and techniques to generate programs in various proprietary robot programming environments by emulating user interactions in those environments. The “Assembly” tool is used to generate robot-independent intermediary program models, which are translated into robot-specific programs using a graphical user interface automation toolchain. The generalizability of the approach to list, tree, and block-based programming is assessed using three different robot programming environments, two of which are proprietary. The results of this evaluation suggest that the proposed approach is feasible for an entire range of programming models and thus enables the generation of programs in various proprietary robot programming environments. In educational settings, the automated generation of programs fosters learning different robot programming models by example. For experts, the proposed approach provides a means for generating program (or task) templates, which can be adjusted to the needs of the application at hand on the shop floor.
Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements
