Abstract
In this paper, we address the system evaluation issue for commercial term extraction tools from the users’
perspective. We first revisit the gold standard approach commonly practised among researchers, and discuss the challenges it may
pose on end users, taking translators as a typical example. Considering the very different motivations and needs of users and
researchers, a user-driven approach is proposed as a variation and alternative to the gold standard approach to allow users to
assess and understand the performance of commercial tools more objectively. Its feasibility and usefulness are demonstrated by
deploying a benchmarking dataset of English-Chinese financial terms, produced by multiple annotators, in a case study with SDL
MultiTerm Extract. The results also provide insight for future development of term extractors designed for translators, which will
hopefully generate more accurate candidates, offer more customised features, enable better user experience, and enjoy wider
popularity as a computer-aided translation tool.