This chapter discusses design challenges encountered when developing a conversational pedagogical agent. By tracing the historical roots of pedagogical agents in Intelligent Tutoring Systems (ITS), we discern central developments in creating an agent that is both knowledgeable and fosters a social relationship with the learner. Main challenges faced when attempting to develop a pedagogical agent of this kind relate to: i) learners’ expectations on the agent’s knowledge and social profile, ii) dealing with learners’ engagement in off-task conversation and iii) managing potential abuse of the agent. We discuss these challenges and possible ways to address them, with reference to an ongoing Research & Development project, and with a focus on the design of a pedagogical agent’s visual embodiment and its conversational capabilities.