Beyond Counting Clients and Surveying Satisfaction: Directly Measuring the Impacts of CTL Consultations
Evidence-based practice in educational development includes leveraging data to iteratively refine CTL services. However, CTL data collection is often limited to counts and satisfaction surveys, rather than direct measures of outcomes. To directly assess impacts of CTL consultations on course and syllabus design practices, we analyzed 94 clients’ syllabi (n=32 faculty, n=62 graduate students and postdocs), before and after CTL consultations. Faculty and non-faculty clients demonstrated significant change following consultations (6% and 10% gains in syllabus rubric scores, representing 50% and 31% of possible gains, respectively). We compared faculty clients to quasi-experimental control groups who did not receive consultations. Syllabi from non-clients scored lower and did not demonstrate similar changes across semesters. Attendance at a CTL seminar on course and syllabus design did not explain variation in clients’ syllabi. We discuss implications for assessment of CTL services. Additionally, we compare and contrast the affordances of syllabi and other teaching artifacts as data sources in direct assessments of CTL impacts.