From questions to clarity: Using AI to transform student service, Charlotte Miller & Leanne Ruiz
Summary
Charlotte Miller (Assistant Registrar for Strategic Communications and Marketing) and Leanna Ruiz (Associate Director of Business Transformation) from McMaster University's Office of the Registrar presented their multi-year journey transforming student support services using a generative AI chatbot. The session was delivered at EvolveDigital Toronto in March 2026.
McMaster, a research-intensive Canadian university with over 37,000 students, faced escalating student inquiry volumes that were burning out frontline staff who repeatedly answered the same transactional questions. The team's core insight was that students benchmark their digital experience against any online service—not just other universities—demanding fast, accurate, 24/7 responses. The goal became freeing staff from repetitive questions so they could focus on complex, emotionally demanding, and student-specific cases.
The office's chatbot evolution spanned several years: a 20-person live chat service launched during the 2020 pandemic, a rule-based intent/keyword chatbot introduced in 2022 (which required manually writing 25–30 variations of every question and performed poorly with natural language), and finally a generative AI chatbot piloted from March to September 2025 using retrieval-augmented generation (RAG) against the Registrar's Office website.
A key early decision was to limit the AI's data sources to public content only—starting with the Registrar's website (already well-maintained and accurate) and later adding the 900-page undergraduate academic calendar as a second source. The calendar was ingested in under two minutes and immediately improved answers for admissions-related questions asked outside business hours. The chatbot did not connect to any internal student data systems, keeping it compliant and privacy-safe.
Before launch, the team completed a privacy impact assessment and IT security review—a process that took close to a year and involved over 300 IT security checklist items. They included AI disclaimers and source attribution in the chatbot interface. A two-week soft launch revealed that the confidence threshold for the legacy intent system was too high (80–85%), causing it to intercept queries the generative AI should have handled. Raising the threshold to ~95% shifted the balance so that 70–80% of answers were delivered by the generative AI.
The team ran weekly review cycles throughout the pilot: monitoring unanswered questions, identifying content gaps, editing website copy, and measuring results. They tracked AI-answered vs. intent-answered interactions, question-rephrasing rates, thumbs-up/thumbs-down ratings, live-chat escalation rates, and a prompted yes/no accuracy question. Sentiment analysis was also run by AI over interaction logs to supplement sparse user ratings.
Pilot results after six months: positive interactions increased 19%; generative AI responses were rated accurate 69% of the time—more than double the accuracy rating of the previous canned intent responses; generative AI handled approximately 62% of all student inquiries; and live-chat escalations to staff dropped measurably. A student reaching out at 12:23 AM asking how to pay for university without personal funds received three relevant resources instantly—an interaction that a year earlier would have required manually searching the website.
Beyond deflection metrics, the chatbot surfaced strategic content intelligence: it revealed where digital journeys broke down, which language confused students, and where navigation failed. This data directly informed the communications team's content strategy. The team concluded with five lessons: start with content mapping, not technology; pilot before scaling; define success metrics in advance; use institutional governance processes; and lead with privacy and security.
Key Takeaways
- •Start with content mapping before selecting technology: identify what questions students ask, what information exists to answer them, and where gaps are.
- •Run a soft launch before the official pilot to identify and fix configuration issues—McMaster discovered the intent confidence threshold needed to be raised to ~95% to allow the generative AI to handle the majority of queries.
- •Use public, well-maintained content as the initial RAG data source; the Registrar's website's accuracy was the foundation for chatbot trustworthiness.
- •A static PDF (McMaster's 900-page undergraduate calendar) can be added as a RAG source quickly—ingested in under two minutes—to expand coverage without major overhead.
- •Monitor chatbot interactions weekly during the pilot: track unanswered questions, AI misinterpretations, and content gaps, then iterate on website copy in response.
- •Define success metrics before launch, including both quantitative data points (accuracy ratings, escalation rates, volume) and qualitative signals (staff and student observations).
- •Lead with privacy and security: complete a privacy impact assessment and IT security review before piloting—expect this process to take close to a year at a large institution.
- •AI accuracy ratings (69%) can more than double canned-response accuracy ratings while eliminating the manual labour of writing dozens of intent variations per topic.
- •Chatbot interaction data is a strategic content intelligence tool: it shows where digital journeys break down and which language confuses students, directly informing content strategy.
- •Restrict the chatbot to public information only and ensure it does not connect to internal student data systems to maintain privacy compliance and build institutional trust.
Transcript
Um, so here's a question. How can higher education deliver faster, more accurate student support without adding to staff's workload? So today, Charlotte and Leanne from um McMaster University Office of the Register to talk to us about how AI became part of the story and outline the journey. They will talk about implementing of AI chatbot and the connections with the registars's website and their results and they will provide some advice along the way that to anyone who is interested in using AI ...
