Human-AI Collaboration for Online Course Quality Assurance: A Pilot Study
Applying quality assurance (QA) models to online and hybrid programs can be time-intensive and challenging for faculty and course designers. What if Artifical Intelligence (AI) could make QA testing more automated and scalable for online learning units? This lightening talk shares a year-long pilot study that explores how AI can enhance, rather than replcae human expertise in the QA process. It examines the development of an AI-assisted QA process based on established quality frameworks from Quality Matters (QM), the Online Learning Consotrium, and CAST Universal Design for Learning (UDL).
Attendees will observe a rapid demonstration of the pilot workflow, lessons learned, and early findings, followed by an interactive Q&A exploring how AI can be intentionally integrated into QA processes while preserving human creativity and pedagogical judgment. Participants will leave with practical considerations and adaptable strategies for responsible AI use in course review and instructional design.
Learning Outcomes
Participants will be able to:
- Identify opportunities and challenges of using AI-assisted tools in online course quality assurance.
- Describe key learnings from comparing human and AI review results across multiple tools.
- Discuss strategies for ethically and intentionally integrating AI into QA workflows at their own institutions.
Rae Mancilla, University of Pittsburgh, US
Rae Mancilla is the Executive Director of University Digital Education at the University of Pittsburgh Center for Excellence in Digital Education (Pitt EDGE). She leads institution-wide strategy and innovation in digital learning, advancing online and hybrid program development, academic excellence and cross-campus collaboration. A dynamic leader in higher education, she works closely with Pitt’s 16 schools and industry partners to strengthen the quality, accessibility and impact of digital education across the University.
With over two decades of experience in instructional design, curriculum development, faculty development, program evaluation and assessment, Mancilla is a nationally recognized expert in online learning and digital accessibility. She currently serves as vice chair of the Quality Matters Instructional Design Association, a bilingual peer evaluator for the Middle States Commission on Higher Education, a Quality Matters Master Reviewer and Research Colleague, and an invited scholar for the International Board of Standards for Training, Performance and Instruction.
Mancilla’s research interests include instructional designer professional development, digital accessibility and program evaluation. She is the co-editor of Guide to Digital Accessibility: Policies, Practices and Professional Development (2023, Stylus) and co-author of Mentoring Instructional Designers in Higher Education (forthcoming 2026, Routledge).
Natalia Echeverry, MS, University of Pittsburgh, US
Natalia Echeverry is an instructional designer at the Center for Excellence in Digital Education (Pitt EDGE) at the University of Pittsburgh. She holds an MS in Human-Computer Interaction from the Rochester Institute of Technology. Her work centers on designing engaging, evidence-based learning experiences for adult learners. She is currently exploring the integration of AI systems into instructional design workflows to enhance efficiency and innovation.
