Evaluating RAG Systems for University Syllabus QA

​

This study investigates Retrieval-Augmented Generation (RAG) systems for university syllabus question answering (QA), using 7,368 syllabus documents from Sophia University as the target dataset. We systematically compare multiple RAG strategies and propose a novel hybrid method to address a critical real-world challenge.

In Experiment A, we evaluate standard RAG baselines against RAFT (Retrieval-Augmented Fine-Tuning) on 100 questions targeting uniquely-named courses. RAFT with distractor-based training achieves the best performance (Token-F1: 0.695), significantly outperforming Naive RAG (Token-F1: 0.268), as confirmed by Wilcoxon signed-rank tests (p < 0.0001).

Post-analysis reveals a fundamental limitation of conventional RAG: same-name course ambiguity, where multiple course sections share identical names but differ in instructor, schedule, or credits. To address this, Experiment B introduces a mixed benchmark of 50 unique-course and 50 ambiguous-course questions. We propose Metadata+RAFT, which combines rule-based metadata filtering — extracting instructor name, semester, and period from the query to pre-filter the syllabus database — with RAFT-based answer generation. Metadata+RAFT achieves the highest performance across all metrics (CM: 78%, Token-F1: 0.821, BERTScore: 0.935), significantly outperforming all baselines (p < 0.05). These results demonstrate that same-name course ambiguity is a critical blind spot for conventional RAG systems, and that metadata-aware retrieval combined with domain-specific fine-tuning provides an effective solution.