In compliance with national legislation, Greek tertiary education institutions assess educational quality often using a standardized anonymous questionnaire completed by students. This questionnaire aims to independently evaluate various course components, including content organization, instructor quality, facilities, infrastructure, and grading methods. Despite widespread use across universities, the questionnaire’s validity remains unexamined. This study addresses this gap by analyzing 48,000+ questionnaire responses from the University of the Peloponnese (2014-2022), encompassing 68 undergraduate and graduate programs. Confirmatory factor analyses were used to assess the quality of the questionnaire, while exploratory factor analyses were used to assess the dimensions of the tool based on the data. Both analyses reveal shortcomings: confirmatory analysis detects strong correlations between supposedly different factors, and exploratory analysis identifies dimensions inconsistent with expected structure. These findings question the questionnaire’s quality and the validity of drawn conclusions, while additionally identify opportunities for reducing the number of questions, which can contribute to increased questionnaire submission rates. Given its common use across Greek universities and its influence on shaping courses, urgent redesigning of the questionnaire for tertiary education evaluation is recommended.