How to Conduct a Metaevaluation?
A Metaevaluation Practice
DOI:
https://doi.org/10.3138/cjpe.71619Keywords:
metaevaluation, metaevaluation practice, evaluating the evaluation, program evaluationAbstract
A metaevaluation is a quality cross-check to examine the conduct of an evaluation and validate the results. Of the few metaevaluation studies, almost none have reported on the metaevaluation procedure through a practical example evaluation. This study reports on the strengths and weaknesses of a program evaluation study in terms of the four main standards: utility, feasibility, propriety, and accuracy. It includes a metaevaluation process that involves both quantitative and qualitative analysis of data from eight meta-evaluators. It was found that while the evaluation study had very good utility and accuracy standards, the feasibility and propriety standards were only fair.
References
Alkin, M.C. (Ed.). (2012). Evaluation roots: A wider perspective of theorists’ views and influences. Sage.
Association for Evaluation and Accreditation of Teacher Education Programs. (2016). EPDAD Teacher education undergraduate programs standards. https://epdad.org.tr/data/genel/pdf/standartlar.pdf
Association for Evaluation and Accreditation of Teacher Education Programs. (2021). EPDAD Teacher education undergraduate programs standards guide. https://epdad.org.tr/data/genel/pdf/EPDAD_Standartlar_Surum_1.1_Kilavuz.pdf
Aydın-Ceran, S. (2021). Öğretim yöntemlerine dayanan fen eğitimi araştırmalarında güncel eğilimler: İlkokul düzeyinde bir analiz [Current trends in science education research based on teaching methods: A primary school level analysis]. Journal of Individual Differences in Education, 3(2), 113–131.
Berk, Ş. (2012). Evaluation of the modular system implemented in vocational and technical secondary schools by using Provus’ Discrepancy Model. Doctoral dissertation, Anadolu University. Available at Turkish National Thesis Center.
Berk, Ş. (2018a). Modüler sistemin değerlendirilmesi: Modüler sistem, program değerlendirme, mesleki ve teknik eğitim [Evaluation of modular system: Modular system, program evaluation, vocational and technical education]. Pegem Akademi.
Berk, Ş. (2018b). Assessment of public schools’ out-of-school time academic support programs with participant-oriented evaluation. Journal of Education and Learning, 7(3), 159–175.
Bobin, K. (2017). Meta-evaluation: A synthesis of evaluation studies 2005–2016 (Evaluation Series No. 104). The Commonwealth Secretariat. http://meb.ai/JhU2sY
Bourgeois, I., & Whynot, J. (2018). Strategic evaluation utilization in the Canadian federal government. Canadian Journal of Program Evaluation, 32(3).
Cooksy, L. J., & Caracelli, V.J. (2009). Metaevaluation in practice: Selection and application of criteria. Journal of MultiDisciplinary Evaluation, 6(11), 1–15.
Daloğlu, A. (1996). A case study on evaluating the certificate for overseas teachers of English curriculum at Bilkent University. [Unpublished doctoral dissertation, Middle East Technical University]. Available at Turkish National Thesis Center.
Demirel, Ö. (2017). Eğitimde program geliştirme: Kuramdan uygulamaya [Curriculum development: From theory to practice]. Ankara: Pegem A Yayıncılık.
Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2010). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Pearson.
Gardner, D. (2019). Beyond the four levels: An evaluation model for growth and sustainability [Doctoral dissertation, Wayne State University]. Available at ProQuest Dissertations & Theses Global.
Gedik, N. B. (2017). 3. Sınıf fen bilimleri dersi öğretim programının öğretmen görüşlerine dayalı olarak değerlendirilmesi [Evaluation of elementary school third grade science curriculum based on teachers’ views; Unpublished master’s thesis, Adıyaman Univesity]. Available at Turkish National Thesis Center.
Gökmenoğlu, T. (2015). The wide angle: Program evaluation studies in Turkey in terms of models and approaches. International Journal of Curriculum and Instructional Studies, 4(7), 55–70.
Güven, G. (2016). 3. sınıf fen bilimleri dersi öğretim programına ilişkin öğretmen görüşleri [The opinions of teachers about the 3rd class science curriculum; Unpublished master’s thesis, Mustafa Kemal University]. Available at Turkish National Thesis Center.
Jacob, S., & Desautels, G. (2014). Assessing the quality of Aboriginal program evaluations. Canadian Journal of Program Evaluation, 295(1), 62–86.
Kürüm-Yapıcıoğlu, D., Atik-Kara, D., & Sever, D. (2016). Türkiye’de program değerlendirme çalışmalarında eğilimler ve sorunlar: Alan uzmanlarının gözüyle [Trends and problems in curriculum evaluation studies in Turkey: The perspective of domain experts]. Uluslararası Eğitim Programları ve Öğretim Çalışmaları Dergisi, 6(12), 91–113.
Kuru, S. (1987). Mesleki eğitim fakültesi giyim endüstrisi ve giyim eğitimi bölümü “giyim teknikleri ve üretimi” dersinin değerlendirilmesi [Evaluation of the course “clothing techniques and production” in the department of clothing industry and clothing education, faculty of vocational education; Unpublished master’s thesis, Gazi University]. Available at Turkish National Thesis Center.
Leeuw, F.L., & Cooksy, L.J. (2005). Evaluating the performance of development agencies: The role of metaevaluations. In G. K. Pitman, O. N. Feinstein, & G. K. Ingram (Eds.), Evaluating development effectiveness (World Bank Series on Evaluation and Development, Vol. 7, pp. 95–108). Transaction.
Lynch, D. C., Greer, A. G., Larson, L. C., Cummings, D. M., Harriett, B. S., Dreyfus, K. S., & Clay, M. C. (2003). Descriptive metaevaluation: Case study of an interdisciplinary curriculum. Evaluation & the Health Professions, 26(4), 447–461.
Ministry of National Education. (2013). İlköğretim kurumları (ilkokullar ve ortaokullar) fen bilimleri dersi (3, 4, 5, 6, 7 ve 8. sınıflar) öğretim programı [Science curriculum (primary and middle schools) (grades 3, 4, 5, 6, 7 and 8)].
Morris, M. (Ed.). (2007). Evaluation ethics for best practice: Cases and commentaries. Guilford Press.
Organisation for Economic Co-operation and Development. (2011). Strengthening accountability in aid for trade. The development dimension. OECD Publishing.
Özdemir, S. M. (2009). Eğitimde program değerlendirme ve Türkiye’de eğitim programlarını değerlendirme çalışmalarının incelenmesi [Curriculum evaluation and examination of curriculum evaluation studies in Turkey]. Van Yüzüncü Yıl Üniversitesi Eğitim Fakültesi Dergisi, 6(2), 126–149.
Patton, M. Q. (2001). Evaluation, knowledge management, best practices, and high quality lessons learned. American Journal of Evaluation, 22(3), 329–336.
Patton, M. Q. (2012). Meta-evaluation: Evaluating the evaluation of the Paris declaration. The Canadian Journal of Program Evaluation, 27(3), 147–171.
Patton, M. Q. (2017). Principles-focused evaluation: The guide. Guilford Publications.
Patton, M. Q. (2018). Evaluation science. American Journal of Evaluation, 39(2), 183–200.
Perry, K. M. (2008). A reaction to and mental metaevaluation of the experiential learning evaluation project. American Journal of Evaluation, 29(3), 352–357.
Russ-Eft, D., & Preskill, H. (2008). Improving the quality of evaluation participation: A meta-evaluation. Human Resource Development International, 11(1), 35–50.
Sanders, J. R. (1999). Metaevaluation of “the effectiveness of comprehensive, case management interventions: Evidence from the National Evaluation of the Comprehensive Child Development Program.” American Journal of Evaluation, 20(3), 577–582.
Scriven, M. (1969). An introduction to meta-evaluation. Educational Product Report, 2(5), 36–38.
Scriven, M. (1975). Evaluation bias and its control. The Evaluation Center, Western Michigan University.
Scriven, M. (2009). Meta-evaluation revisited. Journal of Multidisciplinary Evaluation, 6(11), 3–8.
Şentürk, Ö. (2017). İlkokul 3. sınıf fen bilimleri dersi öğretim programı'nın değerlendirilmesi [An evaluation of the third-grade science curriculum in elementary school; Unpublished master’s thesis, Marmara University]. Available at Turkish National Thesis Center.
Şentürk, Ö., & Berk, Ş. (2019). İlkokul 3. sınıf fen bilimleri dersi öğretim programının değerlendirilmesi [Evaluation of the 3rd grade science curriculum in primary schools]. Marmara Üniversitesi Atatürk Eğitim Fakültesi Eğitim Bilimleri Dergisi, 49(49), 144–166.
Snow, D. (2001). Communicating quality. In A. Benson, D. M. Hinn, & C. Lloyd (Eds.), Visions of quality: How evaluators define, understand and represent program quality (pp. 29–42). JAI Press.
Stufflebeam, D. L. (1999). Program evaluations metaevaluation checklist (Evaluation Checklists Project). National Science Foundation (NSF) & The Evaluation Center, Western Michigan University. http://meb.ai/fI2tIF
Stufflebeam, D. L. (2000). The methodology of metaevaluation as reflected in metaevaluations by the Western Michigan University Evaluation Center. Journal of Personnel Evaluation in Education, 14(1), 95–125.
Stufflebeam, D. L. (2001). The metaevaluation imperative. American Journal of Evaluation, 22(2), 183–209.
Stufflebeam, D. L. (2004). A note on the purposes, development, and applicability of the Joint Committee Evaluation Standards. American Journal of Evaluation, 25(1), 99–102.
Stufflebeam, D. L., & Coryn, C. L. (2014). Evaluation theory, models, and applications. John Wiley & Sons.
Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. Jossey-Bass.
Türk, N. (2019). Design, implementation and evaluation of science, technology, engineering and mathematics (STEM) curriculum for undergraduate programs of faculty of education. [Unpublished doctoral dissertation, Gazi University]. Available at Turkish National Thesis Center.
ÜAK Code of Ethics. (2019). Üniversitelerarası kurul bilimsel araştırma ve yayın etiği yönergesi [Interuniversity council directive on scientific research and publication ethics]. http://meb.ai/fQT2sN
Wingate, L. A. (2009). The program evaluation standards applied for metaevaluation purposes: Investigating interrater reliability and implications for use. [Doctoral dissertation, Western Michigan University]. Available at ProQuest Dissertations & Theses Global.
Yarbrough, D. B. (2017). Developing the program evaluation utility standards: Scholarly foundations and collaborative processes. Canadian Journal of Program Evaluation, 31(3), 284–304.
Yarbrough, D. B., Shula, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The program evaluation standards: A guide for evaluators and evaluation users (3rd. ed). Corwin Press.
Yüksel, İ. (2010). Türkiye için program değerlendirme standartları oluşturma çalışması [Development of Turkish program evaluation standards; Unpublished doctoral dissertation, Anadolu University]. Available at Turkish National Thesis Center.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Esra Kerimoğlu, Muazzez Nihal Öykü Ülker, Şaban Berk
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
