Evaluation Utility Metrics (EUMs) in Reflective Practice
DOI:
https://doi.org/10.3138/cjpe.72386Keywords:
evaluation, evaluation recommendations, metrics, reflective practice, utilityAbstract
The article proposes three evaluation utility metrics to assist evaluators in evaluating the quality of their evaluation. After an overview of reflective practice in evaluation, the different ways in which evaluators can hold themselves accountable are discussed. It is argued that reflective practice requires evaluators to go beyond evaluation quality (i.e., technical quality and methodological rigor) when assessing evaluation practice to include an evaluation of evaluation utility (i.e., specific actions taken in response to evaluation recommendations). Three Evaluation Utility Metrics (EUMs) are proposed to evaluate utility: whether recommendations are considered (EUMc), adopted (EUMa), and (if adopted) the level of influence of recommendations (EUMli). The authors then reflect on their experience in using the EUMs, noting the importance of managing expectations through negotiation to ensure that EUM data are collected and the need to consider contextual nuances (e.g., adoption and influence of recommendations are influenced by multiple factors beyond the control of the evaluators). Recommendations for increasing EUM rates by paying attention to the frequency and timing of recommendations are also shared. Results of implementing these EUMs in a real-world evaluation provide evidence of their potential value: practice tips led to an EUMc of 100% and and EUMa of over 80%. Methods for considering and applying all three EUMs together to facilitate practice improvement are also discussed.
References
American Evaluation Association (AEA). (2021). Guiding principles. https://www.eval.org/Portals/0/Docs/AEA_289398-18_GuidingPrinciples_Brochure_2.pdf
Alkin, M. C., & Taut, S. M. (2002). Unbundling evaluation use. Studies in Educational Evaluation, 29(1), 1 – 12 . https://doi.org/10.1016/s0191-491x(03)90001-0
Anderson, R. (2004). Intuitive inquiry: An epistemology of the heart for scientific inquiry. The Humanistic Psychologist, 32(4), 307 – 341. https://doi.org/10.1080/08873267.2004.9961758
Bashir, N. Y., Wilson, A. E., Lockwood, P., Chasteen , A. L., & Alisat, S. (2014). The time for action is now: Subjective temporal proximity enhances pursuit of remote-future goals. Social Cognition, 32(1), 83 – 93 . https://doi.org/10.1521/soco.2014.32.1.83
Brusset, E., Cosgrave, J., & MacDonald, W. (2010). Real-time evaluation in humanitarian emergencies. New Directions for Evaluation, 2010(126), 9 – 20. https://doi.org/10.1002/ev.326
Bundi, P., Frey, K., & Widmer, T. (2021). Does evaluation quality enhance evaluation use? Evidence & Policy: A Journal of Research, Debate and Practice, 17(4), 661 – 687. https://doi.org/10.1332/174426421x16141794148067
Bustelo, M. (2002). Metaevaluation as a tool for the improvement and development of the evaluation function in public administrations. https://evaluationcanada.ca/distribution/20021010_bustelo_maria.pdf
Canadian Evaluation Society (CES). (2018, October 19). Competencies for Canadian evaluation practice. https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice_2018.pdf
Cousins, J. B., & Leithwood, K. A. (1986). Current empirical research on evaluation utilization. Review of Educational Research, 56(3), 331 – 364. https://doi.org/10.3102/00346543056003331
Employment and Training Administration (ETA). (2021). Government performance and results act. https://www.dol.gov/agencies/eta/performance/goals/gpra
Friedman, M. (2009). Video introduction to result based accountability. https://youtu.be/RtBC89F3Xi4
Glass, G. V., & Ellett, F. S., Jr. (1980). Evaluation research. Annual Review of Psychology, 31(1), 211 – 228 . https://doi.org/10.1146/annurev.ps.31.020180.001235
Henry, G. T. (2000). Why not use? New Directions for Evaluation, 2000(88), 85 – 98 . https://doi.org/10.1002/ev.1193
Henry, G. T. (2003). Influential evaluations. American Journal of Evaluation, 24(4), 515 – 524 . https://doi.org/10.1016/j.ameval.2003.10.001
Henry, G. T., & Mark, M. M. (2003). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24(3), 293 – 314. https://doi.org/10.1177/109821400302400302
Henry, G. T., Mark, M. M., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving public and nonproft policies and programs. Wiley.
Iriti, J. E., Bickel, W. E., & Nelson, C. A. (2005). Using recommendations in evaluation: A decision-making framework for evaluators. American Journal of Evaluation, 26(4), 464 – 479. https://doi.org/10.1177/1098214005281444
King, J. A., & Pechman, E. M. (1982). The process of evaluation use in local school settings (Final Report of NIE Grant 81-0900). Orleans Parish School Board (ERIC Document Reproduction Service No. ED 233 037).
Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F., & Bolkov, B. (2009). Research on evaluation use: A review of the empirical literature from 1986– 2005. American Journal of Evaluation, 30(3), 377 – 410. https://doi.org/10.1177/1098214009341660
Mark, M. M., & Henry, G. T. (2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10(1), 35 – 57 . https://doi.org/10.1177/1356389004042326
McCormick, E. R. (1997). Factors influencing the results of evaluation use (Unpublished Master’s thesis). University of Minnesota.
McShane, S., & Von Glinow, M. (2009). Organizational behavior: Emerging knowledge and practice for the real world (5th ed.). McGraw Hill.
Meadows, D. (2008). Thinking in systems: A primer. Chelsea Green.
Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Sage.
Patton, M. Q. (2013). Utilization-Focused Evaluation (UFE) checklist. https://wmich.edu/sites/default/fles/attachments/u350/2014/UFE_checklist_2013.pdf
Renger, R., Basson, M. D., Hart, G., Van Eck, R., Souvannasacd, E., Renger, J., & Foltysova, J. (2020). Lessons learned in evaluating the infrastructure of a Centre for Translational Research. Evaluation Journal of Australasia, 20(1), 6 – 22 . https://doi.org/10.1177/1035719x20909910. Medline:34163122
Renger, R., Renger, J., Basson, M. D., Van Eck, R. N., Renger, J., Souvannasacd, E., & Hart, G. (2021). Using the Homeland Security and Exercise Evaluation Program (HSEEP) building block approach to implement System Evaluation Teory (SET). American Journal of Evaluation, 42(4), 586 – 601. https://doi.org/10.1177/1098214020986619. Medline:34966242
Rickinson, M., Sharples, J., & Lovell, O. (2020). Towards an understanding of quality of evidence use. In S. Gorard (Ed.), Getting evidence into education: Evaluating the routes to policy and practice (pp. 218-233 ). Routledge. https://doi.org/10.4324/9780429290343
Scriven, M. (1991). The evaluation thesaurus (4th ed.). Sage .
Scriven, M. (2007). Key evaluation checklist (KEC). https://evaluationcanada.ca/distribution/20060523_scriven_michael.pdf
Smith, T. L., Barlow, P. B. , Peters , J. M. , & Skolits, G. J. (2015). Demystifying reflective practice: Using the DATA model to enhance evaluators’ professional activities. Evaluation and Program Planning, 52, 142 – 147. https://doi.org/10.1016/j.evalprogplan.2015.04.004. Medline:26051793
Stufebeam, D. L. (1974). Meta-evaluation. Journal of MultiDisciplinary Evaluation, 7(15), 99 – 158 .
van Draanen, J. (2017). Introducing reflexivity to evaluation practice: An in-depth case study. American Journal of Evaluation, 38(3), 360 – 375. https://doi.org/10.1177/1098214016668401
Weinger, M. B., Slagle, J., Jain, S., & Ordonez, N. (2003). Retrospective data collection and analytical techniques for patient safety studies. Journal of Biomedical Informatics, 36(1–2), 106 – 119. https://doi.org/10.1016/j.jbi.2003.08.002. Medline:14552852
Wolski, M. (2020). System requirements: The context and boundary of a system. https://michael.wolski.pro/2018/01/system-requirements-the-context-and-boundary-of-the-system/
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
