Toward an Evidence-Based Approach to Building Evaluation Capacity
DOI:
https://doi.org/10.3138/cjpe.69191Keywords:
assessment tool, early childhood development field, evaluation capacity, evaluation capacity assessment, measurement, structural equation modellingAbstract
Organizations are required to evaluate their programs for both learning and accountability purposes, which has increased the need to build their internal evaluation capacity. A remaining challenge is access to tools that lead to valid evidence supporting internal capacity development. The authors share practical insights from the development and use of the Evaluation Capacity Needs Assessment tool and framework and implications for using its data to make concrete decisions within Canadian contexts. The article refers to validity evidence generated from factor analyses and structural equation modelling and describes how applying the framework can be used to identify individual and organizational evaluation capacity strengths and gaps, concluding with practice considerations and future directions for this work.
References
Abma, T. A. (2006). The social relations of evaluation. In I. F. Shaw, J. C. Greene, & M. M. Mark (Eds.), The Sage handbook of evaluation (pp. 184 – 199). Thousand Oaks, CA: SAGE.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Bourgeois, I., Toews, E., Whynot, J., & Lamarche, M. K. (2013). Measuring organizational evaluation capacity in the Canadian federal government. Canadian Journal of Program Evaluation, 28(2), 1 – 19. Retrieved from https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/30828
Buchanan, H., & Kuji-Shikatani, K. (2014). Evaluator competencies: The Canadian experience. Canadian Journal of Program Evaluation, 28(3), 29 – 47. Retrieved from https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/30950
Burke, W. W. (2017). Organization change: Theory and practice. Thousand Oaks, CA: SAGE.
Burns, G. N., Kotrba , L. M., & Denison, D. R. (2013). Leader-culture fit: Aligning leadership and corporate culture. In J. Passmore (Ed.), The Wiley-Blackwell handbook of the psychology of leadership, change, and organizational development (pp. 113 – 128). Chichester, England: John Wiley & Sons.
Chelimsky, E. (2006). The purposes of evaluation in a democratic society. In I. F. Shaw, J. C. Greene, & M. M. Mark (Eds.), The SAGE handbook of evaluation (pp. 33 – 55). Thousand Oaks, CA: SAGE.
Cheng, S.- H., & King, J. A. (2016). Exploring organizational evaluation capacity and evaluation capacity building: A Delphi study of Taiwanese elementary and junior high schools. American Journal of Evaluation, 38(4), 521 – 539. https://doi.org/10.1177/1098214016672344
Compton, D., Baizerman, M., Preskill, H., Rieker, P., & Miner, K. (2001). Developing evaluation capacity while improving evaluation training in public health: The American Cancer Society’s Collaborative Evaluation Fellows Project. Evaluation and Program Planning, 24(1), 33 – 40. https://doi.org/10.1016/S0149-7189(00)00045-8
Cousins, J. B., Elliott, C., Amo, C., Bourgeois, I., Chouinard, J., Goh, S. C., & Lahey, R. (2008). Organizational capacity to do and use evaluation: Results of a pan-Canadian survey of evaluators. Canadian Journal of Program Evaluation, 23(3), 1 – 35. Retrieved from https://evaluationcanada.ca/system/files/cjpe-entries/23-3-001.pdf
Cousins, J. B., Goh, S. C., Elliott, C. J., & Bourgeois, I. (2014). Framing the capacity to do and use evaluation. New Directions for Evaluation, 2014(141), 7 – 23. https://doi.org/10.1002/ev.20076
Davies, P., Newcomer, K., & Soydan, H. (2006). Government as structural context for evaluation. In I. Shaw, J. Greene, & M. Mark (Eds.), The Sage handbook of evaluation (pp. 163 – 183). Thousand Oaks, CA: Sage.
El Hassar, B. (2019). Development and Validation of an Evaluation Capacity Instrument for the Early Childhood Field (Unpublished doctoral dissertation). University of Alberta, Edmonton, Alberta.
Fierro, L. A. (2012). Clarifying the connections: Evaluation capacity and intended outcomes (Unpublished doctoral dissertation). Claremont Graduate University, Claremont, CA.
Gagnon, F., Aubry, T., Cousins, J. B., Goh, S. C., & Elliott, C. (2018). Validation of the evaluation capacity in organizations questionnaire. Evaluation and Program Planning, 68, 166 – 175. https://doi.org/10.1016/j.evalprogplan.2018.01.002
Gokiert, R., Kingsley, B., Poth, C., Edwards, K., El Hassar, B., Tink, L., … Hopkins, S. (2017). Developing an evaluation capacity building network in the field of early childhood development. Engaged Scholar Journal: Community-Engaged Research, Teaching, and Learning, 3(2), 59 – 80.
King, J. A. (2020). Putting evaluation capacity building in context: Reflections on the Ontario Brain Institute’s Evaluation Support Program. Evaluation and Program Planning, 80. https://doi.org/10.1016/j.evalprogplan.2017.05.013. Medline:28641901
Labin, S. N. (2014). Developing common measures in evaluation capacity building: An iterative science and practice process. American Journal of Evaluation, 35(1), 107 – 115. https://doi.org/10.1177/1098214013499965
Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3), 307 – 338. https://doi.org/10.1177/1098214011434608
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed. , pp. 13 – 104). New York, NY: Macmillan.
Minkler, M., & Wallerstein, N. (2011). Community-based participatory research for health: From process to outcomes (2nd ed.). San Francisco, CA: John Wiley & Sons.
Moss, P. A. (1998). The role of consequences in validity theory. Educational Measurement: Issues and Practice, 17(2), 6 – 12.
Nielsen, S. B., Lemire, S., & Christie, C. A. (2018). The evaluation marketplace and its industry. New Directions for Evaluation, 2018(160), 13 – 28. https://doi.org/10.1002/ev.20344
Nielsen, S. B., Lemire, S., & Skov, M. (2011). Measuring evaluation capacity: Results and implications of a Danish study. American Journal of Evaluation, 32(3), 324 – 344. https://doi.org/10.1177/1098214010396075
Patton, M. Q. (2011). Essentials of utilization-focused evaluation. Thousand Oaks, CA: SAGE.
Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443 – 459. https://doi.org/10.1177/1098214008324182
Rivers, D. C., Meade, A. W., & Fuller, W. L. (2009). Examining question and context effects in organization survey data using item response theory. Organizational Research Methods, 12(3), 529 – 553. https://doi.org/10.1177/1094428108315864
Stockdill, H. S., Baizerman, M., & Compton, D. W. (2002). Toward a definition of the ECB process: A conversation with the ECB literature. New Directions for Evaluation, 2002(93), 7 – 26. https://doi.org/10.1002/ev.39
Taylor-Ritzler, T., Suarez-Balcazar, Y., Garcia-Iriarte, E., Henry, D. B., & Balcazar, F. E. (2013). Understanding and measuring evaluation capacity: A model and instrument validation study. American Journal of Evaluation, 34(2), 190 – 206. https://doi.org/10.1177/1098214012471421
Weitzman, B. C., & Silver, D. (2013). Good evaluation measures: More than their psychometric properties. American Journal of Evaluation, 34(1), 115 – 119. https://doi.org/10.1177/1098214012461628
Wilcox, Y., & King, J. A. (2013). A professional grounding and history of the development and formal use of evaluator competencies. Canadian Journal of Program Evaluation, 28(3), 1 – 28. Retrieved from https://journalhosting.ucalgary.ca/index.php/cjpe/article/view/30949
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Btissam El Hassar, Cheryl Poth, Rebecca Gokiert, Okan Bulut
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
