Capturing Evaluation Capacity
Findings from a Mapping of Evaluation Capacity Instruments
DOI:
https://doi.org/10.3138/cjpe-2024-0001Keywords:
evaluation capacity, evaluation capacity building, measurement, research on evaluation, validityAbstract
This article surveys the literature on organizational evaluation capacity (EC) instruments. Over the past 25 years, articles have been published in four waves covering the development of organizational evaluation capacity models, surveys of the evaluation capacity and practice landscape, the development of EC instruments, and their replication and adaptation. Altogether, the authors identified 20 studies applying 16 EC instruments. Several studies applied variants of the same instrument. We found instruments of three types: checklists, rubrics, and questionnaires. The instruments vary in terms of type, purpose, scoring, dimensions, number of items, and validation processes. In general, the validation studies demonstrated acceptable face, content, construct validity, and internal consistency. Only two studies tested for predictive validity. No studies covered concurrent validity and test–retest, or inter-rater reliability.
References
Allen, M., & Yen, W. (2002). Introduction to measurement theory. Waveland Press.
Baizerman, M., Compton, D. W., & Stockdill, S. H. (2002a). Editors’ notes. The art, craft, and science of evaluation capacity building. New Directions for Evaluation, 2002(93), 1–6.
Baizerman, M., Compton, D. W., & Stockdill, S. H. (2002b). Summary and analysis of the case studies: Themes across the cases. New Directions for Evaluation, 2022(93), 101–108.
Baizerman, M., Compton, D. W., & Stockdill, S. H. (2002c). New directions for ECB. New Directions for Evaluation, 2002(93), 109–119.
Bourgeois, I., & Cousins, J. B. (2013). Understanding Dimensions of Organizational Evaluation Capacity. American Journal of Evaluation, 34(3), 299–319.
Bourgeois, I., Lemire, S. T., Fierro, L. A., Castleman, A. M., & Cho, M. (2023). Laying a solid foundation for the next generation of evaluation capacity building: Findings from an Integrative Review. American Journal of Evaluation, 44(1), 29–49.
Bourgeois, I., Simmons, L. Hotte, N., & Osseni, R. (2016). Measuring organizational evaluation capacity in Ontario public health units: An empirical investigation. Canadian Journal of Program Evaluation, 31(2), 165–183.
Bourgeois, I., Toews, E., Whynot, J., & Lamarche, M. K. (2013). Measuring organizational evaluation capacity in the Canadian federal government. Canadian Journal of Program Evaluation, 28(2), 1–19.
Boyle, R., &Lemaire, D. (Eds.). (1999). Building effective evaluation capacity: Lessons from practice. Transaction Publishers.
Buetti, D., Bourgeois, I., & Savard, S. (2019). L’étude des capacités en évaluation des organismes communautaires du Québec: proposition d’un cadre conceptuel et d’une grille d’analyse organisationnelle [Study of evaluation capacity in community organizations in Quebec: Proposal for a conceptual framework and an organizational analysis matrix]. Intervention, 150, 25–46.
Carman, J. G. (2007). Evaluation practice among community-based organizations: Research into the reality. American Journal of Evaluation, 28(1), 60–75.
Carman, J. G., & Fredericks, K. A. (2010). Evaluation capacity and nonprofit organizations: Is the glass half-empty or half-full? American Journal of Evaluation, 31(1), 84–104.
Cheng, S.-H., & King, J. A. (2017). Exploring organizational evaluation capacity and evaluation capacity building: A Delphi study of Taiwanese elementary and junior high schools. American Journal of Evaluation, 38(4), 521–539.
Cousins, J. B., Elliott, C., Amo, C., Bourgeois, I., Goh, S., & Lahey, R. (2008). Organizational capacity to do and use evaluation: Results of a pan-Canadian survey of evaluators. Canadian Journal of Program Evaluation, 23(3), 1–35.
Cousins, J. B., Goh, S. C., Clark, S., & Lee, L. E. (2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19(2), 99–141.
de Ries, K. E. Schaap, H. van Loon, A. M. M. J. A. P., Kral, M. M. H., & Meijer, P. C. (2022). A literature review of open-ended concept maps as a research instrument to study knowledge and learning. Quality & Quantity, 56, 73–107.
DeVellis, R. F., & Thorpe, C. T. (2021). Scale development: Theory and applications. Sage Publications.
El Hassar, B. (2019). Development and validation of an evaluation capacity instrument for the early childhood field [Unpublished doctoral dissertation]. University of Alberta. https://era.library.ualberta.ca/items/e7c67baf-611f-46fa-bf0e-fac6984ef350/view/5be396d0-411a-43cc-8baf-2e9f36a4be9d/ElHassar_Btissam_201901_PhD.pdf
Fierro, L. A. (2012). Clarifying the connections: Evaluation capacity and intended outcomes [Unpublished doctoral dissertation]. Claremont Graduate University.
Furubo, J-E., & Sandahl, R. (2002). A diffusion perspective on global developments in evaluation. In J-E. Furubo, R. C. Rist, & R. Sandahl (Eds.), International atlas of evaluation (pp. 1–23). Transaction Publishers.
Gagnon, F., Aubry, T., Cousins, J. B., Goh, S. C., & Elliott, C. (2018). Validation of the evaluation capacity in organizations questionnaire. Evaluation and Program Planning, 68, 166–175.
Kalkbrenner, M. T. (2021). A practical guide to instrument development and score validation in the social sciences: The MEASURE approach. Practical Assessment, Research, and Evaluation, 26(1), Article 1.
King, J. A., & Volkov, B. (2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 35(3), 10–16.
KREVI – Danish Evaluation Institute for Local Governments. (2008). Evaluering i danske kommuner. En kvantitativ kortlægning af kommunernes evalueringskapacitet [Evaluation in Danish local governments. A quantitative mapping of evaluation capacity in local governments]. https://www.vive.dk/media/pure/0xgdggxk/2061213
Leavitt, H. J. (1978). Managerial psychology. University of Chicago Press.
Lemire, S., Porowski, A., & Mumma, K. (2023). How we model matters – visualizing program theories. Abt method guide. Abt Associates. https://www.abtassociates.com/sites/default/files/files/insights/white-papers/2023/visualizing-program-theory.pdf
Mackay, K. (2002). The World Bank’s ECB experience. New Directions for Evaluation, 2002(93), 81–99.
McDonald, B., Rogers, P., & Kefford, B. (2003). Teaching people to fish? Building the evaluation capability of public sector organizations. Evaluation, 9(1), 9–29.
Milstein, B., & Cotton, D. (2000). Defining concepts for the presidential strand on building evaluation capacity. American Evaluation Association.
Ngai, S. S.-y., Cheung, C.-k., Li, Y., Zhao, L., Wang, L., Jiang, S., Tang, H.-y., & Yu, E. N.-h. (2022). Validating the evaluation capacity scale among practitioners in non-governmental organizations. Frontiers in Psychology, 13, 1082313.
Nielsen, S. B., Lemire, S., & Montague, S. (2023). Using evaluative information sensibly. The enduring contributions of John Mayne. Canadian Journal of Program Evaluation, 37(3), 473–493.
Nielsen, S. B., Lemire, S., & Skov, M. (2011). Measuring evaluation capacity: Results and implications of a Danish study. American Journal of Evaluation, 32(3), 324–44.
Preskill, H. (2014). Now for the hard stuff: Next steps in ECB research and practice. American Journal of Evaluation, 35(1), 116–119.
Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459.
Preskill, H., & Torres, R. T. (1999). Evaluative inquiry for learning in organizations. Sage.
Preskill, H., Torres, R. T., & Martinez-Papponi, B. (1999, November). Assessing an organization’s readiness for learning from evaluative inquiry [Paper presentation]. American Evaluation Association annual conference, Orlando, Florida, United States.
Royer, F., Bélanger, J., Janosz, M., & Pascal, S. (2022). Validation du questionnaire sur la capacité évaluative des organisations et ses déterminants (QCEOD) [Validation of the questionnaire on evaluation capacity of organizations and its determinants (QCEOD)]. Canadian Journal of Program Evaluation, 37(2), 232–255.
Schwarzman, J., Bauman, A., Gabbe, B. J., Rissel, C., Shilton, T., & Smith, B. J. (2019). Understanding the factors that influence health promotion evaluation: The development and validation of the evaluation practice analysis survey. Evaluation and Program Planning, 74, 76–83.
Stufflebeam, D. L. (2002). Institutionalizing evaluation checklist. https://wmich.edu/evaluation/checklists
Suarez-Balcazar, Y., & Taylor-Ritzler, T. (2013). Moving from science to practice in evaluation capacity building. American Journal of Evaluation, 35(1), 95–99.
Suarez-Balcazar, Y., Taylor-Ritzler, T., Garcia-Iriarte, E., Keys, C. B., Kinney, L., Ruch-Ross, H. & Curtin, G. (2010). Evaluation capacity building: A culturally- and contextually-grounded interactive framework and exemplar. In F. Balcazar, Y. Suarez-Balcazar, T. Taylor-Ritzler, & C. B. Keys (Eds.), Race, culture and disability: Rehabilitation science and practice (pp. 307–324). Jones and Bartlett.
Taut, S. (2007a). Studying self-evaluation capacity building in a large international development organization. American Journal of Evaluation, 28(1), 45–59.
Taut, S. (2007b). Defining evaluation capacity building: Utility considerations. American Journal of Evaluation, 28(1), 120.
Taylor-Ritzler, T., Suarez-Balcazar, Y., Garcia-Iriarte, E., Henry, D. B., & Balcazar, F. E. (2013). Understanding and measuring evaluation capacity: A model and instrument validation study. American Journal of Evaluation, 34(2), 190–206.
Tseng, S. H. (2011). An assessment scale of ECB: Perspectives from the faculty and staff of technical colleges in Taiwan. Evaluation and Program Planning, 34(2), 154–159.
van Voorst, S. (2017). Evaluation capacity in the European commission. Evaluation, 23(1), 24–41.
Volkov, B. B., & King, J. K. (2007). Checklist for building organizational evaluation capacity. https://wmich.edu/evaluation/checklists
Walker-Egea, C. F. (2014). Design and validation of an evaluation checklist for organizational readiness for evaluation capacity development [Unpublished doctoral dissertation, University of South Florida]. Digital Commons @ University of South Florida. https://digitalcommons.usf.edu/etd/5399
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Steffen Bohni Nielsen, Sebastian Lemire, Isabelle Bourgeois, Leslie A. Fierro
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
