Validation du questionnaire sur la capacité évaluative des organisations et ses déterminants (QCEOD)
DOI:
https://doi.org/10.3138/cjpe.73451Keywords:
evaluation capacity, evaluation capacity building, validation, conceptual framework, questionnaireAbstract
This article describes the design and validation process of the “questionnaire sur la capacité évaluative des organisations et ses déterminants” (QCEOD). In total, 282 respondents from two different organizations completed the questionnaire which, in its final version, contains between 70 and 108 items. Internal consistency analysis combined with confirmatory factor analysis confirmed the validity and reliability of the questionnaire. The limits and implications of these results for research and practice, particularly in relation to evaluation capacity building (ECB), are discussed.
References
Abrami, P. C., Dagenais, C., Bernard, R. M., Janosz, M., Pigeon, M.-E., Lysenko, L. V. et Levesque, J. (2007). Questionnaire sur l’utilisation des connaissances issues de la recherche dans la pratique professionnelle.
Alimo-Metcalfe, B. et Alban-Metcalfe, J. (2005). The crucial role of leadership in meeting the challenges of change. Vision: Th e Journal of Business Perspective, 9 (2), 27–39. doi: https://doi.org/10.1177/097226290500900205
Atwood, M. A., Mora, J. W. et Kaplan, A. W. (2010). Learning to lead: evaluating leadership and organizational learning. Leadership & Organization Development Journal, 31(7), 576–595. doi: https://doi.org/10.1108/01437731011079637
Bess, K. D., Perkins, D. D. et McCown, D. L. (2010). Testing a measure of organizational learning capacity and readiness for transformational change in human services. Journal of prevention & intervention in the community, 39(1), 35–49. doi: https://doi.org/10.1080/10852352.2011.530164
Bourgeois, I., Chouinard, J. A. et Cousins, J. B. (2008). Understanding Organization Capacity for Evaluation: Synthesis and Integration. Canadian Journal of Program Evaluation, 23(3), 225–237.
Bourgeois, I. et Cousins, J. B. (2013). Understanding Dimensions of Organizational Evaluation Capacity. American Journal of Evaluation, 34 (3), 299–319. doi: https://doi.org/10.1177/1098214013477235
Bourgeois, I. et Hurteau, M. (2018). L’implication des parties prenantes dans la démarche évaluative: facteurs de succès et leçons à retenir. Canadian Journal of Program Evaluation, 33(2).
Bourgeois, I., Simmons, L., Hotte, N. et Osseni, R. (2016). Measuring evaluation capacity in Ontario public health units. Canadian Journal of Program Evaluation, 31(2).
Bowen, G. L., Rose, R. A. et Ware, W. B. (2006). The reliability and validity of the School Success Profile Learning Organization measure. Evaluation and Program Planning, 29(1), 97–104. doi: https://doi.org/10.1016/j.evalprogplan.2005.08.005
Brown, T., McCracken, M. et O’Kane, P. (2011). ‘Don’t forget to write’: how reflective learning journals can help to facilitate, assess and evaluate training transfer. Human Resource Development International, 14 (4), 465–481. doi: https://doi.org/10.1080/13678868.2011.601595
Brown, T. A. (2015). Confirmatory factor analysis for applied research. New York, NY: Guilford Publications.
Bruner Foundation. (2010). The Modified Evaluative Th inking Assessment Tool. Retrieved from: http://www.evaluativethinking.org/docs/sample.report.pdf
Chagnon, F., Bardon, C., Proulx, R., Houlfort, N., Briand-Lamarche, M. et Labelle, P. (2012). Outil réflexif pour améliorer l’utilisation des connaissances dans les organisations de services sociaux et de santé. Montréal: Chaire CJM-IU-UQAM d’étude sur l’application des connaissances dans le domaine des jeunes et des familles en difficulté.
Chiva, R., Alegre, J. et Lapiedra, R. (2007). Measuring organisational learning capability among the workforce. International Journal of Manpower, 28 (3/4), 224–242. doi: https://doi.org/10.1108/01437720710755227
Compton, D. W. et Baizerman, M. (2007). Defining evaluation capacity building. American Journal of Evaluation, 28(1), 118–119.
Comrey, A. L. et Lee, H. B. (1992). A first course in factor analysis. Hillsdale, NJ: Erlbaum.
Corn, J. O., Byrom, E., Knestis, K., Matzen, N. et Th rift , B. (2012). Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders. Evaluation and Program Planning, 35(4), 535–542. doi: https://doi.org/10.1016/j.evalprogplan.2011.12.008
Cousins, J. B., Elliott, C., Amo, C., Bourgeois, I., Chouinard, J., Goh, S. C. et Lahey, R. (2008). Organizational Capacity to Do and Use Evaluation: Results of a Pan-Canadian Survey of Evaluators. Canadian Journal of Program Evaluation, 23(3).
Cousins, J. B., Goh, S., Clark, S. et Lee, L. (2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19 (2), 99–141.
Cousins, J. B., Goh, S. C., Elliott, C., Aubry, T. et Gilbert, N. (2014). Government and voluntary sector differences in organizational capacity to do and use evaluation. Evaluation and program planning, 44 , 1–13. doi: https://doi.org/10.1016/j.evalprogplan.2013.12.001
Cousins, J. B., Goh, S. C., Elliott, C. J. et Bourgeois, I. (2014). Framing the Capacity to Do and Use Evaluation. New Directions for Evaluation, 2014 (141), 7–23. doi: https://doi.org/10.1002/ev.20076
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. psychometrika, 16 (3), 297–334.
Eccles, J. S. et Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual review of psychology, 53 (1), 109–132. doi: https://doi.org/10.1146/annurev.psych.53.100901.135153
El Hassar, B., Poth, C., Gokiert, R. et Bulut, O. (2020). Toward an Evidence-Based Approach to Building Evaluation Capacity. Canadian Journal of Program Evaluation, 36(1), 82–94. doi: https://doi.org/10.3138/cjpe.69191
Everitt, B. S. (1975). Multivariate analysis: Th e need for data, and other problems. The British Journal of Psychiatry, 126(3), 237–240.
Expósito-Langa, M., Molina-Morales, F. X. et Tomás-Miquel, J.-V. (2015). How shared vision moderates the effects of absorptive capacity and networking on clustered firms’ innovation. Scandinavian Journal of Management, 31 (3), 293–302. doi: https://doi.org/10.1016/j.scaman.2015.06.001
Fullan, M., Cuttress, C. et Kilcher, A. (2005). 8 forces for leaders of change. Journal of Staff Development, 26 , 54–58.
Gagnon, F., Aubry, T., Cousins, J. B., Goh, S. C. et Elliott, C. (2018). Validation of the evaluation capacity in organizations questionnaire. Evaluation and program planning, 68, 166–175. doi: https://doi.org/10.1016/j.evalprogplan.2018.01.002
Goh, S. C. (1998). Toward a learning organization: Th e strategic building blocks. SAM Advanced Management Journal, 63 , 15–22.
Goh, S. C. (2003). Improving organizational learning capability: lessons from two case studies. The Learning Organization, 10(4), 216–227. doi: https://doi.org/10.1108/09696470310476981
Gouvernement du Québec. (2000). Loi sur l’administration publique.
Harris, A. (2006). Opening up the ‘black box’of leadership practice: Taking a distributed leadership perspective. Leadership and Management, 34 (2), 37–45.
Janosz, M. et Bouthillier, C. (2007). Rapport de validation du Questionnaire sur l’Environnement Socioéducatif des écoles secondaires (QES-secondaire). Montréal: Université de Montréal.
Janzen, R., Ochocka, J., Turner, L., Cook, T., Franklin, M. et Deichert, D. (2017). Building a community-based culture of evaluation. Evaluation and Program Planning, 65 , 163–170.
Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F. et Volkov, B. (2009). Research on Evaluation Use A Review of the Empirical Literature From 1986 to 2005. American Journal of Evaluation, 30 (3), 377–410. doi: https://doi.org/10.1177/1098214009341660
Johnson, L. I. (2021). Evaluation Capacity Case Study: Organizational Assessment and Recommendations to Build Evaluation Capacity.
King, J. A. (2020). Putting evaluation capacity building in context: Reflections on the Ontario Brain Institute’s Evaluation Support Program. Evaluation and program planning, 80, 101452.
King, J. A. et Volkov, B. (2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 35 (3), 10–16.
Kuzmin, A. (2009). Use of evaluation training in evaluation capacity building. From policies to results, 240.
Labin, S. N. (2014). Developing Common Measures in Evaluation Capacity Building An Iterative Science and Practice Process. American Journal of Evaluation, 35 (1), 107–115. doi: https://doi.org/10.1177/1098214013499965
Labin, S. N., Duff y, J. L., Meyers, D. C., Wandersman, A. et Lesesne, C. A. (2012). A Research Synthesis of the Evaluation Capacity Building Literature. American Journal of Evaluation, 33(3), 307–338. doi: https://doi.org/10.1177/1098214011434608
Mark, M. M. et Henry, G. T. (2004). Th e mechanisms and outcomes of evaluation influence. Evaluation, 10(1), 35–57. doi: https://doi.org/10.1177/1356389004042326
Marsick, V. J. et Watkins, K. E. (2003). Demonstrating the value of an organization’s learning culture: the dimensions of the learning organization questionnaire. Advances in developing human resources, 5 (2), 132–151. doi: https://doi.org/10.1177/1523422303005002002
McIntosh, J. S., Buckley, J. et Archibald, T. (2020). Refi ning and measuring the construct of evaluative thinking: An exploratory factor analysis of the Evaluative Thinking Inventory. Journal of MultiDisciplinary Evaluation, 16 (34), 104–117.
Nielsen, S. B., Lemire, S. et Skov, M. (2011). Measuring Evaluation Capacity—Results and Implications of a Danish Study. American Journal of Evaluation, 32 (3), 324–344. doi: https://doi.org/10.1177/1098214010396075
Osborne, J. W. et Costello, A. B. (2009). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Pan-Pacific Management Review, 12 (2), 131–146. doi: https://doi.org/10.7275/jyj1-4868
Patton, M. Q. (2008). Utilization-focused evaluation. Thousand Oaks, CA: Sage publications.
Preskill, H. et Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29 (4), 443–459. doi: https://doi.org/10.1177/1098214008324182
Preskill, H. et Torres, R. (2000). Readiness for organizational learning and evaluation instrument. Available from H. Preskill, Hallie. preskill@fsg.org.
Preskill, H. et Torres, R. T. (1999). Building capacity for organizational learning through evaluative inquiry. Evaluation, 5 (1), 42–60. doi: https://doi.org/10.1177/135638909900500104
Schein, E. H. (2010). Organizational culture and leadership. San Francisco, CA: John Wiley & Sons.
Smits, P. et Jacob, S. (2014). La fonction d’évaluation dans l’administration publique québécoise: analyse de la cohérence du système d’actions. Canadian Public Administration, 57(1), 71–96. doi: https://doi.org/10.1111/capa.12059
Spector, J. M. et Davidsen, P. I. (2006). How can organizational learning be modeled and measured? Evaluation and program planning, 29 (1), 63–69. doi: https://doi.org/10.1016/j.evalprogplan.2005.08.001
Stockdill, S. H., Baizerman, M. et Compton, D. W. (2002). Toward a defi nition of the ECB process: A conversation with the ECB literature. New Directions for Evaluation, 2002 (93), 7–26. doi: https://doi.org/10.1002/ev.39
Suarez-Balcazar, Y., Taylor-Ritzler, T., Garcia-Iriarte, E., Keys, C. B., Kinney, L., & Ruch-Ross, H., . . . Curtin, G. (2010). Evaluation capacity building: A culturally and contextually grounded interactive framework and exemplar. Dans Y. S.-B. F. Balcazar, T. Taylor-Ritzler, & C. B. Keys (dir.), Race, culture and disability: Rehabilitation science and practice (p. 307–324). Sudbury, MA: Jones and Bartlett.
Swope, M. (2021). From Checked Out to Bought In: Exploring Individual Level Motivation in Evaluation Capacity Building. (The Claremont Graduate University).
Tabachnick, B. G. et Fidell, L. S. (2007). Using multivariate statistics . Boston, MA: Pearson.
Taylor-Powell, E. et Boyd, H. H. (2008). Evaluation capacity building in complex organizations. New Directions for Evaluation, 2008 (120), 55–69. doi: https://doi.org/10.1002/ev.276
Taylor-Ritzler, T., Suarez-Balcazar, Y., Garcia-Iriarte, E., Henry, D. B. et Balcazar, F. E. (2013). Understanding and Measuring Evaluation Capacity A Model and Instrument
Validation Study. American Journal of Evaluation, 34 (2), 190–206. doi: https://doi.org/10.1177/1098214012471421
Tohidi, H., Seyedaliakbar, S. M. et Mandegari, M. (2012). Organizational learning measurement and the effect on firm innovation. Journal of Enterprise Information Management, 25 (3), 219–245. doi: https://doi.org/10.1108/17410391211224390
Volkov, B. et King, J. A. (2007). A checklist for building organizational evaluation capacity. The Evaluation Center: Western Michigan University. Accessed February, 5, 2015.
Wandersman, A. (2014). Moving Forward With the Science and Practice of Evaluation Capacity Building (ECB) The Why, How, What, and Outcomes of ECB. American Journal of Evaluation, 35 (1), 87–89.
Wolf, E. J., Harrington, K. M., Clark, S. L. et Miller, M. W. (2013). Sample size requirements for structural equation models: An evaluation of power, bias, and solution propriety. Educational and psychological measurement, 73 (6), 913–934.
Yang, B. (2003). Identifying valid and reliable measures for dimensions of a learning culture. Advances in Developing Human Resources, 5 (2), 152–162. doi: https://doi.org/10.1177/1523422303005002003
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 François, Jean Bélanger, Michel Janosz, Sophie Pascal
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
