Evaluation Literacy and Use in Community Organizations
Insights and Implications for Evaluation Capacity Building
DOI:
https://doi.org/10.3138/cjpe-2024-0002Keywords:
community organizations, evaluation capacity building, evaluation use, Organizational Evaluation Capacity Assessment Instrument (OECAI)Abstract
Past research on evaluation capacity in community organizations has focused on their capacity to conduct rather than use evaluation. This study sought to address this gap by exploring evaluation use and the supporting processes, systems, and structures in the community sector. The authors administered an online survey to 102 individuals working or involved in community organizations to measure their perceptions of their organization’s capacity to use evaluation. The survey was based on an adaptation of Bourgeois and Cousins’ evaluation capacity framework and a related assessment tool, the Organizational Evaluation Capacity Assessment Instrument, for the community sector context. Their findings offer empirical insights into three critical dimensions of the adapted framework associated with evaluation use in community organizations: evaluation literacy, organizational decision-making, and learning benefits. This exploratory study generated several key findings, including the need for tools and policies that establish a strong link between evaluation and decision-making, expanding the scope of engagement in data interpretation beyond primary users, and extending the dissemination of findings to external stakeholders. Implications for practice and research are discussed.
References
Ahrens, V., Cruz, E., Pasturczak, M., Bakken, L. L., & Moore, T. R. (2023). Between funding requirements and community priorities: Centro Hispano of Dane County’s transformative approach to program evaluation. Journal of Multidisciplinary Evaluation, 19(44). 197–212.
Akintobi, T. H., Yancey, E. M., Daniels, P., Mayberry, R. M., Jacobs, D., & Berry, J. (2012). Using evaluability assessment and evaluation capacity-building to strengthen community-based prevention initiatives. Journal of Health Care for the Poor and Underserved, 23(2), 33–48.
Al Hudib, H., & Cousins, J. B. (2022). Evaluation policy and organizational evaluation capacity building: a study of international aid agency evaluation policies. New Directions for Evaluation, 2022(173), 29–48.
Alkin M. C., Taut S. M. (2002). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1–12.
Bhawra, J., & Skinner, K. (2020). Examination of tools associated with the evaluation of knowledge uptake and utilization: A scoping review. Evaluation and Program Planning, 83, 101875.
Bourgeois, I., & Cousins, J. B. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34(3), 299–319.
Bourgeois, I., Whynot, J., & Thériault, É. (2015). Application of an organizational evaluation capacity self-assessment instrument to different organizations: Similarities and lessons learned. Evaluation and Program Planning, 50, 47–55.
Bourgeois, I., Lemire, S. T., Fierro, L. A., Castleman, A. M., & Cho, M. (2023). Laying a Solid Foundation for the Next Generation of Evaluation Capacity Building: Findings from an Integrative Review. American Journal of Evaluation, 44(1), 29–49.
Buetti, D., Bourgeois, I., & Savard, S. (2019a). L’étude des capacités en évaluation des organismes communautaires du Québec : proposition d’un cadre conceptuel et d’une grille d’analyse organisationnelle [Understanding Evaluation Capacity in Community-Based Organizations of Quebec : Presentation of a Conceptual Framework and a Measurement Instrument]. Intervention, 150, 25–42.
Buetti, D., Bourgeois, I., & Savard, S. (2019b). Modélisation des capacités organisationnelles en évaluation dans le secteur communautaire et implications pour le contexte québécois [Modelisation of Evaluation Capacity in Community-Based Organizations]. Canadian Journal of Nonprofit and Social Economy Research, 10(2), 16–39.
Carman, J. G. (2007). Evaluation practice among CBO: Research into the reality. American Journal of Evaluation, 28(1), 60–75.
Carman, J., & Fredericks, K. (2010). Evaluation capacity and nonprofit organizations: Is the glass half-empty or half-full? American Journal of Evaluation, 31(1), 84–104.
Carnochan, S., Samples, M., Myers, M., & Austin, M. J. (2014). Performance Measurement Challenges in Nonprofit Human Service Organizations. Nonprofit and Voluntary Sector Quarterly, 43(6), 1014–1032.
Cheverton, J. (2007). Holding our own: Value and performance in nonprofit organisations. Australian Journal of Social Issues, 42(3), 427–436.
Cousins, J. B. (2003). Utilization effects of participatory evaluation. In T. Kellaghan & D. L. Stufflebeam (Eds.), International handbook of educational evaluation (pp. 245–265). Springer.
Cousins, J.B., Goh, S.C., Clark. S., & Lee, L. (2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19(2), 99–141.
Cousins, J. B., Goh, S. C., Elliott, C., Aubry, T., & Gilbert, N. (2014). Government and voluntary sector differences in organizational capacity to do and use evaluation. Evaluation and Program Planning, 44, 1–13.
Edwards, C., Jensen, E., & Muermann, S. (2022). Core funding supporting nonprofit missions, resilience, and impact. Imagine Canada. https://www.imaginecanada.ca/sites/default/files/core-funding-brief.pdf
Hunter, S. B., Chinman, M., Ebener, P., Imm, P., Wandersman, A., & Ryan, G. W. (2009). Technical assistance as a prevention capacity-building tool: a demonstration using the Getting To Outcomes® Framework. Health Education & Behavior, 36(5), 810–828.
Imagine Canada. (2019). The state of evaluation measurement and evaluation practices in Canada’s charitable sector. https://www.imaginecanada.ca/sites/default/files/2019-06/state_of_evaluation-national_report.pdf
Janzen, R., Ochocka, J., Turner, L., Cook, T., Franklin, M., & Deichert, D. (2017). Building a community-based culture of evaluation. Evaluation and Program Planning, 65, 163–170.
LaMarre, Riley, B., Ruchika J., Zupko, B., & Buetti, D. (2020) Chronic disease prevention evaluation in Ontario’s public health system: A qualitative needs assessment. Canadian Journal of Public Health, 111(6), 1002–1010.
Landes-Cobb, L., Kramer, K., & Smith Milway, K. (2015, October 22). The nonprofit leadership development deficit. Stanford Social Innovation Review. https://ssir.org/articles/entry/the_nonprofit_leadership_development_deficit
McDuff, M. D. (2001). Building the capacity of grassroots conservation organizations to conduct participatory evaluation. Environmental Management, 27(5), 715–727.
Murphy, J. W. (2014). Community-based interventions: Philosophy and action. Springer.
Nielsen, S. B., & Ejler, N. (2008). Improving performance? exploring the complementarities between evaluation and performance management. Evaluation, 14(2), 171–192.
Nielsen, S. B., Lemire. S.T., Bourgeois, I., & Fierro, L. (2024). Capturing evaluation capacity: Findings from a mapping of evaluation capacity instruments. Canadian Journal of Program Evaluation, 38(3), 433–460.
Okul, E. O., & Nyonje, R. O. (2020). Examining stakeholder involvement in the evaluation process for program improvement. International Journal of Research in Business and Social Science, 9(5), 179–191.
Rohacek, M. (2017). Research and Evaluation Capacity Self-Assessment tool and discussion guide. https://www.urban.org/sites/default/files/publication/97146/research_and_evaluation_capacity-_self-assessment_tool_and_discussion_guide_for_ccdf_lead_agencies.pdf
Sieppert, J. D. (2005). Evaluation in Canada’s social services: Progress, rifts, and challenges. Canadian Journal of Program Evaluation, 20 (3), 101–121.
St-Germain, L., Grenier, J., Bourque, M., & Pelland, D. (2017). Enjeux d’autonomie de l’action communautaire autonome (ACA) à partir de l’analyse des discours de rapports d’activités et des acteurs [Autonomy challenges for community action based on the analysis of of activity reports and actors]. Nouvelles pratiques sociales, 29(1–2), 102–120.
Tello-Rozas, S., Léonard, M., & Lussier-Lejeune, F. (2022). Mise à jour du portrait des pratiques d’évaluation dans les organismes communautaires [Update of the overview of evaluation practices in community organizations]. https://sac.uqam.ca/upload/files/Miseajourportraiteval_Rapport_v5_Web.pdf
Wilson, M. G., Lavis, J. N., & Guta, A. (2012). Community-based organizations in the health sector: a scoping review. Health Research Policy and Systems, 10(1), 36–43.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 David Buetti, Isabelle Bourgeois, Sébastien Savard
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
