Examining Youth Sector Stakeholders’ Experiences in an Online Program Evaluation Certificate
DOI:
https://doi.org/10.3138/cjpe-2023-0012Keywords:
online learning, education, capacity building, quantitativeAbstract
Evaluation is a critical skill for practitioners, yet many youth workers lack sufficient training. The current study examined youth sector stakeholders’ experiences in an online program evaluation certificate. A total of 233 participants from six cohorts (i.e., three years) completed pre- and post-certificate surveys measuring their satisfaction and learnings from the certificate. Results revealed that participants were generally satisfied with the certificate and reported significant improvements in their perceived knowledge after participation (p < .001). Findings point to the utility of online certificates for enhancing evaluation capacity in the Canadian youth work sphere and inform future directions for research and practice.
References
Ahmady, A. E., Barker, M., Fahim, M., Dragonetti, R., & Selby, P. (2017). Evaluation of web-based continuing professional development courses: Aggregate mixed-methods model. JMIR Medical Education, 3(2), e7480.
Ager, W., Dow, J., & Gee, M. (2005). Grassroots networks: A model for promoting the influence of service users and carers in social work education. Social Work Education, 24(4), 467–476.
Allen, I., & Seaman, J. (2007). Changing the landscape: More institutions pursue online offerings. On the Horizon, 15(3), 130–138.
Amo, C. J. & Cousins, J. B. (2007). Going through the process: An examination of the operationalization of process use in empirical research on evaluation. New Directions for Evaluation, 116, 5–26.
Arnold, M. E., Cater, M., & Braverman, M. T. (2016). Rethinking evaluation capacity in youth development programs: A new approach for engaging youth workers in program evaluation. In K. Pozzoboni & B. Kirshner (Eds.), The changing landscape of youth work: Theory and practice for an evolving field (pp. 193–194). Information Age Publishing.
Bakken, L. L., Núñez, J., & Couture, C. (2014). A course model for building evaluation capacity through a university–community partnership. American Journal of Evaluation, 35(4), 579–593.
Baker, L. R., Stephens, F., & Hitchcock, L. (2010). Social work practitioners and practice evaluation: How are we doing? Journal of Human Behaviour in the Social Environment, 20(8), 963–973.
Bangert., A. W. (2006). Identifying factors underlying the quality of online teaching effectiveness: An exploratory study. Journal of Computing in Higher Education, 17(2), 79–99.
Bean, C., Rahmani, A. & Anucha, U. (2020). A mixed-methods process evaluation of an online program evaluation certificate for youth sector stakeholders. Hawaii International Conference on Education, 113.
Becerra, Z. M., Fereydooni, N., Kun, A. L., McKerral, A., Riener, A., Schartmuller, C., … Kun, A. L. (2021). Interactive workshops in a pandemic: The real benefits of virtual spaces. IEEE Pervasive Computing, 20(1), 35–39.
Beere, D. (2005). Evaluation capacity-building: A tale of value-adding. Evaluation Journal of Australasia, 5(2), 41–47.
Bourgeois, I., Chouinard, J. A., & Bradley Cousins, J. (2008). Understanding organization capacity for evaluation: Synthesis and integration. Canadian Journal of Program Evaluation, 23(3), 225–237.
Bourgeois, I., Simmons, L., Hotte, N., & Osseni, R. (2016). Measuring evaluation capacity in Ontario public health units. Canadian Journal of Program Evaluation, 31(2), 165–183.
Broad, K., & Evans, M. (2006). A review of literature on professional development content and delivery modes for experienced teachers. Ontario Institute for Studies in Education, University of Toronto. https://shorturl.at/sCIR7
Brown, W. A., Andersson, F. O., & Jo, S. (2016). Dimensions of capacity in nonprofit human service organizations. Voluntas, 27(6), 2889–2912.
Cairns, B., Harris, M., & Young, P. (2005). Building the capacity of the voluntary nonprofit sector: Challenges of theory and practice. International Journal of Public Administration, 28(9–10), 869–885.
Carman, J. G. (2007). Evaluation practice among community-based organizations: Research into the reality. The American Journal of Evaluation, 28(1), 60–75.
Carman, J. G., & Fredericks, K. A. (2010). Evaluation capacity and nonprofit organizations: Is the glass half-empty or half-full? American Journal of Evaluation, 31(1), 84–104.
Cole, D. C., Aslanyan, G., Dunn, A., Boyd, A., & Bates, I. (2014). Dilemmas of evaluation: Health research capacity initiatives. Bulletin of the World Health Organization, 92(12), 920–921.
Cousins, J. B., Goh, S. C., Clark, S., & Lee, L. E. (2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19(2), 99–141.
Dabbagh, N. (2007). The online learner: Characteristics and pedagogical implications. Contemporary Issues in Technology and Teacher Education, 7(3), 217–226.
Davis, M. (2006). Teaching practical public health evaluation methods. American Journal of Evaluation, 27, 247–256.
Doherty, A., Misener, K., & Cuskelly, G. (2014). Toward a multidimensional framework of capacity in community sport clubs. Nonprofit and Voluntary Sector Quarterly, 43(2), 124–142.
Eom, S. B., Wen, H. J., & Ashill, N. (2006). The determinants of students’ perceived learning outcomes and satisfaction in university online education: An empirical investigation. Decision Sciences Journal of Innovative Education, 4(2), 215–235.
Forgey, M. A. & Ortega-Williams, A. (2016). Effectively teaching social work practice online: Moving beyond can to how. Advances in Social Work, 17(1), 59–77.
Hall, M. H., Andrukow, A., Barr, C., Brock, K., de Wit, M., Embuldeniya, D., … Vallaincourt, Y. (2003). The capacity to serve: A qualitative study of the challenges facing Canada’s nonprofit and voluntary organizations. Canadian Centre for Philanthropy. Retrieved from https://shorturl.at/ckoCD
Horton, D. (2002). Planning, implementing, and evaluating capacity development. Research in Agricultural & Applied Economics.
Horton, D., Alexaki, A., Bennett-Lartey, S., Brice, K. N., Campilan, D., Carden, F., … Watts, J. (2003). Evaluating capacity development: Experiences from research and development organizations around the world. International Service for National Agriculture Research.
Ke, F., & Xie, K. (2009). Toward deep learning for adult students in online courses. The Internet and Higher Education, 12(3), 136–145.
Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick’s four levels of training evaluation. Association for Talent Development.
Kirkpatrick, J. D., & Kirkpatrick, W. K. (2023). Kirkpatrick’s four levels of training evaluation. Association for Talent Development.
Kuh, G. D., & Hu, S. (2001). The relationships between computer and information technology use, student learning, and other college experiences. Journal of College Student Development, 42, 217–232.
Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. The American Journal of Evaluation, 33(3), 307–338.
Lee, C., & Nowell, B. (2015). A framework for assessing the performance of nonprofit organizations. American Journal of Evaluation, 36(3), 299–319.
Liket, K. C., Maas, K., & Rey Garcia, M. (2014). Why aren’t evaluations working and what to do about it: A framework for negotiating meaningful evaluation in nonprofits. The American Journal of Evaluation, 35(2), 171–188.
Lovell, A., Anucha, U., Houwer, R., & Galley, A. (2016). Beyond measure? The state of evaluation and action in Ontario’s youth sector. Youth Research & Evaluation eXchange (YouthREX). Retrieved from https://YouthREX.com/wp-content/uploads/2017/12/YouthREX-BeyondMeasure-Report.pdf
McDonald, B., Rogers, P., & Kefford, B. (2003). Teaching people to fish? Building the evaluation capability of public sector organizations. Evaluation, 9(1), 9–29.
McDuff, M. D. (2001). Building the capacity of grassroots conservation organizations to conduct participatory evaluation. Environmental Management, 27(5), 715–727.
McRoberts, R., & Leitch, R. (1998). Best of the old alongside the new: evaluating an integrated training programme for part-time youth workers. Journal of Vocational Education & Training, 50(1), 21–40.
Means, B. (2010). Technology and education change: Focus on student learning. Journal of Research on Technology in Education, 42(3), 285–307.
Metz, A. J. R. (2007). Why conduct program evaluation? Five reasons why evaluation can help an out-of-school time program. Child Trends. https://cyfar.org/sites/default/files/Child_Trends-2007_10_01_RB_WhyProgEval.pdf
Misener, K., & Doherty, A. (2009). A case study of organizational capacity in nonprofit community sport. Journal of Sport Management, 23(4), 457–482.
Moore, M. G. (1991). Editorial: Distance education theory. The American Journal of Distance Education, 5(3), 1–6.
Muirhead, B. (2004). Encouraging interaction in online classes. International Journal of Instructional Technology and Distance Learning, 1(6), 45–50.
Naccarella, L., Pirkis, J., Kohn, F., Morley, B., Burgess, P., & Blashki, G. (2007). Building evaluation capacity: Definitional and practical implications from an Australian case study. Evaluation and Program Planning, 30(3), 231–236.
Owen, J. M. (2003). Evaluation culture: A definition and analysis of its development within organisations. Evaluation Journal of Australasia, 3(1), 43–47.
Park, J. H., & Choi, H. J. (2009). Factors influencing adult learners’ decision to drop out or persist in online learning. Educational Technology & Society, 12(4), 207–217.
Patton, M. Q. (2018). Evaluation science. American Journal of Evaluation, 39(2), 183–200.
Parsons, C., & Hick, S. F. (2008). Moving from the digital divide to digital inclusion. Currents: Scholarship in the Human Services, 7(2), 1–16.
Powers, J. L., & Tiffany, J. S. (2006). Engaging youth in participatory research and evaluation. Journal of Public Health Management and Practice, 12(6), S79–S87.
Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 24(4), 443–459.
Quinney, A., Hutchings, M., & Scammell, J. (2008). Student and staff experiences of using a virtual community, Wessex Bay, to support interprofessional learning: Messages for collaborative practice. Social Work Education, 27(6), 658–664.
Ruey, S. (2010). A case study of constructivist instructional strategies for adult online learning. British Journal of Educational Technology, 41(5), 706–720.
Sarti, A. J., Sutherland, S., Landriault, A., DesRosier, K., Brien, S., & Cardinal, P. (2017). Understanding of evaluation capacity building in practice: A case study of a national medical education organization. Advances in Medical Education and Practice, 8, 761–767.
Shaikh, M., Bean, C., & Forneris, T. (2020). Six recommendations for youth sport stakeholders when evaluating their programs. Journal of Sport Psychology in Action, 11(3), 165–182.
Sharpe, E. K. (2006). Resources at the grassroots of recreation: Organizational capacity and quality of experience in a community sport organization. Leisure Sciences, 28(4), 385–401.
Shearer, R. L. (2003). Interaction in distance education. Atwood Publishing.
Song, L., Singleton, E. S., Hill, J. R., & Koh, M. H. (2004). Improving online learning: Student perceptions of useful and challenging characteristics. The Internet and Higher Education, 7(1), 59–70
Stockdill, S., Baizerman, M., & Compton, D. W. (2002). Toward a definition of the ECB process: A conversation with the ECB literature. New Directions for Evaluation, 2002(93), 7–26.
Sundar, P., Kasprzak, S., Halsall, T., & Woltman, H. (2011). Using web-based technologies to increase evaluation capacity in organizations providing child and youth mental health services. The Canadian Journal of Program Evaluation, 25, 91–112.
Wicker, P., & Breuer, C. (2011). Scarcity of resources in German non-profit sport clubs. Sport Management Review, 14(2), 188–201.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Corliss Bean, Caroline Hummell, Dorrie Fiissel, David Galano, Uzo Anucha
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
