Professionalism in Program Evaluators
A Comparison of American and Canadian Evaluators
DOI:
https://doi.org/10.3138/cjpe.71300Keywords:
credentialing, ethics, expertise, innovation and research, professional autonomy, professionalism, professionalizationAbstract
The professionalization of evaluation means different things to different people, and as a result, the field lacks a clear understanding of how to empirically assess evaluator professionalism. This exploratory study used a sociological model to study the behaviours of practicing evaluators in Canada and the United States using five concepts: expertise, ethical disposition, professional autonomy, innovation and research, and credentialing. Results from 27 in-depth interviews and a survey of 456 respondents demonstrated that, depending on their demographic characteristics, perceptions of professionalism differed between evaluators in the two countries. The study determined that theory-based sociological models can provide insights into the professionalization of program evaluation.
References
Altschuld , J. W. , & Engle , M. (2015). Te inexorable historical press of the developing evaluation profession. New Directions for Evaluation, 2015 (145), 5 – 19. https://doi. org/10.1002/ev.20108
American Evaluation Association (AEA). (n.d.). http://www.eval.org
Ayoo , S. (2020). Professionalization of program evaluation: A comparative mixed methods case study of Canada and the United States (Unpublished doctoral dissertation). University of Minnesota.
Brody , H. , & Doukas , D. (2014). Professionalism: A framework to guide medical education . Medical Education, 48( 10 ), 980 – 987 . https://doi.org/10.1111/medu.12520. Medline:25200018
Cohen , J. (1988). Statistical power analysis for the behavioral sciences. Te New York Academy of Sciences.
Corbin , J. , & Strauss , A. (2008). Strategies for qualitative data analysis. In Basics of qualitative research: Techniques and procedures for developing grounded theory (3rd ed., pp. 65-86). Sage.
Cousins, J. B. , Cullen, J. , Malik, S. , & Maicher, B. (2009). Debating professional designations for evaluators. Journal of Multidisciplinary Evaluation, 6(11), 71 – 82 .
Creswell , J. W. , & Plano Clark , V. L. (2018). Designing and conducting mixed methods research. Sage .
Davidson-Shivers , G. V. , & Barrington , M. E. (2004). Revisiting the professional status of instructional design and technology and the specializations within. Association for Educational Communications and Technology. https://eric.ed.gov/?id=ED485073
Dillman , D. A. , Smyth , J. D. , & Christian , L. M. (2014). Internet, phone, mail and mixed-mode surveys: Te tailored design method ( 4th ed.). Wiley .
Fetters , M. D. , & Molina-Azorin , J. F. (2017). The Journal of Mixed Methods Research starts a new decade: Te mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11(3), 291 – 307 . https://doi.org/10.1177/1558689817714066
Fierro , L. A. , Galport , N. , Hunt , A. , Codd , H. , & Donaldson , S. I. (2016). Canadian Evaluation Society credentialed evaluator designation program: Evaluation report. https://evaluationcanada.ca/txt/2016_pdp_evalrep_en.pdf
Forsyth , P. B. , & Danisiewicz , T. J. (1985). Toward a theory of professionalization . Work and Occupations, 12( 1 ), 59 – 76 . https://doi.org/10.1177/0730888485012001004
Foundations for Evidence-Based Policymaking Act. (2018). H.R. 4174. 115th Congress (2017–2018). https://www.congress.gov/bill/115th-congress/house-bill/4174
Freidson , E. (1994). Professionalism reborn: Teory, prophecy, and policy. University of Chicago Press .
Freidson , E. (2001). Professionalism, the third logic: On the practice of knowledge. University of Chicago Press .
Galport , N. , & Azzam , T. (2017). Evaluator training needs and competencies: A gap analysis . American Journal of Evaluation, 38(1), 80 – 100 . https://doi.org/10.1177/1098214016643183
Gauthier , B. , Kishchuk , N. , Borys , S. , & Roy , S. N. (2015). Te CES professional designations program: Views from members . Canadian Journal of Program Evaluation, 29(3), 98 – 133 .
Gauthier , B. , Lahey , R. E. , & Jacob , S. (in press). Te institutionalization of evaluation in Canada . In Evaluation GLOBE, Americas. Palgrave Macmillan .
House , E. R. (1990). Trends in evaluation . Educational Researcher, 19(3), 24 – 28 . https://doi.org/10.2307/1176068
International Organization for Cooperation in Evaluation (IOCE) . (n.d.). https://www.ioce.net/
Irby , D. M. , & Hamstra , S. J. (2016 ). Parting the clouds: Tree professionalism frameworks in medical education . Academic Medicine, 91( 12 ), 1606 – 1611. https://doi.org/10.1097/acm.0000000000001190. Medline:27119331
Johnson , S. M. ( 2018 ). Te development and use of interpersonal competencies by evaluators (Unpublished doctoral dissertation). University of Minnesota .
Kay , D. , Berry , A. , & Coles , N. A. ( 2019 ). What experiences in medical school trigger professional identity development? Teaching and Learning in Medicine, 31( 1 ), 17 – 25 . https://doi.org/10.1080/10401334.2018.1444487 . Medline:29608109
Kelley , K. A. , Stanke , L. D. , Rabi , S. M. , Kuba , S. E. , & Janke , K. K. (2011). Cross-validation of an instrument for measuring professionalism behaviors . American Journal of Pharmaceutical Education, 75(9), 1 – 10 . https://doi.org/10.5688/ajpe759179 . Medline:22171107
King, J. A. (2003). Evaluating educational programs and projects in the USA . In T. Kellaghan & D. L. Stuf ebeam (Eds.), International handbook of educational evaluation (pp. 721-732 ). Kluwer Academic Publishers .
King , J. A. , & Stevahn , L. (2015). Competencies for program evaluators in light of adaptive action: What? So what? Now what? New Directions for Evaluation, 2015(145), 21 – 37 . https://doi.org/10.1002/ev.20109
Kishchuk , N. , Gauthier , B. , Barrington , G. V. , Cummings , H. , & Farina , S. (2019, May). Evolution of the Canadian Evaluation Society’s professional designation, 2010–2019: Trends and emerging issues/opportunities [PowerPoint presentation]. CES 2019 Conference, Halifax, NS.
Kuji-Shikatani , K. , Tompson , M. , & Matthew , M. (2015). Launching the credentialed evaluator (CE) designation. Canadian Journal of Program Evaluation, 29(3), 70 – 85 .
LaVelle , J. M. , & Donaldson , S. I. (2015). Te state of preparing evaluators. New Directions for Evaluation, 2015(145), 39 – 52 . https://doi.org/10.1002/ev.20110
Lawson , K. , Hunter , T. , & McDavid , J. (2020). Predicting credentialed evaluator status: Characteristics, comparisons, and implications for the CE Program . Canadian Journal of Program Evaluation, 35(1). https://doi.org/10.3138/cjpe.61841
Lombarts , K. M. J. M. H. , Plochg , T. , Tompson , C. A. , & Arah , O. A. , on behalf of the DUQuE Project Consortium. (2014). Measuring professionalism in medicine and nursing: Results of a European survey. PLoS One, 9(5), e97069. https://doi.org/10.1371/journal.pone.0097069. Medline:24849320
Madaus , G. F. , Scriven , M. , & Stufebeam , D. L. (Eds.). (2012). Evaluation models: Viewpoints on educational and human services evaluation (Vol. 6 ). Kluwet-Nijhof Publishing.
Modarresi , S. (1998). Te profession of program evaluation: theory and practice (Unpublished doctoral dissertation). University at Albany, State University of New York .
Patton , M. Q. ( 2015 ). Qualitative research and methods: Integrating theory and practice. Sage.
Pavalko , R. M. ( 1988 ). Sociology of occupations and professions (2nd ed.). F. E. Peacock Publishers .
Picciotto , R. ( 2011 ). Te logic of evaluation professionalism . Evaluation, 17(2), 165 – 180. https://doi.org/10.1177/1356389011403362
Russ-Ef , D. , & Preskill , H. (2009). Evaluation in organizations a systematic approach: A systematic approach to enhancing learning, performance, and change. Basic Books .
Schwandt , T. A. (2017). Professionalization, ethics, and fdelity to an evaluation ethos . American Journal of Evaluation, 38(4), 546 – 553 . https://doi.org/10.1177/1098214017728578
Sou , G. (2008). Historical review of program evaluation for educational quality assurance. School of Graduate Studies Asia International Open University Macau Special Administrative Region.
Stevahn , L. , King , J. A. , Ghere , G. , & Minnema , J. (2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26(1), 43 – 59 . https://doi.org/10.1177/1098214004273180
VERBI Sofware. (2019). MAXQDA 2020 [computer sofware]. https://maxqda.com.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Sandra Ayoo
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
