Breaking the Mould? Expanding our Conceptualization of Evaluator Competencies
DOI:
https://doi.org/10.3138/cjpe-2023-0021Keywords:
evaluation capacity building, evaluator competencies, research on evaluation, professionalizationAbstract
Evaluation capacity building (ECB) is an area of practice that many evaluators engage in. Previous ECB scholars have described ECB as distinct from the practice of evaluation, although the two are related. This research empirically examined competencies needed for practicing ECB, why those competencies are important, and how expert practitioners obtained those competencies. The findings suggest the role of an evaluation capacity builder is different from the role of an evaluator. The authors attempt to make the case that recognizing this difference is critical for the professionalization of the field.
References
Ajose, L., & Kimball, K. (2010). Eval10 session 217: Assessing evaluation capacity - using the evaluation capacity diagnostic tool. [Professional development workshop]. American Evaluation Association, San Antonio, TX, United States. https://comm.eval.org/viewdocument/eval10-session-217
American Evaluation Association. (2018). The 2018 AEA evaluator competencies. https://www.eval.org/About/Competencies-Standards/AEA-Evaluator-Competencies
American Evaluation Association. (2021, February 12). 2020 TIG membership analysis. https://www.eval.org/DesktopModules/DnnSharp/SearchBoost/UrlFwd.ashx?url=https%3a%2f%2fwww.eval.org%2fDesktopModules%2fDnnSharp%2fSearchBoost%2fFileDownload.ashx%3ffile%3d3868%26sb-bhvr%3d1&ukey=yfmLJsa60CqqJbD4EbfmJ5Ddd5DCuwMHiUpKXRNsQog7rrNe1gMitJN%2f2chA6E%2flfAKXljoa4FnhSnAclQFjyW49JsQ%2by3CLaAySTmqEAIo%3d&logid=12060-9dp0lxgubwgu1r59
American Evaluation Association. (n.d.). Topical interest group membership count. Retrieved March 25, 2022, from https://www.eval.org/Community/Topical-Interest-Groups/TIG-Member-Count
Aotearoa New Zealand Evaluation Association. (2011). Evaluator competencies. https://www.anzea.org.nz/app/uploads/2019/04/110801_anzea_evaluator_competencies_final.pdf
Ayoo, S., Wilcox, Y., LaVelle, J. M., Podems, D., & Barrington, G. V. (2020). Grounding the 2018 AEA evaluator competencies in the broader context of professionalization. New Directions for Evaluation, 2020(168), 13–30.
Beyer, S. (1990). Gender differences in the accuracy of self-evaluations of performance. Journal of Personality & Social Psychology, 59(5), 960–970.
Bourgeois, I., Lemire, S., Fierro, L. A., Castleman, A. M., & Cho, M. (2023). Laying a solid foundation for the next generation of evaluation capacity building: Findings from an integrative review. American Journal of Evaluation, 44(1), 29–49.
Buetti, D., Bourgeois, I., & Jafary, M. (2023). Examining the competencies required by evaluation capacity builders in community-based organizations. Evaluation and Program Planning, 97, 102242.
Canadian Evaluation Society. (2018). Competencies for Canadian evaluation practice. https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice.pdf
Carifio, J., & Perla, R. (2008). Resolving the 50-year debate around using and misusing Likert scales. Medical Education, 42(12), 1150–1152. https://evaluationcanada.ca/files/pdf/2_competencies_cdn_evaluation_practice_2018.pdf
Chaplowe, S. G., & Cousins, J. B. (2016). Monitoring and evaluation training: A systematic approach. Sage Publications.
Cho, M. (2021). Applying the AEA evaluator competencies to improve evaluation training programs [Unpublished master’s thesis]. Claremont Graduate University.
Christie, C. A., Quiñones, P., & Fierro, L. A. (2014). Informing the discussion on evaluator training: A look at evaluators’ course taking. American Journal of Evaluation, 35(2), 274–290.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
Coryn, C. L. S., Becho, L. W., Westine, C. D., Mateu, P. F., Abu-Obaid, R. N., Hobson, K. A., … Ramlow, M. (2019). Material incentives and other potential factors associated with response rates to internet surveys of American Evaluation Association members: Findings from a randomized experiment. American Journal of Evaluation, 41, 277–296.
Cousins, J. B., Elliot, C., Amo, C., Bourgeois, I., Chouinard, J. A., Goh, S. C., & Lahey, R. (2008). Organizational capacity to do and use evaluation: Results of a pan-Canadian survey of evaluators. Canadian Journal of Program Evaluation, 23(3), 1–35.
Cousins, J. B., Goh, S. C., Clark, S., & Lee, L. E. (2004). Integrating evaluative inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19(2), 99–141.
Cousins, J. B., Goh, S. C., Elliot, C. J., & Bourgeois, I. (2014). Framing the capacity to do and use evaluation. New Directions for Evaluation, 2014(141), 7–23.
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). Sage Publications.
Dillman, D. A. (2013). Mail and internet surveys: The tailored design method – 2007 update with new internet, visual, and mixed-mode guide. Wiley.
Dunning, D., Heath, C., & Suls, J. M. (2004). Flawed self-assessment. Psychological Science in the Public Interest, 5(3), 69–106.
European Evaluation Society. (n.d.). The EES evaluation capabilities framework. Retrieved March 30, 2022, from https://europeanevaluation.org/wp-content/uploads/2020/03/EES-EVALUATION-CAPABILITIES-FRAMEWORK.pdf
Fierro, L. A. (2019, Spring). Evaluation capacity building. [Course syllabus]. School of Social Science, Policy, and Evaluation, Claremont Graduate University.
Fierro, L. A. (2020, January 13–14). Evaluation capacity building in organizations [Professional development workshop]. The Evaluator’s Institute, Claremont, CA, United States. https://tei.cgu.edu
Fierro, L. A. & Christie, C. A. (2017). Evaluator and program manager perceptions of evaluation capacity and evaluation practice. American Journal of Evaluation, 38(3), 376–392.
Fierro, L. A., Codd, H. D., & Castleman, A. M. (2023). Building evaluation capacity in organizations [Professional development workshop]. American Evaluation Association, Indianapolis, IN, United States.
Fierro, L. A., Galport, N., Hunt, A., Codd, H., & Donaldson, S. I. (2016). Canadian evaluation society credentialed evaluator designation program evaluation report.
Fleischer, D. N., Christie, C. A., & LaVelle, K. B. (2008). Perceptions of evaluation capacity building in the United States: A descriptive study of American evaluation association members. Canadian Journal of Program Evaluation, 23, 37–60.
George, D., & Mallery, P. (2003). SPSS for windows step by step: A simple guide and reference, 11.0 update (4th ed.). Allyn and Bacon.
Hsieh, H.-F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.
International Development Evaluation Association. (2012). Competencies for development evaluation evaluators, managers, and commissioners. https://ideas-global.org/the-competencies-framework/
King, J. A. (Ed.). (2020). The American Evaluation Association’s program evaluator competencies. New Directions for Evaluation, 2020(168), 1–172.
Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed). Guilford Press.
Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A research synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3), 307–338.
LaVelle, J. M., & Galport, N. (2020). Using the 2018 AEA evaluator competencies for evaluator education and professional development. New Directions for Evaluation, 2020(168), 99–116.
Maicher, B., & Frank, C. (2015). The development and initial validation of competencies and descriptors for Canadian evaluation practice. Canadian Journal of Program Evaluation, 29(3), 54–69.
Norman, G. (2010). Likert scales, levels of measurement and the “laws” of statistics. Advances in Health Sciences Education: Theory and Practice, 15(5), 625–32.
Norton, S., Milat, A., Edwards, B., & Giffin, M. (2016). Narrative review of strategies by organizations for building evaluation capacity. Evaluation and Program Planning, 58, 1–19.
Patton, M. Q. (2014). Qualitative research and evaluation methods: Integrating theory and practice (4th ed.). Sage Publications.
Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459.
Scherpereel, C. M., & Bowers, M. Y. (2008). Back to the future: gender differences in self-ratings of team performance criteria. Developments in Business Simulation and Experiential Learning: Proceedings of the Annual ABSEL Conference, 35, 170–179. https://journals.tdl.org/absel/index.php/absel/article/view/401
South African Monitoring and Evaluation Association. (2020). South African Monitoring and Evaluation Association’s evaluator competencies framework. https://oldwww.samea.org.za/summernotefile/dump?summernotefile_id=193
Stockdill, S. H., Baizerman, M., & Compton, D. W. (2002). Toward a definition of the ECB process: A conversation with the ECB literature. New Directions for Evaluation, 2002(93), 7–26.
Sudman, S., Bradburn, N. M., & Schwarz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. Jossey-Bass.
Taylor-Powell, E. (2010, November 8–9). Conference PDW #8: Building evaluation capacity of community organizations. [Professional development workshop]. American Evaluation Association, San Antonio, TX, United States. https://comm.eval.org/viewdocument/conference-pdw-8-b
Taylor-Powell, E. (2011, June 13). SI11 session 6: Building evaluation capacity: Tips and strategies. [Professional development workshop]. AEA/CDC Summer Evaluation Institute, Atlanta, GA, United States. https://comm.eval.org/viewdocument/si11-session-6-buil
Treasury Board of Canada Secretariat. (2020, July 23). Evaluation competencies. https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/evaluation-government-canada/evaluation-competencies.html
Tucker, S. A., Barela, E., Miller, R. L., & Podems, D. (2020). The story of the AEA Competencies Task Force (2015–2018). New Direction for Evaluation, 2020(168), 31–48.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Ann Marie Castleman, Leslie A. Fierro
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
