Evaluation Literacy
Perspectives of Internal Evaluators in Non-Government Organizations
DOI:
https://doi.org/10.3138/cjpe.42190Keywords:
evaluation, evaluation literacy, internal evaluation, non-government organization, not-for-profitAbstract
While there is an abundance of literature on evaluation use, there has been little discussion regarding internal evaluators’ role in promoting evaluation use. Evaluation can be undervalued if context is not taken into consideration. Evaluation literacy is needed to make evaluation more appropriate, understandable, and accessible, particularly in non-government organizations (NGOs) where there is a growing focus on demonstrable outcomes. Evaluation literacy refers to an individual’s understanding and knowledge of evaluation and is an essential component of embedding evaluation into organizational culture. In recognition of the value of the internal perspective, a small exploratory exercise was undertaken to reveal internal evaluator roles and ways of engaging with colleagues around evaluation. Th e exercise examined a key question: What is the role of evaluation literacy in internal evaluation in the non-government sector? Three Australian auto-narrative examples from internal evaluators highlight evaluation literacy and locate it among the multiplicity of roles required for optimal evaluation uptake. Analysis of the narratives revealed the underlying issues affecting evaluation use in NGOs and the skills needed to motivate and enable others to access, understand, and use evaluation information. Responding to the call for expanded research into internal evaluation from a practice perspective, the authors hope that the findings will stimulate a wider conversation and further advance understanding of evaluation literacy.
References
Alaimo, S. P. (2008). Nonprofits and evaluation: Managing expectations from the leader’s perspective. New Directions for Evaluation, 119(Fall), 73–92. https://doi.org/10.1002/ev.269
Alkin, M. C., & Taut, S. M. (2003). Unbundling evaluation use. Studies in Educational Evaluation, 29 (1), 1–12. https://doi.org/10.1016/S0191-491X(03)90001-0
Amo, C., & Cousins, J. B. (2007). Going through the process: An examination of the operationalization of process use in empirical research on evaluation. New Directions for Evaluation, 2007 (116), 5–26. https://doi.org/10.1002/ev.240
Baizerman, M., Compton, D. W., & Stockdill, S. H. (2002). New directions for ECB. New Directions for Evaluation, 2002 (93), 109–120. https://doi.org/10.1002/ev.45
Baron, M. E. (2011). Designing internal evaluation for a small organization with limited resources. New Directions for Evaluation, 2011(132), 87–99. https://doi.org/10.1002/ev.398
Bourgeois, I. (2008). Understanding the dimensions of organizational evaluation capacity (Doctoral dissertation). Faculty of Graduate and Postdoctoral Studies, University of Ottawa, ON, Canada. Retrieved from https://ruor.uottawa.ca/handle/10393/29565
Bourgeois, I. (2016). Performance measurement as precursor to organizational evaluation capacity building. Evaluation Journal of Australasia, 16(1), 11. https://doi.org/10.1177/1035719X1601600103
Bourgeois, I., & Cousins, J. B. (2008). Informing evaluation capacity building through profiling organizational capacity for evaluation: An empirical examination of four Canadian federal government organizations. Canadian Journal of Program Evaluation, 23 (3), 127.
Bourgeois, I., & Cousins, J. B. (2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 3(34), 299–319. https://doi.org/10.1177/1098214013477235
Bourgeois, I., Toews, E., Whynot, J., & Lamarche, M. K. (2013). Measuring organizational evaluation capacity in the Canadian federal government. Canadian Journal of Program Evaluation, 28(2).
Bourgeois, I., Whynot, J., & Thériault, E. (2015). Application of an organizational evaluation capacity self-assessment instrument to different organizations: Similarities and lessons learned. Evaluation and Program Planning, 50, 47–55. https://doi.org/10.1016/j.evalprogplan.2015.01.004. Medline:25757074
Brady, B., Canavan, J., & Redmond, S. (2016). Bridging the gap: Using Veerman and Van Yperen’s (2007) framework to conceptualise and develop evidence informed practice in an Irish youth work organisation. Evaluation and Program Planning, 55 , 128–133. https://doi.org/10.1016/j.evalprogplan.2016.01.003. Medline:26844427
Braverman, M. T. (2013). Negotiating measurement: Methodological and interpersonal considerations in the choice and interpretation of instruments. American Journal of Evaluation, 34 (1), 99–114. https://doi.org/10.1177/1098214012460565
Campbell, D. A., Lambright, K. T., & Bronstein, L. R. (2012). In the eyes of the beholders. Public Performance & Management Review, 36 (1), 7–30. Carman, J. G. (2007). Evaluation practice among community-based organizations: Research into the reality. American Journal of Evaluation, 28(1), 60–75. https://doi.org/10.1177/1098214006296245
Carman, J. G., & Fredericks, K. A. (2010). Evaluation capacity and nonprofit organizations: Is the glass half-empty or half-full? American Journal of Evaluation, 31 (1), 84–104. https://doi.org/10.1177/1098214009352361
Clandinin, D. J., & Connelly, M. (2004). Knowledge, narrative and self-study. In J. J. Loughran, M. L. Hamilton, V. K. LaBoskey, & T. Russell (Eds.), International handbook of self-study of teaching and teacher education practices. Springer International Handbooks of Education, Vol. 12. Dordrecht, Netherlands: Springer.
Coffey, A., & Atkinson, P. (1996). Narratives and stories. In Making sense of qualitative data: Complementary research strategies (pp. 54–82). Thousand Oaks, CA: SAGE Publications.
Cousins, J. B., Goh, S. C., Elliott, C. J., & Bourgeois, I. (2014). Framing the capacity to do and use evaluation. New Directions for Evaluation, 2014(141), 7–23. https://doi.org/10.1002/ev.20076
Doherty, B., Eccleston, R., Hansen, E., Natalier, K., & Churchill, B. (2015). Building evaluation capacity in micro community organisations—more burden than benefit? Evaluation Journal of Australasia, 15(4), 29–37. https://doi.org/10.1177/1035719X1501500404
Fleischer, D. N., & Christie, C. A. (2009). Evaluation use: Results from a survey of U.S. American Evaluation Association members. American Journal of Evaluation, 30(2), 158–175. https://doi.org/10.1177/1098214008331009
Foucault, M. (2013). Archaeology of knowledge (2nd ed.). Hoboken, NJ: Taylor & Francis. Guijt, I., Shutt, C., Eyben, R., & Roche, C. (2015). The politics of evidence and results in international development: Playing the game to change the rules? Rugby, England: Practical Action Publishing.
Kelly, L. M. (2016a). Reflections of an evaluator navigating between community development and welfare paradigms. Development in Practice, 26(4), 517–521. https://doi.org/10.1080/09614524.2016.1164123
Kelly, L. M. (2016b). Perceptions of poverty and the perpetuation of the capability-judgement cycle in outer southeast Melbourne. Community Development, 47(5), 584–601. https://doi.org/10.1080/15575330.2016.1226913
King, J. A. (2016). Taking what action for change? Dare evaluation build a new social order? Visitor Studies, 19 (1), 3–11. https://doi.org/10.1080/10645578.2016.1144022
King, J. A. (2017). Putting evaluation capacity building in context: Reflections on the On-tario Brain Institute’s Evaluation Support Program. Evaluation and Program Planning. https://doi.org/10.1016/j.evalprogplan.2017.05.013. Medline:28641901
King, J. A., & Volkov, B. B. (2005). A framework for building evaluation capacity based on the experiences of three organizations. CURA Reporter, 3 (35), 10–16.
Kirkhart, K. E. (2000). Reconceptualizing evaluation use: An integrated theory of infl u¬ence. New Directions for Evaluation, 2000 (88), 5–23. https://doi.org/10.1002/ev.1188
Labin, S. N., Duffy, J. L., Meyers, D. C., Wandersman, A., & Lesesne, C. A. (2012). A re-search synthesis of the evaluation capacity building literature. American Journal of Evaluation, 33(3). https://doi.org/10.1177/1098214011434608
Lee, C., & Nowell, B. (2015). A framework for assessing the performance of non-profit organizations. American Journal of Evaluation, 36(3), 299–319. https://doi.org/10.1177/1098214014545828
Leviton, L. C. (2014). Some underexamined aspects of evaluation capacity building. American Journal of Evaluation, 35 (1), 90–94. https://doi.org/10.1177/1098214013502844
Lincoln, Y., & Guba, E. (1989). Fourth generation evaluation. Newbury Park, CA: SAGE. Love, A. J. (1991). Internal evaluation: Building organizations from within. Newbury Park, CA: SAGE.
Lyons, M. (2001). Third sector: The contribution of non-profit and cooperative enterprises in Australia. Sydney, Australia: Allen & Unwin.
Mathison, S. (2011). Internal evaluation, historically speaking. New Directions for Evaluation, 2011 (132), 13–23. https://doi.org/10.1002/ev.393
May, S. (2012). Case studies in organizational communication: Ethical perspectives and practices. Thousand Oaks, CA: SAGE.
Mayne, J. (2014). Issues in enhancing evaluation use. In M. Läubli Loud & J. Mayne (Eds.), Enhancing evaluation use: Insights from internal evaluation units (pp. 1–14). Thousand Oaks, CA: SAGE.
McCoy, A., Rose, D., & Connolly, M. (2013). Developing evaluation cultures in human service organisations. Evaluation Journal of Australasia, 13(1), 15. https://doi.org/10.1177/1035719X1301300103
McCoy, A., Rose, D., & Connolly, M. (2014). Approaches to evaluation in Australian child and family welfare organizations. Evaluation and Program Planning, 44 , 68–74. https://doi.org/10.1016/j.evalprogplan.2014.02.004. Medline:24607886
Milstein, B., Chapel, T. J., Wetterhall, S. F., & Cotton, D. A. (2002). Building capacity for program evaluation at the Centers for Disease Control and Prevention. New Directions for Evaluation, 2002 (93), 27–46. https://doi.org/10.1002/ev.40
Milstein, B., & Wetterhall, S. (2000). A framework featuring steps and standards for program evaluation. Health Promotion Practice, 1(3), 221–228. https://doi.org/10.1177/152483990000100304
Moxham, C. (2014). Understanding third sector performance measurement system design: A literature review. International Journal of Productivity and Performance Management, 63 (6), 704–726. https://doi.org/10.1108/IJPPM-08-2013-0143
Naccarella, L., Pirkis, J., Kohn, F., Morley, B., Burgess, P., & Blashki, G. (2007). Building evaluation capacity: Definitional and practical implications from an Australian case study. Evaluation and Program Planning, 30(3), 231–236. https://doi.org/10.1016/j.evalprogplan.2007.05.001. Medline:17689328
Nevo, D. (2001). School evaluation: Internal or external? Studies in Educational Evaluation, 27 (2), 95–106. https://doi.org/10.1016/S0191-491X(01)00016-5
Nevo, D. (2002). Dialogue evaluation: Combining internal and external evaluation. In School-based evaluation: An international perspective (Vol. 8, pp. 3–16). Bingley, England: Emerald Group.
Olejniczak, K., Raimondo, E., & Kupiec, T. (2016). Evaluation units as knowledge brokers: Testing and calibrating an innovative framework. Evaluation, 22(2), 168. https://doi.org/10.1177/1356389016638752
Owen, J. M., & Lambert, F. C. (1995). Roles for evaluation in learning organizations. Evalu-ation, 1 (2), 237–250. https://doi.org/10.1177/135638909500100207
Patton, M. (2008). Utilization-focused evaluation . Thousand Oaks, CA: SAGE. Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459. https://doi.org/10.1177/1098214008324182
Preskill, H., & Torres, R. T. (2000). The learning dimension of evaluation use. New Directions for Evaluation, 2000 (88), 25–37. https://doi.org/10.1002/ev.1189
Rogers, A., Bower, M., Malla, C., Manhire, S., & Rhodes, D. (2017). Developing a cultural protocol for evaluation. Evaluation Journal of Australasia, 17(2), 11–19. https://doi.org/10.1177/1035719X1701700203
Sobeck, J., & Agius, E. (2007). Organizational capacity building: Addressing a research and practice gap. Evaluation and Program Planning, 30(3), 237–246. https://doi.org/10.1016/j.evalprogplan.2007.04.003. Medline:17689329
Sonnichsen, R. C. (2000). High impact internal evaluation: A practitioner’s guide to evaluating and consulting inside organizations . Thousand Oaks, CA: SAGE.
Stockdill, S. H., Baizerman, M., & Compton, D. W. (2002). Toward a definition of the ECB process: A conversation with the ECB literature. New Directions for Evaluation, 2002 (93), 7–26. https://doi.org/10.1002/ev.39
Vo, A. (2015). Foreword. In C. Christie & A. Vo (Eds.), Evaluation use and decision making in society: A tribute to Marvin C. Alkin (pp. vii–xviii). Charlotte, NC: Information Age Publishing.
Volkov, B. B. (2011a). Internal evaluation a quarter-century later: A conversation with Arnold J. Love. New Directions for Evaluation, 2011(132), 5–12. https://doi.org/10.1002/ev.392
Volkov, B. B. (2011b). Beyond being an evaluator: The multiplicity of roles of the internal evaluator. New Directions for Evaluation, 2011(132), 25–42. https://doi.org/10.1002/ev.394
Volkov, B. B., & Baron, M. E. (2011). Issues in internal evaluation: Implications for practice, training, and research. New Directions for Evaluation, 2011(132), 101–111. https://doi.org/10.1002/ev.399
Weiss, C. (2000). What we have learned from 25 years of knowledge utilization. In R. Henke (Ed.), Final report for the international conference on social science and governance: The linkage between social science research and social policy (pp. 20–21). Zeist, Netherlands: Netherlands Commission for UNESCO, Management of Social Transformations.
World Health Organisation (WHO). (1998). Health promotion glossary. Geneva, Switzerland. Retrieved from www.who.int/healthpromotion/about/HPR%20Glossary%201998.pdf
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2019 Alison Rogers, Alicia McCoy, Leanne M. Kelly
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
