Using Evaluative Information Sensibly
The Enduring Contributions of John Mayne
DOI:
https://doi.org/10.3138/cjpe.75444Keywords:
evaluation utilization, results-based management, organizational learning, accountability, contribution analysisAbstract
In this concluding article, we take stock of the diverse and stimulating contributions comprising this special issue. Using concept mapping, we identify eight evaluation themes and concepts central to John Mayne’s collective work: evaluation utilization, results-based management, organizational learning, accountability, evaluation culture, contribution analysis, theory-based evaluation, and causation. The overarching contribution story is that John’s work served to bridge the gaps between evaluation practice and theory; to promote cross-disciplinary synergies across program evaluation, performance auditing, and monitoring; and to translate central themes in evaluation into a cogent system for using evaluative information more sensibly. In so doing, John left a significant institutional and academic legacy in evaluation and in results-based management.
References
Abbott, A. (1988). The system of professions. An essay on the division of expert labor. Chicago: University of Chicago Press.
Archibald, T. (2013, November 11). Evaluative thinking. Free Range Evaluation. https://tgarchibald.wordpress.com/2013/11/11/18/
Baker, A., & Bruner, B. (2012). Integrating evaluative capacity into organizational practice. The Bruner Foundation. Retrieved from http://www.evaluativethinking.org/docs/Integ_Eval_Capacity_Final.pdf
Barrados, M. (2023). John Mayne’s contribution to performance audit. Canadian Journal of Program Evaluation, 37(3), 461–472.
Barrados, M., & Mayne, J. (2003). Can public sector organisations learn? OECD Journal of Budgeting, 3(3), 87–103.
Barrados, M., Mayne, J., & Wileman, T. (2000). Accountability for collaborative programme delivery arrangements in Canada’s federal government: Some consequences of sharing the business of government. International Review of Administrative Sciences, 66(3), 495–511.
Befani, B., & Mayne, J. (2014). Process tracing and contribution analysis: A combined approach to generative causal inference for impact evaluation. IDS Bulletin, 45(6), 17–36.
Bourgeois, I. & Whynot, J. (2023). Bridging evaluation theory and practice: The contributions of John Mayne to Canadian federal evaluation. Canadian Journal of Program Evaluation, 37(3), 324–339.
Chaytor, K. (2023). Building evaluation culture—the missing link. Canadian Journal of Program Evaluation, 37(3), 422–438.
de Ries, K. E., Schaap, H., van Loon, A. M. M. J. A. P., Kral, M. M. H., & Meijer, P. C. (2022). A literature review of open-ended concept maps as a research instrument to study knowledge and learning. Quality & Quantity, 56, 73–107.
Douthwaite, B., Mayne, J., McDougall, C., & Paz-Ybarnegaray R. (2017). Evaluating complex interventions: A theory-driven realist-informed approach. Evaluation, 23(3), 294–311.
Dybdal, L., Nielsen, S. B., & Lemire, S. (2010). Contribution analysis applied: Reflections on scope and methodology. Canadian Journal of Program Evaluation, 25(2), 29–57.
Gauthier, B., Barrington, G., Bozzo, S. L., Chaytor, K., Dignard, A., Lahey, R., Malatest, R., McDavid, J. C., Mason, G., Mayne, J., Porteous, N. L., & Roy, S. (2010). The lay of the land: Evaluation practice in Canada in 2009. Canadian Journal of Program Evaluation, 24(1), 1–49.
Gray, A., Jenkins, B., Leeuw, F. L., & Mayne, J. (eds.) (2003). Collaboration in Public Services: The Challenge for Evaluation. London, UK: Routledge.
Lahey, R. (2010). The Canadian M&E system: Lessons learned from 30 years of development. Evaluation Capacity Development Working Paper Series, No. 23. Washington, DC: World Bank.
Lahey, R. (2023). John Mayne and the origins of evaluation in the public sector in Canada: A shaping of both evaluation and the evaluator. Canadian Journal of Program Evaluation, 37(3), 340–354.
Lahey, R., & Nielsen, S. B. (2013). Rethinking the relationship among monitoring, evaluation, and results-based management: Observations from Canada. New Directions for Evaluation, 137, 45–56.
Läubli-Loud, M., & Mayne, J. (eds.) (2013). Enhancing evaluation use. Insights from internal evaluation units. London, UK: Sage Publishers.
Leeuw, F. L. (2023). John Mayne and rules of thumb for contribution analysis: A comparison with two related approaches. Canadian Journal of Program Evaluation, 37(3), 403–421.
Mayne, J. (1994). Utilizing evaluation in organizations: The balancing act. In F. L. Leeuw, R. C. Rist, & R. C. Sonnichsen, (Eds.), Can governments learn? (pp. 17–43). London, UK: Transaction.
Mayne, J. (1997). Accountability for program performance: A key to effective performance monitoring and reporting. In Mayne, J., & Zapico-Gõni, E. (Eds.), Monitoring performance in the public sector (pp. 157–176). London, UK: Routledge.
Mayne, J. (1999). Addressing attribution through contribution analysis: Using performance measures sensibly (discussion paper). Ottawa, ON: Office of the Auditor General of Canada. Retrieved from: http://dsp-psd.pwgsc.gc.ca/Collection/FA3-31-1999E.pdf
Mayne, J. (2001). Addressing attribution through contribution analysis: Using performance measures sensibly. Canadian Journal of Program Evaluation, 16(1), 1–24.
Mayne, J. (2004). Reporting on outcomes: Setting performance expectations and telling performance stories. Canadian Journal of Program Evaluation, 19(1), 31–60.
Mayne, J. (2006). Performance studies: The missing link? Canadian Journal of Program Evaluation, 21(2), 201–208.
Mayne, J. (2007). Challenges and lessons in implementing results-based management. Evaluation, 13(1), 87–109.
Mayne, J. (2010a). Building an evaluative culture: The key to effective evaluation and results management. Canadian Journal of Program Evaluation, 24(2), 1–30.
Mayne, J. (2010b). Performance auditing: Cozy, comfortable and in need of challenge. Optimum, 40(3), 1–6.
Mayne, J. (2010c). Results management: Can results evidence gain a foothold in the public sector? In O. Rieper, F. L. Leeuw, & T. Ling (Eds.), The evidence book (pp. 117–149). New Brunswick, NJ: Transaction.
Mayne, J. (2011). Addressing cause and effect in simple and complex settings through contribution analysis. In R. Schwartz, K. Forss, & M. Marra (Eds.), Evaluating the complex. Attribution, contribution and beyond. New York, NY: Transaction.
Mayne, J. (2012). Independence in evaluation and the role of culture. In J.-C. Barbier & P. Hawkins (Eds.), Evaluation and culture. Sense-making in complex times. London, UK: Routledge.
Mayne, J. (2013). Issues in enhancing evaluation use. In M. Läubli-Loud & J. Mayne (Eds.), Enhancing evaluation use: Insights from internal evaluation units (p. 1–14). London, UK: Sage.
Mayne, J. (2015). Useful theory of change models. Canadian Journal of Program Evaluation. 30(2), 119–142.
Mayne, J. (2019). Revisiting contribution analysis. Canadian Journal of Program Evaluation. 34(2), 171–191.
Mayne, J. (2020). Building evaluative culture in community services: Caring for evidence. Evaluation and Program Planning, 80, 101450. Medline:29126610
Mayne, J., & Mayne, R. S. (1983). Will program evaluation be used in formulating policy, in M. M. Atkinson & M. A. Chandler (Eds.), The politics of Canadian public policy (pp. 267–283). Toronto, Canada: University of Toronto Press.
Mayne, J., & Rist, R. C. (2006). Studies are not enough: The necessary transformation of evaluation. Canadian Journal of Program Evaluation, 21(3), 93–120.
Mayne, J., & Wilkins, P. (2005). “Believe it or not?”: The emergence of performance information auditing: Seeking confidence in evaluating, auditing, and performance reporting. In R. Schwartz & J. Mayne (Eds.), Quality matters. Seeking confidence in evaluation, auditing and performance reporting (pp. 237–259). London, UK: Routledge.
McDavid, J. (2023). Enduring themes in John Mayne’s work: Implications for evaluation practice. Canadian Journal of Program Evaluation, 37(3), 355–370.
Miles, M., & Huberman, M. (1994). Qualitative data analysis: An expanded sourcebook. Thousand Oaks, CA: Sage.
Nielsen S. B., Lemire, S., & Montague, S. (2023). Remembering John Mayne—A practical thinker and a thinking practitioner. Canadian Journal of Program Evaluation, 37(3), 297–305.
Nielsen S. B., & Montague, S. (2023). John Mayne’s contribution to result-based management thinking: A leadership legacy. Canadian Journal of Program Evaluation, 37(3), 439–460.
Nielsen, S. B., Tangsig, S., & Lemire, S. (2023). Mapping the contributions of John Mayne: Bridging the gaps between evaluation, auditing, and performance monitoring. Canadian Journal of Program Evaluation, 37(3), 306–323.
Office of the Auditor General of Canada (OAG) (2002). Modernizing accountability in the public sector. In Report of the Auditor General of Canada to the House of Commons. Retrieved from https://publications.gc.ca/site/eng/search/advancedSearch.html?
Office of the Comptroller General, Government of Canada. (1981). Guide on the program evaluation function. Ottawa: Treasury Board of Canada, Office of the Comptroller General of Canada.
Palenberg, M. (2023). Causal claims in Contribution Analysis. Canadian Journal of Program Evaluation, 37(3), 389–402.
Schmitt, J. (2020). The causal mechanism claim in evaluation. Does the prophecy fulfill? New Directions for Evaluation, 167, 11–26.
Schwartz, R., & Mayne, J. (2005). Assuring the quality of evaluative information: Theory and practice. Evaluation and Program Planning, 28(1), 1–14.
Sridharan, S., Nakaima, A., & Pereira, A. (2023). Causality and complexity in evaluating equity interventions: Celebrating contribution analysis. Canadian Journal of Program Evaluation, 37(3), 371–388.
Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (2012). Broadening the range of designs and methods for impact evaluations. Report of a study commissioned by the Department for International Development. Working Paper 38. DFID.
Sørensen, O. H., Bjørner, J., Dyreborg, J., Kristiansen, J., Sørli, J. B., & Nielsen, S. B. (2022). Measuring societal impact of research. Developing and validating an impact instrument for occupational health and safety. Research Evaluation, 31(1), 118–131.
Treasury Board Secretariat & Office of the Auditor General, Government of Canada. (2003). The Managing for Results Self-Assessment Tool. Ottawa: Treasury Board of Canada Secretariat, 2003. Retrieved from: https://publications.gc.ca/site/eng/244385/publication.html
Ward, G., & Haigh, M. (2017). Challenges and changes: developing teachers’ and initial teacher education students’ understandings of the nature of science. Research in Science Education, 47(6), 1233–1254.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Steffen Bohni Nielsen, Sebastian Lemire, Steve Montague
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
