John Mayne and Rules of Thumb for Contribution Analysis
A Comparison With Two Related Approaches
DOI:
https://doi.org/10.3138/cjpe.75448Keywords:
contribution analysis, theory-driven evaluations, realist evaluation, policy-scientific approach, John MayneAbstract
This article discusses differences and similarities between (methodological) rules of thumb of contribution analysis, realist evaluation, and the policy-scientific approach to (program) evaluations. John Mayne’s work and his operating procedures are presented and structured. One of the conclusions is that the three approaches form a ‘family.’ This ‘family’ can substantially contribute to at least six of the 10 “declarations of the Program Theory Manifesto” presented in 2019.
References
Bemelmans-Videc, M.-L., Rist, R. C., & Vedung, E. (Eds.). (1998). Carrots, sticks & sermons: Policy instruments and their evaluation. New Brunswick, NJ: Transaction.
Booth, A., Wright, J., & Briscoe, S. (2018). Scoping and searching to support realist approaches. In N. Emmel et al. (Eds.), Doing realist research (pp. 147–167). London, UK: Sage.
Budhwani, S., & McDavid, J. (2017). Contribution analysis: Theoretical and practical challenges and prospects for evaluators. Canadian Journal of Program Evaluation, 32(1), 1–24.
Carvalho, S., & White H. (2003). Theory-based evaluation. The case of social funds. American Journal of Evaluation, 25(2), 141–160.
Dybdal, L., Nielsen, S. B., & Lemire, S. (2010). Contributions analysis applied: Reflections on scope and methodology. Canadian Journal of Program Evaluation, 25(2), 29–57.
Ehren, M., Leeuw, F. L., & Scheerens, J. (2005). On the impact of the Dutch Educational Supervision Act: Analyzing assumptions concerning the Inspection of Primary Education. American Journal of Evaluation, 26(1), 60–76.
Emmel, N., Greenhalgh, J., Manzano, A., Monaghan, M., & Dalkin, S. (Eds.). (2018). Doing realist research. London, UK: Sage.
Evalsed [European Commission] (2012). Sourcebook: methods and techniques. Chapter 8.1. Brussels: DG Regional and Urban Policies. Retrieved from https://ec.europa.eu/regional_policy/sources/evaluation/guide/evaluation_sourcebook.pdf
Gilmore, B., McAuliffe, J., Power, J., & Vallieres, F. (2019). Data analysis and synthesis within a realist evaluation: Toward more transparent methodological approaches. International Journal of Qualitative Methods, 18, 1–11.
Hood, C., & Margetts, H. (2007). The tools of government in the digital age. London, UK: Palgrave Macmillan.
Hoogerwerf, A. (1987). Beleid berust op veronderstellingen: De beleidstheorie. Acta Politica, (19), 493–531.
Klein Haarhuis, C. (2005). Promoting anti-corruption reforms: Evaluating the implementation of a World Bank anti-corruption program in seven African countries (1999–2001). Unpublished dissertation, Utrecht: Utrecht University.
Kruisbergen, E. (2007). Voorlichting: doen of laten? Theorie van afschrikwekkende voorlichtingscampagnes toegepast op de casus van bolletjesslikkers. Beleidswetenschap 19(3), 38–51.
van der Laan, A., Zeijlmans, K., Beerthuizen, M., & Prop, L. (2021). Evaluatie van het adolescentenstrafrecht. Een multi-criteria evaluatie. Den Haag: WODC.
Lahey, R. (2023). John Mayne and the origins of evaluation in the public sector in Canada: A shaping of both evaluation and the evaluator. Canadian Journal of Program Evaluation, 37(3), 340–354.
Leeuw, F. L. (1991a). Aspects méthodologiques de la reconstruction et de l’evaluation des théories de comportement qui soustendent une politique demographique. Politiques de Population, (4), 5–43.
Leeuw, F. L. (1991b). Policy theories, knowledge utilization, and evaluation. Knowledge and Policy, 4(3), 73–91.
Leeuw, F. L. (2003). Reconstructing program theories: Methods available and problems to be solved. American Journal of Evaluation, 24(1), 5–20.
Leeuw, F. L. (2012). Linking theory-based evaluation and contribution analysis: Three problems and a few solutions. Evaluation, 18(3), 348–363.
Leeuw, F. L. (2020). Program evaluation B: Evaluation, big data, and artificial intelligence: two sides of one coin. In E. Vigoda-Gadot, & D. R. Vashdi (Eds.), Handbook of research methods in public administration, management and policy (pp. 277–297). Cheltenham: EE.
Leeuw, F. L., & Donaldson, S. (2015). Theory in evaluation: Reducing confusion and encouraging debate. Evaluation, 21(4), 467–480.
Leeuw, F. L., & Schmeets, H. (2016). Empirical legal research: A guidance book for lawyers, legislators and regulators. Cheltenham: EE.
Lemire, S., Kwako, A., Nielsen, S.B., Christie, C., Donaldson, S., & Leeuw, F. (2020). What is this thing called a mechanism? Findings from a review of realist evaluations. In J. Schmitt (Ed.), “Special Issue: Causal Mechanisms in Program Evaluation.” New Directions for Evaluation, 167, 73–86.
Lemire, S., Whynot, J. & Montague, S. (2019). How we model matters: A manifesto for the next generation of program theorizing. Canadian Journal of Program Evaluation, 33(3), 414–433.
Leuuw, F. L., & Waltermann, A. M. (2022). On identifying assumptions underlying legal arrangements: Some conceptual and methodological considerations. Law and Method, 11, 1–21.
Mannheim, K. (1935). Mensch und gesellschaft im zeitalter des umbaus. Leiden: Sijthoff.
Marchal, B., Kegels, G., & Van Belle, S. (2018). Theory and realist methods. In N. Emmel et al. (Eds.), Doing realist research (pp. 79–91). London, UK: Sage.
Mason, I., & Mitroff, I. (1981). Challenging strategic planning assumptions. New York, NY: Wiley.
Mayne, J. (1999). Addressing attribution through contribution analysis: Using performance measures sensibly (discussion paper). Ottawa, ON: Office of the Auditor General of Canada.
Mayne, J. (2001). Addressing attribution through contribution analysis: Using performance measures sensibly. Canadian Journal of Program Evaluation, 16(1), 1–24.
Mayne, J. (2008). Contribution analysis: An approach to exploring cause and effect (ILAC Brief 16). Rome, Italy: Institutional Learning and Change (ILAC) Initiative. Retrieved at http://www.inteval-group.org/IMG/pdf/ ILAC_Contribution_Analysis-2.pdf
Mayne, J. (2011). Contribution analysis: Addressing cause and effect. In R. Schwartz, K. Forss, & M. Marra (Eds.), Evaluating the complex: Attribution, contribution and beyond (pp. 53–97). New York, NY: Transaction.
Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18(3), 270–280.
Mayne, J. (2015). Useful theory of change models. Canadian Journal of Program Evaluation, 30(2), 119–142.
Mayne, J. (2017). Theory of change analysis: Building robust theories of change. Canadian Journal of Program Evaluation, 32(2), 155–173.
Mayne, J. (2018). Contribution analysis and the long-term perspective: Challenges and opportunities, Unpublished paper, July 14, 2018.
Mayne, J. (2019). Revisiting contribution analysis. Canadian Journal of Program Evaluation, 34(2), 171–191.
McDavid, J. (2023). Enduring themes in John Mayne’s work: Implications for evaluation practice. Canadian Journal of Program Evaluation, 37(3), 355–370.
Merton, R. K. (1968). On sociological theories of the middle range. In R. Merton, Social theory and social structure (pp. 39–73). New York, NY: Free Press.
Nagtegaal, M. (2020). Wet langdurig toezicht. Reconstructie van de beleidstheorie, eerste evidentie en nadere onderzoekthema’s. Den Haag: WODC.
Nielsen, S. B., Lemire, S., & Tangsig, S. (2022). Unpacking context in realist evaluations: Findings from a comprehensive review. Evaluation, 28(1), 91–112.
Noije, L. v., & Wittebrood, K. (2010). Fighting crime by fighting misconceptions and blind spots in policy theories: An evidence-based evaluation of interventions and assumed causal mechanisms. American Journal of Evaluation, 31(4), 499–516.
Palenberg, M. A. (2023). Causal claims in contribution analysis. Canadian Journal of Program Evaluation, 37(3), 389–402.
Pawson, R. (2002a). Evidence-based policy: The promise of ‘realist synthesis.’ Evaluation, 8(3), 340–358.
Pawson, R. (2002b). Evidence and policy and naming and shaming. Policy Studies, 23(3), 211–230.
Pawson, R. (2006). Evidence-based policy: A realist perspective. London, UK: Sage.
Pawson, R. (2013). The science of evaluation. London, UK: Sage.
Pawson, R. (2021). The coronavirus response: Boxed in by models. Evaluation, 27(2), 149–167.
Pawson, R., & Sridharan, S. (2010). Theory-driven evaluation of public health programmes, in A. Killoran and M. Kelly (Eds.), Evidence-based public health: Effectiveness and efficiency (pp. 43–62). Oxford, UK: Oxford University Press.
Popper, K. (1935). Logik der Forschung. Vienna: Julius Springer.
Rameses (2013). RAMESES I (Realist and Meta-narrative Evidence Syntheses: Evolving Standards) http://ramesesproject.org/
Rossi, P., Freeman, H., & Lipsey, M. (1999). Evaluation: A systematic approach. Thousand Oaks, CA: Sage.
Siebert, P., & P. Myles (2019). Eliciting and reconstructing programme theory: An exercise in translating theory into practice. Evaluation, 25(4), 469–476.
Schmitt, J. (2020). The causal mechanism claim in evaluation: Does the prophecy fulfill? In J. Schmitt (Ed.), Causal mechanisms in program evaluation. New Directions for Evaluation, 167, 11–26.
Sridharan, S., Nakaima, A., & Pereira, A. (2023). Causality and complexity in evaluating equity interventions: Celebrating contribution analysis. Canadian Journal of Program Evaluation, 37(3), 371–388.
Toulmin, S. (1958). The uses of argument. Cambridge, UK: Cambridge University Press.
Westhorp, G. (2014). Realist impact evaluation. An introduction. A methods lab publication. London, UK: Overseas development Institute.
Westhorp, G. (2018). Understanding mechanisms in realist evaluation and research. In N. Emmel et al. (Eds.), Doing realist research (pp. 41–59). London, UK: Sage.
Wippler, R. (1978). The structural-individualistic approach in Dutch sociology: Toward an explanatory social science. The Netherlands Journal of Sociology, 14(2), 135–155.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Frans L. Leeuw
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
