Revisiting Contribution Analysis
DOI:
https://doi.org/10.3138/cjpe.68004Keywords:
causal factors, causal narratives, contribution analysis, contribution claims, contributory causes, theories of changeAbstract
The basic ideas behind contribution analysis were set out in 2001. Since then, interest in the approach has grown and contribution analysis has been operationalized in different ways. In addition, several reviews of the approach have been published and raise a few concerns. In this article, I clarify several of the key concepts behind contribution analysis, including contributory causes and contribution claims. I discuss the need for reasonably robust theories of change and the use of nested theories of change to unpack complex settings. On contribution claims, I argue the need for causal narratives to arrive at credible claims, the limited role that external causal factors play in arriving at contribution claims, the use of robust theories of change to avoid bias, and the fact that opinions of stakeholders on the contribution made are not central in arriving at contribution claims.
References
Befani, B., & Mayne, J. (2014). Process tracing and contribution analysis: A combined approach to generative causal inference for impact evaluation. IDS Bulletin, 45(6), 17–36. https://doi.org/10.1111/1759-5436.12110
Befani, B., & Stedman-Bryce, G. (2016). Process tracing and Bayesian updating for impact evaluation. Evaluation, 23 (1), 42–60. https://doi.org/10.1177/1356389016654584
Buckley, A. P. (2016). Using Contribution Analysis to evaluate small & medium enterprise support policy. Evaluation, 22 (2), 129–148. https://doi.org/10.1177/1356389016638625
Budhwani, S., & McDavid, J. C. (2017). Contribution analysis: Theoretical and practical challenges and prospects for evaluators. Canadian Journal of Program Evaluation, 32 (1), 1–24. https://doi.org/10.3138/cjpe.31121
Buregeya, J. M., Brousselle, A., Nour, K., & Loignon, C. (2017). Comment évaluer les eff ets des évaluations d’impact sur la santé : le potentiel de l’analyse de contribution. Canadian Journal of Program Evaluation, 32 (1), 25–45. https://doi.org/10.3138/cjpe.31151
Byrne, D. (2013). Evaluating complex social interventions in a complex world. Evaluation, 19 (3), 217–228. https://doi.org/10.1177/1356389013495617
Cartwright, N., & Hardie, J. (2012). Evidence-based policy: Doing it better. A practical guide to predicting if a policy will work for you. Oxford, England: Oxford University Press.
Davidson, E. J. (2009). Causation inference: Nuts and bolts. A Mini Workshop for the ANZEA Wellington branch. Welington, New Zealand. Retrieved from http://realevaluation.com/pres/causation-anzea09.pdf
Davidson, E. J. (2013). Understanding causes and outcomes of impacts . BetterEvaluation coffee break webinars. Retrieved from https://www.betterevaluation.org/events/coffee_break_webinars_2013#webinarPart5
Davies, R. (2012). Criteria for assessing the evaluability of Theories of Change. Rick on the road [blog]. Retrieved from http://mandenews.blogspot.com/2012/04/criteria-forassessing-evaluablity-of.html
Delahais, T., & Toulemonde, J. (2012). Applying contribution analysis: Lessons from five years of real life experience. Evaluation, 18(3), 281–293. https://doi.org/10.1177/1356389012450810
Delahais, T., & Toulemonde, J. (2017). Making rigorous causal claims in a real-life context: Has research contributed to sustainable forest management? Evaluation, 23(4), 370–388. https://doi.org/10.1177/1356389017733211
Douthwaite, B., Mayne, J., McDougall, C., & Paz-Ybarnegaray, R. (2017). Evaluating complex interventions: A theory-driven realist-informed approach. Evaluation, 23(3), 294–311. https://doi.org/10.1177/1356389017714382
Downes, A., Novicki, E., & Howard, J. (2018). Using the contribution analysis approach to evaluate science impact: A case study of the national institute for occupational safety and health. American Journal of Evaluation, 40(2), 177–189. https://doi.org/10.1177/1098214018767046. Medline:30518992
Dybdal, L., Nielsen, S. B., & Lemire, S. (2010). Contribution analysis applied: Refl ections on scope and method. Canadian Journal of Program Evaluation, 25 (2), 29–57.
Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping. Ottawa, ON: International Development Research Centre.
Freeman, T., Mayhew, S., Matsinhe, C., & Nazza, D. A. (2014). Evaluation of the Danish strategy for the promotion of sexual and reproductive health and rights 2006–2013— Pathways to change in SRHR: Synthesis report. Ministry of Foreign Affairs of Denmark. Retrieved from http://www.netpublikationer.dk/um/14_evaluation_2014_03/Pdf/evaluation_2014_03.pdf
Gates, E., & Dyson, L. (2017). Implications of the changing conversation about causality for evaluators. American Journal of Evaluation, 38(1), 29–46. https://doi.org/10.1177/1098214016644068
Gerrits, L., & Verweij, S. (2015). Taking stock of complexity in evaluation: A discussion of three recent publications. Evaluation, 21(4), 481–491. https://doi.org/10.1177/1356389015605204
Kane, R., Levine, C., Orians, C., & Reinelt, C. (2017). Contribution analysis in policy work: Assessing advocacy’s infl uence. Centre for Evaluation Innovation. Retrieved from https://www.evaluationinnovation.org/wp-content/uploads/2017/11/Contribution-Analysis_0.pdf
Koleros, A., & Mayne, J. (2019). Using actor-based theories of change to conduct robust contribution analysis in complex settings. Canadian Journal of Program Evaluation, 33 (3), 292–315. https://doi.org/10.3138/cjpe.52946
Lemire, S. T., Nielsen, S. B., & Dybdal, L. (2012). Making contribution analysis work: A practical framework for handling influencing factors and alternative explanations. Evaluation, 18 (3), 294–309. https://doi.org/10.1177/1356389012450654
Mackie, J. L. (1974). The cement of the universe: A study of causation. Oxford, England: Oxford University Press.
Mahoney, J. (2008). Toward a unified theory of causality. Comparative Political Studies, 41 (4/5), 412–436. https://doi.org/10.1177/0010414007313115
Mayne, J. (2001). Addressing attribution through contribution analysis: Using performance measures sensibly. Canadian Journal of Program Evaluation, 16(1), 1–24. Retrieved from https://evaluationcanada.ca/system/files/cjpe-entries/16-1-001.pdf
Mayne, J. (2011). Contribution analysis: Addressing cause and effect. In R. Schwartz, K. Forss, & M. Marra (Eds.), Evaluating the complex (pp. 53–96). New Brunswick, NJ: Transaction Publishers.
Mayne, J. (2012). Contribution analysis: Coming of age? Evaluation, 18 (3), 270–280. https://doi.org/10.1177/1356389012451663
Mayne, J. (2015). Useful theory of change models. Canadian Journal of Program Evaluation, 30 (2), 119–142. https://doi.org/10.3138/cjpe.230
Mayne, J. (2017). Theory of change analysis: Building robust theories of change. Canadian Journal of Program Evaluation, 32 (2), 155–173. https://doi.org/10.3138/cjpe.31122
Mayne, J. (2018). The COM-B theory of change model. Retrieved from https://www.researchgate.net/publication/323868561_The_COMB_ToC_Model4
Mayne, J. (2019). Assessing the relative importance of causal factors. CDI Practice Paper 21. Brighton, England: IDS. Retrieved from https://opendocs.ids.ac.uk/opendocs/handle/123456789/14647
Mayne, J. (Forthcoming). Realistic commissioning of impact evaluations: Getting what you ask for? In A. Paulson & M. Palenberg (Eds.), Evaluation and the pursuit of impact. Abingdon, England: Taylor and Francis.
Mayne, J., & Johnson, N. (2015). Using theories of change in the CGIAR Research Program on Agriculture for Nutrition and Health. Evaluation, 21(4), 407–428. https://doi.org/10.1177/1356389015605198
Michie, S., Atkins, L., & West, R. (2014). The behaviour change wheel: A guide to designing interventions. London, England: Silverback Publishing. Retrieved from http://www.behaviourchangewheel.com/
Noltze, M., Gaisbauer, F., Schwedersky, T., & Krapp, S. (2014). Contribution analysis as an evaluation strategy in the context of a sector-wide approach: Performance-based health financing in Rwanda. African Evaluation Journal, 2(1), 8 pp. Retrieved from http://www.aejonline.org/index.php/aej/article/view/81
Palladium. (2015). Independent evaluation of the Security Sector Accountability and Police Reform Programme: Final evaluation report. DFID. Retrieved from http://r4d.dfid.gov.uk/pdf/outputs/SSAPRP-DRC/SSAPR_Complete_Impact_Evaluation_Rep_Final-Eng.pdf
Pawson, R., & Tilley, N. (1997). Realistic evaluation. London, England: SAGE.
Paz-Ybarnegaray, R., & Douthwaite, B. (2016). Outcome evidencing: A method for enabling and evaluating program intervention in complex systems. American Journal of Evaluation, 38 (2), 275–293. https://doi.org/10.1177/1098214016676573
Pearl, J., & Mackenzie, D. (2018). The book of why: The new science of cause and effect. New York, NY: Basic Books.
Punton, M., & Barnett, C. (2018). Contribution analysis and Bayesian confidence updating: A brief introduction. Itad briefing paper. Retrieved from https://itad.com/wp-content/uploads/2019/06/CDC-Brief-contribution-analysis-and-Bayesian-Updating.pdf
Punton, M., & Welle, K. (2015). Straws-in-the-wind, hoops and smoking guns: What can process tracing offer to impact evaluation? CDI Practice Paper, No. 10. Centre for Development Impact. Retrieved from http://opendocs.ids.ac.uk/opendocs/bitstream/handle/123456789/5997/CDIPracticePaper_10.pdf?sequence=1
Riley, B. L., Kernoghan, A., Stockton, L., Montague, S., Yessis, J., & Willis, C. D. (2018). Using contribution analysis to evaluate the impacts of research on policy: Getting to “good enough.” Research Evaluation, 27(1), 16–27. https://doi.org/10.1093/reseval/rvx037. Retrieved from https://academic.oup.com/rev/article/27/1/16/4554784
Schmitt, J., & Beach, D. (2015). The contribution of process tracing to theory-based evaluations of complex aid instruments. Evaluation, 21(4), 429–447. https://doi.org/10.1177/1356389015607739
Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (2012). Broadening the range of designs and methods for impact evaluations. DFID Working Paper, 38. London, England: DFID. Retrieved from http://r4d.dfid.gov.uk/Output/189575/
Ton, G. (2017). Contribution analysis of a Bolivian innovation grant fund: Mixing methods to verify relevance, effi ciency and eff ectiveness. Journal of Development Effectiveness, 9 (1), 120–143. http://doi.org/10.1080/19439342.2016.1231702
Ton, G., Mayne, J., Delahais, T., Morell, J., Befani, B., Apgar, M., & O’Flynn, P. (2019). Contribution analysis and estimating the size of effects: Can we reconcile the possible with the impossible? CDI Practice Paper, Number 20. Centre for Development Impact. Retrieved from https://www.cdimpact.org/publications/contribution-analysis-andestimating-size-effects-can-we-reconcile-possible-impossible
Weiss, C. H. (1995). Nothing as practical as good theory: Exploring theory-based evaluation for comprehensive community initiatives for children and families. In J. P. Connell, A. C. Kubisch, L. B. Schorr, & C. H. Weiss (Eds.), New approaches to evaluating community initiatives: Volume 1: Concepts, methods and contexts (pp. 65–92). Washington, DC: The Aspen Institute.
Westhorp, G. (2014). Realist impact evaluation: An introduction. Methods Lab. Retrieved from http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinionfiles/9138.pdf
White, H. (2009). Theory-based impact evaluation: Principles and practice. Working Paper 3. International Initiative for Impact Evaluation (3ie). Retrieved from https://www.3ieimpact.org/sites/default/files/2017-11/Working_Paper_3.pdf
Wilson-Grau, R., & Britt, H. (2012). Outcome harvesting. Retrieved from http://www.outcomemapping.ca/resource/resource.php?id=374
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2019 John Mayne
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
