Realistic Evaluation and the Process-Tracing Method

A Combined Approach to Scrutinizing Causal Mechanisms

Authors

  • Salah eddine Bouyousfi Groupe Institut Supérieur de Commerce et d’Administration des Entreprises, Casablanca, Morocco https://orcid.org/0000-0002-2414-1399
  • Mohamed Sabar Groupe Institut Supérieur de Commerce et d’Administration des Entreprises, Casablanca, Morocco

DOI:

https://doi.org/10.3138/cjpe.71050

Keywords:

causal mechanisms, process tracing method, professional integration, realistic evaluation

Abstract

This article proposes a methodological contribution for causal mechanism scrutiny by combining realistic evaluation (RE) and the process-tracing method (PTM). To overcome some limits of RE’s operationalization, the combination RE-PTM reinforces RE by breaking down the causal mechanisms and testing them with PTM. This combination was tested as part of a project for the professional integration of young people in difficulty. We developed a theoretical framework involving the framing of RE and used Bayesian logic to test the RE hypotheses. Semi-structured interviews served as the main evidence source. The results of the study support the fact that the RE-PTM combination can address some RE operationalization challenges. The RE-PTM procedure shows that PTM can provide RE with guidance on how to collect and assess data that support the contribution made by an intervention in achieving the expected outcomes.

References

Allard, R., & Ouellette, J. G. (2002). Vers un modèle macroscopique des facteurs déterminants de l’insertion socioprofessionnelle des jeunes. Carriérologie, 8 (3), 497 - 517.

Astbury, B., & Leeuw, F. L. (2010). Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation, 31 (3), 363 – 381. https://doi.org/10.1177/1098214010371972

Beach, D., & Pedersen, R. B. (2013). Process tracing methods: Foundation and guidelines. University of Michigan Press.

Beach, D., & Pedersen, R. B. (2019). Process-tracing methods: Foundations and guidelines (2nd ed.). University of Michigan Press. https://www.press.umich.edu/10072208/process_tracing_methods%0A

Befani, B., & Mayne, J. (2014). Process tracing and contribution analysis: A combined approach to generative causal inference for impact evaluation. IDS Bulletin, 45 (6), 17 – 36. https://doi.org/10.1111/1759-5436.12110

Befani, B., & Stedman-Bryce, G. (2016). Process tracing and Bayesian updating for impact evaluation. Evaluation, 23 (1), 42 – 60. https://doi.org/10.1177/1356389016654584

Bhaskar, R. (1979). The possibility of naturalism: A philosophical critique of the contemporary human sciences. Humanities Press.

Brousselle, A., & Buregeya, J. - M. (2018). Theory-based evaluations: Framing the existence of a new theory in evaluation and the rise of the 5th generation. Evaluation, 24 (2), 153 – 168. https://doi.org/10.1177/1356389018765487

Chen, H.-T. (2005). Practical program evaluation. Sage. https://doi.org/10.4135/9781412985444

Corbin, J., & Strauss, A. (2008). Basics of qualitative research: Techniques and procedures for developing grounded theory (3rd ed.). Sage.

De Souza, D. E. (2015). Critical realism and realist review: Analyzing complexity in educational restructuring and the limits of generalizing program theories across borders. Ameri-can Journal of Evaluation, 37 (2), 1 – 22. https://doi.org/10.1177/1098214015605175

Devaux-Spatarakis, A. (2014). L’évaluation « basée sur la théorie » entre rigueur scienti-fi que et contexte politique. Politiques et Management Public, 31 (1), 75 – 91. https://doi.org/10.3166/pmp.31.51-68

Fairfield, T., & Charman, A. (2019). A dialogue with the data: The Bayesian foundations of iterative research in qualitative social science. Perspectives on Politics, 17 (1), 154 – 167. https://doi.org/10.1017/s1537592718002177

Fletcher, A., Farah, J., Moore, G., Evans, R. E., Murphy, S., & Bonell, C. (2016). Realist complex intervention science: Applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation,22 (3), 286 – 303. https://doi.org/10.1177/1356389016652743. Medline:27478401

Fournier, G., & Bourassa, B. (2001). Disparition ou transformation des formes de l'emploi? Entretien du CRIEVAT-Laval avec José Rose. Trajectoires professionnelles et marché du travail contemporain. Presses de l’Université Laval.

Fournier, G., & Monette, M. (2000). L’insertion socioprofessionnelle: un jeu de stratégie ou un jeu de hasard? Presses de l’Université Laval. Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. Jossey-Bass.

Hay, C. (2016). Process tracing: A laudable aim or a high-tariff methodology? New Political Economy, 21 (5), 500 – 504. https://doi.org/10.1080/13563467.2016.1201806

Heavey, C., Halliday, S. V., Gilbert, D., & Murphy, E. (2011). Enhancing performance: Bringing trust, commitment and motivation together in organisations. Journal of General Management, 36 (3), 1 – 18. https://doi.org/10.1177/030630701103600301

Hsieh, H. - F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15 (9), 1277 – 1288. https://doi.org/10.1177/1049732305276687. Medline:16204405

Leeuw, F. L. (2012). Linking theory-based evaluation and contribution analysis: Three problems and a few solutions. Evaluation, 18 (3), 348 – 363. https://doi.org/10.1177/1356389012452051

Lemire, S. T., Nielsen, S. B., & Dybdal, L. (2012). Making contribution analysis work: A practical framework for handling influencing factors and alternative explanations. Evaluation, 18 (3), 294 - 309. https://doi.org/10.1177/1356389012450654

Mackenzie, M., O’Donnell, C., Halliday, E., Sridharan, S., & Platt, S. (2010). Do health improvement programmes fit with MRC guidance on evaluating complex interventions? BMJ, 340 (c185). https://doi.org/10.1136/bmj.c185. Medline:20123834

Manzano, A. (2016). The craft of interviewing in realist evaluation. Evaluation,22 (3), 342 – 360. https://doi.org//10.1177/1356389016638615

Mayne, J. (2015). Useful theory of change models. Canadian Journal of Program Evaluation,30 (2), 119 – 142. https://doi.org/10.3138/cjpe.230 Mayne, J. (2018). The COM-B Theory of Change Model (V4). https://www.researchgate.net/publication/323868561_The_COMB_ToC_Model4

Michie, S., West, R., & van Stralen, M. M. (2011). The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation Science, 6, article 42. https://doi.org/10.1186/1748-5908-6-42. Medline:21513547

Pawson, R. (2003). Nothing as practical as a good theory. Evaluation, 9 (4), 471 – 490.

Pawson, R. (2013). The science of evaluation: A realist manifesto. Sage.

Pawson, R., & Manzano-Santaella, A. (2012). A realist diagnostic workshop. Evaluation,18 (2), 176 – 191. https://doi.org/10.1177/1356389012440912

Pawson, R., & Tilley, N. (1997). Realistic evaluation. Sage.

Porter, S. (2015). Realist evaluation: An immanent critique. Nursing Philosophy, 16 (4), 239 – 251. https://doi.org/10.1111/nup.12100. Medline:26392234

Punton, M., Vogel, I., & Lloyd, R. (2016). Refl ections from a realist evaluation in progress: Scaling ladders and stitching theory. CDI Practice Paper 18. IDS.

Ridde, V., Robert, E., Guichard, A., Blaise, P., & van Olmen, J. (2012). L’approche Realist à l’épreuve du réel de l’évaluation des programmes. Th e Canadian Journal of Program Evaluation, 26 (3), 37 – 59.

Rohlfing, I. (2014). Comparative hypothesis testing via process tracing. Sociological Methods & Research, 43 (4), 606 – 642. https://doi.org/10.1177/0049124113503142

Rolfe, S. (2019). Combining theories of change and realist evaluation in practice: Les-sons from a research on evaluation study. Evaluation,25 (3), 294 - 316. https://doi.org/10.1177/1356389019835229

Salter, K. L., & Kothari, A. (2014). Using realist evaluation to open the black box of knowledge translation: A state-of-the-art review. Implementation Science, 9 (1), 115. https://doi.org/10.1186/s13012-014-0115-y. Medline:25190100

Smith, C., & Elger, T. (2014). Critical realism and interviewing subjects. In P. K. Edwards, J. O’Mahoney, & S. Vincent (Eds.), Studying organizations using critical realism: A practical guide (1st ed., pp. 109 - 131). Oxford University Press.

Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., Befani, B., & Thévenet, M. (2012). Broadening the range of designs and methods for impact evaluations. Report of a study commissioned by the Department for International Development. Working Paper 38. Editions d’organisation.

UK Department for Business, Energy & Industrial Strategy. (2017). Evaluation of the transitional arrangements: Phase 2. https://www.gov.uk/government/publications/evaluation-of-the-transitional-arrangements-phase-1

Wadeson, A., Monzani, B., & Aston, T. (2020, March 25). Process tracing as a practical evaluation method: Comparative learning from six evaluations. Monitoring and Evaluation News. https://mande.co.uk/2020/media-3/unpublished-paper/process-tracing-as-a-practical-evaluation-method-comparative-learning-from-six-evaluations/?utm_source=dlvr.it&utm_medium=twitter

Wand, T., White, K., & Patching, J. (2011). Realistic evaluation of an emergency department-based mental health nurse practitioner outpatient service in Australia. Nursing and Health Sciences,13 (2), 199 – 206. https://doi.org/10.1111/j.1442-2018.2011.00601.x. Medline:21595814

Westhorp, G. (2018). Understanding mechanisms in realist evaluation and research. In N. Emmel, J. Greenhalgh, A. Manzano, M. Monaghan, & S. Dalkin (Eds.), Doing realist research (pp. 41 - 58). Sage.

Zhu, Y., & Akhtar, S. (2014). How transformational leadership influences follower helping behavior: The role of trust and prosocial motivation. Journal of Organizational Behavior, 35, 373 – 392. https://doi.org/10.1002/job.1884

Downloads

Published

2023-04-21

How to Cite

Bouyousfi, S. eddine, & Sabar, M. (2023). Realistic Evaluation and the Process-Tracing Method: A Combined Approach to Scrutinizing Causal Mechanisms. Canadian Journal of Program Evaluation, 37(1), 83–103. https://doi.org/10.3138/cjpe.71050

Issue

Vol. 37 No. 1 (2022): Spring

Section

Articles

License

Copyright (c) 2022 Salah eddine Bouyousfi, Mohamed Sabar

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.

Authors retain copyright of their work and grant the journal right of first publication.

Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.