A Multi-Stage Approach to Qualitative Sampling within a Mixed Methods Evaluation
Some Reflections on Purpose and Process
DOI:
https://doi.org/10.3138/cjpe.71237Keywords:
India, international development, mixed methods, program evaluationAbstract
We share experiences from a mixed methods evaluation in rural India that combines a randomized controlled trial (RCT) of 400 villages with embedded case studies in four villages. Specifically, we present two lessons from the multi-stage sampling approach adopted to select the four case-study villages, which first prioritized key-informant observations regarding intervention status in order to shortlist locations and subsequently used data from the RCT’s baseline survey to select the final sample. In doing so, we highlight how large-scale mixed methods program evaluations in education can go beyond questions of “what works” to answering those of “how,” “why,” and “why not.”
References
Cartwright, N., & Hardie, J. (2012). Evidence-based policy: A practical guide to doing it better. Oxford University Press. https://doi.org/10.1093/acprof:osobl/9780199841608.001.0001
Collins, K. M. T., Onwuegbuzie, A. J., & Jiao, Q. G. (2006). Prevalence of mixed-methods sampling designs in social science research. Evaluation & Research in Education, 19(2), 83 – 101. https://doi.org/10.2167/eri421.0
Collins, K. M. T., Onwuegbuzie, A. J., & Jiao, Q. G. (2007). A mixed methods investigation of mixed methods sampling designs in social and health science research. Journal of Mixed Methods Research, 1(3), 267 – 294. https://doi.org/10.1177/1558689807299526
Corrigan, A. J., & Onwuegbuzie, A. J. (2020). Toward a meta-framework for conducting mixed methods representation analyses to optimize meta-inferences. The Qualitative Report, 25(3), 785 – 812. https://doi.org/10.46743/2160-3715/2020.3579
Hammersley, M., & Atkinson, P. (2007). The process of analysis. In Ethnography: Principles in practice (pp. 158 – 190). Routledge. https://doi.org/10.4324/9780203944769
Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945 – 960. https://doi.org/10.1080/01621459.1986.10478354
Kabeer, N. (2019). Randomized control trials and qualitative evaluations of a multifaceted programme for women in extreme poverty: Empirical findings and methodological reflections. Journal of Human Development and Capabilities, 20(2), 197 – 217. https://doi.org/10.1080/19452829.2018.1536696
Mason, J. (2018). Qualitative researching. Sage.
Newman, I., Ridenour, C. S., Newman, C., & DeMarco, G. M. P. (2003). A typology of re-search purposes and its relationship to mixed methods. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 167 – 188). Sage. https://doi.org/10.4135/9781506335193
Onwuegbuzie, A. J., & Hitchcock, J. H. (2017). A meta-framework for conducting mixed methods impact evaluations: Implications for altering practice and the teaching of evaluation. Studies in Educational Evaluation, 53, 55 – 68. https://doi.org/10.1016/j.stueduc.2017.02.001
Onwuegbuzie, A. J., & Leech, N. L. (2007). Sampling designs in qualitative research: Making the sampling process more public. The Qualitative Report, 12(2), 238 – 254. https://doi.org/10.46743/2160-3715/2007.1636
Outhwaite, L. A., Gulliford, A., & Pitchford, N. J. (2020). A new methodological approach for evaluating the impact of educational intervention implementation on learning outcomes. International Journal of Research & Method in Education, 43(3), 225 – 242. https://doi.org/10.1080/1743727x.2019.1657081
Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 42(5), 533 – 544. https://doi.org/10.1007/s10488-013-0528-y. Medline:24193818
Patton, M. Q. (1990). Qualitative research and evaluation methods (2nd ed.). Sage.
Peterson, A. (2016). Getting “what works” working: Building blocks for the integration of experimental and improvement science. International Journal of Research & Method in Education, 39(3), 299 – 313. https://doi.org/10.1080/1743727x.2016.1170114
Poth, C. N. (2018). Innovation in mixed methods research: A practical guide to integrative thinking with complexity. Sage.
Rog, D. J. (2012). When background becomes foreground: Toward context-sensitive evaluation practice. New Directions for Evaluation, 2012(135), 25 – 40. https://doi.org/10.1002/ev.20025
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and non-randomized studies. Journal of Educational Psychology, 66(5), 688 – 701. https://doi.org/10.1037/h0037350
White, H. (2009). Theory-based impact evaluation: Principles and practice. Journal of Development Effectiveness, 1(3), 271 – 284. https://doi.org/10.1080/19439340903114628
Yin, R. (2003). Case study research: Design and methods. Sage.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Purnima Ramanujan, Suman Bhattacharjea, Benjamin Alcott
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
