Principles, Approaches, and Methods for Evaluation in Indigenous Contexts
A Grey Literature Scoping Review
DOI:
https://doi.org/10.3138/cjpe.43050Keywords:
evaluation, Indigenous communities, Indigenous context, scoping literature reviewAbstract
This article describes findings from a scoping review of the grey literature to identify principles, approaches, methods, tools, and frameworks for conducting program evaluation in Indigenous contexts, reported from 2000–2015 in Canada, the United States, New Zealand, and Australia. It includes consultation with key informants to validate and enrich interpretation of findings. The fifteen guiding principles, and the approaches, methods, tools, and frameworks identified through this review may be used as a starting point for evaluators and communities to initiate discussion about how to conduct their evaluation in their communities, and which approaches, methods, tools, or frameworks would be contextually appropriate.
References
Aboriginal Family and Community Literacy Curriculum. (2011). Reading: Circle of courage frameworks – Aboriginal family and community literacy curriculum. Vancouver, BC: University of British Columbia.
Armitage, A. (1995). Comparing the policy of aboriginal assimilation: Australia, Canada, and New Zealand. Vancouver, BC: University of British Columbia Press.
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10. 1080/1364557032000119616
Athabasca University. (2015). Aboriginal organizations. Athabasca, AB: Author. Retrieved from http://indigenous.athabascau.ca/resources/library/aboriginal/index.php
Atlantic Council for International Cooperation. (2007). Medicine wheel evaluation framework. Halifax, NS: Atlantic Council for International Cooperation. Retrieved from http://static1.1.sqspcdn.com/static/f/1590842/20068323/1349103128263/Medicine_Wheel_Evaluation_Framework.pdf?token=llFfXrrug9OHQlTc5Eff288RuN4%3D
Ball, J., & Janyst, P. (2008). Enacting research ethics in partnerships with Indigenous communities in Canada: “Do it in a good way.” Journal of Empirical Research on Human Research Ethics, 3(2), 33–51. https://doi.org/10.1525/jer.2008.3.2.33. Med¬line:19385744
Better Evaluation. (2012). Approaches. Melbourne, Australia: RMIT University. Retrieved from http://www.betterevaluation.org/en/approaches
Canadian Institute of Health Research (CIHR), Natural Sciences and Engineering Research Council of Canada (NSERC), and Social Sciences and Humanities Research Council of Canada (SSHRC). (2014). Tri-Council policy statement: Ethical conduct for research involving humans. Ottawa, ON: Government of Canada. Retrieved from http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/Default/
Canadian Parks and Recreation Association. (2000). Toolkits: Everybody gets to play community mobilization tool kit. Ottawa, ON: Canadian Parks and Recreation Association.
Castellano, M. B. (2004). Ethics of Aboriginal research. International Journal of Indigenous Health, 1(1), 98. Retrieved from https://www.nvit.ca/docs/ethics%20of%20aboriginal%20research.pdf
Chesterton, P. (2003). Balancing ethical principles in evaluation: A case study. The Canadian Journal of Program Evaluation, 18 (1), 49–60.
Chouinard, J. A., & Cousins, J. B. (2007). Culturally competent evaluation for Aboriginal communities: A review of the empirical literature. Journal of MultiDisciplinary Evaluation, 4 (8), 40–57.
Cochran, P. A. L., Marshall, C. A., Garcia-Downing, C., Kendall, E., Cook, D., McCubbin, L., & Gover, R. M. S. (2008). Indigenous ways of knowing: Implications for participatory research and community. American Journal of Public Health, 98 (1), 22–27. https://doi.org/10.2105/AJPH.2006.093641
Crabtree, B. F., & Miller, W. L. (Eds.). (1999). Doing qualitative research. Newbury Park, CA: SAGE. Dapice, A. N. (2006). The medicine wheel. Journal of Transcultural Nursing, 17 (3), 251–260. https://doi.org/10.1177/1043659606288383
Drawson, A. S., Toombs, E., & Mushquash, C. J. (2017). Indigenous research methods: A systematic review. International Indigenous Policy Journal, 8 (2), 1–27.
Elliott, S. J. (1999). And the question shall determine the method. Professional Geographer, 51 (2), 240–243.
First Nations Centre. (2007). OCAP: Ownership, control, access and possession. Ottawa, ON: National Aboriginal Health Organization. Retrieved from http://cahr.uvic.ca/nearbc/documents/2009/FNC-OCAP.pdf
First Nations Information Governance Centre (FNIGC). (2015). First Nations Regional Health Survey (RHS) 2008/10: National report on adults, youth and children living in First Nations communities. Ottawa, ON: First Nations Information Governance Centre. Retrieved from https://fnigc.ca/wp-content/uploads/2020/09/8df51b653b65f2c39930156f72db642d_first_nations_regional_health_survey_rhs_2008-10_-_national_report_youth.pdf
Flicker, S., & Worthington, C. A. (2012). Public health research involving Aboriginal Peoples: Research ethics board stakeholders’ reflections on ethics principles and research processes. Canadian Journal of Public Health, 103(1), 19–22. https://journal.cpha.ca/index.php/cjph/article/viewFile/2896/2570. Medline:22338323
George, M. A., Masotti, P., MacLeod, S., Van Bibber, M., Loock, C., Fleming, M., … Smith, C. (2007). Bridging the research gap: Aboriginal and academic collaboration in FASD prevention. The Healthy Communities, Mothers, and Children Project. Aboriginal and academic collaboration in FASD prevention. Alaska Medicine, 49 (Suppl. 2), 139–141.
Gray, J., Gideon, V., Tournier, C., & Schnarch, B. (2004). Our voice, our survey, our future: The “survey of choice” by First Nations in Canada. Proceedings from Workshop on Data Collection and Disaggregation on Indigenous peoples, Permanent Forum on Indigenous Issues, New York, NY.
Greenhalgh, T., & Peacock, R. (2005). Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. British Medical Journal, 331(7524), 1064–1065. https://doi.org/10.1136/bmj.38636.593461.68. Medline:16230312
Grover, J. G. (2010). Challenges in applying Indigenous evaluation practices in mainstream grant programs to Indigenous communities. Canadian Journal of Program Evaluation, 23 (2), 33–50.
Hart-Wasekeesikaw, F. (2009). Cultural competence and cultural safety in nursing education: A framework for First Nations, Inuit, and Métis nursing. Ottawa, ON: Canadian Indigenous Nurses Association. Retrieved from https://cna-aiic.ca/~/media/cna/page-content/pdf-en/first_nations_framework_e.pdf
Health Canada. (2015). First Nations mental wellness continuum framework. Ottawa, ON: Author. Retrieved from http://publications.gc.ca/collections/collection_2015/sc-hc/H34-278-1-2014-eng.pdf
Hovey, R. B., Delormier, T., McComber, A. M., Lévesque, L., & Martin, D. (2017). En¬hancing Indigenous health promotion research through two-eyed seeing: A hermeneutic relational process. Qualitative Health Research, 27(9), 1–10. https://doi.org/10.1177/1049732317697948. Medline:28682710
Hudson, M., Milne, M., Reynolds, P., Russell, K., & Smith, B. (2010). Te ara tika: Guidelines for Māori research ethics: A framework for researchers and ethics committee members. Auckland, New Zealand: Health Research Council of New Zealand on behalf of the Putaiora Writing Group. Retrieved from http://www.hrc.govt.nz/sites/default/files/Te%20Ara%20Tika%20Guidelines%20for%20Maori%20Research%20Ethics.pdf
Jeffery, B., Abonyi, S., Hamilton, C., Bird, S., Denechezhe, M., Lidguerre, T., ... & Whitecap, Z. (2006). Community health indicators toolkit. Prince Albert, SK: University of Regina and University of Saskatchewan. Johnston Research Inc. (2015). Toronto, ON: Author. Retrieved from http://www.johnstonresearch.ca/
Kahnawake Schools Diabetes Prevention Project. (2007). Code of research ethics. Kahnawake, QC: Author. Retrieved from http://www.ksdpp.org/media/ksdpp_code_of_research_ethics2007.pdf
Katz, I., Newton, B. J., Bates, S., & Raven, M. (2016). Evaluation theories and approaches; Relevance for Aboriginal contexts. Sydney, Australia: Social Policy Research Centre, UNSW Australia.
Kovach, M. E. (2010). Indigenous methodologies: Characteristics, conversations, and con-texts. Toronto, ON: University of Toronto Press.
LaFrance, J. (2004). Culturally competent evaluation in Indian country. New Directions for Evaluation, 2004 (102), 39–50. https://doi.org/10.1002/ev.114
LaFrance, J., & Nichols, R. (2008). Reframing evaluation: Defining an Indigenous evaluation framework. Canadian Journal of Program Evaluation, 23 (2), 13.
LaFrance, J., Nichols, R., & Kirkhart, K. E. (2012). Culture writes the script: On the centrality of context in Indigenous evaluation. New Directions for Evaluation, 2012(135), 59–74. https://doi.org/10.1002/ev.20027
Lavallée, L. F. (2009). Practical application of an Indigenous research framework and two qualitative Indigenous research methods: Sharing circles and Anishnaabe symbol-based reflection. International Journal of Qualitative Methods, 8(1), 21–40. https://doi.org/10.1177/160940690900800103
Lavallée, L., & Lévesque, L. (2013). Two-eyed seeing: Physical activity, sport, and recreation promotion in Indigenous communities. In J. Forsyth & A. R. Giles (Eds.), Aboriginal peoples and sport in Canada: Historical foundations and contemporary Issues (pp. 206–228). Vancouver, BC: University of British Columbia Press.
Laycock, A., Walker, D., Harrison, N., & Brands, J. (2011). Researching Indigenous health: A practical guide for researchers. Carlton, Australia: The Lowitja Institute.
Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scoping studies: Advancing the methodology. Implementation Science, 5 (1), 69. https://doi.org/10.1186/1748-5908-5-69
Maddox, R., Blais, G., Monchalin, R., Firestone, M., Ziegler, C., Mashford-Pringle, A., & Smylie, J. (in press). Evaluation of Indigenous health service and program evaluations: A systematic review (Unpublished manuscript). Toronto, ON: Well Living House Action Research Centre.
Mikkonen, J., & Raphael, D. (2010). Social determinants of health: The Canadian facts. Toronto, ON: York University, School of Health Policy and Management.
Miles, M. B., & Huberman, A. M. (1994). Conceptual framework. In Qualitative data analysis: An expanded sourcebook (pp. 40–49). Thousand Oaks, CA: SAGE.
National Aboriginal Health Organization. (2003). Winds of change: A strategy for health policy research and analysis. Ottawa, ON: National Aboriginal Health Organization.
National Collaborating Centre for Aboriginal Health. (2013). Indigenous approaches to program evaluation. Prince George, BC: National Collaborating Centre for Aboriginal Health. Retrieved from https://www.nccih.ca/docs/context/FS-IndigenousApproachesProgramEvaluation-EN.pdf
National Collaborating Centre for Methods and Tools. (2017). Glossary. Hamilton, ON: McMaster University. Retrieved from http://www.nccmt.ca/glossary
Ontario Federation of Indigenous Friendship Centres. (2012). USAI (utility, self-voicing, access, and inter-relationality framework) research framework. Toronto, ON: Ontario Federation of Indigenous Friendship Centres. Retrieved from http://ofifc.org/sites/default/files/docs/USAI%20Research%20Framework%20Booklet%202012.pdf
Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & McEwen, S. A. (2014). A scoping review of scoping reviews: Advancing the approach and enhancing the consistency. Research synthesis methods, 5(4), 371–385. https://doi.org/10.1002/jrsm.1123 . Medline:26052958
Poole, N., Rutman, D., Van Bibber, M., Hummerstey, C., & Hume, S. (2014). Evaluating FASD prevention and support programs: Tools to support planning and evaluation. Vancouver, BC: British Columbia Centre of Excellence for Women’s Health. Retrieved from http://www.fasd-evaluation.ca/wp-content/uploads/2012/10/FASD-Evaluation-Framework-Webinar-Slides-Feb-25-2014.pdf
Principle. In Oxford English Dictionary. Retrieved from https://en.oxforddictionaries.com/definition/principle
Public Health Agency of Canada (PHAC). (2015). Ways tried and true: Aboriginal methodological framework for the Canadian best practices initiative. Ottawa, ON: Public Health Agency of Canada. Retrieved from http://publications.gc.ca/collections/collection_2015/aspc-phac/HP35-59-2015-eng.pdf
Putt, J. (2013). Conducting research with Indigenous people and communities. Canberra City, Australia: Indigenous Justice Clearinghouse. Retrieved from http://www.indigenousjustice.gov.au/briefs/brief015.pdf
Reading, C. L., & Wien, F. (2009). Health inequalities and social determinants of Aboriginal peoples’ health. Prince George, BC: National Collaborating Centre for Aboriginal Health. Retrieved from http://www.nccah-ccnsa.ca/Publications/Lists/Publications/Attachments/46/health_inequalities_EN_web.pdf
Reading, J. (2009). The crisis of chronic disease among Aboriginal Peoples: A challenge for public health, population health and social policy. Victoria, BC: Centre for Aboriginal Health Research, University of Victoria.
Riddell, J. K., Salamanca, A., Pepler, D. J., Cardinal, S., & McIvor, O. (2017). Laying the groundwork: A practical guide for ethical research with Indigenous communities. International Indigenous Policy Journal, 8 (2), 6. https://doi.org/10.18584/iipj.2017.8.2.6
Robeson, P. (2009). Networking in public health: Exploring the value of networks to the National Collaborating Centres for Public Health. Hamilton, ON: National Collaborating Centre for Methods and Tools. Retrieved from http://www.nccmt.ca/pubs/NetworkingPaperApr09EN_WEB.pdf
Rossi, P. H., Lipsey, M. W., & Freeman, H. E. (2003). Evaluation: A systematic approach. Thousand Oaks, CA: SAGE.
Rutman, D., Poole, N., Hume, S., Hubberstey, C., & Van Bibber, M. (2014). Building a framework for evaluation of fetal alcohol spectrum disorder prevention and support programs: A collaborative Canadian project. International Journal of Alcohol and Drug Research, 3 (1), 81–89. https://doi.org/10.7895/ijadr.v3i1.122
Simon Fraser University Library. (2018). Grey literature: What it is & how to find it. Vancouver, BC: Simon Fraser University. Retrieved from https://www.lib.sfu.ca/help/research-assistance/format-type/grey-literature
Smith, L. (1999). Decolonizing methodologies: Research and Indigenous peoples. London, England: Zed Books.
Smylie, J., & Anderson, M. (2006). Understanding the health of Indigenous peoples in Canada: Key methodological and conceptual challenges. Canadian Medical Association Journal, 175(6), 602. https://doi.org/10.1503/cmaj.060940 . Medline:16966664
Smylie, J., Firestone, M., Cochran, L., Prince, C., Maracle, S., Morley, M., ... & McPherson, B. (2011). Our health counts: Urban Aboriginal database research project: Community report. Hamilton, ON: Well Living House Action Research Centre.
Smylie, J., Lofters, A., Firestone, M., & O’Campo, P. (2011). Population-based data and community empowerment. In P. O’Campo & J. R. Dunn (Eds.), Rethinking social epidemiology: Towards a science of change (pp. 67–92). Dordrecht, Netherlands: Springer Science & Business Media.
Statistics Canada. (2016). Canadian Community Health Survey—annual component (CCHS). Ottawa, ON: Statistics Canada. Retrieved from http://www23.statcan.gc.ca/imdb/p2SV.pl?Function=getSurvey&SDDS=3226
Statistics Canada. (2017a). Aboriginal peoples survey 2012. Ottawa, ON: Statistics Canada, Indigenous and Northern Affairs Canada. Retrieved from http://www.aadnc-aandc.gc.ca/eng/1321384019753/1322059098232#2012
Statistics Canada. (2017b). First Nations regional early childhood, education and employment survey. Ottawa, ON: Statistics Canada, Indigenous and Northern Affairs Canada. Retrieved from http://www.aadnc-aandc.gc.ca/eng/1321384019753/1322059098232#Enq2012
Suri, H. (2011). Purposeful sampling in qualitative research synthesis. Qualitative Research Journal, 11 (2), 63–75.
Taylor, R. (2003). An Indigenous perspective on evaluations in the inter-cultural context: How far can one throw a Moree boomerang? Evaluation Journal of Australasia, 3(2), 44–53. https://doi.org/10.1177/1035719X0300300211
Truth and Reconciliation Commission of Canada. (2015). Honouring the truth, reconciling for the future: Summary report of the final report of the Truth and Reconciliation Commission of Canada. Winnipeg, MB: National Centre for Truth and Reconciliation. Retrieved from http://nctr.ca/assets/reports/Final%20Reports/Executive_Summary_English_Web.pdf
United Nations. (2008). United Nations declaration on the rights of Indigenous peoples. Geneva, Switzerland: United Nations. Retrieved from http://www.un.org/esa/socdev/unpfii/documents/DRIPS_en.pdf
Wilson , S. (2008). Research is ceremony: Indigenous research methods. Blackpoint, NS: Fernwood Publishing.
Young, N. L., Wabano, M. J., Burke, T. A., Ritchie, S. D., Mishibinijima, D., & Corbiere, R. G. (2013). A process for creating the Aboriginal children’s health and well-being measure (ACHWM). Canadian Journal of Public Health, 104(2), 136–141. https://journal.cpha.ca/index.php/cjph/article/view/3636 . Medline:23618206
Zorzi, R., Perrin, B., Mcguire, M., Long, B., & Lee, L. (2002). Defining the benefits, outputs, and knowledge elements of program evaluation. Canadian Journal of Program Evaluation, 17 (3), 143–150.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2019 Kriti Chandna, Michelle M. Vine, Susan J. Snelling, Rachel Harris, Janet Smylie, Heather Manson
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
