Catching the Wave
Harnessing Data Science to Support Evaluation’s Capacity for Making a Transformational Contribution to Sustainable Development
DOI:
https://doi.org/10.3138/cjpe.71527Keywords:
Artificial Intelligence, big data, development, digital twins, evaluation, machine learningAbstract
In this article we set ourselves three challenges: first, to examine the potential of the data revolution to aid the transformational change required to achieve sustainable development goals second, and the ability of evaluation to contribute to greater engagement with data science technologies; and third, the capacity of data science to further evaluation as an innovative and progressive field of inquiry. We also discuss the political, economic, cultural, and ethical challenges that data science presents to sustainable development and evaluation. We conclude that data science and evaluation can enhance each other to address the key development challenges of our time.
References
Alber, M., Tepole, A. B., Cannon, W. R., De, S., Dura-Bernal, S., Garikipati, K., . . . Kuhl, E. (2019). Integrating machine learning and multiscale modelling: Perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences. NPJ Digital Medicine, 2(1), 115. https://doi.org/10.1038/s41746-019-0193-y
ARUP. (2019). Digital twin: Towards a meaningful framework. Author.
Asian Development Bank (ADB). (2020). Mapping poverty through data integration and artificial intelligence: A special supplement of the key indicators for Asia and the Pacific. Author.
Ayasse, A. K., Dennison, P. E., Foote, M., Thorpe, A. K., Joshi, S., Green, R. O., . . . Roberts, D. A. (2019). Methane mapping with future satellite imaging spectrometers. Remote Sensing, 11(24), 3054. https://doi.org/10.3390/rs11243054
Bamberger, M., Vaessen, J., & Raimondo, E. (2016). Dealing with complexity in development evaluation: A practical approach. SAGE.
Barricelli, B. R., Casiraghi, E., Gliozzo, J., Petrini, A., & Valtolina, S. (2020). Human digital twin for fitness management. IEEE Access, 8, 26637 – 26664. https://doi.org/10.1109/access.2020.2971576
Befani, B., Ramalingam, B., & Stern, E. (2015). Introduction: Towards systemic approaches to evaluation and impact. IDS Bulletin, 46 (1), 1 – 6. https://doi.org/10.1111/1759-5436.12116
Bertermann, K., Robinson, A., Bamberger, M., Higdon, G. L., & Raftree, L. (2020). Big Data to data science: Moving from “what” to “how” in the MERL Tech Space. MERL Tech.
Bicket, M., Christie, I., Gilbert, N., Penn, A., Hills, D., & Wilkinson, H. (2020). Complexity, and what it means for policy design, implementation, and evaluation. A CECAN Evaluation Policy and Practice Note for Policy Analyst and Evaluators. Note No. 16. Centre for the Evaluation of Complexity across the Nexus.
Blumenstock, J., Cadamuro, G., & On, R. (2015). Predicting poverty and wealth from mobile phone metadata. Science, 350 (6264), 1073 – 1076. https://doi.org/10.1126/science.aac4420. Medline:26612950
BMU, & WRI. (2020). Global dialogue on responding to the COVID-19 pandemic and economic crisis: Building back better aligned to the SDGs and the Paris agreement. Federal Ministry of the Environment, Nature Conservation and Nuclear Safety; and the World Resources Institute. https://reliefweb.int/report/world/global-dialogue-responding-covid-19-pandemic-and-economic-crisis-building-back-better
Broadband Commission for Sustainable Development. (2017). The state of broadband: Broadband catalyzing sustainable development. ITU and UNESCO.
Bruce, K., Gandhi, V. J., & Vandelenotte, J. (2020). Emerging technologies and approaches in monitoring, evaluation, research, and learning for international development programs. MERL Tech.
Bruynseels, K., Santoni de Sio, S., & van den Hoven, J. (2018). Digital twins in healthcare: Ethical implications of an emerging engineering paradigm. Frontiers in Genetics, 9 , 31. https://doi.org/10.3389/fgene.2018.00031. Medline: 29487613
Caffrey, L., & Munro, E. (2017). A systems approach to policy evaluation. Evaluation, 23 (4), 463 – 478. https://doi.org/10.1177/1356389017730727
Carugi, C., & Bryant, H. (2020). A joint evaluation with lessons for the Sustainable Development Goals era: The joint GEF-UNDP evaluation of the small grants programme. American Journal of Evaluation, 41(2), 182 – 200. https://doi.org/10.1177/1098214019865936
CECAN. (2019). Complexity evaluation framework: Recognising complexity and key considerations for complexity-appropriate evaluation in the Department for Environment, Food and Rural Affairs, DEFRA. Commissioned by the Department for Environment, Food and Rural Affairs (Defra) Delivered by CECAN Ltd with Risk Solutions and the Tavistock Institute of Human Relations. https://www.betterevaluation.org/en/resources/complexity-evaluation-framework-recognising-complexity-key-considerations-complexity
Chaplowe, S. G., Hejnowicz, A. P., & Laeubli-Loud, M. (2021). Evaluation as a pathway to transformation: Lessons from Sustainable Development. In A. Nicolaides, S. Eschenbacher, P. Buergelt, Y. Gilpin-Jackson, M. Welch, & M. Misawa (Eds.), Palgrave handbook on learning for transformation. Palgrave Macmillan.
Cohen, J. L. , & Kharas , H. (2018). Using Big Data and artificial intelligence to accelerate global development. Brookings Institution. https://www.brookings.edu/research/using-big-data-and-artificial-intelligence-to-accelerate-global-development/
Cox, J. (2019). How does the commissioning process inhibit the uptake of complexity-appropriate evaluation? Centre for the Evaluation of Complexity across the Nexus.
Delli Paoli, A. & Addeo, F. (2021). Big Data to support Sustainable Development Goals (SDGs). In C. Bevilacqua, F. Calabrò, & L. Della Spina (Eds.), New metropolitan perspectives. Smart innovation, systems and technologies, 178 , 738 – 748. https://doi.org/10.1007/978-3-030-48279-4_69
Deloitte Insights. (2020). Tech trends 2020. Deloitte Development LLC.
Del Río Castro, G., Fernández, M. C. G., & Colsa, A. U. (2021). Unleashing the convergence and digitalization and sustainability towards pursuing the Sustainable Development Goals (SDGs): A holistic review . Journal of Cleaner Production, 280, 122204. https://doi.org/10.1016/j.jclepro.2020.122204
De Nadai, M., Letouzé, E., González, M. C., & Lepri, B. (2018). Characterizing and analyzing urban dynamics in Bogota. AFD Research Paper Series, No. 2018-70.
Elghamrawy, S. M., Darwish, A., & Hassanien, A. E. (2021). Monitoring COVID-19 disease using Big Data and artificial intelligence-driven tools. In A. E. Hassanien & A. Darwish (Eds.), Digital transformation and emerging technologies for fighting COVID-19 pandemic: Innovative approaches. Studies in systems, decision and control, 322, 163 – 174. https://doi.org/10.1007/978-3-030-63307-3_10
Eppel, E. A., & Rhodes, M. L. (2018). Complexity theory and public management: A “becoming” field. Public Management Review, 20(7), 949 – 959. https://doi.org/10.1080/14719037.2017.1364414
Everard, M., Johnston, P., Santillo, D., & Staddon, C. (2020). The role of ecosystems in mitigation and management of Covid-19 and other zoonoses. Environmental Science & Policy, 111, 7 – 17. https://doi.org/10.1016/j.envsci.2020.05.017. Medline:32501392
Fu, B., Wang, S., Zhang, J., Hou, Z., & Li, J. (2019). Unravelling the complexity in achieving the 17 sustainable-development goals. National Science Review, 6(3), 386 – 388. https://doi.org/10.1093/nsr/nwz038
García, O. A., & Kotturi, P. (Eds.). (2020). Information and communication technologies for development evaluation (1st ed.). Routledge Studies in Sustainable Development, Routledge. Taylor & Francis Group.
Gartner Research and Advisory Company. (2020). Gartner Glossary: Digital Twin. https://www.gartner.com/en/information-technology/glossary/digital-twin
Gates, E. F. (2016). Making sense of the emerging conversation in evaluation about systems thinking and complexity science. Evaluation and Program Planning, 59, 62 – 73. https://doi.org/10.1016/j.evalprogplan.2016.08.004. Medline: 27591941
Geyer, R. (2012). Can complexity move UK policy beyond “evidence-based policy making” and the “audit culture”? Applying a “complexity cascade” to education and health policy. Political Studies, 60(1), 20 – 43. https://doi.org/10.1111/j.1467-9248.2011.00903.x
Geyer, R., & Cairney, P. (Eds.). (2015). Handbook on complexity and public policy. Edward Elgar Press.
Ghaffarzadegan, N., Lyneis, J., & Richardson, G. P. (2011). How small system dynamics models can help the public policy process. System Dynamics Review, 27(1), 22 – 44. https://doi.org/10.1002/sdr.442
Global Environment Facility. (2018). Strengthening monitoring and evaluation of climate change adaptation. https://www.thegef.org/publications/strengthening-monitoring-and-evaluation-climate-change-adaptation
Goralski, M. A., & Tan, T. K. (2020). Artificial intelligence and sustainable development. International Journal of Management Education, 18(1), 100330. https://doi.org/10.1016/j.ijme.2019.100330
Guidehouse Insights. (2019). Leveraging digital twin approach for sustainable manufacturing. https://guidehouseinsights.com/reports/leveraging-digital-twin-approach-for-sustainable-manufacturing
Hagendorff, T. (2020). The ethics of AI ethics: An evaluation of guidelines. Minds and Machines, 30 (1), 99 – 120. https://doi.org/10.1007/s11023-020-09517-8
Head, B. W. (2019). Forty years of wicked problems literature: Forging closer links to policy studies. Policy and Society, 38(2), 180 – 197. https://doi.org/10.1080/14494035.2018.1488797
Hejnowicz, A. P., Hartley, S. E., & Gilbert, N. (2018). Build two-way rapport for better policymaking. Nature, 556(7700), 174 . https://doi.org/10.1038/d41586-018-04167-4. Medline:29643493.
Hilbert, M., & Mann, S. (2019). Artificial Intelligence for development: AI4D. In IATT (Ed.), Science-policy briefs on the impact of rapid technological change on the SDGs (pp. 55 - 58). Contributions to the Technology Facilitation Mechanism by individual experts. Interagency Task Team on Science, Technology and Innovation for the Sustainable Development Goals.
Hopson, R., & Cram, F. (Eds.). (2018). Tackling wicked problems in complex ecologies: The role of evaluation. Stanford University Press.
Hummelbrunner, R. (2015). Learning, systems concepts and values in evaluation: Proposal for an exploratory framework to improve coherence. IDS Bulletin , 46 (1), 17 – 29. https://doi.org/10.1111/1759-5436.12118
IATT . (Ed.). (2019). Science-policy briefs on the impact of rapid technological change on the SDGs. Contributions to the technology facilitation mechanism by individual experts. Interagency Task Team on Science, Technology and Innovation for the Sustainable Development Goals.
Jean, N., Burke, M., Xie, M., Davis, W. M., Lobell, D. B., & Ermon, S. (2016). Combining satellite imaging and machine learning to predict poverty. Science, 353(6301), 790 – 794. https://doi.org/10.1126/science.aaf7894. Medline: 27540167
Jones, D., Snider, C., Nassechi, A., Yon, J., & Hicks, B. (2020). Characterising the digital twin: A systematic literature review. CIRP Journal of Manufacturing Science and Technology, 29, 36 – 52. https://doi.org/10.1016/j.cirpj.2020.02.002
Kaur, M. J., Mishra, V. P., & Maheshwari, P. (2020). Th e convergence of digital twin, IoT, and machine learning: Transforming data into action. In M. Farsi, A. Daneshkhah, A. Hosseinian-Far, & H. Jahankhani (Eds.), Digital twin technologies and smart cities (pp. 3 - 17). Springer .
Kong, L., Liu, Z., & Wu, J. (2020). A systematic review of big data-based urban sustainability research: State-of-the-science and future directions. Journal of Cleaner Production, 273, 123142. https://doi.org/10.1016/j.jclepro.2020.123142
Lambert, H., Gupte, J., Fletcher, H., Hammond, L., Lowe, N., Pelling, M. . . . Shanks, K. (2020). Covid-19 as a global challenge: Towards an inclusive and sustainable future. The Lancet Planetary Health, 4(8), e312 – e314. https://doi.org/10.1016/s2542-5196(20)30168-6
Letouzé, E., Areias, A., & Jackson, S. (2016). The evaluation of complex development interventions in the age of Big Data1. In M. Bamberger, J. Vaessen, & E. Raimondo (Eds.), Dealing with complexity in development evaluation (pp. 221 – 250). SAGE.
Letouzé, F., Stock, M., Chiara, F. De., Lizzi, A., & Mazariegos, C. (2019). Harnessing innovation data and technology to measure development effectiveness. Southern Voice.
Levin, K., Cashore, B., Bernstein, S., & Auld, G. (2012). Overcoming the tragedy of super wicked problems: Constraining our future selves to ameliorate global climate change. Policy Sciences, 45(2), 123 – 152. https://doi.org/10.1007/s11077-012-9151-0
Mertens, D. M. (2018). Transformative mixed methods and policy evaluation. Diritto & Questioni Pubbliche, 18(1), 247 – 264.
Morrell, J. A. (2010). Evaluation in the face of uncertainty: Anticipating surprise and responding to the inevitable. The Guilford Press.
Morrell, J. A. (2018). Systematic iteration between model and methodology: A proposed approach to evaluating unintended consequences. Evaluation and Program Planning, 68, 243 – 252. https://doi.org/10.1016/j.evalprogplan.2017.09.006. Medline:28947067
Nara, E. O. B., da Costa, M. B., Basierk, I. C., Schaefer, J. L., Benitez, G. B., Lima do Santos, L. M. A., & Benitez, L. B. (2021). Expected impact of industry 4.0 technologies on sustainable development: A study in the context of Brazil’s plastic industry. Sustainable Production and Consumption, 25, 102 – 122. https://doi.org/10.1016/j.spc.2020.07.018
Nikita, E., van Kemenade, S., Lena, P., Eirini, S., & Stone, Z. (2019). Artificial intelligence for aid NGOs. In IATT (Ed.), Science-policy briefs on the impact of rapid technological change on the SDGs (pp. 15 - 20). Contributions to the Technology Facilitation Mechanism by individual experts. Interagency Task Team on Science, Technology and Innovation for the Sustainable Development Goals.
Nilsson, M., Chisholm, E., Griggs, D., Howden-Chapman, P., McCollum, D., Messerli, P. . . . Stafford-Smith, M. (2018). Mapping interactions between the sustainable development goals: Lessons learned and ways forward. Sustainability Science, 13(6), 1489 – 1503. https://doi.org/10.1007/s11625-018-0604-z. Medline:30546483
Nochta, T., Wan, L., Schooling, J. M., & Parlikad, A. K. (2021). A socio-technical perspective on urban analytics: The case of city-scale digital twins. Journal of Urban Technology, 28(1–2), 263 – 287. https://doi.org/10.1080/10630732.2020.1798177
O’Brien, D. (2014). Cultural value, measurement and policy making. Arts and Humanities in Higher Education, 14(1), 79 – 94. https://doi.org/10.1177/1474022214533892
Ofir, Z., & Rugg, D. (2021). American Journal of Evaluation section on international developments in evaluation: Transforming evaluation for times of global transformation. American Journal of Evaluation, 42(1), 47 – 52. https://doi.org/10.1177/1098214020979070
Ofir, Z., Schwandt, T., D’Errico, S., El-Saddik, K., & Lucks, D. (2016). Five considerations for national evaluation agendas informed by the SDGs. IIED Briefing. International Institute of Environment and Development.
Oughton, E. (2018). Building a digital twin: Testing the effectiveness of telecommunication policies in a virtual world. TPRC 46: The 46th Research Conference on Communication Information and Internet Policy, Washington, D.C., September 21 -22. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3141527
Parsons, B., Dhillon, L., & Keene, M. (2020). Visionary evaluation for a sustainable, equitable future. Information Age.
Patton, M. Q. (2019a). Transformation to global sustainability: Implications for evaluation and evaluators. In G. Julnes (Ed.), Evaluating sustainability: Evaluative support for managing processes in the public interest. New Directions for Evaluation, 162, 103 – 117. https://doi.org/10.1002/ev.20362
Patton, M. Q. (2019b). Blue marble evaluation: Premises and principles (1st ed.). Guilford Press.
Patton, M. Q. (2020). Evaluation criteria for evaluating transformation: Implications for the coronavirus pandemic and the global climate emergency. American Journal of Evaluation, 42(1), 53 – 89. https://doi.org/10.1177/1098214020933689
Picciotto, R. (2020). Evaluation and the Big Data challenge. American Journal of Evaluation, 41(2), 166 – 181. https://doi.org/10.1177/1098214019850334
Pichler, M., Schaffartzik, A., Haberl, H., & Gӧrg, C. (2017). Drivers of society-nature relations in the Anthropocene and their implications for sustainability transformations. Current Opinion in Environmental Sustainability, 26–27, 32 – 36. https://doi.org/10.1016/j.cosust.2017.01.017
Quayle, A., & Kelly, B. (2019). Building informal knowledge-sharing relationships between policy makers and academics: Insights from a PM&C engagement project. Australian Journal of Public Administration, 78(2), 311 – 318. https://doi.org/10.1111/1467-8500.12341
Raftree, L., & Bamberger, M. (2014). Emerging opportunities: Monitoring and evaluation in a tech-enabled world. The Rockefeller Foundation Evaluation Office.
Ramos, G., Hynes, W., Müller, J.-M., & Lees, M. (2019). Systemic thinking for policy making: The potential of systems analysis for addressing global policy challenges in the 21st century. OECD-IIASA.
Rathinam, F., Khatua, S., Siddiqui, Z., Malik, M., Duggal, P., Watson, S., & Vollenweider, X. (2020). Using big data for evaluating development outcomes: A systematic map. CEDIL Methods Working Paper 2. Centre of Excellence for Development Impact and Learning (CEDIL). https://doi.org/10.51744/CMWp2
Rowe, A. (2019). Sustainability-ready evaluation: A call to action. In G. Julnes (Ed.), Evaluating sustainability: Evaluative support for managing processes in the public interest. New Directions for Evaluation, 162, 29 – 48. https://doi.org/10.1002/ev.20365
Saadia, D. (2021). Integration of cloud computing, big data, artificial intelligence, and internet of things: Review and open research issues. International Journal of Web-Based Learning and Teaching Technologies, 16(1), 10 – 17. https://doi.org/10.4018/ijwltt.2021010102
Schwab, K. (2021, March 23). The fourth industrial revolution. Encyclopedia Britannica. https://www.britannica.com/topic/The-Fourth-Industrial-Revolution-2119734
Schwandt, T. A., Ofir, Z., Lucks, D., El-Saddik, K., & D’Errico, S. (2016). Evaluation: A crucial ingredient for SDG success: An IIED briefing. International Institute for Environment and Development.
Schwarz, B., Tellman, B., Sullivan, J., Kuhn, C., Mahtta, R., Pandey, B. . . . Pestre, G. (2017). Socio-physical vulnerability to flooding in Senegal: An exploratory analysis with new data and Google Earth Engine. AFD Technical Report No. 2017-25. https://www.afd.fr/en/ressources/socio-physical-vulnerability-flooding-senegal
Slingshot Simulations. (2020). https://www.slingshotsimulations.co.uk/digital-twins/digital-twin-series-coming-soon/
Statistia.com. (2021). Volume of data/information created, captured, copied, and consumed worldwide from 2010 to 2025. https://www.statista.com/statistics/871513/worldwide-data-created/
Steffen, W., Rockström, J., Richardson, K., Lenton, T. M., Folke, C., Liverman, D. . . . Schellnhuber, H. J. (2018). Trajectories of the earth system in the Anthropocene. Proceedings of the National Academy of Sciences, 115(33), 8252 – 8259. https://doi.org/10.1073/pnas.1810141115. Medline:30082409
Sterner, T., Barbier, E. B., Bateman , I. , van den Bijgaart , I. , Crépin , A.-S. , Edenhofer , O. . . . Robinson , A. (2019). Policy design for the Anthropocene. Nature Sustainability, 2(1),14 – 21. https://doi.org/10.1038/s41893-018-0194-x
Talbot, C., & Talbot, C. (2015). Bridging the academic-policy-making gap: Practice and policy issues. Public Money & Management, 35(3), 187 – 194. https://doi.org/10.1080/09540962.2015.1027491
Truby, J. (2020). Governing artificial intelligence to benefit the UN Sustainable Development Goals. Sustainable Development, 28(4), 946 – 959. https://doi.org/10.1002/sd.2048
2030Vision—Global Goals Technology Forum. (2019). AI & the Sustainable Development Goals: The state of play. SustainAbility.
United Nations (UN). (2012). Future we want: Outcome document. Author. https://sustainabledevelopment.un.org/rio20/futurewewant
United Nations (UN). (2015). Transforming our world: The 2030 agenda for sustainable development. United Nations General Assembly, Seventh Session, A/RES/70/1. Author. https://sustainabledevelopment.un.org/post2015/transformingourworld/publication
United Nations Conference on Trade and Development (UNCTAD). (2018). The “fourth industrial revolution” can power sustainable development, if we get it right. https://unctad.org/news/fourth-industrial-revolution-can-power-sustainable-development-if-we-get-it-right
United Nations Economics and Social Commission for Asia and the Pacific, (UN ESCAP), & Google. (2019). Artificial intelligence in the delivery of public services. Author. https://www.unescap.org/publications/artificial-intelligence-delivery-public-services
United Nations Independent Expert Advisory Group (UN IEAG). (2014). A world that counts: Mobilising the data revolution for sustainable development. A report prepared at the Request of the United Nations Secretary-General, by the Independent Expert Advisory Group on a Data Revolution for Sustainable Development. https://www.undatarevolution.org/wp-content/uploads/2014/11/A-World-That-Counts.pdf
United Nations Research Institute for Social Development (UNRISD). (2016). Policy innovations for transformative change: UNRISD flagship report 2016. Author.
United States Agency for International Development (USAID). (2019). Reflecting the past, shaping the future: Making AI work for international development. Author.
van den Berg, R. D., Magro, C., & Mulder, S. S. (Eds.). (2019). Evaluation for transformational change: Opportunities and challenges for the sustainable development goals. International Development Evaluation Association (IDEAS). https://ideas-global.org/wp-content/uploads/2019/11/2019-11-05-Final_IDEAS_EvaluationForTransformationalChange.pdf
van Soest, H. L., van Vuuren, D. P., Hilaire, J., Minx, J. C., Harmsen, M. J. H. M., Krey ,V. . . . Luderer, G. (2019). Analysing interactions among Sustainable Development Goals with Integrated Assessment Models. Global Transitions, 1, 210 – 225. https://doi.org/10.1016/j.glt.2019.10.004
Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S. . . . Nerini, F. F. (2020). The role of artificial intelligence in achieving the Sustainable Development Goals. Nature Communications, 11(1), 233. https://doi.org/10.1038/s41467-019-14108-y. Medline:31932590
Vogel, I. (2012). Review of the use of “theory of change” in international development. Review Report for the UK Department of International Development.
Waddock, S., Waddell, S., Goldstein, B., Linner, B., Schapke, N., & Vogel, C. (2020). Transformation: How to spur radical change. In A. Scrutton (Ed.), Our future on earth report (pp. 82 – 90). Future Earth.
Widianingsih, I., & Mertens, D. M. (2019). Transformative research and the Sustainable Development Goals: Challenges and a vision from Bandung, West Java. International Journal of Transformative Research, 6(1), 27 – 35. https://doi.org/10.1515/ijtr-2019-0005
Williams, B. (2015). Prosaic or profound? The adoption of systems ideas by impact evaluation. IDS Bulletin, 46(1), 7 – 16. https://doi.org/10.1111/1759-5436.12117
Williams, B., & Hummelbrunner, R. (2010). Systems concepts in action: A practitioner’s toolkit. Stanford Business Books. Stanford University Press.
World Bank. (2015). World development report 2016: Digital dividends overview. A World Bank Group Flagship Report. International Bank for Reconstruction and Development/The World Bank.
World Commission on Environment and Development (WCED). (1987). Our common future. Oxford University Press.
Yayboke, E., Nealer, E., & Rice, C. (2017). Harnessing the data revolution to achieve the Sustainable Development Goals: Enabling frogs to leap. Centre for Strategic and International Studies. https://www.csis.org/analysis/harnessing-data-revolution-achieve-sustainable-development-goals
York, P., & Bamberger, M. (2020). Measuring results and impact in the age of big data: The nexus of evaluation, analytics, and digital technology. The Rockefeller Foundation.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Adam Hejnowicz, Scott Chaplowe
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
