Expert Panels in Evaluation
An Update From the Field Using the DATA Model
DOI:
https://doi.org/10.3138/cjpe-2023-0047Keywords:
evaluation, policy capacity, political science, practitioner, public administration, public policy, qualitative researchAbstract
In this practice note, the authors reflect on the use and utility of expert panels in evaluation. They apply the describe, analyze, theorize, act model using interviews with evaluators, insights from peer-reviewed literature and their own professional observations. Connections are made to larger evaluation discourses regarding reflection, expert opinion, expertise, epistemic authority, and lived experience. It was found that expert panels are generally underutilized in evaluation due to a lack of awareness among evaluators as well as the perceived complexity associated with this method. However, the literature and interviews were clear that, when managed properly, expert panels can add tremendous value to an evaluation. There is therefore merit for more seriously considering panels in future evaluations. This note provides recommendations for evaluators and the evaluation community at large.
References
Abbatangelo-Gray, J., Cole, G., & Kennedy, M. (2007). Guidance for evaluating mass communication health initiatives. Evaluation & the Health Professions, 30(3), 229–253.
Althaus, C. (2020). Different paradigms of evidence and knowledge: Recognising, honouring, and celebrating Indigenous ways of knowing and being. Australian Journal of Public Administration, 79(2), 187–207.
Anderson, E. (1995). Knowledge, human interests, and objectivity in feminist epistemology. Philosophical Topics, 23(2), 27–58.
Averch, H. (2004). Using expert judgement. In J. Wholey, H. Hatry, & K. Newcomer (Eds.), Handbook of practical program evaluation (2nd ed., pp. 292–309). Wiley & Sons.
Azevedo, L., Shi, W., Medina, P., & Bagwell, M. (2020). Examining junior faculty work–life balance in public affairs programs in the United States. Journal of Public Affairs Education, 26(4), 416–436.
Bamberger, M. (2016). The importance of a mixed methods approach for evaluating complexity. In M. Bamberger, J. Vaessen, & E. Raimondo (Eds.), Dealing with complexity in development evaluation: A practical approach (pp. 144–163). Sage Publications.
Baxter, B. (1996). Consultancy expertise: A post-modern perspective. In H. Scarbrough (Ed.), The management of expertise. Management, work and organisations (pp. 66–92). Palgrave.
Bechar, S., & Mero-Jaffe, I. (2013). Who is afraid of evaluation? Ethics in evaluation research as a way to cope with excessive evaluation anxiety. American Journal of Evaluation, 35(3), 364–376.
Bédard, J., & Chi, M. (1992). Expertise. Current Directions in Psychological Science, 1(4), 135–139.
Berg, M., & Seeber, B. (2016). The slow professor: Challenging the culture of speed in the academy. University of Toronto Press.
Bertrand, F., & Radford, K. (2022, June 13–15). Using expert review committees increase diversity and utilization! The experience of the National Research Council of Canada [Paper presentation]. Canadian Evaluation Society Annual Conference, Winnipeg, Manitoba, Canada.
BetterEvaluation. (n.d.-a). Advisory group. Retrieved September 14, 2023, from https://www.betterevaluation.org/methods-approaches/methods/advisory-group
BetterEvaluation. (n.d.-b). Check results match expert predictions. Retrieved September 14, 2023, from https://www.betterevaluation.org/methods-approaches/methods/check-results-match-expert-predictions
Bozeman, B. (1993). Peer review and evaluation of R&D impacts. In B. Bozeman & J. Melkers (Eds.), Evaluating R&D impacts: Methods and practice (pp. 79–98). Springer.
Bremner, L. (2019). Creating new stories: The role of evaluation in truth and reconciliation. Canadian Journal of Program Evaluation, 34(2), 331–342.
Bruijn, J., & Heuvelhof, E. (1999). Scientific expertise in complex decision-making processes. Science and Public Policy, 26(3), 179–184.
Cameron, B. (2022). Leadership, change and conflict: An examination of informal human resources theory for policy capacity. Canadian Political Science Review, 16(1), 42–59.
Cameron, B. (2023). Building capacity with evaluation standards and guidelines in Prince Edward Island: Responding to academics’ “call to action”. Canadian Journal of Program Evaluation, 38(1), 79–98.
Canadian Evaluation Society. (2018). Competencies for Canadian evaluation practice. https://evaluationcanada.ca/files/pdf/2_competencies_cdn_evaluation_practice_2018.pdf
Canadian Food Inspection Agency. (2018). Evaluation of the CFIA’s meat programs. Government of Canada. https://inspection.canada.ca/about-cfia/transparency/corporate-management-reporting/audits-reviews-and-evaluations/evaluation-of-the-meat-programs/final-report/eng/1517982099910/1517983124769
Canadian Heritage. (2017). Horizontal evaluation of the Roadmap for Canada’s Official Languages 2013–2018: Education, immigration, communities and evaluation of the Coordination Program. Government of Canada. https://www.canada.ca/en/canadian-heritage/corporate/publications/evaluations/official-languages-coordination-roadmap-2013-2018.html
Canadian Heritage. (2022a). Evaluation of the Aboriginal Peoples’ Program 2014–15 to 2018–19. Government of Canada. https://www.canada.ca/en/canadian-heritage/corporate/publications/evaluations/evaluation-aboriginal-peoples-program-2014-2019.html
Canadian Heritage. (2022b). Evaluation of the Official Languages Support Programs 2003–04 to 2020–21. Government of Canada. https://www.canada.ca/en/canadian-heritage/corporate/publications/evaluations/evaluation-official-languages-2003-2021.html
Canadian Institutes of Health Research. (2020). Vanier Canada Graduate Scholarships Program evaluation. Government of Canada. https://cihr-irsc.gc.ca/e/52497.html
Carton, G., & Ungureanu, P. (2018). Bridging the research–practice divide: A study of scholar-practitioners’ multiple role management strategies and knowledge spillovers across roles. Journal of Management Inquiry, 27(4), 436–453.
Cerovac, I. (2016). The role of experts in a democratic society. Journal of Education Culture and Society, 7(2), 75–88.
Collins, H., & Evans, R. (2007). Rethinking expertise. University of Chicago Press.
Coryn, C., & Scriven, M. (2008). Editors’ notes. New Directions for Evaluation, 2008(118), 1–5.
Cozzens, S. (1987). Expert review in evaluating programs. Science and Public Policy, 14(2), 71–81.
de Muijnck, W. (2011). Normative authority for empirical science. Philosophical Explorations, 14(3), 263–275.
Dietz, T., & Pfund, A. (1988). An impact identification method for development program evaluation. Impact Assessment, 6(3–4), 137–145.
Dighe, S., & Matthias, C. (2023). Deconstructing the Imperial Episteme: Decolonizing Knowledge Production in Program Evaluation. Journal of MultiDisciplinary Evaluation, 19(44), 117–130.
Donaldson, S., Gooler, L., & Scriven, M. (2002). Strategies for managing evaluation anxiety: Toward a psychology of program evaluation. The American Journal of Evaluation, 23(3), 261–273.
Douglas, H. (2008). The role of values in expert reasoning. Public Affairs Quarterly, 22(1), 1–18.
Employment and Social Development Canada. (2007). Interim evaluation of the National Literacy Program. Government of Canada. https://publications.gc.ca/site/eng/323742/publication.html
Employment and Social Development Canada. (2017a). Evaluation of learning and labour market information as disseminated by Employment and Social Development Canada using a web-based consolidated approach. Government of Canada. https://www.canada.ca/en/employment-social-development/corporate/reports/evaluations/learning-labour-information-web-approach.html
Employment and Social Development Canada. (2017b). Summative evaluation of the Canada pension plan—Retirement pension and survivor benefits. Government of Canada. https://www.canada.ca/en/employment-social-development/corporate/reports/evaluations/2016-summative-cpp-retirement-pension-survivor-benefits.html
Employment and Social Development Canada. (2020). Evaluation of the Foreign Credential Recognition Program. Government of Canada. https://www.canada.ca/en/employment-social-development/corporate/reports/evaluations/foreign-credential-recognition-program.html
Employment and Social Development Canada. (2022). Evaluation of the Enabling Accessibility Fund—2016 to 2021. Government of Canada. https://www.canada.ca/en/employment-social-development/corporate/reports/evaluations/2016-2021-enabling-accessibility-fund.html
Feinstein, D. (2011). Editor’s note: Thoughts on experts—From a non-expert perspective. Antitrust, 25(3), 5–7.
Firestone, M. (2020). Indigenous health service: Principles and guidelines from a provincial “three ribbon” expert panel. Canadian Journal of Program Evaluation, 34(3).
Fulton, R. (1991). Getting ideas into print: Some tips for practitioners. New Directions for Adult and Continuing Education, 51, 49–58.
Geva-May, I., & Thorngate, W. (2003). Reducing anxiety and resistance in policy and programme evaluations. Evaluation, 9(2), 205–227.
Gibson, R., & Dale, J. (2023). Do more, with less: The realities of local government in rural Ontario. The Canadian Geographer, 67(1), 176–187.
Gill, R. (2003). Change management—or change leadership? Journal of Change Management, 3(4), 307–318.
Goldman, A. (2016). Expertise. Topoi, 37(1), 3–10.
Goodyear, L. (2011). Building a community of evaluation practice within a multisite program. New Directions for Evaluation, 2011(129), 97–105.
Grandisson, M., Hébert, M., & Thibeault, R. (2016). Practice guidelines for program evaluation in community-based rehabilitation. Disability and Rehabilitation, 39(12), 1243–1251.
Grundmann, R. (2023). Making sense of expertise: Cases from law, medicine, journalism, Covid-19, and climate change. Routledge.
Harding, S. (1991). Whose science? Whose knowledge? Thinking from women’s lives. Cornell University Press.
Herling, R. (2000). Operational definitions of expertise and competence. Advances in Developing Human Resources, 2(1), 8–21.
Hewitt, K., Tovey, T., Hall, C., & Weikart, C. (2022). Improving a rural partnership-based leadership preparation program using the DATA model. International Journal of Educational Leadership Preparation, 17(1), 189–202. https://www.icpel.org/uploads/1/5/6/2/15622000/ijelp_volume_17_number_1_spring_2022.pdf
Holst, C., & Molander, A. (2019). Epistemic democracy and the role of experts. Contemporary Political Theory, 18(4), 541–561.
Hotvedt, R., Lossius, H., Kristiansen, I., Steen, P., Søreide, E., & Førde, O. (2003). Are expert panel judgements of medical benefits reliable? An evaluation of emergency medical support programs. International Journal of Technology Assessment in Health Care, 19(1), 158–167.
Immigration, Refugees and Citizenship Canada. (2020). Evaluation of language training services. Government of Canada. https://www.canada.ca/en/immigration-refugees-citizenship/corporate/reports-statistics/evaluations/eval-language-training-services.html
Janack, M. (1997). Standpoint epistemology without the “standpoint”?: An examination of epistemic privilege and epistemic authority. Hypatia, 12(2), 125–139.
Janssen, M., & Estevez, E. (2013). Lean government and platform-based governance—Doing more with less. Government Information Quarterly, 30(Suppl. 1), 1–8.
Johnson, T. (1993). Expertise and the state. In M. Gane & T. Johnson (Eds.), Foucault’s new domains (pp. 139–152). Routledge.
Justice Canada. (2012). Initiative in support of access to justice in both official languages evaluation. Government of Canada. https://www.justice.gc.ca/eng/rp-pr/cp-pm/eval/rep-rap/12/bole-edlo/toc-tdm.html
Kallemeyn, L., Roseveare, C., Steinberg, M., & Tovey, T. (2021). What does implementing case‐centered teaching of evaluation look like in practice? New Directions for Evaluation, 2021(172), 69–84.
Kandiuk, M. (2015, August 12–14). Building success in research and scholarship through peer mentoring [Paper presentation]. IFLA WLIC 2015, Cape Town, South Africa. https://library.ifla.org/id/eprint/1918/1/S02-2015-kandiuk-en.pdf
Kitchenham, B., Turner, M., Budgen, D., Linkman, S., Charters, S., & Brereton, P. (2007). Large-scale software engineering questions—expert opinion or empirical evidence? IET Software, 1(5), 161–171.
Kocaoglu, D. (1983). A participative approach to program evaluation. IEEE Transactions on Engineering Management, 30(3), 112–118.
Krick, E. (2015). Negotiated expertise in policy-making: How governments use hybrid advisory committees. Science and Public Policy, 42(4), 487–500.
Kruglanski, A. (1989). Lay epistemics and human knowledge: Cognitive motivational bases. Springer Science+Business Media.
Lachapelle, E., Montpetit, É., & Gauvin, J. (2014). Public perceptions of expert credibility on policy issues: The role of expert framing and political worldviews. Policy Studies Journal, 42(4), 674–697.
Larrick, R., & Feiler, D. (2015). Expertise in decision making. In G. Keren & G. Wu (Eds.), The Wiley Blackwell handbook of judgment and decision making, II. Wiley & Sons.
Lau, F. (2019). Cultivating eHealth evaluation leadership. Healthcare Management Forum, 32(4), 192–196.
Lawrenz, F., & Huffman, D. (2006). Methodological pluralism: The gold standard of STEM evaluation. New Directions for Evaluation, 2006(109), 19–34.
Lawrenz, F., Thao, M., & Johnson, K. (2012). Expert panel reviews of research centers: The site visit process. Evaluation and Program Planning, 35(3), 390–397.
Lee, C., Reissing, E., & Dobson, D. (2009). Work-life balance for early career Canadian psychologists in professional programs. Canadian Psychology, 50(2), 74–82.
Liao, Y. (2021). The legitimacy crisis of arguments from expert opinion: Can’t we trust experts? Argumentation, 35, 265–286.
Logsdon, J., & Rubin, C. (1988). Research evaluation activities of ten federal agencies. Evaluation and Program Planning, 11(1), 1–11.
Luhmann, N., & Collins, H. (2011). The hierarchy of expertise. In N. Stehr & R. Grundmann (Eds.), Experts: The knowledge and power of expertise (pp. 61–92). Routledge.
McKegg, K. (2019). White privilege and the decolonization work needed in evaluation to support Indigenous sovereignty and self-determination. Canadian Journal of Program Evaluation, 34(2), 357–367.
McLean, R., Graham, I., Bosompra, K., Choudhry, Y., Coen, S., MacLeod, M., Manuel, C., McCarthy, R., Mota, A., Peckham, D., Tetroe, J., & Tucker, J. (2012). Understanding the performance and impact of public knowledge translation funding interventions: Protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs. Implementation Science, 7(1).
Mertens, D. (2023). Research and evaluation in education and psychology: Integrating diversity within quantitative, qualitative, and mixed methods (6th ed.). Sage Publications.
Miciak, M., Lavoie, M., & Barrington, G. (2021). Reflective practice: Moving intention into action. Canadian Journal of Program Evaluation, 36(1), 95–105.
Mizrahi, M. (2018). Arguments from expert opinion and persistent bias. Argumentation, 32(2), 175–195.
Nakamura, G., Soares, B., Pillar, V., Alexandre, J., & Duarte, L. (2023). Three pathways to better recognize the expertise of Global South researchers. NPJ Biodiversity, 2(1), 1–17.
Natural Resources Canada. (2016). Evaluation report: EcoENERGY for alternative fuels program. Government of Canada. https://natural-resources.canada.ca/maps-tools-publications/publications/minerals-mining-publications/evaluation-report-ecoenergy-alternative-fuels-program/18646
Nevo, D. (1985, March 31–April 4). Experts’ opinion: A powerful evaluation tool [Paper presentation]. 69th Annual Meeting of the American Educational Research Association, Chicago, Illinois, United States. https://eric.ed.gov/?id=ED258994
Nevo, D. (1989). Expert opinion in program evaluation. New Directions for Program Evaluation, 1989(42), 85–93.
Nielsen, S., Lemire, S., & Montague, S. (2023). Remembering John Mayne—A practical thinker and a thinking practitioner. Canadian Journal of Program Evaluation, 37(3), 297–305.
National Research Council Canada. (2018a). Evaluation of NRC’s Energy, Mining and Environment Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-nrcs-energy-mining-environment-research-centre
National Research Council Canada. (2018b). Evaluation of NRC’s Digital Technologies Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-nrcs-digital-technologies-research-centre
National Research Council Canada. (2018c). Evaluation of NRC’s Security and Disruptive Technologies Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-nrcs-security-disruptive-technologies-research-centre-evaluation-report-summary
National Research Council Canada. (2019). Evaluation of the NRC’s Medical Devices Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-nrcs-medical-devices-research-centre
National Research Council Canada. (2020a). Evaluation of the Aerospace Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-aerospace-research-centre
National Research Council Canada. (2020b). Evaluation of the Automotive and Surface Transportation Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-automotive-surface-transportation-research-centre
National Research Council Canada. (2020c). Evaluation of the Human Health Therapeutics Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-human-health-therapeutics-research-centre
National Research Council Canada. (2021). Evaluation of the Herzberg Astronomy and Astrophysics Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-herzberg-astronomy-astrophysics-research-centre
National Research Council Canada. (2022a). Evaluation of NRC’s Aquatic and Crop Resource Development Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-nrcs-aquatic-crop-resource-development-research-centre
National Research Council Canada. (2022b). Evaluation of the Nanotechnology Research Centre. Government of Canada. https://nrc.canada.ca/en/corporate/planning-reporting/evaluation-nanotechnology-research-centre
Obrecht, M. (1999). Distilling stakeholder input for program evaluation priority setting. The Canadian Journal of Program Evaluation, 14(2), 119–134.
Organisation for Economic Co-operation and Development. (2014). Compendium of good practices for integrity in public procurement (GOV/PGC/ETH(2014)2/REV1). https://www.oecd.org/gov/public-procurement/compendium-for-good-practices-for-integrity-in-public-procurement.pdf
Palinkas, L., Mendon, S., & Hamilton, A. (2019). Innovations in mixed methods evaluations. Annual Review of Public Health, 40, 423–442.
Pallas, A. (2001). Preparing education doctoral students for epistemological diversity. Educational Researcher, 30(5), 1–6.
Peters, J. (1991). Strategies for reflective practice. New Directions for Adult and Continuing Education, 1991(51), 89–96.
Peters, J. (1999, July). The DATA-DATA model. Paper presented at the International Human Science Research Conference, Sheffield, UK.
Peters, J. (2002, November 6–8). Action research: An approach called DATA-DATA [Paper presentation]. Mid-South Educational Research Association Annual Meeting, Chattanooga, Tennessee, United States. https://eric.ed.gov/?id=ED471300
Peters, J. (2009). DATA-DATA: A model for practitioner-researchers. International Journal of Interdisciplinary Social Sciences, 4(3), 147–158.
Pierson, R. (1994). The epistemic authority of expertise. PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, 1994, 398–405.
Roberts, G. (2010). Western epistemology: A stranger in a strange land? Pacific Health Dialogue, 16(2), 112–118.
Robey, D., Taylor, W., & Grabowski, L. (2018). Pragmatic rigor: Principles and criteria for conducting and evaluating practitioner scholarship. Engaged Management ReView, 2(3).
Schinker, R. (2005). Managing extreme evaluation anxiety through nonverbal communication. Journal of Multidisciplinary Evaluation, 2(2), 76–80.
Schwandt, T., & Burgon, H. (2006). Evaluation and the study of lived experience. In I. Shaw, J. Greene & M. Mark (Eds.), The SAGE handbook of evaluation: Policies, programs and practices (pp. 98–117). Sage Publications.
Scott, S. (2008). Drawing on Indigenous ways of knowing: Reflections from a community evaluator. The Canadian Journal of Program Evaluation, 23(2), 73–92.
Sechrest, L., Babcock, J., & Smith, B. (1993). An invitation to methodological pluralism. Evaluation Practice, 14(3), 227–235.
Selinger, E., & Crease, R. (Eds.). (2006). The philosophy of expertise. Columbia University Press.
Shepherd, R. (2020). Evaluation in Indigenous contexts: An introduction to practice. Canadian Journal of Program Evaluation, 34(3).
Sinduja, J., & Priyadharshini, R. (2023, March 7–8). Comparison of exposure to various research methodologies, willingness, and involvement of undergraduate students in research projects between Indian and international students—A KAP survey [Paper presentation]. International Conference on Business Analytics for Technology and Security (ICBATS), Dubai, United Arab Emirates. https://ieeexplore.ieee.org/abstract/document/10111443
Smith, T., Barlow, P., Peters, J., & Skolits, G. (2015). Demystifying reflective practice: Using the DATA model to enhance evaluators’ professional activities. Evaluation and Program Planning, 52, 142–147.
Staudt, M., Dulmus, C., & Bennett, G. (2003). Facilitating writing by practitioners: Survey of practitioners who have published. Social Work, 48(1), 75–83.
Stufflebeam, D. (2000). The CIPP model for evaluation. In D. Stufflebeam, C. Madam, & T. Kellaghan (Eds.), Evaluation models: Viewpoints on educational and human services evaluation (pp. 279–317). Springer.
Tovey, T., & Archibald, T. (2023). The relationship between reflective practice, evaluative thinking and practical wisdom. In M. Hurteau & T. Archibald (Eds.), Practical wisdom for an ethical evaluation practice (pp. 87–101). Information Age Publishing.
Tovey, T., & Skolits, G. (2021). Conceptualizing and engaging in reflective practice: Experienced evaluators’ perspectives. American Journal of Evaluation, 43(1), 5–25.
United States National Research Council. (2007). Prospective evaluation of applied energy research and development at DOE (phase two). The National Academic Press. https://nap.nationalacademies.org/read/11806/chapter/1
Valentino, A. (2021). Applied behavior analysis: Research made easy. New Harbinger Publications.
Vázquez, E., Kim, M., & Santaella, M. (2023). Lived experience experts: A name created by us for us. Expert Review of Hematology, 16(Suppl. 1), 7–11.
Voegeli, C., Schmitt-Boshnick, M., & Krupa, E. (2021). Evaluation advisory groups: Considerations for design and management. Canadian Journal of Program Evaluation, 36(2), 227–235.
Voronka, J. (2016). The politics of “people with lived experience”: Experiential authority and the risks of strategic essentialism. Philosophy, Psychiatry, & Psychology, 23(3–4), 189–201.
Wagenaar, W., & Keren, G. (1986). Does the expert know? The reliability of predictions and confidence ratings of experts. In E. Hollnagel, G. Mancini, & D. D. Woods (Eds.), Intelligent decision support in process environments (NATO ASI Series, Vol. 21, pp. 87–103). Springer.
Welch, J. (2009). Interdisciplinarity and the history of Western epistemology. Issues in Integrative Studies, 27, 35–69. http://hdl.handle.net/10323/4447
Wolfenden, H., Sercombe, H., & Tucker, P. (2019). Making practice publishable: What practice academics need to do to get their work published, and what that tells us about the theory–practice gap. Social Epistemology, 33(6), 555–573.
Yates, J., & Tschirhat, M. (2006). Decision-making expertise. In K. Ericsson, N. Charness, P. Feltovich, & R. Hoffman (Eds.), Cambridge handbook of expertise and expert performance (pp. 421–438). Cambridge University Press.
Yedidia, M., Chou, J., Brownlee, S., Flynn, L., & Tanner, C. (2014). Association of faculty perceptions of work-life with emotional exhaustion and intent to leave academic nursing: Report on a national survey of nurse faculty. The Journal of Nursing Education, 53(10), 569–579.
Zagzebski, L. (2012). Epistemic authority: A theory of trust, authority, and autonomy in belief. Oxford University Press.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Bobby Thomas Cameron, Clinton Hynes
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
