Employing Mixed-Methods Citation Analysis to Investigate Transnational Influence in Evaluation Theory
DOI:
https://doi.org/10.3138/cjpe-2023-0009Keywords:
citation analysisAbstract
Mixed-methods citation analysis can be used to conduct research on evaluation (RoE) studies with a cross-country focus. In particular, this RoE study employs mixed-methods citation analysis to investigate the influence of an evaluation theorist’s concepts transnationally. We use Jennifer Greene’s program evaluation theories as a case example for observing the imprint of a theorists’ concepts on evaluators theorizing and practicing outside of the United States. Conducting this case analysis highlights the methodological affordances and shortcomings of citation analysis. The article addresses equity and diversity considerations in using citation analysis, particularly for investigating multidirectional transnational influences in evaluation theory.
References
Alkin, M. C. (2013). Evaluation roots: A wider perspective of theorists’ views and influences, second edition. Thousand Oaks, CA: SAGE.
Alkin, M. C., & King, J. A. (2016). The historical development of evaluation use. American Journal of Evaluation, 37(4), 568–579.
Alkin, M. C., & Christie, C. A. (2019). Theorists’ models in action: A second look. New Directions for Evaluation, 2019(163), 11–18.
Antell, K., Strothmann, M., Chen, X., & O’Kelly, K. (2013). Cross-examining Google scholar. Reference & User Services Quarterly, 52(4), 279–82.
Cakici, H. (2016). Getting to the roots of evaluation capacity building in the Global South: Multiple streams model to frame the agenda status of evaluation in Turkey. Canadian Journal of Program Evaluation, 30(3), 277–295.
Cavino, H. M. (2013). Across the colonial divide: Conversations about evaluation in Indigenous contexts. American Journal of Evaluation 34(3): 339–355.
Chilisa, B. Major, T. E., Gaotlhobogwe, M., & Mokgolodi, H. (2015). Decolonizing and Indigenizing evaluation practice in Africa: Toward African relational evaluation approaches. Canadian Journal of Program Evaluation, 30(3): 313–328.
Christie, C. A. (2011). Advancing empirical scholarship to further develop evaluation theory and practice. Canadian Journal of Program Evaluation, 26(1), 1–18.
Coryn, C., Wilson, L., Westine, C., Hobson, K., Ozeki, S., Fiekowsky, E., … Schröter, D. (2017). A decade of research on evaluation: A systemic review of research on evaluation published between 2005 and 2014. American Journal of Evaluation, 38(3), 329–347.
Donovan, C. (2011). State of the art in assessing research impact: introduction to a special issue. Research Evaluation, 20(3), 175–179.
Emerson, R. O. (2020). Evaluation use and evaluator competencies in low-income countries: A three article dissertation (PhD Thesis). University of Massachusetts, Lowell.
Erfanmanesh, M., Tahira, M., & Abrizah, A. (2017). The publication success of 102 nations in Scopus and the performance of their Scopus-indexed journals. Publishing Research Quarterly, 33(4), 421–432.
EvalPartners. (2016). Executive summary: Global evaluation agenda 2016–2020. Retrieved from https://www.evalpartners.org/sites/default/files/documents/ExecutiveSummary.pdf
Gildemyn, M. (2014). Understanding the influence of independent civil society monitoring and evaluation at the district level. American Journal of Evaluation, 35(4), 507–524.
Goodnight, M. R. (2022). “We went in favor of rebellion”: The decisions that made India’s Annual Status of Education Report (ASER). International Journal of Educational Development, 94.
Goodnight, M. R. (2023). Research on evaluation influence in India: Theorizing beyond process and results to design. American Journal of Evaluation, 44(4), 629–648.
Goodnight, M. R., & Avent, C. M. (2023). In pursuit of democratic values: Transnational influences on Jennifer Greene. In J. N. Hall, A. Boyce, & R. Hopson (Eds.), Disrupting program evaluation and mixed methods research for a more just society: The contributions of Jennifer C. Greene (pp. 165–180). Information Age Publishing.
Goodnight, M. R., Avent, C. M., & Kumaran, R. (2023). Tracing the transnational influence of Jennifer Greene’s ideas: Research on evaluation theory across countries. Journal of MultiDisciplinary Evaluation 19(45), 81–98.
Greene, J. C. (1997). Evaluation as advocacy. American Journal of Evaluation, 18(1), 25–35.
Greene, J. C. (2012). Values-engaged evaluation. In M. Sergone (Ed.), Evaluation for Equitable Development Results (pp. 192–207) UNICEF.
Greene, J. C. (2015a) Culture and evaluation: From a transcultural belvedere. In S. Hood, R. Hopson, & H. Frierson (Eds.), Continuing the Journey to Reposition Culture and Cultural Context in Evaluation Theory and Practice (pp. 91–107). Charlotte: Information Age Publishing.
Greene, J. C. (2015b). The emergence of mixing methods in the field of evaluation. Qualitative Health Research, 25(6), 746–750.
Greene, J. C. (2016). Advancing equity: Cultivating an evaluation habit. In S. Donaldson & R. Picciotto (Eds.), Evaluation for an equitable society (pp. 49–66). Charlotte: Information Age Publishing.
Greenseid, L. O., & Lawrenz, F. (2011). Using citation analysis methods to assess the influence of science, technology, engineering, and mathematics education evaluations. American Journal of Evaluation, 32(3), 392–407.
Hall, J. N., Boyce, A. S., & Hopson, R. K. (2023). Disrupting program evaluation and mixed methods research for a more just society: The contributions of Jennifer C. Greene (Evaluation and Society Series). Charlotte: Information Age Publishing.
Hall, M. (2004, November) Persuasive language, responsive design: Towards broader evaluation utilization. In K. Kirkhart (Chair), What is the nature of evaluation use/influence? American Evaluation Association conference, Atlanta, GA.
Heberger, A. E., Christie, C. A., & Alkin, M. C. (2010). A bibliometric analysis of the academic influences of and on evaluation theorists’ published work. American Journal of Evaluation, 31(1), 24–44.
Henry, G. T., & Mark, M. M. (2003). Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation, 24(3), 293–314.
Hood, S. (1998). Responsive evaluation Amistad Style: Perspectives of one African American evaluator. In R. Davis (ed.), Proceedings of the stake symposium on educational evaluation. Urbana-Champaign: University of Illinois.
Hood, S. (2001). Nobody knows my name: In praise of African American evaluators who were responsive. In J. C. Greene & T. Abma (Eds.), Responsive evaluation. New directions for evaluation (pp. 31–44).
Kirkhart, K. E. (2000). Reconceptualizing evaluation use: An integrated theory of influence. In V. J. Caracelli & H. Preskill (Eds.), The expanding scope of evaluation use (pp. 5–23). New Directions for Evaluation.
Kirkhart, K. E. (2010) Eyes on the prize: Multicultural validity and evaluation theory. American Journal of Evaluation, 31(3), 400–413.
Kirkhart, K. E. (2011). Culture and influence in multisite evaluation. In J. King & F. Lawrenz (Eds.), Multisite evaluation practice: Lessons and reflections from four cases. New Directions for Evaluation (pp. 73–85).
Lindgren, L. (2011). If Robert Merton said it, it must be true: A citation analysis in the field of performance measurement. Evaluation, 17(1), 7–19.
Mark, M. M., & Henry, G. T. (2004). The mechanisms and outcomes of evaluation influence. Evaluation, 10(1), 35–57.
Martín-Martínm, A., Orduna-Malea, E., Thelwall, M., & López-Cózar, E. D. (2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of informetrics 12(4), 1160–1177.
Mathie, A., & Greene, J. C. (1997). Stakeholder participation in evaluation: How important is diversity. Evaluation and Program Planning, 20(3), 279–285.
Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13): 2105–2125.
Mertens, D. M., & Wilson, A. T. (2019) Program evaluation theory and practice: A comprehensive guide, (2nd Ed.). New York, NY: Guilford Press.
Moed, H. F. (2005). Citation Analysis in Research Evaluation. Netherlands: Springer.
Morrow, E. M., Goreham, H., & Ross, F. (2017). Exploring research impact in the assessment of leadership, governance and management research. Evaluation, 23(4), 407–431.
Noruzi, A. (2005). Google scholar: The new generation of citation indexes. LIBRI, 55(4): 170–180.
Quiedeville, S. (2017). Ex-post assessment of impacts of research on innovations for organic farming: Issues, methods, tools, and instruments (PhD Thesis). Université De Montpellier.
Roseland, D., Greenseid, L. O., Volkov, B. B., & Lawrenz, F. (2011). Documenting the impact of multisite evaluations on the science, technology, engineering, and mathematics field. New Directions for Evaluation, 129, 39–48.
Sahoo, B., & Kumar, A. (2018). Indexing and abstracting bibliographic electronic database: A comparative analysis. International Journal of Information Dissemination and Technology, 8(2), 99–103.
Shadish, W. R. (1998). Evaluation theory is who we are. American Journal of Evaluation, 19(1), 1–19.
Shah, S. R. U., & Mahmood, K. (2017). Review of Google scholar, Web of Science, and Scopus search results: The case of inclusive education research. Library Philosophy and Practice (e-journal). 1544.
Tijssen, R. J. W., Mouton, J., Van Leeuwen, T. N. & Boshoff, N. (2006). How relevant are local scholarly journals in global science? A case study of South Africa. Research Evaluation 15(3), 163–174.
Vallin, L. M., Philippoff, J., Pierce, S., & Brandon, P. R. (2015). Research-on-evaluation articles published in the American Journal of Evaluation, 1998–2014. In P. R. Brandon (Ed.), Research on evaluation. New Directions for Evaluation (pp.7–15).
Walters, W. H. (2016). Information sources and indicators for the assessment of journal reputation and impact. The Reference Librarian, 57(1), 13–22.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Melissa Rae Goodnight, Cherie M. Avent, Ramya Kumaran
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
