Comparing and Contrasting a Program versus System Approach to Evaluation
The Example of a Cardiac Care System
DOI:
https://doi.org/10.3138/cjpe.68127Keywords:
evaluation, systems, systems thinkingAbstract
This article examines the differences between approaching an evaluation problem from a program perspective and doing so from a systems perspective. The terms program, systems, systems thinking, and systems concepts are first defined. Then, using an actual evaluation of a cardiac care system, it is shown how initial investments in a program theory approach were deemed inadequate to account for the influence of external factors on patient outcomes. It was decided that a systems thinking approach was more appropriate for evaluating the interactions between several agencies comprising the cardiac care system. It is then shown how System Evaluation Theory (SET) was used to systematically apply different systems concepts to define and evaluate the cardiac care system. The discussion compares and contrasts the program and system evaluation approaches, noting the conditions under which each is more appropriate. It concludes by noting scope and cost differences between the two approaches.
References
Ackoff, R. L. (1971). Towards a system of systems concepts. Management Science, 7(11), 661 – 671. https://doi.org/10.1287/mnsc.17.11.661
American Evaluation Association (AEA). (2018). Principles for effective use of systems thinking in evaluation. Systems in Evaluation TIG. Retrieved from https://www.systemsinevaluation.com/wp-content/uploads/2018/10/SETIG-Principles-FINALDRAFT-2018-9-9.pdf
Astbury, B., & Hawkins, A. (2019). The consolations of evaluation theory. Presentation given at the Australian Evaluation Society Annual Conference, Sydney, Australia.
Bohm, K., Rosenqvist, M., Herlitz, J., Hollenberg, J., & Svensson, L. (2007). Survival is similar after standard treatment and chest compression only in out-of-hospital bystander cardiopulmonary resuscitation. Circulation, 116, 2908 – 2912. Retrieved https://ahajournals.org/doi/full/10.1161/circulationaha.107.710194
Chen, H. T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage.
Chen, H. T. (2014). Practical program evaluation: Theory-driven evaluation and the integrated perspective. Newbury Park, CA: Sage.
Collier, A. (1994). Critical realism: An introduction to Roy Bhaskar’s philosophy. London, England: Verso.
Department of Defense. (2016). Relationship between effective performance management and performance feedback. Retrieved from https://slideplayer.com/slide/14350612/89/images/6/and+Performance+Feedback.jpg
Donaldson, S. I. (2003). Theory-driven program evaluation in the new millennium. In S. I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the new millennium. Mahwah, NJ: Lawrence Erlbaum.
Donaldson, S. I. (2007). Program theory-driven evaluation science: Strategies and applications. New York, NY: Psychology Press.
Eisenberg, M. (2013). Resuscitate! How your community can improve survival from sudden cardiac arrest (2nd ed.). Seattle, WA: University of Washington Press.
Gesellschaft für Internationale Zusammenarbeit (GIZ). (2011). GIZ International Conference on Systemic Approaches in Evaluation, Eschborn, Germany , January 25 - 26. Retrieved from https://sites.google.com/a/cpwf.info/l2i/blog/gizconferenceonsystemicapproachestoevaluation
Granillo, B., Renger, R., McPherson, M., Dalbey, D., & Foltysova, J. (2014). Redfield-Aberdeen-Sioux falls cardiac arrest drill after-action report/improvement plan [Unpublished technical report]. Grant Forks, ND: Center for Rural Health, University of North Dakota.
Green, L. W., Kreuter, M. W., Deeds, S. G., Partridge, K. B., & Bartlett, E. (1980). Health education planning: A diagnostic approach. Palo Alto, CA: Mayfield Publishing .
Guzy, P. M., Pearce, M. L., & Greenfield, S. (1983). The survival benefit of bystander cardiopulmonary resuscitation is a paramedic served metropolitan area. American Journal of Public Health, 73(7), 766 – 769. https://doi.org/10.2105/ajph.73.7.766. Medline:6859360
Harness, D. S., Sullivan, S., Keegan, J., Taylor, C., Neibauer, T., Jones, I., ... Renger, J. (2019, September 3). Coordinating a ground-air intercept: Lessons learned. Journal of Emergency Medical Services. Retrieved from https://www.jems.com/articles/2019/08/coordinating-ground-air-intercept.html
Hawkins, A. J. (2016). Realist evaluation and randomised controlled trials for testing program theory in complex social systems. Evaluation, 22(3), 270 – 285. https://doi.org/10.1177/1356389016652744
Helmsley. (2019). The Leona M. and Harry B. Charitable Trust Rural Healthcare Program. Retrieved from https://helmsleytrust.org/programs/health-rural-healthcare
HSEEP. (2013). Homeland security and exercise evaluation program. Department of Homeland Security. Retrieved from https://www.fema.gov/media-library-data/20130726-1914-25045-8890/hseep_apr13_.pdf
Hummelbrunner, R. (2011). Systems thinking and evaluation. Evaluation, 17(4), 395 – 403. https://doi.org/10.1177/1356389011421935
Manghani, K. (2011). Quality assurance: Importance of systems and standard operating procedures. Quality, 2(1), 34 – 37. https://doi.org/10.4103/2229-3485.76288. Medline:21584180
Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco, CA: Jossey-Bass.
Meadows, D. H., & Wright, D. (2008). Thinking in systems: A primer. White River Junction, VT: Chelsea Green Publishing.
Meaney, P. A., Bobrow, B. J., Mancini, M. E., Christenson, J., de Caen, A. R., Bhanji, F., ... Leary, M. (2013). Cardiopulmonary resuscitation quality: Improving cardiac resuscitation outcomes both inside and outside the hospital. A consensus statement from the American Heart Association. Circulation, 128(4), 417 – 435. https://doi.org/10.1161/cir.0b013e31829d8654. Medline:23801105
Mohan, R. (2014). Evaluator advocacy: It is all in a day’s work. American Journal of Evaluation, 35(3), 397 – 403. https://doi.org/10.1177/1098214014532167
National Institutes of Health (NIH). (2019). Sudden cardiac arrest. Retrieved from https://www.nhlbi.nih.gov/health-topics/sudden-cardiac-arrest
Pawson, R. (2013). Science of evaluation: A realist manisfesto. London, England: Sage.
Pawson, R., & Tilley, N. (1997). Realistic Evaluation. London, England: Sage.
Physio-control. (2016). The LUCAS™2 training manual. Retrieved from http://www.lucas-cpr.com/web_training_center/en/pdf2/LUCAS_2_Web_Training_Center_CHECK%20LIST_EN_R05.pdf
Pima County Health Department. (2018). Program coordinator job description. Retrieved from https://www.governmentjobs.com/careers/pima?department%5B0%5D=Healt h&sort=PositionTitle%7CAscending
Renger, R. (2015). System Evaluation Theory (SET): A practical framework for evaluators to meet the challenges of system evaluation. Evaluation Journal of Australasia, 15(4), 16 – 28. https://doi.org/10.1177/1035719x1501500403
Renger, R. (2016). Illustrating the evaluation of system feedback mechanisms using System Evaluation Theory (SET). Evaluation Journal of Australasia, 16(4), 15 – 21. https://doi.org/10.1177/1035719x1601600403
Renger, R., Atkinson, L., Renger, J., Renger, J., & Hart, G. (2019). The connection between logic models and systems thinking concepts. Evaluation Journal of Australasia, 19(2), 79 – 87. https://doi.org/10.1177/1035719x19853660
Renger, R., Foltysova, J., Ienuso, S., Renger, R., & Booze, W. (2017). Evaluating system cascading failures. Evaluation Journal of Australasia, 17(2), 29 – 36. https://doi.org/10.1177/1035719x1701700205
Renger, R., Foltysova, J., Renger, J., & Booze, W. (2017). Defining systems to evaluate system efficiency and effectiveness. Evaluation Journal of Australasia, 17(3), 4 – 13. https://doi.org/10.1177/1035719x1701700302
Renger, R., Harness, D. S., Souvannasacd, E., & Granillo, B. (2018). The ALS intercept: The need for standard operating procedures (SOPs) and mutual aid agreements (MAAs) to improve patient care and attenuate the impact of billing constraints. Journal of Emergency Medical Services. Retrieved from https://www.jems.com/articles/2018/12/rethinking-als-intercepts-and-mutual-aide-agreements.html
Renger, R., Keogh, B., Hawkins, A., Foltysova, J., & Souvannasacd, E. (2018). Reworks: A robust system efficiency measure. Evaluation Journal of Australasia, 18(3), 183 – 191. https://doi.org/10.1177/1035719x18796611
Renger, R., McPherson, M., Kontz-Bartels, T., & Becker, K. L. (2016) Process flow mapping for systems improvement: Lessons learned. Canadian Journal of Program Evaluation, 31(1), 109 – 121. https://doi.org/10.3138/cjpe.267
Renger, R., & Titcomb, A. (2002). A three-step approach to teaching logic models. American Journal of Evaluation, 23(4), 493 – 503. https://doi.org/10.1177/109821400202300409
Renger, R., Wood, S., Williamson, S., & Krapp, S. (2011). Systemic evaluation, impact evaluation and logic models. Evaluation Journal of Australasia, 11(2), 24 – 30. https://doi.org/10.1177/1035719x1101100204
Rogers, P. J. (2008). Using programme theory to evaluate complicated and complex aspects of interventions . Evaluation, 14(1), 29 – 48.
Sandroni, C., & Geocadin, R. G. (2016). Neurological prognostication after cardiac arrest. Current Opinion in Critical Care, 21(3), 209 – 214. https://doi.org/10.1097/mcc.0000000000000202. Medline:25922894
Sutton, R. M., Maltese, M. R., Niles, D., French, B., Nishisaki, A., Arbogast, K. B., ... Nadkarni, V. (2009). Quantitative analysis of chest compression interruptions during in-hospital resuscitation of older children and adolescents. Resuscitation, 80(11), 1259 – 1263. https://doi.org/10.1016/j.resuscitation.2009.08.009. Medline:19733427
Ulrich , W. ( 2002 ). Boundary critique . In H. G. Daellenbach & R. L. Flood (Eds.), Th e informed student guide to management science. London, England : Thomson Learning .
Verheij, B. (2005). Evaluating arguments based on Toulmin’s scheme. Argumentation, 19(3), 347 – 371. https://doi.org/10.1007/s10503-005-4421-z
Williams, B. (2010). Fitting the key to the lock: Matching systems methods to evaluation questions. Paper presented at the 24th Annual Conference of the American Evaluation Association, San Antonio, TX.
Williams, B., & Britt, H. (2014). Systemic thinking for monitoring: Attending to interrelationships, perspectives and boundaries. Washington, DC: United States Agency for International Development. Retrieved from https://usaidlearninglab.org/library/attending-interrelationships-perspectives-andboundaries-complexity-aware-monitoring
Williams, B., & Hummelbrunner, R. (2010). Systems concepts in action: A practitioner’s toolkit. Stanford, CA: Stanford University Press.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2020 Ralph Renger, Jessica Renger, Stewart I. Donaldson, Jirina Renger, Gary Hart, Andrew Hawkins
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.
Authors retain copyright of their work and grant the journal right of first publication.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
