New Insights for Tracking and Reporting Milestones

Authors

  • Jessica Renger Kaiser Permanente, Pomona, California
  • Ralph Renger Just Evaluation Services, LLC, Vail, Arizona
  • Richard N. Van Eck School of Medicine and Health Sciences, University of North Dakota, Grand Forks, North Dakota
  • Marc D. Basson School of Medicine and Health Sciences, University of North Dakota, Grand Forks, North Dakota
  • Jirina Renger Walden University, Minneapolis, Minnesota

DOI:

https://doi.org/10.3138/cjpe.74487

Keywords:

milestones, monitoring, evaluation standards

Abstract

Despite being a common evaluation metric, milestones are poorly defined, and guidance for their use is underdeveloped in the evaluation literature. This practice note attempts to address this oversight by providing evaluation-specific guidance for tracking and reporting milestones. Using a case study, we share our initial approach to tracking and reporting milestones, as well as how critical reflection guided by the evaluation standards provided us with key insights that allowed us to make improvements to our initial process. We hope this practice note acts as a springboard for elevating the critical dialogue surrounding milestones in the evaluation literature.

References

Accreditation Council for Graduate Medical Education (ACGME). (2021). The next steps in the outcomes-based accreditation project. Retrieved from https://www.acgme.org/Portals/0/PFAssets/bulletin/bulletin5_08.pdf

Bjerke, M. B., & Renger, R. (2017). Being smart about writing SMART objectives. Evaluation and program planning, 61, 125–127.

Canadian Evaluation Society. (2010).Competencies for Canadian evaluation practice . Retrieved from https://evaluationcanada.ca/txt/2_competencies_cdn_evaluation_practice_ 2010.pdf

Coşkun, R., Akande, A., & Renger, R. (2012). Using root cause analysis for evaluating program improvement. Evaluation Journal of Australasia, 12(2), 4–14.

Edgar, L., McLean, S., Hogan, S. O., Hamstra, S., & Holmboe, E. S. (2020). Th e milestones guidebook (version 2020). Retrieved from Accreditation Council for Graduate Medical Education website: https://www.acgme.org/globalassets/milestonesguidebook.pdf

Gawande, Atul. (2009) The checklist manifesto: How to get things right. Metropolitan Books. New York, NY.

Green, L., & Kreuter, M. W. (2005). Health program planning: An educational and ecological approach (4th ed.). New York: McGraw Hill.

Harned, M. S., Schmidt, S. C., Korslund, K. E., & Gallop, R. J. (2021). Does adding the Dialectical Behavior Therapy Prolonged Exposure (DBT PE) protocol for PTSD to DBT improve outcomes in public mental health settings? A pilot nonrandomized effectiveness trial with benchmarking. Behavior therapy, 52(3), 639–655.

Heath, J. K., Edgar, L., & Guralnick, S. (2021). Assessment of learning, for learning: Operationalizing milestones data for program level improvement. Journal of Graduate Medical Education, 13(Suppl. 2), 120–123. https://doi.org/10.4300/JGME-D-20-00849.1

Lurie, N. H., & Swaminathan, J. M. (2009). Is timely information always better? Th e eff ect of feedback frequency on decision making. Organizational Behavior and Human Decision Processes, 108 (2), 315–329.

Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: An integrated framework for understanding, guiding, and improving policies and programs. San Francisco, CA: Jossey-Bass.

Merriam Webster (2021). Milestone. Retrieved from https://www.merriam-webster.com/dictionary/milestone

National Institutes of Health. (2017). Institutional Development Award (IDeA) program infra- structure for clinical and translational research (IDeA-CTR) (PAR-17-304). Retrieved from https://grants.nih.gov/grants/guide/pa-files/par-17-304.html

Renger, R., Foltysova, J., Ienuso, S., Renger, J., & Booze, W. (2017). Evaluating system cascading failures. Evaluation Journal of Australasia, 17(2), 29–36.

Scriven, M. (1991). Evaluation Thesaurus (4th ed.). SAGE. Newbury Park, CA.

Tovey, T. L. S, & Skolits, G. J. (2021, December). Conceptualizing and engaging in reflective practice: Experienced evaluators’ perspectives. American Journal of Evaluation. https://doi.org/10.1177/1098214020983926

University of North Dakota (2021). DaCCoTA: What is Clinical Translational Research? Retrieved from https://med.und.edu/daccota/

Vaneman, W., Carlson, R., & Wolfgeher, C. (2019). Defining a model-based systems engineering approach for milestone technical reviews. Acquisition Research Program (NPS-SE-20-016). Retrieved from https://dair.nps.edu/handle/123456789/4168

Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). Th e program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). SAGE. Thousand Oaks, CA.

Downloads

Published

2023-04-21

How to Cite

Renger, J., Renger, R., Van Eck, R. N., Basson, M. D., & Renger, J. (2023). New Insights for Tracking and Reporting Milestones. Canadian Journal of Program Evaluation, 37(2), 273–282. https://doi.org/10.3138/cjpe.74487

Issue

Vol. 37 No. 2 (2022): Fall

Section

Research and Practice Notes

License

Copyright (c) 2023 Jessica Renger, Ralph Renger, Richard N. Van Eck, Marc D. Basson, Jirina Renger

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Authors contributing to The Canadian Journal of Program Evaluation agree to release their articles under the Creative Commons Attribution-Noncommercial 4.0 (CC-BY-NC) license. This licence allows this work to be copied, distributed, remixed, transformed, and built upon for any purpose provided that appropriate attribution is given, a link is provided to the license, and changes made were indicated.

Authors retain copyright of their work and grant the journal right of first publication.

Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.