Asthma-related Systems Change Project Evaluation

The following evaluation plan is for the asthma-related systems change project, including the Controlling Asthma in American Cities funded clinics and the Medica Foundation funded clinics. Jill Heins Nesvold will serve as the lead internal evaluator. CAACP project staff recognize that internal evaluators may come with biases. Therefore, the CAACP Evaluation Team will provide evaluation oversight, review evaluation tools, and review evaluation findings prior to justifying any conclusions from the findings. It is the intent of this evaluation to utilize the process evaluation data on an ongoing basis to continually improve the project and assist the partner clinics in achieving their asthma-related goals.

Process and Outcome Evaluation

Guiding Process Evaluation Questions	Measure	Evaluation Method	Data Sources
What resources are needed to assist clinics in undertaking a asthma-related systems change project (including trainings, tools, equipment, financial resources)?	# and type of trainings # and type of tools # and type of equipment Amount of financial resources	Qualitative and quantitative analysis of program documents Staff interview Jill…Many records will not provide sufficient detail to clarify human resources required (number of hours spent on various tasks, etc.) Even when our forms are set up to collect these types of data , staff are notorious for underestimating the time spent on various efforts. You may need to supplement the registration and accounting records with a key informant interview to ensure accuracy of data collected via the written reports.	Participant registration Accounting records
To what degree were clinics able to gain buy-in from their clinic administration, providers, and other staff?	Description of methods to gain buy-in	Qualitative analysis of program documents Site visit	Clinic project team Documentation of letters of commitment Documentation of letters to patients
What barriers were encountered by CAACP project staff?	Reported staff barriers	Qualitative analysis of program documents Staff interview	Project staff
What barriers were encountered by clinic?	Reported clinic barriers	Qualitative analysis of program documents Site visit	Clinic staff lead and asthma champion
What unexpected opportunities were encountered during this project?	Reported opportunities	Qualitative analysis of program documents	Clinic staff lead and asthma champion Project staff
To what extent was the baseline assessment conducted at each clinic as proposed?	Use of spirometry Assignment of severity rating Prescription for controller meds Written asthma action plans Asthma education	Chart audit form Asthma services assessment Organizational readiness to change survey	Chart audits Clinic staff lead and asthma champion
How were individual clinic baseline findings used to develop their goals?	Individual clinic goals	Qualitative analysis of program documents Site visit	Clinic staff lead and asthma champion
How is work of each clinic team communicated with other clinic staff members?	# and types of project communication	Qualitative analysis of program documents Site visit	Clinic staff lead and asthma champion
To what degree is the mentor/coach utilized by the individual clinics?	# of mentor/coach requests for assistance	Qualitative analysis of program documents	Project staff
To what degree did the clinics find the joint clinic meetings useful?	Measure of participant satisfaction	Survey	Joint clinic meeting participants
To what extent were the monthly measures and monthly clinic status report forms completed by each clinic?	# of completed monthly measures and monthly clinic status report forms completed	Analysis of reporting forms	Monthly measures and monthly clinic status report forms completed by each clinic
To what degree were individual clinics able to define what they wanted to sustain beyond the one-year project?	Description of desired sustained components	Qualitative analysis of program documents	Clinic staff lead and asthma champion
Guiding Outcome Evaluation Questions
How many months (time) did it take for clinics to see a change in indicators after implementation of the project?	Length of time between project implementation and measured results	Analysis of reporting forms	Monthly measures and monthly clinic status report forms completed by each clinic
What was the change in indicators from baseline to the end of the project?	Use of spirometry Assignment of severity rating Prescription for controller meds Written asthma action plans Asthma education	Chart audit form	Chart audits
What processes were sustained at the end of the project?	# and type of processes changed	Qualitative analysis of program documents	Clinic staff lead and asthma champion