Developing an Evaluation Plan
Evaluation of technical assistance does not have to be daunting. The two primary purposes are to:
- Collect information for yourself as you provide TA so you can make adjustments as needed or plan overall improvements for your project
- Collect information for required reports (e.g., for OSEP or your state department of education)
The type and intensity of evaluation depends on what you need and want to collect to serve these two purposes and is influenced by a variety of factors, including the intensity of the TA you are providing, where your project is housed, and your project’s internal procedures
Determine what you will evaluate and why
The following are common areas of TA evaluation:
Team/agency readiness (13)
The degree to which a team and school or other agency are willing and able to engage in TA and implement new practices. It is typically evaluated during the Exploration Phase, but you may want to reassess it during the Initial Implementation Phase to identify ongoing or emerging problems.
Fidelity of the TA process
The degree to which your own TA is consistent with good TA practice and conducted as outlined in the TA agreement or plan.
Technical assistance content
The degree to which the information or training delivered through the TA is relevant, useful, and of high quality.
Change in knowledge and skills (12)
The degree to which service providers and families are progressing in areas where a need for knowledge and skill development has been identified.
Fidelity of implementation (1, 3)
The degree to which service providers and family members are implementing key components of effective practices as intended (see Exploration Phase: Selecting Practices or Programs for the Team to Implement).
Child change (1)
The degree to which goals and outcomes for a child are being achieved. This can be communicated to a team to support implementation or reported to state agencies.
Determine data collection sources and methods
Depending on your needs, you may collect data from parents, teachers, administrators, or other team members. Apply your experience working with various groups to determine the most effective and efficient ways to gather information. For example, you may have found that parents prefer direct conversations, while service providers do better with short surveys.
There are a variety of different ways to collect data, including:
- Self-report (e.g., via survey)
- Pre/post tests
- Observation
- Informal conversations and written exchanges
- Analysis of TA plans and notes
The methods you choose are highly dependent upon available resources, including:
- Time (this is especially important if you plan to gather observational or qualitative data that is time consuming to collect and summarize)
- Existing surveys or checklists
- Electronic survey software
- TA logs and summaries
- Observation tools (e.g., Classroom Observation Instrument)
Advice from Colleagues
Measuring change in service provider, family, or child outcomes can be time consuming and intensive in nature. While observational data is richer and more reliable, it is probably not realistic to gather it extensively.
Tools and Resources
Determine how you will analyze your data to meet your needs
The data you collect will likely be in several forms (e.g., ratings, narrative, multiple choice responses). Decide how to organize it so that you can:
- Keep track of it over time
- Provide feedback to guide practice
- Produce annual reports
- Produce summaries for stakeholders
Create a data collection timeline
Determine when and how frequently you will collect specific data. The timing is dependent on your reasons for collecting the data. For example:
- During the Initial Implementation Phase, you are likely to be collecting data to provide immediate feedback on how things are going or to identify new needs that have arisen so you can adapt your TA accordingly.
- During the Full Implementation Phase, you may collect data to include in your annual report.