Tips for Evaluation

By evaluating both the process of implementation and the outcomes, you can better understand what impact implementation is having and why. By evaluating how things are going along the way, rather than just at the end, you may also be able to pick-up on issues and address these early-on. An organisation may want to conduct their own internal evaluation or work with an outside partner like a university. Here we provide links to different tools you could choose from depending what it is your evaluation will focus on.

The ImpRes tool is a user-friendly step-by-step guide to planning an implementation study. It introduces key concepts and tools in implementation science along the way.

Implementation happens in real-world settings rather than a laboratory, so the context in each organisation will be unique. There are different ways you can keep track of what is going on. Here are some:

  • Facilitator reports – we provide a template in Walk the Talk for facilitators to systematically take notes about the process of implementation planning. 
  • Involvement evaluation form for Implementation Team members – This template allows Implementation Team members to record their reflections after being involved at a meeting or event. Results can help the facilitator improve Implementation Team meetings.

Recovery Self-Assessment (RSA) aims to measure the implementation of recovery-oriented elements in services for people with mental health problems that support or impede recovery. This scale has adapted versions to elicit the perceptions of managers, staff, users of mental health services and significant others.

https://medicine.yale.edu/psychiatry/prch/tools/rec_selfassessment/

Recovery Knowledge Inventory is a 20-item questionnaire that assesses the knowledge and attitudes towards recovery of mental health service providers.

A review by Shanks and colleagues identified thirteen tools for measuring personal recovery. Here are links to some identified in the review:

Khanam and colleagues in 2013 published a detailed Toolkit for Mental Health Providers in New York City that summarises 40 measures of recovery: 23 at the individual level and 17 at the program level. The toolkit includes a wealth of information pertinent to each measure including the stakeholder that completes the measure, whether it’s free, copyrighted, or requires permission to be used, and links for more information. 

Access it here

The Consolidated Framework for Implementation Research (CFIR) identifies five groups of factors( domains) that could hinder or facilitate the implementation of a healthcare intervention: factors related to the intervention, (e.g., evidence strength and quality); the outer setting (e.g., patient needs and resources); the inner setting (e.g., culture, leadership engagement); the characteristics of the individuals involved; and the process (e.g., planning). The CFIR guide website provides a range of data collection tools,

As part of the Walk the Talk toolkit we have have created the Involvement Evaluation Form based on the “Sample involvement evaluation form for service user representative/project advisors” from the “Mental health researchers’ toolkit for involving service users in the research process” (Mental Health Research Network, June 2011). It is a great way to evaluate how the Implementation Team process is going and what can be improved. 

Proctor and colleagues in 2011 identified eight implementation outcomes (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3068522/). Here we provide links to some tools for assessing each possible implementation outcome. For a wider selection of implementation measures per implementation outcome, you could read the review by Mettert and colleagues in 2020  (https://journals.sagepub.com/doi/full/10.1177/2633489520936644) who identified 102 different measures for implementation outcomes.