The P4C Research Study

Partnering for Change Evaluation Process

The Partnering for Change model was evaluated over two years to enable collection of important information that will guide further strategic planning and program implementation within school health. Evaluation consisted of measurement of child, family, educator, occupational therapist and health system outcomes using multiple standardized assessments, questionnaires, focus groups and interviews.

  • Qualitative and quantitative methodologies were used to enable multiple perspectives to be explored and to ensure a thorough understanding of the processes, outcomes and benefits of the P4C model.
  • Data were collected from educators, families, occupational therapists, and children in all elementary schools that received the P4C service.

Research Objectives

Research objectives were identified at the beginning of the study and were addressed and evaluated during the two years:

  • Build capacity in school boards, schools and among educators
  • Build capacity among families for self-management
  • Describe children who are identified and served using the P4C Model
  • Describe any change in individual child outcomes
  • Identify any barriers to implementation and developing solutions
  • Determine the cost of Partnering for Change service delivery

Methods of Evaluation: Quantitative

Quantitative evidence was collected from parents, educators and OTs over two school years: Year 1 (November 2013 to June 2014) and Year 2 (September 2014 to June 2015).

Describing the Children, Individual Child Outcomes and Parent Capacity-Building

Families were recruited into the study during Year 1. Data were collected from parents in the first year to describe the children and in the second year, to understand the specific impact for children receiving the P4C service. OTs seeking consent for a child to receive health care services asked families if they were willing to hear more about the research study. Families were informed that participation in the study was voluntary and would not impact delivery of the P4C health service. Those families who agreed to hear more information about the study received a phone call from a project coordinator describing the study. 392 families agreed to receive information about the study. The project coordinator (rather than the OT) obtained additional consent from families for participation in the study.

The OTs obtained health care consent for 592 children to receive the P4C service:

  • 346 families who provided health care consent did not want to be research participants.
  • 246 families who provided health care consent also agreed to be part of the two year study.

Parents who consented to participate in the study completed pre- and post-standardized measures in Year 1 (T1) and Year 2 (T2); pre-measures were completed in the first year as parents enrolled in the study and post measures were completed in the spring/summer of the second year.

They were asked to complete questionnaires either in hard copy or electronically in order to:

  • capture demographic information about their family,
  • describe their child’s coordination issues in activities at home (DCD-Questionnaire) (collected once),
  • describe their knowledge about DCD (Parent Knowledge Questionnaire),
  • determine their child’s participation in different settings: Participation and Environment Measure — Children and Youth (PEM-CY),
  • report observations of their child’s behaviour: Strengths and Difficulties Questionnaire (SDQ).

At the end of the study, parents were asked some additional questions about capacity and their satisfaction with the P4C OT service.

At the beginning of Year 2, 30 research participants were lost to follow-up. Most of these were children who had moved to middle school or had moved to non-P4C schools. This left 216 participants for the remainder of the study.

The premise of P4C is that OTs can use their knowledge of health and development to observe children’s difficulties and do not need to administer standardized assessments in order to suggest and try out solutions in context. For the purpose of describing the children who were served in this study, however, additional consent was obtained from parents to enable a trained research assessor to complete the Movement Assessment Battery for Children (MABC-2) near the end of Year 2. Permission was sought again only from parents who had returned parent packages in Year 1 and who had provided consent. Trained research assistants completed the MABC-2 with 152 children whose families had provided consent for this additional activity.

Educators of the children who were in the study were asked about their Knowledge, Skill and Experience, and about their observations of the child’s behaviour (Strengths and Difficulties Questionnaire) and the child’s participation in the school (School Function Assessment).

Educators completed these measures at 3 time points:

  • when the child was referred into the study in Year 1 (T1)
  • the fall of Year 2 when the child was in the next grade (T2)
  • the spring of Year 2, usually by the same teacher (T3)

Building Capacity Among Educators

Educators at the 40 research schools who did not have research children in their classroom were also asked to complete an Educator Knowledge, Skill and Experience Questionnaire at four time points:

  • fall of 2013; at the start of Year 1 (T1)
  • spring of 2014; at the end of Year 1 (T2)
  • fall of 2014; at the start of Year 2 (T3)
  • spring of 2015; at the end of Year 2 (T4)

Building Capacity Among Occupational Therapists

Twenty-two different occupational therapists delivered the P4C service during the two year study: ten (10) OTs delivered the P4C service for two full years; twelve (12) OTs delivered it in either Year 1, Year 2 or parts of both years.

All OTs providing the service were asked to complete an Occupational Therapist Knowledge, Skills and Experience Questionnaire at 3 time points:

  • fall of 2013; at the start of Year 1 (T1)
  • spring of 2014; at the end of Year 1 (T2)
  • spring of 2015; at the end of Year 2 (T3)

At the end of Year 2, the OTs completed an Effectiveness of Training measure, to evaluate the effectiveness of the training and mentorship provided. The OTs also completed a Goal Attainment Scale at the end of Year 2, to report their reflection on the extent to which they achieved the goals of P4C in each school.

Describing the Cost of Delivering Partnering for Change

OTs completed an electronic Daily Log for every day they provided P4C services at their assigned research school to record OT services delivered at all levels of the P4C pyramid. A total of 1207 daily logs were collected in Year 1 and 1443 daily logs in Year 2, indicating the total number of days that OTs provided service each year.

OTs also noted on the electronic Daily Log requests they were receiving regarding children with special needs who were not currently receiving P4C but who could be served with the P4C model. In Year 2, these other types of children were added to the P4C OT workload and more children were transferred from the CCAC waitlist. New children continued to be added so that 806 children were considered to be clients of the CCAC across the period of the study (530 in Central West; 276 in HNHB).

Methods of Evaluation: Qualitative

In order to truly understand the implementation process, qualitative evidence was collected from many key stakeholders across the two years. Qualitative data was obtained via stakeholder interviews and focus groups to understand the barriers encountered and to derive potential solutions.

Focus Groups

The occupational therapists (OTs) delivering the P4C service participated in a focus group during each year of service delivery and provided “Pearls of Wisdom” at the conclusion of the study for future OTs; future funders and managers of a P4C model of service; and the P4C research team.

Interviews

Semi-structured qualitative interviews were completed with key stakeholders from the health care and education systems during both years of the project to enable increased understanding of how implementation of the model was experienced. An Implementation Science Framework was used to guide the selection of interview questions. Thematic analysis of the Year 1 interviews informed revision of training and dissemination materials used during Year 2 of the project.

Across the two year study, 51 interviews were conducted by investigators and/or trained staff with:

  • 14 education stakeholders (e.g. school board superintendents, principals, special education teachers),
  • 12 health care stakeholders (e.g. decision-makers, managers, clinical coordinators),
  • 5 occupational therapists,
  • 3 research team members,
  • 17 educators who had opportunities for collaboration with P4C OTs in support of children with special needs in their classrooms.

Additional data and information was also collected through regular email interactions, phone calls and meetings at multiple levels within the CCACs regarding: health system change, value for money, integration of the service approach and access to timely care.