Project Questionnaire for the Public and Patient Engagement Evaluation Tool

ASSESSMENT INSTRUMENT BACKGROUND

Context of instrument development/use
The articles highlight the importance of public and patient engagement (PPE) in quality improvement efforts and that evaluating PPE often requires a balance between “relevance to practitioner needs” and “application of rigorous methods.”⁴ The articles discuss the development of PPEET, which leverages a 3-year collaboration between Canadian researchers and practitioners.^1,4  PPEET “was launched as a simple-to-administer tool intended for use by a wide range of health system organizations to assess the quality and impacts of engagement, with the goal of contributing to both the practice and the science of public and patient engagement.”¹ PPEET consists of three questionnaires to evaluate public and patient engagement: the Project Questionnaire (described here), the Organization Questionnaire (described in another assessment instrument summary), and the Participant Questionnaire (described in another assessment instrument summary).

Instrument description/purpose

The Project Questionnaire of the PPEET allows for capturing insights from those responsible for “planning, execution or sponsoring of engagement activities within organizations.”¹ The instrument is for engagement practitioners and users, such as directors and managers of organizations or sponsors. The Project Questionnaire assess three areas:

• Integrity of design and process
• Influence and impact
• Collaboration and common purpose

The Project Questionnaire contains 28 questions that use a combination of open-ended, yes/no/don’t know, and five-point Likert scale responses. The Likert scale ranges from “strongly agree” to “strongly disagree.”⁴

 The guidance for administering the instruments and the English and the French translations for the three questionnaires in the PPEET, including the Participant Questionnaire, can be accessed here: https://ppe.mcmaster.ca/resources/public-and-patient-engagement-evaluation-tool/. Please contact [email protected] to request the other language translations.

Engagement involved in developing, implementing, or evaluating the assessment instrument
“A pan-Canadian partnership of PPE practitioners and researchers” with the shared goal of developing a common evaluation tool formed through two consecutive research grants from the Canadian Institutes of Health Research. This research–practice collaborative “included representation from seven provinces, six regional health authorities and two provincial and local health organizations.”⁴

The development of the tool took place over a 3-year period. After a review of the literature, collaborative members engaged using structured e-mail, telephone, and face-to-face exchanges at workshops. A process with iterative rounds of review, also known as a modified Delphi process, was also used to review and prioritize insights. These activities contributed to the “identification of a set of overarching principles for carrying out high quality PPE activities that would serve as the foundation for the evaluation tool.”  The workshops used break-out sessions and reporting back to the larger group, as well as larger group discussions, to identify and agree on a core set of outcomes. The core principles were mapped to outcomes and prioritized for inclusion in the tool. After developing “three discrete evaluation questionnaires for three different respondent groups,” the collaborative “tested the usability of the questionnaires preceding final revisions to the tool.”⁴

Patients and members of the public were only directly involved in the usability-testing phase. Participants, project managers and senior organizational personnel in two health regions tested the usability of the questionnaires.⁴ The tool underwent “additional feasibility testing in seven health system organizations in Ontario in collaboration with staff and patient partner representatives from each organization.” The PPEET was modified based on the results and the revised instruments, which launched in August 2018, were tailored to the specific respondent groups, had separate modules for different types and stages of engagement “(e.g., one time versus ongoing and planning versus implementation),” and included an increased balance in response options with opportunities for more in-depth follow up.¹

Additional information on populations engaged in instrument use
Usability testing for the questionnaires took place with practice partners from two health regions and provinces. The Project Questionnaire “was distributed to 28 directors and managers across both organizations with 14 responses received.”⁴

Notes

• Potential limitations: The authors suggest that the focus throughout the process of developing the tool was on user needs (i.e., usability) rather than on psychometric properties, which may have led to a less robust evaluation tool. Since the tool development process was influenced by having short and easy to administer questionnaires, this may have “compromised the tool’s validity (e.g., number and specificity of statements used to assess a particular domain of practice, use of a 5-point vs. a 7-point scale).” Additionally, patient and citizen perspectives were not directly included in the development process for the PPEET beyond the usability testing phase. The PPEET’s focus on the health care context of Canada may limit its generalizability and applicability to “non-Canadian settings and to health-care organizations that focus on smaller and more specialized populations;” however, the extensive international literature review and participating partner organizations from major urban and regional referral centers that informed the tool included large and highly diverse populations.⁴
• Important findings: The authors indicate that to their knowledge, “this is the first collaboration of researchers and practitioners in the co-design of a comprehensive evaluation tool aimed at assessing the quality and impact of episodic and on-going PPE activities in health system organizations from three distinct perspectives – public and patient participants, sponsors and managers of PPE projects and organizational leaders responsible for PPE.” The tool strikes a balance between “the application of rigorous methods and relevance to practitioner needs,” Based on usability testing results, revisions were made to the Project Questionnaire to improve accessibility (e.g., clarity, layout).⁴
• Future research needed: Additional research and testing of the questionnaires is needed to understand if any weaknesses exist in the PPEET’s validity. Further testing is also needed on the feasibility of applying the tool to every type, level, and degree of PPE.⁴
• Supplemental information: The modified version of the PPEET, including the Project Questionnaire, released in 2018, can be accessed here: https://healthsci.mcmaster.ca/docs/librariesprovider61/default-document-library/ppeet-complete-set-final.pdf?sfvrsn=d1617fe6_2. Additional information on other settings this assessment instrument has been used in (i.e., emergency settings), populations in which the instrument has been tested (i.e., children with developmental delays, women with heart diseases) and modifications made can be found in the following articles:
  • Ogourtsova, T., M. E. O’Donnell, J. H. Filliter, K. Wittmeier, Bright Coaching Group, and A. Majnemer. 2021. Patient engagement in an online coaching intervention for parents of children with suspected developmental delays. Developmental Medicine & Child Neurology 63 (6):668-674. https://doi.org/10.1111/dmcn.14810.
  • Teed, M., J. Ianiro, C. Culhane, J. Monaghan, J. Takacs, G. Arthur, and A. Nash. 2021. Engaging Women With Lived Experience: A Novel Cross- Canada Approach. Journal of Patient Experience 8:1-7. https://doi.org/10.1177/23743735211008300.
  • Bhati, D. K., M. Fitzgerald, C. Kendall, and S. Dahrouge. 2020. Patients’ engagement in primary care research: a case study in a Canadian context. Research Involvement and Engagement 6:1-12. https://doi.org/10.1186/s40900-020-00238-x.
  • Drebit, S., K. Eggers, C. Archibald, R. Abu-Laban, K. Ho, A. Khazei, R. Lindstrom, J. Marsden, E. Martin, and J. Christenson. 2020. Evaluation of Patient Engagement in a Clinical Emergency Care Network: Findings From the BC Emergency Medicine Network. Journal of Patient Experience 7(6):937-940. https://doi.org/10.1177/2374373520925721.
  • Thompson, A. P., S. E. MacDonald, E. Wine, and S. D. Scott. 2020. An Evaluation of Parents’ Experiences of Patient Engagement in Research to Develop a Digital Knowledge Translation Tool: Protocol for a Multi-Method Study. JMIR Research Protocols 9(8). https://doi.org/10.2196/19108.