Organization Questionnaire for the Public and Patient Engagement Evaluation Tool

ASSESSMENT INSTRUMENT BACKGROUND

Context of instrument development/use
The articles highlight the importance of public and patient engagement (PPE) in quality improvement efforts and that evaluating PPE often requires a balance between “relevance to practitioner needs” and “application of rigorous methods.”⁴ The articles discuss the development of PPEET, which leverages a 3-year collaboration between Canadian researchers and practitioners.^1,4  PPEET “was launched as a simple-to-administer tool intended for use by a wide range of health system organizations to assess the quality and impacts of engagement, with the goal of contributing to both the practice and the science of public and patient engagement.”¹ PPEET consists of three questionnaires to evaluate public and patient engagement: the Organization Questionnaire (described here), the Participant Questionnaire (described in another assessment instrument summary), and the Project Questionnaire (described in another assessment instrument summary).

Instrument description/purpose
The Organization Questionnaire of the PPEET assesses how organizations are conducting engagement as an organizational activity and responsibility. It is completed by “those providing the leadership and capacity for public and patient engagement within their organizations (organizational leadership),” including health board members, senior management team members, and directors.¹ The Organization Questionnaire assesses:

• Collaboration and common purpose
• Influence and impact
• Participatory culture
• Policies and practices that support planning and implementation

The Organization Questionnaire contains 32 questions and uses a combination of open-ended, yes/no, and various five-point Likert scales with response options ranging from “strongly agree” to “strongly disagree” and “all of the time” to “don’t know.”⁴

The guidance for administering the instruments and the English and the French translations for the three questionnaires in the PPEET, including the Organization Questionnaire, can be accessed here: https://ppe.mcmaster.ca/resources/public-and-patient-engagement-evaluation-tool/. Please contact [email protected] to request the other language translations.

Engagement involved in developing, implementing, or evaluating the assessment instrument

“A pan-Canadian partnership of PPE practitioners and researchers” with the shared goal of developing a common evaluation tool formed through two consecutive research grants from the Canadian Institutes of Health Research. This research–practice collaborative “included representation from seven provinces, six regional health authorities and two provincial and local health organizations.”⁴

The development of the tool took place over a 3-year period. After a review of the literature, collaborative members engaged using structured e-mail, telephone, and face-to-face exchanges at workshops. A process with iterative rounds of review, also known as a modified Delphi process, was also used to review and prioritize insights. These activities contributed to the “identification of a set of overarching principles for carrying out high quality PPE activities that would serve as the foundation for the evaluation tool.”  The workshops used break-out sessions and reporting back to the larger group, as well as larger group discussions, to identify and agree on a core set of outcomes. The core principles were mapped to outcomes and prioritized for inclusion in the tool. After developing “three discrete evaluation questionnaires for three different respondent groups,” the collaborative “tested the usability of the questionnaires preceding final revisions to the tool.”⁴

Patients and members of the public were only directly involved in the usability-testing phase. Participants, project managers and senior organizational personnel in two health regions tested the usability of the questionnaires.⁴ The tool underwent “additional feasibility testing in seven health system organizations in Ontario in collaboration with staff and patient partner representatives from each organization.” The PPEET was modified based on the results and the revised instruments, which launched in August 2018, were tailored to the specific respondent groups, had separate modules for different types and stages of engagement “(e.g., one time versus ongoing and planning versus implementation),” and included an increased balance in response options with opportunities for more in-depth follow up.¹

Additional information on populations engaged in instrument use
Usability testing for the questionnaires took place with practice partners from two health regions and provinces. The Organization Questionnaire “was distributed to 75 health board member and senior management team members and directors across the two organizations with 28 responses received.”⁴

Notes

• Potential limitations: The authors suggest that the focus throughout the process of developing the tool was on user needs (i.e., usability) rather than on psychometric properties, which may have led to a less robust evaluation tool. Since the tool development process was influenced by having short and easy to administer questionnaires, this may have “compromised the tool’s validity (e.g., number and specificity of statements used to assess a particular domain of practice, use of a 5-point vs. a 7-point scale).” Additionally, patient and citizen perspectives were not directly included in the development process for the PPEET beyond the usability testing phase. The PPEET’s focus on the health care context of Canada may limit its generalizability and applicability to “non-Canadian settings and to health-care organizations that focus on smaller and more specialized populations;” however, the extensive international literature review and participating partner organizations from major urban and regional referral centers that informed the tool included large and highly diverse populations.⁴
• Important findings: The authors indicate that to their knowledge, “this is the first collaboration of researchers and practitioners in the co-design of a comprehensive evaluation tool aimed at assessing the quality and impact of episodic and on-going PPE activities in health system organizations from three distinct perspectives – public and patient participants, sponsors and managers of PPE projects and organizational leaders responsible for PPE.” The tool strikes a balance between “the application of rigorous methods and relevance to practitioner needs,” Based on usability testing results, revisions were made to the Organization Questionnaire to improve accessibility (e.g., clarity, layout).⁴
• Future research needed: Additional research and testing of the questionnaires is needed to understand if any weaknesses exist in the PPEET’s validity. Further testing is also needed on the feasibility of applying the tool to every type, level, and degree of PPE.⁴
• Supplemental information: The modified version of the PPEET, including the Organization Questionnaire, released in 2018, can be accessed here: https://healthsci.mcmaster.ca/docs/librariesprovider61/default-document-library/ppeet-complete-set-final.pdf?sfvrsn=d1617fe6_2. Additional information on other settings this assessment instrument has been used in (i.e., emergency settings), populations in which the instrument has been tested (i.e., children with developmental delays, women with heart diseases) and modifications made can be found in the following articles:
  • Ogourtsova, T., M. E. O’Donnell, J. H. Filliter, K. Wittmeier, Bright Coaching Group, and A. Majnemer. 2021. Patient engagement in an online coaching intervention for parents of children with suspected developmental delays. Developmental Medicine & Child Neurology 63 (6):668-674. https://doi.org/10.1111/dmcn.14810.
  • Teed, M., J. Ianiro, C. Culhane, J. Monaghan, J. Takacs, G. Arthur, and A. Nash. 2021. Engaging Women With Lived Experience: A Novel Cross- Canada Approach. Journal of Patient Experience 8:1-7. https://doi.org/10.1177/23743735211008300.
  • Bhati, D. K., M. Fitzgerald, C. Kendall, and S. Dahrouge. 2020. Patients’ engagement in primary care research: a case study in a Canadian context. Research Involvement and Engagement 6:1-12. https://doi.org/10.1186/s40900-020-00238-x.
  • Drebit, S., K. Eggers, C. Archibald, R. Abu-Laban, K. Ho, A. Khazei, R. Lindstrom, J. Marsden, E. Martin, and J. Christenson. 2020. Evaluation of Patient Engagement in a Clinical Emergency Care Network: Findings From the BC Emergency Medicine Network. Journal of Patient Experience 7(6):937-940. https://doi.org/10.1177/2374373520925721.
  • Thompson, A. P., S. E. MacDonald, E. Wine, and S. D. Scott. 2020. An Evaluation of Parents’ Experiences of Patient Engagement in Research to Develop a Digital Knowledge Translation Tool: Protocol for a Multi-Method Study. JMIR Research Protocols 9(8). https://doi.org/10.2196/19108.