Assessing Meaningful Community Engagement
Early Partnership Indicators
KEY FEATURES
COMMUNITY/ GEOGRAPHY
Policymakers
Researchers
Ontario, Canada
COMMUNITY ENGAGEMENT OUTCOMES
Strengthened partnerships + alliances
Broad alignment
Diversity + inclusivity
Sustained relationships
Mutual value
Trust
Shared power
Structural supports for community engagement
Expanded knowledge
Community-ready information
Improved health + health care programs + policies
Broad alignment
PLACE(S) OF INSTRUMENT USE
Government agency
Academic/research institution/university
LANGUAGE TRANSLATIONS
Not specified
PSYCHOMETRIC PROPERTIES
Content validity
Face validity
YEAR OF USE
2000-2002
Assessment Instrument Overview
The Early Partnership Indicators1,2 has 21 questions and should be used by policy makers and health researchers. It supports the management of collaborative knowledge generation and assesses the performance of a partnership, with focus on discussion of research findings, negotiation of partnership factors, and enhancement of the partnership itself. The Early Partnership Indicators is part of a set of three instruments that also includes the Common Partnership Indicators and the Mature Partnership Indicators.
Alignment with Assessing Meaningful Community Engagement Conceptual Model
The questions in Early Partnership Indicators were aligned to the Assessing Community Engagement Conceptual Model. Figure 1 displays the alignment of the Early Partnership Indicators with the Conceptual Model domain(s) and indicator(s). Where an instrument is mapped broadly with a domain or with a specific indicator, the figure shows the alignment in blue font.
Table 1 displays the alignment of the Early Partnership Indicators with the Conceptual Model domain(s) and indicator(s). The table shows, from left to right, the aligned Conceptual Model domain(s) and indicator(s) and the individual questions from the Early Partnership Indicators transcribed as they appear in the instrument (with minor formatting changes for clarity).
| CONCEPTUAL MODEL DOMAIN(S) AND INDICATOR(S) | ASSESSMENT INSTRUMENT QUESTIONS |
STRENGTHENED PARTNERSHIPS + ALLIANCES; Broad alignment with all indicators in this domain | 1.0. Clear leadership with respect to partnership management |
STRENGTHENED PARTNERSHIPS + ALLIANCES; Diversity + inclusivity | 1.1 Key players and senior management, where relevant, are visibly involved and supportive |
STRENGTHENED PARTNERSHIPS + ALLIANCES; Sustained relationships | 2.1 Discussion of potential long-term plans or structure to ensure continuity of relationship 3.0. Early engagement of people 3.1 Staff with previous linkages with each other are incorporated into partnership |
| STRENGTHENED PARTNERSHIPS + ALLIANCES; Mutual value | 1.2 Written terms of reference for research project (or similar document)* 2.0. Development of team mentality |
| STRENGTHENED PARTNERSHIPS + ALLIANCES; Trust | 1.1 Roles and responsibilities are documented 2.1 Requirements for deliverables and timelines are documented 4.0 Exposure to team/organization structures of research partners 4.1 Discussion of respective organizational realities of research partners |
| STRENGTHENED PARTNERSHIPS + ALLIANCES; Shared power | 1.0 Negotiation occurs at various stages of the research process 2.0 Negotiated items are clearly understood by all 2.2 Partners make their needs explicit (i.e., in terms of accountabilities, priorities, and long-term interest) |
STRENGTHENED PARTNERSHIPS + ALLIANCES; Structural supports for community engagement | 1.2 Written terms of reference for research project (or similar document)* 2.3 Partners document the above needs |
Expanded knowledge; Community-ready information | 1.2 Implications of findings are understood by all |
IMPROVED HEALTH + HEALTH CARE PROGRAMS + POLICIES; Broad alignment with all indicators in this domain | 1.0 Research findings are discussed in policy deliberations 1.1 Research findings are presented in policy-related format and language 1.3 Documentation of feedback to researchers 1.4 Ministry senior staff are aware of research findings 1.5 Research findings are discussed or are reflected in government meeting material and research documents |
*Note that these questions are duplicated to reflect their alignment with multiple domains and/or indicators in the Conceptual Model.
Table 1 | Early Partnership Indicators questions and alignment with the domain(s) and indicator(s) of the Assessing Community Engagement Conceptual Model
ASSESSMENT INSTRUMENT BACKGROUND
Context of instrument development/use
The article describes a study to “examine research receptor capacity and research utilization needs within the Ontario Ministry of Health and Long Term Care (MOHLTC).”1,2 The study explored the “abilities of Ministry staff to find, understand and use evidence-based research in policy development processes.” The Health System-Linked Research Unit (HSLRU), which had experience engaging with Ministry partners and developing research directly intended for transfer into government decision-making, supported the development of instruments. The instruments reflect both processes and outcomes that can be used to “manage collaborative knowledge generation or assess the performance of a partnership between health researchers and policymakers.” The study led to the development of a set of instruments: the Early Partnership Indicators (discussed here), as well as the Common Partnership Indicators and the Mature Partnership Indicators (discussed in other assessment instrument summaries).2
Instrument description/purpose
The Early Partnership Indicators instrument focuses on three key themes related to the early stages of partnerships:
- “Research findings are discussed in policy deliberations
- Negotiation occurs at various stages of the research process [and] negotiated items are clearly understood by all
- Partnership enhancement, [including] clear leadership with respect to partnership management, development of team mentality, early engagement of people, [and] exposure to team/organization structures of research partners”
The Early Partnership Indicators has 21 questions. The possible response options to the questions were not presented in the article.2
The Early Partnership Indicators instrument can be accessed here: https://doi.org/10.1057/kmrp.2011.16.
Engagement involved in developing, implementing, or evaluating the assessment instrument
The Early Partnership Indicators were developed using a cross-sectional survey followed by qualitative interviews, which provided “detailed recommendations to improve access to research information, enhance use of the information once accessed, and promote an organizational culture supportive of research utilization.” Study participants involved in developing and validating the instruments included “all eight of Ontario’s HSLRUs and their designated partners at the Ministry of Health and Long Term Care.” Semi-structured telephone interviews were conducted with eight Research Unit directors (or their designee) and their eight Ministry partners. Using the interview findings and findings from a literature review, the instruments were drafted and then tested with focus groups of HSLRU participants and one Ministry partner (the majority of whom also participated in the interviews) to examine “clarity, feasibility, credibility, relevance, level of specificity, and their ability to support each evaluation question.”2
Additional information on populations engaged in instrument use
The study participants – the HSLRU researchers and Ministry partners – conduct health research in a wide range of areas with policy implications, including “community health, cancer, dental health, rehabilitation, child health, arthritis, mental health, health information.” The partnerships often involved multiple projects, and included engagement with community, government, and research partners depending on the content area. Project activities were also wide-ranging and “included literature reviews, surveys, programme and service evaluation, costing estimates for policy initiatives, policy analysis, health system human resource analysis, intervention studies, knowledge dissemination to government and community, and knowledge transfer studies.”2
Notes
- Important findings:
- The Early Partnership Indicators, as well as the Mature Partnership Indicators and the Common Partnership Indicators (discussed in other assessment instrument summaries), support improved understanding of knowledge translation partnerships, providing opportunities to measure success at each stage of partnership development. The authors maintain that the results of this study are applicable beyond the partners who tested the instruments, especially given the broad range of research content and type of research conducted by the participants.2
- The authors noted that having good partnerships allowed for overcoming “actual differences of values and ideologies that might have impeded the work.” They suggested that a shared commitment to collaboration and to the work was critical and could lead to the development and maintenance of communication, rapport, and negotiation. Without these key success elements, difficulties in the partnerships ensued.2
- Negotiation [one of the three dimensions in the Early Partnership Indicators instrument] “was an explicit dimension in the partnership” and marked by the need for continuous ‘giving and taking’ during the research process. The importance of negotiation, and the understanding of what is being negotiated, has “received minimal attention in the researcher-policymaker literature.”2
- When considering the maturity of partnerships, the length of time working as partners may influence the characteristics displayed or exhibited among partners. In addition to the Common Partnership Indicators, Early Partnership Indicators, and Mature Partnership Indicators being used to evaluate relationships, they could also be used to monitor partnership processes and guide a set of deliverables that could be included in negotiated agreements.2
- Future research needed: Future prospective studies could provide evidence on the applicability of the instrument in practice. “Other future studies using these indicators might focus on prioritizing them, determining optimal frequency of measurement, usefulness in modifying the partnership midway through the partnership, or determining the extent to which they predict the use of research by policymakers. Alternatively, one might study which indicators are better suited for partnerships with bureaucrats, and which are better for collaborations with elected officials. Validation and reliability work would be required to optimize issues of reliability, validity, and generalizability. Such a study would also want to consider whether there are instances in which the indicators may obstruct the partnership.” Another area to study for the future would be the maturation of such partnerships, with considerations for the time frames needed to show a shift in early versus mature partnerships.2
We want to hear from you!
Assessing community engagement involves the participation of many stakeholders. Click here to share feedback on these resources, or email [email protected] and include “measure engagement” in the subject line to learn more about the NAM’s Assessing Community Engagement project.
