Internal Coalition Effectiveness Instrument

Sandoval, J. A., J. Lucero, J. Oetzel, M. Avila, L. Belone, M. Mau, C. Pearson, G. Tafoya, B. Duran, L I. Rios, and N. Wallerstein. 2012. Process and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health Education Research 27(4):680-690. https://doi.org/10.1093/her/cyr087.¹

Cramer, M. E., J. R. Atwood, and J. A. Stoner. 2006. Measuring Community Coalition Effectiveness Using the ICE Instrument. Public Health Nursing 23(1):74-87. https://doi.org/10.1111/j.0737-1209.2006.230111.x.²

ASSESSMENT INSTRUMENT OVERVIEW
The Internal Coalition Effectiveness (ICE)^1,2 instrument has 30 questions and is used by public health nurses. It evaluates the strengths and areas of improvement in community coalitions.​

ALIGNMENT WITH ASSESSING COMMUNITY ENGAGEMENT CONCEPTUAL MODEL
The questions in ICE were realigned to the Assessing Community Engagement Conceptual Model. Figure 1 displays the alignment of ICE with the Conceptual Model domain(s) and indicator(s). Where an instrument is mapped broadly with a domain or with a specific indicator, the figure shows the alignment in blue font.

Figure 1 | Alignment of the Internal Coalition Effectiveness instrument with the Assessing Community Engagement Conceptual Model

Table 1 displays the alignment of ICE’s individual questions and validated focus areas with the Conceptual Model domain(s) and indicator(s). The table shows, from left to right, the aligned Conceptual Model domain(s) and indicator(s), the individual questions from ICE transcribed as they appear in the instrument (with minor formatting changes for clarity), and the validated focus area(s) presented in the article.

CONCEPTUAL MODEL DOMAIN(S) AND INDICATOR(S)	ASSESSMENT INSTRUMENT QUESTIONS	VALIDATED FOCUS AREA(S)
STRENGTHENED PARTNERSHIPS + ALLIANCES; Diversity + inclusivity	Section 2: 16. Leaders of my coalition work for coalition success by…promoting the involvement of a broad base of members in the work of the coalition.	Efficient practices
	Section 1: 8. Members of my coalition…work together to establish positive relationships with community members whom the coalition wants to engage and mobilize. Section 2: 22. Leaders of my coalition work for coalition success by…establishing positive relationships with community members that the coalitions want to engage and mobilize.	Relationships
	Section 1: 11. Members of my coalition…have a sense of inclusivity that engages a variety of public and private individuals from the community in the coalition – from elected officials to community leaders and residents. Leaders of my coalition work for coalition success by… Section 2: 27. facilitating a sense of inclusivity that engages a variety of public and private individuals from the community in the coalition – from elected officials to community leaders and residents. Section 2: 28. working to engage a broad cross section of the community to participate in the coalition’s work.	Participation
STRENGTHENED PARTNERSHIPS + ALLIANCES; Partnerships + opportunities	Section 2: 18. Leaders of my coalition work for coalition success by…” developing other leaders within the coalition.*	Efficient practices
	Section 2: 21. Leaders of my coalition work for coalition success by…providing resources to develop leadership skills among coalition members.*	Knowledge and training
	Section 2: 23. Leaders of my coalition work for coalition success by…facilitating positive community relationships with other local key players and stakeholders involved in the issues.	Relationships
STRENGTHENED PARTNERSHIPS + ALLIANCES; Acknowledgment, visibility, recognition	Section 2: 24. Leaders of my coalition work for coalition success by…building respectful relationships between the coalition and the community.	Relationships
STRENGTHENED PARTNERSHIPS + ALLIANCES; Sustained relationships	Section 1: 4. Members of my coalition…work together to coordinate coalition activities to avoid duplication of services and efforts.	Efficient practices
	Section 1: 13. Members of my coalition…take the necessary corrective action when problems arise regarding lack of activity implementation by other coalition members.* Section 2: 30. Leaders of my coalition work for coalition success by…taking the necessary corrective action when problems arise regarding lack of activity implementation by individual coalition members.*	Activities
STRENGTHENED PARTNERSHIPS + ALLIANCES; Mutual value	Section 1: 5. Members of my coalition…work together to strengthen each other’s advocacy efforts.	Efficient practices
	Section 1: 13. Members of my coalition…take the necessary corrective action when problems arise regarding lack of activity implementation by other coalition members.* Section 2: 30. Leaders of my coalition work for coalition success by…taking the necessary corrective action when problems arise regarding lack of activity implementation by individual coalition members.*	Activities
STRENGTHENED PARTNERSHIPS + ALLIANCES; Trust	Section 2: 26. Leaders of my coalition work for coalition success by…facilitating open communication within the coalition and with the coalition leaders.	Participation
	Section 1: 13. Members of my coalition…take the necessary corrective action when problems arise regarding lack of activity implementation by other coalition members.* Section 2: 30. Leaders of my coalition work for coalition success by…taking the necessary corrective action when problems arise regarding lack of activity implementation by individual coalition members.*	Activities
STRENGTHENED PARTNERSHIPS + ALLIANCES; Shared power	Members of my coalition… Section 1: 1. have a shared social vision. Section 1: 2. agree with our coalition’s mission and purpose. Leaders of my coalition work for coalition success by… Section 2: 14. facilitating a shared social vision among coalition members. Section 2: 15. facilitating the process of developing agreement among coalition members about the mission and purpose.	Social vision
	Section 1: 3.Members of my coalition…work together to make the coalition’s financial resources go substantially further. Section 2: 18. Leaders of my coalition work for coalition success by…” developing other leaders within the coalition.*	Efficient practices
	Leaders of my coalition work for coalition success by… Section 2: 19. providing resources to keep coalition members current on issue‐ related legislation. Section 2: 21. providing resources to develop leadership skills among coalition members.*	Knowledge and training
	Members of my coalition… Section 1: 9. encourage each other to actively participate in the coalition’s decision‐making process. Section 1: 10. encourage each other to identify issues, analyze problems, select interventions and evaluate interventions. Section 2: 25. Leaders of my coalition work for coalition success by…encouraging members’ active participation in the coalition’s decision‐making processes.	Participation
EXPANDED KNOWLEDGE; Bi-directional learning	Members of my coalition… Section 1: 6. work together to expand each member’s knowledge and potential for addressing the issues. Section 1: 7. enrich each other’s abilities and skills in the issues. Section 2: 20. Leaders of my coalition work for coalition success by…providing resources to keep coalition members informed about best practices on the issues.	Knowledge and training
Not aligned with Conceptual Model	Section 2: 17. Leaders of my coalition work for coalition success by…repositioning coalition assets, competencies, and resources to address changing needs and priorities.	Efficient practices
	Section 1: 12. Members of my coalition…successfully implement the vast majority of coalition’s work plan on a timely basis. Section 2: 29. Leaders of my coalition work for coalition success by…providing necessary organizational oversight to the coalition based on evaluation data to ensure that the vast majority of the work plan is implemented on a timely basis.	Activities

*Note that these questions are duplicated to reflect their alignment with multiple domains and/or indicators in the Conceptual Model.

Table 1 | Internal Coalition Effectiveness Instrument questions and alignment with the domain(s) and indicator(s) of the Assessing Community Engagement Conceptual Model

ASSESSMENT INSTRUMENT BACKGROUND

Context of instrument development/use
The article identifies a critical role for public health nurses and faculty in “evaluation, program planning, communications, relationship development, and community development.” These professionals are often asked to serve as evaluators for coalitions engaged in developing health programs for communities. ICE was developed to be used by public health nurses who participate in and evaluate community coalitions.²

Instrument description/purpose

ICE is based on the Internal Coalition Outcome Hierarch (ICOH) conceptual model and assesses strengths and areas of improvement for community coalitions using seven validated (i.e., construct) focus areas:

• Social vision
• Efficient practices
• Knowledge and training
• Relationships
• Participation
• Activities
• Resources

ICE consists of 30 questions. The scoring information indicates that ICE is organized into two sections that first ask the respondent “to consider how well members work together to achieve common goals and objectives” and second, “to consider how well collation leaders are effective in facilitating the work of the coalition.” Each focus area question is scored using a five-point Likert scale ranging from “strongly disagree” to “strongly agree.”

Individual scores for each validated focus area are obtained by calculating the mean of the focus area responses and calculating the mean of the responses to all 30 items provides a score of the overall coalition effectiveness.

ICE can be accessed here: https://nam.edu/wp-content/uploads/2023/01/ICE-Title-Page-and-Instrument-v2.pdf.

Engagement involved in developing, implementing, or evaluating the assessment instrument
ICOH represented a three-year long effort that was seeking two years of continuation funding. The initial step to developing ICE was a thorough literature review that identified dimensions of effective coalitions. Content validity testing took place by matching the 61 items generated from the literature and “their corresponding theoretical constructs.” Next, an eight-person panel consisting of seven faculty from universities with experience working with coalitions in the areas of cooperative extension, tobacco, and substance abuse and one expert from a state health department working with local community coalitions were assembled. The panel reviewed the constructs and rated the degree of relevance between the item and the corresponding construct. The panel also considered if there were missing components for each theoretical construct. The 61 items from the literature analysis were reduced to 41 items, and the final instrument with 30 items was found to be psychometrically sound. Members and leaders of a large Midwest coalition focused on “tobacco prevention among youth and exposure reduction to second hand some in public places” participated in assessing the internal consistency and construct validity of ICE.²

Additional information on populations engaged in instrument use
ICE was mailed to 61 coalition members and leaders and had a 77% response rate. All the leaders and 67% of the members completed the instrument.²

Notes

• Potential limitations: The study demonstrates that the ICE can be applicable for use among “public health nurses working as evaluators for coalitions engaged in community health programing.” The limited sample size of this study may impact the ability to detect differences in responses from either members or leaders. Additionally, since the study focused its analysis on the individual coalition, it may be reasonable to expect that members and leaders from the same coalition would respond more similarly than those from different coalitions. Ultimately, the study and the ICE illustrate the importance of measuring perceptions of both members and leaders.²
• Important findings: The study findings demonstrate that the 30-item ICE is psychometrically sound. If there is a lack of congruence between the views of the members and leaders in each of the constructs, it could indicate a problem within a coalition. “The ICE provided coalition members and leaders with useful information for understanding various aspects of their internal effectiveness,” as well as “promoting coalition sustainability by identifying internal strengths and areas for improvement.”²
• Supplemental information: Additional research on tobacco control and on other topics (i.e., childhood injury, youth agricultural safety) has been conducted using the ICE. The findings from the research can be found in the following articles:
  • Cooper, T. V., J. A. Cabriales, T. Taylor, N. Hernandez, J. Law, and M. Kelly. 2015. Internal Structure Analysis of a Tobacco Control Network on the U.S.-Mexico Border. Health Promotion Practices 16(5):707-714. https://doi.org/10.1177/1524839914558513.
  • Cramer, M. E., and M. J. Wendl. 2015. Children’s Agricultural Safety Network: Evaluating Organizational Effectiveness and Impacts. Journal of Agromedicine 20(2):105-115. https://doi.org/10.1080/1059924X.2015.1010067.
  • Wendl, M. J., and M. E. Cramer. 2018. Evaluating Effective Leadership and Governance in a Midwestern Agricultural Safety and Health Coalition. Workplace Health Safety 66(2):84-94. https://doi.org/10.1177/2165079917729172.

We want to hear from you!

Assessing community engagement involves the participation of many stakeholders. Click here to share feedback on these resources, or email leadershipconsortium@nas.edu and include “measure engagement” in the subject line to learn more about the NAM’s Assessing Community Engagement project.