Assessment Instrument Summary

Assessing Meaningful Community Engagement

Coalition Self-Assessment Survey

Andrews, M. L., V. Sánchez, C. Carrillo, B. Allen-Ananins, and Y. B. Cruz. 2014. Using a participatory evaluation design to create an online data collection and monitoring system for New Mexico’s Community Health Councils. Evaluation and Program Planning 42:32-42. https://doi.org/10.1016/j.evalprogplan.2013.09.003.¹

Sandoval, J. A., J. Lucero, J. Oetzel, M. Avila, L. Belone, M. Mau, C. Pearson, G. Tafoya, B. Duran, L I. Rios, N. Wallerstein. 2012. Process and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health Education Research 27(4):680-690. https://doi.org/10.1093/her/cyr087.²

Allies Against Asthma. 2003. Allies Against Asthma Evaluation Instruments.³

KEY FEATURES

COMMUNITY/ GEOGRAPHY
Community coalitions
Health councils
Pediatric asthma
New Mexico
United States

COMMUNITY ENGAGEMENT OUTCOMES
Strengthened partnerships + alliances
Broad alignment
Diversity + inclusivity
Partnerships + opportunities
Acknowledgment, visibility, recognition
Sustained relationships
Mutual value
Trust
Shared power
Structural supports for community engagement

Expanded knowledge
Broad alignment

Improved health + health care programs + policies
Sustainable solutions

PLACE(S) OF INSTRUMENT USE
Community/community-based organization
Local government agency

LANGUAGE TRANSLATIONS
Spanish

PSYCHOMETRIC PROPERTIES
Reliability

YEAR OF USE
2006-2007

Assessment Instrument Overview

The Coalition Self-Assessment Survey (CSAS)^1-3 has over 140 questions and sub-questions and is used by community coalitions. It assesses coalition functioning, leadership, and effectiveness of effort.

Alignment with Assessing Meaningful Community Engagement Conceptual Model

The questions in CSAS were realigned to the Assessing Community Engagement Conceptual Model. Figure 1 displays the alignment of the CSAS with the Conceptual Model domains and indicators. Where an instrument is mapped broadly with a domain or with a specific indicator, the figure shows the alignment in blue font.

Table 1 displays the alignment of the questions of the CSAS with the Conceptual Model domain(s) and indicator(s). The table shows, from left to right, the aligned Conceptual Model domain(s) and indicator(s) and the individual questions from the CSAS transcribed as they appear in the instrument (with minor formatting changes for clarity).

CONCEPTUAL MODEL DOMAIN(S) AND INDICATOR(S)	ASSESSMENT INSTRUMENT QUESTIONS
STRENGTHENED PARTNERSHIPS + ALLIANCES; Broad alignment with all indicators in this domain	Q19. Please circle a number to show how much you agree or disagree with each statement: a) The coalition is well managed Q26. Please circle a number to show how much you agree or disagree with the following statements: c) I am satisfied with how the coalition operates Q29. Please circle a number to show to what extent each of the following have been problems for your participation or your organization’s participation in the coalition: a) Coalition activities do not reach my primary constituency Q39. Please circle a number to show how much you agree or disagree with the following statements. c) In general I am satisfied with the coalition
STRENGTHENED PARTNERSHIPS + ALLIANCES; Diversity + inclusivity	Q6. In your opinion, does your coalition have sufficient representation from groups, organizations, and/or schools in your community to accomplish the objectives of the coalition? Q6a. If you answered “no” above, in your opinion, which type of the following groups, organizations and/or schools listed are NOT well represented on the coalition? Circle all that apply. Q6b. If you have circled one or more groups above as being not well represented, please select the single group you think is most important to add to the coalition at this time. Write the number of the group in this box: Q6c. Why do you think the group identified as most important to add to the coalition is not well represented at this time? (circle all that apply): Q20. Please circle a number to show whether the following functions are major, minor, not a function, or you don’t know: b) network with concerned citizens Q29. Please circle a number to show to what extent each of the following have been problems for your participation or your organization’s participation in the coalition: g) i am often the only voice representing my viewpoint
STRENGTHENED PARTNERSHIPS + ALLIANCES; Partnerships + opportunities	Q7. Is your coalition actively recruiting new members? Q20. Please circle a number to show whether the following functions are major, minor, not a function, or you don’t know: a) Network with other professionals Q21. Please circle a number to show how much you agree or disagree with the following statements: a) Relationships among coalition members go beyond individuals at the table, to include member organizations Q25. Please circle a number to show how many times over the last year you personally have done the following for the coalition: a) Recruited new members b) Served as a spokesperson c) Attempted to get outside support for coalition positions on key issues d) Worked on implementing activities or events sponsored by the coalition (other than coalition meetings) e) Acquired funding or other resources for the coalition Q27. Please circle a number to show how much you agree or disagree with the following statements. a) Staff from my organization contribute time to the coalition b) Volunteers from my organization contribute time to the coalition c) My organization supports the positions of the coalition publicly d) Overall, my organization is committed to the work of the coalition e) My organization contributes funds to support the coalition Q28. Please circle a number to show to what extent each of the following has been a benefit to your participation or your organization’s participation on the coalition: a) Developing collaborative relationships with other agencies g) Increasing my professional skills and knowledge* i) Getting access to key policy makers* Q29. Please circle a number to show to what extent each of the following have been problems for your participation or your organization’s participation in the coalition: e) My (or my organization’s) opinion is not valued
STRENGTHENED PARTNERSHIPS + ALLIANCES; Acknowledgment, visibility, recognition	Q14b. Circle the number that best represents your opinion of how much conflict within the coalition was caused by each of the following factors: g) Differences in opinion about who gets public exposure and recognition Q17. With respect to the leadership you just identified, please circle a number to show how much you agree or disagree with the following statements: b) Is respected in the community d) Is respected in the coalition Q29. Please circle a number to show to what extent each of the following have been problems for your participation or your organization’s participation in the coalition: b) My organization doesn’t get enough public recognition for our work on the coalition
STRENGTHENED PARTNERSHIPS + ALLIANCES; Sustained relationships	Q23. Please circle a number to show how much you agree or disagree with the following statements: a) Notification of meetings is timely b) Background materials needed for meetings are prepared & distributed in advance of meetings (agendas, minutes, study documents) c) Informative committee and/or task force reports are routinely made to the entire coalition Q26. Please circle a number to show how much you agree or disagree with the following statements: b) I go to coalition meetings only because it is part of my job Q32. Please circle a number to show how much you agree or disagree with the following statements: a) The current method for communication between coalition staff/leadership and its members is effective b) Members can communicate between themselves as necessary or desired c) The coalition staff facilitates communication between coalition members d) The coalition staff effectively and efficiently notifies me of meetings, agenda items, etc. Q38. Please circle a number to show how much you agree or disagree with the following statements: a) The coalition is making plans to continue operating after current funding is terminated d) The coalition will continue to exist beyond the Robert Wood Johnson Foundation grant period
STRENGTHENED PARTNERSHIPS + ALLIANCES; Mutual value	Q14b. Circle the number that best represents your opinion of how much conflict within the coalition was caused by each of the following factors: d) Personality clashes Q21. Please circle a number to show how much you agree or disagree with the following statements: f) Coalition members respect each others’ points of view even if they might disagree* Q28. Please circle a number to show to what extent each of the following has been a benefit to your participation or your organization’s participation on the coalition: b) Helping my organization move toward our goals c) Getting access to target populations with whom we have previously had little contact d) Getting funding for my organization e) Getting services for our clients f) Getting client referrals from others g) Increasing my professional skills and knowledge* h) Staying well informed in a rapidly changing environment* i) Getting access to key policy makers* j) Increasing my sense that others share my goals and concerns k) Getting support for policy issues our organization feels strongly about Q29. Please circle a number to show to what extent each of the following have been problems for your participation or your organization’s participation in the coalition: j) The coalition is competing with my organization Q30. From your organization’s perspective (if applicable), do the benefits of participation in the coalition appear to outweigh the costs at this point? Q31. From your own professional and/or personal perspective, do the benefits of participation in the coalition appear to outweigh the costs at this point? Q36. Has your coalition brought benefit to your community?
STRENGTHENED PARTNERSHIPS + ALLIANCES; Trust	Q14. Please circle a number to show how much you agree or disagree with the following statements: b) The coalition follows standard procedures for making decisions c) The decision-making process used by the coalition is fair d) The decision-making process used by the coalition is timely e) The coalition makes good decisions Q14b. Circle the number that best represents your opinion of how much conflict within the coalition was caused by each of the following factors: h) Procedures used for completing the work Q19. Please circle a number to show how much you agree or disagree with each statement: c) People know the roles of staff as compared to coalition members Q21. Please circle a number to show how much you agree or disagree with the following statements: b) I am comfortable requesting assistance from the other coalition members when I feel their input could be of value c) I can talk openly and honestly at the coalition meetings d) I am comfortable expressing my point of view even if they might disagree e) I am comfortable bringing up new ideas at coalition meetings f) Coalition members respect each others’ points of view even if they might disagree* g) My opinion is listened to and considered by other members* Q26. Please circle a number to show how much you agree or disagree with the following statements: d) I feel a strong sense of “loyalty” to the coalition
STRENGTHENED PARTNERSHIPS + ALLIANCES; Shared power	Q9. Of those that represent organizations, please circle the number which best represents your opinion about the number of members who participate in your coalition who have enough authority to make commitments of resources or other support for the coalition. Q10. Please circle the number below that shows how much influence you think the person or group has in deciding on the actions and policies for your coalition: a) Coalition Chair b) Coalition Officers or Committee Chairs c) Lead Staff d) Coalition Members Q11. Please circle a number to show how much influence you personally have in making coalition decisions. Q12. How are decisions usually made regarding coalition priorities, policies and actions? Circle the number of the main way(s) you think decisions are usually made. (CIRCLE NO MORE THAN TWO): 1. Coalition members vote, with majority rule 2. Coalition members discuss the issue and come to consensus 3. The coalition chair makes final decisions 4. The coalition executive or steering committee makes final decisions 5. The lead agency for the project makes the decisions 6. Don’t know Q13. Please circle a number to show how comfortable you are overall with the coalition decision-making process. Q14b. Circle the number that best represents your opinion of how much conflict within the coalition was caused by each of the following factors: a) Differences in opinion about coalition mission and goals b) Differences in opinion about specific objectives c) Differences in opinion about the best strategies to achieve coalition goals and objectives e) Fighting for power, prestige and/or influence f) Fighting for resources i) People aren’t sufficiently included in coalition processes/decision-making j) Member(s) who dominate the coalition meetings and impede proper collaboration Q15. Please circle the main strategy your coalition has used to address conflicts that occur. (CIRLCE NO MORE THAN TWO): Q16. Who do you think is most significant in providing leadership for your coalition? (CIRCLE ONLY ONE NUMBER): Q17. With respect to the leadership you just identified, please circle a number to show how much you agree or disagree with the following statements: f) Intentionally seeks other’s views g) Utilizes the skills and talents of many, not just a few h) Creates an appropriate balance of responsibility between leaders, staff and embers j) Builds consensus on key decisions k) Works collaboratively with coalition members n) Is skillful in resolving conflict o) Is ethical Q18. Who actually sets the agenda for meetings of the coalition and its committee/task forces? (PLEASE CIRCLE ALL THAT APPLY): Q19. Please circle a number to show how much you agree or disagree with each statement: d) Coalition members take responsibility for getting the work done Q21. Please circle a number to show how much you agree or disagree with the following statements: g) My opinion is listened to and considered by other members* Q22. Please circle a number to show how much you agree or disagree with the following statements: a) Our coalition has a clear and shared understanding of the problems we are trying to address b) There is a general agreement with respect to the mission of the coalition c) There is general agreement with respect to the priorities of the coalition d) Members agree on the strategies the coalition should use in pursuing its priorities e) Our action plan defines well the roles, responsibilities and timelines for conducting the activities that work towards achieving the stated mission of the coalition Q26. Please circle a number to show how much you agree or disagree with the following statements: a) I feel that I have a voice in what the coalition decides Q29. Please circle a number to show to what extent each of the following have been problems for your participation or your organization’s participation in the coalition: d) My skills and time are not well-used f) The coalition is not taking any meaningful action
STRENGTHENED PARTNERSHIPS + ALLIANCES; Structural supports for community engagement	Q8. In your opinion, do new members receive adequate orientation to be effective members of the coalition? Q14. Please circle a number to show how much you agree or disagree with the following statements: a) The coalition has clear and explicit procedures for making important decisions Q29. Please circle a number to show to what extent each of the following have been problems for your participation or your organization’s participation in the coalition: h) The financial burden of traveling to coalition meetings is too high i) The financial burden of participating in coalition activities (barring travel) is too high Q38. Please circle a number to show how much you agree or disagree with the following statements: b) The coalition has begun to find resources to continue operating after current funding is terminated
EXPANDED KNOWLEDGE; Broad alignment with all indicators in this domain	Q28. Please circle a number to show to what extent each of the following has been a benefit to your participation or your organization’s participation on the coalition: h) Staying well informed in a rapidly changing environment*
Improved Health + Health care Programs + Policies; Sustainable solutions	Q38. Please circle a number to show how much you agree or disagree with the following statements: c) Resources are being identified to support the systemic, programmatic changes implemented through the work of the coalition

*Note that these questions are duplicated to reflect their alignment with multiple domains and/or indicators in the Conceptual Model.

Table 1 | Coalition Self-Assessment Survey questions and alignment with the domain(s) and indicator(s) of the Assessing Community Engagement Conceptual Model

ASSESSMENT INSTRUMENT BACKGROUND

Context of instrument development/use
Two of the articles describe how CSAS was developed and implemented by two different groups to evaluate their initiatives. The first initiative, Allies Against Asthma, consisted of seven community coalitions focused on developing and sustaining “community-wide pediatric asthma control systems.” Funded by the Robert Wood Johnson Foundation, Allies Against Asthma used CSAS to conduct a cross-site evaluation of the community coalitions.³

In the other initiative, CSAS was implemented in New Mexico as part of the state’s aim to evaluate the impact of 32 county health councils’ “actions on local community health systems and health status outcomes.” Health councils in the state have received “legislative funding since 1991;” however, a state-wide evaluation to examine the effectiveness of the councils and the investment “(i.e., did health councils increase service integration or help change health behaviors in their communities)” had never been conducted. The University of New Mexico Master of Public Health Program, the New Mexico Department of Health Office of Health Promotion and Community Health Improvement, health council coordinators, and members of a community-based participatory research (CBPR) project developed a multistage participatory evaluation within the state’s health council system. As part of the participatory evaluation process to develop an online reporting system, CSAS was administered during stakeholder meetings, serving as an important foundational part of the effort.¹

Instrument description/purpose
Allies Against Asthma used CSAS “to capture quantitative information from coalition members on coalition structure and processes including coalition functioning, leadership, and effectiveness of effort.”¹ In New Mexico, CSAS was used to “assess internal council functioning and the council’s relationship with the Department of Health.”

CSAS assesses four areas:

Decision making
Leadership
Communication
Conflict resolution

CSAS includes over 140 questions and sub-questions. Questions use “yes/no” response options, as well as various Likert scale response options.³

The CSAS instrument in English and Spanish can be accessed here: http://www.asthma.umich.edu/media/eval_autogen/CSAS.pdf.

Engagement involved in developing, implementing, or evaluating the assessment instrument
Allies Against Asthma used an evaluation approach that was collaboratively designed with participation from “leaders from all seven community coalitions, the program’s National Advisory Committee members, and the Allies National Program Office staff.”³

The development of New Mexico’s health council web-based data collection and monitoring plan was collaborative. It was co-developed with representatives of the health councils, including coordinators, members from each region, and Department of Health staff. During the first of eight evaluation planning meetings, the statewide coalition survey was developed to help answer the two questions: 1) “What organizational structures and processes do county health councils have in place that support council development and council actions?” and 2) “What strategies and actions had county health councils implemented in their action plans for FY 07?”¹

Subsequent meetings were held in Public Health Division regions and included 15 or less participants or were held during statewide events where sessions included 25-100 coordinators, council members, and Department of Health staff.¹

Additional information on populations engaged in instrument use
Not specified.

Notes

Potential limitations: Several challenges emerged during the development of the online reporting, which led to important lessons learned, including: arriving at shared understanding of definitions and terminology; accommodating councils located in remote areas of the state with limited resources to travel; creating functionality and training to address technical issues; noting differences in voluntary participation among health council representatives and staff and satisfaction with the final product; and responding to increased work levels and satisfaction of council coordinators.¹
Important findings: The collaborative process of developing the evaluation and online reporting system fostered “a sense of connectedness among health councils” and allowed them to see their statewide impact. The credibility of the evaluation and the adoption of the reporting system may have been influenced by the voluntary involvement of health council coordinators who were experienced and knowledgeable. Further, “health council representatives were involved in all phases of system development,” placing “respect and inclusion of their perspectives” at the forefront of the process. “Through the online system, health councils reported data on intermediate outcomes, including policy changes and funds leveraged. The system captured data that were common across the health council system, yet was also flexible so that councils could report their unique accomplishments at the county level.”¹
Supplemental information: Additional research has been conducted using the CSAS with other populations. The findings and the resulting modifications can be found in the following articles:
- Rockler, B. E., S. B. Procter, D. Contreras, A. Gold, A. Keim, A. R. Mobley, R. Oscarson, P. Peters, V. Remig, and C. Smathers. 2019. Communities Partnering With Researchers: An Evaluation of Coalition Function in a Community-Engaged Research Approach. Progress in Community Health Partnerships 13(1):105-114. https://doi.org/10.1353/cpr.2019.0013.
- Sanchez, V., M. Sanders, M. L. Andrews, R. Hale, and C. Carrillo. 2014. Community health coalitions in context: associations between geographic context, member type and length of membership with coalition functions. Health Education Research 29(5):715-29. https://doi.org/10.1093/her/cyu028.

We want to hear from you!

Assessing community engagement involves the participation of many stakeholders. Click here to share feedback on these resources, or email [email protected] and include “measure engagement” in the subject line to learn more about the NAM’s Assessing Community Engagement project.