Participatory Evaluation Measurement Instrument

ASSESSMENT INSTRUMENT BACKGROUND

Context of instrument development/use
The articles noted that participatory evaluation (PE) approaches aiming to measure stakeholder participation in evaluation are increasing. The articles also emphasized that instruments are needed to understand the “necessary conditions for distinguishing participatory evaluation from nonparticipatory evaluation.”^1-3  ​

Instrument description/purpose
PEMI was developed to measure the participation level taking place in the program evaluation process.¹ PEMI is intended to be used to frame discussions about and assess stakeholder participation in evaluation using the following three validated (i.e., convergent, discriminant) focus areas:

• Extent of involvement
• Diversity of participants
• Control of the evaluation process

PEMI consists of three questions which are measured on a five-point ordinal scale ranging from .00 (absence of this instance of PE) to 1.00 (full presence of this instance of PE).³ In the original version of the PEMI, stakeholder participation was the minimum or lowest score of the three focus areas. In the revised version, the overall participation score is calculated by determining the average on the three focus areas.²

This instrument can be accessed here: https://nam.edu/wp-content/uploads/2023/01/PEMI-Title-Page-and-Instrument-v2.pdf.

Engagement involved in developing, implementing, or evaluating the assessment instrument
PEMI “was pilot-tested for clarity and readability by two university professors with significant expertise in program evaluation in general and stakeholder participation in particular.” After slight modification to the instrument, PEMI was sent to authors representing 40 case studies published between 1985 and 2010.³ Case studies were selected based on consideration for “diversity in terms of policy domains, origins of authors, and journals.” Case studies were diverse with respect to evaluation and stakeholder involvement approaches (e.g., collaborative, empowerment, stakeholder-based, utilization-focused, democratic-deliberative, community-based).² Case studies addressed “many policy domains, mainly education, health and human services, but also agriculture, local governance, environment, and international development.”³ The majority of author respondents (91.6%) completed a qualitative open-ended question of “why” to allow for “elaboration, enhancement, illustration, [and] clarification” of the quantitative score. The responses were analyzed and generated evidence for using “a less conservative concept structure for the revised version of the instrument.”²

Additional information on populations engaged in instrument use
The majority of study authors responding to the instrument had institutional affiliations in the United States (76.5%), with Canada (14.7%), Australia (5.9%) and South Korea (2.9%) also represented. 76.5% of study authors had university/academic affiliations, and 20.6% and 2.9% had non-academic and mixed affiliations, respectively.¹

Notes

• Potential limitations: The small, purposive sample of 40 cases, the limited qualitative data, and recall bias associated with cases published more than 25 years ago represented limitations. In addition, it was the “inferences derived from the instrument for this particular sample that were validated, not the instrument itself.” The last limitation cited was the “lack of sensitivity of the three-point quantitative scale to measure agreement.”²
• Important findings: “PEMI is a nonnormative measurement instrument in the sense that it does not assume that stakeholder participation is desirable (or undesirable). Yet it seems that respondents have high expectations toward their score for overall participation that cannot only be explained by a conservative bias in the PEMI.” During the completion of PEMI, respondents were asked to complete open-ended questions and participate in informal email exchanges about their responses to the instrument. The unexpected qualitative data (authors only expected a few responses to the open-ended questions, but received an abundant amount of qualitative data) were reviewed, underwent thematic analysis, and were used to revise the instrument and collect additional quantitative data for analysis. PEMI, with the inclusion of a modest quantitative component, addresses concepts of participation and aligns with respondents’ beliefs about the level of participation of their project evaluations.² The study results demonstrate that PEMI scores are both reliable and valid.^2,3

Three overarching themes emerged from the data: 1) there appeared to be positive alignment between PEMI case scores and respondent opinions of the level of stakeholder participation that took place during the evaluation; 2) where there was disagreement of the participation score, respondents unanimously believed the score was too low; 3) “many respondents explicitly mentioned or alluded to the normative power of stakeholder participation, either to embrace or criticize it.” Ultimately, respondents suggested that PEMI underrepresented stakeholder participation. PEMI was therefore revised to support a less conservative concept structure.²

• Future research needed: “Further empirical studies are certainly needed to establish the robustness of the findings presented. This is especially so with respect to the validation of the revised version of the instrument. The quantitative evidence reported here—although going in the expected direction—remains quite modest.”²