Evaluating the Community-Engaged Organization

Chapter 14

Community Engaged Organizational (CEO) Evaluation

An assessment of an organization based upon data, its structure, activities and interactions that provide a foundation to measure performance, outcomes and impact within the scope of intended and unintended results

Crucial part of the planning process

Used to provide information necessary for informed decisions to shape future directions

The Need to Evaluate

Evaluate Community-Engaged Organizations (CEO) to:

Become more effective, efficient and attractive to funders

To provide increased accountability

To build a case for future investments in the organization or a particular program

Serve as a means to help organizations get the information they need to develop and execute strategies

Improve and maintain the quality of all programs, services and organizations

Determine here opportunities and challenges have occurred

Determine what accounted for success & failures

Determine what lessons were learned & how past lessons were applied

Strengthen organizational operations

Provide pathways to strengthen staff capabilities and performance

Framing The Evaluation

Framework must be established before an evaluator is identified to establish qualifications needed (internal vs. external)

Must establish scope (boundaries) of what is to be evaluated

The scope is used to align administration, stakeholders, staff and participants as to what to expect in the evaluation report

Living document / used and referred to when making organizational decisions

www.cdc.gov/eval/framework/index.htm

Review the above website for a framework for program evaluation

The Evaluator

Evaluator should be skilled and impartial

Internal Evaluator External Evaluator
Advantages: Low Costs Intimate knowledge of organization Advantages: Free from internal influences of organization Brings a fresh perspective
Disadvantage: Previously drawn conclusions Feels like an employee Disadvantage: High Costs Not as well versed in the organization’s climate
Infrastructure of the Community Engaged Organization (CEO)

Evaluation of an CEO answers the question, “Is the current organizational structure conducive to efficiently and effectively meeting the organization’s stated mission?”

Assesses the board, management and interactions with stakeholders

Can be done by external or internal evaluator

Usually involves staff members as data collectors

Data may include visions, missions, types of activities and involvement into the community

Needs Assessment

The evaluation may reexamine the very need for the Community Engaged Organization (CEO)

Usually there is a needs assessment of the community to be served and serves as the rationale for the organization’s existence

SWOT analysis is a tool used to analyze the needs assessment of the CEO and to revisit its purpose and has it changed, does the CEO still have the capacity to meet the community’s needs?

Strategies

Strategies are the assumptions and approaches a Community Engaged Organization (CEO) believes will lead to successful outcomes for the target population

Evaluating strategies of the CEO is the second thrust of the evaluation process

Are the strategies effective for program implementation and intended outcomes

Evaluation should provide continuous and timely feedback on meeting goals and objectives

Evaluation should address CEO’s mission, functions, methods and structure

Evaluations of CEO Programs and Strategies should answer the following questions:

How was success measured?

Were the strategies used appropriate?

Were the measures used appropriate given the stated strategy?

How successful have organizational strategies been in meeting past needs?

What changes are required in the strategies to meet identified needs?

Were objectives met?

What were the outcomes – short term, intermediate and long term?

Logic Model Example

Evaluation Logic Model Example

Goal:
Objective/Research Question Information Source Evaluation Activity Tool Analysis
What are board members’ perceptions of community needs? Board Members Interviews Interview protocol Qualitative assessments
How successful has the organization been in meeting its objectives? Program reports Annual reports Program data files Community Board Review program objectives and data analyses: focus groups interview Activity counts, inferential stats, focus groups and interview protocol Qualitative assessments and statistical testing
Was the program successful? Program Objectives Outcomes Participants Community Program data and activity information; focus group protocol Qualitative assessments And statistical testing
Evaluation Plan

Contains:

The elements of the Logic Model

The specifics needed to implement the evaluation

Data

Information sources

Data-gathering methods

Data-gathering instruments

Data analysis

Data reporting

Information Sources

Surveys

Interviews

Focus groups

Data files

Randomized sampling should come from a subgroup that best mirrors the larger population that is being studied

Consent forms

Providing an overview of the evaluation, guaranteed confidentiality and anonymity, risk factors and assurances of nonparticipation or withdrawal without penalty or threat of retaliation

Data Collection

Guided by the evaluation questions & outcomes that are to be measured

Information gathering should be only facts & perceptions that relate to the evaluation questions and outcomes

Evaluations for CEO usually contain both quantitative and qualitative data

Data-Gathering Methods and Instruments

Protocols should be developed for consistency

Interviews / Focus groups

First key question should lead into your topic

Next four-five questions should speak directly to your objectives for conducting the focus group

Last question is summary of what you heard

Survey administrators should make sure instruments are easy to code for data entry

Other sources may include organizational files, reports, case management reports, …..

All data should be kept confidential

Data Analyses

Descriptive section

History, composition, place in the community, achievements, future plans

Evaluation Questions or objective

Data

Records, interviews, focus groups, survey questions

Qualitative and/or quantitative data analyses

Findings and conclusions

The Evaluation Report

Report is for assessment of performance and effectiveness of CEO and how to improve their efforts for their community, better programs and better advocates

Report can provide framework that can be used to adjust strategies to accomplish goals and objectives

Outline for an evaluation report:

Executive summary

Introduction

Evaluation framework

Evaluation Plan

Findings regarding infrastructure and strategies

Conclusions, implications, and recommendations

Planning Process

Never ending

Begins with needs assessment & SWOT analysis

Vision Mission Goals Objectives

Strategies for reaching objectives

Strategies are then evaluated for feedback

Feedback is then used to reassess information found in needs assessment and SWOT analysis and process begins again

Situation

Goal

Rationales

Assumptions

Kentucky Worksite Wellness Tax Credit (HB 74)

To show the impact of the Kentucky Worksite

Wellness Tax Credit (HB 74)

Motivate companies to

implement effective

wellness programs

resulting in improved

employee health

Healthier families,

including spouses and

children

Health care costs could

decline, while job

opportunities could

increase

Healthier worksites

could lead to healthier

communities

Responsible

agency

Technical

assistance

Develop

certification

process

Expert at state

level

Marketing

Create employee

wellness model

Create awards

Conduct trainings

Tracking

Increase in number of

companies offering

wellness programs

Increase in number of

comprehensive wellness

programs

Increased number of

health promoting

policies in workplaces

Capacity for job growth

Increased access to

wellness programs

Healthier families,

including spouses and

children

Improved social

cohesion internally and

externally of worksites

in Kentucky

Resources

Activities

Outcomes

Blog A