EVALUATION

Since BEST was a new initiative, the FFC and BEST staff felt it was very important to place learning and evaluative thinking at the center of the experience. It was determined early in the project that the evaluation would be conducted by external professionals and that the process would:

measure success in meeting the primary goals and objectives; and
generate a report that could be shared with funders, participant organizations and capacity building providers.

Evaluation data would be vital in determining whether to continue and/or expand the collaborative.

Evaluation Design

BEST has engaged the services of two nationally known organizations in separate, but connected evaluative processes: the TCC Group (formerly The Conservation Company) and the Center for Nonprofit Management (CNM).

Project effectiveness effectiveness. TCC Group is measuring the effectiveness and impact of the project as a whole over time. Findings can be broken down by cohort, and also considered in the aggregate. The work undertaken by TCC may be described as a meta-evaluation of the BEST Project: that is, an evaluation of the whole initiative that will help us learn from all processes and elements that have gone into forming and implementing BEST. Included in this work are pre- and post-assessments for comparison and evaluation.
Agency Development Development. CNM conducts a quantitative assessment of each nonprofit agency and its board via a written survey. The report, when used in conjunction with the interviewing process, provides either confirmation of findings or opportunities for further exploration. CNM is conducting a survey-based formative evaluation with each nonprofit agency in order to enable both the nonprofit's and the organizers of BEST to learn from their experiences. Each agency receives a customized report and analysis that includes indicators for growth and development as well as those that need attention. Consultants work with agencies to integrate findings into their qualitative assessments and reports.

This quantitative assessment is re-administered after the second year of capacity building to provide a comparative analysis of pre- and post-BEST engagement. In addition, CNM has agreed to share raw data from internal and external assessments (while maintaining confidentiality of the data) with TCC, so that the final evaluation of BEST will reflect these critical quantitative measures. Each evaluation component provides perspectives that the FFC and BEST staff need to be effective in serving the community. The TCC evaluation contributes to understanding the cumulative effect of the initiative, which involves nonprofit's, the consultants who work with them, the Flint Funders Collaborative, project administrators, and the greater community. The CNM data provides a perspective on the effects BEST is having within each cohort of nonprofits, through data from pre- and post-capacity building surveys.

Evaluations

What BEST has learned, and is still learning, from the evaluation verifies the need for intentional capacity building with nonprofit agencies in the context of local realities. BEST is highly customized, expensive, and time-consuming, and it relies on careful attention to detail. The early data confirm the success of the BEST approach. BEST staff has collected and delivered informal and anecdotal information on a regular basis to the FFC. In turn, this information has been used to make program and strategy adjustments. Agencies have provided candid and honest feedback on programming which has been critical to the ongoing revision of the program design. These lessons help BEST leadership understand the processes of the initiative, make adjustments in programming, and engage in productive discussions. BEST is currently exploring ways to capture agencies’ feedback about their experiences and interactions with consultants, to understand more about the micro-level impact of the work on a more timely basis.

BEST is learning and adapting continuously and expects to do so for the life of the project. Just as communities and people change, so do needs. To be effective, philanthropists must demonstrate the same kind of leadership, management, technical and adaptive capacity as their grantees and technical assistance providers. Continuous learning, strategic management and evaluative thinking demand reliable information and a sound process and methodologies.

The evaluations help to show where improvements and adjustments are needed, as well as identify things that are important to the constituents of this project; things that need to continue to make the project a success.

For instance, by 2005 approximately two-thirds of BEST nonprofits reported increased communication and improved knowledge of capacity building throughout their organizations as a result of the assessment phase and capacity building activities. Over half the executives reported improvement in their ability to make organizational decisions and solve problems as a result of this involvement. As one executive stated: “The capacity building has taught us what high-functioning, structured leadership looks like.”

Questions for Consideration

What do we want to learn from this initiative?
What areas can be measured?
How can the impact of a collaborative effort be documented?
How important is it to be able to compare local results to models implemented elsewhere in the region or country?
What time period will give meaningful results?
What are the benchmarks for success?

--Project Guide Table of Contents--

Download the entire Project Guide (11.9 MB)