Skip Navigation

Comprehensive Community Initiatives, Improving the lives of youth and families through systems change, a toolkit for federal managers
HOME
How the toolkit was created What is a CCI? CCI Tools for Federal Staff
Develop your CCI Project
Guidelines to design evaluation
1. Integrate evaluation into the overall design and management of the CCI.
Why is it important to integrate evaluation into the CCI?

To reap the full benefits of evaluation, incorporate it into each phase of the initiative--from design and planning through implementation and the transition to sustainability. Integrating the evaluation into the CCI makes it possible to:

  • Demonstrate to stakeholders--including Congress, Federal agency leaders, local leaders, and others-- what the initiative has accomplished through collective efforts in the community.
  • Obtain feedback that enables funders and sites to make continuous adjustments and improvements.
  • Build a body of evidence about best practices that can inform future initiatives.
Why is a theory of change useful for integrating evaluation into a CCI?

A theory of change (TOC) is a useful starting point for evaluating a CCI because it:

  • Lays out the assumptions underlying the initiative.
  • States what results are expected and why the initiative's activities are expected to lead to those results.
  • Clarifies goals and strategies.

Usually a TOC is depicted visually in a logic model that shows how activities connect to results. For further information about theories of change, see "Evaluation Design," Page 14 in the Literature Review and http://www.aspeninstitute.org/policy-work/community-change/about-roundtable.

Theory of Change Approach to Evaluation
Evaluation experts advise using the theory of change (TOC) model as a starting point for evaluation of complex systems initiatives, and a number of foundations encourage this approach (Anderson, 2004, p. 3). A TOC is a "concrete statement of plausible, testable pathways of change that can both guide actions and explain their impact" (Kubisch et al., 2002, pp. 75). It specifies stakeholders' underlying assumptions about what will change as a result of an initiative and why. It provides a roadmap depicting the relationships between the initiative's strategies, interim outcomes, and long-term impacts, and produces testable assumptions regarding those relationships. Articulating a TOC requires the partners to specify why they are selecting a particular activity and what the intended outcomes are. It often involves backward mapping, where stakeholders think in backward steps from the long-term goal to the intermediate and early outcomes that would be required to accomplish the desired long-term outcomes. A TOC should be developed as part of the initiative design so that it can inform the development of initiative strategies. And the evaluation design should be tailored to the TOC, measuring the outcomes and impacts of each strategy (Coffman, 2007; Association for the Study and Development of Community, 2001; Kubisch et al., 2002; Anderson, 2004; Auspos and Kubisch, 2004).
How can I design the evaluation to serve as a tool I can use during implementation?
To make evaluation helpful throughout implementation...
  • Involve an evaluator in discussions with funders to map out a theory of change and create a logic model.
  • Build evaluation into the design and operation of the CCI from the beginning.
  • Plan to revisit and refine the evaluation process.
  • Consolidate performance measures.
  • Develop a process to report interim findings, and establish a feedback loop.

Involve an evaluator in early discussions with funders to help you map out a theory of change and create a logic model. The process of designing the evaluation will help you to think through the design of the CCI and ensure that the logic of the evaluation aligns with the rationale underlying the CCI. Allow ample time and resources to develop the theory of change and logic model. Plan to go through several iterations. (See A Framework for Evaluating Systems Initiatives.)

The changing role of the evaluator
New approaches to evaluating CCIs challenge traditional roles. When evaluations are built on TOCs [theories of change], evaluators are engaged with initiatives much earlier than they have been in the past. As they facilitate CCIs' efforts to define outcomes, activities, and pathways of change, the line between evaluation and technical assistance becomes blurred (Kubisch et al., 2002, pp. 70-72).

Build evaluation into the design and operation of the CCI from the beginning. If evaluation is an afterthought, you won't be able to gather the information for accountability, effective management, and knowledge development. Planning from the beginning is especially significant because of a CCI's complexity, as you'll be monitoring two kinds of change--people and systems. Fund the multisite evaluation before selecting sites. See Accountability Systems: Improving Results for Young Children and Guidelines for Completing the Performance Measures Section of OJJDP's Solicitations to learn more about integrating evaluation into the CCI.

Plan to revisit and refine the evaluation process as you learn more about what you are measuring. After you award funding, ask the evaluator to work with funders and community stakeholders to revisit the theory of change and the community's goals. Make sure that the logic still holds and the theory of change is still viable.

Consolidate performance measures. When there are multiple Federal partners, each will have its own congressional mandate, standards for accountability, and reporting requirements. Ensure that there are common performance measures or indicators that will address the requirements of all Federal partners supporting the CCI.

Develop a process to report interim evaluation findings, and establish a feedback loop. For a case example of how interim evaluation findings can be used to improve an initiative, see Midcourse Corrections to a Major Initiative (pages 5-6), and for an example of a website that helps grantees report their interim evaluation findings see, http://www.ojjdp.gov/grantees/pm/index.html .

  • The structure for interim reporting should mirror significant milestones in the implementation of the CCI. Example: You might ask sites to report at three points--after the first six months of implementing demonstration sites, at the end of funded planning phases, and at interim goal-achievement markers. Front-load the feedback with greater frequency at the outset.
  • Develop a process by which interim findings are presented periodically to Federal stakeholders and sites to create a feedback loop. Create processes and open communication for grantees to report back to their program officer.
  • "Focusing on pathways of change shifts the nature of evaluation away from a purely summative accounting of the CCI to a continuous process of providing feedback that can guide management and decision-making" (Kubisch, et al., 2002, pp. 70-72).
  • Use what you are learning along the way--an evaluation learning strategy--to make quality improvements in the supports you are providing to sites. Draw on evaluation data--along with an analysis of what's going well and what is not--to inform TA services.

References

Anderson, A.A. 2004. Theory of Change as a Tool for Strategic Planning: A Report on Early Experiences. Washington, DC: Aspen Institute.

Association for the Study and Development of Community. 2001. Principles for Evaluating Comprehensive Community Initiatives. Gaithersburg, MD: Association for the Study and Development of Community.

Auspos, P., and Kubisch, A.C. 2004. Building Knowledge about Community Change: Moving Beyond Evaluations. New York, NY: Aspen Institute Roundtable on Community Change.

Coffman, J. 2007. A Framework for Evaluating Systems Initiatives. Build Strong Foundations for Our Youngest Children Initiative. http://www.buildinitiative.org/docs/BuildInitiative.pdf

Kubisch, A. et al. 2002. Voices from the Field II: Reflections on Comprehensive Community Change. Washington, DC: Aspen Institute.