Develop your CCI Project
Guidelines to design evaluation
1. Integrate evaluation into the overall design and management of the CCI.
Why is it important to integrate evaluation into the CCI?
To reap the full benefits of evaluation, incorporate it into each phase of the initiative--from design and planning through implementation and the transition to sustainability. Integrating the evaluation into the CCI makes it possible to:
Why is a theory of change useful for integrating evaluation into a CCI?
A theory of change (TOC) is a useful starting point for evaluating a CCI because it:
Usually a TOC is depicted visually in a logic model that shows how activities connect to results. For further information about theories of change, see "Evaluation Design," Page 14 in the Literature Review and http://www.aspeninstitute.org/policy-work/community-change/about-roundtable.
Theory of Change Approach to EvaluationEvaluation experts advise using the theory of change (TOC) model as a starting point for evaluation of complex systems initiatives, and a number of foundations encourage this approach (Anderson, 2004, p. 3). A TOC is a "concrete statement of plausible, testable pathways of change that can both guide actions and explain their impact" (Kubisch et al., 2002, pp. 75). It specifies stakeholders' underlying assumptions about what will change as a result of an initiative and why. It provides a roadmap depicting the relationships between the initiative's strategies, interim outcomes, and long-term impacts, and produces testable assumptions regarding those relationships. Articulating a TOC requires the partners to specify why they are selecting a particular activity and what the intended outcomes are. It often involves backward mapping, where stakeholders think in backward steps from the long-term goal to the intermediate and early outcomes that would be required to accomplish the desired long-term outcomes. A TOC should be developed as part of the initiative design so that it can inform the development of initiative strategies. And the evaluation design should be tailored to the TOC, measuring the outcomes and impacts of each strategy (Coffman, 2007; Association for the Study and Development of Community, 2001; Kubisch et al., 2002; Anderson, 2004; Auspos and Kubisch, 2004).
How can I design the evaluation to serve as a tool I can use during implementation?
To make evaluation helpful throughout implementation...
Involve an evaluator in early discussions with funders to help you map out a theory of change and create a logic model. The process of designing the evaluation will help you to think through the design of the CCI and ensure that the logic of the evaluation aligns with the rationale underlying the CCI. Allow ample time and resources to develop the theory of change and logic model. Plan to go through several iterations. (See A Framework for Evaluating Systems Initiatives.)
The changing role of the evaluatorNew approaches to evaluating CCIs challenge traditional roles. When evaluations are built on TOCs [theories of change], evaluators are engaged with initiatives much earlier than they have been in the past. As they facilitate CCIs' efforts to define outcomes, activities, and pathways of change, the line between evaluation and technical assistance becomes blurred (Kubisch et al., 2002, pp. 70-72).
Build evaluation into the design and operation of the CCI from the beginning. If evaluation is an afterthought, you won't be able to gather the information for accountability, effective management, and knowledge development. Planning from the beginning is especially significant because of a CCI's complexity, as you'll be monitoring two kinds of change--people and systems. Fund the multisite evaluation before selecting sites. See Accountability Systems: Improving Results for Young Children and Guidelines for Completing the Performance Measures Section of OJJDP's Solicitations to learn more about integrating evaluation into the CCI.
Plan to revisit and refine the evaluation process as you learn more about what you are measuring. After you award funding, ask the evaluator to work with funders and community stakeholders to revisit the theory of change and the community's goals. Make sure that the logic still holds and the theory of change is still viable.
Consolidate performance measures. When there are multiple Federal partners, each will have its own congressional mandate, standards for accountability, and reporting requirements. Ensure that there are common performance measures or indicators that will address the requirements of all Federal partners supporting the CCI.
Develop a process to report interim evaluation findings, and establish a feedback loop. For a case example of how interim evaluation findings can be used to improve an initiative, see Midcourse Corrections to a Major Initiative (pages 5-6), and for an example of a website that helps grantees report their interim evaluation findings see, http://www.ojjdp.gov/grantees/pm/index.html .
"Focusing on pathways of change shifts the nature of evaluation away from a purely summative accounting of the CCI to a continuous process of providing feedback that can guide management and decision-making" (Kubisch, et al., 2002, pp. 70-72).
Anderson, A.A. 2004. Theory of Change as a Tool for Strategic Planning: A Report on Early Experiences. Washington, DC: Aspen Institute.
Association for the Study and Development of Community. 2001. Principles for Evaluating Comprehensive Community Initiatives. Gaithersburg, MD: Association for the Study and Development of Community.
Auspos, P., and Kubisch, A.C. 2004. Building Knowledge about Community Change: Moving Beyond Evaluations. New York, NY: Aspen Institute Roundtable on Community Change.
Coffman, J. 2007. A Framework for Evaluating Systems Initiatives. Build Strong Foundations for Our Youngest Children Initiative. http://www.buildinitiative.org/docs/BuildInitiative.pdf
Kubisch, A. et al. 2002. Voices from the Field II: Reflections on Comprehensive Community Change. Washington, DC: Aspen Institute.
This Web site is maintained with assistance from the Office of Juvenile Justice and Delinquency Prevention,
a component of the Office of Justice Programs, U.S. Department of Justice,