Making Evaluation Meaningful

This checklist was developed to aid evaluators in making evaluation more meaningful to the diverse makeup of education stakeholders in the current marketplace. With education reform in the forefront and accountability a key issue, community involvement in education has greatly evolved in the past few years. As a result, the makeup of education stakeholders has also changed dramatically. Collaborations, partnerships, and collective action are buzzwords in education today. But businesses, government, parents, media, and community partners do not speak the same language as the education community. It is almost the tale of two different worlds that overlap the public and private sectors. Even though both are focused on issues of accountability, measurable outcomes, and connecting evaluation to strategic planning in school reform, the ideology and modus operandi of the education sector and larger community are quite different.

Evaluators need to recognize the current makeup of education stakeholders and format their products (reports) to be more meaningful and understandable to noneducation stakeholders in the community. This is critical in order to attract and sustain corporate funding and involvement in education, assist school board members in making important decisions about goals and objectives, and optimize the partnerships for overall gains in schools. By using an appropriate evaluation design and delivery method, evaluators can create a win-win situation for all and can impact major policy decisions.

The following checklist is organized on three levels:

Assessing the Customer Base (preevaluation)
Formatting the Evaluation (during and postevaluation)
Disseminating the Information and Educating the Stakeholders (postevaluation)

return to Checklists	Making Evaluation Meaningful to all Education Stakeholders pdf Paula Gangopadhyay[1] September 2002
This checklist was developed to aid evaluators in making evaluation more meaningful to the diverse makeup of education stakeholders in the current marketplace. With education reform in the forefront and accountability a key issue, community involvement in education has greatly evolved in the past few years. As a result, the makeup of education stakeholders has also changed dramatically. Collaborations, partnerships, and collective action are buzzwords in education today. But businesses, government, parents, media, and community partners do not speak the same language as the education community. It is almost the tale of two different worlds that overlap the public and private sectors. Even though both are focused on issues of accountability, measurable outcomes, and connecting evaluation to strategic planning in school reform, the ideology and modus operandi of the education sector and larger community are quite different. Evaluators need to recognize the current makeup of education stakeholders and format their products (reports) to be more meaningful and understandable to noneducation stakeholders in the community. This is critical in order to attract and sustain corporate funding and involvement in education, assist school board members in making important decisions about goals and objectives, and optimize the partnerships for overall gains in schools. By using an appropriate evaluation design and delivery method, evaluators can create a win-win situation for all and can impact major policy decisions. The following checklist is organized on three levels: Assessing the Customer Base (preevaluation) Formatting the Evaluation (during and postevaluation) Disseminating the Information and Educating the Stakeholders (postevaluation) The rationale supporting each recommended step is provided in parenthesis.
1. Assessing the Customer Base.View the evaluation work from the stakeholders’ perspectives. Consider your evaluation as ‘the product’ and take into account the larger audience who will be using your product.
Determine who the evaluation’s stakeholders are—school systems, government, corporations, parents, media, etc. (This is critical to know up front, because it clearly delineates the context in which the evaluation results will be used.)
Examine the stakeholders' vested interests in the evaluation. (Vested interests of community stakeholders may be tied to the local economy, politics, and other social issues indirectly connected to the project being evaluated. It is in the best interest of the evaluator to assess up front some of these indirect links that may affect the use of the evaluation in the future.)
Identify who is funding the evaluation. (It is important for the evaluator to know the funding organization’s goals and involvement in the project. This may help set up the framework for intended use of the evaluation.)
Find out if the evaluation will be built in from the inception of the project. (In many instances projects do not involve evaluators while the project is ongoing; rather, they bring in evaluators late in the project to do summative evaluation with data collected by project staff. This limits the evaluation’s potential to help improve the project. Also, without early joint planning with the evaluator, the project staff may not collect all the information needed to assess the project’s merit and worth.)
Examine the existing research on the given issue. (It is important to be informed about the subject area in which you are evaluating as well as about similar previous evaluations. This will help you identify approaches, theories, and tools that may help you conduct the evaluation. It will also help you identify gaps in the research that your evaluation may fill.)
Find out the main points of any contrary research. (For your evaluation outcomes to challenge an existing point of view or prevalent assumptions, it is critical to address the claims of pertinent contrary research. Remember, these folks may challenge your evaluation findings, so place yourself in a strategic position by fully educating yourself about the opposition’s main points and rationale.)
Identify what policy implications the evaluation research may have. (This will help you assess how the evaluation may best generate long-term impacts.)
Find out which local legislators will be interested in your evaluation findings. (Every evaluation aims to impact policymaking. Local legislators are the actors who can make use of the findings effectively and advocate for the cause. They can become strong and highly influential proponents of your evaluation findings.)
Assess the local, state, and national media’s awareness, perception, and knowledge of the given issue. (Remember, media can make or break the public’s perception of a given issue.)
2. Formatting the Evaluation Report.Make sure your evaluation report is comprehensible to all stakeholders.
Package your evaluation findings in a CASCADING MANNER to help the reader navigate from the key points (top) to the detailed format (bottom). (Most nonacademic stakeholders will glance through the key points and will only refer to your actual detailed report “if need be.” Suggested sequence: (1) title page with project title and evaluation question(s); (2) table of contents; (3) author’s notes on approach taken/logic model; (4) one-page, bulleted summary of findings and related policy implications (most important; (5) executive summary; (6) at-a-glance bar graphs/pie charts delineating main findings; (7) other supporting research in the field; (8) your “in advance” counterpoints challenging any prospective opposing views; (9) full detailed report; (10) glossary of terms used in your evaluation report.)
Include simple, at-a-glance bar graphs, pie charts, and other graphic data displays. These are helpful visual aids for stakeholders, and business people especially are used to seeing information presented graphically.)
Clearly delineate the objectives for conducting the evaluation and provide author’s notes on how you approached the topic and the rationale for using certain research methods. (Assume that the reader does not know your rationale and approach.)
Always prepare a one-page, bulleted summary of the evaluation’s findings and related policy implications. (This is an alternative approach to the traditional executive summary. Most noneducation stakeholders are looking for a single page that identifies the evaluation’s main findings and their relevance to policymaking or education reform. This is your chance to make the first and lasting impression.)
List references that support or endorse your findings. (To demonstrate that your evaluation aligns with findings of other notable researchers.)
Cite your critics’ potential counterpoints and support your findings with arguments based on empirical data. (Most evaluators wait for critics to challenge their findings, then counter them with their own support points. Remember, the media and community stakeholders will not be interested in following a series of exchanges with supporting and counterarguments. To impact public opinion, make your point the first time.)
Address strands of education reform related to your evaluation results, and delineate the impact your findings may have on those issues. (Additionally, these can be excerpted to be included in the popular and widely read national- and state-level education publications and media reports.)
Make specific recommendations on how to use the evaluation findings effectively and connect them to the overall strategic planning and change-management process of the project/organization. (This will help the organization make effective use of the evaluation findings for long-term systemic change.)
Attach a glossary of research methods and terms used in your report. (Assume 80 percent of your readers will not understand the evaluation vernacular. Educate them to make your work more meaningful.)
3. Disseminating the Information and Educating the Stakeholders. Your role as the evaluator does not end with completing the report. To make a lasting and powerful impact on policy issues and in order to bring in systemic social change, your proactive involvement is needed in the postevaluation stages also. But remember, you can make or break your case depending on how you convey the message.
With your client’s approval, disseminate your evaluation to all the stakeholders and not just your client. (This helps a wider audience become aware of your research and opens the door for your involvement in future related projects.)
Disseminate your findings in a variety of ways to a wider and nontraditional audience. (Consider disseminating your evaluation findings through local/regional Chamber of Commerce newsletters, letters to newspaper editors, school newsletters, legislative briefs, Web sites, and feedback workshops with various stakeholder groups and other means. Remember the wider the outreach, the greater the impact.)
Make press releases catchy rather than detailed and wordy—-provide the one-page bulleted findings rather than the executive summary. (Chances of misreading bulleted points are far less than gleaning points from executive summary.)
Keep a log of who is accessing, citing, and using your evaluation report. (This is important for tapping into future clients as well as keeping track of how and where your evaluation findings are being used.)

[1] Author’s Profile: As the ex-Executive Director of Commission for Lansing Schools Success (CLASS), a major school reform initiative in Lansing, Paula Gangopadhyay gained valuable experience in helping disintegrate and interpret available data in a readily understandable format to various constituencies. Paula calls it “packaging in a user-friendly manner.” Her recommended format was applauded at the National Partners in Education conference as well as utilized by school districts for school reform action plans. Ms. Gangopadhyay is also an Education Policy Fellow from the Institute of Education Policy and has valuable experience in connecting evaluation outcomes to policy development processes. She is currently involved in evaluating the Michigan Middle Start Public Policy Public Engagement program as well as the Kalamazoo United Way Youth Development Initiative.

This checklist is being provided as a free service to the user. The provider of the checklist has not modified or adapted the checklist to fit the specific needs of the user and the user is executing his or her own discretion and judgment in using the checklist. The provider of the checklist makes no representations or warranties that this checklist is fit for the particular purpose contemplated by user and specifically disclaims any such warranties or representations.