Presenters for the 2002 Summer Institute were asked to provide us with a tentative outline of their scheduled presentations at the institute. We asked them for specific information including goals, objectives, etc. Our intention was to provide a preview to institute participants as well as a means for communication of content among presenters.

What follows is the material we have received to date in the form submitted to us. Please remember that the specifics of presentations will likely vary some from these early outlines.

Project Management: A Key to Successful Evaluation

Presenter: Dr. Nanette Keiser, Senior Research Associate/Project Manager

Session Goal: To provide participants with basic knowledge, skills, tools, and other resources related to project management, with an emphasis on managing evaluation-related projects.

Outcomes: Participants will:

Learn basic project management knowledge and skills for any type of project.

Learn basic project management skills specific to evaluation-related projects through hands-on exercises drawn from the general management of a National Science Foundation program evaluation, its web-based surveys, and site visits.

Be introduced to tools and other resources for managing evaluation-related projects (e.g., software–Groupwise, Excel, Project 2000; checklists).

Hands-On Activities/Supporting Materials:

The session will use a mix of a PowerPoint slide presentation, demonstration, and interactive hands-on activities to enhance interest in the topic and learning through application. Brief demonstrations of various key features of software packages related to the program management principles presented will be interwoven throughout the session. Questions will be welcomed throughout the session.

Basic project management principles will be presented. A hands-on exercise in which participants will apply the principles presented to a scenario will follow.

These principles will then be related to evaluation-type projects. Overall project management issues for an evaluation will be discussed (e.g., managing deadlines, information, personnel, budget, meetings, other resources). After this discussion, project management related to two different methodologies often used in evaluations (surveys and site visits) will be presented. What worked and didn't work in managing a web-based survey system and 13 site visits across the U.S. for the NSF's Advanced Technological Education (ATE) program evaluation over the late three years will be linked to the presented project management principles. After each of the three examples is reviewed, the participants will be asked to form groups (or work in their existing groups) and work on various scenarios to apply the principles reviewed.

As a follow-up activity after this session, participants will be asked to briefly describe an evaluation project (or materials development project) they are working on and how they would organize its management using the principles and tools presented in this session. This will be provided to the instructor via email within one week of the session.

Feedback Activities:

Feedback will be provided to participants by the instructor during the hands-on activities described above. Feedback will also be provided to participants by the instructor via email for the follow-up activity described above within one week of receipt of their documents.

Materials to be Provided include:

Presentation Outline
Scenarios
Basic Project Management Principles/Checklist
List of Software Useful in Project Management
Resource List for Project Management
A Checklist for Managing Web-Based Surveys
Resource List for Web-Based Surveys
A Checklist for Effective Site Visits
Resource List for Site Visits
Copy of PowerPoint presentation

Summer Institute Sessions

June 18, Utilization-Focused Evaluation.

Overall Goal: Participants will learn an evaluation process, Utilization-Focused Evaluation, that enhances use of evaluation from beginning to end.

Specific Outcomes: Participants will learn

• the fundamental premises of utilization-focused evaluation;
• the implications of focusing an evaluation on intended use by intended users;
• options for evaluation design and methods based on situational responsiveness, adaptability and creativity; and
• how to use the utilization-focused evaluation checklist and flowchart.

Feedback processes: Portions of the sessions will involve small group work in which participants report out to the full group and get instructor feedback.

Activities and materials: The sessions will combine lectures on utilization-focused evaluation with concrete examples and case analyses, including cases provided by the participants. Exhibits from the textbook Utilization-Focused Evaluation (Patton, 1997) will be provided.

Hands-on activities: The morning will include a small group case analysis. The afternoon will include small group work on participants' cases.

June 19, Qualitative Evaluation Methods.

Overall Goal: Participants will learn the strengths and weaknesses of qualitative evaluation methods, and how to apply those methods in evaluations.

Specific Outcomes: Participants will learn

• the strategic themes of qualitative inquiry;
• the kinds of evaluation questions for which qualitative inquiry is particularly appropriate, and mixed methods approaches;
• purposeful sampling strategies and issues;
• interviewing and observation methods; and
• analytical and communication approaches that support useful evaluation.

Feedback processes: Portions of the sessions will involve small group work in which participants report out to the full group and get instructor feedback.

Activities and materials: The sessions will combine lectures on qualitative inquiry with concrete examples, including problem situations provided by the participants. Exhibits from the textbook Qualitative Research and Evaluation Methods (Patton, 2002) will be provided.

Hands-on activities: The morning will include a small group analysis exercise. The afternoon will include small group design work on participants' cases.

Christopher Nelson

Session Goals and Outcomes:

Recent federal legislation and administrative directives seek to elevate the scientific status of educational evaluations through greater attention to scientific rigor. The No Child Left Behind Act, the Bush administration's strategic plan for the Department of Education, and various pieces of pending legislation, for instance, promote the use of randomized experiments in the determination educational programs' merit and worth. This session seeks to explore the role of scientific rigor in educational evaluations through consideration of the following questions:

• What is scientific rigor?
• Must scientific rigor entail quantification?
• What other criteria should guide educational evaluation (e.g., relevance, utility) and how should evaluators and evaluation users balance these criteria?
• How should these standards inform (a) the synthesis of previous evaluation findings and (b) decisions about how to invest in future evaluation activities?

Feedback and Hands-on Activities:

The session will begin with an interactive lecture in which the issues discussed above will be presented and developed. During the middle portion of the session participants may break up into small groups to discuss a number of brief cases that involve the application of standards of scientific rigor. A final plenary session will allow the groups to present and receive feedback on their ideas from the instructor and other participants.

Activities, Supporting Materials, and Follow-up:

Materials will include a manuscript on the topic (a work in progress) and a one-page handout with a checklist of the key issues and criteria developed during the presentation. Session participants will be encouraged to provide the instructor will feedback on both checklist and the manuscript.

Lesson Plan for June 14, 2002

Literature Review

Arlen Gullickson

Instruction Goal: The session is designed for participants to learn how to access and use literature sources to inform and assist the evaluator in framing evaluation plans and preparing evaluation measures, protocols, and materials.

We will address five general issues:

• What constitutes a good review of literature?
• How do you access and assure long-term viability of a reference source?
• What are viable mechanisms to organize a literature review?
• What steps should be employed to establish literature review objectives?
• What are viable ways to use diverse and sometimes conflicting information?

Expected Outcomes: From this exercise each participant should gain (a) a set of criteria that can be applied to address each of the five identified points and (b) sound, clear strategies for conducting literature reviews that meet these criteria. Secondarily, if data are gathered and prepared effectively, the findings from the class can be used to frame a set of papers for presentation and/or publication.

Feedback Activities: Each participant's contributions will be used as part of the foundation for group presentations. As such the presentations provide some feedback regarding the substance of individual's knowledge and skill in the targeted area. Additionally, Dr. Gullickson will be available to meet with participant groups prior to the class session (as presentations are being prepared) and after the class session. The last part of the class session will be devoted to feedback as part of the general discussion on the five topics.

Specific Activities: This session requires pre-institute preparation on the part of participants. Each participant is asked to conduct the following homework and prepare and submit (in electronic form) the following materials by June 1, 2002:

1. Interview one person to obtain that person's perspectives on the five above points. From that interview the participant should:
   1. Briefly describe the qualifications of the interviewee.
   2. Take notes on and summarize the interviewee's response for each of the five points (list the item number and question preceding your summary).
2. Select a literature review source (or a study that effectively uses literature sources) that you deem to be good. (You may choose to use only part of a referenced study, e.g., literature that makes the case for an instrument or evaluation protocol)
   1. Cite the review reference.
   2. Provide a brief abstract/summary of the review (bring a copy of the referenced material with you to the Institute).
   3. Report how this literature review addresses each of the five general issues. Answer each separately (precede each answer with the item number and question statement).

Attention to Hands-on Involvement by Participants: Between June 1 and June 10 all homework will be organized into the five issue categories. On June 10 the submitted materials will be given to participants. Each group of three participants will receive the submitted information for one of the issues along with directions for planning and presenting on that topic. On June 14 each group will report their "literature review" findings and conclusions for the assigned issue (5 to 10 minute presentation). Following these reports, we will discuss what has been uncovered and learned. Criteria participant groups should apply in preparation of their reports include: identification and description of key points pertinent to the issue, utility of the information for conducting reviews, and relevance to evaluators.

Developing Logic Models

Carolyn Sullins, Ph.D.

Goal: To gain the skills to create viable logic models with one's evaluation clients, to be used in part for developing more effective evaluations.

Expected Outcomes:

• Participants will be able to determine under what circumstances a logic model will be most helpful, and how complex the logic model should be.
• Participants will be able to articulate how developing a logic model can help the evaluator learn more about the program, and the program stakeholders understand the evaluation processes and results.
• Participants will be able to define the following components of a logic model, and explain why they are often important when developing an evaluation: mission, values, assumptions, goals, processes, intended outcomes (short term, intermediate, long term), and potential impact.
• Participants will develop skills in creating a logic model with their evaluation clients.

Activities and Supporting Materials:

I will start by presenting a brief power-point presentation covering: 1) the basic definition of a logic model and its various components; 2) the disadvantages and (primarily) the advantages of developing a logic model with the clients; 3) how to include the clients in the development of a logic model; 4) how to use the logic model to develop the evaluation.

Hand-outs covering the main points of the presentation, with room for note-taking, will be provided.

The presentation will be followed by role-playing activities (see next section for details).

After the role-playing, the session will be concluded with a whole-group question and answer session.

Interactive/participatory activities – Role-playing:

Participants will take turns role-playing evaluators and evaluation clients.

For the first example I will invite a volunteer to play the role of the evaluator, and I will play the role of a somewhat challenging evaluation client. The rest of the participants will observe. After this role play I will encourage the participants to give feedback to the "evaluator" and ask questions of both of us.

I will then ask the participants to pair up and take turns playing the role of the evaluator and the client.

In all cases, the evaluators role will include the following:

• Introducing the concept of the "logic model" to the client
• Explaining the benefits of the logic model to the client
• Describing each step of the logic model
• Soliciting input into developing the logic model

Feedback activities:

During and immediately after my powerpoint presentation, I will allow time to answer brief questions about logic models, in order to further facilitate the participants' understanding of the issues.

The role-playing sessions will allow plenty of time for feedback amongst the participants. There will also be opportunities to ask me questions regarding their specific role-played situations, and how the "evaluators" in these role-plays can most effectively communicate their ideas to their "clients."

After the role-playing, there will be time allotted for the participants to reflect on the experience to the entire group, and to ask me more questions about the processes of developing logic models. I will answer each of these questions to the entire group, so that everyone can benefit.

2002 SUMMER INSTITUTE

Session on Reporting, Editing, Recordkeeping

Gullickson/Veeder

June 17, 2002

1. Goal for the Session: To provide participants with opportunities to think about, extract, and identify elements that are most useful in reporting, editing, and recordkeeping in their own evaluation practices and how these elements relate to specific Program Evaluation Standards
2. Expected Outcomes for Participants: Each participant should be able to (1) identify key concepts/factors of reporting, editing, and recordkeeping to use in their own settings and contexts and (2) develop a matrix of key factors and use it in making their team presentations (use of matrices should be part of presentation evaluation).
3. Explicit Feedback Activities: Gullickson and Veeder will look over the matrices developed by each team and will provide formative feedback.
4. Specific Activities and Supporting Materials to be Used: Example matrix (provided in advance of the session), source books, The Program Evaluation Standards, transparencies, Feedback Workshop Checklist, references to different kinds of reporting options (e.g., PowerPoint presentations), example reporting formats
5. Hands-On Activities and Interaction During Session: Gullickson and Veeder will spend 9-10 minutes per cell in a 9-cell matrix focusing on key attributes in a report; participants will spend about 5 minutes per cell identifying essential elements and begin developing a personal report matrix, taking into consideration their own contextual issues. When all the matrices are completed, they will be turned in to Gullickson and Veeder for feedback. The outline will be presented to the external evaluation team.

2002 SUMMER INSTITUTE

Session on Joint Committee Evaluation Standards

Sanders/Veeder/Wingate

June 12, 2002

1. Goal for the Session: Participants will become familiar with the Joint Committee Standards
   
   
2. Expected Outcomes for Participants: Participants will learn how the Joint Committee Standards were developed, learn the content of the Program Evaluation Standards, apply the Program Standards to a case, identify issues and limitations of the Program Evaluation Standards, and identify uses of the Program Standards.
   
   
3. Explicit Feedback Activities: Participants will apply the Standards to a case. Strengths and weaknesses of the case related to each standard will be reported to the group.
   
   
4. Specific Activities and Supporting Materials to be Used: The primary activity will be group application of the Standards to a case. Supporting materials include a background paper on the Joint Committee, a handout describing procedures for developing national standards, a case study, and lists of standard statements for all three sets of standards.
   
   
5. Hands-On Activities and Interaction During Session: Approximately two-thirds of the session will be devoted to hands-on activities and interactions with participants based on a case study application of the Program Evaluation Standards.

Evaluation Resources

Outline for the 2002 Summer Evaluation Institute

Dale Farland

Monday, June 10, 2002

Goals: The participants will learn about evaluation resources available on The Evaluation Center Web site and about some of the additional materials linked from the Center's Web site. Participants also will be invited to contribute additional items and suggest improvements to the site.

Outcomes: Participants will

• explore the Center Web site, particularly the Support Services section.
• add one or more annotated bibliography entries to the Web resources.
• recommend improvements for the Web site.

Feedback: Participants will share their Web resource additions with the opportunity to receive feedback. Recommendations for improvements to the Web site will be compiled and redistributed as the basis of a discussion of the recommendations.

Activities and supporting materials: An activity sheet will be distributed to invited participants prior to the institute. Participants will be asked to review selected parts of the Web site and return recommendations. Participants also will be asked to prepare Web resource additions prior to arrival at the institute. Session time will be spent discussing contributed resources and Web site improvement.

Planning for the 2002 MTS Summer Institute

Presenter/Facilitator: Jerry Horn

Topics:

Project Planning (to include evaluating and responding to RFPs, preparing the proposal, and contracting), Budgeting, and Staffing

Time/Date:

8:30 a.m. - 12:00 p.m. and 1:00 p.m. - 3:00 p.m. on Monday, June 24,
and 8:30 a.m. - 10:00 a.m. on Tuesday, June 25

Approach:

Rather than attempt to divide these as separate and distinct topics, I will integrate them into a single, continuous learning session that combines oral presentations (supported by visual slides) with discussions of real examples and work on sample cases or practice projects.

Materials:

• The Program Evaluation Standards (The Joint Committee on Standards for Educational Standards publication)
  
  
• Discussion/talking points slides (hard copies of slides to be handed out at the beginning of the applicable session for note taking and future reference)
  
  
• Samples of work plans, budget worksheets, and budgets
  
  
• Abbreviated descriptions of sample evaluation projects and RFPs
  
  
• User-Friendly Handbook for Project Evaluation (NSF publication)
  
  
• User-Friendly Handbook for Mixed Method Evaluations (NSF publication)
  
  
• Checklists (Checklist for Developing and Evaluating Evaluation Budgets, Evaluation Contracts Checklist, and Evaluating Plans and Operations Checklist)

Goal:

The goal of this session is that all participants will be knowledgeable enough about RFPs and evaluation planning to respond in an appropriate, professional, and competitive manner.

Objectives/Outcomes:

Participants will be able to:

1. Evaluate RFPs using appropriate Program Evaluation Standards and determine whether or not to respond
   
   
2. Develop an evaluation work plan for sample cases
   
   
3. Determine tasks and project costs and integrate into a workable budget
   
   
4. Analyze evaluation plans and propose a staffing configuration that addresses the needs of the work plan and is responsive to the budgetary constraints
   
   
5. Recognize the differences between grants and fixed- and cost-reimbursable contracts
   
   
6. Use checklists for budgeting and contracting for projects as a part of a practical activity

I anticipate that the various topics will be opened with a brief (15 minute or so) introduction to the topic followed by an examination of sample products of applicable materials, i.e., copies of RFPs, sample work plans, proposed or actual budgets of EC projects, etc. After an opportunity for participant questions and requests for further explanation, groups of no larger than 5 persons will be formulated to work on sample assignments on such tasks as developing an evaluation work plan, a functional summary and detailed budget for a potential project, etc. It is planned that about equal time will be devoted to presentations by the facilitator, reviewing sample materials, and developing plans, budgets, etc. in small groups.

Participants will receive feedback through (1) questions and answer opportunities, (2) individual reaction to work in progress (by facilitator) on such tasks as developing an evaluation work plan, staffing arrangement, and budget, and (3) group and facilitator reaction to plans presented to the group and facilitator.

Most of the time made available to this presenter/facilitator on Tuesday morning (second day) will be devoted to an expanded question and answer session. In effect, this session will be largely responsive to the needs/interests of the participants. They will be able to ask questions about topics/issues for which they feel a need for more information, clarification, or information about other learning/informative materials. Group work on budgets will collected at noon on Tuesday, June 25, 2202, and the facilitator will make notes and evaluative comments on them before they are handed back and/or discussed with participant groups before Friday, June 28, 2002. Further, the facilitator will be available in the EC during the week to meet with participants and discuss individual inquiries or issues.

The Materials Development Process

• Demonstrate an understanding of the relationship between the curriculum design process and the evaluation of educational materials
• Analyze the appropriateness of various assessment methods to the evaluation process
• Develop an assessment plan for evaluation of the efficacy of educational materials

• All handouts
• Assessment methods with glossary
• Materials development white paper (including materials development checklist)
• Scenarios

Survey Sampling

June 17, 2002

Session Goal: To provide participants with basic knowledge, skills, and insights related to writing evaluation reports.

Specific Objectives and Outcomes:
Participants will:

• learn how to structure and organize reports
• review logistical details associated with report writing
• gain insights into how to budget for report writing and the dissemination of reports
• appreciate and understand the importance of linking reporting format to various audiences

Feedback Activities: Group work (brainstorming sessions), sharing of experiences by participants, and the opportunity to raise questions will provide multiple routes for dialogue and feedback.

Supporting Materials: Sample evaluation reports will be passed around during the session to illustrate the need to target specific reports to different audiences. A research assistant will synthesize results from brainstorming activities which will be projected on the screen during the session.

Hands-on Activities: The session will include group interaction in the form of brainstorming about the contents of typical evaluation reports, actual costs for preparing and disseminating reports, and logistical issues related to report writing. Immediately following the session the results from the brainstorming sessions will be edited and summarized by the instructor and shared with the participants.

Return to top

Instructional Design and the Evaluation of Instructional Materials

Expected Outcomes:

Participants will be able to -
• Explain the basic progression of human performance and the performance equation;
• Describe the role and purpose of instruction as it supports human performance (as compared to the roles of between information and education);
• Name the key activities of instructional analysis and instructional design;
• Describe and explain the key elements of instruction;
• List the major blocks to acquisition and transfer of learning beyond the classroom;
• Name the four levels of evaluation and their relationship to instructional design processes;
• Discuss the role that training plays in the overall evaluation of a project;
• Describe the relationship between instructional design and evaluation, and explain the importance of this relationship;
• Apply the performance-based evaluation model for the evaluation of instructional projects.

Session Activities/Follow-Up/Resources:

1. Prior to the session, participants will complete pre-reading from Mager (1999) What Every Manager Should Know About Training
2. During the session, participants will
   1. Engage in a topic overview by the presenter
   2. Discuss the following questions regarding the presentation overview:
      • What ideas presented here have changed your understanding of the purpose of instruction?
      • What ideas presented here have changed your understanding of the purpose of evaluation?
      • What implications does this presentation have for the evaluation process?
   3. Apply the performance-based evaluation model to a case study of an instructional project
   4. Discuss the strengths and weaknesses of the performance-based evaluation model for the evaluation of instructional projects.
3. After the conclusion of the session, participants will
   1. Apply the key elements of instructional to develop materials for the MTS Summer Evaluation Institute;
   2. Consider a current evaluation project and identify appropriate changes to the evaluation strategy through the application of the performance-based evaluation model;
   3. Critique the performance-based evaluation model based on lessons learned during the MTS Summer Evaluation Institute about the Program Evaluation Standards
   4. Participants will select one of the three activities above and send their work to the presenter via e-mail.
4. Materials to be provided to the students for this session include:
   • Robert Mager (1999), What Every Manager Should Know About Training
   • Presentation outline
   • Topic materials
   • Performance-based evaluation model framework
   • Performance-based evaluation method protocol
   • Case study materials
   • Activity sheets for session activities

Feedback Activities:

Participants will receive feedback from the presenter for the following activities:
• Topic discussion during the presentation
• Case study activity during presentation
• Post-session application activity (via e-mail)