JOINT COMMITTEE ON STANDARDS FOR EDUCATIONAL EVALUATION

SEPTEMBER 26 - 28, 1991

NORDIC HILLS, ITASCA, IL

MEETING MINUTES

Thursday, September 27

Dr. Sanders opened the meeting by welcoming everyone. For the benefit of new members of The Joint Committee, he asked Dr. Stufflebeam to give a brief history of the Joint Committee, its beginning, its purpose, and where it is today.

The primary purpose of this year's meeting was to work on a new version of the program evaluation standards. The new version will include applications to educational settings such as business and industry, military, government, higher education, legal, and health. New and combined standards will also be considered. The development of the new version will follow procedures specified in the Operating Procedures that were approved by the American National Standards Institute (ANSI), the agency accreditating the Joint Committee.

Dr. Sanders thanked Dr. Rosalie Torres for stepping in as Acting Staff Director earlier this past year when Dr. Edgar Kelley became ill. He also thanked Dr. Bruce Gould (Validation Panel Chair) and the members of the Validation Panel for attending this year's meeting. Dr. Gould said that he sees the Validation Panel as serving in a formative role. The Panel members will observe the task groups and then submit a report on the work done on the new program evaluation standards. They will be applying the new proposed metaevaluation standard during this process. They will also draft a mission statement and hope to have this to the group by the end of the meeting. Dr. Gould then introduced the members of the Validation Panel.

Dr. Sanders then asked the Committee members to introduce themselves and state what organization they represent. The members of the Executive Committee are Dr. Diana Pullin (Vice Chair), Dr. Ralph Alexander, Dr. Esther Diamond, and Dr. Sheila Simmons. Since Dr. Simmons has been replaced on the Joint Committee, an election was planned during the meeting to replace her on the Executive Committee. Dr. Sanders asked Dr. Pullin to chair a nomination committee with the other Executive Committee members serving with her as committee members.

Meeting objectives and a revised agenda were then reviewed. Much of the meeting would be spend on small group work. There will be a final business segment on Saturday to discuss items that are being reviewed by small groups earlier in the meeting.

Staff Director's Report

Dr. Torres noted that there are five new representatives on the Joint Committee. They are from the American Association of School Administrators (AASA), the American Evaluation Association (AEA), American Federation of Teachers (AFT), Council on Postsecondary Accreditation (COPA), and the National Education Association (NEA). Two current organizations, AASPA and NSBA, have vacancies at this time.

The Michigan School Boards Association has developed a checklist based on The Personnel Evaluation Standards and the Joint Committee has received permission to use this checklist for workshops. Dr. Torres stated that this checklist can be used for any scheduled workshops. This checklist was used at a workshop done under The Evaluation Center's Center for Evaluation and Accountability in Teacher Education (CREATE) at Western Michigan University by Drs. Daniel Stufflebeam and Michael Scriven.

This past year, Joint Committee members nominated writers for new cases for the program evaluation standards. Out of the 80 names received, 25 agreed to participate. We received 40 new cases from the 25 writers.

The Evaluation Center received $25,000 from the W. K. Kellogg Foundation to support work on the new program evaluation standards. These funds will be used to support the work of Dr. Rosalie Torres, Mary Ramlow, and the Validation Panel members in 1991. A new request for 1992 support is planned.

Other items of note: Dr. Stufflebeam wrote a chapter on professional evaluation standards. A graduate student intern from Western Michigan University, Prudence Pollard, developed a list of issues based on past feedback on the program evaluation standards. ANSI has been informed of the Joint Committee's work being conducted on the program evaluation standards.

Dr. Torres stated that a proposal was submitted to six foundations to develop standards for evaluating student needs and performance. This proposal received no favorable response. Responses stated that this proposal fell outside of the foundation's priorities.

Financial Report

Mary Ramlow reviewed the financial report. The current fund balance for the Joint Committee is $15,139. Sales of the program evaluation standards through December, 1990 were $15,196. Sales of The Personnel Evaluation Standards through December, 1990 were $6,828. She noted that The Evaluation Center has purchased 1,000 copies of each book that are not reflected in the sales report. This purchase reflects a dissemination goal supported by Project CREATE. Concerns about McGraw-Hill not publicizing the program evaluation standards and stating that the book is not available were discussed.

The New Program Evaluation Standards

Introduction

Drs. Berger and Bracey agreed to work on the new Introduction. They recorded discussions held in the large group, and incorporated them into the draft. Items discussed were passive or active voice, grammatical errors, expansion to cover the added fields, and whether the Standards are international or national. It was moved and seconded to have the Introduction written in APA format and use active voice. The motion carried. Grammatical errors and expansion to include the new fields should be done as Berger and Bracey write the Introduction. Even though the Standards are now in four languages, it is not international. Certain Standards are not applicable in some countries. The large group reviewed the old Introduction page by page. One goal of this discussion was to make the program evaluation standards parallel to The Personnel Evaluation Standards. With that in mind, the history of the program evaluation standards would be removed from the Introduction and incorporated into a Preface. This will be written by Dr. Sanders.

Dr. Stufflebeam stated that more response might be generated from the Invitation to Users if there was an occasional paper published referencing the user.

After lunch, discussion continued on the Introduction and the other parts of the book around the Standards.

It was suggested that footnotes be added were Standards have been changed significantly. Sanders asked that the members of small groups look at the illustrative cases and discuss them among themselves. Then decide if you want one, two, or a hybrid of 1 or 2 cases for each standard. Discussion continued on justification for the number of cases for each standard, a suggestion section for each standard, and the use of the checklist.

Task Assignments

Utility Standards

Philip Hosford (Chair)

Esther Diamond

Gretchen McDowell

Accuracy Standards

Todd Rogers (Chair)

Ross Traub

Ralph Alexander

Feasibility Standards

William Mays (Chair)

Daniel Stufflebeam

Edgar Kelley

Introduction Section

Marsha Berger

Gerald Bracey

Propriety Standards

Rosalie Torres (Chair)

Thurston Manning

Constance Filling

Dr. Sanders asked that members give their comments on the Standards to the appropriate chair. Put your name on those sections that you will want back. The group broke into small groups.

Friday, September 27

Dr. Sanders asked that the small groups reconvene until 11:45 a.m. At 11:45 a.m. the large group met for re-assignment to new tasks. The new tasks and assignments were as follows:

Group A

Daniel Stufflebeam (Chair)

William Mays

Diana Pullin

Assignment: Develop plans for review panels for the new standards, letters to be sent to reviewers, and forms or instruments. What was done before was given to the group. The task group was asked to make sure that we have representatives from all the settings that are being represented now. The reviewers will be asked to look at the entire book. The Validation Panel members may have names that we can use for the new settings.

Group B

Ross Traub (Chair)

Esther Diamond

Constance Filling

Assignment: Develop a plan for field testing, contact letters, forms and/or instruments, and other documents.

Group C

Philip Hosford (Chair)

Gretchen McDowell

Marsha Berger

Assignment: National Hearings. Develop a plan, where they should be held (such as association meetings, and if so, which ones?), and advertisements. Keep in mind costs, any forms or letters etc. that are necessary.

Dr. Gould expressed concern that the window is very narrow between this meeting and when these hearings are to be conducted. Keep in mind the organization groups, their meeting times, etc. Dr. Sanders anticipated being ready for spring hearings. Dr. Hosford clarified that the goal is to get feedback by the next Joint Committee meeting. Concerns were discussed about the turnaround time and when the hearings would be scheduled. We want to make sure that all settings are represented in these hearings, through sponsoring organizations and/or cooperating organizations.

Group D

Thurston Manning (Chair)

Gerald Bracey

Todd Rogers

Assignment: Review what we have in regards to proposals and what can be done to get more funds. Dr. Sanders asked Dr. Walser to join this group for input. The Foundation Directory was made available for any additional sources. The goal is to plan what should be done this year i.e., rewrite contact letters, different forms, etc.

Discussion continued on various ways to get additional funds.

Group E

Ralph Alexander

Edgar Kelley

Assignment: Work on Part Two of the Applications exercise and get to Mary Ramlow for handout on Saturday.

The large group dispersed into the new small groups and were asked to reconvene at 3 p.m. to check on group progress.

GROUP REPORTS

Group A: Review Panel

Stufflebeam reported that the letters and forms were basically sound but should be reviewed and corrected for the new program evaluation standards. The letters should also be reviewed to read from the other areas not just the schools. They suggested that there be an equal number of reviewers from North America and from international countries. They recognized that the interest from these newly added groups should represent at least 25 percent of members on the review panel. Group A questioned if the reviewers should read the whole book or just specific sections. It was agreed that the review panel should review the whole book. Discussion continued on the best way to contact possible members for this panel.

Group B: Field Test

Ross Traub reported that the group read the materials and considered the plan that was included. That plan gave a three prong approach but the group suggested using a plan with a two prong approach, volunteer and actively recruited categories. Principles for recruiting and role vs. use were discussed. Questions reviewed were how would the evaluator make use of the Standards, hire the contractors, and hire metaevaluators. Other categories could be internal vs. external evaluation, quantity vs. quality, and geographical location. Group B suggested that the Committee attempt to conduct 15 field tests. The group reviewed most of the documents but there are more to do. One concern stated by the Validation Panel was that it be specified in the letter to field testers what is a successful field test. These are specified in the implementation of the field test.

Group C: National Hearings

The past hearings' materials were reviewed. Dr. Hosford reported that the group felt that the objective of the hearings was to get a perspective from everybody and anybody. Four kinds of input would be 1) written responses, 2) oral input from the different meetings that could be scheduled and invite all to come and respond, 3) offer the opportunity to telephone and give their response(s) to the staff, or 4) after they have responded one way or another, that a phone call be available for additional input.

The group suggested that 1) assignments be made to make sure that some people from the organization respond to the book; 2) that a different letter be drafted for those people in the other fields with a history of the Joint Committee, where we are, and where we are trying to go; and 3) everyone knows someone back home that a third letter, that would be personalized, would used to follow-up a phone call made by the recommender. Suggested timeline would be that the letters go out November 15 with a deadline to respond by January 15 and how they are to respond to the materials. Then the manuscripts be sent out by February 1 and all returns no later than May 1 allowing time for phone follow-up to be done.

Dr. Hosford reported that the group recommended three meeting sites, ASCD, AERA and in Washington, DC. Washington would be a great place but deadlines are now past to schedule a meeting. We would get good attendance especially if people have been notified in advance of the meeting date and time.

Concerns discussed were the length of phone calls, and hearings being open to anyone in the area not just at the conference. It was suggested to get a copy of the mailing lists from organizations representing the new areas to assist in getting these areas covered. Advertising in those new area's organization newsletters and flyers announcing the hearings and inviting their participation would be other avenues to consider in reaching those people. Also using computer bulletin boarding would allow for easy access by all. These different approaches were discussed. Dr. Sanders asked the Validation Panel for assistance in placing those notices or announcements.

Dr. Sanders asked the Committee if we wanted to do the hearings at the same time as the field tests. His concern was that we will be getting two types of feedback on the same document. Is that what we want to do? There will not be major changes so if they are run at the same time, this would allow the office to only make one revision versus two. Discussion continued on the effectiveness of the hearings, ANSI requirements, and the time frame.

Group D: Funding Proposals

Dr. Manning reported that the group reviewed the student evaluation standards proposal. It needs to be more precise. He suggested that the proposal be sent to the American Society of Testing and Materials (ASTM), which established standards in the field and encourage developing standards in the social sciences. They undertake the secondary support while the Joint Committee is conducting the research. The Committee would loose control of the revision process because we would have to do it their way. ASTM would have the right to publish and would receive all royalties.

The group asked the Joint Committee staff to look into other sources for money. They suggested 1) looking where we can get a smaller amount of money, 2) including more explicit details, 3) having the proposal prepared with more documentation specifying how the student evaluation standards will improve things, 4) adding an abstract or an executive summary in the proposal not in the letter, 5) strengthening the budget by noting what happens to all the money (inkind, contributions, paying for group representative to attend annual Joint Committee meetings, and 6) making a personal contact before the actual proposal is sent is more appreciated. Discussion continued on possible sources for funds (advance in royalty from publisher, Barbara Nelson-Hadden from Ford Foundation) and soliciting assistance from Research and Development offices at Western Michigan University in development of the proposal.

Group E: Part Two of the Applications Section

They needed more time to complete this task.

Dr. Sanders adjourned Friday's meeting and asked the members to return at 8:30 a.m. Saturday morning to review the Validation Panel report, the Introduction, and conducted the final business meeting. He also asked the task groups that have work yet to finish to meet this evening to complete their charge.

Saturday, September 29

Dr. Sanders opened the meeting by asking Dr. Gould to present the Validation Panel report. Dr. Gould stated that the Panel was presented with five goals. The Panel revised those goals and wanted to make sure that they are in agreement with the Joint Committee. The statement Dr. Gould read is in Appendix I of these minutes.

Discussion ensued on the purpose of having international reviewers for the new program evaluation standards. Dr. Stufflebeam stated that the purpose was to get the widest possible perspective that we can get.

The Validation Panel report included seven topic areas. They were to 1) expanded context of the Standards, 2) data collection, 3) modification of the illustrative cases to include the added fields, 4) revision of the Application section, 5) following ANSI guidelines to maintain ANSI certification, 6) legal obligation/liability, and 7) review of the Invitation to Users section of the book.

Dr. Walser has the ANSI and ASTM guidelines for standards revision and the Validation Panel will review them to see if there are any vague areas in them that the Joint Committee should address. The Committee discussed the report with the Panel. Concerns mentioned were the short time frame, new standards to be more explicit, not losing critical comments in the development process, attempts to solicit demographic data, and how users could or would use the Standards.

Business Meeting

Dr. Sanders reported that Sheila Simmons was replaced on the Joint Committee because she has a new assignment at NEA. She asked that Dr. Sanders send apologies for her in not being able to attend this meeting. Her resignation left a vacancy on the Executive Committee. Earlier at this meeting, the Executive Committee (chaired by Dr. Pullin) was charged to prepare nominations for a replacement for Sheila.

The members of the Executive Committee nominated William Mays. He agreed to serve if elected. Dr. Pullin moved and Dr. Hosford seconded the motion to elect William Mays to serve on the Executive Committee. Hearing no objection, the motion carried.

Dr. Sanders reported that he received, and very sadly accepted, a resignation from Edgar Kelley as Staff Director. In accordance with the Operating Procedures, Dr. Sanders has asked Rosalie Torres to serve in that capacity. Dr. Torres agreed to do so.

Next year's meeting was then discussed. In order to allow more time to get assigned tasks done, Dr. Sanders suggested that next year's meeting time be moved to the end of October (10/29 - 11/1, 1992). He also suggested that the meeting be held at Nordic Hills again next year since it is close and cost effective for the staff. Check-in will be on October 28 with the meeting starting at 8 a.m. on October 29, ending at noon on November 1, 1992. It was also suggested to reduce the meals and alter the eating times to 1 p.m. for lunch and 7 p.m for dinner. With no further discussion desired by the Committee, Dr. Sanders asked for a vote. All members present agreed unanimously to the dates and location.

The next topic discussed was the contract with McGraw-Hill. Ben Diamond and Diana Pullin reviewed the current contract. Concerns mentioned were (1) McGraw-Hill stating that the book is out-of-print, (2) it's not being advertised, and (3) its not being available at conferences. On the other hand, Sage publishers have performed admirably in handling The Personnel Standards book. They have also mentioned an interest in marketing the two books as a set. Discussion continued on publisher rights, author rights, etc. Ben Diamond and Diana Pullin agreed that they will need copies of the original RFP and copies of the copyright agreements for the Program Standards, to check on the wording since the Program Standards are in the public domain. This is because the Standards were developed using public funds and therefore become public domain after ten years. Ben Diamond and Diana Pullin will continue to review the materials on this issue and report back to Dr. Sanders.

Dr. Sanders asked for an update on the Applications Section. Dr. Alexander reported that the decision was made to make an indepth revision of the Applications section. Dr. Kelley had recommended that a specific case would be very good as an overall case for this section. But only three of the four authors would agree to let us use it. It was suggested that Dr. Alexander write a new case in the spirit of the recommended case. Edgar Kelley, Marie Kelley, and Ralph Alexander have worked through one case. They have done a second case but will not have it done until after the meeting. Dr. Alexander will get it to Mary Ramlow to include in the review panel draft.

The second case is an example of the development of the evaluation process. The three suggested that a third case be developed for the application of the Standard to a training program that is being conducted concurrently. This would give a good overview and they are hoping to have it ready to put in the review process. Marsha Berger volunteered her office and their services to get that case done. Concerns were stated with accepting this offer and the rights to control.

Dr. Gould will meet with Dr. Sanders and the staff to discuss roles of the Joint Committee, the Validation Panel, and the staff.

Dr. Sanders reviewed the Operating Procedures and noted that the agreement with the hosting institution needs to revisited. This agreement can be renewed once. Sanders asked that the Executive Committee review that requirement and report back to him.

Dr. Pullin asked if the length of term of the chair was limited to only one renewal. After discussion of terms, Dr. Sanders asked the Executive Committee to review that requirement and submit a recommendation.

Consequently, Dr. Pullin moved that the language on the top of page 13 of the Procedures be amended as follows:

The phase "may be re-elected for additional terms" that the last sentence be removed.

Dr. Stufflebeam seconded. (This is segment 4.2.3.1 last paragraph). Dr. Stufflebeam added that similar wording in the by-laws be changed accordingly. Dr. Alexander noted that these changes should be sent to ANSI. Dr. Pullin stated that the Executive Committee should be the group to develop this change to the by-laws and withdrew her motion. Dr. Stufflebeam withdrew his seconded.

Dr. Alexander moved that The Evaluation Center should continue as a host institution of the Joint Committee. Dr. Berger seconded. Dr. Pullin stated that according to the Operating Procedures there is suppose to be a contract on file. One has never been developed. Dr. Pullin said that one needs to be developed and sent to Dr. Stufflebeam for his review. Dr. Berger hated to wait until next year on this item and asked if the Executive Committee could handle this. Dr. Alexander revised his motion to ask the Executive Committee to get an agreement with Western Michigan University for the host institution. With no further discussion desired by the Committee, the motion was unanimously approved.

Dr. Pullin also noted that an election will need to be held next year for Joint Committee chair.

Dr. Torres then reported on the Propriety Standards Group. They had major new content to recommend. She said the new sections involved grouping some of the standards, name changes to closer reflect their purpose, and development of a new service orientation standard. Discussion continued on the reasons for these changes. Dr. Sanders asked Dr. Manning to chair the writing of that new standard. Drs. Torres and Diamond volunteered to assist Manning. Dr. Manning stated that they will draft the standard and give a copy to the whole Joint Committee.

Dr. Hosford reported on the Utility Standards Group. The only concerns that they had questions on were that one standard used only schools as a setting in their cases and the use of Black or African American. Black or Black American are preferred.

The members unanimously agreed that the lettering of the standards be changed to U, P, F, and A versus A, B, C, and D. Discussion continued on the timeline for the completion of the revision of the program evaluation standards.

The Introduction was discussed next. After reviewing the changes made by Drs. Berger and Bracey, Dr. Sanders asked each member of the Committee to review this last draft and submit their revisions to Dr. Torres for compilation into a final form.

Dr. Sanders asked that each member also check the sponsoring organization list distributed earlier and make sure their information is current and correct. If not, please send corrections to Mary Ramlow. He also asked them to submit their fax numbers or electronic network numbers to Mary Ramlow to be added to the directory.

With no further items to discuss, the meeting was adjourned.

Minutes.Jc JC ANNUAL MTG #2