ANNUAL MEETING OF THE

JOINT COMMITTEE ON STANDARDS

FOR EDUCATIONAL EVALUATION

Hilton Head, South Carolina

September 30 - October 2, 1994

Attendees:

Joint Committee Members:

Dr. James R. Sanders, Chair

Dr. Robert Baker (APA)

Dr. Edith Beatty (ASCD)

Dr. Barbara Clements (CCSSO) (substitute)

Dr. Oliver Cummings (AEA)

Dr. Dorothy Fenwick (CORPA)

Dr. Henry Johnson (AASA)

Ms. Gwen Keith (CES)

Mr. William Mays, Jr. (NAESP)

Dr. Diana Pullin (Member-at-large)

Dr. Rodney Riffel (NEA)

Dr. W. Todd Rogers (CSSE)

Dr. Bruce Thompson (NCME)

Dr. W. Eugene Werner (NASSP)

Joint Committee Staff:

Ms. Mary Ramlow, Secretary

JOINT COMMITTEE ON STANDARDS FOR EDUCATIONAL EVALUATION

MEETING MINUTES

SEPTEMBER 29 - OCTOBER 1, 1994

HILTON HEAD, SOUTH CAROLINA

Friday, September 29, 1994

Dr. Sanders welcomed everyone to the 20th annual meeting of the Joint Committee. The objectives of this meeting were (1) to conduct the business of the Joint Committee on Standards for Educational Evaluation, (2) to review the final report from the Validation Panel for The Program Evaluation Standards, (3) to receive and act on recommendations from the Task Force on revising The Personnel Evaluation Standards and from the Task Force on future governance of the Joint Committee, (4) to develop a strategic plan for funding the next Joint Committee project, and (5) to develop a strategic plan for dissemination and training in the use of the Standards.

Dr. Sanders asked members of the Joint Committee to introduce themselves. He then reviewed the contents of the meeting packet. Dr. Sanders then gave a brief report on the history of the Joint Committee.

He identified the members of the Executive Committee. They are Dr. Diana Pullin (Vice Chair), Mr. William Mays, Jr., Dr. Bruce Thompson, and Dr. Gene Werner. The Executive Committee is the group that takes action on routine matters between annual meetings of the entire Joint Committee. As the members' terms end, replacements are elected to the Executive Committee.

Dr. Sanders welcomed the Canadian Evaluation Society (CES) to the Committee. They accepted our invitation to join this past year. Their representative is Gwen Keith. He also noted that COPA has continued their membership on the Joint Committee but under a new name and structure. The new organization is called CORPA (Council On Recognition on Postsecondary Accreditation). Their representative is Dorothy Fenwick.

Dr. Fenwick gave a brief update on the reorganization of CORPA. Ms. Keith gave a brief background on CES and handed out a booklet on CES.

In the Chairman's report, Dr. Sanders summarized highlights of work this past year. The drafted Program Evaluation Standards were reviewed, revised, and voted on by the members at last year's meeting. Since then, there was an additional review and final vote. This resulted in a document that was taken to Sage Publications, the publisher chosen at last year's meeting. A contract was reviewed and then signed with Sage, and the book was published on time according to our specifications.

Sage has been very good to work with. They have widely advertised both Standards books. The contract states that the Joint Committee receives 15 percent royalties. Joint Committee members can receive a 40 percent discount on the cost of the books if they are not resold. Dr. Sanders then reviewed the discounts available through regular orders that are described on the Sage order form. The price of the book is as low as we could get in order to make it available to as many people as possible. Sage sponsored a reception at AERA/NCME in honor of the new book. Sessions on The Program Evaluation Standards were held at AEA, AERA, APA, CSSE, and NCME.

The Program Evaluation Standards was reviewed in Educational and Psychological Measurement, and Evaluation Practice. Sage sent the book to a long list of organizations so there may be more reviews coming.

Dr. Sanders informed the members that a professor from China is using the Standards in one of his courses. The World Bank put Dr. Sanders in touch with this professor

Gwen Keith mentioned several organizations/committees that are working on evaluation, assessment, and/or standards, and that should be informed about the Standards. Dr. Sanders asked Ms. Keith to draft a letter to these organizations/committees to inform them of our activities. He also asked her to include a list of names of people to contact.

The ERIC Clearinghouse on Assessment and Evaluation sent out an announcement about the Standards. CSSE has requested 900 copies of the Sage flyer for the Standards that will be distributed throughout Canada. Dr. Sanders emphasized that it is important to keep the Standards in front of people so they will be used. Ms. Keith stated that their last newsletter noted the new Standards.

The Nebraska Extension AGSTAT will do a broadcast on The Program Evaluation Standards in January. They have 300 students who will receive the broadcast.

Louisiana has a policy that as an evaluator you must be certified using The Program Evaluation Standards before they can serve as an evaluator using state funds. Dr. Sanders would like to see that requirement in every state. Dr. Clements said that she would see what she could do about that through the CSSO.

There were displays at the annual meetings of NAESP, NASSP, AEA, AERA, APA, and CES this past year. Dr. Sanders asked the representatives from each organization to plan on doing that this year. He will contact Sage and have them get in touch with the representatives with materials for displays.

Last year, Dr. Joy Frechtling of WESTAT, contracted with the Joint Committee to develop workshop materials for The Program Evaluation Standards. The monies were used to develop a training manual and a pocket guide. Nine different federal agencies were involved in the workshops.

Dr. Stufflebeam wrote an article that is in the new International Encyclopedia. There were also four recent court cases that used The Personnel Evaluation Standards in their arguments.

The Center for Research on Educational Accountability and Teacher Evaluation (CREATE) has been using both sets of the Standards for research and development work in administrator, teacher, and school evaluation. CREATE will be doing mini-institutes in Texas and Hawaii using The Personnel Evaluation Standards. CREATE has been getting the word out about the Standards.

The Standards have been used in dissertations this past year at the University of Illinois, University of Regina in Saskatchewan, and University of Zurich.

Pre-proposals were sent to the National Science Foundation (NSF) this past year seeking support

for Joint Committee projects. Neither pre-proposal was supported.

Dr. Sanders was asked last year to extend an invitation to ASTD to join the Joint Committee. They declined because they are pulling back on their commitments and therefore are not accepting new invitations. The Association for Assessment in Counseling (AAC) is not represented this year. The American Counseling Association has had financial difficulties and they would like to suspend their status until they can get back on their feet. The American Educational Research Association (AERA) was not represented this year because their representative had not yet been appointed.

Dr. Sanders received a letter from Dr. Daniel Stufflebeam resigning as an at-large-member of the Joint Committee. Dr. Stufflebeam has been involved with the Joint Committee since its inception in 1974.

The Validation Panel for the new Program Evaluation Standards was to have their final report done in time for this year's meeting. Dr. Sanders gave a brief background of the Panel members. The Panel members have discussed the report several times but they were unable to get it finalized in time for the meeting. Dr. Gould, Validation Panel Chair, plans on having it to Dr. Sanders by the end of this calendar year.

An item discussed at last year's meeting was to conduct a vote to change the size of the Joint Committee from 18 to 21 sponsoring organizations. Dr. Sanders delayed doing so with the changes in the current sponsoring organizations, with Dr. Stufflebeam's resignation and with the ASTD seat on hold. Dr. Sanders noted that the Joint Committee needs to be careful in enlarging the group. There is a balance in the membership between consumer and technical perspectives that needs to be maintained. Ideas on organizations to be invited and which perspective to select them from was discussed. There was also discussion of adding organizations as sponsoring organizations or cooperating organizations. This discussion was then tabled until later in the meeting.

Financial Report

Ms. Ramlow reviewed the submitted financial report, noting a balance of $14,243.03 as of 9/21/94. Dr. Sanders noted that the Joint Committee needs about $10,000 per year for basic operations and the Annual Meeting. Ms. Ramlow also reviewed the McGraw-Hill and Sage book sales reports; 17,791 copies of the 1981 Standards and 8,484 copies of the 1988 Personnel Evaluation Standards have been sold. The sales summaries and royalties from Sage are only received once a year.

TASK FORCE REPORTS

Dr. Sanders noted that at last year's meeting, two task force groups were appointed. One had a charge of conducting a five year review of The Personnel Evaluation Standards to see if they need to be revised. The other was charged with reviewing the future governance of the Joint Committee. The following are their reports.

Task Force on the Five Year Review of The Personnel Evaluation Standards

Dr. Cummings chaired this task force, that also included Drs. Bader, Baker, and Rogers. They met last year via teleconference. Subsequent to that meeting, Dr. Cummings gathered data over the course of the year. The task force met again at the 1994 Annual Meeting and reviewed their recommendation, which was based on interviews and literature review. The Task Force said that there was no compelling reason for a revision of The Personnel Evaluation Standards at this time. The only changes would be cosmetic versus fundamental. The Task Force expressed a concern about waiting another five years to review this question again. They recommended that it be reviewed again no later than two years from now so that the Standards do not become obsolete. Dr. Sanders noted that the five year review cycle is part of the operating procedures accredited by ANSI but that does not mean that a revision process can not be done sooner. Discussion turned toward the issue of acquiring funds to conduct the revision, but it was decided that whether or not to revise the Standards should not be based on whether we can acquire the necessary funds. Dr. Cummings moved and Dr. Baker seconded that we reaffirm The Personnel Evaluation Standards but that we revisit the issue of revising the current version of The Personnel Evaluation Standards by having the question on next year's agenda. The motion carried unanimously.

Discussion then turned to sending The Personnel Evaluation Standards to ANSI for their approval. This would be a way to get information for the review process. Concern was stated of what would happen if the ANSI review resulted in changes to the book which would require funding for the revision. Mr. Mays moved and Dr. Thompson seconded that The Personnel Evaluation Standards not be sent to ANSI at this time. The motion carried unanimously.

Task Force on Future Governance of the Joint Committee

Mr. Mays, Jr. chaired this Task Force with its members being Drs. Pullin and Stufflebeam. The Task Force on the future governance of The Joint Committee recommended the following:

1. That Dr. James Sanders be asked to serve an additional three year term, after this one ends, as Chair of The Joint Committee.

2. That from one year from now, we bring onto The Joint Committee as a member at large Dr. Arlen Gullickson, who is Chief of Staff at The Evaluation Center at Western Michigan University. Dr. Gullickson would come to The Joint Committee as the replacement for Diana Pullin as Vice Chair assuming that Dr. Pullin is leaving The Joint Committee after this year.

3. That during Dr. Sanders' third year of his last term, Dr. Gullickson may be nominated for election as the new Chair.

The Task Force believed that these recommendations would assure The Joint Committee of remaining at The Evaluation Center and relieving the anxiety and worry of finding a new location for a host institution for The Joint Committee. This would also provide a transition period for the chairmanship.

Dr. Baker moved, and Dr. Johnson seconded, that the recommendation be accepted. The motion carried unanimously.

Location of the 1995 Annual Meeting

Several locations were discussed. The dates for 1995 will be September 29, 30 and October 1 Friday, Saturday, and Sunday). The two locations that the members wanted Dr. Sanders to consider were Phoenix, Arizona and Denver, Colorado. Dr. Baker will send some possibilities of locations for the Phoenix area. Dr. Rogers indicated that he may look into Banff as a third possibility and will send information if it looks feasible. Dr. Sanders will collect data on the sites and make a decision based on cost and room availability.

WORKING GROUPS

Student Evaluation Standards

Six years ago, a series of papers were developed to look at the need for standards for evaluations of students. Issues that were identified in the papers were discussed and then members were asked if this is something that the Joint Committee should take on. Several pros and cons were discussed including a discussion of what other organizations have developed or are working on developing. It was noted that there are no organizations on the Joint Committee now to represent students or parents. Appropriate organizations could be invited to join the Joint Committee.

Dr. Rogers stated that we need to look at student evaluation broadly using the four attributes of sound evaluation: utility, propriety, feasibility, and accuracy.

Dr. Sanders identified two issues: (1) the need for standards for evaluating students and (2) what the domain should include. The working group was asked to come back with a recommendation on whether we should develop student evaluation standards and a funding strategy if we decide we should develop the standards.

Dissemination of the Standards

Dr. Sanders reported on dissemination that has taken place. But dissemination has not been systematic or widespread. There is much to be done. The Standards are not getting into the hands of the people who are directly involved with education. At last year's meeting, a list of dissemination tasks was developed. Many of these tasks were done, but they have not gotten us very far. How to get the Standards disseminated was the assignment of the second working group.

Dr. Sanders asked each Committee member to work on a "homework" assignment to record ideas about what each person can do to get the Standards into the awareness of the leaders and members of their own organizations. Who in the association knows a contact person in a funding agency, who might be willing to listen to and support what the Joint Committee is doing? How can we achieve the vision of getting the Standards into practice? The meeting adjourned with members dispersing into their working groups. Members of the two working groups were as follows:

Saturday, October 1, 1994

Dr. Sanders opened the meeting by asking the members to develop plans for the year for working with their organization to get the Standards into the awareness or into use by the organization leaders, and to support people who are trying to use the Standards. He also asked members to list people in their organizations who have contacts with funding agencies that would have an interest in what the Joint Committee does.

Reports from the Working Groups

Dr. Pullin reported that the task of this group was to develop a strategic plan for dissemination and training on the use of both sets of standards. The group discussed what had been done as well as what could be done. Issues developed include: having people know the standards exist; having people know the importance of applying the standards to the work they do; and how to persuade people to get involved in using the standards.

A key is convincing key actors or groups about the importance of the standards. This could have a ripple effect. Five goals were identified for such an effort.

1. To enhance utilization of the Standards,

2. To require and encourage the use of Standards in funded projects particularly in the evaluation components of funded projects, particularly through OERI and the U.S. Department of Education, but also possibly state departments of education;

3. To seek some form of state and/or federal legislation incorporating the commitment to utilization of the Standards:

4. To encourage other standard-setting bodies to take into account the works of the Joint Committee

5. To promote the availability of workshops or institutes on the use of the Standards.

A list of proposed activities was submitted, (see attached Proposed Activities).

Discussion occurred on marketing of the Standards including the full use of technology to disseminate information, i.e., electronic bulletin board, Gopher, and use of advertising with the Standards pamphlet.

Another proposed initiative was to ask federal officials to write legislation requiring use of the Standards in education reform. This may include accessing member organizations' government liaisons, educating them about the Standards so they may advocate for the Standards and stress importance of building in evaluation at the beginning of new systems. Committee members will identify and submit to Dr. Sanders names of people from member organizations who are government liaisons. Dr. Pullin will work with Dr. Sanders to develop the letter to OERI.

Key influential persons in Canada are members of the Canadian Evaluation Society (CES) Professional Development Committee as well as members of CES who are Canadian federal officials.

Dr. Baker suggested contacting CEDAR, directors of labs and centers, project officers running large evaluation programs, technical assistance centers, CRESST, Deans/Superintendents Supervisory Committee, and the Southern California Superintendents Research Council. California has myriad initiatives where good evaluation programs are needed.

University summer instructional programs may be a viable option for dissemination of Standards information. Another option is training at conferences such as AERA, NCME, APA, or AEA. The training could be before or after the conferences. It was recommended that it be done every year, with one person as lead for each organization for efficiency. Usually if you do a good presentation one year, the organization keeps you on the list for the following year. It was noted that maybe Dr. Gullickson could help with this project. With this in mind, the manual for The Program Evaluation Standards needs to be used as a model for The Personnel Evaluation Standards so that we have a training manual for both standards.

The working group on standards for evaluations of students submitted a draft position statement that went through several revisions. The discussion continued into Sunday and after the Annual Meeting when the attached concept statement was prepared.

Expansion of Joint Committee Members

There was a discussion on addition of members to The Joint Committee, with a consensus that 3 to 6 new members should be added. A motion was made to change the Bylaws to allow addition of up to 6 new members, for a total of up to 24 total members. Discussion followed on the requirement of a paper ballot to amend Bylaws and the notification requirements. The motion was withdrawn, and Drs. Baker and Johnson will submit to Dr. Sanders a written request for a ballot to change the Bylaws.

Invitation to At-Large Joint Committee Members

Dr. Pullin moved and Dr. Beatty seconded that if the ballot to extend membership to 24 is approved, the Chair of the Joint Committee will send an invitation to American Council on Education (ACE), National Parent Teacher Association (NPTA), and the National Coalition of Advocacy for Students (NCAS), and if any of the three decline the invitation, the Chair then will begin efforts to identify a replacement organization which would represent a similar perspective. The addition of a new at-large member would be subject to the approval of The Joint Committee. There was discussion and a vote taken, with the motion caring unanimously.

Addition of Cooperating Members of the Joint Committee

Dr. Thompson made a motion to extend an invitation to become cooperating (nonvoting) members to five organizations: National Conference of State Legislatures (NCSL), National Governors Association (NGA), National Association of State Boards of Education (NASBE), Council of Great City Schools (CGCS), and the National Association of Independent Schools (NAIS). Motion was seconded by Dr. Pullin. Concern was voiced about size of the group becoming too large to be effective. The vote was taken; 11-yes, 1-no. The motion carried.

Dr. Sanders requested members make a reference list of foundations that their organizations had worked with in the past and list contact persons at those organizations. Suggestions included Spencer, Carnegie Corporation, Danforth, Lilly, Robert Wood-Johnson and MacArthur. Additional suggestions from others included Lee Smith Reynolds, Reynolds Tobacco, Reynolds Aluminum.

Dr. Rogers reported on further work by the working group on student evaluation standards. The Discussion that followed resulted in several conclusions:

1. There are many efforts, but there is a need for a broad conceptual umbrella.

2. This will include a broad range of stakeholders and the issues they are concerned with.

3. This effort will establish American National Standards which will influence professional practice in education.

4. It will develop underlying principles that address education in the broad sense of all students in all educational settings.

5. The process used is to be open, participatory, and public.

6. This product will not be time bound.

7. It will be a living document that will continue to guide practice and continue to be reviewed.

There was discussion and agreement to include this list in the conceptual report from the working group with emphasis to include cooperative efforts to create a product that does not duplicate other work being done, and the Joint Committee's track record of bringing together diverse organizations and reaching consensus.

Sunday, October 2, 1994

Dr. Sanders opened the meeting by distributing a list of the associations and asking each member to verify their association's annual meeting date and if the contact persons were correct.

Discussion of statement to get common understanding and direction for the student evaluation standards.

Dr. Sanders opened the discussion by summarizing earlier discussions.

THE PROJECT

The discussion began by suggestions that a conceptual statement should stress the concept of uniqueness. Narrative should be replaced with bullets for effect. We should address stakeholder issues. An overarching statement is needed with broad coverage of stakeholder's concerns.

Stakeholders that should be included that are not presently included are mayors, parents, employers, and policymakers.

We need to include student "needs," not only instruction and outcome measures. We are using the word "evaluation" not "assessment".

We should emphasize efforts to create national standards, to have a living document which will continue to be revised every five years, to avoid threatening other standard setting organizations, to be inclusive and unique, not redundant or parallel with other organizations.

We need to distinguish among our standards, content standards, and competency standards.

Five goals for dissemination and training and commitments

Copies were distributed by Dr. Pullin as confirmation of commitments for action.

GOALS FOR DISSEMINATION, TRAINING, AND COMMITMENTS

1. Enhance utilization of standards

2. Require/encourage use of standards in evaluation section funded projects

A. All RFPs from OERI and Department of Education--perhaps state departments too.

"Proposers are encouraged to address the 1994 ANSI approved Program Evaluation Standards within their proposed evaluation plans" Further, proposers are encouraged to apply appropriate segments of The Program Evaluation Standards at the initial stages of the overall project planning, as opposed to superimposing an evaluation plan on a project design already completed.

3. State or federal legislation should be sought incorporating a commitment to utilization of relevant professional evaluation standards in education

4. Encourage other standards-setting bodies to take into account The Joint Committee Standards

5. Promote the availability of workshop or institutes on use of the Standards

Standardized communication interfaces between organizations.

List of fax numbers and e-mail designations were gathered and will be distributed.

Individuals' Summary Comments

Overall consensus that this was a productive session with good cooperation and appreciation for quality contributions by new members, leading to clarification of responsibilities and crucial nature of our work. There is a desire to move as quickly as possible on the student evaluation standards to have the work available for the maximum impact while student evaluation has widespread public attention. There was discussion of meeting twice a year if budget allows.

With no further items to discuss, Dr. Sanders thanked everyone for their hard work and adjourned the meeting.

PROPOSED ACTIVITIES

1. Article for Kappan - Jim? Bruce?

2. Article for Education Week - Barbara/Jim

Overview of all standards efforts

3. Perhaps AASA journal - Barbara/Henry?

4. Create a Joint Committee letterhead with organization names on it - Mary R.

5. OERI letter - Diana & Jim

6. Integration of standards into CES essential skills project - Gwen

(Perhaps workshop for CES)

7. Contact association government liaison and executive director - each Joint Committee member Jim to coordinate?

8. Talk to Dena Stoner regarding the regional labs - Barbara

9. Talk to National Goals Panel - Barbara

10. Formulate a strategy to influence Canadian provincial and federal policy - Gwen & Todd

11. Target for workshop presentations (over years) by Jim & Arlen

AEA; CES; AERA/NCME; Division 5, 15 of APA

12. Provide copies of all inservice materials to Joint Committee at The Evaluation Center

13. Provide copies of all correspondence and implementation documents to Joint Committee at The Evaluation Center

14. Gather information for a Louisiana case study - Bruce/Jim

Use for Kappan article?

15. By Sage, a dear colleague from Jim to University faculty

Check to see if Sage has targeted the Association of Institutional Researchers in higher education

16. Letter/contact to key congressional staff people - Barbara/Diana/Jim Kris Gilbert E. Guiney

Enclose pocket guide

17. Information into our Association newsletters and publications - each Joint Committee member (uniform write-up)

18. ERIC - is it in Clearinghouse by Larry Rudner?