ANNUAL MEETING OF THE

JOINT COMMITTEE ON STANDARDS

FOR EDUCATIONAL EVALUATION

Kalamazoo, Michigan

September 30 - October 1, 1995

Attendees:

Joint Committee Members:

Dr. Edith Beatty (ASCD)

Dr. Rolf Blank (CCSSO)

Dr. Glenn Cutlip (NEA)

Dr. Mark Davison, (APA)

Dr. Arlen Gullickson (Member-at-large)

Ms. Gwen Keith (CES)

Mr. William Mays, Jr. (NAESP)

Dr. Dianna Newman (AEA)

Dr. Diana Pullin (Member-at-large)

Dr. W. Todd Rogers (CSSE)

Dr. James R. Sanders, Chair

Dr. Bruce Thompson (NCME)

Dr. William Webster, (AERA)

Dr. W. Eugene Werner (NASSP)

Joint Committee Staff:

Ms. Mary Ramlow, Secretary

JOINT COMMITTEE ON STANDARDS FOR EDUCATIONAL EVALUATION

MEETING MINUTES

SEPTEMBER 30 - OCTOBER 1, 1995

KALAMAZOO, MICHIGAN

Friday, September 30, 1995

Meeting Objectives

1. To conduct the business of the Joint Committee on Standards for Educational Evaluation

2. To review the final report of the Validation Panel for The Program Evaluation Standards

3. To review the funding strategy for The Student Evaluation Standards

4. To review the dissemination and training strategies for The Program Evaluation Standards and The Personnel Evaluation Standards

Dr. Sanders welcomed everyone to the 21st annual meeting of the Joint Committee. For the benefit of those who have not been serving the Joint Committee, Dr. Sanders gave a brief oral history of what the Joint Committee is, where it came from, and where it is heading.

History of the Joint Committee

The Joint Committee was created out of a task force that was appointed in 1974 when the APA, AERA, NCME test standards were undergoing revision. There was a proposal in that revision process to include a section on the use of tests in program evaluation, and the task force was created to look at how that might be added to the test standards. The recommendation from that task force was that program evaluation is a lot bigger than just the use of tests. If we are to develop standards for program evaluation, we ought to be looking at a wider set of issues than just testing issues. Out of that recommendation came an appointment of a new committee from APA, AERA and NCME to investigate the development of standards for evaluations.

The first meeting of that new committee was in 1975. They elected Dan Stufflebeam, who was one of three representatives from NCME, to chair that meeting. It was an illustrious group; for example, Donald Campbell was representing APA and Egon Guba was representing AERA.

At the meeting there were 3 representatives from each of the 3 organizations. They decided then that there was a group of people in program evaluation not represented, and these were the people who use program evaluation in education settings. They are participants in the evaluation process and should have as much a say in standards as the groups represented there. So they voted to enlarge the committee from 9 people to 18. They had 9 people already on the committee from the 3 organizations, so they added 9 more people to balance the technical perspective with other perspectives. A list of organizations was then developed and invited to join this committee, and that was the beginnings of this Joint Committee.

In 1981 the Joint Committee published Standards for Evaluations of Educational Programs, Projects and Materials. They purposely stayed away from teacher evaluation or personnel evaluation in general and institutional evaluation, like accreditation, because the area of personnel evaluation was just too controversial for the committee to handle, and they thought they needed to take on a manageable project and have a success first, before diving into controversial issues. With institutional evaluation they thought it was redundant with what accrediting bodies were already doing. In 1981 when the first standards were published, the Joint Committee decided that if these standards were to be kept current and maintained over the years, that a continuing group needed to be monitoring the use of standards, new issues, and the need for revision. Consequently, the Joint Committee voted to incorporate as a nonprofit organization that would be a continuing standard-maintaining and standard-setting group, modeled after a similar body that establishes and monitors CPA standards. The Joint Committee incorporated as a 501(c)(3) tax status organization, and developed a set of By-laws and Principles, to govern the organization. After that, they did decide to take on the task of looking at teacher evaluation and administrator evaluation in education and were able to secure funding from foundations to support that work. In 1988 they published The Personnel Evaluation Standards. In 1989, at the invitation of the American National Standards Institute (ANSI), the Joint Committee applied to become an accredited committee that would submit its standards for approval through ANSI. Such standards become American National Standards if they are approved by ANSI, and follow the Joint Committee Operating Procedures. This 1989 application to become an ANSI accredited organization was approved by ANSI and by the Joint Committee. We submit our standards to ANSI when we develop them, and if approved, they do become American National Standards in international cooperation for standard setting as well as here in the U.S. Also in 1988 the Joint Committee considered issues that were on the forefront of evaluation and areas in which we should be working. One discussion was that we ought to be thinking about revisions of the 1981 Standards, because of new methodological developments, qualitative evaluation methods, stakeholder involvement in evaluation. Another suggestion was that we ought to be looking at evaluation standards for evaluations of student performance and student needs because there were a lot of issues that include testing, but go beyond testing, too, that we ought to be addressing as a standard-setting organization. The Joint Committee was able to get funding from the Kellogg Foundation to support the revision of the 1981 Standards and that then sent us in the direction of putting aside the student evaluation standards project for the time being to work on the revision. The Program Evaluation Standards were published in 1994 and submitted to ANSI as the first set of Joint Committee standards submitted to ANSI. They became the first American National Standards that were published by the Joint Committee.

Last year at the annual meeting we revisited the need for standards for student evaluations, and unanimously decided that this was still a pressing need. The Joint Committee has a role to play in the practice of student evaluation and we should seek support to develop a third set of standards for educational evaluation. This third set would cover the primary clients of education; that is, students.

The Joint Committee is a working committee, and the annual meeting is the one time during the year when we get to talk to each other face to face to discuss issues, decide the direction that we are heading, and prepare draft products. The Joint Committee's By-laws indicate that we do not only establish standards in educational evaluation, but we also serve in a brokerage role to help people who are doing evaluation to come in contact with the standards. We serve an information role in terms of being a depository for papers, articles, and reports written about use of standards in educational evaluation. We serve in a role of research and development in terms of taking on projects that will further the application of standards that have been developed or will develop new standards. Finally, we serve in a technical assistance role, such as providing testimony in court, showing people how to use the Standards. Training is a part of that technical assistance role where we provide conferences or workshops to help people to make use of the Standards.

The Joint Committee is the only committee that brings the voices we have represented here around the table to talk about evaluation, problems with evaluation in education, and the practice of evaluation. It is a diverse group of people. It is an important committee for that very reason.

Organization of the Joint Committee

Dr. Sanders has been serving as Chair since 1988. Dr. Pullin has been serving as Vice Chair since 1988, and this will be Dr. Pullin's final year on the Joint Committee. Dr. Sanders thanked Dr. Pullin for her service to the Joint Committee. The elected members of the executive committee for the past 3 years have been Mr. William Mays, Dr. Bruce Thompson and Dr. Eugene Werner.

Dr. Sanders briefly reviewed the agenda. Review of the Joint Committee's host contract with The Evaluation Center was added to Saturday morning's agenda at 8:30 a.m.

Also on Saturday, discussion will take place on our relationship with two governmental organizations that have expressed an interest in participating in some role with the Joint Committee. One is the General Accounting Office in Washington; the Comptroller General has indicated that he would like to establish some relationship with the Joint Committee. The other group is the National Legislative Program Evaluation Society (NLPES). These are state employees who do audits of state government programs and also do program evaluation of state government supported programs. A question was raised about whether individuals are members of the Joint Committee or only organizations. Dr. Sanders reported that we have 2 at-large individuals on the Joint Committee. Dr. Pullin has been serving since 1985 and was invited because she has a background in legal aspects of education. When the Joint Committee was getting involved in personnel evaluation, that was an important perspective that was needed on the Joint Committee. Dr. Arlen Gullickson has joined us this past year as an at-large member of the Joint Committee as it looks at future governance of the Joint Committee and the role The Evaluation Center at Western will play. The Joint Committee pays for at-large members to attend the annual meeting.

Chair's Report

The most obvious change that occurred after last year's meeting was the creation of letterhead for the Joint Committee. It was decided last year that an official letterhead was needed.

The Joint Committee voted to expand the size of the Joint Committee from 18 to 24 sponsoring organizations last year. We had reached our limit in terms of 18, and we were not able to add new sponsoring organizations unless we expanded the size of the Joint Committee. We voted to expand the size to 24 with the understanding that we would continue have a balance between the technical organizations and the practitioner organizations in education.

Sponsoring Organization invitations were sent to the National PTA, American Council on Education, and The National Council of Advocacy for Students. Those three organizations were extended invitations based on last year's discussion that as we move into standards for evaluations of student performance, voices from parents, policy organization, and students were needed on the Joint Committee. There was no response from any of those three. National PTA had been on the Joint Committee and opted to drop off because of priorities; they were overextended in their national activities, and this was not high enough on their priority list. They participated in the development of the Personnel Evaluation Standards in the 1980s.

After discussion Dr. Pullin volunteered to make a personal call to the National Coalition of Advocates for Students and Mr. Mays volunteered to contact Arnold Fege of the National PTA.

Invitations to organizations to join us as cooperating organizations were also extended. A cooperating organization is one that we keep informed about our activities. We send them notice of our annual meeting and, minutes of our annual meeting, but they don't usually attend the annual meeting. They would have no voting rights on the Joint Committee, and they would not be listed on the letterhead as a sponsoring organization. It is a communication arrangement that we make with these organizations. Invitations for cooperating status were extended to National Governors Association, National Association of State Boards of Education, Council of Great City Schools, and the National Association of Independent Schools. Another organization that we wanted to extend that invitation to was the National Legislative Program Evaluation Society (NLPES). Because they had expressed an interest in becoming a sponsoring organization, Dr. Sanders held that one until the Committee decides which invitation they wish to extend. We received a response from the National Association of State Boards of Education; they said they thought this was important, and they would put it on their meeting in late October, and then they would respond to us after that meeting. The Council of Great City Schools accepted the invitation and thanked us for the invitation. We did not get a response from National Governors Association or the National Association of Independent Schools. Dr. Blank will make a phone call follow-up with the National Governors Association. No follow up will be done on National Association of Independent Schools.

Dr. Sanders sent correspondence to the APA, AERA, and NCME Tests Standards Committee after last years meeting to inform them of our decision to develop standards in the area of student evaluations. He did not receive a reply. They have been informed of our intentions.

Several revisions have been made in our proposal for the student evaluation standards, and we now have a proposal ready to go as soon as a funding organization expresses an interest. Contacts were made with the Z. Smith Reynolds Foundation through Dr. Henry Johnson. They fund North Carolina projects. Dr. Sanders talked with the Pew Charitable Trust about support for the project, and they said they would be interested in entertaining a two-page letter proposal, which Dr. Sanders sent to them. Dr. Pullin has a contact with the Spencer Foundation and is scheduled to meet in October to talk with them about this proposal.

Both sets of Joint Committee Standards are now accessible electronically through the American Evaluation Association on a listserv called EVAL INFO and also by the National Council of Measurement in Education. The ERIC-TM clearinghouse has both sets of Standards online.

Two changes this past year to the by-laws were: (1) to increase the size of the Joint Committee from 18 to 24 members, and (2) to approve unlimited terms for the Chair of the Joint Committee to allow for continuity over many years.

Dr. Sanders had conversations with the NLPES and GAO organizations. Dr. Sanders met with the Assistant Controller of GAO and with the Executive Board of NLPES. Both are very interested in creating some formal relationship with the Joint Committee. Dr. Sanders also presented at the annual meeting of the NLPES group on the Joint Committee Program Evaluation Standards.

We proposed an all day presession, and it was accepted for the International Evaluation Conference that will be held in Vancouver, British Columbia, November 1st through the 5th of this year. Diana Newman, the representative on the Joint Committee from the American Evaluation Association, Gwen Keith, the representative from the Canadian Evaluation Society, and James Sanders will do the presession. There is also a paper session on the Joint Committee's Standards scheduled at that International Conference. The session deals with the state of Louisiana's use of The Program Evaluation Standards to train and certify consultants in that state.

An article was developed by Drs. Thompson and Sanders this past year on the Standards, and it was submitted to the Phi Delta Kappan. They turned it down, and we submitted it to Educational Evaluation and Policy Analysis (EEPA). Two other articles were published, one in the Evaluation Journal of Australia on The Program Evaluation Standards, and the other in New Directions for Program Evaluation, a publication of the American Evaluation Association.

Dr. Sanders gave permission to translate The Program Evaluation Standards into German and Spanish. He put the interested parties in contact with Sage so that they could get copyright contracts written for international use. He asked them to keep us informed of the publication of those translations.

The American Evaluation Association listserv called EVALTALK has been discussing The Program Evaluation Standards with very positive discussion about how useful they are in an international context. They are not just American standards, as the Australians pointed out and the United Kingdom people agreed, and they were indeed useful.

We have had correspondence and telephone discussion with the American National Standards Institute (ANSI). They will be starting a new process of auditing accredited organizations to make sure they are following their operating procedures and method of operation. The accredited organization would have to pay the expense of the audit which could amount to several thousand dollars, if not more. Dr. Sanders appealed because we are a voluntary standard setting organization, and we do not have an endowment to finance an audit. They accepted the appeal and made a special case for us. They said that the audit would be done, but it would be done through the mail. The Joint Committee has followed its operating procedures, which the Validation Panel will support. We will allocate up to $100.00 to support expenses for the audit. The audit for us will probably occur in 1998. We are down the list of organizations to be audited.

The state of Hawaii is using The Personnel Evaluation Standards to develop a statewide teacher evaluation process. They ordered 300 copies of the Personnel Evaluation Standards so that every school in Hawaii has copies. Dr. Arlen Gullickson has been working with them.

The University of Nebraska did a telecast this past year on The Program Evaluation Standards as part of a course they were offering through the AGSTAT satellite system for extension programs. It was a one-hour presentation that had phone call ins for questions from over 300 sites around the country.

Papers were presented this past year at the National Organization on Legal Problems in Education on the use of the Personnel Evaluation Standards in litigation in education.

The Dallas Public Schools is making use of the Personnel Evaluation Standards as a basis for a new teacher evaluation system. Dr. Bill Webster is in the middle of developing and implementing this new system.

The research and development center on teacher evaluation called CREATE, sponsored by OERI, has based a lot of its work on the Personnel Evaluation Standards and The Program Evaluation Standards. Through CREATE there have been one day conferences held this past year in Kansas City, Boston, and Chicago to demonstrate to educators how to use the Personnel Evaluation Standards for developing their own evaluation systems. There is also a new project through The Evaluation Center at Western Michigan University to evaluate the Marine Corps personnel evaluation system using the Personnel Evaluation Standards. This could be a high visibility project and also lead to a number of applications outside of the traditional education setting if it's successful.

A paper was written this past year on what the Program Evaluation Standards say about social justice. Articles were published in the Canadian Evaluation Society Newsletter and also in the Journal of Experimental Education on the work of the Joint Committee. Following last year's meeting and following Dr. Pullin's report, we submitted a response to the request for comments on the reorganization of The Office of Education Research and Improvement (OERI). Dr. Sanders had copies of that statement to share.

Dr. Pullin reported that the Massachusetts Legislature and Department of Education were going through a process to try to "reform" personnel preparation in Massachusetts. After some negotiations with the state, the Standards were included in their requirements concerning the evaluation of teachers. These happen to include a requirement that those evaluations be done according to "appropriate professional standards". They didn't want to write in the title of the Standards, because they thought that was too specific, but they did write in this general requirement that it be done according to professional standards. Dr. Pullin tried to make sure that the relevant people in the state had copies of the Standards. They were completely unaware of the Standards, and were quite impressed when they saw them. This exemplifies the problem that people are not sufficiently aware of the standards and what role they can play.

Dr. Sanders commented that the personal contact turned that around, and that written communications probably would not have gotten a response. It was mentioned in the Chair's report that letters were sent which had no response, but where there was personal contact, like with NLPES and GAO, for example, it does work. Dr. Sanders encouraged all members to on the Joint Committee to make personal contacts with people in places that can influence use of the Standards.

Various activities and dissemination ideas from the past year were reviewed, including distributing copies of The Program Evaluation Standards, which has been a valuable marketing tool. A suggestion was made to develop some instructional packages that could be printed in a professional journal, such as Evaluation Practice. A training manual for teaching the Standards, that could be published by Sage was suggested by Dr. Newman. Another suggestion was made to develop a video presentation on the Personnel Evaluation Standards and The Program Evaluation Standards. Dr. Beatty agreed to approach ASCD about taking the lead on developing a video product.

Dr. Sanders stated that the Joint Committee received a $20,000 grant last year from the National Science Foundation (NSF) to develop evaluation training materials, and a training manual was developed. Dr. Sanders conducted an evaluation training workshop for representatives of nine different federal agencies, and gave them the training manual to take with them. He followed up with each workshop participant to explore how the Committee could work within their agencies to continue this training. Due to other priorities in Washington, the agencies' representatives said this was not the right time to do something new, and none of the nine agencies were interested in pursuing further work with the Joint Committee. However, discussions with GAO indicated they recognize the need for federal agencies to be doing better evaluation and using standards like The Program Evaluation Standards. Dr. Sanders stated that the GAO is considered an independent neutral party to do evaluation training as well as evaluation, and if the Joint Committee were to get involved in a partnership with GAO, the Standards would be used more by federal agencies.

It was suggested that involvement with the Washington Evaluation Association (WEA) or with GAO could provide programs that have some continuity through changes in administration. Dr. Blank, who is a member of WEA, agreed to talk with them about developing a relationship with the Joint Committee.

Dr. Gullickson stated that The Evaluation Center has just received a two-year grant through the National Science Foundation (NSF) to evaluate the NSF Science Education Program. The evaluation of their projects will be based mainly on The Program Evaluation Standards. The evaluation information will be presented at a set of conferences and two institutes. Following that, the information will be available on Internet and World Wide Web. This is one of several ways that NSF has taken an active role in promoting the Standards.

It was suggested that if a collection of all of the different ways that the standards are employed in various documents were gathered, a variety of medium materials would emerge which would be helpful. It was suggested that this could also be helpful with fund raising. If the Joint Committee could develop some graphic portfolio on the utilizations of standards, it could help convince funding sources that this investment would be well utilized given past evidence of utilization of the standards.

A process was described by Dr. Gullickson of developing a manual on how schools should employ teacher evaluation methods. The manual became a powerful statement of what the Standards can do. It is compelling evidence that there is a way to conduct teacher evaluation, based on the Standards, which is logical and gives a strong assurance that evaluation is being done appropriately.

Providing a manual to law firms that represent affiliates of NEA or teachers' attorneys, either through collective bargaining or through representing teachers around evaluation issues, would help them to understand that there are some professional standards that they could utilize in their advocacy. Providing a copy of The Personnel Evaluation Standards along with a copy of the teacher evaluation manual also would be useful to get a foot in the door in those types of associations.

Dr. Davison stated that his purpose for raising this was to say there are dozens of these kinds of things going on, and the Joint Committee members do not have a good sense of the webbing that is occurring. Gathering a sense of that would be very useful to help make the system grow. Dr. Cutlip agreed to pursue this with NEA.

Dr. Sanders summarized by saying that from his perspective much has been done this past year, but not as much as he would have hoped. In 1996 he hopes that funding will be established for student evaluation standards.

Financial Report

Ms. Ramlow stated that last year a survey was done of five financial institutions, including two credit unions, to find the most favorable interest rates. After a review of the information by Ms. Ramlow and Dr. Sanders, it was decided to close out the Old Kent account and move the funds to School Employees' Credit Union, in a "Gold Money Market" account. This is reflected in the Interest Income item dated August 1, 1995.

JOINT COMMITTEE

BANK ACCOUNT STATEMENT AS OF SEPTEMBER 22, 1995

Ms. Ramlow also explained that the expenses Dr. Sanders referred to in his Chair Report regarding the support services offered by the Evaluation Center were cumulative totals since last year, including the amount of money spent on typing, mailing, producing, photo copying, Ms. Ramlow's time and the cost of supporting participation of at large members, Dr. Sanders, and Ms. Ramlow at the Annual Meeting. That total comes to $12,036.24.

Dr. Thompson asked that a detailed breakdown of costs be reported in the future. Ms. Ramlow said that she would do so.

Dr. Sanders said that our balance in normal times when we don't have a project going on usually fluctuates between $12,000 to $16,000. So we are about where we usually are in terms of our financial status, which is borderline poverty. If for some reason the royalty payments were to drop suddenly, the Joint Committee would have trouble holding the annual meeting. This is one reason we need to keep seeking outside support for projects which we think are important.

Dr. Sanders said that having the Annual Meeting in the Kalamazoo area is a cost saving because travel for local at-large members and staff such as Dr. Gullickson, himself, and Ms. Ramlow, is virtually free. Dr. Pullin will be the only travel expense that we have this year.

It was suggested it would be helpful to have information that tracks sales of the Standards across years. The Joint Committee should look at patterns of whether sales are steady or are growing or not. This is important in terms of knowing how the dissemination efforts are going and whether the standards are still perceived to be current. They should be split out by the two Standards now that Sage is publishing both of them, so we know what is going on with the Personnel Evaluation Standards versus The Program Evaluation Standards. Ms. Ramlow indicated that at the present time we do not have the actual total sale information, but she will follow up with Deborah Lawton and send the information to everyone by e-mail.

A question was asked whether there was any other income from workshop materials, which could be utilized as an indicator of whether or not it is a good avenue to pursue, or perhaps if it requires additional effort in terms of development and marketing.

Ms. Ramlow indicated that the Joint Committee expenses from The Evaluation Center also include any credits received for anything the Center sold, such as the Manuals or the Step Guides. Those credits are reflected on statements from The Evaluation Center.

Dr. Sanders explained that when items are printed, the Evaluation Center is charged for the printing, and when it sells items, the sale is credited to The Evaluation Center. Some profits have been made, but not much. Dr. Sanders also indicated that notebook sales have not been successful.

It was suggested that the Joint Committee might advertise to groups, like professors, who are teaching The Program Evaluation Standards. This would be a good audience for sales of the manual, transparencies and evaluation materials. Identifying which college book stores are buying The Program Evaluation Standards to sell to their students could specify a targeted audience. Dr. Newman offered to work on a training manual to accompany the Standards if Sage would be interested in publishing it. Dr. Sanders will check with Sage and then contact Dr. Newman if they are interested.

A question was raised about the pocket guides. Dr. Sanders said that pocket guides are not income generating. He said the reaction we are getting from the pocket guides is that everybody wants them, but they don't want to pay for them. There are limited funds to be able to respond when people want 1000 copies for distribution.

It was suggested that confusion is created when some materials, i.e. pocket guides and manuals, are sold by The Evaluation Center, while the Standards come from Sage. The pocket guides are a marketing device to lead to purchase of the book.

Dr. Sanders agreed that a big piece of the dissemination work is to decide what other products, such as the step guide, can be bundled together with the Standards for SAGE to market, but the Joint Committee needs to draw on the marketing expertise of SAGE.

Arrangements for 1996 Annual Meeting

Dr. Sanders stated that traditionally the Joint Committee Annual Meeting has been the last weekend in September, not earlier because of the start up of higher education classes and not later because there are conflicts with conferences that are going on in October, and then the weather turns bad. After discussion of conflicting schedules, it was agreed that the 1996 Annual Meeting will be held on October 4, 5, and 6. Members would come in Thursday night, October 3. Sunday morning will be reserved in case the student evaluation standards are funded, as that will be a critical time. Dr. Sanders said that if funding is not obtained and a Sunday work session is not needed, members can plan to leave Saturday evening. However, he suggested an optimistic view which anticipates funding will be available for the student evaluation project, and that the Committee is going to need as much time together as possible.

Dr. Sanders opened discussion on location for the 1996 Annual Meeting, stating that we try to find a location that is pleasant, where we are not locked up in hotel but also is affordable and accessible. He said that the site must be chosen a year in advance to ensure appropriate space accommodations due to the size of the group. He stated the meeting site at Yarrow was chosen for 1995 to keep travel costs down for Dr. Gullickson, Ms. Ramlow and himself. Due to uncertainty of funding for next year, Dr. Sanders suggested meeting at Yarrow in Kalamazoo again next year. One objection was raised to the Kalamazoo site due to difficulty with commuter flights on both ends, and the Chicago Nordic Hills was suggested as an alternative. A third suggestion was made to consider a retreat site in Washington DC if there is funding to support transportation for the staff for the Joint Committee.

Dr. Sanders agreed that a Chicago site would be possible, as it was within driving distance of Kalamazoo to save travel costs. Dr. Sanders will report later on the location of next year's meeting.

Validation Panel Report

Dr. Sanders opened discussion on the Validation Panel Report by stating that the Committee's task at this point is to raise any questions or issues for the validation panel.

A motion was made and seconded to accept the Validation Panel Report. Discussion included the point that the Committee had considered virtually all comments that had come in during field testing, but that it was just inevitable that not all concerns could be addressed.

Discussion continued on content of the Validation Report, wording intent, and impact on the Joint Committee. Dr. Sanders said it was a generally positive report. He will review with Dr. Gould, the Chair of the Validation Panel, the paragraphs on which the Committee has concerns and/or questions and have a final report sent to all Committee members. It was the consensus of the Committee members to accept the Validation Report with the noted changes.

The Committee discussed publishing the report in one of the following:

Evaluation Practice, by the American Evaluation Association

Chronicle of Higher Education

Education Research

EEA - Evaluation Information document depository

ERIC

Newsletters - EP, NEA

Creating a shorter version or an abstract to accommodate space requirements will be necessary, either by Dr. Gould or a Committee member. Dr. Newman will write an abstract of the report to be published in various newsletters.

A discussion of the need for revision of The Personnel Evaluation Standards was next on the agenda. The Committee discussed the need for a more recent copyright; updating; reference; revisions to gain ANSI accreditation; revisions to address new developments in the field, i.e., potential use of performance and portfolio assessments for educators; creating a revised edition vs. a new edition to keep references current without a formal edition change process; changes necessary in policies or by-laws to enable the Joint Committee to allow a more brief revision process; and an update of case studies.

Funding and time issues were discussed with concern about difficulty working on two projects simultaneously, i.e., full-scale revision of The Personnel Evaluation Standards and creation of Student Evaluation Standards.

Dr. Gullickson suggested that The Personnel Evaluation Standards could be used to create the Student Evaluation Standards due to the similarities, and then running the two projects simultaneously may be possible.

Funding from sources which benefit from the standards and funding from the original sources was brought up and tabled for afternoon discussion.

The agenda for Friday afternoon was to break into work groups: (1) on development of a funding strategy for developing Student Evaluation Standards, chaired by Dr. Rogers, and (2) on dissemination and training for the standards, chaired by Dr. Pullin.

Group Reports

Group 1: Development of a Funding of Strategy for Student Evaluation Standards

After reconvening into the large group, Dr. Rogers reported on the work of group 1. They were asked to look at strategies for obtaining funds and to look at the draft student evaluation proposal that was mailed to the Committee members. They met as a group for the first part and then split into two subgroups, one subgroup looking at obtaining funds, and the second subgroup looking at the proposal.

Dr. Werner, chair of subgroup one, reported that they had two charges--one is to identify the group of associations and/or testing companies that might consider to join in partnership to help finance the student assessment project. They identified at least a dozen national educational associations but no names of the testing companies. But most members know 4 or 5 that would be funding prospects to contribute to some type of partnership contribution. Then the other half of their charge was to work on the letter that would be going to the prospects to grab their attention, try to create a concern or help them grasp the seriousness of student evaluation issues and maybe the current state of affairs, which is not good at all in terms of quality and the need for standards for student evaluations. Most of their time was spent trying to develop a letter with a strong beginning to get their attention about the seriousness of this problem, and then proposing to them what we felt could be done with a partnership.

They thought if we could get 15 contributors at $5,000 per year for a 3-year period, we could easily do a 60/40 matching -60 percent by a foundation and 40 percent by professional educational associations.

Dr. Sanders commented that in the past the Joint Committee has shown foundations that our associations have contributed to the development of standards through the volunteer work of people who respond to drafts of the standards, through our volunteer work as committee members, and it is considerable. It is about an equal amount of money that our associations are already contributing to the effort when hours are added up that people put in to this. So we already have a really good argument with foundations that there is other money being put into these, but its just not cash. As soon as the letter is finished, they will bring it to the group for review.

The second subgroup decided to rewrite the proposal. They recommended the introduction should start out with 3 illustrations with the first 2 being classroom examples of a pervasive problem. The examples should be something the reader of the proposal can relate to, for example, a situation which happened to me or to my children, describe what happened and what the negative impact is, what malpractice occurred, and where it impacted negatively. And then the 3rd illustration would be one of external testing, because we anticipate the standards will be useful across a range of different kinds of tests and who is doing the testing. Dr. Stiggins and Dr. Rogers were suggested as sources for illustrations 1 and 2, and Dr. Webster will help with illustration 3, using the Texas situation.

The introduction paragraph has been rewritten so that the it talks about what the standards are going to do with their audiences. Following that is the proposal and the proposed set of work. It was suggested that it be written in terms of the tasks that are going to be done. What are we going to produce as we march through this? Then it moves into the Joint Committee's approved operating procedures, using the panel of writers who provide the materials, and then into the international and national reviews and public hearings, and so on. That will stay the same, but we are recasting it not in terms of years, but in terms of a sequence of activities. Then a time chart, will indicate when a particular activities will be completed in Year 1, and Year 2 and so on.

Group 2: Dissemination and Training for the Standards

The group took the last year's list of tasks with what had been done and tried making comparison between what was done and not done last year, what needed further work, what were new ideas, etc.

Also discussed was a press release in terms of the Validation Panel report. It will also be useful to call the Educational Researcher, and Dr. Blank volunteered to either do it himself or have somebody from his press office do that to talk to them through why this is really important.

They also talked more generally about ideas to start broadening people's knowledge of evaluation in education. They want to interest people not in just the standards themselves, but why education evaluation is important. The purpose is not to sell books, but to actually get people thinking about evaluation.

They talked about many of the questions that people have about education in general, and the reaction. A lot of it has to do with people making decisions and conclusions without much information, without thinking it through. So it seems to be sort of an educational function.

An opinion was given that a lot of this was provoked by a discussion of the on-going problem with people not being aware of the Standards or why they would be important to them, given the role they play. And it may be that some of this language that Dr. Blank is going to propose, could be useful for the proposal itself. There was discussion on how to get people's attention, and if they really can come up with some good language for that, it could go into a proposal or some kind of a brochure.

Other items discussed were issues that educators may have on their minds, and how evaluation connects to what they worry about. It was suggested that the Committee give some ideas to people who are going to be working on the press release and actually try to draft it, and then tomorrow have the whole group look at it. A suggestion for a statement about what the annual meeting, for example, was: "The Joint Committee had its annual meeting, discussed issues facing educators, decision leaders and the public in 1995, and how recent work on standards for good education evaluation can be used. The committee identified several key areas that can benefit from knowledge of sound evaluation." Some suggested items include design of good use of school accountability and reporting, evaluating student learning to fairly assess progress, combining things that the Committee has done and is working toward. Also suggested were: how to assess performance of teachers and administrators against high professional standards of teaching quality.

Abstract of Validation Panel Group Report

This group included Dr. Beatty, Dr. Pullin and Dr. Blank. Dr. Beatty will write up 2-page abstract and circulate it among the group and get it to Dr. Sanders.

They will develop and prepare two kinds of press release. One will emphasize the validation of the Standards and will be written in a manner appropriate to newsletters like Division H, the newsletter for American Evaluation Association, Professors of Education Research, NCME, etc. The second will include more lay language and emphasize the movement into the student personnel standards; Dr. Cutlip will take the lead on drafting that one.

The committee discussed a definition of Student Evaluation differing from other existing evaluation methods and how to communicate that either in a subtitle or list of bullits in the document.

Personnel Evaluation Standards Group Report

This Group included Dr. Thompson, Dr. Rogers, and Dr. Pullin. They made a recommendation to do some minor revisions, and check with Sage on their willingness to publish a revised edition. The group said the Validity Standard was examined and approved as current, but the Reliability Standard could use some work. Additional case studies were recommended to be more current and broader in coverage. The literature review should be updated.

Procedures of the Committee should include a small number of targeted hearings, particularly involving groups such as NEA and AFT and the principal organizations, and have a small review panel go over the revisions, including areas of more movement such as IO area of psychology and teacher organizations.

It is anticipated that a modest amount of funding will be required, but that will be necessary to prevent loss of income on an outdated product. The group recommends establishing a small working group to bring recommendations for revisions to the Joint Committee next year, with total revision time covering 2 years.

Discussion continued on the submission of revised Standards to ANSI. Dan Koretz at the Urban Institute was suggested as contact person on portfolio assessment; and Dr. Sanders will contact Sage regarding details of publishing a revised edition of the Standards.

Dr. Sanders referred to 2 documents to review next. The Committee reviewed and revised the news release on the Joint Committee annual meeting activities for newsletters of members' organizations. Revision of the contact letter to foundations and how this ties in with funding searches, matching with foundations, timing and methods of personal contact were then discussed. Dr. Sanders agreed to develop a work plan with time line for a funding search.

Discussion continued on promotion of The Program Evaluation Standards and the Personnel Evaluation Standards at annual meetings of member organization, states' education departments, regional research organizations affiliated with AERA, targeting professors to use the Standards as text.

Other Business

1996 Annual Meeting

Dr. Sanders announced that next year's annual meeting would be held at Yarrow in Kalamazoo over 3 full days, Thursday--Saturday, October 3, 4, and 5, 1996. The Committee then conducted an election for new term of Chair, Vice Chair and three-person executive committee. In agreement with last year's task force on future governance of the Joint Committee, Dr. Sanders has agreed to serve as Chair for one more term, and Dr. Gullickson has agreed serve as Vice-Chair.

Executive Committee

Dr. Pullin, as outgoing Vice-Chair, chaired the elections. For executive committee the following people were elected: Dr. Beatty, Mr. Mays and Dr. Thompson.

A motion and second was made to accept recommendations of last year's task force, that Dr. Sanders remain Chair, and Dr. Gullickson will serve as Vice-Chair. There was no further discussion, and there was a unanimous vote to elect them a three year term.

The Evaluation Center Contract

A motion was made and seconded to accept and renew the contract for the next 3-year period. Discussion was opened on revision of the current contract with The Evaluation Center. A motion to renew the contract with minor revisions was approved unanimously.

Invitations to New Organizations

The Committee discussed relationships with GAO, and National Legislative Program Evaluation Society (NLPES). The Joint Committee recommended that we explore a cooperative working relationship with GAO. A vote was taken in favor of inviting NLPES to become a sponsoring organization on the Joint Committee. The Joint Committee also voted to extend an invitation to The Association for Institutional Research (AIR) and the American Association of Higher Education (AAHE) to become cooperating organizations. Dr. Gullickson agreed to chair a group to look at the by-laws and principles to suggest wording changes or additions that will provide better direction for membership on the Joint Committee. Other members of the task force will be Dr. Webster, Mr. Mays, Dr. Thompson, and Dr. Bader. The Committee also discussed the addition of two members-at-large if finances permit.

With no further items to be discussed, Dr. Sanders adjourned the meeting.

1995 JC ANNUAL MEETING\MINUTES.95\WP