JOINT COMMITTEE ON STANDARDS FOR EDUCATIONAL EVALUATION

MEETING MINUTES

OCTOBER 29 - NOVEMBER 1, 1992

NORDIC HILLS, ITASCA, ILLINOIS

Thursday, October 29

Dr. Sanders welcomed everyone to the 18th annual meeting of the Joint Committee. The objectives of this meeting were (1) to conduct the business of the Joint Committee on Standards for Educational Evaluation, (2) to review the results of national and international reviews of the draft Program Evaluation Standards, (3) to review plans for the field test and national hearings of the draft Program Evaluation Standards, (4) to resolve issues raised during the development of the first and second drafts of Program Evaluation Standards, and (5) to hear a report from the Validation Panel for the new Program Evaluation Standards.

Dr. Sanders asked members of the Joint Committee to introduce themselves, and he then identified members of the Executive Committee. They were Dr. Diana Pullin (Vice Chair), Dr. Esther Diamond, Mr. William Mays, Jr., and Dr. Ralph Alexander. The Executive Committee is the group that takes action if anything needs to be done between meetings of the entire Joint Committee. Since Dr. Alexander is no longer a representative on the Joint Committee, he needed to be replaced. That election took place later at this meeting.

Since there were several new members appointed to the Joint Committee in 1992, Dr. Sanders asked Dr. Stufflebeam to give a summary of the history of the Joint Committee--where we came from and why we are here. Afterwards, Dr. Hosford thanked Dr. Stufflebeam for his service over the years to the Joint Committee. Mr. Mays supported that statement.

Chair Election

This past year two issues were raised relating to the governance of the Joint Committee. One was that our operating procedures call for an election of a chair every three years and secondly, our by-laws indicated that chairs could succeed themselves once. The members voted during the year to change the by-laws to read that "the chair may be re-elected for additional terms." Dr. Sanders noted that an election for a chair needed to be done and had asked Dr. Pullin to chair the election committee, with the members of the committee being the Executive Committee (Pullin, Diamond, and Mays). Dr. Pullin met with the members of the election committee and nominated Dr. James Sanders to continue as Chair of the Joint Committee. The committee felt that he has done an excellent job. Mr. Mays seconded the nomination. Being no other nominations, Mr. Mays moved that nominations be closed. Dr. Diamond seconded. Vote was conducted by a show of hands. The motion carried unanimously. Dr. Sanders was elected to continue as Chair of the Joint Committee for another three years.

Staff Director's Report

Dr. Torres reported that this past year was very busy with rewriting and revising the Standards, the introduction, and illustrative cases. At the last meeting, the membership recommended persons to participate as reviewers. This resulted in 120 national and 40 international persons being invited to participate as reviewers. We had 91 of the 120 and 24 of the 40 agreeing to be reviewers. The reviewers were sent the May draft of the Program Standards. We received a response rate of 81 percent of the national and 54 percent of the international reviewers. From these responses, the staff developed the field test (August draft) version of the new Program Evaluation Standards.

The Joint Committee used a $25,000 grant received from the W. K. Kellogg Foundation for preparation of the first draft of the new Program Standards. The staff submitted a second proposal to the W. K. Kellogg Foundation requesting continued funding of this work. They granted the Joint Committee an additional $39,500 in March 1992 for this effort.

Mary Piontek is doing her internship with the Joint Committee. She has been involved in setting up the field tests of the new Program Evaluation Standards. Dr. Sanders asked that if the members have any editorial changes that need to be made to the August draft of the book to give them to Ms. Piontek.

Agenda Review

Dr. Sanders then reviewed the agenda for this year's meeting. This agenda called for a report on the national and international reviews of the May draft of the new Program Standards book. There were several issues raised by the reviewers on which neither staff nor Dr. Sanders felt comfortable making decisions so they brought the issues to the Joint Committee to make recommendations, allow for discussion, and to resolve. Work on the issues was scheduled for Thursday and Friday. On Saturday, the agenda called for discussion of a plan for working with publishers, discussion of plans for field tests currently being conducted, and discussion of plans for the national hearings scheduled for next spring. Dr. Sanders also will be recommending that next year's meeting be held in August in order to make final changes to the book and to vote on each standard. This will allow us to give a finished manuscript to the publisher in September and hopefully meet our initially scheduled timeline of printing the book in 1993. Dr. Gould and the Validation Panel will be giving the Joint Committee feedback on Saturday on their observations of the process and for things that they feel the Joint Committee needs to work on. Dr. Sanders then asked Dr. Gould to explain to the members the mission of the Validation Panel.

Validation Panel

The Validation Panel is conducting an outside evaluation of the Joint Committee's revision of the Program Standards. The members of the Panel are representative of most of the new areas being emphasized in this revision of the Program Standards. Dr. Gould noted that it is important to keep a balance between formative and summative evaluation of the process. The Panel hopes that they are providing constructive criticisms and suggestions during the process. They are looking at the process, the quality and utility of the final product, and then reporting publicly on that final product. At this meeting, they will be joining the small groups in order to understand the process that the Joint Committee is going through so that they can comment on it. They are also hoping to attend some of the public hearings and to review interim products. The Validation Panel is also looking at the ANSI (American National Standards Institute) requirements for revision.

Discussion continued on how the Validation Panel plans on publicly reporting their findings. It was suggested that they publish the report in a journal that has a wide circulation. The Panel was also considering giving a copy of the report to the sponsoring organizations for publication in the various journals. Dr. Gould asked that if that avenue is selected that the representatives let them know what limitations are required for the various publications so that the report can be modified to meet those requirements.

Financial Report

Mary Ramlow reviewed the Joint Committee bank account statement (Attachment A) noting the royalty and interest payments for the year and explained the expenses paid from that account. Accompanying the bank statement was a listing of the sales for the program and personnel standards books. Dr. Stufflebeam noted the difference in the sales for the first three years of each book. Dr. Torres stated that was probably due to the fact that more people are in need of a book on evaluation standards for programs, projects and materials than those that are in need for a book on evaluation standards for personnel. Most educational institutions have their own personnel evaluation standards and therefore may not have a need for that type of book.

The Evaluation Center Agreement

During the past year, the Joint Committee has been working with the Evaluation Center to develop a formal agreement for housing the Joint Committee at the Center. Drs. Pullin, Stufflebeam, and Sanders have developed a formal agreement. Dr. Pullin reported that there was nothing that documented the past relationship between the two entities. This sets a formal agreement documenting the relationship and is renewable every three years.

Joint Committee Membership

Membership in the Joint Committee as a sponsoring organization was then discussed. This past year the National Parent Teacher Association withdrew their membership due to financial priorities. This leaves a vacancy for adding a new organization to the Committee. Dr. Sanders also recommended that the Joint Committee drop the American Association of School Personnel Administrators. They have not participated at an annual meeting for the last four years. Dr. Sanders has written them each year sharing with them the Joint Committee's policy that states that it is vital that all sponsoring organizations be represented at the annual meetings. They said they understood that and would like to continue their membership but then wrote and stated that they could not attend this years meeting. Therefore Dr. Sanders moved, seconded by Ms. Wheeler that the American Association of School Personnel Administrators be terminated as a sponsoring organization of the Joint Committee. Discussion followed on actions taken in previous years, a sponsoring organization's financial commitment to the Joint Committee, and an organization's ability to request reestablishment of membership later on. Mr. Mays' called for the question. The motion carried unanimously. Dr. Sanders will correspond with the American Association of School Personnel Administrators.

This leaves two sponsoring organization openings. The Joint Committee is not required to fill those vacancies but Dr. Sanders recommended, and then moved, that an invitation to join the Joint Committee as a sponsoring organization be extended to the Council of Chief State School Officers. They represent the state school superintendents. Dr. Johnson seconded the motion. It was felt that this was a very good recommendation. After discussion, the motion carried unanimously.

Other organizations recommended for consideration were ASTD (American Society for Training and Development) and NSPI (National Society for Performance and Instruction). These are organizations that would represent persons from the areas that we are trying to include in our Program Standards revision. Dr. Gould stated that NSPI had contacted him about the revision process and asked him to come and give an informal presentation about the Standards. Dr. Torres mentioned that when NSPI was contacted for possible field testers, they were very helpful. Other organizations mentioned were Education Commission of the States, National Governors Council, and the National Association of Early Childhood and Young Children (NAECYC). Dr. Sanders suggested that in order to make an informed decision about another organization that we review information about the organization. Dr. Pullin suggested higher education associations be considered as well since they are being asked to be more accountable with this revision of the Standards.

Future Joint Committee Ventures

The Joint Committee has developed a proposal for standards for evaluations of students that has been put on the back burner due to the revision of the Program Standards. Dr. Sanders sees this as something the Joint Committee will be working on in the future as well as reviewing the Personnel Standards to see if they need to be revised. The review cycle for the books is every five years. Dr. Sanders clarified that even though standards are to be reviewed every five years the policy does not necessarily mean that they will need to be revised. It was also suggested that another objective for the future might be to make both sets of standards user friendly. This in combination with the review process should keep this group very busy. Dr. Stufflebeam noted that in the Joint Committee By-laws it states that task force groups may be developed to produce items for the Joint Committee's review. They could be asked to produce first drafts as an example.

Report of the National and International Reviews of the First Draft of the new Program Evaluation Standards

Dr. Torres reviewed the Executive Summary of her report on the National and International Reviews. There were invitations sent to 120 national and 40 international persons to participate in the review, 115 agreed to participate of which 87 actual responses were received. The overall response rate was 75.7 percent. The reviewers were asked for written comments in eight areas: (1) responsiveness to current concerns/issues in program evaluation, (2) scope/comprehensiveness of the 30 standards, (3) utility of the standards, (4) political soundness/viability of the standards, (5) appropriateness of language, (6) organization of the document, (7) any other comments/reactions/suggestions, and (8) to make any editorial changes and suggestions for improvement on the document itself. Dr. Torres took the changes and suggestions and applied them to The Ethnograph software, a computer program for the analysis of text-based data. She then explained how the software worked. Dr. Sanders then reviewed each marked copy and combined the comments onto one copy of the draft to produce the August 1992 field test draft.

Dr. Torres received generally positive responses to the standards. It was noted that the reviewers liked the applications of the standards to settings beyond school settings. Reviewers also commented on how the Standards should be used, their impact on the profession, the approach to evaluation supported by the Standards, and to specifics about evaluation methodology. Issues that the staff didn't feel comfortable in handling were then assigned to task groups at this annual meeting of the Joint Committee for recommendations.

The task group assignments and members were as follows.

Task Group 1

The issues were:

1. Standard A5 (Valid Measurement) addresses both measurement and design validity. Should it? If so, should it be renamed Valid Information Collection, or simply Validity?

2. Standards A5, A6 (Reliable Measurement--should it be Reliable Information Collection or Reliability?) and A11 (Objective Reporting--Fair Reporting?) are antithetical to the conceptual frameworks used by credible qualitative methodologists. There is a growing segment of the evaluation field that sees valid measurement, reliable measurement, and objectivity as inconsistent with their ways of evaluation. Should we change the names of these standards and rewrite the overviews, guidelines, common errors, and illustrative cases to accommodate this segment of the field? if we do, should we combine A8 and A9, Analysis of Quantitative and Qualitative Information into one standards (Appropriate Analysis of Information?) that reflect both methodologies?

The members of this task group were:

Esther Diamond Daniel Stufflebeam (Chair)

Joy Frechtling Bruce Thompson

W. Todd Rogers Rosalie Torres

Task Group 2

The issues were:

1. In our move to consolidate certain standards, the timeliness of reporting concept has become buried in U6 (Report Dissemination). Both dissemination targets and timeliness of reporting are critical concepts and there are those who believe they should be kept separate (as in the 1981 Standards, A6 and A7).

2. The appropriateness of new standards, P1 (Service Orientation), and A12 (Metaevaluation). There are some who view P1 as unnecessary since evaluation is a service. Others believed that P1 is covered by the introduction and standards A3, A8, and A12. One reviewer commented that P1 moves us out of evaluation and into educational psychology (teaching and learning). With A12, the concern is that metaevaluation is a function comparable to the other functions in our functional table of contents, i.e., it is an important part of the evaluation process rather than a principle of good evaluation practice. To back this up, one reviewer noted that A12 is included in 9 out of 10 functions in the functional table of contents. Why not take it out of the list of standards and make it a function?

The members of this task group were:

Oliver Cummings James Sanders

Philip Hosford (Chair) Barbara Wheeler

Henry Johnson

Task Group 3

The issues were:

1. If the Standards are going to be used, they need to be user friendly. Some see this document as cumbersome and not something that will be picked up and used by most educators (and many evaluators.) We seem to be violating our own principles. How can we make it more user friendly or should we?

2. The limits of application we should claim for these standards. Some feel strongly that the Standards apply to evaluations of (1) social programs, (2) institutions, e.g., accreditation, and (3) state and local accountability systems. What changes, if any, should be made to address this issue.

The members of this task group were:

Mary Piontek Jeffrey Wadelin

Diana Pullin W. Eugene Werner

William Mays, Jr. (Chair)

Dr. Rogers noted that people get informed about things through other people. He brought up the question of what responsibilities do we have to disseminate this book within our associations/ organizations? This dissemination question would be discussed later.

Dr. Frechtling asked the question of who the audience was for the book. What are their needs? And does this book meet them? People that use the Standards need to be made aware when the book is available. Dr. McGuire thought that students in training for evaluation and professional evaluators are the main audiences. This document sets forth what the evaluators are to be doing. The contractor and the consumer are users. If others are to be using this book, there was concern that they won't see it as a user friendly document. It may need to be organized by role to get to those people. Task group 3 would be addressing this issue.

Friday, October 30

Dr. Sanders convened Friday's meeting by asking for a progress report from each of the three task groups.

Dr. Stufflebeam asked if it was correct that the groups were doing fine tuning at this point versus major rewrites. Dr. Sanders agreed. The groups should make major revisions only if there are major flaws in this version. Previous revisions included major rewrites. The primary focus at this point is the task group responses to their assignments.

Task Group I: Quantitative/Qualitative Issue

Dr. Rogers reported that the group discussed the issues across the standards listed. Then each worked on editing a standard to incorporate as much as possible a balance of quantitative and qualitative perspectives. They will be reconvening to review those edits as a group and report back to the Joint Committee.

Dr. Pullin asked if someone was going to review the bibliography. Dr. Sanders asked all members to help with this. Dr. Torres and Mary Piontek have been working to update the bibliography but any help would be appreciated. Dr. Pullin asked if ERIC could do the work for us. Dr. Frechtling suggested that Larry Rudner could be contacted to see if he can help. Dr. Stufflebeam inquired as to whether it would be possible to do an organized search according to each of the standards. Dr. Sanders stated that it may be possible through Western's Waldo Library. Dr. Frechtling also stated that John Forsythe at the Educational Resource Center (ERS) collects all reports that come out of school systems and then indexes them, if we want that type of input. Dr. Frechtling was asked to contact him.

Group 2: To look at the consolidation of some of the standards to see if anything has been lost due to the revisions. And also to look at the two new standards (Service Orientation and Metaevaluation) to see if they should be kept, not kept, or revised.

Dr. Johnson reported a group recommendation that Timeliness and Dissemination-U6 should not be two separate standards but should revise the U6 heading to Report Timeliness and Dissemination. They suggested that the wording in the body of U6 be revised to more reflect the issue of timeliness. The suggested wording was reviewed. They also suggested revisions of Service Orientation (P1) and Metaevaluation (A12).

Group 3: To look at the user-friendly issue across the document and to look at the limits of what we want to claim for application of the standards.

Dr. Werner reported that in order for this document to be user-friendly, they had to define who the user was. If it's from the perspective of the accrediting agencies, state departments of education, or professional educators, then it is not user-friendly at that level. Do the standards need to be user-friendly for the contractor who needs to be comfortable with the speed and direction of the evaluation, for the consumer who is affected by the result(s) of the evaluation, or for the first-time user looking at the standards as principles guiding the evaluation? Questions discussed by the group were: (1) Should the evaluator go to the back section first to find out how to use the standards during the evaluation so that the standards can be used correctly? If the answer is yes, then why not place the second part in the front where it will be read first. (2) Does Part II fulfill user needs for application? If the answer is no, more specific, detailed steps need to be added for the application. (3) Should the group consider a redesign of Part II? After considerable discussion, the group decided that it certainly should be done. About 10-12 pages of Part II added to parts of the drafted Introduction give better direction to that do-it-yourselfer or the first-time user. (4) Should that be a separate document outside of the present document? The feeling was yes but that it be one or two pages with reference to the main document. (5) Are there other modifications that could be made in Part II to make it more user-friendly? Yes, the group felt that there could be an expanded and improved citation form to include the step-by-step process and to develop a generic, abbreviated illustrative case that represents an evaluation that teachers/principals can relate to. Dr. Rogers also stated that there should be a limit of only two illustrative cases for each standard. Some have one and some have three. They should also be in a positive tone throughout the standards. These questions were discussed at length by the group.

All of this is impacted by how far the Joint Committee wants to go to make the document "user-friendly" for all of the various consumer groups. Several pages were referred to in the Introduction that define who the users of the Standards are. Those audiences should also be clear throughout the document. Discussion continued on the questions stated above, modifications to the citation form with inclusion of a column stating "that the standard was applicable but not met", and the document itself being in two parts. Dr. Pullin felt that this division diverts you from the purpose of the book. Dr. Rogers concurred. Dr. Stufflebeam thought that having all the pieces stating how to do the evaluation in the front of the book would be a good idea.

Dr. Stufflebeam mentioned concern about the standards being readable once the document is in final form size (approx. 5" x 7"). Dr. Frechtling mentioned that there was no way to make the document usable instantly by everyone. Companion pieces would assist in doing that. But to do that at this time would delay reaching the production date and affect the budget. Ideas such as a detachable booklet accompanying the book, and an interpretation piece or a computer disk giving step-by-step directions were mentioned as possible companion-awareness pieces. Who will be buying this book? These pieces should be aimed towards them. Concern was mentioned regarding the licensing and liability of those type of documents.

Dr. Sanders asked the members if the changes in the Introduction were what they want. Dr. Stufflebeam asked the members to read the modified introduction in its entirety prior to making that decision. Dr. Basarab had copies made for the members review. Dr. Sanders asked this task group to do two things: (1) to make a recommendation regarding companion-awareness pieces and what they would look like, i.e., user guide, outline form, etc.; and (2) to make a recommendation on changes to the introduction especially in the sections concerning audiences. At this point, Dr. Sanders asked the Joint Committee to break into small groups for the rest of the day and to reconvene as a large group on Saturday.

Saturday, October 31

Dr. Sanders reconvened the Joint Committee with a number of business items to discuss. He asked that editing to the Standards be given to Mary Piontek for compilation. One of the business items to discuss was a vacancy on the Executive Committee due to Dr. Ralph Alexander's resignation. Dr. Pullin chaired the elections committee to fill that vacancy. Dr. Diamond moved that Dr. Todd Rogers be nominated to fill that vacancy. Dr. Mays seconded. Being no other nominations, Dr. Sanders asked for a vote on the motion. The motion carried unanimously. Dr. Sanders welcomed Dr. Rogers to the Executive Committee.

Next discussed was the 1993 meeting. Dr. Sanders suggested that next year's meeting be held in August to give the Standards its final review. This will allow the Committee to go through the Standards one by one and vote on each one individually as a Committee. By doing so, Dr. Sanders is hoping that the manuscript will be to the publisher by September in order to have it printed in 1993. Dr. Sanders recommended meeting August 12-15, 1993, in Chicago because of its ease of access, and it is cheaper for some of the members, the Joint Committee staff and the Validation Panel members to attend. He was open to suggestions on the location. The date was first discussed. Keeping in mind APA conference dates, it was recommended to hold the meeting August 12 to 15, 1993. Being no discussion on the place, Dr. Sanders will look at various places in the Chicago area and will inform the Committee once a decision has been made. The Task Group Reports were then heard.

Task Group 1: To look at the standards on valid measurement, objective reporting, and qualitative and quantitative analysis.

Dr. Rogers reported that the group recommended changing the name on A11 (Objective Reporting) to Impartial Reporting. The second recommendation was to retain A5, A6, A8, and A9 as separate standards. They did some editing to better balance the qualitative and quantitative aspects of those standards. The group recommended that when there are references to reports, they should be cross referenced to Standard U4.

Task Group 2: To look at the old standard relating to Timeliness and Reporting to see if something was lost in the revision process going from the old to the new, and to look at the new standards P1 (Service Orientation) and A12 (Metaevaluation) to see if there are recommendations that should be made regarding their inclusion and the way they were written.

Dr. Johnson reported that the first recommendation was to modify U6 to read "Report Timeliness and Dissemination" with that modification made throughout the text of the standard. The second recommendation was in modifying the P1 text to pick up the concept of community and society in the standard, overview, and guidelines and leaving references to needs within the standard. Those modifications were discussed. The third recommendation was that A12 (Metaevaluation) should remain a part of the document, but the text was modified. Discussion took place regarding whether A12 is a function or a standard. It was agreed that it should stay as a standard. Dr. Stufflebeam recommended that the staff think of these recommendations as working hypotheses since the field testers are using a different version of this document than what will come out of the meeting.

Task Group 3: To look at the question of user-friendliness of the standards and to the limits to which we might want to make claims about the applicability of the standards.

Dr. Werner reported that the group identified three items that needed to be revised in the Introduction. One was that different audiences will have different uses for the Standards. Two, the task group added a different lead into the case study. Three, the task group edited and moved Appendix B into the Introduction.

Dr. Sanders felt that the Joint Committee has resolved some very difficult issues. Recommendations will be incorporated into the next revision. The next item for discussion was the issue of publishers. After negotiations with Sage have taken place, a letter to McGraw-Hill will be sent asking for termination of our current agreement and that the 1981 Standards be taken out of print. Dr. Sanders asked if the Joint Committee felt that user-friendly, computer-assisted programming to accompany the Standards was important. Dr. Hosford asked Dr. Sanders to check with Sage to see if they have any potential to put together a computer program to accompany this book. It was also suggested that Dr. Sanders consider revising the publishing date to 1994 so that we don't lose a year of life from the copyright. Dr. Stufflebeam noted that if it is published after October 1, that you can use the next year's date for copyrighting purposes.

Dr. Sanders then asked the members to look at the field testing that is currently underway. Dr. Sanders asked Mary Piontek to describe the procedures that were followed and to talk about who is doing the field tests, in what kinds of settings, and what they are to do. She reviewed how the participants were selected and publications announcements that were used. There are 70 persons doing the field testing from across the U.S. The reports from the field testing are due December 15, 1992. Dr. Hosford commented that this was an excellent report and asked if Mary Piontek had a sense of how many of these people would be doing a metaevaluation versus a current evaluation. Ms. Piontek said that many of them are going to go back to an evaluation that had already been done. Dr. Torres stated that there were so many phone calls that they started targeting the field tests so that input from a good cross section of settings and uses would result.

The next item for discussion was the National Hearings. Dr. Sanders reported that hearings at two sites were currently scheduled. One is at AERA in April. The other is at NEA in May. NEA is willing to provide facilities and to share a mailing list of associations in the Washington area. He then asked for volunteers from the Joint Committee to conduct hearings in conjunction with their organization's annual meeting or at some other location. Dr. Sanders will develop an announcement that will be sent to each member to put into their organization's newsletter. A question was raised about whether there are any restrictions on the number of people that can be involved in this process. Dr. Sanders stated that a cap will have to be placed in order to have funds to produce and distribute the copies. Dr. Stufflebeam commented that Dr. Sanders may want to set a ceiling and then offer additional copies to others at cost. Concern was stated that the scheduled hearings are only on the east coast. Something should be held on the west coast. Dr. Frechtling stated that the NSBA conference will be held in Anaheim, California and asked if Dr. Wheeler would also be going and maybe they could assist in holding a hearing. Dr. Sanders took Drs. Frechtling and Wheeler up on their offer to hold a hearing in Anaheim. Dr. Stufflebeam stated that in order for these hearings to be cost effective, we should make sure that people will go to the hearings and that testimonies will be given. Dr. Thompson suggested that we should make the Standards available in other ways at these conferences such as at a booth where the books can be on display. This would make more people aware of the document, what is going on, and possibly get their input. Dr. Sanders informed the members on how these hearings are conducted. He asked for the members view on this process. A question was raised about ANSI requirements for this process. Dr. Sanders said that ANSI approved the Joint Committee's operating procedures. Strategy was discussed on when the hearings should be scheduled, how long they are to be, and who will be able to help. A question was raised about when these hearings need to be completed. Dr. Sanders said that he hoped they would be done by the end of May so that he can review the testimony, make revisions, and then get an updated version of the new Standards to the members for their review one month in advance of the August meeting. Dr. Stufflebeam stated that The Evaluation Center's CREATE project will be having a booth at AERA and they had the Standards displayed there last year and would be willing to do so again this year.

Dr. Stufflebeam suggested that maybe a contingency plan should be developed if something is uncovered at the hearings that would affect the publication timeline. Dr. Thompson thought it might be a good idea to have a few people go up to Michigan and assist in the development of the final draft for the August meeting. Dr. Frechtling asked for Dr. Stufflebeam's opinion on how large a volume these hearings could generate. Dr. Stufflebeam reported that they received quality information since they were able to make sure that particular people were involved in the testimonies at the hearings. But they also received numerous comments on the backs of envelopes, slips of paper, etc. and all of these pieces need to be compiled for review and possible inclusion in the final draft. Dr. Torres reviewed the timeline from her perspective of results of the field tests and hearings. The time between the hearings and getting the final draft to the members for the next meeting will be a very short period. On Dr. Rogers recommendation, Dr. Sanders asked if any of the members would object to receiving the final draft of the document around August 1, 1993. Hearing no objection, Dr. Sanders will keep that in mind. Dr. Pullin suggested setting an October meeting time to get it on member's calendar just in case the August deadline can't be met. October 14-17, 1993, were suggested as contingency dates. The group then adjourned for lunch.

Dr. Sanders then asked Dr. Gould to give the Validation Panel Report. Dr. Gould said that when he arrived at this meeting, he had a set of concerns and questions that he felt the Committee needed to address during this process. During the last few days, these concerns and questions plus ones from other members of the Panel were systematically addressed. This is a strong endorsement of the process the Joint Committee uses, the way it goes about reaching consensus and draws from a wide range of expertise. The Committee has adhered to its own standards and in the process developed standard A12 (Metaevaluation) and then commissioned the Validation Panel to help meet that standard. He suggested writing about how the development of the standards was faithful to the standards. Dr. Gould was puzzled over how to write that in without breaking the flow. Maybe there should be a section that addresses in a paragraph or so what the Joint Committee's process was to ensure content validity of the standards. One thing that the Joint Committee has not addressed is what happens when we come to an impasse. Dr. Gould asked Dr. Basarab to state a suggestion for that concern. Dr. Basarab stated his view as an outside observer. A buzz word in the business world today is empowered work teams. What he has used at decision-making blocks is a consensus method. The chairman watches the discussion and if after three rounds of discussion there is no new information added, the chairman calls for a decision. It makes it a little bit quicker to go on if all you are doing is rehashing the same points. There are other ways but that is the simplest.

The Panel next addressed the applicability of the standards through a variety of audiences. These standards can't do everything for everybody listed as audiences but as Group 3 made the standards easier to use most concerns were erased. They suggested that we need to address not only who the users are but define what the needs are of some of those users. Dr. Basarab had an idea on how to go about doing that. Dr. Basarab and the Panel members recommended that the Joint Committee consider getting an external editor who does not live and breathe the standards and charge the editor with looking at grammar and passive voice. The Panel recommended contacting someone who is not an evaluator, who is not familiar with the standards, and who is not looking for content but is looking to bring the document to an easy readability level.

Dr. Gould was very impressed to see how the staff organized the reviewer results for inclusion in this current draft of the report. This is so important, that he strongly invited the staff to use the same degree of rigor for the field test results.

Dr. Basarab wanted to comment on Dr. Stufflebeam's earlier concern about the timeline. Deadlines should be moved in order to follow the process rigorously versus trying to cut corners so that you can still make the deadline. It is more important to do what is already set as a precedent. Dr. Basarab stated that we should not state that the standards are applicable in an arena if they have not been field tested in that arena. There will be no data to back it up.

Dr. Gould commended the staff in their coverage of the various audiences in the field testing, reviewers, and process. Keep an eye on the smaller groups so we don't lose their views. On the hearings, Dr. Gould felt that having other members of the Committee conduct those hearings was a good way to not over extend the staff. Dr. Gould also requested that when the schedule of the hearings is finalized that the Validation Panel get a copy so if they are in the area they would be able to attend.

On user feedback and dissemination issues, Dr. Gould felt that we had touched on the concerns the Validation Panel had in the beginning. A formal plan for dissemination would allow everyone to give suggestions. The revised citation form presented at this meeting is an outstanding idea but it is his understanding that the response from the current set of standards has been somewhat marginal. The people who use the Standards are not using that form and sending it to the Joint Committee. An issue to address is how that can be resolved. Dr. Gould suggested getting a mailing/label list from the publisher covering who orders the book and follow up with a letter to consumers letting them know that we really want their feedback if they used the Standards. This can also be done through sponsoring organization's newsletters. People who have the old books need to know that the new book will be available. Maybe the new book can be marketed before it is printed by stating that they can get it at a reduced price if they order it before it hits the shelves.

The Panel also felt that the Committee made a good effort to address last year's problem of positive versus negative illustrative cases. Dr. Gould suggested that each illustrative case be reviewed to add positive notes to a negative case. A final point that was discussed was where the Validation Panel's final report should be disseminated. Dr. Rogers gave Dr. Gould earlier in the year page counts, publications, and recommendations for possible outlets. Dr. Gould asked that the other members get that same information to them for disseminating their final, public report. The Panel will probably write an executive summary and a full report that can be modified for any page restrictions that might be necessary.

Dr. Frechtling raised a concern with the Panel's recommendation for an outside editor for this document. She felt that language such as quantitative and qualitative, etc. may become more political when edited by an outsider. Dr. Basarab stated that the Panel was only coming from the perspective of those "technical" words that should be defined in laymen's terms so that the Standards will be used. With no other comments or concerns, Dr. Sanders thanked the Validation Panel members and stated that their points are well taken.

Dr. Sanders noted that he will produce a summary of what has transpired at this annual meeting. He will leave it up to the members to get it published in their organization's publication. Mary Ramlow will produce more detailed minutes.

Dr. Gould commented that he looked at the reviewer's comments that he felt were noteworthy and checked to see how they were handled in the August draft of the document. Issues that he thought were worth addressing were covered in the August document. He felt that this was a very strong endorsement for what the staff did with those results.

Next discussed was the dissemination issue. A dissemination plan was developed in 1987 and was reviewed at that time by the Joint Committee. A funding proposal was produced to get support to carry out this plan. Funding requests were not successful. Dissemination is a concern and as user friendly issues were discussed and ideas presented, we should consider user friendly action to take. Dr. Wheeler felt that people have to know that the Standards exist in order to use them. A very inexpensive way to get the word out about the Standards, is to have a panel at the national conventions, and have people comment on how they will be effective in changing or evaluating programs. People who attend these panels will go back and say there is a tool that can be used to increase their effectiveness. With little or no effort and no cost to the Joint Committee, Dr. Wheeler felt that a panel could be arranged at NSBA and that AASA would also be agreeable. Some of the larger state associations are looking for things that they can show to their constituents to demonstrate that they are rigorously evaluating what they are doing. Dr. Sanders stated that in past year's, members were asked to come to the annual meeting with a plan for presentation of the Joint Committee documents/results to their organization. Dr. Wheeler said that she could ask her superintendent to put a panel together and ask NSBA to put us on the national convention program. Dr. Sanders asked the members to bring a plan to the 1993 meeting with dates and a proposal of what they would like to do the following year.

Dr. Pullin asked if there was some way that several of these national organizations could get this information to the state and/or regional programs for conferences. She felt that it would widely increase the number of people who would be interested in an opportunity to learn about the Standards. If there was some way to encourage presentation at state levels by the organizations, we would reach the audiences that we are least likely to reach with other dissemination plans. Several ideas were discussed. Dr. Johnson mentioned that he would be able to contact various levels of his organization. Dr. Stufflebeam recommended using the reviewers and field testers who are real knowledgeable about the Standards to help us with these types of sessions. Dr. Rogers mentioned that he had already looked at the list of reviewers and noted two Canadians who he was going to ask for help in getting the word out about the Standards.

Dr. Thompson mentioned that another idea would be to have people do book reviews at the same time the book is being typeset. That way the book reviews would come out about the same time the book itself is released so they could reinforce each other. Dr. Basarab suggested that Sage do a marketing brochure to promote the work. Dr. Stufflebeam mentioned that Sage does a wonderful job of that. Dr. Rogers recommended having a pocket book size summary that could be taken anywhere. He also asked that when we contact Sage we ask them for ideas to promote the book.

Dr. Pullin inquired whether the Joint Committee has any affiliate organizations. Dr. Sanders reported that the Joint Committee has defined a cooperating organization status. These are organizations that are non-voting but are informed about the work of the Joint Committee. At this time, there are no cooperating organizations. Dr. Pullin stated that getting some of those organizations would dramatically expand our repertoire of contacts. Not being sure what a cooperating organization is, Dr. Pullin suggested that they could become "active" participants in disseminating the word about the existence of the Standards. Dr. Sanders read the section on cooperative organizations from the bylaws. Dr. Pullin was thinking of the organization having input into the process and participating in the dissemination. Producing a press release for the Chronicle and other outlets would be another avenue to investigate as well as the bulletin board systems on computers.

Dr. Sanders thanked everyone for their participation in the annual meeting. It was a very productive meeting. A special thank you to Dr. Henry Johnson for filling in at the last minute for AASA. Dr. Sanders stated he was looking forward to a productive year and looking forward to meeting the group next August 12-15, 1993. Dr. Rogers thanked the staff for their hard work and wished them good luck in their future work. With no other business to discuss, Dr. Sanders adjourned the meeting.

Respectfully submitted,

Mary E. Ramlow, CPS

Secretary to the Joint Committee