ANNUAL MEETING OF THE

JOINT COMMITTEE ON STANDARDS

FOR EDUCATIONAL EVALUATION

Nordic Hills Resort

Itasca, Illinois

August 12 - 15, 1993

Attendees:

Joint Committee Members:

Dr. James R. Sanders, Chair

Dr. Beth Bader (NEA)

Dr. Robert Baker (APA)

Dr. Oliver Cummings (AEA)

Dr. Esther E. Diamond (AAC)

Dr. Joy Frechtling (AERA)

Dr. Philip L. Hosford (ASCD)

Mr. William Mays, Jr. (NAESP)

Dr. Diana C. Pullin (Member-at-large)

Dr. Rodney Riffel (NEA)

Dr. W. Todd Rogers (CSSE)

Dr. Daniel L. Stufflebeam (Member-at-large)

Dr. Bruce Thompson (NCME)

Dr. W. Eugene Werner (NASSP)

Validation Panel Members:

Mr. David Basarab

Dr. R. Bruce Gould

Dr. Phil Robinson

Dr. LeRoy Walser

Dr. Alexandria Wigdor

Joint Committee Staff:

Ms. Mary E. Ramlow, Secretary

JOINT COMMITTEE ON STANDARDS FOR EDUCATIONAL EVALUATION

MEETING MINUTES

AUGUST 12 - 15, 1993

NORDIC HILLS, ITASCA, ILLINOIS

Thursday, August 12, 1993

Dr. Sanders welcomed everyone to the 19th annual meeting of the Joint Committee. The objectives of this meeting were (1) to conduct the business of the Joint Committee on Standards for Educational Evaluation, (2) to review the results of the field tests and public hearings for the draft Program Evaluation Standards, (3) to make final revisions and approve the publication of the Program Evaluation Standards, (4) to hear a report from the Validation Panel for the Program Evaluation Standards, and (5) to plan Joint Committee activities for 1993-94.

Dr. Sanders asked members of the Joint Committee to introduce themselves, and he then identified the members of the Executive Committee. They were Dr. Diana Pullin (Vice Chair), Dr. Esther Diamond, Mr. William Mays, Jr., and Dr. Todd Rogers. The Executive Committee is the group that takes action if anything needs to be done between annual meetings of the entire Joint Committee. Since Dr. Diamond and Dr. Rogers will not be representatives on the Joint Committee after this meeting, an election of their replacements was scheduled for later at this meeting.

Since there were several new members appointed to the Joint Committee in 1993, Dr. Sanders asked Dr. Stufflebeam to give a summary of the history of the Joint Committee--where we came from and why we are here. When he had finished, Dr. Walser added that the Program Standards are accredited by the American National Standards Institute (ANSI). ANSI accredits standards setting groups. Dr. Stufflebeam said that he hoped that the revised Program Evaluation Standards will be approved by ANSI. It was stated that the Joint Committee review process is in 100 percent compliance with ANSI guidelines. Dr. Sanders noted that once this draft is finalized, it will be sent to ANSI for national and international review. They require about 90 days to complete the process. Any concerns will be sent to the Joint Committee for adjudication. The Personnel Standards have not been approved by ANSI since the Joint Committee had published them at the time they were given ANSI accreditation. If The Personnel Standards are revised, then they will be submitted for review and approval by ANSI. Dr. Walser reminded the members that ANSI accreditation is only good for five years. A five year review needs to be done on accredited standards to see if they need to be revised. This process is parallel to procedures used by the Joint Committee.

Dr. Sanders thanked Dr. Stufflebeam and mentioned that the Joint Committee is a working committee and much work needs to be done at this meeting. In doing the final revisions of the Standards, Dr. Sanders suggested that they provide comments and discussion as a large group then break into small groups of two to review the Standards individually making the changes according to the large group discussions. The edits will then be finalized in word processing and the final product will be used as the copy to be given to the publisher.

The Validation Panel was introduced. Dr. Gould asked for names and addresses for places to publish the Validation Panel report. Any ideas would be appreciated. A draft of the Validation Panel report will be sent to the Joint Committee prior to publishing. The last Validation Panel report, done for the Personnel Standards, was published in the Journal of Personnel and Educational Evaluation.

Dr. Sanders then updated the members on the issue of publishers for the revised Program Standards. He has received notification from McGraw-Hill that it will take approximately six weeks to double check on whether there are no outstanding rights to the original Standards. That was about the middle of July, 1993.

Dr. Sanders distributed a memo from Meg Blinkiewicz dated July 29, 1993, that relayed her conversation with attorney Mr. Gary Tibble on liability insurance as requested by the members at the last meeting. That memo stated that most organizations usually carry a policy on their members but also indicated that individual members could buy an umbrella $100,000 policy that would cost approximately $125 per year. Dr. Pullin asked the members to check with their organizations on that coverage. She did not feel that it was the norm for most organizations.

Dr. Sanders then informed the members that Dr. Frechtling was awarded a contract from the National Science Foundation that has the Joint Committee Standards will be used for evaluation training at the federal level.

Dr. Sanders then asked Mary Ramlow for the financial report of the Joint Committee. Mary distributed and reviewed her report. The financial resources of the Joint Committee as of July 31, 1993 were $19,517.73. From this point forward, expenses will be charged to this account since there are no other funds available.

The Field Test Report from The Program Evaluation Standards revision process was next discussed. The February, 1993, draft of the revised Standards was based from this report and on the results of the 1992 Joint Committee annual meeting. The February 1993 draft was used for the hearings process. Four Hearings were held; Washington, DC; Atlanta, Georgia; Banff, Alberta, CANADA; and Anaheim, California. The hearings results were used to develop the August, 1993, draft of the Standards.

Dr. Sanders reviewed the comments that were received from participants in the hearings, and the action that was taken for each, if any. Discussion followed. During the review, Dr. Sanders referred to a memo he received from Meg Blinkiewicz on her analysis of the illustrative cases in this revision. Her analysis confirmed comments made by those at the hearings that diversity had been well served. Dr. Gould mentioned that Dr. Sanders and members captured most of the comments made at the Hearings except for the ones that stated the Standards were good and needed. Dr. Sanders responded that the hearings report was used to identify issues and actions, but that a good majority of comments at the hearings were indeed laudatory.

One of the biggest concerns raised at the hearings was that the Standards are intimidating because of size and substance. They need to be more user friendly. Dr. Sanders mentioned that people at the National School Boards hearing especially noted a need for derivative documents.

Dr. Frechtling raised a concern mentioned at the Hearings about the use of the word of stakeholder. Do we mean it? When the Standards are reviewed, we must carefully look at how we use the word stakeholder and make sure that is what we mean. Dr. Baker mentioned that another comment made at the Hearings was whether reports should be written or oral. If we are talking about a group coming in and evaluating our program, we need a written report. If they do an oral report, there is no accountability. Dr. Thompson was surprised at the Hearings with the AERA and NCME that people were not aware of the Standards even though they have been out since 1981. It was mentioned that not everyone rushes out and gets a copy of the book. Dr. Baker stated that this is echoed throughout the students in his doctoral program. These students are teachers with four to five years of experience and 50-60 percent of them are in administrative positions in the schools. They did not know the original book was even available. We really need to spend time on the derivative items and make them user friendly. Time was scheduled on the agenda to discuss this on Saturday.

When returning from break, Dr. Stufflebeam commented that this document is a substantial improvement over what was here last year. At previous Joint Committee meetings on the other Standards, they were reviewed line by line and then the members voted on each standard which wasted valuable time in making a great document. Dr. Stufflebeam made the motion that the August 1993 draft of the Standards be accepted by the Joint Committee in its entirety, knowing that some changes will be made at this meeting. A new document will be compiled by Dr. Sanders and then the final document will be sent to each member after this meeting with a ballot to vote their approval of the document for publishing. Dr. Hosford seconded the motion. The motion carried unanimously.

Dr. Sanders then assigned Joint Committee members into the following teams to take notes on discussion and to make final revisions:

Drs. Bader and Baker will do standards U1 through U4.

Drs. Cummings and Diamond with do standards U5 through F1.

Drs. Frechtling and Hosford will do standards F2 through P2.

Drs. Johnson and Werner will do standards P3 through P6.

Drs. Mays and Pullin will do standards P7 through A2.

Drs. Riffel and Rogers will do standards A3 through A7.

Drs. Thompson and Wheeler will do standards A8 through A12.

Drs. Sanders and Stufflebeam will do the Preface, Introduction, and Appendix.

Dr. Stufflebeam will assist Dr. Sanders in reviewing these compiled reports and producing the final document. Dr. Sanders then reviewed the procedure to follow in reviewing the Standards. We will go through the Standards section by section allowing for discussion of each piece. If you have written items down on your copy of the Standards, tear them out and give them to the appropriate team member for the team to add to their work. Team's are also to do the proofing of their section as well.

Dr. Gould gave some Validation Panel observations to consider while reviewing the standards. The issues were: (1) indicators/criteria should be set before review so it can be implemented by everyone; (2) a Standards Guide versus a How To document, make it clear throughout, and identify possible solutions; (3) define target audience; (4) do the Standards stand the test of time, independent of changes in the world where the Standards operate; and (5) the use of ideal cases versus reality. Dr. Wigdor felt that item one could be handled by raising the issues in the Introduction and then clarifying them in the Standards. The book is not a how-to-do book. How are you really supposed to use these Standards? Maybe the way to address this is to specifically state the uses.

The members then reviewed the book together. The following are a list of items that were discussed for the beginning section (table of contents, preface, and introduction).

1. Dr. Pullin suggested adding a paragraph on page 7 that states how the functional table of contents should be used.

2. On page 11, 2-4 foundations are listed as supporters of our work. These were for the first edition not this revised work. The wording should be revised to reflect this.

3. The citation form is used more internally than in formal situations. Dr. Thompson suggested that we send people the citation form since we received very few in the past. Dr. Frechtling stated that the citation is not an inviting form to use. We need to find a way to make it more appealing. Dr. Pullin suggested that having more space between the Standards would make it easier to read. A reference on page 19 states how the form should be used. Dr. Hosford suggested putting the address information on a separate page. Wording was discussed (e.g., applicable versus addressed, met versus applicable). Dr. Rogers suggested that Invitation to Users on page 12, pages 43-45, and page 301 should be put together and placed at the end of the introduction. Dr. Hosford felt that the solicitation should be in the back of the book but that the Citation Form should be with the Applying the Standards section. Dr. Pullin had an insert for page 16, last paragraph. It was read and agreed to be used with minor revisions.

4. Dr. Werner felt that using educators was misleading. Is it being used in the broadest sense? Dr. Walser recommended using trainers since that is more commonly used among the people in the new areas. It was agreed to use both.

5. Page 18, paragraph 3. Dr. Stufflebeam thought it might be more useful to describe what the standards are versus what they are not. Dr. Frechtling asked if this would be the point that we add that this is a "guide" not a "how-to" book. Dr. Rogers stated that it needs to be consistent with the functional table of contents in the book.

6. It was noted that clients and stakeholders should be defined. (See U4 and A1.)

7. Dr. Stufflebeam noted that on page 21 formal agreements should be used versus contracts.

8. Referring to page 27, Dr. Hosford said that adding an applications section to the Introduction was good but to reinforce it the applications section should be listed in the table of contents. Dr. Baker said that the purpose of the evaluation should be listed with the applications section. Dr. Stufflebeam thought the applications section might be richer by adding the following questions: (1) sufficient ground to do the evaluation?, (2) the evaluation clearly identified?, (3) the evaluation appropriately staffed?, (4) can the process be improved?, (5) completely defensible results?, and (6) adequate assistance to the stakeholder to implement the findings?. Dr. Thompson recommended wording as "the standards facilitate questions such as . . ."

9. In the second paragraph, propriety should not be traded off. It needs to be emphasized. Dr. Thompson concurred. This could be done by bolding or italicizing.

10. Dr. Riffel said that on page 32, ". . . in light . . ." should be ". . . in consideration . . ."

11. Dr. Stufflebeam noted that on page 35 it starts off with summative but by the end of the paragraph, it becomes formative.

12. Dr. Thompson stated that the case needed to be supplemented with additional material. Otherwise it is not bad. The case has a narrow focus in the beginning and then it is criticized in the end. Dr. Bader felt it was hard to follow--who is the evaluator? Internal and external evaluators are used.

13. Dr. Rogers said that the italicized section on page 19 should also be added on page 43 or maybe the page after it. He was concerned that people will skip the front section and go right to the standards. Dr. Sanders was not sure where to put it. Dr. Baker thought that maybe the publisher's editor could help with where to put that type of thing.

Dr. Frechtling felt that throughout all the Standards, we need to be clear in the usage of client and stakeholder or client and other stakeholders.

U1 TO U4 TEAM OF BADER AND BAKER

U1 Check the use of stakeholder, persons, and clients--clarify usage. Eliminate reference to language barrier.

U2 Address the issue of maintaining a balance. On page 58 item #F, switch from overinvesting to achieve the evaluation.

U3 On page 65 item #B was misplaced and should be moved to U5 guidelines. Different wording for item #H: Failing to clarify clients needs. Wording needs to be clarified for item #F: Collecting information that is extraneous to the central purpose of the evaluation. Add a reference to the Nowakowski, et al Handbook.

U4 On page 71, change Title 1 to Chapter 1. The object of the investigation should be changed to program being evaluated. Common error #E should be deleted and reference to Madaus, Scriven, and Stufflebeam should be added.

U5 to F1 TEAM OF CUMMINGS AND DIAMOND

U5 Add clearly in the standard after should and before describe.

U6 On page 64, second line from the bottom "Establish contingency . . ." should be guideline item #E.

U7 Common error item #H: change wording to "Believe that every evaluation job is complete once an evaluation . . . been made." Add reference to Stufflebeam, et al, Educational Evaluation and Decision Making.

F1 The case study on page 101 led to discussion on why there was no comparative analysis and why the client was refusing to have a case study done. On page 100, last sentence of the case description should be deleted.

F2 to P2 TEAM OF FRECHTLING AND HOSFORD

F2 Add three references: Cronbach, et al, Toward A Reformation of Evaluation; House, E., The Politics of Education; Scriven, M., Hard Won Lessons. Classroomless needs to be clarified--it was decided to have it deleted. On page 108 in the analysis, second paragraph, needs to have wording added that reflects the problems that you can have with unions and evaluations.

The Joint Committee adjourned for the day at this point.

Friday, August 13, 1993

The Committee resumed review of the remaining standards.

F2 to P2 TEAM OF FRECHTLING AND HOSFORD

F3 Add "be efficient and" to the standard between should and produce information. At the end of the overview, something should be written in regards to how important it is to be prudent and efficient in expending resources for evaluation.

P1 Reword the standard to say "Evaluations should be designed to assist organizations to address and effectively serve the needs of the full range of targeted participants". Guideline B should be reworded to "Inform the stakeholders of the purposes of the evaluation (see A3, Described Purposes and Procedures)." Merge Guideline B and C together and reword to reflect the purposes of the evaluation. Rewrite Guideline F to say "Examine program effects against the assessed needs of the targeted participants or other beneficiaries". Rewrite Guideline I to state "Minimize the interruption of the instructional process while still acknowledging the purposes of the evaluation".

P2 Change title to "Formal Agreements". Change it also on inside of front and back covers, in the table of contents, in the functional table of contents, and in the Propriety standards summary. Add reference to House, Rivers, Stufflebeam article on Michigan accountability.

P3 to P6 TEAM OF JOHNSON AND WERNER

P3 Add parents and guardians to e.g., in Guideline I. Delete "whenever possible" from Guideline P.

P4 Add language differences to Guideline A and B. Concern was discussed about standard not starting with a "So that . . ." We must say what we want to say and not be concerned with having a balanced format.

P5 Change title to "Complete and Fair Assessment". Change it also on inside of front and back covers, in the table of contents, in the functional table of contents, and in the propriety standards summary. Change full to complete in standard. Take ". . . or to further or protect the evaluator's personal interests or biases" of Common Error A and make it Common Error B. Add to the description of illustrative case #2 that "The administrator described the evaluator's purpose generally saying that the use of out-of-state external . . . of their evaluations." (beginning of the second sentence) reflects reference later in analysis.

P6 Replace standard with "The formal parties to an evaluation should ensure that the full set of evaluation findings along with pertinent limitations are made accessible to the persons affected by the evaluation, and any others with expressed legal rights to receive the results." Change obligation on page 157, end of second paragraph to agreement. Change it also in cross reference following and in Guideline A. Modify Common Error H to state ". . . limitations of the evaluation that the evaluation looses credibility." On page 162, second paragraph, line four modify it to say ". . . to provide assessments yielding valid information about the faculty clinicians."

P7 to A2 TEAM OF MAYS AND PULLIN

P7 Have format in P7 match that of A5 in regards to setting off examples. Dr. Pullin will rewrite overview opening to clarify conflict of interest. Also add clarifying examples. Change Obligations to Agreements on page 166.

P8 Add Guideline F: Be frugal in expending resources for the evaluation. On page 176 at the end of the sentence of paragraph one, add ". . . to the site visit or to conduct analyses comparing the characteristics of respondents to nonrespondents or to the total sample." In the Illustrative Case--Analysis, connect paragraph one and two together.

A1 Change title to Program Documentation. Change it also on inside of front and back covers, in the table of contents, in the functional table of contents, and in the accuracy standards summary. Modify standard to say ". . . sufficiently described and documented so that the program being evaluated can be . . ." Delete opening paragraph of Overview and first sentence of the second paragraph. Move third and fourth sentences of that paragraph to the beginning of Overview. Replace Common Error #F with: Assuming that the program was uniformly implemented as intended.

A2 No changes.

A3 to A7 TEAM OF RIFFEL AND ROGERS

A3 No changes.

A4 No changes.

A5 Change title to Valid Information. Change it also on inside of front and back covers, in the table of contents, in the functional table of contents, and in the accuracy standards summary. Reword standard to state: "The information gathering procedures . . . arrived at is valid for the intended use." Discussion took place on the pros and cons of "information" and "data". On page 205 first main paragraph modify as follows: "From the preceding elements, it is incorrect to say that a specific instrument or information collection procedure is valid. Instead, it is the inference that is validated. Further, it is not appropriate to conclude . . . that the inferences drawn . . . Move Guideline J to A10.

A6 Change standards to say "The information-gathering . . ." Put definition of information and data in Glossary. Balance the use between data and information throughout the standard. On page 216 in second paragraph, rewrite to state "assess reliability analyses that focus on individuals as the limit of analyses. Similarly, . . ." Modify the end of Guideline A to read: ". . . for their intended uses; however, the generalizability of previously favorable reliability results may not be simply assumed." Move Guideline B as second sentence of Guideline A. Rewrite Common Error A for clarity.

A7 Change title to Systematic Information Control. Change it also on inside of front and back covers, in the table of contents, in the functional table of contents, and in the accuracy standards summary. Add as new paragraph before Guidelines: "The information collected and the results of the analyses of this information should be kept secure to avoid subsequent tampering. In this way, the completeness and accuracy of the information and results will be maintained." Rewrite Guidelines D and E as follows: "D. Monitor outside agencies or individuals responsible for information collecting, scoring and categorization, and/or quantitative or qualitative analyses." and "E. Maintain control of original information and results so that their integrity can be protected."

A8 to A12 TEAM OF THOMPSON AND WHEELER

A8 Change end of standard to read ". . . systematically analyzed so that evaluation questions are effectively answered." Delete "intended to address . . . the evaluation" from the first sentence in the Overview. Delete A7 cross reference in second paragraph of Overview.

A9 Change end of standard to read ". . . systematically analyzed so that evaluation questions are effectively answered." Add reference at end: Yin (1988-89?) Case Study Analysis.

A10 Move Guideline J from page 208 in with Guidelines.

A11 In standard, add "caused" between distortion and by on first line.

A12 Modify Guideline G with cross reference at end: (See P2, Formal Agreements).

Before breaking into teams to complete work on the book, Dr. Sanders noted that in the back of the book there are names of people with no location. If anyone knows where these people are, please let him know so that it can be updated. Dr. Sanders then modified the schedule for Saturday. Activities planned for Saturday afternoon will be discussed Saturday morning.

Saturday, August 14, 1993

Dr. Sanders opened the meeting by reviewing the revised schedule for Saturday.

Dr. Sanders announced that Drs. Frechtling and Rogers will be leaving the Joint Committee. They will be replaced by others from their organizations. Dr. Sanders thanked them for their help over the past three years.

The date and location of next year's meeting was then discussed. The dates were set for September 30, October 1 and 2, 1994. Several options were mentioned for location of next year's meeting (1) return to Nordic Hills, (2) retreat in the Washington, DC area, or (3) go west to Tanqe Verde or Mayan Ranch. It was agreed that the Mayan Ranch was too difficult to get to but that the next meeting should be out west. Concern was stated in regards to the cost of the staff to go west. Dr. Rogers stated that it should be left to Dr. Sanders to decide keeping costs in mind since there are limited funds.

There are 17 members of the Joint Committee; we are allowed 18 by our bylaws. The Canadian Evaluation Society (CES) has written to Dr. Sanders indicating an interest in joining. Dr. Sanders gave a brief background on that organization. He also gave a brief background on two other organizations that would fit well with the Joint Committee. They were the National Society for Performance and Instruction (NSPI) and American Society for Training and Development (ASTD). Concerns discussed were on the number of groups that we can have and the pros and cons of expanding. A motion was made and seconded to extend an invitation to the Canadian Evaluation Society (CES) to join the Joint Committee as a sponsoring organization. After discussion, the motion carried. A second motion was made and seconded to increase the number of sponsoring organizations from 18 to 21 in our bylaws to allow room for growth. The motion carried (9 to 2). A third motion was made and seconded to allow the Chair to offer the invitation to American Society for Training and Development (ASTD) should the opportunity become available. The motion carried. The bylaws can only be amended through a process described in the principles and bylaws. That process will be followed this year to make the amendment on number of sponsoring organizations.

After break, Dr. Sanders asked the members if they had any new items to add to the program evaluation standards that need to be discussed by the members. Hearing none the revisions submitted will be incorporated into the final form and sent to the members for their final approval.

Committee Reports

Dissemination

Sales of the Standards are not what we would like them to be. Those people that should be aware of the Standards are not. The Standards are not user friendly, there is a need for derivative documents. Dr. Sanders listed on the chalkboard the items that the members have done in the past for dissemination. The following are the results of the discussion on dissemination.

DISSEMINATION PLAN

JOINT COMMITTEE 1993 MEETING

1. Executive Summaries-targeted audiences & modules (NCME), Practitioner's Guide-How do you use?, Video?

Dr. Sanders

2. Kappan, EEPA, Educational Leadership, Canadian School Executive.

Dr. Frechtling

3. NCATE, Holmes: train examiners, newsletters

Dr. Pullin

4. Book Reviews

Dr. Thompson

Dr. Pullin will do the lawyers

5. Newsletters and/or Journals of Sponsoring Organizations

Getting space in the Journals for the promotional ad for the book.

ALL MEMBERS

6. Job Aids

Derivative Documents

Dr. Sanders

7. Urban School Districts, Council of Greater City Schools, Directors of Research and Evaluation Programs

Dr. Baker, Dr. Frechtling, and Mr. Mays

8. Regional Workshops, policy seminars, MIEM

Mr. Mays, Dr. Sanders, and Ms. Wheeler

9. Sessions at annual meetings, training

Dr. Cummings, Dr. Diamond, Dr. Frechtling, and Dr. Thompson

Dr. Riffel will do ECS conference

10. Work with States/Provinces, LA model, inservice departments, state accreditation agencies Mr. Mays and Dr. Rogers

11. Annual summer institutes, advertised nationally, probably weekends, given time and place, different components

Dr. Cummings (Arthur Andersen), Dr. Sanders, and Dr. Stufflebeam

12. Displays at annual meetings

ALL MEMBERS

13. OERI (Dr. Stufflebeam), labs (Dr. Bader), CEDR (Dr. Riffel), OSERS, newsletters

14. Private Consulting Organizations

Dr. Bader will get a list from OERI of contacts for their RFPs.

15. University continuing education areas

Dr. Baker

16. Unions

Dr. Bader and Dr. Riffel

17. Outreach - NCSL

Dr. Riffel

18. SAGE, press release, Education Weekly, Teacher, Chronicle of Higher Education, press conference in Washington, DC

Dr. Sanders

19. ERIC-TM, newsletter, co-market

Dr. Thompson

20. ANSI

Dr. Sanders

DERIVATIVE DOCUMENTS:

It was mentioned that ASCD does a good job in doing derivative items. Dr. Richard Manatt did an item for the Personnel Standards. Ron Brandt with Gene Carter did a video. The Joint Committee could go in partnership with them to develop these materials for this revision. Dr. Hosford stated concern about doing programs at the larger conferences and about having room for 400 people and only 20-30 people show up. There's no way to make sure that there will be a large group in attendance because of having multiple workshops all held at the same time. Concern was also mentioned that if we don't have these documents set up prior to the book being published that we may need to get the publisher's permission before proceeding. Dr. Thompson stated that training materials need to be developed and disseminated to show people how to use the Standards. With more people using PCs and laptops, may be we need to put it on disk for word search accessibility by the users. But all of this went back to concern of whether or not we needed the publisher's approval before proceeding. Dr. Thompson asked Dr. Sanders to check on this and let them know.

This brought up the subject of the contract with the publisher. Dr. Stufflebeam wanted to know if the members will see the contract prior to it being signed. Dr. Sanders stated that he was planning on working with the Executive Committee when it was finalized. Dr. Stufflebeam asked that a draft be sent to all the members for review since there are several concerns with the derivative documents and the wording in the contract.

To get people to attend to the Standards we need to address common questions such as how do you improve student achievement by using the Standards and how can the Standards strengthen curriculum and instruction. We need to use examples of how vulnerable the personnel field is without standards. That would get their attention. Dr. Bader said that promoting the Standards by showing concern for human subjects and taking the stakeholders position when developing these Standards is a good way to attract people from her organization. Dr. Sanders stated that getting their attention is the first thing to do and then show how the Standards can help them. Site-based decision making is being tried and is something that Foundations might like to study using the Standards. Any derivative document is great as long as it doesn't erode the book itself. Use the derivative document to instill interest in the book itself. One suggestion made was to copy the Standards from the inside of the covers and use them as a derivative document. But that raised the question of whether they would undermine the sale of the Standards themselves. Maybe a marketing agency would be able to assist in this. Dr. Rogers agreed with that but questioned how do you get a copy of the book without the expense of the cost of the book? Maybe the publisher needs to be involved in the developing of the derivative document. A specialist is needed to work with this. Discussion continued on additional ideas for derivative documents, how to produce them, who produces them (Joint Committee or a publisher?), and whether the publisher should or should not be involved in the development of the derivative items.

Would we be better off to market the book ourselves and then work with the organizations to sell it? The Joint Committee will be receiving a copy of the marketing plan that Dr. Wigdor will be sending that might help us. Last year ASCD set up an ACE program to develop a users manual and an administrators guide that members can use. Dr. Thompson recommended that the Joint Committee print the book itself (it would cost approximately $5 each to produce) and then the sales would go directly to the Joint Committee. After we sell approximately 1,000 copies, the cost of the Standards would level out and the Joint Committee would make money. He felt it would almost be double the cost to print the books commercially. Especially if we can get the organizations to do free advertising. You need to spend money up front but over the years we could make money. Dr. Cummings stated that if we printed the book with a spiral comb binding, we could cut the costs of printing. If the point is to get the Standards into the hands of the people, maybe it could be something that is cheaper to print and not as fancy. Do we want it to be comparable to the Personnel Standards or not? If the organizations are really sponsoring organizations, then why not have them become partners with the production or marketing or both? It was mentioned that the Joint Committee could serve as an outlet from the publisher for copies of the book. The members then took a break.

Dr. Sanders announced that Dr. Cummings has been asked to chair a committee to look at The Personnel Evaluation Standards and determine if they need to be revised. Drs. Bader, Baker and Rogers will work with Dr. Cummings on this topic throughout the next year. Please let Dr. Cummings know if you have any information that would help them also if you want to be on the Committee.

Next discussed was the Nominations Committee report. Dr. Pullin stated that the nominees would be willing to serve on the Executive Committee if elected. She moved that the slate of Drs. Bruce Thompson and Gene Werner be elected to fill the vacancies on the Executive Committee that will open at the end of this meeting. Dr. Stufflebeam seconded. The motion carried unanimously. The new Executive Board members will be Dr. Pullin (Vice Chair), Mr. Mays, Dr. Thompson, Dr. Werner, and Dr. Sanders (Chair). Dr. Sanders thanked Dr. Pullin and Mr. Mays for their work.

Dr. Sanders asked the members to keep the Joint Committee work in front of our constituents. Please think of how the Standards can be used with your organization.

Dr. Sanders mentioned that we do have a proposal for development of standards for evaluations of students but that it has not been funded. If you know of a funding source, please let him know. Dr. Pullin suggested recontacting those sources since there are now new views on

education.

Validation Panel Report

Dr. Gould reported on points that were raised prior to the August meeting or during the meeting:

(1) Add a summary in the preface of what is in this book that is not in the first edition; it may make the difference of whether someone buys the book.

(2) Specify what the Validation Panel is, does, in the front of the book. The breadth of the representation of the panel would be helpful to others.

(3) When talking about expanding the scope, most of that expansion used the words training and education. So on the cover change the wording to educational/training programs.

(4) A priority of the Standards was to get people to recognize the amount of detail in each

Standard. State that there is a difference in using a standard in all of its detail from when you are looking at the whole set. Looking through and finding those standards that apply to them might be quite different than what they used them for in the beginning.

(5) Pages 16 and 43 were taken as a set versus taken as a whole.

(6) ANSI - (a) Add a place describing the ANSI approval process. Dr. Walser has some materials on this and will help whenever possible. (b) Since this process is very time consuming, they suggested taking the draft that the Committee approves and send it to ANSI as soon as possible. This might enable us to state it has ANSI approval in the published book. Dr. Thompson suggested putting that approval in a box on the same page as the copyright.

(7) The hearings results were incorporated into this new draft. The hearings report was different from the field test report. The Panel can't see how the field test results were addressed. Dr. Sanders responded that the results of the field tests were used to make the February 1993 draft. The February 1993 draft was used for the hearings. The input from the hearings resulted in the August 1993 draft. The difference between the August 1992 draft and the February 1993 draft was the result of responding to field test results and revisions recommended by the Joint Committee at the 1992 annual meeting. Dr. Sanders will go back and document changes made as a result of the field tests.

(8) As soon as the manuscript is completed, a copy should be sent to ANSI and a copy to the Validation Panel so they can do their report. An editor should be hired to make the language as understandable as possible to the broadest range of people as possible, making it consistent throughout, and as interesting as possible.

(9) May want to have a sentence or two stating what is the acceptable definition of the word "standard" included in the Glossary.

Dr. Gould then made general comments. The Validation Panel is to report publicly on the evaluation of the Standards. Dr. Gould is doing an outline of the Validation Panel's report for the Committee's review, will send it to the Committee members for their comments, and finalize the report by the end of December 1993. That will be sent to Dr. Sanders to identify possible errors, etc., and to get it back to the Validation Panel by the end of February, 1994. Once finalized, the report will be sent to Dr. Sanders. They are also expecting to send a smaller report to the sponsoring organizations for announcement purposes. The Panel's report will be summarized for publication. The Validation Panel does not want to publish it in an obscure journal. After publication, if we make ANSI certification, an announcement can be placed in the ANSI Reporter. Individual members can use the report as they wish. The Panel will use the appropriate journal to print the Validation Panel's report once it is ready for publication. The Panel recommended developing procedures on how to deal with the feedback from this review. Dr. Sanders thanked Dr. Gould for his leadership on the Validation Panel.

Dr. Sanders then announced that Mr. Mays is chairing a Task Force, with Drs. Pullin and Stufflebeam as members, to look at the future of the Joint Committee. Mr. Mays reported that Dr. Sanders ends his chairmanship of the Joint Committee in two years. If Dr. Sanders accepts being a chair for another three years, then there should be a Vice Chair for those last three years who will become the next chair (to have continuity). Status of matters like this for the future of the Joint Committee will be discussed by the Task Force. Mr. Mays plans on bringing the Task Force's recommendation to next year's meeting.

A question was raised about how much it costs to operate the Joint committee. If we are involved in developing new standards, it would be necessary to have $150,000 per year versus $50,000 per year if doing a revision. It could be as little as $8-10,000 if there was no development activity. One idea was to ask sponsoring organizations to provide support. Dr. Sanders stated that $1,000 per year would be no problem for some organizations but would be for smaller organizations that can't afford it. Dr. Thompson stated that even $500 may cause a sponsoring organization to pause in becoming or staying a member of the Joint Committee. Dr. Cummings stated that since the Joint Committee is a non-profit organization, people could write donations off their income taxes. Several other options and concerns were raised. Dr. Sanders noted that these are ideas for the Task Force and that they should be creative in their thinking and recommendations.

With no other items to discuss, Dr. Sanders adjourned the meeting.