Twenty-Eighth Annual Meeting of
The Joint Committee on Standards for Educational Evaluation
May 9-11, 2002
Raddisson Hotel
Kalamazoo, MI
Attendees:
Joint Committee Members:
Edith Beatty (ASCD)
Flora Caruthers (NLPES)
Jim Cullen (CES)
Glen Cutlip (NEA)
Paula Egelson (CREATE)
Arlen Gullickson, Chair (At Large)
Patricia McDivitt (American Counseling Association)
Dianna Newman (AEA)
Gary Wegenke (AASA)
Bob Wilson (CSSE)
Mark Wilson (AERA)
Donald Yarbrough (NCME)
Validation Panel:
Dr. Robert Linn, Chair
Kathleen Boundy
Placido Garcia
Doug Harris
Audrey Kleinsasser
Victor Willson
Joint Committee Staff:
Dale Farland, Acting Secretary
Thursday, May 9, 2002, 9 AM
Dr. Gullickson called the meeting to order.
Dr. Gullickson introduced the Validation Panel (all were present).
Dr. Linn thanked the panel members for their work and described their efforts in very complimentary language.
Dr. Gullickson asked Joint Committee members to introduce themselves.
Dr. Gullickson
§ Thanked the Joint Committee members for their work
§ Thanked the Validation Panel members for their work
§ Encouraged the Joint Committee and the Validation Panel to carry on the message of the Student Standards
Dr. Gullickson distributed the agenda and asked if there were changes or additions.
Yarbrough/M. Wilson moved/seconded to approve the agenda—passed
Dr. Newman reported that students at Nova University purchased over 700 Program Standards.
Dr. Gullickson mentioned that without grant support, much of the Joint Committee work has been contributed by The Evaluation Center.
Dr. Gullickson distributed the financial report and reported on it.
Beatty/Yarbrough moved/seconded to approve financial report—passed
Individual Joint Committee members reported on their organization’s activities related to the work of the Joint Committee.
Yarbrough/McDivitt moved/seconded that the 2003 meeting dates be September 25-27, 2003—passed
Dr. Linn distributed and discussed the Validation Panel report.
§ The product has much potential utility.
§ The book should be supplemented to insure the work is likely to be used.
§ The fifth conclusion stresses the need for a clear link to test standards for high stakes users.
§ Additional distribution/marketing options should be considered.
In response to a question about the report, Dr. Linn stated that diversity has been a consideration of the Joint Committee from the beginning (considering a broad view of diversity). The data from the field tests were not very revealing regarding diversity of sites. The Joint Committee could encourage sponsoring organizations to appoint Joint Committee members who are more representative of diversity. The Joint Committee should formulate a specific plan for progress.
Dr. M. Wilson asked Dr. Gullickson about options of CD and Web-based publication of The Student Evaluation Standards.
Dr. Harris was invited by Dr. Gullickson to comment regarding other options.
One option is the recommendation in the Validation Panel report to provide scoring rubrics.
For electronic publication, he recommended Web-based, database-driven access to the standards. This should be searchable in flexible ways to answer specific questions. Tailor professional development to specific questions (just-in-time, on-line approach to providing training materials). Use a Web approach to collecting data regarding questions and use. (He reported that it costs about $3 per entry to do meta-tagging for libraries). Dr. Harris thinks that the financial return to the Joint Committee will not be from sale of CDs, etc., but rather from training and support.
Mr. Cullen asked the Validation Panel about the “voice” of the standards—whom they spoke to. The Validation Panel (Garcia) thought they were very understandable to a nontechnical reader.
Dr. M. Wilson stated that he agreed with the recommendation that providing “context” materials for the standards is important.
Ms. Boundy suggested that exclusion by silence (not mentioning specific diversity issues) could be easily improved in a final edit.
Dr. Gullickson asked Dr. Linn to speak on the recommendations from the Validation Panel to modify the operating procedures. Linn suggested that it is especially important to consider revision of section 5.4.
Dr. Willson suggested that the Joint Committee could have presented the characterization of the teachers and students represented in the field trials.
Ms. Boundy recommended citing a 2001 Office of Civil Rights document on high stakes testing in The Student Evaluation Standards.
Dr. Willson suggested adding a note in the Student Standards book that updated references would be posted on the Joint Committee Web site.
Dr. Gullickson began his chair’s report by commenting on the influence of the individuals on the Validation Panel. In addition, he made the following points:
§ He thought we were at the end of the road in development of The Student Evaluation Standards
§ The Joint Committee should give more consideration the mission statement. Items that have not been central enough to actions of the Joint Committee are training, dissemination, and revision.
§ Selected agenda items were reviewed.
§ He stressed the importance of having the book ready by the November 2002 AEA meeting in Washington, DC.
§ The ANSI audit is scheduled for late 2003. It may be possible to secure financial support of the Joint Committee from industries involved in standards.
§ The Joint Committee needs to be involved in training and training certification.
§ A draft facilitators guide is available for review by the Joint Committee.
§ There are concerns about the time involved in developing standards; a different strategy is needed. ANSI considers a 2-year development cycle to be a long time.
§ The Joint Committee has not done a good job of studying the impact of standards.
§ Sponsoring organization involvement needs to be encouraged, and ways to improve involvement should be considered.
§ Elections (past) and current (executive committee, chair, and vice chair)
§ The host organization contract ends; a new contract needed to be considered.
§ The intention was to complete work on The Student Evaluation Standards that day.
Dr. Yarbrough raised questions about the approval process for the standards. His concern is to facilitate approval of the standards, but allow executive judgment in deciding on the details.
A question was raised about the meaning of “standards” from the perspective of ANSI and of other stakeholders. Dr. Gullickson also presented the distinction between technical standards and advisory standards as seen by ANSI.
The minutes of the 2001 Joint Committee meeting were presented. A correction (to add Gary Wegenke, AASA) was pointed out.
Yarbrough/Newman moved/seconded to approve the minutes as corrected—passed
Mr. Farland presented a summary of the standards adoption process through December 2002.
Dr. Newman reported about the national hearing at the American Evaluation Association meeting.
Dr. Gullickson presented the national hearing report from the CCSSO state assessment officer meeting.
Mr. Cullen asked about the Joint Committee structure; he commented about the time it takes to develop standards.
Dr. M. Wilson reported on the national hearing panel presentation at NCME/AERA. The audience was positive and wanted the standards to get into users’ hands with supporting materials.
Dr. Yarbrough would have expected any technical comments to come from this group, but there were none.
Dr. Newman was at both the AEA and NCME hearings and was pleased that people who five years ago were very critical, are now very supportive.
Dr. M. Wilson asked that the presentation materials be included.
M. Wilson/McDivitt moved/seconded approval of the hearing reports with edit of and inclusion of the presentation materials—passed
Mr. Farland summarized the ANSI public Review process including procedures for responding to comments. No comments resulted from the public review period. There were 190 total hits on the public review comment Web page.
Dr. Gullickson presented a table summarizing the Joint Committee response to specific comments from the National hearings.
Dr. Gullickson raised questions for the Joint Committee.
1. Should names be included in posting the comment summary to the Web? Consensus is to not include the names.
2. The Joint Committee was asked to read and suggest changes to the stated responses in his summary table.
Dr. Gullickson introduced the ballot procedures for the Joint Committee vote on the standards.
Mr. Farland asked about the ANSI requirement for identifying of an interest area for each member of the consensus group. Dr. Gullickson will ask ANSI if designation of organizations is sufficient.
Yarbrough/B. Wilson moved/seconded that the committee vote on the 28 standards individually and as a whole and on the supporting materials. The vote will be yes, no, or abstain (with comments required for a no vote and optional for others). The vote will close on May 15, 2002. Individuals will vote by filling out and submitting to the Joint Committee chair a signed paper ballot. Passed
Ballots were distributed to those members present at the meeting. Those not present will have ballots mailed to them for return to the committee chair.
Following a discussion of constituting a committee to respond to comments during the consensus period, Newman/M. Wilson moved/seconded that Don Yarbrough, Bob Wilson, Edith Beatty, Jerry Horn, Arlen Gullickson, and Todd Rogers be designated the consensus response committee. Passed
Signed ballots on the standards were collected from members present.
Copies of the preface, acknowledgments, and applying the standards were distributed for review by the Joint Committee. The invitation to users was presented via the data projector. Members then voted on each supporting document.
The draft-training guide developed under contract for Corwin Press was distributed. It will be discussed on Friday.
Yarbrough/Newman moved/seconded to approve the appendix to the standards. The chair of the Joint Committee, with help of the consensus committee, will assure that it is accurate—passed
The Joint Committee reconvened for Day 2: Friday, May 10, 2002, 8:30am
Dr. Gullickson asked the group to introduce themselves. Two additional members arrived: Flora Caruthers and Paula Egelson.
Daniel Stufflebeam, Jim Sanders, Nanette Keiser, and Todd Rogers were present as invited guests of the Committee (Todd Rogers arrived Thursday afternoon).
Discussion was held on the supporting materials, especially “applying the standards.” Discussion focused primarily on the tone of the chapter and its location in the book.
Yarbrough/M. Wilson moved/seconded to authorize the chair, in consultation with the consensus committee, to decide about placement of the “applying the standards” chapter—passed
Dr. Gullickson reviewed critical issues for consideration by the Joint Committee and appointed working groups to consider the respective issues:
1. Organization of the Joint Committee (B. Wilson and McDivitt)
2. Revision or confirmation of the program standards and revision of the personnel standards (Caruthers, Beatty, Sanders, Keiser)
3. Dissemination of standards (Cutlip, Egelson, Newman, Cullen, Wegenke) and training
4. The research mission of the Joint Committee (Yarbrough, M. Wilson)
Working groups on each of the four tasks worked the rest of Friday and reported to the Joint Committee on Saturday.
Yarbrough/Beatty moved/seconded that the Joint Committee continue to contract with The Evaluation Center at Western Michigan University to serve as the host organization for the Joint Committee—passed
Yarbrough/Newman moved/seconded that the Joint Committee affirm Arlen Gullickson as chair of the Joint Committee for the next term as specified in the operating procedures—passed
The Joint Committee moved into working groups and spent most of Friday working on the four tasks.
Upon reconvening at 4 PM Friday, the subgroup (Yarbrough and M. Wilson) working on the research agenda reported to the Joint Committee. They intend to further develop their research group agenda over the next several months.
The subgroup (B. Wilson and McDivitt) working on the organization of the Joint Committee reported to the Joint Committee (a printed draft was provided). They suggested that the Joint Committee bylaws do not provide for subcommittees. They distinguished between a task force (group with a specific assignment where the group exists until the task is completed) and a subcommittee that may be a continuing body of the organization.
The third report was from the subgroup (Caruthers, Beatty, Sanders, Keiser) considering review and revision of the program and personnel standards. They proposed tasks and a timeline for review and revision of the program and personnel standards.
The dissemination and training subgroup (Cutlip, Egelson, Newman, Cullen, Wegenke) reported on their deliberation. Though couched in terms of The Student Standards, their report applies to all three standards. They proposed a number of activities for dissemination and training.
Day 3: Saturday morning, May 11, 2002, 9:00am
Dr. Gullickson opened the meeting with an overview of the prior sessions and of the business remaining.
Elections for Executive Committee
Yarbrough/Beatty nominated Todd Rogers for a 3-year at-large membership on the Joint Committee—passed
Yarbrough/Newman nominated Todd Rogers as vicechair of the Joint Committee—passed by acclimation
Yarbrough/B. Wilson nominated Patty McDivitt for a 3-year term as a member of the executive committee of the Joint Committee—passed
Cullen nominated Don Yarbrough to a 1-year term on the executive committee—passed
Newman nominated Edith Beatty to a 2-year term on the executive committee—passed by acclimation
There was an extensive discussion of Joint Committee meeting attendance, Joint Committee sponsoring organization membership, and strategies to encourage both.
Yarbrough/Newman moved/seconded that NAESP and NASSP be designated as cooperating members if they do not send a representative to the next meeting, and the chair is instructed to make all diligent and appropriate efforts to encourage the organizations to appoint effective representatives.
Beatty/Newman moved/seconded to table the previous motion with the chair making all due effort to solicit input from the two organizations—passed
There was discussion of actively seeking increased diversity on the Joint Committee and in its work as suggested by the Validation Panel report. Efforts to enhance diversity should consider race, ethnicity, disabilities, and language and should extend to representation on task forces and writing panels and other functions of the Joint Committee.
Members are encouraged to suggest contacts in organizations that can help with the effort.
Yarbrough/McDivitt moved/seconded that the chair appoint a task force with at least one Joint Committee member to explore all options for better incorporating diversity perspectives in the collaboration and products of the Joint Committee. The task force will report at the next annual meeting. That task force will explore at least the following as possible actions:
1. Task force membership; chairs of task forces
2. Panel of writers
3. Contacting special interest groups and minority group member organizations
4. Make special effort to recruit membership from
a. Council of Exceptional Children
b. Association for Rehabilitation Counseling
c. Holmes Scholars
d. Holmes Partnership
5. An advisory committee for diversity
6. Cooperating memberships
Passed
Dr. Newman was appointed chair of the task force, and Dr. Egelson will represent the Joint Committee on the task force.
Discussion ensued of the recommendations from the Validation Panel regarding enhancing the standards.
Beatty/McDivitt moved/seconded that the chair of the Joint Committee appoint a task force to address enhancing the value and use of The Student Evaluation Standards through formatting, targeting users, marketing approaches, etc., appreciating the recommendations of the validation panel. The task force will report at the September 2003 annual meeting—passed
The next discussion dealt with the desirability that supplemental materials on The Student Evaluation Standards include emphasis on school improvement and current educational issues as recommended in the Validation Panel report.
Dr. Cutlip recommended that the standards and supplemental materials include a discussion emphasizing that the purpose of evaluation and testing is to improve student learning.
Ms. McDivitt recommended that the Joint Committee encourage organizations such as CCSSO and the Educational Learning Council Conference to require that The Student Evaluation Standards be included in their request for proposal processes.
Regarding the Validation Panel recommended procedures for evaluation, interpretation, and revisions of the standards, the consensus of the Joint Committee was that revision of those procedures was under way through efforts of that subcommittee.
Yarbrough/B. Wilson moved/seconded that the executive committee revise the operating procedures in response to the Validation Panel recommendation (and propose changes to address this recommendation). The executive committee will report to the Joint Committee either at or prior to the next annual meeting—passed
The title for the student evaluation standards book was discussed.
Beatty/Caruthers moved/seconded that the title of the book be “The Student Evaluation Standards”—passed
Dr. Gullickson announced that it was his understanding that the consensus committee will determine the subtitle.
One comment was received on one ballot on the standards. After discussing the comment, consensus was that the standards statements had been approved and would not be changed at this point.
Discussion of the proposed training guide from Corwin Press was begun.
McDivitt/Newman moved/seconded that the Joint Committee not accept the training guide from Corwin Press and that the dissemination task force be charged with pursuing additional training materials—passed unanimously
Yarbrough/Beatty moved/seconded to reaffirm the Program Standards for a period of three years—passed
McDivitt/Yarbrough moved/seconded that when the Joint Committee begins a revision of a set of standards or begins work on a new set of standards, it will instruct a task force to do the development work based on a precise, detailed task description provided by the Joint Committee. That detailed description will include but will not be limited to:
1. A description of the task
2. The qualities and qualifications of the writers
3. Assigning tasks to writers based on a matching process
4. Review of work completed
5. Creation of a unified document
6. Submission of a draft document to the Joint Committee for review at an annual meeting.
When a task force is created, it will include a member of The Evaluation Center to serve as a liaison between the task force and the Joint Committee—passed
The meeting adjourned at about 1 PM Saturday.
The Joint Committee continued as a committee of the whole.
Discussion continued on the structure and functions of the Joint Committee.
Dr. Newman offered to send the report of the dissemination subcommittee to the chair of the Joint Committee in digital form.
Dr. Yarbrough recommended that the dissemination subcommittee move quickly to develop a plan for developing training and certifying trainers with remuneration returning to the Joint Committee.
Dr. B. Wilson encouraged the Joint Committee to continue the direction begun by the subcommittee on dissemination.
Dr. Beatty suggested that the committee consider cooperating with various training organizations, such as NSDC, ASCD, or ASTD for developing support materials.
Dr. Cutlip cautioned that cooperation might require revenue sharing.
Dr. Caruthers and Dr. Beatty expressed concern about the short time frame in which training materials are required.
Dr. Beatty suggested a multiple wave approach where the first ripple would be immediate, ad hoc training and the second wave might require grants to support development.
Dr. Rogers suggested having available a list of awareness persons to respond to initial requests.
Dr. Gullickson suggested that the persons who did field tests and national hearings may be a good resource for awareness.
Dr. Rogers and Dr. Newman encouraged development of a quick training outline for the first awareness sessions.
Dr. Yarbrough has a student developing a proposal on training relative to the Student Evaluation Standards.
Dr. Newman and Dr. Yarbrough stated that the Joint Committee needs a plan for dissemination and documentation about success of the plan.
Dr. Newman suggested that if an advertisement goes out announcing The Student Evaluation Standards to school administrators, it should also refer to the Program Standards.
Dr. Newman suggested that book review vehicles of organizations would be a good way to promote the book.
Dr. Rogers suggested that the Student Evaluation Standards should be promoted to teacher educators.
Dr. Gullickson said that the Department of Education should be a prime vehicle for promoting the Student Evaluation Standards.
Dr. Gullickson suggested that The Evaluation Center conduct training but that other organizations need to be involved. Dr. Newman volunteered.
Dr. Yarbrough suggested that the Joint Committee establish a policy about Committee members presenting about The Student Evaluation Standards for an established fee, with a portion returned to the Joint Committee. Dr. Gullickson wants to distinguish a personal request to a Joint Committee member for speaking versus a request to the Joint Committee.
A further suggestion was that requests to the Joint Committee for speakers be processed through The Evaluation Center, with billing based on school size or some other basis with some percentage going to the Joint Committee.
Dr. Yarbrough suggested establishing a development fund to support addressing diversity in speaking engagements. Dr. Gullickson will develop a proposal in consultation with Dr. Wegenke.
The Joint Committee turned to a follow-up discussion of the report from the subcommittee on revision of standards.
The seven studies recommended for the program and personnel standards need to be completed by the 2003 meeting. These will serve as a basis for decisions regarding next steps for each respective standard.
Ms. Caruthers agreed to serve on a task force to direct studies regarding the program standards.
Dr. B. Wilson reminded the committee to consider the differences between minor and major revisions of standards.
The executive committee will consider the Joint Committee operating procedures and their relationship to ANSI requirements.
Dr. Gullickson will ask Dr. Egelson to work on the personnel standards.
Dr. Yarbrough will talk with Dr. Nagliari about organizational psychology representation from APA on the personnel standards.
Dr. Jane Davidson was asked to serve on a task force to consider the personnel standards (she is also a member of the Society of Industrial and Organizational Psychology).
The subcommittee also recommended that Dr. Nanette Keiser be asked to serve on the task force considering the program standards.
Discussion turned to consideration of the report of the research subcommittee.
Dr. Yarbrough reiterated that he and Dr. M. Wilson hope to submit a report to the Joint Committee by the end of summer.
Mr. Cullen reported that the new editor (Brad Cousins) of the Canadian Journal of Evaluation will look for papers on the effects of evaluation.
Dr. Yarbrough is working with students to submit an article to the Journal of Teacher Education (based upon field trial data). He also will submit an article to NCME on the history of the Joint Committee.
Dr. Gullickson asked about including an annotated bibliography in the student standards book. The consensus was not to include it in the book.
Dr. Gullickson offered to build a system to store the bibliography and solicit more entries from users.
The book will reference the Web site.
Mr. Cullen asked that the new task force structure be summarized as soon as possible and sent to the Joint Committee.
The Joint Committee concluded its deliberations for the 2002 annual meeting.
Dale S. Farland