Water on the Web:

 

A Formative Evaluation of the Website ^*

 

 

 

 

Holly Miller

Evaluation Intern

For

The Evaluation Center,

Kalamazoo, MI

September 1999

 

Background

 

Project overview

Water on the Web (WOW) is an interactive website that combines environmental monitoring technology with user-friendly science and math curriculum modules to provide inquiry-based learning to high school and community college students.  An increased need to use technology to manage environmental needs has placed a greater demand for more technologically competent professionals.  The purpose of WOW is therefore to aid in the education and training of such students.  The WOW website uses four remote sensing instruments (RUSS - Remote Underwater Sampling System) to gather lake data that is recorded and available on the website.  Students can use the data/technology for designing experiments, programming RUSS to collect certain data, and interpreting results using the different curriculum modules.  Curriculum modules found on the site are designed to be used independently or integrated into existing curriculum. 

 

 

Workshop description

Over the course of the summer, WOW staff had the opportunity to present the website to a captive audience of science & math teachers, researchers and scientists at the 1999 Operation Pathfinder Summer Course.  The course participants were introduced to the website and given time to review the website on their own.   They were asked to review one of the lessons on the site and provide feedback on both the lesson and the website in general (see Appendix A for a copy of the instrument in PDF format).    The following information was provided from the workshop held in Duluth MN held from June 17 - July 1, 1999.

 

 

Participant Demographics

Twenty-three workshop participants returned a partially completed or completed survey.  Of the twenty-three respondents, 65% were high school teachers while the other 35% indicated other professions (middle school teachers, outreach educators, K-3 special education, and students).

 

The range of courses taught by the high school teachers was diverse, ranging from General Science to Math to Language Arts.  Almost all of the teachers, 93%, taught at least one science or math class.  Those respondents who were high school teachers taught grade levels that ranges from 7^th grade to 12^th grade, with a few teachers also indicating lower grades.  Table 1 displays the courses taught and the percentage of high school teachers teaching each course.  Table 2 summarizes the grades levels taught by the same group of teachers.

 

 

 

 

 

 

Table 1.

 

 

           

                                    Table 2.

 

 

 

When asked what classes these teachers would use Water on the Web with, the following courses were listed:

·        Earth Science (n=6)

·        General Science (n=3)

·        Math (n=3)

·        Biology (n=2)

·        Life Science (n=2)

·        Chemistry (n=2)

·        History (n=1)

·        English (n=1)

·        Environmental  (n=1)

·        Oceanography (n=1)

·        Physical Science (n=1)

 

Primary Findings

 

Lessons reviewed

The workshop participants selected one of the following lessons, or part of a lesson to review:

 

1.      Lesson 3:  Diel Temperature

2.      Lesson 4:  Heat Budget of Lakes

3.      Lesson 9:  Thermal Stratification

4.      Lesson 10:  To Stock or Not to Stock

 

 

Results of Survey

The course participants were asked to review the WOW website and one of the lessons located on the website.  They were then asked to fill out a feedback form.  The survey was divided into three main components: Lessons, Website and Demographics.  The following information was collected from twenty-three course participants.  (See Appendix A for a copy of the survey instrument.)  Data was reviewed for all participants, by participants that were high school teachers, and was also analyzed by individual lessons. 

 

 

Feedback on the Lessons: By all participants

Goals and Outcomes.  The majority of those who responded indicated that the goals for the lessons were realistic for the education level of their students (61%), and these goals also met curriculum requirements for 94% of those who responded.  71% of course participants who responded agreed the outcomes listed for each lesson were outcomes they would typically cover in their classrooms.  Four of the twenty-one respondents suggested simplifying or narrowing the number of outcomes for the lessons.

 

Lesson Prerequisites and Resources.  Feedback on the lesson procedures and workbook was not as positive as the lesson goals and outcomes.  Although the lessons were all realistic, educationally appropriate and included typical outcomes for science and math curriculum, 96% of the participants stated their students would need to do further preparations before receiving the lesson.  The majority of respondents (83%) have the facilities and resources needed to conduct the lessons.  One respondent did not have computer access while another would need DRG maps.  When asked what ratio of computer to students they thought would be appropriate for the lessons, 56% indicated 1:2, 13% indicated 1:1 while another 13% indicated 1:3.  Nineteen of the twenty-three course participants indicated they had the resources needed to use the lessons.

 

Lesson Procedure and Student Workbook.  The majority (70%) did not think the instructions for the lessons were straightforward.    Only 39% of respondents agreed the activities developed for the lessons were reasonable to accomplish in the identified time.  Additionally, only 21% agreed that the student directions and/or student worksheet provided the necessary resources and guidance to complete the lessons.  48% of course participants indicated they would use the student worksheet provided with the lesson.  When asked how the activities could be changed to better fit the curriculum, course participants responded:

·        Improve the lesson procedure to better suit the students (n=2)

·        Simplify the expectations to fit the appropriate grade and age of students (n=2)

·        Better instructions and information on graphing (n=2)

·        Fewer variable to compare (n=1)

·        Provide more directions on how to move/graph data in excel (n=1)

·        More up-front information (n=1)

·        Provide better instructions when using other parts of the website with lesson (n=1)

·        Provide a lesson on technology, or provide more technology information (n=1)

·        Select certain days where there will be things to see in the data (n=1)

·        May only do parts of lessons (n=1)

 

When asked what additional resources or information they would need to complete the lessons, the respondents indicated:

·        More step-by-step instructions (n=4)

·        More introductions and background information (n=3)

·        Provide formulas (volume, heat capacity, etc.) (n=2)

·        More technical practice (n=2)

·        Information on what you will need for the lesson (n=1)

·        More explanation on the logistics on how to move  data to graphs with multiple dates (n=1)

·        Better links between site sections (n=1)

·        DRG problems (n=1)

 

When asked what types of changes might improve the lesson to better fit their curriculum, the responses were:

·        Better instructions (n=3)

·        Lower the level of the lesson (n=2)

·        Graph different comparisons (n=1)

·        Provide more visual images (n=1)

·        Only use the lesson (or site) for science basics (n=1)

·        Create a lesson sequence (n=1)

·        Provide background or a lesson on technology (n=1)

 

 

Feedback on Site: By all Participants

WOW Website and Lessons.  Participants were asked which of the lessons on the site they would use in their classrooms.   Only five participants had time to review other lessons and three indicated specific lessons they would use.  The lessons indicated were:

1.      Storm surges - land use and lake turbidity

2.      Effects of pH on aquatic organisms (n=2)

3.      Part/sections of each lesson. (n=1)

 

 

The participants were asked whether or not they anticipated using either the lessons and/or the lake.  Of the seventeen responses, 61% said they would use some part of WOW in their classroom. 

 

The course participants were asked what parts of the website they would use to enhance their own classroom curriculum, their responses were:

·        Physical Temperature Characteristics (n=3)

·        Data/Russ information (n=3)

·        Graphics/Visuals (n=3)

·        Everything (n=2)

·        Samples (n=1)

·        Glossary (n=1)

 

The course participants were also asked what they like best about the site, the responses were as follows:

·        Data (n=4)

·        Russ component/Profiler (n=4)

·        Everything (n=2)

·        Graphs (n=2)

·        Home Page (n=1)

 

When asked what suggestion they have to improve the site, their responses were as follows:

·        Step-by-step directions/instructions (n=6)

·        Better between site links (n=3)

·        List topics/lessons by grade-level (n=2)

·        More background on technology (n=2)

·        Easier accessibility to data (n=2)

·        Correct data on site (n=2)

·        More user-friendly site (n=1)

·        More information on background of each lesson (n=1)

·        Better color (n=1)

 

 

 

Lesson and Site Feedback: By High School Teachers Only

The analysis for high school teachers looks quite similar to that of the entire group.  Only responses that had results with a difference greater than 5% are discussed below. 

 

Resources.  The high school teachers had fewer resources available to complete the lessons, with only 73% of respondents indicating they had the necessary facilities and resources.  In addition, none of the teachers indicated they would use a 1:1 ratio of computers to students.  Of those who responded 93% selected a 1:2 ratio and 7% (n=1) indicated a 1:3 ratio. 

 

Lesson Procedure and Student Workbook.  When asked if they believed the student directions and/or worksheet(s) provide the necessary resources and guidance needed to complete the lesson, fewer teachers indicated yes, with only 13% versus the 22% response by all participants.  However, a higher percentage of teachers indicated they would use the student worksheet included with the lesson (60% of teachers and only 48% of all respondents).

 

Lessons.  And finally, a higher number of teachers anticipate using either the lessons or the lake data with their classroom lessons, with 67% indicating 'yes'.   (Only 61% of all respondents anticipated using the site for lessons or lake data.)

 

 

Feedback on Individual Lessons

Goals and Outcomes.  Frequencies and descriptive statistics were ran for each of the four individual lessons and compared to one another and the group's responses.   (See Table 3. for a summary of the results.)  In terms of realistic goals for the educational level of their students, the percent of responses that agreed ranged from 50% to 75% with "To Stock or Not to Stock" receiving the highest percent of agreement (75%). 

 

Looking at whether each lesson meets curriculum requirements, each lesson received at least 50% agreement, with all responses ranging from 50% to 100%.    All participants reviewing the Diel Temperature lesson agreed the lesson met curriculum requirements.  The range of responses for outcomes typically covered in their own classrooms was larger, from 33% to 100%.  Again, the Diel Temperature lesson had 100% of respondents agreeing that all of the listed outcomes would typically be covered in their classroom.  The lesson on Thermal Stratification received the lowest rating. 

 

Lesson Prerequisites and Resources.  When asked if the students would need further preparation prior to receiving the lesson, the statistics were similar for all lessons.  Additionally, the respondents indicated the instructions were not straightforward. Thermal Stratification had the highest percentage agreeing the instructions were straightforward (67%).    When asked whether or not the activities were reasonable to accomplish in the time stated, responses were somewhat varied ranging from 0% to 50%.  No one reviewing the Diel Temperature lesson believed the activities explained were reasonable to finish within the allocated time.

 

Lesson Procedures and Student Workbook.  Looking at the student directions and worksheets, again, the responses were relatively similar.  21% of the whole group agreed that the student directions and/or worksheet provided the necessary resources and guidance to complete the lessons.  Individual lesson responses ranged from 0% to 50%, with the lesson on Heat Budget of Lakes not receiving any positive response.  In terms of teachers using the student worksheets in their own classroom, the overall score was 48% while individual lessons ranged from 0% to 71%.  The lesson on Thermal Stratification did not receive any endorsement of their worksheet.

 

Overall, the responses to the lessons mirrored that of the whole group.  However, looking at each lesson individually will show the strengths and weaknesses of each.

 

 

Strengths

Because the website is primarily directed to high school and community college teachers and with those responding to the survey primarily high school teachers, most of the strengths will be drawn from this population.

 

Although still in its piloting stage, the results of the survey demonstrate the need for this website, and others like it. While its primary audience may seem rather specific (high school and community college teachers and professors), the opportunity and needs of each teacher is diverse.  The website allows for teachers in many different areas of science and math to use the site for their curriculum.  The survey indicates that the types of classes taught by these teachers are broad, as is the grade levels and therefore substantiating the utility of the site for many different academic settings.

 

Survey results also indicate the materials found on WOW are appropriate for the intended audiences.  Course participants, and specifically high school teachers, indicate the goals set for the different lessons were both realistic for the educational level of students and curriculum requirements (60% agreement in terms of educational level and 80% for curriculum requirements, both indicated by all the high school teachers).  The outcomes of each lesson are also typical outcomes teachers would cover with their own curriculum.  Of those who responded, 82% said they intend to either use the lessons or the lake data in their classrooms.  Additionally, 73% of the teachers indicated they had the resources and materials available to use the lessons and lake data, demonstrating the opportunity for teacher to use WOW in their classrooms. 

 

 

Limitations

The evaluation responses and feedback demonstrate the demand and need for information and interactive website like this one.  The foundation of Water on the Web is strong; however, respondents overwhelmingly indicated the need for more detailed information and background materials.

 

Although a majority of responses indicated the goals and outcomes of the lessons are educationally appropriate and covered in their own curriculum, 100% of the high school teachers indicated their students would need further preparation before receiving the different lessons.   In addition to more preparation for lessons, the teachers also indicated the student directions and worksheets would not provide the necessary guidance needed for students to complete the lessons. 

 

When looking at how the teachers would like to see the website improved, the results indicate the site contents are satisfactory but the details of the site need refining.  The majority of responses asked for better directions, better links to different areas of the site and more background on lessons and technology.  The suggestions ask for improved guidance and setup of the site.

 

Recommendations

Before this website can focus on the small details of the site, it needs to first lay a strong foundation in terms of information, content and activities.  Feedback from the survey indicates Water on the Web has done just that.  Next, the WOW staff needs to look at the more intricate details and running of the site including, site navigation, and details of the lessons and other site logistics.

 

The site provides a number of lessons that can be used in their entirety or in parts.  The teachers indicated their interest in using these lessons.  The teachers also indicate they need (and perhaps their students will need) more background and direction to using the technology associated with some of the lessons and lake data.  Feedback specifically focused on transferring data to excel, selecting specific dates, and creating comparison tables.  Either including an entire lesson for the teacher and/or students on the technology associated with the site, or better instructions within each lesson, would help facilitate the use of technology within specific lessons and the lake data activities.

 

In addition to more thorough directions in using website technology, the course participants indicated they would like more background information on the lessons posted on the site.  Although the lessons had appropriate outcomes and educational levels for students, teachers wanted to see the lesson expectations simplified, outcomes narrowed, and better procedures available for students to follow.  In addition, teachers felt they needed more information about each lesson, and more background.  Providing more information up front with each lesson may help teachers better understand the lesson, and how appropriate it is for their own students and help students complete the lessons.

 

Finally, participants requested a few details and additional information.  A couple of teachers were interested in having the lessons listed by grade level.  Similarly, one teacher inquired whether or not there was a lesson sequence, or suggested providing a sequence. Other teachers and participants suggested general improvement to make the site more user-friendly and easier accessibility to the data. 

 

Overall, the course participants were happy with the website and indicated they would use the site for their work or classrooms.  The suggestions they provided for the site focused on the details of lessons and the ability to easily navigate throughout the site.

The recommendations focused on the needs of the high school teachers who filled out the Website Evaluation.  Asking a more diverse audience of teachers and researchers to review the site may help confirm the recommendations for this website.   

 

^*  Addendum:  Significant changes have been made since the formative evaluation was completed. Since the time of the evaluation we have totally overhauled the architecture and the content of our website and the curriculum it contains.