There are many definitions of evaluation. According to the book Program Evaluation: Alternative Approaches and Practical Guidelines, many experts say it is an "...assessment of the extent to which specific OBJECTIVES have been attained" (1997) or they see it as "professional judgment ... or ... quality control" (1997). Still others believe that the definition of evaluation is "...the act of collecting and providing information to enable decision makers to function more intelligently" (1997). Authors Worthen, Sanders, and Fitzpatrick lean toward Scriven's definition which describes evaluation as the act of "...judging the worth or merit of something" (1997). All of these make sense.

So, what is my own definition of evaluation? In answering this question, it is important to remember that there are many purposes for evaluations as well as many different kinds. As the book Program Evaluation: Alternative Approaches and Practical Guidelines states, "...one's definition of evaluation is the product of what one believes the purpose of evaluation to be" (1997). For example, some purposes for evaluation might be for grades (teacher/student), employment (administrator/teacher), or program adoption/continuation (expert/program) as well as many other purposes. 

Likewise, there are many different kinds of evaluations. Teachers use both formal and informal evaluations in order to assign a grade to a student's performance. For example, for some evaluations I collect data and use my years of experience along with predetermined criteria to make decisions about the worth of an assignment and/or group of assignments in order to assign a grade. Although administrators do not place grades on a teacher's performance, they still use informal and formal methods to make judgments on the merit of an instructor. When decisions of program adoption/continuation are required, formative and/or summative evaluations may be employed. There are several other kinds of evaluations, like internal/external and pre-ordinate/responsive, that can be used in program evaluation as well (Worthen, Sanders, Fitzpatrick, 1997; DeStefano Lecture).

With all of this in mind, I have created my own definition of evaluation. My personal definition of evaluation is as follows: 
 

Issues

What must be considered when implementing a learning technology? When contemplating this question, several issues come to mind.

First, before implementing a learning technology, there must be a need for it and interest must be shown. Many times, educators have recognized a need for one program or another. However, all too often, the needed program is put off or eliminated simply because of the lack of administrative interest. 

A second issue to consider is the development of a viable plan for the program in question. Interested educators must show the need for the program through creating a well-organized plan, which the powers-that-be will consider. This plan should take into consideration the expectations of all those who are involved. In a school setting, those involved would include the school board, the administration, the faculty, the students and possibly even the community, depending upon the type of technology being proposed.

The well-organized plan mentioned above should include a section that thoroughly explains the financial aspects involved. This idea brings in the third major issue to be considered when preparing a learning technology for educational use: the budget. Realism hits when an educator's vision and desire to commit to a given learning technology collide with the fact that his/her district has limited resources to make the vision a reality. Obviously, computer availability, software needs, and the instructional time given to the facilitators through inservice all cost money. In order for a learning technology program to succeed, one must give ample time to plan a realistic budget for it.

A fourth issue to discuss, evaluation of the selected program, is actually one of the most important. It is important to periodically evaluate a program in order to keep it functioning. Also, it is imperative that the evaluator be an expert in the field that is being evaluated and be free from bias (Worthen, Sanders, and Fitzpatrick, 1997; DeStefano Lecture, Week 1b). 

The idea of evaluation actually presents several salient issues within itself. To begin, there should be a balance among several evaluation methods. For example, technology is growing so rapidly that using a formative evaluation mixed with a summative one may be the most productive way to get quick feedback (DeStefano Lecture; Week 1c). 

Other evaluation methods that may work could be internal or external evaluations. According to Worthen, Sanders, and Fitzpatrick, an advantage to an internal evaluation is that the evaluator is directly involved with the evaluand, knowing the details of the program. This can provide "...important contextual information..." (1997) about the evaluand. However, one might feel that employing an external evaluator might be a way to preserve objectivity (Worthen, Sanders, and Fitzpatrick, 1997). 

There are other issues involved in the evaluation stage of a program as well. For example, evaluations help us use our resources wisely (DeStefano Lecture, Week 1c; Worthen, Sanders, and Fitzpatrick, 1997). Also, people are able to judge teacher/student satisfaction with the program by conducting evaluations (DeStefano Lecture, Week 1b). In addition, past evaluations create a history for the evaluand, giving important information about it (Robert Stake Lecture, Week 1g). 

The last issue to be discussed in regards to implementing a learning technology would be the outcome of the evaluation. After evaluating a program, one can decide whether or not the evaluand met the expectations of those involved. Are the users and facilitators satisfied with its performance (DeStefano Lecture, Week 1b)? If they are, then the program will most likely continue as is, with very few adjustments. However, if they are not satisfied, the program will need to be refined in some way or dropped all together. 

Obviously, one can see that there are several issues worthy of thought when implementing a learning technology. To summarize, there must be a need and interest, a well-organized plan, ongoing evaluation (which has issues of its own), and a commitment to refinement after the evaluation. These issues must be considered in order to foster the implementation of a learning technology.
 

My proposal is to evaluate the learning technology called Education and Career Opportunities System (ECOS). This Internet-based program is sponsored by the Eastern Illinois Education for Employment System (EIEFES) and is supervised by Dennis Hutchings, the Student Services Coordinator for EIEFES. 

ECOS is an online learning technology published by Embark.  Embark was founded in 1981 and its network is ". . . now used by millions of students, name brad businesses, and leading universities worldwide" (Embark, 2002).  Soon, another company, The Princeton Review, joined Embark.  Both companies feel the same way when it comes to technology.  They have a ". . . belief that technology can greatly enhance the college admissions process and make a positive difference in the experience of students, parents, counselors and colleges" (Embark, 2002).

The Princeton Review is a leader in college admissions and services.  Over 1.5 million people use their services monthly.  The Review helps over 1,000 colleges with recruitment of students.  In addition, more than 1,400 high school counselors use their services to help parents and students understand admissions forms.

More information about all of these can be found by logging on to the web site at http://www.ecos.embark.com . While ECOS itself is a fairly new Internet-based program that does not have much written history, information about Embark and The Princeton Review can be found easily.  Again, to access that information, one can log on to the ECOS web site and click on "About Us". 

Description of Evaluand

The Education and Career Opportunities System (ECOS), a program is being used by Cumberland Unit District #77, is an Internet-based program that deals with many issues that concern high school students. Because of this, it is important to look at the program from different angles and points of view.  A true description of a program brings forth both the positive and the negative aspects of that program.

To begin, there are many advantages to using this program.  For example, it is an excellent resource for junior and senior students because 
it provides learning opportunities in the following areas: careers, colleges, majors, scholarships, and ACT and SAT preparation. In addition, it provides each student with a personal planner/calendar, an email account, and a locker for storing information. Likewise, the teacher can benefit from using ECOS. For instance, the teacher/facilitator can track his/her students throughout the program. He/She has access to every student's user name and password, which gives him/her access to each student's information throughout the program. An added benefit is that class assignments can be posted on the home page.  That way, a student who missed class or forgot the assignment can log on and check the daily assignment.

While there are many advantages to using this program, there are also issues that need to be considered before full implementation can occur.  One of these issues is the availability of the program to the students.  Do all students have access to a computer and Internet service?  A second issue of concern is that ECOS supplies email access for the students.  The intended use of this feature is to provide the opportunity for the students to communicate with colleges, counselors, and teachers. That is a great idea; however, in the Cumberland district, students are not allowed to email anyone from school computers.  That is a real problem that will have to be solved before this program can be permanently added to the curriculum. Another issue to be addressed is the fact that the business department at Cumberland High School already has a career program called CIS that is used with the younger students in the school.  Will the business department be willing to change to ECOS?  Would it be worth the change to those involved?  Finally, for now, the use of ECOS at Cumberland High School has been paid for by Eastern Illinois Education for Employment System (EIEFES).  However, that sponsorship only lasts through the 2002-2003 school year.  If   the program is incorporated into the tech prep classes and the sponsor pulls out for some reason, Cumberland District will have to pay for it.  Obviously, having a thorough evaluation before full implementation into the curriculum is needed.

Key Stakeholders

There are several key stake holders in this evaluation.  A primary stakeholder is, of course, is Eastern Illinois Education for Employment System (EIEFES), Cumberland's sponsor.  Another stake holder is Dennis Hutchings, the Student Services Coordinator for EIEFES. Other stakeholders include the administration and staff at Cumberland High School.
 

Evaluation Questions

To begin an evaluation, one must understand the needs of the client and the stakeholders.  For this particular evaluation, the client, Dennis Hutchings, listed several issues he would like to see covered.  These issues can be divided into two categories: Student Domain and Teacher Domain. The chart below illustrates the areas to be covered for the client.

There are several other questions that will be addressed in this evaluation as well.  The chart below illustrates some general questions concerning cost..

Process

My evaluation of the ECOS program will be a combination of several different types.  It will be descriptive and summative with focus on the utilization of the system.  In addition, a consumer-oriented evaluation will be provided.  These concepts as they relate to this evaluation will be explained throughout this section.

Rationale for Evaluation

The main reason for conducting this evaluation is to determine if the ECOS site is advantageous for the students at Cumberland High School.  Secondary reasons for this evaluation are to help ECOS provide a learning technology that is user-friendly and helpful to high school students and to determine if it is worth the money spent on it.

Methods
Subjects | Instrumentation | Procedure for Data Collection | Analysis |

Subjects

The subjects (junior and senior students, faculty members, and administrators at Cumberland High School) will be selected by a combination of convenience sampling and purposive sampling.  There is a possibility that I may select some subjects from other school districts in the state of Illinois who have used the program; however, this is not definite. In addition to the surveys, the subjects will be randomly interviewed about the ECOS system.

Instrumentation
Web Site Worksheet | Checklist for Evaluations | Survey | Interview | Log Book

In order to collect the information needed for this evaluation, several different instruments will be used to cross-check the results.  This will help validate the findings.  The instruments used will be as follows:
 

Examples of these are given below:

Because this is primarily a consumer-oriented evaluation of an online program, a web site evaluation worksheet and a checklist will be used. These can be seen below:

With the client's questions in mind, the online survey, using a free online service (http://free-online-surveys.co.uk), was created in order to begin gathering data for my evaluation.  The survey web site was a great site as it provided the basic tools needed to create a survey.

However, it only covered the basics. Unfortunately, when trying to change the font size to accommodate the vision impaired (Bullock lecture, Wk 6b), one has to contact the company and get special permission and instructions. Also,  thirty questions and  fifty  respondents was the limit of the free service.  To ask more questions or to collect more than 50 responses at a time requires an upgrade. Therefore, one would pay for the service rather than receiving it free. (However, there is a way around this. While the  30 questions limit is firm,  more respondents can be accepted by recording the results and then clearing them after each group responds. This way, there can be more than 50 respondents to the same survey.)

While creating the survey, the advice given by experts was definitely in mind. For example, Lizanne DeStefano and Cheryl Bullock mentioned several points that needed to be considered when creating a survey or a questionnaire. For instance, both said it is important to list easy questions (like gender) at the beginning and move slowly into the more thought-provoking questions later. They feel it is important that the survey is attractive yet covers the focus of the evaluation. Also, they feel strongly that surveys need to be piloted in order to check for clarity (DeStefano lecture, Wk 4:8; Worthen, Sanders, and Fitzpatrick, 1997; Bullock lecture, Wk 6c). These areas were covered when creating and preparing the  survey for use.

The next step was to send the survey's URL to others for review. The survey was sent to the client, to the high school counselor,  to classmate Heidi Adreon, and to Merrill Chandler . It was also piloted, using six students. 

Having others look at the survey before actually implementing it proved to be a valuable thing to do. Taking their comments into consideration, several questions were adjusted and even deleted while others were added. For example, one of the questions reads as follows: How often do you visit the Careers module? The original answer choices were as follows: once a week or more; once every 2 weeks; once a month; less than once a month; (and) other. After reviewing the survey, Heidi suggested that another answer option be given: I have not visited this module. The adjustment was subsequently made (Adreon critique, 2002). 

Other adjustments resulted from Merrill's suggestions. Basically, he noticed that, at times, that too much was packed into single questions (Chandler critique, 2002; DeStefano lecture, Wk 4:9; Worthen, Sanders, and Fitzpatrick, 1997). So, the suggested questions were reviewed and adjustments were made. He also noticed instances where there was too much knowledge expected from the respondents (Chandler critique, 2002; DeStefano lecture, Wk 4:9; Worthen, Sanders, and Fitzpatrick, 1997). Therefore, the questions involved were changed. 

Finally, by piloting the survey with a few students, it was found that my questions were clear and understandable. This will help the accuracy of the results (DeStefano lecture, Wk 4:9; Worthen, Sanders, and Fitzpatrick, 1997).
 

CRITIQUE OF SURVEY

Upon completion, the collection and analysis of the results became the issue. Upon further review, there is evidence of the hierarchical nature of the levels of measurement or scale of data (Bullock lecture, Wk 6a; DeStefano lecture, Wk 4:18). For example, there are questions that would be considered nominal in that they ask the gender of the respondent. Obviously, this requires very little thought. There are also ordinal questions in that the question of whether they are students (juniors or seniors) or parents is asked. More evidence of the levels of measurements is seen later in the survey when it asks how useful a given module is. The respondents are asked  to rate the usefulness from very useful to not useful at all. This is the interval level (DeStefano lecture, Wk 4:18; Bullock lecture, Wk 6a). So, several different levels of measurement have been covered.

After the survey is completed, the results will help determine the questions for the interviews.  At this point, the tentative interview questions are listed below:

Procedures for Data Collection

Before collection takes place, permission for collection is needed.  In order to obtain that permission, one must first send in an IRB-1 Form, a form for review of research involving human subjects.  With that form, the evaluator must provide sample copies of consent forms both for children and adults.  Below are the consent forms attached to the approved IRB-1 Form.
 
 

The data will be collected in a number of ways.  Not only will it be collected in the more traditional way, but also by unobtrusive measures such as the number of hits on this web site. The worksheet and checklist will be completed by the evaluator so the collection procedure is obvious.  As for the online survey, the subjects will be picked in a purposeful manner. This evaluator will collect the data immediately after the subjects complete the survey.  It will be available immediately and for ten days after the surveys are taken. If the results are not picked up from the online site within ten days, a service charge will be assessed before results will be displayed.  Continuing, the interviews will be held in a classroom at Cumberland High School and audiotaped. Then, the transcripts will be provided to the interviewees for verification and clarification.  Finally, a log book will be maintained in order to document the different events of this evaluation. If used, the focus group will be videotaped in order to be able to read the body language and emotions that are attached to the words of the subjects.

To insure the validity of the responses, special attention will be given to the anonymity and confidentiality of the subjects.  For instance, the online survey has no place for any specific information about the subjects. They simply log on, point and click, and log off.  As for the paper/pen surveys, anyone taking those will be asked to refrain from putting his/her name on the paper and to place the completed survey in an envelope, which will hold all the paper/pen responses in no particular order.  Also, audiotaping and videotaping will not be done without the written permission of those involved. 

Analysis

As for the results, at this time I feel I will use the descriptive type of analysis. The results of my survey will return as percentages so I will have easy access to that. I will also be able to provide simple summaries of the meanings of these percentages as well as the results of the open-ended, essay-type questions that are scattered throughout the survey. In addition, I will be able to simply describe what the results mean or what the data shows (DeStefano lecture, Wk 4:17; Bullock lecture Wk 6a).

Timeline/Schedule

The web site worksheet and the checklist was completed in late February.  So was the pilot survey. The online surveys were completed in early March as was the pilot interview.  Student and staff interviews will take place later in March while the focus group (if needed) will be held in the middle of April.  The log book is the continuous responsibility of this evaluator.

Adequacy of Methods

The methods used during this evaluation are effective in many ways.  They are comprehensive, legal and ethical, and cost-effective.  There is a minimal amount of disruption to the daily routines of the subjects involved.  Also, these methods do not take an enormous amount of the subjects' time to complete.  Because there are several methods, results will be able to be cross-checked, which leads to reliability and validity. 

Reporting Plans

A formal report of the evaluation results will be given to the client, Dennis Hutchings and to anyone else who requests a copy.  Informal reports will also be given to all participants.  This includes Cumberland High School administration, staff, and students. 

Interpretation Procedure

Basically, the evaluation will answer the questions: Is the ECOS program user friendly? Does it meet any state educational goals? Is it worth the cost? Naturally, other question will arise during the evaluation and will be considered at that time.

Audience

The primary audience for this evaluation is the Eastern Illinois Education for Employment System (EIEFES) and  Dennis Hutchings, Student Services Coordinator. The secondary audience would be Cumberland District #77 administration, faculty, staff, students, and community.

Content

The content of this evaluation will help answer the questions:  Is this learning technology user friendly?  Is it worth its cost?  What changes need to be made?

Presentation Schedule

In the middle of April meetings will be conducted to discuss the evaluation results.  There will be one meeting with Cumberland High School's counselor, Mrs. Diann Bomer,  to discuss her interpretation of results and to make any necessary changes. A technical report will be presented to Dennis Hutchings, the Student Services Coordinator for Eastern Illinois Education for Employment System.  As many meetings as are needed will be scheduled with Mr. Hutchings to discuss the evaluation results. There will also be one meeting with Cumberland High School administrators to verbally report the findings. An  informal report will be given to the Cumberland High School staff who participated. The  release of written formal report is scheduled for late April. Copies of the written report will be made available to the public upon request.
 

Resource Estimation

Below is an estimation of the budget for this evaluation.  While it is comprehensive, it is still only an estimate.

Evaluator's Time: $25/hour
Personnel (staff and consultants): $25/hour
Internet Access Fees: $25/month
Online Survey Fees:  $10 
Participation Incentives: $25
Printing and Duplicating: $50
Mailings: $30
Misc. Office Supplies: $25

Other expenses will be added as they arise.
 

Meta-Evaluation

In the eyes of this evaluator, meta-evaluation is an on-going event.  This evaluation is, among other things, an internal one. Because of this, special precautions should be in place. Actually, according to Worthen, Sanders, and Fitzpatrick, an advantage to an internal evaluation is that the evaluator is directly involved with the evaluand, knowing the details of the program. This can provide "...important contextual information..." (1997) about the evaluand. However, a disadvantage to the internal evaluation method is that the evaluator most often is so involved in the program that he/she may miss some important discrepancies.  To avoid this, a log of the events of this evaluation will be kept. That way, it can be reviewed at any moment to ensure non-bias results.  In addition, constant contact will be kept with an external volunteer evaluator whose job will be to evaluate the evaluator. 

Which departments use it? (English, social studies, counseling etc.)

How many students are involved?

Evaluation period

Email addresses of the client, the sponsor, and the adult subjects

Other components as they arise
 

Checklist - late February

Pilot Survey - late February 

Surveys - early March 

Pilot Interview - March 

Interviews - late March

Focus Group (if used) - middle of April

Secondary - Cumberland administration, faculty, and staff, community

Release report - late April

I.  Web Site Evaluation Worksheet

The evaluation of the ECOS program was best suited to begin with the Web Site Evaluation Worksheet.  This evaluation worksheet was adapted from Dr. Nancy Everhart's 1996 worksheet. The worksheet covers several areas:  currency, content/information, authority, navigation, experience, multimedia, treatment,access, and a miscellaneous section.

Currency
The ECOS web site did not score very highly for this section. The worksheet asked if the "...site has the date of last revision posted" (Web Site Evaluation Worksheet, 1996).  While searching the site, a revision date was not found on it anywhere.  All that was available was the copyright date.  Also, there was no definite proof that the site had been recently update; however, it did seem to be current at the time of this evaluation.  Unfortunately, there was no mention of possible times of expected updates (Web Site Evaluation Worksheet, 1996).  So, of a possible fifteen points, ECOS scored a 6.3 on this section.

Content/Information
This section of the worksheet was quite extensive.  It covered such topics as the web sites usefulness to the curriculum and the extent of student interest.  It also asked about whether or not the information provided on the site is available in the school's resource center. Along with questioning the site's accuracy and thoroughness, the worksheet asks if the "...purpose of the page is obvious . . . (if) the information is in good taste . . . (and if) the page uses correct spelling and grammar" (Web Site Evaluation Worksheet, 1996).  It was determined that this site fills all of the mentioned requirements, scoring 14.7 points of a possible 15 points.

Authority
This section of the worksheet covered specific details about the authors.  It asked if  it is easy to find who authored the site and if they are knowledgable in the content of the site.  Also, it seemed important to be able ". .  . to contact the author(s) via e-mail or traditional mail" (Web Site Evaluation Worksheet, 1996).  Finally, it asked whether or not it is easy to recognize the site's domain.  Upon review, this site scores 9.5 points of a possible 10 points.

Navigation
When following the worksheet, an evaluator will find that the navigation of the ECOS site is extremely easy.  There is obvious visual organization as well as consistent layouts of pages.    The font and colors help visitors read and understand the set up of the site and complement the functional, meaningful links that are available (Web Site Evaluation Worksheet, 1996).  This site received 10 points of a possible 10 points in this area.

Experience
This section covers the student's experience when visiting this site.  Subjects such as purpose, worth, and attractiveness are addressed.  Did the site do what it intended to do?  Was the visit ". . . worth the time (spent)" (Web Site Evaluation Worksheet, 1996)?  Was it attrative and inviting to the visitor (Web Site Evaluation Worksheet, 1996)?   Again, the site scored well on this subject, receiving a 9.8 of a possible 10 points.

Multimedia
This part of the worksheet evaluates the multimedia elements on this site.  The site has only a few graphics, and no sounds or video enhancements were found.  Therefore, it receives just five points of a total of 10.

Treatment
For this section the evaluator must check for  bias, stereotyping, and audience awareness.  Upon studying the information provided, it was determined that treatment of the information on this site is ethical and age appropriate.  Therefore, it received 10 points of a possible 10 points.

Access
When evaluating the ease of access, one must keep in mind several things.  How long does it take to log on? Is the site a link on search engines?  How long does it take for the page to load?  Is there an option to choose a variety of downloading options?  When considering these questions, it is learned that the log on time is not long at all.  Also, it is believed that the site can be found on search engines; however, one must have a subscription, a user name, and a password to access the site. Because this is a large site, it takes a little longer than others to load. However, the wait is not as long as what would be expected from a site of this size. Unfortunately, the visitor may not choose the type of download he/she desires.  With all of this in mind, the ECOS site receives a 3.25 of a total of 5 points.

Miscellaneous
This section covers many different areas that do not fall into any other categories.  The following subjects are covered: Is it an award-winning site? (Yes; several different kinds) Does it cost any money in order to use it? (Yes, a group subscription usually paid for by a sponsor)  Is the users privacy protected? (Absolutely!)  Does the user have to change the configuration of his/her printer in order to obtain a copy of information found on the site? (Yes, but only for color).  Is the information ". . . presented in short enough segments so it can be printed without backing up the system for othre users? (Yes)  Does the site itself have an internal search engine? (Yes)  As seen by the answers to the questions, this section scores high with 14.3 points assigned of a possible 15.

TOTAL POINTS:  82.88  =  GOOD

Scoring:
90 - 100   Excellent
80 - 89     Good
70 - 79     Average
60 - 69     Borderline Acceptable
Below 60 Unacceptable

Since this is an evaluation of a web site, an informal, unobtrusive measure was taken. This involved the number of hits on the ECOS web site.  Only a teacher or a counselor can access this information.  Doing this simply provided another way to evaluate whether or not the site was being visited on a regular basis. 

The results of this unobtrusive measure prove that the ECOS web site is visited quite often. The total number of users at the time of this evaluation was 124, and, within a span of two months, the total number of logons was 1514. When breaking this number down by gender, it was found that a total of 72 males and 52 females used this program.  At the time of this evaluation, the males had logged on 865 times while the females had visited 649 times.

In order to report on the findings of the survey, one must consider the order of the presentation of information.  Because this evaluation  is extensive, it is logical to present the results of the student survey in the order in which the modules are presented on the ECOS web site. However, the report of the findings of this survey will begin with an overview of the respondents involved and will also include comments on select features of some modules.

A.  Frequency of Use and Access

Several students responded to the electronic survey which was posted on the ECOS web site.  In all, eight-two students contributed their thoughts about this learning technology.  The characteristics of the respondents involve three major concepts: gender, grade, and educational path. In raw numbers, there were thirty-six female respondents (44%) and forty-six male respondents (56%). (See Figure 1.) The differences in the percentages of respondents is related to the male/female ratio within the school. The survey was restricted to upperclassmen due to convenience. Figure 2 shows that most (91%) of the respondents were juniors, while 9% were seniors.  This lopsided statistic is also due to convenience as this evaluator had easy access to most of the juniors at this evaluation site. Finally, the education path of the respondent is an important factor. Actually, it could be seeen as the most important factor.  For example, 28% of the respondents are enrolled in tech. prep. classes and/or buisness English.  These are classes that focus on preparing students to enter the workforce immediately after high school. Therefore, these students may not have an interest in certain parts of the ECOS program like the test prep section, the college module, the major module, and the Financial Aid module.  This fact may affect the frequency of use, which is part of this evaluation.

All of those who replied to  this particular survey said they received access to the ECOS program through a teacher (Figure 3) even though they could have received access through the school counselor.  This leads one to believe that there are teachers in the building who are using and promoting this learning technology.

Thinking logically, one would realize that the next three results of the student survey are directly related to each other.  Obviously, if access and navigation are easily done, there will be a higher frequency of logging on.  Figures 4, 5, and 6 show the correlation among these three categories.  Apparently, ECOS is very easy to access, and students have no problem with navigation of the site.  Therefore, there is a high frequency of logins.  As Figure 4 shows, 81% of the respondents log on at least once every 2 weeks with 32% of these logging on at least once a week.  Only 19% log on once a month or less. 

This is proof that the site is easy to access and easy to use.  In fact, 94% of the students who responded said that accessing this system was an easy thing to do.  That is an overwhelming statistic.  Obviously, the authors have done their job in this area.  While this is true, one respondent did express the fact that he/she did not have a computer, which made it difficult to use the program.
 

Gender   Figure 1	Grade Figure 2	Type of Access   Figure 3
Log on Frequency Figure 4	Ease of Access Figure 5	Ease of Navigation Figure 6

B.  Module Usage and Helpfulness (Overall).

There are six different modules with which the students can work.  They are as follows:  Portfolio, Career, Major, College, Financial Aid, and Life Skills.  The student survey helped to pinpoint which modules were being used more than the others and which ones were used less . 

Most Used
Obviously, when thinking about the future, a lot of students ask themselves, "What am I planning to do after high school?"   When answering, two things naturally come to mind:  careers and college.  Figure 7 shows that the Careers module is the most used module in the program.  That module is used by 100% of the students who answered the survey.  This is not a surprise when one considers the characteristics of the respondents.  For example, 91% of the respondents were juniors.  At that age, most students are just beginning to think about life after high school.  In doing so, their thoughts naturally turn to possible careers.  Another reason that these results show a strong usage pattern of the Careers module is because 37% (27 of 82) of the students who responded are enrolled in a high school class that focuses on using English skills in the work place.  Therefore, those students were more likely to be pulled toward the Careers module rather than the other modules. 

Most Helpful
Figure 7 also shows a comparison of the helpfulness of each module. Here, one can see that not only is the Careers module the one most used, but it also is the one that is rated the most helpful by a wide margin.  In support of this module, one student responded, saying, "It helps me to choose a career that is right for me" (Anonymous, ECOS Evaluation Survey: Q14, 2002).  However, while the Portfolio is the second most used module (86%), it apparently is not the second most useful module.  On the contrary, with 24% of the respondents choosing it, the College module proved to take second in the most helpful category.  Again, this could relate to the fact that the respondents were mostly juniors, looking toward what to do after high school. 

Least Helpful
Also, it was determined that the Life Skills module is one of the lesser used modules and is the least helpful to the respondents (See Figure 7).  In order to explain this, one student responded, "...I don't have to worry about that kind of stuff right now" (Anonymous, ECOS Evaluation Survey: Q14, 2002).  Another respondent commented that he/she simply didn't "...need it" (Anonymous, ECOS Evaluation Survey:Q14, 2002).  This finding may show cause to evaluate that module more closely to see if there are any changes that could be made to show its worth to the students. 
 
 

ModuleUsage Figure 7

C.  Individual Modules Frequency of Use and Helpfulness.
 

Although it has a high frequency of use, it is not ranked as one of the top modules in regards to helpfulness. (See Figure 9.)  This is easily explained.  Logic states that the students at this age will not use the features available in the Portfolio module as much as some of the features in the other modules at this time.  The Portfolio module has sections that include a resume writer and a place to record course credits and other information that must be stored in order to fully use the other modules and other features of ECOS.  Therefore, since this module is basically a bank of stored information, there really is not a way to use it other than updating the information from time to time.  This fact obviously explains why it is not used as often as the other more interactive modules.
 

The Careers module proved to be the most used and the most helpful module in the ECOS program.  This module has obviously caught the eyes of the students. 

As mentioned earlier in this report, the fact that the students are attracted to this module is not a surprise, considering the characteristics of the respondents. For example, of a total of 82 respondents, 75 of them were juniors in high school.  This age group is just beginning to think about the future. When these thoughts enter their minds, students naturally ask, "What do I want to do after high school?"  As expected, many answer, "I want a good career."  So, when the resources are available, they research different careers. ECOS provides the students with the resources. Logically speaking, a careers search would be the place to start, even if they plan to go to college directly after high school.  They need to find out what their interests are so they can plan the other parts of their life after high school.

Again, with all this in mind, it is obvious why the Careers module has been chosen through this survey as the most used and most helpful module that is available in the ECOS program.  As Figure 10 (below) shows, 44% of the respondents use this module at least once every two weeks, with 17% using it once a week or more.   Likewise, shown in Figure 11, 35% (29 of 82) of the respondents rank the Careers module as "very useful".  Clearly, this is one of the strongest sections within the ECOS program.
 
 

Career Module (Usage) Figure 10

Careers Module (Helpfulness) Figure 11

According to the student survey, the Major module ranks fourth, behind the Careers module, the Portfolio module, and the College module, in regards to usage.  This makes sense in that the natural progression of thought for students in this age group would follow this path.  First, they would think about what they want to do (career). Next, they would gather some personal information about themselves like contact information, references, class rank, and grades in order to fill out college applications (portfolio). Following that, they would obtain information about different colleges (college), and then start exploring a major interest of study (major). So, it is no surprise that the Major module ranks forth in usage.

The charts below illustrate the respondents' usage of the Major module as well as the helpfulness of that same module.  As shown in Figure 12, most of the respondents, 56% (46 of 82), visit the Major module once a month or even less than once a month.  In fact, ten (12%) of the respondents had not visited the Major module at all at the time of the survey.  On the other hand, these percentages are balanced by the 26% (21 of 82) that visit this module at least once every two weeks.   Figure 13 (below) shows that most of the respondents (37%) feel that this module has only an average amount of helpfulness while 16% of them have no opinion at all about the helpfulness of this module. 

These statistics lead one to believe that this is a module that is not as important to the students as the Careers module, the Portfolio module, or the College module.  It is simply in the middle of the group of modules in the areas of usage and helpfulness. 
 

Major Module (Usage) Figure 12

Major Module  (Helpfulness) Figure 13

As shown in Figure 7, the College module is one to which the respondents are more attracted.  However, Figure 14 shows that 
that the majority of the respondents visit this module once a month or less than once a month.  To be more specific, this is 67%, or 54 of the 82 total respondents.  The category marked "Other" could have a combination of respondents who visit this site more than once a week and those who visit far less than once a month.  Therefore, it is not included in the percentage mentioned earlier. It should be recognized that 12% (10) of the respondents have not visited this module at all.

Figure 15 shows how helpful this module is to those who visit it.  As seen, 59% of the students who answered the survey rate this module as being "very helpful" and/or "somewhat helpful".  In contrast, 13% of the respondents had no opinion at all about this module.  This is probably because they are the ones who have not visited the module yet.  At any rate, it can be assume by the information collected that if the module is visited, it is helpful to the students.  As further proof, only 6% of the respondents stated that this module is of no use at all to them.
 

College Module (Usage) Figure 14

College Module  (Helpfulness) Figure 15

The Financial Aid module is much more helpful than the students realize.  They do not seem to spend much time here, maybe because they simply do not think to look up ways to save money on college.  On the contrary, the parents are probably more attracted to this than the students for obvious reasons; parents pay for college. While this is true, this evaluation is focused on student usage, not parental usage.  Therefore, the results of the survey cover the frequency of student usage and the amount of help the students receive from this module.

According to Figure 16, one can see that this module is simply not used as much as the others.  In fact, 50% of the respondents had not even visited this module at the time of the student survey.  Only 11% of the students responding to the survey visit this module once every two weeks or more with a minuscule 2% visiting once a week.

Obviously, if 50% of the respondents had not visited this module, one can only assume that the students would not have much of an opinion about it.  This holds true as seen in Figure 17.  According to that chart, 57% of the respondents had "no opinion" when asked about this module's usefulness.  Howeever, once these students come closer to actually going to college, this module will become more important to them.

Financial Aid Module (Usage) Figure 16

Financial Aid Module (Helpfulness) Figure 17

The Life Skills module is another module that is not visited very frequently.  In fact, only 2% of the respondents visit this module once a week or more. Furthermore, 41% of the students who answered the survey reported that they have never visisted this module.  (See Figure 18.)

Because it is not visited often, over half (54%) of the respondents did not have an opinion about its helpfulness.   However,  40% visited this module and believed it to be of average (or more) use to them.  (See Figure 19.)  Therefore, it can be assumed from this survey that this module is somewhat effective if it is visited. 

Reasons for the lack of interest in this module can only be of the speculative nature.  One can determine by the comments received on this survey (See Question 14) that the respondents have not thought about the different aspects of the Life Skills module yet.  They are focused on other parts of their life that seem more important to them.  This is summed up by one respondent's comment: "Life skills is the least helpful to me right now because I am not looking for information on life skills" (Anonymous, ECOS Evaluation Survey: Q14, 2002).

Life Skills Module Figure 18

Life Skills Module Figure 19

D.  Calendar.

According to the chart below, several of the respondents use the calendar which is available. Figure 20 proves that 63% of the students who responded to the survey use this feature at least once a month, including 33% who visit it once every two weeks. Also, Figure 20 shows that all of the respondents have visited the calendar at least once.  One student responded, "The calander is helpful to keep me organized and updated on things that my school is doing" (Anonymous, ECOS Evaluation Survey; Q30 2002). From this information it can be assumed that the students appreciate the features that are available to them within this section of the ECOS program. 
 
 

Calendar Figure 20

E.  ACT/PSAT Test Preparation.

Considering the characteristics of the respondents, the test preparation section is extremely important to the college-bound students.  Judging by the chart below (Figure 21), an overwhelming 82% of the respondents conveyed that they felt that this section of ECOS is at least of average help or more.  Of this 82%, 29% feel it is somewhat helpful while 21% feel it is very helpful. 

Again, considering the characteristics of the respondents, it is not surprising that 11% have no opinion and/or have not visited this section. (See Figure 21.) This is probably linked to the percentage of students who are enrolled in the business English/tech. prep courses coupled with the number of seniors who responded to this survey.    To explain, after high school, the tech prep students, in general, plan to go directly into the work place instead of attending college. Therefore, college testing is not one of their priorities.  Also, the seniors are not as interested in preparing for the ACT or the SAT as the juniors are because, more often than not, these tests are taken during the students' junior year. Consequently, the seniors have probably already taken the test if they are going to college after high school.

To explore more deeply the feelings of the respondents, one must turn to their comments in regard to this section.  One supporter of this section said, "I believe that the ECOS systemis a very good place for high school students to go and get help and information . . . . It helps with the SAT and ACT tests so that you know what to expect . . . " (Anonymous, ECOS Evaluation Survey: Q14, 2002).  However, not all comments were positive in regards to this section.  One respondent wrote, " ...the ACT test prep is a bit confusing with the answer sheet and the questions so close, it's a little hard to keep scrolling up and down to find the answer" (Anonymous, ECOS Evaluation Survey: Q14, 2002). On the whole, however, this section of the ECOS program is considered to be helpful to those who visit it. 
 

Test Preparation Figure 21

F.  Student Comments.

To understand the overall feelings of the respondents in regards to the ECOS  program, the students were simply asked what they thought of the program.  The last question on the ECOS Evaluation Survey was as follows:
 
 

When compiling the responses to this question, this evaluator placed them into two groups: positive responses and negative responses. Upon doing so, it became obvious that the overall student opinion of the ECOS program is overwhelmingly positive.  As seen in Figure 22, 90% of the students responding to the survey commented positively. Only 10% had anything negative to report about the ECOS program.  Obviously, the positive comments made in reply to the statement above (#30) far outweigh the negative ones.
 
 
 

Overall Student Opinion Figure 22

The subjects had much of the same feelings as the students showed throughout the survey. For example, as mentioned earlier, the students felt that the Career module and the College module were of great help to them.  In a like manner, the teachers and counsellor felt the same.  On the contrary, when asked which module was of little or no value, teachers and students differed in their opinions.  During the interview, the teachers agreed that all of the modules were of some value; however, during the student survey, the students conveyed that the Life Skills module was the least helpful module.

In question 5, the counselor was asked if he/she ever recommended the use of ECOS to anyone.  The counselor responded positively, saying that the financial aid module is extremely helpful to parents of those students who are considering continuing their education after high school.  Likewise, it seems as though the college module is recommended as well because the students can apply to different colleges online while waiving the application fees.  Finally, the Career module was mentioned as somewhere students are directed in order to give them guidance.

Question 6 asks about any technical and navigational problems that may need to be addressed.  There were major technical problems that seemed to be consistently mentioned.  The first one concerns the login problems for new useres.  Those who help the students login for the first time have mentioned that, at times, they have to use a former student's identification number instead of their own before they are allowed access.  At this time, no one can pinpoint the reason for this problem.  The second problem is during the use of the Test Prep seciton of ECOS.  It has been noticed that when one opens a practice ACT test, the clock begins running before the page is displayed.  This causes the students to become anxious about the time ticking away on them. Likewise, when practicing the ACT, users noticed that the answer sheets do not match the test.  For example, the answer choices on the answer sheets may show "A B C D", but the questions on the test may have the aswer choices as "F G H J."  Contrary to the technical problems mentioned, the navigation of the site seems to be very easy because of the site's design.

Several interview questions deal with teacher/counselor usage of this site.  When asked about the frequency of use, one teacher said the use of ECOS was incorporated into the class lessons during a unit that was two months long.  Students worked with the system once a week during those two months.  This teacher said the students explore the site, practice the ACT, and use the calendar/planner.  Also, the students' assingments are put on the homepage in the Bulletin section.

Other interviewees responded to this question in different ways.  One said emailing was used for quick contacts with the students. In addition, ECOS is used to guide juniors and seniors to careers and colleges.  The other interviewee said that ECOS was not part of the curriculum right now because this teacher is not familiar enough with the program yet.

In answering question 9, the subjects responding in several ways.  One said she used the calendar/planner for monitoring upcoming school activities. She also used the user directory  and the tracking section to monitor the students as well as to check on how they performed on the ACT practice tests.  The reason for this, she said, is obvious; she teaches juniors and the state now assesses juniors using the ACT test.  Another "counselor" mentioned enjoying being able to customize the homepage.  The other subjuct said she simply was not familiar enough with the program at this time in order to comment on it.

Continuing, the subjects commented on each module that is available to them.  The chart below will illustrate their feelings.
 
 

Question 14 of the interview asks the subjects to mention any technical problems they may have encountered along with any possible sollutions they may have for each problem.  The same problems that were mentioned earlier were mentioned again.  They seem to agree that the log in problems for new users and the ACT format are the major areas of concern.  While they had no sollutions at this time for the log in problems, one subject did mention something about the ACT format.  She states, "Maybe if they made it just one page with one scroll bar, it would be more user friendly."

When asked how valuable ECOS is to their students, the subjects expressed definite opinoins.  One subject said, "The students comment on how they enjoy loggin on. They like the fact that there are assessment test in the College and Career modules.  They feel like they can match up their interests to certain careers so it gives them a direction to go in life."  A second interviewee mentioned, " I think it is very valuable.  It helps them find a direction.  We have several students here who need help figuring out what they want to do.  This site helps them with that."  Finally, the third respondent said, "I think the lIfe Skills module looks like it could help my younger students.  It may fit into my curriculum."

Next, they were asked about ECOS's worth to the district.  In general, they all agreed that it is important for the district to continue to expose their students to this program for many reasons.  For the most part, they feel it is important because of the characteristics of many of the students within their district.  One spoke for all when she said, "I feel ECOS is extremely valuable to our district.  We have a high percentage of students who go directly into the workforce after high school.  ECOS helps them with what direction to go.  If they head off in the right direction, they will become responsible adults.  That is our job as an educational institution."  As mentioned, the other interviewees agreed.

Key questions in the client's mind are addressed in questions 17 and 18 of the interview.  In short, question 17 asks if the Eastern Illinois Education for Employment System (EIEFES) should continue sponsoring this program for districts.  In response to this, one responded while another agreed:  "I feel strongly we should still have this program.  I am concerned that the district won't pay for it because we already use a free careers program online.  So, if EIEFES  could continue its sponsorship, it would ensure that more of our students could be exposed to its benefits."  In the same manner, these respondents also said that they would most definitely recommend the purchase of this program if the sponsorship was discontinued. However, they reiterated their concern that the district would opt for the free program instead even though it is obviously not as good a program as ECOS.

The interviewees' overall impressions of the ECOS program are illustrated in the chart below:
 
 

Upon completing this checklist, it became obvious that this web site is definitely worth the time that the students and teachers spend there.   For example, the content ". . . cover(s) a significant portion of the program competencies" (Worthen, Sanders, and Fitzpatrick, 1997).  In addition, the content is current and is geared toward the high school level. 

To continue, the materials available to the users are very interesting.  There are computerized assessment tests within several modules to help the students determine their interests and to help them find for what they are most suited.  There are also some practice standardized college tests (ACT and SAT) so the students can prepare themselves for their future.  As for the teachers and counselors, there is a teachers' section that allows educators to edit the site to make the site more useful to their individual class or school.  The homepage can be personalized and assignments can be placed on a bulletin board.  There are several other features such as management options like user password access and access to the students' lockers (personal accounts).  Lesson plans are also available on this site. In addition, the visitor will soon recognize that the site is set up in a "logical sequence" (Web Site Evaluation Worksheet, 1996).

As for the quality of the site, the user can immediately recognize the high quality of this site.  Most of the activities are accurate and interesting.  The purpose of the site and the student tasks that are available are closely linked. 

While the majority of this site is of high quality, it is not without its problems.  After using the test preparation pages, this evaluator noticed some minor errors through the ACT test.  Also, there is a clock that times the ACT practice tests for students who log on. There is a problem with that clock; it begins immediately after the user clicks the link to the page. The problem with this is that the page itself takes longer to load than the clock does.  Therefore, the students lose testing time as they wait for the questions and answer sheet to appear.  Also, the questions and the answer sheet come up side by side on the screen but have separate controls.  While this is probably the only way to do it, it presents a bit of a problem when the reading material has to be moved before the answer sheet does. Sometimes, this confuses the students, which results in clicking on the wrong line when answering a question.

Completing this checklist helped to cross check the data received from other evaluation instruments. 

In conclusion, one can see that this evaluation is important to all concerned.  It has the potential to affect the curriculum at Cumberland High School, which, in turn, affects over 350 students and approximately 60 staff members. With the information provided in this proposal, this evaluator feels confident that the evaluation of the learning technology Education and Career Opportunities System will be thorough and successful.  This evaluation has the potential to be used immediately upon completion.

Bruce, Chip, Michelle Hinn, and Kevin Leander.  "An EIT Special Guest .....Michael Quinn Patton Utilization-Focused Evaluation:  The Union Institute." Evaluation News Homepage.  10 Feb. 2002 < http://cter.ed.uiuc.edu>.

Bullock, Cheryl. Lectures Week 6 a-c. EdPsy 399 EA: Evaluation and Assessment in Learning Technologies. CTER CD-ROM. Spring 2002.

Chandler, Merrill. Critique. E-mail. 22 Feb. 2002.

DeStefano, Lizanne. Lecture. EdPsy 399 EA: Evaluation and Assessment in Learning Technologies. CD-ROM. Spring 2002.

Embark. N.p.: Princeton Review, 2002. 14 Mar. 2002 < http://www.embark.com >.

Everhart, Nancy. Web Page Evaluation Worksheet. Jamaica: n.p., 1996. 25 Jun. 2001 <http://www.duke.edu/~de1/evaluate.html>.

Free Online Surveys.com. 1 Mar. 2002 < http://www.freeonlinesurveys.com >.

Stake, Robert. Lecture. EdPsy 399 EA: Evaluation and Assessment in Learning Technologies. CD-ROM. Spring 2002.

Worthen, Blaine R., James R. Sanders, and Jody L. Fitzpatrick. Program Evaluation: Alternative Approaches and Practical Guidelines. New York: Longman, 1997.