Distributed Education Task Force

Evaluation Sub-Committee Preliminary Report (REVISED)

Members:

March 2001

TABLE OF CONTENTS

0. Introduction

I. Organizational Structure of Evaluation/Assessment

1. Who will carry out the evaluations/assessments?
2. Who will look at it?
3. Who will take an action, such as cancel, expand, improve etc.?

II. Student Learning Evaluation/Assessment

1. Are they learning what is expected?
2. What are the learning objectives, expected learning outcomes, appropriate assessment methods and tools?
3. Are the students satisfied with their learning?
4. What could be the evaluation/assessment tools?
5. Student support advising

III. Course Design Evaluation

1. How do we design courses for delivery at a distance?
2. What type of courses should be replicated versus re-designed?
3. Do we need a standardization on the format?
4. What are the minimum standards expected on course design?

IV. Faculty/Instructor Evaluation

1. Satisfaction level of instructors using these alternative systems of delivery
2. A systematic mechanism to collect the instructors' opinion and suggestions.

V. Economic Evaluation/Assessment

1. Do they need to be profitable?
2. The program versus course feasibility
3. Market analysis (target groups etc.)
4. What are the economic objectives, expected outcomes, appropriate evaluation methods and tools.
5. What could be the evaluation/assessment tools?

VI. Institutional Evaluation/Assessment

1. Are the individual programs in accordance with the general mission of the University? What are the criteria for this?
2. Are the programs complementing each other not competing?
3. Are the resources provided to each unit equitable? (Same opportunities and support for all units)
4. Institutional cost and benefit analysis
5. What could be the evaluation/assessment tools?
6. Evaluation of public and private partnerships--collaborations

A. Appendices

1. Distance Education Program Development Agreement(DRAFT)

0. Introduction

Charge:

How to measure impact, market, learning outcomes, and learner satisfaction and will discuss such issues as the development of appropriate outcome measures. Also, competitive advantage needs to be considered. (Look at how to measure impact, market, learning outcomes & learning satisfaction. Include development of appropriate outcome measures.)

Overview of Resources:

From NCA Handbook of Accreditation

Guidelines for Distance Education: Any institution offering distance education is expected to meet the requirements of its own regional accrediting body, and be guided by the Western Interstate Commission for Higher Education (WICHE) Principles.

• The institution assesses student capability to succeed in distance education programs and applies this information to admission and recruiting policies and decisions.

• The institution evaluates the educational effectiveness of its distance education programs (including assessments of student learning outcomes, student retention, and student satisfaction) to ensure comparability to campus-bases programs.

• The institution ensures the integrity of student work and the credibility of the degrees and credits it awards.

From Regional Accrediting Commissions Draft Guidelines for the Evaluation of Electronically Offered Degree and Certificate Programs

The Guidelines are divided into the following five separate components, each of which addresses a particular area of institutional activity relevant to distance education. They are:

1. Institutional Context and Commitment
2. Curriculum and Instruction
3. Faculty Support
4. Student Support
5. Evaluation and Assessment.

On the Evaluation and Assessment draft report says: "Both the assessment of student achievement and evaluation of the overall program take on added importance as new techniques evolve. For example, in asynchronous programs the element of seat time is essentially removed from the equation. For these reasons, the institution conducts sustained, evidence-based and participatory inquiry as to whether distance learning programs are achieving objectives. The results of such inquiry are used to guide curriculum design and delivery, pedagogy, and educational processes, and ,may affect future policy and budgets perhaps have implications for the institution's roles and mission."

From National Education Association and Blackboard Inc. Benchmark for Internet Based Distance Learning Study

The benchmarks distilled from this study are divided into seven categories of quality measures currently in use on campuses around the nation. Many are common sense, but the study validates their importance. The categories and benchmarks include:

1. Institutional Support Benchmarks
2. Course Development Benchmarks

1. Teaching/Learning Benchmarks
2. Course Structure Benchmarks
3. Student Support Benchmarks
4. Faculty Support Benchmarks
5. Evaluation and Assessment Benchmarks.

On the evaluation and assessment part the following benchmarks are given:

• The program's educational effectiveness and teaching/learning process is assessed through an evaluation process that uses several methods and applies specific standards.

• Data on enrollment, costs, and successful/innovative uses of technology are used to evaluate program effectiveness.

• Intended learning outcomes are reviewed regularly to ensure clarity, utility, and appropriateness.

The Report:

This report divides the evaluation/assessment of the distributed learning into following six areas: organizational structure, student learning, course design, faculty/instructor, economic, institutional evaluation/assessment. For each one of these areas relevant questions have been stated and possible policies together with potential tools have been presented. Ideas in each section detailed as much as possible. The questions that need to be addressed are; who will be the audience of the report and what type of actions will be taken based on it. In the light of the input that will be received from the Task Force members this report will be modified and detailed.

Organizational Structure of Evaluation

.

Questions:

• Who will carry out the evaluations/assessments?

• Who will look at it?

• Who will take an action, such as cancel, expand, improve etc.?

Policy: Each unit carries out an assessment/evaluation process and report their results and communicate their findings within the system as it is explained in the Figure 1. Each unit will have the ownership of the process. Conflicts, if any, between the units will be resolved by the "University-wide Coordination Board".

Figure 1. UM Draft Plan of the Assessment of Distributed Education

Tools:

University-wide Coordination Board

Membership:

Duties:

Periodic Reports and Communications

Each program should report/communicate the following information:

1. Description of Program
2. How Program Fits to the Units Mission
3. Administrative Structure of the Program
4. Target Student Body
5. Competition
6. Collaboration, Partnerships
7. Funding

1. Marketing
2. Issues, Challenges
3. Benchmarks, Evaluation/Assessment Methods and Tools

II. Student Learning Evaluation/Assessment

.

Questions:

• Are they learning what is expected?

• What are the learning objectives, expected learning outcomes, appropriate assessment methods and tools?

• Are the students satisfied with their learning?

• What could be the evaluation/assessment tools?

Evaluation of Distance Learning Courses:

Courses offered through a distance learning option need to be evaluated just as thoroughly as on-campus courses. Most of the same issues are relevant. Additional issues that should be considered include the suitability of the delivery mechanism for the distance option, suitability of the schedule of the course including assignments and examinations, the availability of important resources (textbook, course packets, library resources, computer resources), the accessibility of the instructor for providing guidance or responding to questions, relative importance and accessibility of peer group input (threaded discussion groups, chat rooms, etc.). Open ended questions directed toward determining what was especially beneficial to the learning experience and in contrast what was perceived as being an obstacle to their learning experience should yield useful information that can be helpful in modifying and refining subsequent offerings.

Since many distance learning options are still in the development stage it is important that participants be given opportunity to respond to the suitability of the delivery mechanism early in each course to reveal possible flaws in sufficient time in order to institute corrective changes. Distance learning students are quite independent and thus may assume responsibility for overcoming delivery method shortcomings that are properly in the domain of the instructor and/or institution.

As the distance learning option becomes more widely used, complete degree or certificate programs may be accomplished without on-campus residence time. This requires a distance learning infrastructure that can deal with the broader issues of meeting degree requirements, the suitability of the distance learning experience in the major, advising, availability of financial aid, resources for finding post--graduation internships or employment, and the like. It is premature to define all of these issues at this time, but within the next five to ten years these matters will probably have to be addressed.

Integrity of the Learning Experience and the Awarding of the Credential:

Concerns have been raised about the integrity of distance learning, that is, how can an instructor determine and/or verify the identity of the person to which the educational credential is eventually awarded. With small and medium enrollment on-campus classes instructors get personally acquainted with the students. However, in large enrollment on-campus classes the same question may apply, that is, how can an instructor be certain that the person claiming the credential is actually the person who did the course work upon which that credential is based. These concerns have been largely resolved through procedures by which all critical assignments and examinations are handled by pre-approved proctors who serve as official contact persons at distant locations.

Evaluation of student performance in the distance learning option is dependent upon the quality of the assignments submitted and the demonstration of knowlege and/or ability to process knowledge as revealed by examinations. This may be in contrast to some on-campus courses in which seat time may contribute to the basis for awarding course credit.

Completion Rate for Distance Learning Students vs On-Campus Students:

When distance learning occurs asynchronously students do not experience the peer pressure that occurs in an on-campus course which may help to keep students on task even when they are faced with competing demands and non-academic counter options. A schedule imposed by a set of firm deadline dates for completion of assignments and/or examinations can provide a significant force to keep students on task and on schedule. The limited connectivity between student and instructor (as well as fellow students) and the more diverse environment in which distance students operate undoubtedly contribute to a lower course completion rate for distance students than for on-campus students.

Policy: Each unit creates assessment of student learning (see Figure 2) and student opinion of teaching evaluation plan as it is expected from traditional programs.

Tools:

Modified Student Opinion on Teaching Forms

(Existing SOT forms should be modified to include questions related with the special characteristics of the program. This will require the approval of the related committees and assemblies)

Additional questions related with the technology/delivery could be:

1. Generally, this Internet course compares favorably with regular courses I have taken.
2. The specific course requirements in this Internet course (the nature and amount of reading and writing assignments) were comparable to regular on-campus courses.
3. The interaction between faculty and student was a valuable part of the Internet course.
4. Access to needed library resources was comparable to regular on-campus courses.
5. The quality of technology in this course was comparable to other Internet resources I have used.

There are some other coming from benchmarks set by the National Education Association:

Teaching/Learning Benchmarks

6. Student interaction with faculty and other students is an essential characteristics and is facilitated through variety of ways, including voice-mail and/or e-mail.
7. Feedback to student assignments and questions is constructive and provided in timely manner.
8. Students are instructed in the proper methods of effective research, including assessment of the validity of resources.

Figure 2. Assessment of Student Learning

III. Course Design Evaluation

.

Questions:

• How do we design courses for delivery at a distance?

• What type of courses should be replicated versus re-designed?

• Do we need a standardization on the format?

• What are the minimum standards expected on course design?

I. Course Design & Support

Instructional Design:

• The University should ensure that existing programs when exported to distant sites maintain coherence and integrity. Ensuring integrity includes determining the appropriateness of the technology used in delivering the course.
• The course should be designed on the basis of the content to be delivered, the characteristics and capabilities of the audience to be served, and the proposed learning outcomes.

• The University should provide appropriate technical, design, and production support for the development of electronically offered courses.
• The University should make relevant competence requirements of authors, consultants, and others that are brought into the course design and development process.

User-Interface: Consideration should be made in the macro-design to the organization of course content as well as presentation flow and in the micro-design to issues of screen layouts, language use, graphic formats and navigational strategies.

Technology: Uses of technology should be evaluated as a part of the evaluation process to provide continuous input about effectiveness so quality related to technology can be assessed and changes and improvements can be applied as needed.

• The University should strive to assure a consistent and coherent technical framework for students and faculty. When a change in technologies is necessary, it is introduced in a way that minimizes the impact on students and faculty.
• The University should conduct planning for and support the development and use of appropriate technologies as a part of higher education's distance learning plan and work toward successful integration of multiple technologies to provide an electronic infrastructure that meets the instructional requirements of the course.
• The selection of technologies should be based on appropriateness for the students and the curriculum. It is recognized that availability, cost, and other issues are often involved, but program documentation should include specific consideration of the match between technology and program.

Faculty Support:

• University-wide faculty support services should work together to ensure coordination and collaboration in developing a comprehensive faculty support plan.
• Faculty should have access to:
• Instructional development support including training for teaching over distance learning systems, instructional design, graphics preparation, video training support, multimedia.
• Technology delivery support including distribution of hard copy media, satellite up linking, broadcast and cable TV delivery, audio conferencing, compressed video bridge, computer access.
• Faculty members should be assisted in the transition from classroom teaching to online instruction and are assessed during the process.
• Instructor training and assistance should continue through the progression of the online course.
• Faculty members should be provided with written resources to understand issues relating to distance learning.

II. Minimum Standards for Courses

Structure:

• Before starting an online program, students should be advised about the program to determine:
• If they possess the self-motivation and commitment to learn at a distance and
• If they have access to the minimal technology required by the course design.
• Students should have access to sufficient library resources.

Content:

• Instructional materials should be reviewed periodically to ensure they meet program standards.
• Courses should be designed to require students to engage in analysis, synthesis, and evaluation as part of their course requirements.
• Learning outcomes, not the availability of existing technology, should determine the technology being used to deliver course content.

Interaction:

• The importance of appropriate interaction (synchronous or asynchronous) between instructor and students and among students should be reflected in the design and be facilitated through a variety of ways.
• Feedback to student assignments and questions should be constructive and provided in a timely manner.
• The instructor should have knowledge regarding how to effectively interact with students at a distance.

Resources:

• The course should ensure that appropriate learning resources are available to students.

Technology:

• The selection of technologies should be based on appropriateness for the students and the curriculum. It is recognized that availability, cost, and other issues are often involved, but program documentation should include specific consideration of the match between technology and course.
• There should be in place a back-up plan for technology failures.
• Review and approval processes should be in place to ensure the appropriateness of the technology being used to meet the program’s objectives.

Policy: Course design including intended learning outcomes should be reviewed internally/externally regularly to ensure clarity, utility, and appropriateness.

Tools:

Internal and External Periodic Review of the Program/Courses

Peer Reviews

IV. Faculty/Instructor Evaluation

.

Questions:

• Satisfaction level of instructors using these alternative systems of delivery

• A systematic mechanism to collect the instructors' opinion and suggestions.

Policy: Each instructor should file out Instructor Course Evaluation Form regularly.

Tools:

Instructor Evaluation Form (similar to the Student Opinion of Teaching Form, but from instructor perspective).

The following are the items that are recommended to include in the faculty/instructor evaluations for distributed learning:

TIME RELATED ISSUES

Time spent learning new technologies

Preparation time required for a TEL course beyond that of a

traditional course

Preparation time required for a previously taught TEL course

Amount of time spent interacting with students using TEL

(vs. traditional course)

Amount of advanced planning required (server space, technical

coordination, etc.)

TECHNOLOGY RELATED ISSUES

Adequate training

Adequate hardware and software access and support

Availability, effectiveness and competency of technical support and consultants

Level of technology used in courses (streaming, WebCT, website, email, etc.)

Access to technology enhanced classrooms

Effectiveness of interaction with students (vs. traditional course)

Satisfaction with technology teaching methods

INFLUENCING FACTORS

Promotion/tenure - influence in decision to use technology in teaching

Support / encouragement from departments, colleges, University

Student needs or requests for distributed and asynchronous learning options

COURSE RELATED ISSUES

Size (large vs. small enrollment)

Level (lower or upper division undergraduate or graduate)

Type of course (interactive or discussion course; lecture course; lab course, etc.)

First taught as a traditional course, or first taught as a TEL course

V. Economic Evaluation/Assessment

.

Questions:

• Do they need to be profitable?

• The program versus course feasibility

• Market analysis (target groups etc.)

• What are the economic objectives, expected outcomes, appropriate evaluation methods and tools.

• What could be the evaluation/assessment tools?

Policy: Each unit decides on the "profitability" criteria. On their report they justify cost, including the impact on the regular programs), and benefit in the light of their mission.

Basic Principle: Unless designated as a part of a unique university mission, distributed learning courses or programs should reach a break-even point in a minimum of 3 years from date of first offering.

Tools: COST/PERFORMANCE ANALYSIS

• Fixed Costs
• Intellectual property development
• Instructional design
• Production/programming
• Infrastructure (equipment, facilities) establishment
• Course-Variable or Stepped Costs
• Instructor costs
• Site rental costs
• Some telecommunication costs

• Learner-Variable Costs
• Materials duplication
• Mailing costs
• Some telecommunication and software license costs
• Learner travel & salary
• Marketing
• Needs assessments
• Web development
• Advertisement
• Market pricing models for tuition/fees
• Registration

• Tuition Generation
• Grants & Contracts
• Dedicated Foundation Funds
• State Specials
• Partnership Cost Sharing

• Comparison Cost Analysis
• Written University Online vs Regular Classroom
• Private/Public Partnerships
• Inter-university Partnerships
• F & A Models Specific to Distributed Learning

VI. Institutional Evaluation/Assessment

.

Questions:

• Are the individual programs in accordance with the general mission of the University? What are the criteria for this?

• Are the programs complementing each other not competing?

• Are the resources provided to each unit equitable? (Same opportunities and support for all units?)

• Institutional cost and benefit analysis

• What could be the evaluation assessment tools?

Policy: University-wide Coordination Board (?) makes sure that general mission of the University is met and various programs are integrated, coordinated and complemented to reduce the cost and to prevent a possible competition which could affect some units negatively.

Tools:

A variety and a combination of tools should be used in the evaluation and assessment of distributed educational offerings. Many tools have been mentioned earlier in this report, and to some extent, they will be mentioned here, along with some explanation of how the data might be significant from an institutional perspective.

Demographic Information . From the institutional perspective, student demographic information is important to determine if: (1) the course/program is serving the intended clientele, (2) the support needs of the students in the course/program are any different than expected or provided, and/or (3) energy should be expended attracting students with a different demographic base. Knowledge of the student should improve the interaction and support provided to encourage completion and ultimate success.

Cost/Benefit Analysis. Analysis of program costs and the benefits of the course/program need to be evaluated by the institution to be sure that efforts and resources are not misplaced. To be fair and objective from the institutional perspective, available or alternative delivery systems should be compared on the basis of the same, or comparable, costs and benefits. Costs and benefits may be evaluated in quantitative and qualitative terms.

Student Performance Data. Several measures of student performance have been given in earlier sections of this report. Analysis can include simple completion and/or time-to-completion data or more elaborate analysis of performance. The institution must be concerned about the quality of graduates who have taken all, or part, of their program through some form of distributed learning format; graduates should be as competent in their field of study as any other graduate of the institution. Competencies should be documented with regard to course/program outcomes as well the identity of individuals being evaluated.

Program Evaluation and Assessment. The institution has an obligation to review a course/program with regard to whether it fits the mission of the institution, whether the course/program has a competitive advantage that will support its continuation, whether the course/program is "rigorous" and perceived by the public as being credible and of high quality, whether students are satisfied with the delivery system and course/program outcomes, and whether peers evaluate the course/program highly. It will be important for the institution to decide the extent to which duplication of efforts across units is acceptable. Again, qualitative and quantitative data can be used in this analysis.

As many as possible of the above program analysis elements/methods should be used to determine the appropriateness and sustainability of the institution's program and course offerings (traditional or distributed). It is possible that non-economic reasons justify continuation, but if a program or course cannot generate the income necessary to pay its costs, the institution must establish the program/course as a priority that is supported by other aspects of the institution's operations.

Distributed Education Data Book

1. Type of student served (PSEO, international, age group, reason for taking the course, and other information describing the student group)
2. Number of students served (enrolled and completed)
3. Course completion ratio (meet expectations?)
4. Average time to completion comparison data (with document of special efforts/incentives)
5. Course evaluation and assessment results (student and faculty)
6. Cost of delivery and income generated

For this tool the Task Force needs to make a decision on the value of comparative groups in the institutional evaluation/assessment of distributed learning programs. Either University of Minnesota sets the standards for itself and decides whether we do or don’t do this type of programming or fallow the standards that are set or will be set by similar institutions.

The effectiveness (cost and operational) of a collaboration should be reviewed and analyzed periodically to be sure that the collaboration is justified. At minimum, the analysis should determine whether the collaboration produces results that are possible, better, or worse than working independently. Data elements as used above should facilitate this analysis.

References

"Quality On The Line", National Education Association (NEA and Blackboard inc. study presents 24 measures of quality in internet-based distance learning)

"Statement of the Regional Accrediting Commissions on the Evaluation of Electronically Offered Degree and Certificate Programs", Draft, Regional Accrediting Commissions

"Guidelines for the Evaluation of Electronically Offered Degree and Certificate Programs", Draft, Regional Accrediting Commissions

DRAFT: Distance Education
Program Development Agreement

(Adapted from Penn State’s Distance Education Agreement)

Program Information:

College _________________________________________________________________

Academic Department _____________________________________________________

Program Name __________________________________________________________

Brief Description of Program _______________________________________________

_______________________________________________________________________

_______________________________________________________________________

Course Title/Number _____________________________________________________

Section Number ____    Number of Credits ____    Number of CEUs ____

Prerequisites ______________________________________________________

Development Plan:

Materials to be Developed _________________________________________________

_______________________________________________________________________

_______________________________________________________________________

Instructional Design Plan _________________________________________________

_______________________________________________________________________

_______________________________________________________________________

Delivery Technology:_____________________________________________________

_______________________________________________________________________

Match between Technology and Course:_______________________________________

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________

Back-up Plan for Technology Failures: ________________________________________

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________

Timeline and Key Milestones ______________________________________________

_______________________________________________________________________

_______________________________________________________________________

_______________________________________________________________________

Development Personnel:

Faculty Member(s) _______________________________________________________

Faculty Development Plan _________________________________________________

_______________________________________________________________________

_______________________________________________________________________

Departmental Approval Point _______________________________________________

College Approval Point ___________________________________________________

Distance Education Team:

Lead Designer ____________________________________________________

Additional Team Members __________________________________________

________________________________________________________________

________________________________________________________________

Lead Delivery Representative ________________________________________

Approval Point ____________________________________________________

Delivery Goal:

Target Audience _________________________________________________________

Estimated Enrollments per Offering or per Year __________________________

Targeted Market Area _____________________________________________________

Client/Receiving Organizations _____________________________________________

Delivery System Requirements _____________________________________________

Presentation Media ________________________________________________

Interaction Media _________________________________________________

Resource Media __________________________________________________

Registration System _______________________________________________

Advising ________________________________________________________

Marketing _______________________________________________________

Logistics ________________________________________________________

Student Support __________________________________________________

Faculty Support __________________________________________________

Instructor _______________________________________________________

Evaluation Plan:

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________

Expense Budget:

College Expenses ________________________________________________________

Distance Education Expenses _______________________________________________

Other Expenses __________________________________________________________

Total __________________________________________________________________

Income Budget:

Site Licenses ___________________________________________________________

Individual Tuition/Fees ___________________________________________________

Break-even Point:

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________

Net Income Sharing Arrangements:

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________

Royalty Sharing Arrangements:

Continuing and Distance Education ___________________________________%

Author _________________________________________________________%

College ________________________________________________________%

Department _____________________________________________________%

Other __________________________________________________________%

Approved:

____________________________                  __________________________
FacultyMember                                           Director,ID&D,
                                                       Distance Education

____________________________                  __________________________
Department Head                                  Director, Program Delivery

____________________________                  __________________________
Associate Dean for C&DE                           Asst Vice President for DE

______________________________
Dean, Undergraduate Education
or
Dean, Graduate School