College of Biological Sciences
College of Biological Sciences
Assistant Vice Chancellor,
University of Minnesota Duluth
College of Education and Human Development
Instructional Technology Fellowship Coordinator,
College of Education & Human Development
University of Minnesota Morris
TABLE OF CONTENTS
I. Organizational Structure of Evaluation/Assessment
II. Student Learning Evaluation/Assessment
III. Course Design Evaluation
IV. Faculty/Instructor Evaluation
V. Economic Evaluation/Assessment
VI. Institutional Evaluation/Assessment
How to measure impact, market, learning outcomes, and learner satisfaction and will discuss such issues as the development of appropriate outcome measures. Also, competitive advantage needs to be considered. (Look at how to measure impact, market, learning outcomes & learning satisfaction. Include development of appropriate outcome measures.)
Overview of Resources:
From NCA Handbook of Accreditation
Guidelines for Distance Education: Any institution offering distance education is expected to meet the requirements of its own regional accrediting body, and be guided by the Western Interstate Commission for Higher Education (WICHE) Principles.
Evaluation and Assessment
The institution assesses student capability to succeed in distance education programs and applies this information to admission and recruiting policies and decisions.
The institution evaluates the educational effectiveness of its distance education programs (including assessments of student learning outcomes, student retention, and student satisfaction) to ensure comparability to campus-bases programs.
The institution ensures the integrity of student work and the credibility of the degrees and credits it awards.
From Regional Accrediting Commissions Draft Guidelines for the Evaluation of Electronically Offered Degree and Certificate Programs
The Guidelines are divided into the following five separate components, each of which addresses a particular area of institutional activity relevant to distance education. They are:
On the Evaluation and Assessment draft report says: "Both the assessment of student achievement and evaluation of the overall program take on added importance as new techniques evolve. For example, in asynchronous programs the element of seat time is essentially removed from the equation. For these reasons, the institution conducts sustained, evidence-based and participatory inquiry as to whether distance learning programs are achieving objectives. The results of such inquiry are used to guide curriculum design and delivery, pedagogy, and educational processes, and ,may affect future policy and budgets perhaps have implications for the institution's roles and mission."
From National Education Association and Blackboard Inc. Benchmark for Internet Based Distance Learning Study
The benchmarks distilled from this study are divided into seven categories of quality measures currently in use on campuses around the nation. Many are common sense, but the study validates their importance. The categories and benchmarks include:
On the evaluation and assessment part the following benchmarks are given:
The program's educational effectiveness and teaching/learning process is assessed through an evaluation process that uses several methods and applies specific standards.
Data on enrollment, costs, and successful/innovative uses of technology are used to evaluate program effectiveness.
Intended learning outcomes are reviewed regularly to ensure clarity, utility, and appropriateness.
This report divides the evaluation/assessment of the distributed learning into following six areas: organizational structure, student learning, course design, faculty/instructor, economic, institutional evaluation/assessment. For each one of these areas relevant questions have been stated and possible policies together with potential tools have been presented. Ideas in each section detailed as much as possible. The questions that need to be addressed are; who will be the audience of the report and what type of actions will be taken based on it. In the light of the input that will be received from the Task Force members this report will be modified and detailed.
Who will carry out the evaluations/assessments?
Who will look at it?
Who will take an action, such as cancel, expand, improve etc.?
Policy: Each unit carries out an assessment/evaluation process and report their results and communicate their findings within the system as it is explained in the Figure 1. Each unit will have the ownership of the process. Conflicts, if any, between the units will be resolved by the "University-wide Coordination Board".
Figure 1. UM Draft Plan of the Assessment of Distributed Education
University-wide Coordination Board
Periodic Reports and Communications
Each program should report/communicate the following information:
Are they learning what is expected?
What are the learning objectives, expected learning outcomes, appropriate assessment methods and tools?
Are the students satisfied with their learning?
What could be the evaluation/assessment tools?
Evaluation of Distance Learning Courses:
Courses offered through a distance learning option need to be evaluated just as thoroughly as on-campus courses. Most of the same issues are relevant. Additional issues that should be considered include the suitability of the delivery mechanism for the distance option, suitability of the schedule of the course including assignments and examinations, the availability of important resources (textbook, course packets, library resources, computer resources), the accessibility of the instructor for providing guidance or responding to questions, relative importance and accessibility of peer group input (threaded discussion groups, chat rooms, etc.). Open ended questions directed toward determining what was especially beneficial to the learning experience and in contrast what was perceived as being an obstacle to their learning experience should yield useful information that can be helpful in modifying and refining subsequent offerings.
Since many distance learning options are still in the development stage it is important that participants be given opportunity to respond to the suitability of the delivery mechanism early in each course to reveal possible flaws in sufficient time in order to institute corrective changes. Distance learning students are quite independent and thus may assume responsibility for overcoming delivery method shortcomings that are properly in the domain of the instructor and/or institution.
As the distance learning option becomes more widely used, complete degree or certificate programs may be accomplished without on-campus residence time. This requires a distance learning infrastructure that can deal with the broader issues of meeting degree requirements, the suitability of the distance learning experience in the major, advising, availability of financial aid, resources for finding post--graduation internships or employment, and the like. It is premature to define all of these issues at this time, but within the next five to ten years these matters will probably have to be addressed.
Integrity of the Learning Experience and the Awarding of the Credential:
Concerns have been raised about the integrity of distance learning, that is, how can an instructor determine and/or verify the identity of the person to which the educational credential is eventually awarded. With small and medium enrollment on-campus classes instructors get personally acquainted with the students. However, in large enrollment on-campus classes the same question may apply, that is, how can an instructor be certain that the person claiming the credential is actually the person who did the course work upon which that credential is based. These concerns have been largely resolved through procedures by which all critical assignments and examinations are handled by pre-approved proctors who serve as official contact persons at distant locations.
Evaluation of student performance in the distance learning option is dependent upon the quality of the assignments submitted and the demonstration of knowlege and/or ability to process knowledge as revealed by examinations. This may be in contrast to some on-campus courses in which seat time may contribute to the basis for awarding course credit.
Completion Rate for Distance Learning Students vs On-Campus Students:
When distance learning occurs asynchronously students do not experience the peer pressure that occurs in an on-campus course which may help to keep students on task even when they are faced with competing demands and non-academic counter options. A schedule imposed by a set of firm deadline dates for completion of assignments and/or examinations can provide a significant force to keep students on task and on schedule. The limited connectivity between student and instructor (as well as fellow students) and the more diverse environment in which distance students operate undoubtedly contribute to a lower course completion rate for distance students than for on-campus students.
Policy: Each unit creates assessment of student learning (see Figure 2) and student opinion of teaching evaluation plan as it is expected from traditional programs.
Modified Student Opinion on Teaching Forms
(Existing SOT forms should be modified to include questions related with the special characteristics of the program. This will require the approval of the related committees and assemblies)
Additional questions related with the technology/delivery could be:
There are some other coming from benchmarks set by the National Education Association:
Figure 2. Assessment of Student Learning
How do we design courses for delivery at a distance?
What type of courses should be replicated versus re-designed?
Do we need a standardization on the format?
What are the minimum standards expected on course design?
I. Course Design & Support
User-Interface: Consideration should be made in the macro-design to the organization of course content as well as presentation flow and in the micro-design to issues of screen layouts, language use, graphic formats and navigational strategies.
Technology: Uses of technology should be evaluated as a part of the evaluation process to provide continuous input about effectiveness so quality related to technology can be assessed and changes and improvements can be applied as needed.
II. Minimum Standards for Courses
Policy: Course design including intended learning outcomes should be reviewed internally/externally regularly to ensure clarity, utility, and appropriateness.
Internal and External Periodic Review of the Program/Courses
Satisfaction level of instructors using these alternative systems of delivery
A systematic mechanism to collect the instructors' opinion and suggestions.
Policy: Each instructor should file out Instructor Course Evaluation Form regularly.
Instructor Evaluation Form (similar to the Student Opinion of Teaching Form, but from instructor perspective).
The following are the items that are recommended to include in the faculty/instructor evaluations for distributed learning:
Time spent learning new technologies
Preparation time required for a TEL course beyond that of a
Preparation time required for a previously taught TEL course
Amount of time spent interacting with students using TEL
(vs. traditional course)
Amount of advanced planning required (server space, technical
Adequate hardware and software access and support
Availability, effectiveness and competency of technical support and consultants
Level of technology used in courses (streaming, WebCT, website, email, etc.)
Access to technology enhanced classrooms
Effectiveness of interaction with students (vs. traditional course)
Satisfaction with technology teaching methods
Promotion/tenure - influence in decision to use technology in teaching
Support / encouragement from departments, colleges, University
Student needs or requests for distributed and asynchronous learning options
Size (large vs. small enrollment)
Level (lower or upper division undergraduate or graduate)
Type of course (interactive or discussion course; lecture course; lab course, etc.)
First taught as a traditional course, or first taught as a TEL course
Do they need to be profitable?
The program versus course feasibility
Market analysis (target groups etc.)
What are the economic objectives, expected outcomes, appropriate evaluation methods and tools.
What could be the evaluation/assessment tools?
Policy: Each unit decides on the "profitability" criteria. On their report they justify cost, including the impact on the regular programs), and benefit in the light of their mission.
Basic Principle: Unless designated as a part of a unique university mission, distributed learning courses or programs should reach a break-even point in a minimum of 3 years from date of first offering.
Tools: COST/PERFORMANCE ANALYSIS
Measure Of Financial Performance And Cost Recovery
Are the individual programs in accordance with the general mission of the University? What are the criteria for this?
Are the programs complementing each other not competing?
Are the resources provided to each unit equitable? (Same opportunities and support for all units?)
Institutional cost and benefit analysis
What could be the evaluation assessment tools?
Policy: University-wide Coordination Board (?) makes sure that general mission of the University is met and various programs are integrated, coordinated and complemented to reduce the cost and to prevent a possible competition which could affect some units negatively.
A variety and a combination of tools should be used in the evaluation and assessment of distributed educational offerings. Many tools have been mentioned earlier in this report, and to some extent, they will be mentioned here, along with some explanation of how the data might be significant from an institutional perspective.
Demographic Information . From the institutional perspective, student demographic information is important to determine if: (1) the course/program is serving the intended clientele, (2) the support needs of the students in the course/program are any different than expected or provided, and/or (3) energy should be expended attracting students with a different demographic base. Knowledge of the student should improve the interaction and support provided to encourage completion and ultimate success.
Cost/Benefit Analysis. Analysis of program costs and the benefits of the course/program need to be evaluated by the institution to be sure that efforts and resources are not misplaced. To be fair and objective from the institutional perspective, available or alternative delivery systems should be compared on the basis of the same, or comparable, costs and benefits. Costs and benefits may be evaluated in quantitative and qualitative terms.
Student Performance Data. Several measures of student performance have been given in earlier sections of this report. Analysis can include simple completion and/or time-to-completion data or more elaborate analysis of performance. The institution must be concerned about the quality of graduates who have taken all, or part, of their program through some form of distributed learning format; graduates should be as competent in their field of study as any other graduate of the institution. Competencies should be documented with regard to course/program outcomes as well the identity of individuals being evaluated.
Program Evaluation and Assessment. The institution has an obligation to review a course/program with regard to whether it fits the mission of the institution, whether the course/program has a competitive advantage that will support its continuation, whether the course/program is "rigorous" and perceived by the public as being credible and of high quality, whether students are satisfied with the delivery system and course/program outcomes, and whether peers evaluate the course/program highly. It will be important for the institution to decide the extent to which duplication of efforts across units is acceptable. Again, qualitative and quantitative data can be used in this analysis.
As many as possible of the above program analysis elements/methods should be used to determine the appropriateness and sustainability of the institution's program and course offerings (traditional or distributed). It is possible that non-economic reasons justify continuation, but if a program or course cannot generate the income necessary to pay its costs, the institution must establish the program/course as a priority that is supported by other aspects of the institution's operations.
Distributed Education Data Book
Formation of Comparative Groups
For this tool the Task Force needs to make a decision on the value of comparative groups in the institutional evaluation/assessment of distributed learning programs. Either University of Minnesota sets the standards for itself and decides whether we do or dont do this type of programming or fallow the standards that are set or will be set by similar institutions.
Evaluation of Collaboration
The effectiveness (cost and operational) of a collaboration should be reviewed and analyzed periodically to be sure that the collaboration is justified. At minimum, the analysis should determine whether the collaboration produces results that are possible, better, or worse than working independently. Data elements as used above should facilitate this analysis.
"Quality On The Line", National Education Association (NEA and Blackboard inc. study presents 24 measures of quality in internet-based distance learning)
"Statement of the Regional Accrediting Commissions on the Evaluation of Electronically Offered Degree and Certificate Programs", Draft, Regional Accrediting Commissions
"Guidelines for the Evaluation of Electronically Offered Degree and Certificate Programs", Draft, Regional Accrediting Commissions
DRAFT: Distance Education
Program Development Agreement
(Adapted from Penn States Distance Education Agreement)
Academic Department _____________________________________________________
Program Name __________________________________________________________
Brief Description of Program _______________________________________________
Course Title/Number _____________________________________________________
Section Number ____ Number of Credits ____ Number of CEUs ____
Materials to be Developed _________________________________________________
Instructional Design Plan _________________________________________________
Match between Technology and Course:_______________________________________
Back-up Plan for Technology Failures: ________________________________________
Timeline and Key Milestones ______________________________________________
Faculty Member(s) _______________________________________________________
Faculty Development Plan _________________________________________________
Departmental Approval Point _______________________________________________
College Approval Point ___________________________________________________
Distance Education Team:
Lead Designer ____________________________________________________
Additional Team Members __________________________________________
Lead Delivery Representative ________________________________________
Approval Point ____________________________________________________
Target Audience _________________________________________________________
Estimated Enrollments per Offering or per Year __________________________
Targeted Market Area _____________________________________________________
Client/Receiving Organizations _____________________________________________
Delivery System Requirements _____________________________________________
Presentation Media ________________________________________________
Interaction Media _________________________________________________
Resource Media __________________________________________________
Registration System _______________________________________________
Student Support __________________________________________________
Faculty Support __________________________________________________
College Expenses ________________________________________________________
Distance Education Expenses _______________________________________________
Other Expenses __________________________________________________________
Site Licenses ___________________________________________________________
Individual Tuition/Fees ___________________________________________________
Net Income Sharing Arrangements:
Royalty Sharing Arrangements:
Continuing and Distance Education ___________________________________%
Department Head Director, Program Delivery
Associate Dean for C&DE Asst Vice President for DE
Dean, Undergraduate Education
Dean, Graduate School