May 14, 2008

 

To:             University of Oregon University Senate

From:         Joint Senate-Academic Affairs Committee on On-Line Course Evaluation Implementation

Subject:      Final Report

 

Introduction

In the fall of 2008, the University Senate and the Office of Academic Affairs charged this committee with the task of revising and implementing the new university-wide questions recommended by the Joint Senate-Academic Affairs Committee on Student Evaluations in 2007, addressing the format and implementation issues associated with moving to an on-line format, and making recommendations about how best to use the resulting data for both promotion and tenure decisions and distribution to students. This report represents the culmination of this process, and we anticipate that the recommendations we have included will be integrated into the course evaluation policies of the Office of Academic Affairs.

 

Two motions were discussed by the Senate as a result of the work of this committee. The first, US 07/08-7, concerning the revised university-wide questions (see below), was passed at the 3/12/08 Senate meeting. The second, US 07/08-19, concerning the wording in the Senate Legislation related to course evaluations will be considered at the 5/14/08 Senate meeting. This latter motion  revises this section of text in the Legislation to reflect the fact that the course evaluations will now be in an on-line as opposed to paper format.

 

COMPOSITION OF COMMITTEE

Paul van Donkelaar, Chair (Senate Vice-President, Human Physiology)

Kassia Dellabough (Architecture and Allied Arts)

Sue Eveland (Registrar's Office)

Raymond King (Associate Dean, Lundquist College of Business)

Bertram Malle (Psychology)

Zoe Roman (Undergraduate Student, Mathematics)

Gordon Sayre (Senate President, English)

Brad Shelton (Chair, FAC, Mathematics)

Priscilla Southwell (CAS Dean’s Office and Political Science)

Russ Tomlin (Vice-Provost for Academic Affairs)

 

 

 

RECOMMENDATIONS

 

I. REQUIRED QUESTIONS

(Approved in motion US 07/08-7)

 

Student evaluations play an important role both for improving teaching and in providing an opportunity for students to contribute to information included in the yearly evaluation of faculty. The 2006-07 Joint Senate-Academic Affairs Committee on Student Evaluations recommended that the four required university-wide questions currently in use be replaced with a more comprehensive set of questions that would more accurately and consistently probe students' evaluations of course and instructor performance. In their report, the committee outlined a new set of questions that the current committee has subsequently slightly revised and vetted by each college/unit and various faculty governance and administrative units.

 

The proposed new required university-wide questions will have a 7-2-3 structure. In particular, the first 7 quantitative questions plus the subsequent 2 qualitative questions will be the only university-wide questions used in the evaluation of faculty in yearly reviews and promotion & tenure cases. The remaining 3 questions related to course attendance, workload, and expected grade will allow instructors to better interpret the patterns of the main evaluation questions. It is important to note that these 3 questions will not be used to evaluate faculty. Finally, students will have access to responses to all of the first 7 quantitative questions. Additional questions beyond those contained in the 7-2-3 structure are the responsibility of individual departments/units.

 

The new questions are as follows:

 

Please share with us your basic perceptions of the course: 

 

1. What was the quality of this course?            

            Exceptional | Good | Adequate | Somewhat inadequate | Unsatisfactory

 

2. What was the quality of the instructor’s teaching?  

            Exceptional | Good | Adequate | Somewhat inadequate | Unsatisfactory 

3. How well organized was this course?    

            Exceptional | Good | Adequate | Somewhat inadequate | Unsatisfactory

 

4. How effective was the instructor’s use of class time?            

            Exceptional | Good | Adequate | Somewhat inadequate | Unsatisfactory

 

5. How available was the instructor for communication outside of class?             

            Exceptional | Good | Adequate | Somewhat inadequate | Unsatisfactory

 

6. How clear were the guidelines for evaluating students work in this course?     

            Exceptional | Good | Adequate | Somewhat inadequate | Unsatisfactory

 

7. The amount that I learned in this course was:              

            Exceptional | Good | Adequate | Somewhat inadequate | Unsatisfactory

 

Please share with us your thoughts on the course:

 

1.   Please comment on the instructor’s strengths and areas for possible improvement.

 

2.   Please comment on the strengths and areas of possible improvement for the course as a whole.

 

Please tell us a little bit about yourself:

 

1. How often did you attend class?  

       90-100%   

       75-90%   

       50-75%   

       25 to 50%   

       Less than 25%   

       Does not apply

 

2. How many hours per week did you spend on this course, other than time in class? 

       More than 10   

       8 to 10    

       6 to 8    

       4 to 6    

       2 to 4    

       Less than 2

 

3. What grade do you expect in this course?   

       A

       B     

       C or P     

       D     

       F or N

 

 

 

II. Revised senate legislation

 

Motion US07/08-19 – Amend US07/08-7 regarding required course evaluations to include explanatory text on the uses of the evaluations

 

Sponsored by: Joint Senate-Academic Affairs Committee on On-Line Course Evaluation Implementation

 

For Senate Action: May 14, 2008

 

Moved to amend US07/08-7, passed March 12, 2008 to replace the Preamble and include the following text (new text shown in italics) to explain implementation and use of the required student course evaluations questions as follows:

 

Student Evaluation of Teaching and Learning

Preamble:

This legislation defines expectations regarding student course evaluations at the University of Oregon, especially as they relate to annual faculty reviews and the promotion, tenure and post-tenure review process.

Certain aspects of teaching, such as the ability to create a positive learning environment, are appropriately and necessarily assessed through student evaluations. Evaluations provided by students can be effectively used by all faculty to gain insight into their teaching, and to identify ways to improve their classroom performance. Evaluative data provided by students include responses to both quantitative and qualitative questions.

 

It is important to emphasize that student evaluation of teaching is not the sole means by which faculty teaching performance is assessed. In particular, peer teaching evaluations are also a required component of every promotion & tenure case.

I. Student Evaluation of Teaching

1. The on-line questionnaire system will be used to evaluate all courses with 5 or more students.

2. The following university-wide questions will be included at the beginning of the evaluation form:

 

(*13 questions given in Section I inserted here)

 

 

a.    Data from the first 7 questions are to be made centrally available to students.

 

b.    Departments may include additional questions beyond these as they see fit.

 

3. Students shall be clearly informed, either verbally or through instructions on the on-line questionnaire, that results of their evaluation play an important role in faculty development, in future promotion and tenure decisions and in post-tenure reviews.

 

4. For the qualitative questions, the on-line forms must indicate that only electronically signed evaluations may be used in promotion/tenure and post-tenure reviews. (ORS 351.065 (f) (g)). In addition, the forms shall clearly state that the faculty member responsible for the course will have access to the written comments, but only after the grades for the course have been submitted.

 

5. A standard course evaluation report shall include the distribution of responses and the mean scores for each of the 7 quantitative university-wide questions. For comparison, the distribution of responses and mean scores from 1) classes of a similar size within the instructor's department; 2) classes of the same level within the instructor's department; and 3) all classes within the instructor's department will also be provided.

II. Procedure for Administration and Use of Student Evaluations.

1. All on-line course evaluations are to be conducted during dead week. Students will not have access to the system prior to or after this period.

2. After grades have been submitted, the faculty member shall be given access to both the quantitative and qualitative evaluations.

3. The department archives the standard course evaluation report and the qualitative evaluations in the permanent personnel file of the faculty member for use in future faculty evaluations.

4. Quantitative evaluations should be analyzed using valid statistical measures and the most relevant comparator groups. Review of the written evaluations should be conducted by a comprehensive reading of the comments.

 

 

III. Use of Data

In addition to revising and implementing the new university-wide questions, a 2^nd main focus of our work was related to how to use the resulting data. The new on-line system allows the possibility for streamlined analysis and comparisons across a number of dimensions. In this section we first outline how different stakeholders will interact with the data provided by the on-line system and then provide some recommendations for potential future uses.

 

Instructors: We propose that each instructor for courses with 5 or more students enrolled be provided with a standard course evaluation report which contains the distribution of responses and the mean scores for each of the 7 quantitative university-wide questions as well as the responses from the qualitative questions. As a basis for comparison, the distribution of responses and mean scores for the 7 quantitative questions will also be provided for courses within the same department during the term in question with 1) the same approximate number of students (number ranges: 5-19, 20-49, 50-100, 100+); 2) the same course level (i.e., 100, 200, 300, 400/500, 600 level courses); and 3) all the courses within the department for that term. The report will provide the number of courses that fall into each of these categories.

 

Because the mean scores represent a mapping of the ordinal response options onto a numerical scale, it is important to interpret the mean scores not in isolation, but with respect to this mapping. In addition, because this mapping is occurring across the relevant comparators outlined in the preceding paragraph, the key aspect to interpreting teaching performance using this tool is to directly assess the mean and distributions of responses for the instructor relative to those from the comparators.

 

Together these sets of numbers will provide an overall sense of how well the instructor performed within a specific course relative to other departmental courses of a similar size or level as well as the department as a whole. For each of the 7 university-wide quantitative questions a table will be generated to provide this information:

 

	Exceptional	Good	Adequate	Somewhat inadequate	Unsatisfactory	Mean	Number of classes
Instructor
Same size
Same level
Department

 

 

In addition, the responses on the final 3 questions of the university-wide forms related to class attendance, expected grade, and course workload can be used by the instructor to gauge how these variables affected responses on the remaining questions. The responses to these questions will not be used in the evaluation of faculty.

 

Departments: The standard reports for each course taught by an individual instructor shall be kept in the permanent personnel file of that individual. These reports will be used in the evaluation of faculty for yearly reviews, merit increase reports, and tenure and promotion cases. Departments are urged to examine trends in the evaluation scores across several years to determine changes in performance across time.

 

Responses to additional questions provided by the Department should also be archived in the permanent personnel file of the individual and may also be used in departmental evaluations of teaching performance.

 

Colleges/Units: Course evaluation responses will be used at the college/unit level as part of the process by which the performance of faculty is examined. The standard reports for the faculty member from the appropriate period of time will be used for this purpose. Thus, every tenure & promotion case will include the standard reports for the 7 university-wide questions. The decision to include additional material rests with each individual department.

 

Students: Students will have access via DuckWeb to the mean responses to the first 7 quantitative questions for each instructor. The data from each term should be made available in a timely manner. In addition, historical data from previous years should also be available at this site.

 

Future Additional Data Analyses: In the future, the vendor providing the on-line course evaluation service should provide the University with a “data dump” at the end of each term to allow in depth analysis of university-wide teaching performance. In addition, this will allow evaluations to be compared across similar departments. When this utility becomes available, an important issue to consider will be the levels of accessibility relative to the administrative level of the user.