Student Ratings of Instruction: A Literature Review

Committee on Teaching Presentation

As Co-Chair of the Committee on Teaching's Subcommittee on Teaching and Course Evaluations, I performed a review of the research literature on student ratings of instruction, and the screencast below captures a summary of that literature review. Please feel free to jump ahead to the portions of the presentation that are of most interest to you, using the time indices below the video. If you have further questions, you can e-mail me or consult the bibliography at the bottom of this page.

Research-Validated Structures
33:26 - 40:32

Research-Validated Administration
40:32 - 41:48

Research-Validated Analysis
41:48 - 44:11

Appropriate Use
44:11 - 46:30

Conclusion
46:30 - 46:51  

Introduction
00:00 - 02:06

Research Questions
02:06 - 04:37

Reliability
04:37 - 07:40

Validity
07:40 - 19:02  

Biases
19:02 - 32:12    
    
Relevance?
32:12 - 33:26


Selected Bibliography

Below is a selection of peer-reviewed articles on student ratings, organized according to common areas of interest. To help guide your reading, we've indicated especially important (well-designed and/or influential) works with a double asterisk prior to the first author's name. As there are thousands of publications on student ratings, it is important to emphasize that what follows is not meant to be comprehensive.

General Introductions, Literature Reviews, and Meta-Analyses

  1. **Benton, Stephen, and Cashin, William E. Student Ratings of Teaching: A Summary of Research and Literature. IDEA Paper. IDEA Center, 2012.
  2. Cohen, Peter A. “Effectiveness of Student-Rating Feedback for Improving College Instruction: A Meta-Analysis of Findings.” Research in Higher Education13, no. 4 (January 1, 1980): 321–41.
  3. **Cohen, Peter A. “Student Ratings of Instruction and Student Achievement: A Meta-Analysis of Multisection Validity Studies.” Review of Educational Research 51, no. 3 (October 1, 1981): 281–309.
  4. Costin, Frank, William T. Greenough, and Robert J. Menges. “Student Ratings of College Teaching: Reliability, Validity, and Usefulness.” Review of Educational Research 41, no. 5 (December 1, 1971): 511–35.
  5. Falchikov, Nancy, and David Boud. “Student Self-Assessment in Higher Education: A Meta-Analysis.” Review of Educational Research 59, no. 4 (December 1, 1989): 395–430.
  6. Marsh, Herbert W. “Students’ Evaluations of University Teaching: Dimensionality, Reliability, Validity, Potential Baises, and Utility.” Journal of Educational Psychology 76, no. 5 (1984): 707–54.
  7. **Marsh, Herbert W. “Students’ Evaluations of University Teaching: Research Findings, Methodological Issues, and Directions for Future Research.”International Journal of Educational Research 11, no. 3 (1987): 253–388.
  8. Ory, John C., and Katherine Ryan. “How Do Student Ratings Measure Up to a New Validity Framework?” New Directions for Institutional Research 2001, no. 109 (March 1, 2001): 27–44.
  9. Spooren, Pieter, Bert Brockx, and Dimitri Mortelmans. “On the Validity of Student Evaluation of Teaching The State of the Art.” Review of Educational Research 83, no. 4 (December 1, 2013): 598–642.
  10. **Theall, Michael, Philip C. Abrami, and Lisa Mets A. The Student Ratings Debate: Are They Valid? How Can We Best  Use Them?. Directions for Institutional Research 109. San Francisco: Jossey-Bass, 2001.

Relationship Between Student Ratings and Student Achievement

  1. **Centra, John A. “Student Ratings of Instruction and Their Relationship to Student Learning.” American Educational Research Journal 14, no. 1 (January 1, 1977): 17–24.
  2. **Feldman, Kenneth A. “The Association between Student Ratings of Specific Instructional Dimensions and Student Achievement: Refining and Extending the Synthesis of Data from Multisection Validity Studies.” Research in Higher Education 30, no. 6 (December 1, 1989): 583–645.
  3. Galbraith, Craig S., Gregory B. Merrill, and Doug M. Kline. “Are Student Evaluations of Teaching Effectiveness Valid for Measuring Student Learning Outcomes in Business Related Classes? A Neural Network and Bayesian Analyses.” Research in Higher Education 53, no. 3 (May 1, 2012): 353–74.
  4. Rodin, Miriam, and Burton Rodin. “Student Evaluations of Teachers.” Science, New Series, 177, no. 4055 (September 29, 1972): 1164–66.
  5. Stehle, Sebastian, Birgit Spinath, and Martina Kadmon. “Measuring Teaching Effectiveness: Correspondence Between Students’ Evaluations of Teaching and Different Measures of Student Learning.” Research in Higher Education 53, no. 8 (December 1, 2012): 888–904.

Relationship Between Student Ratings and Course Grades/Workload

  1. Abrami, Philip C., Wenda J. Dickens, Raymond P. Perry, and Les Leventhal. “Do Teacher Standards for Assigning Grades Affect Student Evaluations of Instruction?” Journal of Educational Psychology 72, no. 1 (1980): 107–18.
  2. **Centra, John A. “Will Teachers Receive Higher Student Evaluations by Giving Higher Grades and Less Course Work?” Research in Higher Education 44, no. 5 (October 1, 2003): 495–518.
  3. Eiszler, Charles F. “College Students’ Evaluations of Teaching and Grade Inflation.” Research in Higher Education 43, no. 4 (August 1, 2002): 483–501.
  4. **Feldman, Kenneth A. “Grades and College Students’ Evaluations of Their Courses and Teachers.” Research in Higher Education 4, no. 1 (January 1, 1976): 69–111.
  5. Greenwald, Anthony G., and Gerald M. Gillmore. “Grading Leniency Is a Removable Contaminant of Student Ratings.” American Psychologist 52, no. 11 (1997): 1209–17.
  6. **Greenwald, Anthony G., and Gerald M. Gillmore. “No Pain, No Gain? The Importance of Measuring Course Workload in Student Ratings of Instruction.”Journal of Educational Psychology 89, no. 4 (1997): 743–51.
  7. Marsh, Herbert W., and Lawrence A. Roche. “Effects of Grading Leniency and Low Workload on Students’ Evaluations of Teaching: Popular Myth, Bias, Validity, or Innocent Bystanders?” Journal of Educational Psychology 92, no. 1 (March 2000): 202–28.
  8. **Marsh, Herbert W. “Distinguishing Between Good (Useful) and Bad Workloads on Students’ Evaluations of Teaching.” American Educational Research Journal 38, no. 1 (March 20, 2001): 183–212.

Relationship Between Student Ratings and Faculty Expressiveness, Attractiveness, and/or Personality

  1. **Abrami, Philip C., Les Leventhal, and Raymond P. Perry. “Educational Seduction.” Review of Educational Research 52, no. 3 (October 1, 1982): 446–64.
  2. Ambady, Nalini, and Robert Rosenthal. “Half a Minute: Predicting Teacher Evaluations from Thin Slices of Nonverbal Behavior and Physical Attractiveness.”Journal of Personality and Social Psychology 64, no. 3 (1993): 431–41.
  3. Feldman, Kenneth A. “The Perceived Instructional Effectiveness of College Teachers as Related to Their Personality and Attitudinal Characteristics: A Review and Synthesis.” Research in Higher Education 24, no. 2 (January 1, 1986): 139–213.
  4. **Marsh, Herbert W., and John E. Ware. “Effects of Expressiveness, Content Coverage, and Incentive on Multidimensional Student Rating Scales: New Interpretations of the Dr. Fox Effect.” Journal of Educational Psychology 74, no. 1 (February 1982): 126–34.
  5. **Peer, Eyal, and Elisha Babad. “The Doctor Fox Research (1973) Rerevisited: ‘Educational Seduction’ Ruled Out.” Journal of Educational Psychology 106, no. 1 (February 2014): 36–45.
  6. **Ware, J. E., and R. G. Williams. “The Dr. Fox Effect: A Study of Lecturer Effectiveness and Ratings of Instruction.” Journal of Medical Education 50, no. 2 (February 1975): 149–56.

Relationship Between Student Ratings and Class Size, Discipline, Gender, and Other Potential Biases

  1. Abrami, Philip C., and Deborah A. Mizener. “Does the Attitude Similarity of College Professors and Their Students Produce ‘Bias’ in Course Evaluations?.”American Educational Research Journal 20, no. 1 (April 1, 1983): 123–36.
  2. Feldman, Kenneth A. “Course Characteristics and College Students’ Ratings of Their Teachers: What We Know and What We Don’t.” Research in Higher Education 9, no. 3 (January 1, 1978): 199–242.
  3. Feldman, Kenneth A. “The Significance of Circumstances for College Students’ Ratings of Their Teachers and Courses.” Research in Higher Education 10, no. 2 (January 1, 1979): 149–72.
  4. Feldman, Kenneth A. “Class Size and College Students’ Evaluations of Teachers and Courses: A Closer Look.” Research in Higher Education 21, no. 1 (January 1, 1984): 45–116.
  5. Feldman, Kenneth A. “College Students’ Views of Male and Female College Teachers: Part I: Evidence from the Social Laboratory and Experiments.” Research in Higher Education 33, no. 3 (June 1, 1992): 317–75.
  6. **Feldman, Kenneth A. “College Students’ Views of Male and Female College Teachers: Part II: Evidence from Students’ Evaluations of Their Classroom Teachers.” Research in Higher Education 34, no. 2 (April 1, 1993): 151–211.
  7. Kember, David, and Doris Y. P. Leung. “Disciplinary Differences in Student Ratings of Teaching Quality.” Research in Higher Education 52, no. 3 (May 1, 2011): 278–99.
  8. **Theall, Michael, and Jennifer Franklin. “Looking for Bias in All the Wrong Places: A Search for Truth or a Witch Hunt in Student Ratings of Instruction?” InThe Student Ratings Debate: Are They Valid? How Can We Best Use Them?, edited by Philip C. Abrami, Lisa Mets A., and Michael Theall, 45–58. San Francisco: Jossey-Bass, 2001.

Global Items and the Dimensionality of Student Ratings

  1. Berk, Ronald. “Should Global Items on Student Rating Scales Be Used for Summative Decisions?” The Journal of Faculty Development 27, no. 1 (January 1, 2013): 63–68.
  2. Cashin, William E., Ronald G. Downey, and Glenn R. Sixbury. “Global and Specific Ratings of Teaching Effectiveness and Their Relation to Course Objectives: Reply to Marsh (1994).” Journal of Educational Psychology 86, no. 4 (1994): 649–57.
  3. Cashin, William E., and Ronald G. Downey. “Using Global Student Rating Items for Summative Evaluation.” Journal of Educational Psychology 84, no. 4 (1992): 563–72. 
  4. Marsh, Herbert W., and Michael J. Dunkin. “Students’ Evaluations of University Teaching: A Multidimensional Perspective.” In Effective Teaching in Higher Education: Research and Practice, 241–320. Bronx, NY: Agathon Press, 1997.
  5. Marsh, Herbert W., and Dennis Hocevar. “The Factorial Invariance of Student Evaluations of College Teaching.” American Educational Research Journal21, no. 2 (July 1, 1984): 341–66. 
  6. Marsh, Herbert W. “Validity of Students’ Evaluations of College Teaching: A Multitrait–multimethod Analysis.” Journal of Educational Psychology 74, no. 2 (April 1982): 264–79. 
  7. Marsh, Herbert W. “Multidimensional Ratings of Teaching Effectiveness by Students from Different Academic Settings and Their Relation to Student/course/instructor Characteristics.” Journal of Educational Psychology 75, no. 1 (1983): 150–66. 
  8. Murray, Harry G. “Low-Inference Classroom Teaching Behaviors and Student Ratings of College Teaching Effectiveness.” Journal of Educational Psychology 75, no. 1 (1983): 138–49. 

Reliability of Student Ratings

  1. Feldman, Kenneth A. “Consistency and Variability among College Students in Rating Their Teachers and Courses: A Review and Analysis.” Research in Higher Education 6, no. 3 (January 1, 1977): 223–74.
  2. Overall, J. U., and Herbert W. Marsh. “Students’ Evaluations of Instruction: A Longitudinal Study of Their Stability.” Journal of Educational Psychology 72, no. 3 (June 1980): 321–25.

Use of Student Ratings in Evaluation of Teaching

  1. **Abrami, Philip C. “How Should We Use Student Ratings to Evaluate Teaching?” Research in Higher Education 30, no. 2 (April 1, 1989): 221–27.
  2. **Arreola, Raoul A. Developing a Comprehensive Faculty Evaluation System: A Handbook for College Faculty and Administrators on Designing and Operating a Comprehensive Faculty Evaluation System. 2 edition. Bolton, MA: Anker Pub Co, 2000.
  3. Berk, Ronald. “Should Student Outcomes Be Used to Evaluate Teaching?” The Journal of Faculty Development 28, no. 2 (May 1, 2014): 87–96.
  4. Berk, Ronald A. Top 10 Flashpoints in Student Ratings and the Evaluation of Teaching: What Faculty and Administrators Must Know to Protect Themselves in Employment Decisions. Sterling, Virginia: Stylus, 2013.
  5. Franklin, Jennifer. “Interpreting the Numbers: Using a Narrative to Help Others Read Student Evaluations of Your Teaching Accurately.” New Directions for Teaching and Learning 2001, no. 87 (September 1, 2001): 85–100.
  6. Kite, Mary E. Effective Evaluation of Teaching: A Guide for Faculty and Administrators. Society for the Teaching of Psychology, 2012.

Alternative Measures of Teaching Effectiveness

  1. Berk, Ronald A. “Beyond Student Ratings: Peer Observation of Classroom and Clinical Teaching.” International Journal of Nursing Education Scholarship1, no. 1 (2004): 1–26.
  2. Centra, John A. “Colleagues as Raters of Classroom Instruction.” The Journal of Higher Education 46, no. 3 (May 1, 1975): 327–37.
  3. **Feldman, Kenneth A. “Instructional Effectiveness of College Teachers as Judged by Teachers Themselves, Current and Former Students, Colleagues, Administrators, and External (Neutral) Observers.” Research in Higher Education 30, no. 2 (April 1, 1989): 137–94.
  4. **Marsh, Herbert W., J. U. Overall, and Steven P. Kesler. “Validity of Student Evaluations of Instructional Effectiveness: A Comparison of Faculty Self-Evaluations and Evaluations by Their Students.” Journal of Educational Psychology 71, no. 2 (April 1979): 149–60.

Interpreting and Using Written Comments

  1. **Braskamp, Larry A., John C. Ory, and David M. Pieper. “Student Written Comments: Dimensions of Instructional Quality.” Journal of Educational Psychology 73, no. 1 (1981): 65–70.
  2. Lewis, Karron G. “Making Sense of Student Written Comments.” New Directions for Teaching and Learning 2001, no. 87 (September 1, 2001): 25–32. 
  3. Rando, William L. “Writing Teaching Assessment Questions for Precision and Reflection.” New Directions for Teaching and Learning 2001, no. 87 (September 1, 2001): 77–83.

Student and Faculty Attitudes About the Evaluation of Teaching

  1. Feldman, Kenneth A. “The Superior College Teacher from the Students’ View.” Research in Higher Education 5, no. 3 (January 1, 1976): 243–88.
  2. Feldman, Kenneth A. “Effective College Teaching from the Students’ and Faculty’s View: Matched or Mismatched Priorities?” Research in Higher Education 28, no. 4 (June 1, 1988): 291–344.
  3. Goldstein, Gary S., and Victor A. Benassi. “Students’ and Instructors’ Beliefs about Excellent Lecturers and Discussion Leaders.” Research in Higher Education 47, no. 6 (September 1, 2006): 685–707.  
  4. Ory, John C. “Faculty Thoughts and Concerns About Student Ratings.” New Directions for Teaching and Learning 2001, no. 87 (September 1, 2001): 3–15.

On-Line vs. Paper Administration

  1. Adams, Meredith, and Paul Umbach. “Nonresponse and Online Student Evaluations of Teaching: Understanding the Influence of Salience, Fatigue, and Academic Environments.” Research in Higher Education 53, no. 5 (August 2012): 576–91. 
  2. Anderson, H.M., J. Cain, and E. Bird. “Online Student Course Evaluations: Review of Literature and a Pilot Study.” American Journal of Pharmaceutical Education 69, no. 1: 34–43.
  3. Avery, Rosemary J., W. Keith Bryant, Alan Mathios, Hyojin Kang, and Duncan Bell. “Electronic Course Evaluations: Does an Online Delivery System Influence Student Evaluations?” Journal of Economic Education 37, no. 1 (Winter 2006): 21–37.
  4. Burton, William B., Adele Civitano, and Penny Steiner-grossman. “Online versus Paper Evaluations: Differences in Both Quantitative and Qualitative Data.”Journal of Computing in Higher Education 24, no. 1 (April 2012): 58–69.
  5. Layne, Benjamin H., Joseph R. DeCristoforo, and Dixie McGinty. “Electronic versus Traditional Student Ratings of Instruction.” Research in Higher Education 40, no. 2 (April 1, 1999): 221–32.
Posted on February 1, 2015 and filed under Evaluation.