What is course evaluation?
In colleges and universities nationwide science faculty are changing how they teach to include more student-active approaches. It is difficult to know how widespread or consequential the changes are, but it appears that funding for course reforms from PKal (Project Kaleidoscope), NSF and others has paid off. Walk into a random college classroom and the science professor will probably be lecturing, but walk into the class next door and it wouldn't be surprising to see students working on open ended questions in groups or engaged in lively discussion, even if the class is large. What you would be seeing are examples of students taking responsibility for their own learning.
Those of us who have taken the leap into student-active teaching are now being urged by our funding sources or workshop leaders to take the next logical step and evaluate our progress. We are being encouraged to ask how do I know whether my changes are working?, or in education language, to do evaluation.
Evaluation of our efforts makes a lot of sense. After all, we are scientists who pose questions that we continually re-examine with data. A problem facing us, though, is the same problem that made changing our views about teaching and learning so difficult in the first place. Most of us have no formal training in pedagogy or theories of cognition and learning, and we have no training in the principles and methods of evaluation either. We are being urged to do something even more foreign to us than experimenting with cooperative groupwork or nontraditional testing.
In my experience most faculty catch on quickly to the concepts and approaches of evaluation if they are given the fundamentals, such as vocabulary definitions, and also specific examples developed and modified by other science teachers. With the basic tools of evaluation I believe most of us can progress quickly towards designing our own evaluation procedures because doing evaluation is in essence like doing science. Evaluators make observations, ask questions, design experiments, get data, and revisit initial questions. Science faculty know how to do this and many enjoy such work.
In this article I will give Ecology 101 readers the same key information about evaluation that I and my colleagues give faculty in our teaching workshops. With this information and these resources I hope that some of you will become interested in asking "how do I know if it's working?" and more confident about how you can find out.
Educators like any other professionals have their own language, and the first hurdle for would-be evaluators is understanding the jargon and concepts of evaluation. I start with "evaluation" vs. "assessment" because this was most confusing to me at first.
Evaluation and Assessment
In my first NSF Division of Undergraduate Education training session, I learned that we evaluate projects or programs and assess student progress (e.g. give them tests). This is similar to the glossary definition of assessment in NSF's evaluation publication (1993): "Assessment is often used as a synonym for evaluation. The term [assessment] is sometimes recommended for restriction to processes that are focused on quantitative and/or testing approaches". To avoid confusion be aware that use of two terms is sometimes reversed. The classic Classroom Assessment Techniques (Angelo and Cross 1993) focuses on ongoing evaluation as a way for teachers ascertain what and how well their students are learning.
Program and Project Evaluation
NSF clearly distinguishes between evaluation of programs (a coordinated collection of projects) and projects (a particular activity; NSF 1993.). This article is about project evaluation.
Formative and Summative Evaluation
Understanding the difference between formative and summative evaluation helped me appreciate what evaluation was really about. Formative evaluation looks at the project (course) all along the way and its purpose is to give ongoing diagnosis and feedback so that professors can change their teaching if needed. Summative evaluation is what we are all familiar with when we give students tests. Or, as evaluator Bob Stake, said: "When the cook tastes the soup, that's formative; when the guest taste the soup, that's summative" (ibid.).
Angelo and Cross (1993) give a good overview of formative evaluation with their 7 basic assumptions of classroom assessment: 1) quality of student learning is directly related to quality of teaching, 2) the first step in getting useful feedback about course goals is to make these goals explicit, 3) students need focused feedback early and often, and they should be taught how to assess their own learning, 4) the most effective assessment addresses problem-directed questions that faculty ask themselves, 5) course assessment is an intellectual challenge and therefore motivating for faculty, 6) assessment does not require special training, 7) collaboration with colleagues and students improves learning and is satisfying.
A website that also nicely explains formative and summative evaluation has been developed by Doug Eder, a biologist at Southern Illinois University (www.siue.edu/assessment/, click on "classroom assessment techniques"). Doug emphasizes that formative evaluation is non-judgmental partly because the focus in on learning as influenced by many factors such as teaching approaches, student's background knowledge, and student motivation. Final assessments (grades) are usually private and anonymous, and the full weight of a grade is placed on the student alone who therefore identifies with them, for better or for worse. The following table modified from Eder's website details the difference between formative assessment and summative assessment (graded tests).
|Usually Goal-Directed||Usually Content-Driven|
Examples of Formative Evaluation Techniques
A good way to begin to understand the process of course evaluation is to simply look at a range of ways that science professors do it. The following is a list of approaches from Eder's site plus Angelo and Cross (1993).
Evaluating Student Knowledge:
Minute Paper — popular because it is a quick diagnostic that helps students reflect on the class and gives the teacher immediate feedback. Questions for a minute paper at the end of a session might be "What was the main point of today's class?", What points were most confusing?", "What points were most interesting?". Faculty who use this come up with their own ways to collect the responses efficiently (e.g. students pick up index cards on the way in and drop them in boxes in the back of the room on the way out). Even with large classes a professor can quickly scan through the cards to get the overall response to the questions. An important point with all formative evaluations is that faculty should bring common or interesting student responses to the next class because students will be much more likely to take this evaluation seriously if they see that the professor respects what they have to say. Minute papers can be used often or infrequently.
Muddiest Point — a modification of the minute paper that allows students to describe ideas or concepts that are most unclear to them.
Transfer and Apply — a way for students to learn how to apply what they have learned to new situations. Application is one of the more difficult critical thinking skills (along with analysis and comparison) that students need to practice. In this evaluation students are asked to list ideas or strategies from the class and then apply them to other situations.
Student Self-evaluation: Groupwork Evaluation by Students
Student-active learning usually involves students working collaboratively in groups on questions or projects in and out of class. In the workshops I've attended or led "groupwork" is second only to "coverage" as a controversial and difficult aspect course reform. In this short article I will only describe one way for students to evaluate how well their group is doing. This example illustrates how and why students, as well as teachers, do formative evaluation.
Most of us have very mixed feelings about asking students to evaluate their own and their colleague's performance and effectiveness in groups. I have only done this a few times and the results were fuzzy, probably because I did not prepare the students well enough. If you want to try this you can use one of the forms available on the web (see sites below) or make up your own. In this assessment students are asked to rank their responses to questions like: "How many group members participate actively in your group most of the time?", "How effectively did your group accomplish this task?", "How would you judge your own effectiveness in this group?" or address open ended questions about uneven participation or how the group could work better together. An important aspect of this evaluation is that it helps students be more reflective about group process and what can be done to improve it. Another critical point is that the teacher must allow class time for discussion of the purpose and ethics of this evaluation.
Evaluating Teaching Effectiveness
Reciprocal Classroom Interview — a formal technique in which two colleagues who know and trust each other interview students in each other's courses. This requires a fair amount of time and includes a meeting before the selected class to outline the focus and questions, another debriefing after class, clear explanation to the students about the purpose of the evaluation (e.g. that it is anonymous and private) and enough classtime.
Using Rubrics in Summative Evaluations (e.g. tests)
Educators frequently talk about rubrics, but most faculty have never heard of them. A rubric is a formal way to explicitly tell your students how you are going to grade or otherwise evaluate them on a test, paper, oral presentation, poster. When I first read about rubrics I immediately understood their utility and (again) how my ignorance about education has impacted my teaching. How could I expect my undergraduates to write good primary papers if I didn't tell them in real detail what I meant by "put your question or hypothesis in context", "compare your results with others'", or "describe your findings in the results section, interpret them in the discussion"?
Writing good rubrics may be one of the most important things you can do for your students and to improve your teaching. Creating the rubric clarifies your thinking about what you consider essential for your students to know and be able to do. For example, faculty in workshops often list "improving critical thinking skills" as an important course goal, but they do not explicitly explain, discuss, or practice what they mean by "critical thinking" in their particular course. Writing a rubric that operationally defines critical thinking helps faculty restructure their teaching to focus more directly on this sophisticated aspect of learning.
Eder's site contains a good example of a rubric for assessment and evaluation of student writing. Listed are aspects he uses in grading such as "uses disciplinary facts correctly" and "provides adequate supporting arguments with reasons, evidence, and examples". These are ranked from excellent to poor. I prefer to develop my own rubrics and I often do this with students because they become much more invested in the goals as a result. Some of these class discussions about good and poor development of arguments, data description, and the like have been invaluable to my students' understanding of these higher level skills.
Action research is a type of formative investigative evaluation done by teachers on their own classes and institutions. According to Elliot (1991) action research is "the study of a social situation with a view to improving the quality of action within it". Essential aspects of this research are that it is reflective, useful, focused on pragmatic issues or questions (that you can do something about), and structured. Action research has stimulated K-12 teachers in particular to professionalize and communicate their reform efforts and has empowered them to change situations in their classrooms and schools.
Action research could also be a powerful tool for college science teachers, but very few do it. Again, components of this research are quite familiar to scientists — focusing and shaping an issue or question plus collecting and reflecting on data. What I have found most foreign about this type of evaluation is that is that the data include fuzzy, qualitative information such as student behavior or interviews. What I have most appreciated is the potential immediate utility of the findings.
As an example from my own teaching, I was especially interested to know whether students in a freshman ecology course recognized the importance of a goal we discussed numerous times over the semester. The objective was for the students to recognize that science is a reiterative process and that messy or unexpected data are not "wrong". To evaluate my student's appreciation of this goal, I asked them to write self evaluations that focused on objectives for the course, and I looked for wording that would indicate their maturity about this aspect process of science.
About half of the students wrote things like "I did learn a valuable lesson that even mistakes made in research are useful..." and "Our experiment ... didn't really work as we wanted it to, but I learned a lot about setting up an experiment, looking for all the variables, and identifying problems". While I was pleased with these comments, I was surprised that more students did not make them. This finding has forced me to think more carefully about how I discuss this aspect of science in the class. Although I thought I was quite explicit, perhaps I was not.
Another example comes from an introductory Oceanography class taught by Richard Yuretich, a geologist at UMass in Amherst, MA. Richard teaches in the most challenging of situations — in a big lecture hall to 300 students who use this course to fulfill a requirement at a state university. Over the past few years Richard has made major changes in this class including cooperative group exams and frequent groupwork in class on open-ended questions (e.g. "think-pair-share"). One way that he has attempted to assess the effect of these changes on student performance is to compare final exam results in 1996 (before the changes) and 1998 (after changes; Yuretich and Leckie 2000). He found that the mean exam score was substantially higher in 1998 and students in the redesigned courses did better on 37 of 38 identical questions. (The topic for question 38 was not covered in 1998). More specifically, fewer students received a "D' or "F" in the second year, indicating that the changes may have helped those with greatest academic difficulty. The questions assessed a range of abilities: recall, calculation, interpretation, and deduction. Additional evidence about improved student attitudes in this class came from end of semester evaluations. Many more students in the reformed class showed interest in oceanography and they acknowledged benefits to them from the new teaching approaches. (For a similar but more thorough study of a large course in biology see Ebert-May et al. 1997).
Reflecting on data like these has been important to Richard. Redesigning this class has been extremely time-consuming. He and I have had many conversations about time investment in teaching vs., research, and so positive feedback that the efforts are really worth it is crucial for him. In addition, he now has evidence that he can show students from their own course when he explains the pedagogical philosophy of the class and talks about how people learn best. Finally, the research provides that bases for further improvements. Written comments from students at the end of the semester emphasized a common problem with groupwork — students who "go along for the ride" and do not participate in discussion. In the future he hopes to use older students as roving monitors in class to help address this issue, and these written comments from students may help him get the extra funds to support this.
If you decide to try doing formative evaluation in a course, start small. Pick an appropriate method for one class session, tell your students what you are up to and why, and then report back to the class, including explaining any adjustments you make.
To learn more about formative evaluation of your own redesigned courses, look through the websites and other resources listed below. Also, I have just completed a commercially produced video (funded by NSF's Division of Undergraduate Education) called How Change Happens: Breaking the Teach As Your Were Taught Cycle in Science and Math that features Richard Yuretich and other faculty from a range of colleges and university settings. How Change Happens follows these teachers into their classroom as they improve their teaching and reflect on their progress, what keeps them going, and how their students have become more reflective, better thinkers. Email me if you would like a free copy of this video. Finally, sign up for the teaching research workshop offered by Diane Ebert-May at the Snowbird 2000 and future ESA meetings.
- Work related to this article was funded by National Science Foundation Grants No. 9653966 and 9654358.
- Adelman, C. (ed.). 1988. Performance and judgment: Essays on principles and practice in the assessment of college student learning. Office of Educational Research and Improvement Publication OR88-514. Washington, D.C.: U.S. Department of Education.
- Altrichter, P., P. Posch, B. Somekh. 1993. Teachers Investigate Their Work: An Introduction to the Methods of Action Research. Routledge, N. Y.
- Angelo, T. A. (ed.). 1991. Classroom Research; Early Lessons From Success. New Directions for Teaching and Learning. Jossey-Bass, San Francisco.
- Angelo, T. A., and K. P. Cross. 1993. Classroom Assessment Techniques: A Handbook for College Teachers. Jossey-Bass, San Fransisco.
- Banta, T. W., J. P. Lund, K. E. Black, and F. W. Oblander 1996. Assessment in practice: Putting principles to work on college campuses. San Francisco: Jossey-Bass.
- Cross, K. P., and M. H. Steadman. 1996. Classroom research: Implementing the scholarship of teaching. San Francisco: Jossey-Bass.
- Diamond, R. M. 1997. Designing and Assessing Courses and Curricula: A Practical Guide. Jossey-Bass, San Fransisco.
- Ebert-May, D., C. Brewer and S. Allred. 1997. Innovation in large lecture teaching for active learning. BioScience 47: 601-607.
- Elliot, J. 1991. Action research for educational change. Milton Keynes and Philadelphia: Open University Press.
- Hutchings, P., and E. Rueben. (1988/July/August). Faculty voices on assessment: Expanding the conversation. Change, 20(4), 48-55.
- National Science Foundation. 1993. User-Friendly Handbook for Project Evaluation. J. Frechtling, ed. NSF publication 93-152.
Student Course Evaluations
This article was originally published in the Fall 2003 issue of the CFT’s newsletter, Teaching Forum.
by Anupama Balasubramanian
This column highlights concrete innovations and insights in teaching and learning across the Vanderbilt campus. In this issue, a Vanderbilt faculty member and teaching assistant discuss their perceptions of student course evaluations, and their strategies for reflecting on them and using them to improve their courses.
Kathleen Hoover-Dempsey is Chair and Associate Professor in the Department of Psychology and Human Development, as well as recipient of the university’s highest teaching honor, a Chair of Teaching Excellence. She is one of the pioneers of the Family-School Partnership Lab at Vanderbilt University, which is dedicated to the scientific investigation of the reciprocal relationships among families, schools and children. She teaches undergraduate courses in the child development major and is currently teaching a graduate level course in Educational Psychology.
How do you respond to your end-semester student evaluations?
With care and caring. I wait until the semester is well over and wait until I can sit down by myself and digest the information, particularly the student comments on the back of each form. I often move from that into trying to identifying themes in the comments. I look particularly for themes that I really need to do something about, especially things that might not have gone as well as anticipated in the eyes of the students. I also look for more generic advice that might help me, perhaps to do a better job balancing certain topics, or required projects, or midterms more effectively. After many years of teaching, I have a pretty good handle on the “rhythms” of courses, so I tend not to get much feedback on those issues. But I’m always looking for things that students identify as strengths of the course and what were weaknesses. I want to take steps to do something about the weaknesses; and work hard to figure out how I’m going to take them into account and address them the next time that I teach that course.
So would you say you give more credence to the comments than to the numerical ranking?
I actually find the numerical ratings on the form very helpful, too and I map them out across semesters. In my role as chair I do that for all my faculty, as well. That information gives me a good sense of trends and progress across semesters; I look especially for upward trends or ‘stalled’ areas. Overall, I think the numerical ratings are really important, but you often need to analyze students’ comments in order to remedy some of the concerns that may underlie lower ratings. That is the reason I also really look hard at the comments.
Have you changed or improved your teaching based on the feedback you have received, and if so, what are some concrete examples?
I definitely think I have improved my courses and my teaching over the years based on student comments and feedback. I have done this for so long (i.e. experimented with changes based on themes in comments), that it’s a little hard to identify particular examples at this point, but I’ll try.
I’ve certainly had comments about exams–for example, the balance of objective and subjective items on exams or preferred approaches to final examinations. Over the years, those comments have been so helpful that I don’t get many suggestions in those areas any more. (That’s one of the benefits of having taught for a very long time!) But certainly, there are often very useful comments about my approaches to a particular session or topic that I may well use in rethinking, for example, the balance of attention I’d give to theory, research and applications related to a particular topic. There are also comments that help me think well about effective ways of engaging students in active work with constructs during class sessions.
Because I sometimes teach large courses and have TAs, I also get very helpful feedback on more effective or efficient ways to engage my TAs in the course. When my TAs teach a session, we gather informal student evaluation comments, and my TAs find these very helpful. We also gather student evaluations for the TAs at the end of the course. These can be particularly helpful because they’re among the first formal comments that my graduate students have on their teaching and direct work with undergraduate students.
In addition, in the middle of the course (generally a week after I return the first midterm), I often ask students to give me a midterm evaluation of the course. These always include useful ideas that I can address while the course is in progress.
Is this mid-semester evaluation form something you create on your own?
Yes. Sometimes I write up an evaluation form; at other times I use copies of the regular evaluation forms. Generally, I tell my students that I want their feedback now so that I can do something about the things that aren’t working as well as they might be. I always come back to the class to tell them what I’ve learned from their evaluations and what I’m going to do about specific suggestions. On occasion, I get very mixed comments, like “You’re going way too fast” and “I love the pace” and “You could speed it up.” When I get mixed responses, I usually summarize their feedback and talk with them about the dilemma this presents for an instructor; I talk then how I plan to address the concerns and why.
Do you think that your students take the end-semester course evaluation forms seriously?
Yes, I do think my students take them very seriously. I think they do in part because I tell them that I take them seriously. I schedule a time when we’ll do the evaluation form; I tell students in advance that I consider it to be very important, and tell them that I really want them all to be present to evaluate the course. I tell them that I read every comment and find the comments extremely useful in thinking about and improving my own teaching. When I give the evaluations forms out I repeat all of those things, and add, “You can never write too much; I value all of the feedback I get, I do read it and it is very important to me.” And then I follow all of the university guidelines (like getting quickly out of the classroom after identifying who’s going to collect and return them to the department office.)
So yes, I get very substantive feedback, which I really value. In many courses, perhaps especially large ones, there is likely to be at least someone who’s not particularly happy with the course. Their feedback can be very, very helpful to thinking about what I might do differently in the course. I think emphasizing that we take student comments very seriously, and find them very helpful, simply increases the likelihood of getting very useful feedback from all students.
Scott Hicks is a graduate teaching assistant in the English Department and served as a Teaching Affiliate at the Center for Teaching in Summer 2003. He currently teaches a 100-level English composition course, which he designed and teaches on his own, under the auspices of the College Writing Program.
How do you respond to your end-semester student evaluations?
I take them really seriously because I think they play a large role in how we are viewed as TAs in our program. TAs are in a sort of an apprenticeship and are still learning the field and their profession, so I think they really do matter. But I do find that they vary in degree of helpfulness. Sometimes the students that I thought did not like the class loved the class, or a few that I thought loved the class maybe did not like it. So I sometimes find out from students that I can’t read how they really felt about it. But usually, the evaluations confirm what I already knew about the class.
The main things I look for in my student evaluations are: (1) The extent to which they feel they have been challenged and (2) Whether or not they found me to be a helpful and communicative teacher. In other words, I want to make sure my classes are always challenging, while still trying to make sure I meet their needs, including what they think they need. And I want to make sure I’m communicating what I expect, because if I don’t do that, they don’t know what to do and I don’t get from them what I need to get from them. When I get the evaluations, I read them but I really don’t stress out about them one way or the other, because I really cannot fix anything for that class. Instead, I can use them as an insight to stop and think about future classes. Since I design and teach my own courses, all the evaluations I get speak to me of things that I can control.
Have you changed or improved your teaching based on the feedback you have received, and if so, what are some concrete examples?
I think the most concrete thing I have changed is modifying the novels that I use based on their response to specific books that they liked or did not like. One semester I remember teaching a memoir that they hated and another dense novel that they did not understand. So in the future semesters, I spent more time on the dense novel, had them do more small group work, discussed more of the plot and the characters to make sure they had a firm foundation, and then got into themes. So in that sense I think it helped because I was able to get rid of something that they were not getting anything out of, and then focus on something that they could get a lot out of if I gave them more guidance and worked on it more in class.
At a less concrete level, one thing I have tried to work on is responding to students on their writing in my comments on papers and in conferences. One thing I always can do better is to communicate to them what I want them to do on paper and what I see as areas they need to improve on. Responding to students more articulately can help them see what they need to do and help me in my teaching. When I grade papers, I give them back written comments and then if we have a conference on it, I try to make sure I am always communicating as fairly and concisely as possible because they have a lot going on. So the evaluations help point things out from both ends: when they evaluate me I want them to be as clear and concise as possible, and when I evaluate them they want me to be the same.
Have you made any drastic changes to your teaching like adopting a totally different method based on the feedback you received?
I have not drastically changed anything because I feel I cannot teach from outside my personality and my core approaches to teaching. If I do anything too different from that, I feel I am not going to do well. What I have found, though, is that a lot of times the most important thing that I can do is to explain why we are doing something. For example, in the first semester I taught, we did a lot of group work, but in the second semester I did a better job of explaining why: when working in groups, better ideas are generated, shared, discussed and critiqued. When one explains to students why some things are done, they are more likely to understand the need for doing it and will not complain because it has a valid, useful purpose.
Any other suggestions for improving the evaluations teachers receive from students?
I think the biggest thing one can do is to let students know that you care about their academic performances as well as their personal lives. This doesn’t mean you have to be a therapist or get involved in their personal lives. Rather, if they see you caring about them as students and as people, their evaluations are going to be filtered through the sense that you are interested in both the class and in them.
In addition, I gather feedback in the middle of the semester. I have used the Center for Teaching’s Small Group Analysis (SGA), which was very helpful. I also have used an evaluation form in the middle of the semester, as well as the end-semester forms.
Is the evaluation form that you give mid-semester something you design?
Sometimes I use one of the forms from the Center for Teaching website. It is a question-short answer kind of a form, because when I do a ranked form, I don’t get the feedback I need to make improvements for the rest of the semester. With a short answer form I get really tangible suggestions like “I hate this novel” or “I did not understand that story,” and I can go back and answer some questions, or change my style of teaching. It is really useful to get this feedback at a point in the semester when something can be done about it.
So this form is primarily something you use on your own to improve your teaching or your course?
Yes. I try to see what students need and help them actually understand what is being done in class and get something out of it. I don’t want to find out about the gaps at the end of the semester when it is too late to fix things. Another good thing about doing the mid-semester evaluations or the small group analysis is that it shows students that you’re concerned about what they’re getting out the class. It is not at all an insult to me personally, but instead enables me to say to my class: “I am really concerned about how things are going and we are going to change these few things,” and then let them know that I want to help them out and change things and make them work. I increase the things that work, and cut out the things that don’t work. Doing so makes it easier for me to prepare for class, as well.