Course Evaluations: More Carrot, Less Stick
I appreciated the candid emails from BB yesterday and today about course evaluations. He reasserted Boalt's dismay that since the school switched to an online system, "the return rate has been slowly dwindling," and he explained how the school has responded.
First, data will only be viewable by students who completed an evaluation the previous semester. (I'm willing to bet there is a giant exception for fall semester 1L's, so if you're lazy, that's who you should coerce. It shouldn't be hard.) BB explains that the school favors this scheme because it has generated a 90% response rate at Northwestern, perhaps because it has a "certain symmetry." It seems at least as likely to me that the high response rate at Northwestern is partly because it's so damn cold and miserable there that students have nothing better to do, but that's pure speculation.
Second, the school will begin releasing all quantitative data for all classes, with the exception of junior professors (but not junior lecturers - yet another short straw to the hard working folks in the Skills Program, no?)
Third, Boalt will begin offering a bigger carrot. Students who complete timely evaluations will be entered into some sort of Amazon.com raffle. See the email in the comments.
Finally, the school is "still discussing" whether to release the written comments from the evaluations. This is, in my opinion, the key to the whole shebang. I understand that there is a California labor law problem with mandatory release of the written responses, but I continue to believe that if that information were released the response rate would skyrocket. No doubt Armen will chime in with lush and nostalgic praise for the system at UCLA, and with good reason: communication is a two way street here, and until we get the sense that WE can benefit from our input, assertions that the school "takes them quite seriously" continue to feel rather empty. I realize the labor law may pose a bit of a stumbling block, but I continue to have faith that if the school wants to, it can someday find a way around it. Where there is a will, there's surely a way, right? I probably shouldn't go there but, think: Prop 209.
Oh yeah, and click here to complete your evaluation.
First, data will only be viewable by students who completed an evaluation the previous semester. (I'm willing to bet there is a giant exception for fall semester 1L's, so if you're lazy, that's who you should coerce. It shouldn't be hard.) BB explains that the school favors this scheme because it has generated a 90% response rate at Northwestern, perhaps because it has a "certain symmetry." It seems at least as likely to me that the high response rate at Northwestern is partly because it's so damn cold and miserable there that students have nothing better to do, but that's pure speculation.
Second, the school will begin releasing all quantitative data for all classes, with the exception of junior professors (but not junior lecturers - yet another short straw to the hard working folks in the Skills Program, no?)
Third, Boalt will begin offering a bigger carrot. Students who complete timely evaluations will be entered into some sort of Amazon.com raffle. See the email in the comments.
Finally, the school is "still discussing" whether to release the written comments from the evaluations. This is, in my opinion, the key to the whole shebang. I understand that there is a California labor law problem with mandatory release of the written responses, but I continue to believe that if that information were released the response rate would skyrocket. No doubt Armen will chime in with lush and nostalgic praise for the system at UCLA, and with good reason: communication is a two way street here, and until we get the sense that WE can benefit from our input, assertions that the school "takes them quite seriously" continue to feel rather empty. I realize the labor law may pose a bit of a stumbling block, but I continue to have faith that if the school wants to, it can someday find a way around it. Where there is a will, there's surely a way, right? I probably shouldn't go there but, think: Prop 209.
Oh yeah, and click here to complete your evaluation.
posted by Patrick Bageant at 6:21 PM
17 Comments:
Email #1Since we switched from paper course evaluations to a web-based system, the return rate has been slowly dwindling. This is a problem. The evaluations are a valuable tool. They provide feedback to the instructor that can help him or her improve as a teacher. The results are also used in judging what adjunct faculty to rehire and they are included when our faculty are considered for promotion or tenure.. The Campus takes them quite seriously. Improving the return rate is a priority. During the fall semester the Curriculum Committee discussed the problem at length. The student members of that committee conducted a survey of students about the issue. In the end, the matter was brought to the faculty for consideration. Here is what we came up with as a solution.
First, drawing on the experience of other law schools who faced the same problem, (there is lots of speculation about why this problem is so widespread), the faculty decided that course evaluation data for a specific semester will only be available to students who complete evaluations for all of his or her courses during the previous semester. This has system has worked other places. At Northwestern, it resulted in a response rates of over 90%. There is a certain symmetry to this plan. You fill out evaluations, you can view evaluations. Recognizing the value of a more market based system, the faculty also agreed that as an additional incentive, those who fill out their evaluations in a timely manner will be entered in a raffle for gift cards. (More details on this to follow).
Second, the Curriculum Committee received feedback from students requesting that evaluation data be made available for all classes, rather than via the current system where it is only made available if the instructor chooses to do so. The faculty decided that the quantitative evaluation data for all courses will be made available. The sole exception is for courses taught the first time by a junior faculty member. In that case, the data will not be disclosed unless the faculty member agrees to do so. Quantitative evaluation data for all courses taught by lecturers will be made available. The first set of courses to which this policy applies will be those taught this semester.
Third, the faculty is still discussing the issue of releasing the qualitative comments written on the evaluations. The fall survey, and the student members of the Curriculum Committee, have let us know that this is something students would like to see. I will keep you posted as that conversation progresses.
Why tell you all this in the sixth week of the semester? There are a few short courses wrapping up even as you read this e-mail. ( BTW thanks for reading this far). If you are in one of those courses, please fill out your evaluation now. I will be back to you at the end of the semester with a more general and pithy exhortation to fill them in for all courses and news of any new developments. Thanks for the great response on the fall survey. It really helped.
Email #2Back in February I wrote to you about the question of how to get more students to fill out course evaluations. Since we switched to the online evaluation system, return rates have been low. We need to get evaluations for many reasons. The faculty can use the feedback. The School needs the data so that we can use it in making personnel decisions about regular faculty (promotions and tenure matters) and it helps us decide what adjunct faculty to ask back to teach again. The data can also help you in choosing courses.
To this end you will be receiving a series of e-mails about incentives that we will offer for those who fill out evaluations (the market incentive has worked at other law schools) and we will be urging faculty to take class time for you to complete the evaluation. I am sorry to report that some of the substantive changes that I described in the February e-mail, access for students to some narrative comments from the form and disclosure of numerical data for all courses, has not happened. It is taking longer to work out some of the details than we anticipated. This was the Curriculum Committee's fault, we thought we did not foresee some of the challenges involved in making the changes. Since I am the chair of that committee this year, I will assume responsibility for the failure. The student members of the committee did a great job in moving things forward, and I think that in the fall we will be able to make much of what we promised a reality. For this semester, though, you will see a form that is much like the old one. Sorry.
In the February e-mail I also announced a new policy that mandated that only those who filled out evaluations for all courses this semester would be able to look at evaluation information next semester. (Other schools have used this method to good end). But since we did not manage to retool the form, we do not think we can implement that policy. If we offer no carrot, we should brandish no stick. Actually, we are going to offer prizes for those who fill out evaluations, a fairly tangible carrot, but we did not fix the form, so fair is fair.
Please fill out your evaluation forms this semester. News of the incentives is on the way.
Email #3Course evaluations started today and end two weeks later on Friday 5/1. The faculty and administration really value your input. We use your evaluations in making promotions and tenure decisions about regular faculty and in deciding what adjunct faculty to ask back again. We hope that the evaluations aid your course decisions as well.
This year we are aiming for an 85% evaluation return rate in every class and will be awarding 20 Amazon.com gift cards worth $50 a piece to students who complete evaluations. Each day, starting today, and continuing through the first week of evaluations, we will choose 4 students at random who have completed evaluations and award each one a gift card. The more evaluations you complete, the more chances you have to win. Starting tomorrow and each day following a drawing, I'll announce the classes from which the winners were chosen.
Help your law school, your classmates, and yourself. Complete an evaluation for every one of your classes.
To be honest, I'm not sure I want the qualitative statements. I had to read some as part of a hiring committee, and they range from useless to vitriolic. Sifting through hundreds of rambling, aimless comments to find one nugget of truth seems like something I'd like to do if I went to school at a place like Northwestern.
Matt, do you think public disclosure would temper some of the vitriol?
Based on what I read - mainly ad hominem attacks - I'd sure hope so! Granted, that was only a handful of evaluations out of a 90-person class, but still.
I'd be more concerned that once they know their words will be available for public consumption, students might be more forgiving and less candid.
Patrick, you know me too well. I still think BHSA [see comment at 9:47 PM] should host a website that allows students to rate the professors and offer qualitative feedback. The ratings currently used are useless. 1-5 Rating and every single professor is between 3.8 and 3.9. Give me a break. Double it to 1-10 and add a few additional categories of review.
Qualitative can be useless, but where a strong theme emerges, then it's a very useful trend. It's a bit beyond tragic at this point that you can get better feedback if you're shopping for a used ipod on ebay than for the education that is to serve as the foundation for the rest of your professional life.
I really think the qualitative responses are the key too -- how helpful would Yelp be if it was only quantitative grades? Who would read Pitchfork if all they did was give a numerical value for each record?
I would even be fine with a situation where the prof has four choices: deny qualitative data altogether, allow all qualitative responses, choose whether or not to publish each response, or allow an unbiased third party to comb through the data and remove anything inflammatory or inappropriate.
Just knowing which option a prof has chosen would tell us a lot about the prof -- a prof willing to share qualitative reviews, warts and all, is probably the sort of professor I want to be studying with. On the other hand, I think the Boalt population could get really good at finding the hidden truths in selectively published comments.
Wait, the "only viewable by students who completed an evaluation the previous semester" doesn't kick in right away right? (See Email # 2: "since we did not manage to retool the form, we do not think we can implement that policy. If we offer no carrot, we should brandish no stick.").
Maybe I read to fast. If so, I reserve the right to use the 3L defense, as this policy would hardly affect me either way.
Also, I agree that response rate would skyrocket if responses were always made available. Or if not that much, I'd at least like to know my effort of filling out the evaluation was serving some purpose, any purpose.
They should tie doing the evaluations to being able to see your grades online... that'll get everyone on board.
I know that's sort of extreme. But honestly, these take practically no time and are of an obvious and important use. Just fill them out, it's really not so hard.
I personally will withhold my evaluations until written comments are also released. I disagree with Matt on their usefulness. Some truth can be gleaned from even the most vitriolic comments.
Quantitative ratings are skewed in that some people will always put "3" or "5", or if the student absolutely hates the professor, a "1" hardly does justice (and indeed gets hidden from the viewer in the class average). The way you evaluate someone that gets all 3's is different than the way you evaluate someone that gets half 1's and half 5's.
I wish I could have seen comments for civ pro 2. I would have known in advance to steer clear and teach it to myself (like I'm doing anyway). Take that Armen and Patrick!
toney, if you're not down with vetter...*sigh* i'm sorry.
haha... I'm "down" with him. He's funny, incredibly smart, etc. I just can't understand him. I suspect I'm not the only one.
I don't hold anything against him. I just wish I had known these things before I took the course.
fwiw, i think CPII is just kind of an unintelligible subject. i had bundy, and while he's got quite the cult of personality on this board, i didnt think he was particularly cogent or clear.
I wish Gilbert, of commercial outline fame, taught a class.
You know, I don't think they could tie it to us receiving our grades, Laura. Even though I think it's silly, they always close down evaluations before finals start. So it's not like we could go back later and do our evals in order to get our grades released. And they can't withhold our grades forever because we missed the two-week evaluation window. Of course, I personally think we should be allowed to evaluate the final exam too. That's a big part of the course. But Boalt apparently disagrees. I think, at most, they could say if we missed the evaluation window, we just couldn't see our grades online that semester and we'd have to wait for a paper report card to come. But that would seem to cause them some new administrative headaches.
toney - it's possible to see a breakdown of what constitutes the mean response. click on the image to the right of the posted mean & a pop-up will indicate the breakdown of responses, the mean, the standard deviation, and the percent of students responding for that question.
I <3 11:19!
I'll chime in with the perspective of someone who rarely fills out his evaluations. (I know. I suck.) I don't dispute that evals are an important resource or that more people (including myself) should do them. And I'm not sure where I come down on releasing the written comments. But I will say that, for me at least, this carrot isn't worth much.
Like for a split-second I was thinking, "Damn. I guess I have to do them this time." Then it quickly dawned on me that I have almost never looked up the course evaluations anyway, and when I have, they weren't all that helpful. Instead, I've most often relied on word-of-mouth and found that it's usually spot on.
I guess my point is that the people who don't usually bother to fill these out aren't very likely to care about not having access to them. And, secondarily, I'd advise anyone who does get shut out by this policy to make some friends.
I really don't think that its a matter of quantitative vs qualitative---personally I'd rather just look at a bar chart on the professors' "ability to stimulate thought" rather than read through 60 comments. But the categories they currently have are so useless! And, as mentioned above, the lead everyone to answer between 3 and 4 for all but the most adored/hated Profs. I also think that the exam HAS to be evaluated as well. I once had a prof I thought was okay until I took the final and it was filled with errors and inconsistencies---everyone I spoke to thought it was a horrible measure of what we had learned.
Post a Comment
<< Home