Berkeley vs. The LSAT
Looks like researchers right here in B-town are hard at work in designing a new law school admissions test that would correlate with success not as a 1L, but as a lawyer. What a novel concept.
If this saves just one future student from having to spend a month learning "tricks" on the logic games, it will have been worth it. Still, I'm skeptical that any 3-hour test can effectively measure the qualities that make a good lawyer. It seems to me the #1 most essential ability is a strong work ethic (not to imply that I have one), which is going to be tough to tease out with multiple choice questions.
What do you guys think of the LSAT? How should a "better" test differ from the one we endured? What are Berkeley's chances of actually affecting any change in the system?
If this saves just one future student from having to spend a month learning "tricks" on the logic games, it will have been worth it. Still, I'm skeptical that any 3-hour test can effectively measure the qualities that make a good lawyer. It seems to me the #1 most essential ability is a strong work ethic (not to imply that I have one), which is going to be tough to tease out with multiple choice questions.
What do you guys think of the LSAT? How should a "better" test differ from the one we endured? What are Berkeley's chances of actually affecting any change in the system?
posted by Dan at 5:51 PM
42 Comments:
I think the LSAT tests three things: inate intelligence, work ethic, and how much money you have to throw at kaplan. The first two are important to law school and career success, the third obviously is not. Keep the LSAT and compensate for the third factor through socio-economic reverse discrimination.
I'd be interested to know how much the Kaplan courses actually help. I'm sure there is research out there, but I'm equally sure a lot of it was paid for by Kaplan.
I studied for the LSAT entirely by myself and definitely improved my score over time. I have always felt that people who do Kaplan might actually be at a disadvantage, since they tend to put an inordinate amount of faith in the program and do little else on their own.
There's no question that Kaplan is better than nothing, but I think a lot of well-disciplined individual studying could actually do more than Kaplan could.
That said, any study program is going to require that one buy or somehow get access to study books and practice quizzes, so financial means will probably still factor in.
Good point Dan. In a sense, I think that much of the benefit of Kaplan is just buying "work ethic"... pay a whole bunch of money and you're forcing yourself to show up and pay attention.
-6:45
I'm very skeptical of the concept that test drafters (or anyone) can define, quantify, or measure what "success as a lawyer" means. More sheep head to firms? High salaries earned? More pro bono performed? Better impact on the world? Trying to even answer these questions strikes me as silly. Trying to measure aptitude for such "success" in three hour diagnostic strikes me as downright ridiculous.
1L grades are at least something you can actually measure. Plus you can quickly determine if your test predicts grades or not. Finding out if you accurately predicted "success as a lawyer" should take a lifetime.
The answer, I think, is just to reduce emphasis on the lsat in admissions. It seems Berkeley already follows this approach to some extent given our average lsats compared to peer schools.
For what it's worth, my lsat score was lower than my not so stellar pre-Kaplan diagnostic test. (Ed Tom gets regular thank yous in my book for nonetheless letting me into this school...)
Is the LSAT really supposed to measure success as a lawyer? I thought it was supposed to be a predictor of law school grades -- particularly first year grades -- and my understanding is that it actually does a decent job of that, at least in the aggregate.
The problem is that first year grades do not seem likely to predict "success as a lawyer." If that is correct, then the curriculum has just as many problems as the LSAT.
Law schools, especially lower ranked schools, are looking for students who will do well in their first year classes and go on to pass the bar. If the current LSAT is a good indicator of this, then it makes sense for these schools to the test scores in admission.
While law schools no doubt strive to produce more successful lawyers, it appears they are obsessed with rankings. (see ATL and every time Edley speaks).
Reflecting on everything I've done so far as a law student, I've got to say, I don't think that the logic games have any place on a test designed to measure law school success. Logical reasoning and reading comprehension, yes; "Andrew, Bernard, and Charlie are in a triathlon," no.
Andrew rode the bike and wore a yellow jersey, Charlie ran in blue shorts, and Bernard swam. Naked.
I win!
How come no one ever suggests taking undergraduate grades out of the admission process? Is there any proof undergradute grades correlate to 1L year grades or "success as a lawyer"?
LSATs are supposed to predict success in law school courses, but I think it's a stretch to say that they predict grades. To ace the LSATs, you need a firm grip of stochastics, logical operators, and contrapositive theory, and need to have the capacity and attention span to tackle reading comprehension. Aside from reading comprehension, nothing in the legal profession is going to require you to regurgitate or practice the rules of logic in 30 seconds without preparation.
I differentiate between success in a course and the grades you get in a course (you could easily define success as good grades). The skills needed to excel at the LSATs probably do predict one's ability to contribute to, understand and retain the material covered in a class. But grades themselves also depend on things like how involved you are during a particular semester, how much you like the classes you're taking, and most importantly, how much you care about grades.
I self-prepared for the LSAT, but I think a 2-day crash course at the beginning of my practicing would have cut my preparation time in half.
The best metric for success as a lawyer is (of course) street cred. Why they haven't factored this in yet, I don't know.
Patrick, you're correct. I thought I implied that with the "predict success not as a 1L, but as a lawyer" bit. The LSAT currently is designed to correlate with 1L grades. The proposed alternative would correlate with "success as a lawyer." Sorry for any confusion.
Toney, actually the LSAT is a very good predictor of 1L grades. They make sure of that. I agree that "success" in a course or as a lawyer is difficult to define, however.
Really my problem with the LSAT is just that the logic games seem to have no bearing on anything. I mean reading comp and understanding argument (logic) are both definitely important. But everyone knows the logic games are the most "learnable" part of the test, which also means they're the least indicative of any innate ability. I suppose there's an argument that they measure work ethic, since you can study hard enough to figure them out (as was my plan). But still, they suck.
I think the LSAT predicts law school grades because both are looking for quickness. 1L exams, in my view, put a huge premium on quantity over quality. The task was to tick off theories or relevant considerations, not to discuss them in any depth.
This ability to do work quickly and fairly accurately is definitely an asset in many lawyer jobs--the billable hour is designed around it, no?--but it also leaves much to be desired for.
There are definitely some outliers wrt to LSAT v 1L grades. I send the same thank-yous to Ed Tom (I would not have gotten in to any other top 10 school w/the lsat i got - low 160's), but I did very well 1L year.
8:21: Yes, there is evidence.
Predicting Law School Academic Performance from LSAT Scores and Undergraduate Grade Point Averages: A Comprehensive Study: "1. The predictive power of the LSAT score, undergraduate GPA and a combination of those two measures in an "index" is about the same, both for first-year and for overall academic performance.
2. Although the differences may be small or statistically insignificant, the LSAT score is a better predictor than GPA of first-year law school performance, and GPA is a better predictor of overall law school performance. An index with a combination of LSAT score and GPA is substantially better than either LSAT score or GPA alone in predicting both first-year and overall law school performance."
The LSAT, Law School Exams, and Meritocracy: The Surprising and Undertheorized Role of Test- Taking Speed:
"[T]he data showed that the LSAT was a relatively robust predictor of in-class exams and a relatively weak predictor of take-home exams and papers. In contrast, undergraduate GPA (UGPA) was a relatively stable predictor of all three testing methods."
The article doesn't note the lead researcher is Marge Shultz. Not that new of a project at all.
Am I the only one who thinks there is an elephant in the room here? Everyone knows that LSAT cutoffs correlate to lower minority enrollment. An attempt to revise the LSAT itself to account for "soft factors" might be a way for DE to increase minority enrollment without causing a slip in the rankings.
Here's the paradox: LSAT scores may correlate to 1L grades in traditional grading systems, but this all goes out the window with relative grading (ie Berkeley's grading). The only way LSAT scores could be predictive is relative to the LSAT scores of the other students. So if all 1Ls had LSATs between 165-175, the predicted 1L grades would as accurate as the grades predicted if all 1Ls had LSAT scores between 145 and 155.
I still think the theory that LSAT scores as predictors of grades is bunk. Work ethic and effort seem far more predictive. In my successful (grade-wise) semester, I put far more effort into school than my less successful semester. And there are several students who have gotten great grades that had poor LSATs. Having said that, LSATs are probably the best quantitative predictors available, but I think essays and interviews give far more insight as to the success of a student in their first year (and beyond).
Toney,
I don't think your Boalt experience necessarily indicates that the LSAT is ineffective. Remember that there is very little variation in LSAT scores at top schools. You have to look at schools where there is a lot of variation, i.e., plenty of 170+ scorers and sub 160 scorers, to see whether the LSAT is useful. And I think studies looking at such schools have shown that the LSAT explains a lot of the variation in grades.
2:14 - I'm not referring to my own experience as the backbone of my claim that LSAT scores are very imperfect predictors of grades (it was just an example).
The problem is, how many schools have plenty of 170+ and sub-160 LSAT scores? That's a pretty huge range, and while there are always outliers, there are outliers to the claimed LSAT-grades correlation as well.
I still think that the LSAT scores/grades relationship is not causal/effectual. But law schools want a number to index us by, and there aren't a lot of other choices.
Any comments on how Boalt didn't make it into the top 25 public interest law schools? See http://taxprof.typepad.com/taxprof_blog/2008/11/top-public-interest-law-schools.html
Yep.
Yeah, talk about not giving credit where credit is due...
Is this that test students were supposed to test out two or three years ago? If it is, that thing was basically a shorter Myers-Briggs. "Adam, Betty, and Carlos go to the movies" may not be relevant to success as a lawyer, but "Do you like to socialize in groups or with one other person only" doesn't seem very helpful either. Granted I hope lots of improvements have been made since I took the beta version, but that test seemed like it would be even easier to game than the current LSAT. Kaplan will be on that in about 30 seconds.
toney... all law schools have relative grading. With the traditional system there are more possible grades and it is easier to tell how someone did in a particular class, but it's graded on a curve in that system just as much as in the Boalt system. And you can measure whether people with high or low LSAT scores do better or worse, whatever school they attend. Of course, you are right that because we depend on the LSAT it artificially limits the range of LSAT scores that appear at a particular school, but this can be controlled for in a well-designed study.
I taught LSAT for a while before law school. It is true that the games section is the easiest to improve on, but false that the other sections are therefore more closely correlated with innate ability. I do agree that the non-games sections involve reasoning that is closer to both law school and lawyering than the games. All are clearly teachable. But it takes time & effort that most people who scored badly never put in. Good instruction helps a lot too, but is very hard to find (I was good toward the end, not earlier).
There is a type of innate ability that helps people learn faster, but it is no more a factor in the LSAT than anything else. The difference is that (1) the preparation material is pretty bad, and (2) most people fail to put in much time studying because they believe the lie that it is an intelligence test (or the lie that it mostly a matter of learning the "tricks", a la the SAT).
I don't think it's a bad test. I did very well on it so maybe you want to take that into account, but I did very well on the SAT too (and taught it) and I think that's about the worst test out there. I'd rather Admissions didn't focus so much on the LSAT (or GPA) and looked at the whole package, but the alternative is to make the process a lot more complicated and time-consuming. And more subjective.
Anyway, a different test probably won't be an improvement.
Toney, I'm sorry but you are dead wrong. There is no "paradox" or nor is there a "theory" to debunk.
The LSAT predicts 1L grades.
That's not speculative or theoretical. It is factual. It's also a big part of the reason law schools use the test in the first place.
9:15 - A couple of things (sorry for the long post):
1. It wouldn't be the first time I've been dead wrong. In fact, it's not even the first time today (I suffered a lexiconic defeat this morning over whether the word 'funnily' is a word - it is).
2. To say it's a fact that LSATs predict 1st year grades (meaning it works every time, without fail) is dead wrong. The fact that I had a great first semester and a lousy second semester proves that. The difference in this one example is that I put less effort into school. My LSATs didn't change.
I assume what you meant is that LSATs are predictive of grades, not factual indicators. I'm willing to concede this a bit, since it probably really is the best existing quantitative index (when combined with undergrad GPA). But the fact still remains that when law schools grade a curve (maybe they all do... in engineering, they don't), the pool that you give yourself is going to perform the same regardless of whether the LSAT range is 165-175 or 145-155.
Look, I did some digging and found this(pdf). This is a study done by LSAC themselves. Look at Table 2. In 2002, the correlation coefficient between LSAT scores and first year grades was .39. Correlation coefficients are given in a range of -1 (pure inverse proportionality - x goes up, y goes down) to 1 (purely proportional - x goes up, y goes up). 0 is random association. .39 is a lot closer to 0 than to 1. Further, there was no correlation between LSAT scores and any law school or professional metric beyond the first year.
My point here is that people put too much weight on the LSAT. If you need a number to index students by, the LSAT is the best you can get. But sitting down and interviewing people, getting an idea of their work ethic, their wit, etc., would be a FAR better predictor of first year grades (and beyond the first year as well).
It's pretty well accepted that LSAT scores (in the aggregate) correlate to first year grades (in the aggregate). Now, what's true of the whole is not necessarily true of any particular part, nor is what is true of a part necessarily true of the whole. I think that 9:15's point, however, is that what is true of the group of high LSAT scorers is also true of the group of people who get good 1L grades -- they're generally (although not in ever case) the same people.
Someone asked above whether attempts to revise the LSAT could be a stealth attempt to broaden admissions criteria without compromising ranking. That's an interesting observation. Someone else remarked that if the LSAT predicts grades, and grades translate to opportunities upon graduation, then revising the LSAT alone isn't going to get us very far. If we want to place the most talented students in the "best" jobs, we should also revise the curriculum to reward students with skill sets other than rapid fire analytic thinking and the ability to apply black letter legal doctrine. That is another interesting observation. But I don't think the LSAT-as-predictor-of-1L-grades question is subject to very much debate.
Toney, I don't mean this in any demeaning way, but have you taken a research methods or statistics course? I only ask because when X is a predictor of Y, it doesn't mean there's a 100% correlation. Why do you assume that? And an eta of .39 is a moderate correlation. You can't say the LSAT does not predict law school grades with a .39!
11:35
"And more subjective."
I don't welcome subjectivity. Subjectivity was introduced in admissions originally as a way to limit Jewish enrollment, and some claim that it now limits Asian enrollment at the undergraduate level.
In my opinion, academia is the last bastion of objectivity (well maybe government jobs as well). There's enough subjectivity in the real world (i.e., the job market). And having worked in the private sector and participated in hiring decisions, subjectivity often leads to curious and often unfair results. Things like who's better looking are discussed, I kid you not.
Armen - I was actually a math major with an emphasis on data modeling.
I haven't ever said that LSATs don't predict 1L grades... I'm just pointing out that there are better ways of doing this, though probably not quantitatively. I do think people put too much weight on their predictive value - and if I knew my potential future was going to based on an eta of .39 vs. my LSAT performance, I would be nervous!
So I'll say it again, and hopefully we can put our arguing behind us: LSATs (when combined with uGPA) are the best quantitative predictor of 1L grades.
Now having said that, it interesting that there is no correlation beyond 1L grades...
Patrick - I've heard a bunch of controversy over just how effective LSATs as predictors of 1L grades are. No one feels they are completely ineffective obviously, but just how effective they are is a source of endless debate and expensive studies, mostly because of the amount of weight put on them in the admissions process.
Now, I'm not complaining too much, because I did well and got into Berkeley. But if you are really interested in improving the student body at law schools, you have to look beyond quantitative measures to adversity, etc.
"The LSAT predicts 1L grades.
That's not speculative or theoretical. It is factual. It's also a big part of the reason law schools use the test in the first place."
LOL. Someone needs a lesson in the definition of "fact."
Hey Toney, where'd you learn to be such an idiot? Idiot school? Oooooh, Burrrrrrn!
12:13 -
The downsides of a more subjective evaluation are easy to rattle off. However, when filling most positions that require significant qualifications to do well, standardized, objective selection criteria only get you so far. That's why no responsible employer would hire you without a series of interviews. You can avoid some forms of unfair discrimination by insisting on purely objective criteria, but by doing that you ensure that you won't have the information to decide who is actually the best candidate. Furthermore, it is possible for the objective criteria to introduce a different form of unfair discrimination into the process (see the criticisms that some groups consistently do worse on some of these standardized tests).
A subjective system could take account of the full range of relevant information about a person. An objective system will always sacrifice relevant information. The question is whether the benefit of the new relevant information would outweigh the time & expense it takes to get it and analyze it, and more importantly, whether we could effectively limit the unfair discrimination that is bound to come with increased subjectivity. I would like to see objective criteria used to make a first cut, and then move to a holistic, subjective evaluation. But maybe you think I am not cynical enough about the potential for abuse.
LSAT grades have nothing to do with 1L grades, unless your 1L tests were multiple choice. A lot of this is because in a multiple choice the issue is laid out for you, but in a law school test you have to come up with the issues yourself. I am a whiz at standardized tests and got a 178 on my LSAT -- my first law school grades? Straight Ps.
New Rule: Anecdotal evidence shall not be used to rebut valid scientific studies establishing a correlation.
E.g., just because you're a firearm dischargin' hick from Idaho who votes for Democrats doesn't mean Idaho's a blue state.
Here's a (beautiful) piece of anecdotal evidence that destroys Armen's assertion.
The LSAT needs a serious overhaul. It can easily be beaten with a little preparation. http://LSATtips.blogspot.com
No offense to Steve, but all the LSAT teachers I know believe that the type of TIPS he gives to EASILY BEAT the LSAT lead to modest (but real) gains. To make significant gains, it requires a lot (not a little) preparation. And the best tip is that people who talk of "tips" and "beating the LSAT" and big gains for little effort tend to want money from you. But that's not only anecdotal, but a blatant argument from authority, so take it for what it's worth.
Sorry about the grammar.
Post a Comment
<< Home