The scores from the September 2017 LSAT have been released and we have the scoring scale in hand. We’ve already done a breakdown of the Logical Reasoning sections from this test. I wanted to take a moment to analyze the scoring scale and what it tells us about the logical difficulty of this exam. As my colleague Dave Killoran has written elsewhere, each LSAT scoring scale is adjusted to fit the difficulty of that particular test.
“Although the number of questions per test has remained relatively constant over the years, the logical difficulty of each test has varied. This is not surprising since the test is made by humans, and there is no precise way to completely predetermine logical difficulty. To account for these variances in test “toughness,” the test makers adjust the Scoring Conversion Chart for each LSAT in order to make similar LSAT scores from different tests mean the same thing.”
So, the looseness or tightness of the scale will reflect the logical difficulty of the exam. A looser scale with a harder exam and a tighter scale with an easier exam. It can have a significant impact on final scores. With that in mind, how did this one turn out?
By itself, it is hard to determine what the scale tells you about the general nature of the exam. You can clearly note individual outcomes. To get a perfect score of 180, you are only able to miss 2 questions. 11 questions to get a 170, 26 questions to get a 160, and so on. But what does that say for the test’s overall difficulty? To answer that we need to consider it relative to other exams.
Comparing to Past Exams
Let’s compare this scale to the scoring scales from prior LSATs. What you’ll find is this scale is fairly normal at the upper end of the scoring range, 165 and up. But, it becomes exceptionally tight, almost record-breaking-ly so, for scores nearer the middle, from 145 to 160. What that means is that this LSAT was easier than average for mid-level scorers. There was a large amount of widely-accessible content. However, at the higher levels, there was enough general difficulty to sufficiently punish those who typically target ~170. Thus softening the scale at and around that number.
Let me elaborate on a few of the more notable aspects. Largely I’m going to compare this test to two past tests. The first is the June 2017 LSAT, which is the last released test. The second is the last test where missing 43 meant a 150 and missing 34 meant a 155: September 2009. Yes, you read that right. There hasn’t been a scale as punishing for those in and around the 150s in 8 years!
Scores of 145, 150, and 155 required minimum raw numbers of 49, 58, and 67, respectively. This is identical to the September 2009 LSAT. When you compare it to the June 2017 test you see notable differences! To achieve those same results in June, you only needed 46, 55, and 65 questions correct, respectively. That’s 2-3 fewer questions required at each scoring outcome. The 2017 September test proved significantly easier for test takers at those levels than the June exam that preceded it.
More questions needed to attain a scaled score means that test takers at that level performed better on the whole (i.e. were challenged slightly less) than test takers at that same level on a different exam. The mid-range on the September 2017 LSAT was not just strict, but the strictest we’ve seen in nearly a decade!
Do the two seemingly disparate exams (June and September 2017) ever cross paths? In fact, they do! At 160, both tests required 75 correct (you could miss 26 total). So, what’s happening here? How have we gone from a test so much easier in the mid-range (September) to catching up to one that was for a long time comparatively much more difficult (June)?
The bulk of the bell curve, those in the 145 to 155 range, found June to be a relative challenge and September to be far simpler. But, the September exam ultimately had more overall or high-level difficulty than June. This pushed more people into the middle range in September. Whereas, with the June test they could break out of it by facing less extreme challenge. Simply put, June had a flatter natural bell curve. To “fatten it up” in the middle, fewer questions correct were required for those scores. September’s results yielded a steeper bell. So, to “mash the peak down,” so to speak, greater requirements were imposed there, squeezing people out to either side. We’ll continue to see the ramifications of this as we climb higher.
Scores of 165, 170, and 175 on this September’s test required a minimum of 83, 90, and 95 questions correct, respectively. Let’s again compare those raw results with their most interesting, and recent, touch point, June 2017. To score a 165 or 170 in June you needed 85 and 92 correct answers, respectively. And what about a 175? That wasn’t a possible score for the June 2017 LSAT. However, 95 questions correct (September 2017’s 175) would have given you just a 173. You should see right away that by the time we reach 165, June’s scale has tightened dramatically, while September’s has gotten far more forgiving.
Of course, if you recall our discussion of the June 2017 scoring scale, you may remember that June wasn’t just strict at the higher levels, it was itself something of a historical standout. Not since the September 2008 LSAT could you miss so few questions to get a 165 and above! The difference we see when we compare this year’s two most recent tests at the upper end is less about September’s belated generosity, and much more a product of June’s dizzyingly-heightened requirements.
The Magic Number
Let’s talk at last about the magic number everyone focuses on when discussing scales: 170. The September 2017 LSAT was a -11 for a 170, which is entirely in line with recent years (aside from, again, the outlier that was June 2017’s singular and miserly -9). Test takers found themselves on familiar, and reasonably friendly, ground. To restate the presumed reason outlined above: this tells us that the September LSAT had enough legitimately daunting content to keep even the best test takers at bay, unlike its predecessor from a few months ago where far less proved formidable.
The Impossible Score(s)
Finally, note that only a single score is impossible in the 170s: a 178. Pretty standard. But again, compare it to June 17, where both a 175 and 179 were impossible. People performed so well at 170+ in June that there simply weren’t enough questions available to allow the usual 10 possible results from 171-180. This forced the test makers to only allow 8 scores above 170. They then gave a single-point buffer to 180, as opposed to September’s more generous -2 allowance. A single score removed in that range is fairly normal. Two is far less and speaks to the vice-like grip at the highest reaches that notorious June 2017 LSAT displayed.
So all in all the September 2017 LSAT’s scale was forgiving for upper-range testers, but punishingly less so for students in the middle.
The takeaway from a bell curve where the middle suffers most is that, for the majority of people out there, this scale hurt. As the test becomes more competitive in the 148-160 range, and more of those traditionally-middled students come better and better prepared—typically by wisely investing in courses and tutoring—the test makers are forced to fight back, raising the bar that traditionally has separated the majority from the elite. In June we saw them target the top scorers with the scale, as the test itself failed to do so. September saw a test better equipped to naturally punish even the best and brightest, but fairly lax on the broader audience, and that’s exactly who LSAC clobbered with this scale as a result.
The only reliable truth then is this: no matter the test, no matter the scale, you’re in a competition with both your peers and the test makers determined to place you orderly among them. If you want to climb your way to the next level you can’t count on a friendly scale or lucky breaks; you have to be better prepared to conquer whatever comes your way. And we can help with that.