The September 2017 LSAT Scoring Scale Analyzed

LSAT Prep

September 2017 Scoring Scale AnalysisThe scores from the September 2017 LSAT have just been released, and the scoring scale is hot off the presses. We’ve already done a breakdown of the Logical Reasoning sections from this test, and will have more section-specific discussions in the days ahead, but in the meantime I wanted to take a moment to analyze the scoring scale and what it tells us about the logical difficulty of this exam. As my colleague Dave Killoran has written elsewhere, each LSAT scoring scale is adjusted to fit the difficulty of that particular test:

"Although the number of questions per test has remained relatively constant over the years, the logical difficulty of each test has varied. This is not surprising since the test is made by humans, and there is no precise way to completely predetermine logical difficulty. To account for these variances in test "toughness," the test makers adjust the Scoring Conversion Chart for each LSAT in order to make similar LSAT scores from different tests mean the same thing."

So, the looseness or tightness of the scale will reflect the logical difficulty of the exam (looser scale with a harder exam, tighter scale with an easier exam), and can have a significant impact on final scores. With that in mind, how did this one turn out?

Here is the scale in full:

 

By itself, it is hard to determine what the scale tells you about the general nature of the exam: you can clearly note individual outcomes—you can miss only 2 question to get a 180, 11 questions to get a 170, 26 questions to get a 160, and so on—but what does that say for the test's overall difficulty? To answer that we need to consider it relative to other exams.

When we compare this scale to the scoring scales from prior LSATs, what we find is that this scale is fairly normal at the upper end of the scoring range (165 and up), but becomes exceptionally tight, almost record-breakingly so, for scores nearer the middle, from 145 to 160. What that means is that this LSAT was easier than average for mid-level scorers—there was a large amount of widely-accessible content—however at the higher levels there was enough general difficulty to sufficiently punish those who typically target ~170, thus softening the scale at and around that number. 

Let me elaborate on a few of the more notable aspects, largely by comparing this test to the last released LSAT (June 2017) and to the last test where you could only miss 43 for a 150, and 34 for a 155: September 2009. Yes, you read that right, there hasn't been a scale as punishing for those in and around the 150s in 8 years!

  • Scores of 145, 150, and 155 required minimum raw numbers of 49, 58, and 67, respectively. This is identical to that September 2009 LSAT, but when you compare it to the June 2017 test you see notable differences! To achieve those same results in June, you only needed 46, 55, and 65 questions correct, respectively. That's 2-3 fewer questions required at each scoring outcome, meaning this September test proved significantly easier for test takers at those levels than did the June exam that preceded it. Again, more questions needed to attain a scaled score means that test takers at that level performed better on the whole (i.e. were challenged slightly less) than test takers at that same level on a different exam. So, again, the mid-range on the September 2017 LSAT was not just strict, but the strictest we've seen in nearly a decade! 
  • Do the two seemingly disparate exams—June and September 2017—ever cross paths? In fact, they do! At 160, both tests required 75 correct (you could miss 26 total). So what's happening here? How, as we've climbed up the scoring scale, have we gone from a test so much easier in the mid-range (September) to suddenly catching up to one that was for a long time comparatively much more difficult (June)? The answer is that while the bulk of the bell curve—those in the 145 to 155 range—found June to be a relative challenge and September to be far simpler, the September exam ultimately had more overall, or high-level difficulty than June, which pushed more people into the middle range in September, while in June they could break out of it by facing less extreme challenge. Simply put, June had a flatter natural bell curve, and so to "fatten it up" in the middle, fewer questions correct were required for those scores. September's results yielded a steeper bell, and so to "mash the peak down," so to speak, greater requirements were imposed there, squeezing people out to either side. And we'll continue to see the ramifications of this as we climb higher...
  • Scores of 165, 170, and 175 on this September's test required a minimum of 83, 90, and 95 questions correct, respectively. Let's again compare those raw results with their most interesting, and recent, touch point, June 2017. To score a 165 or 170 in June you needed 85 and 92 correct answers, respectively. And what about a 175? That wasn't a possible score for the June 2017 LSAT, however 95 questions correct (September 2017's 175) would have given you just a 173. You should see right away that by the time we reach 165, June's scale has tightened dramatically, while September's has gotten far more forgiving. Of course, if you recall our discussion of the June 2017 scoring scale, you may remember that June wasn't just strict at the higher levels, but was itself something of a historical standout: not since the September 2008 LSAT could you miss so few questions to get a 165 and above! So the difference we see when we compare this year's two most recent tests at the upper end is less about September's belated generosity, and much more a product of June's dizzyingly-heightened requirements. 
  • Let's talk at last about the magic number everyone focuses on when discussing scales: 170. The September 2017 LSAT was a -11 for a 170, which is entirely in line with recent years (aside from, again, the outlier that was June 2017's singular and miserly -9). So test takers found themselves on familiar, and reasonably friendly, ground. To restate the presumed reason outlined above: this tells us that the September LSAT had enough legitimately daunting content to keep even the best test takers at bay, unlike its predecessor from a few months ago where far less proved formidable. 
  • Finally, note that only a single score is impossible in the 170s: a 178. Pretty standard. But again compare it to June 17, where both a 175 and 179 were impossible. People performed so well at 170+ in June that there simply weren't enough questions available to allow the usual 10 possible results from 171-180, forcing the test makers to only allow 8 scores above 170 (and then giving just a single-point buffer to 180, as opposed to September's more generous -2 allowance). A single score removed in that range is fairly normal. Two is far less so, and again speaks to the vice-like grip at the highest reaches that notorious June 2017 LSAT displayed. 

So all in all the September 2017 LSAT's scale was forgiving for upper-range testers, but punishingly less so for students in the middle. 

The takeaway from a bell curve where the middle suffers most is that, for the majority of people out there, this scale hurt. As the test becomes more competitive in the 148-160 range, and more of those traditionally-middled students come better and better preparedtypically by wisely investing in courses and tutoring—the test makers are forced to fight back, raising the bar that traditionally has separated the majority from the elite. In June we saw them target the top scorers with the scale, as the test itself failed to do so. September saw a test better equipped to naturally punish even the best and brightest, but fairly lax on the broader audience, and that's exactly who LSAC clobbered with this scale as a result.

The only reliable truth then is this: no matter the test, no matter the scale, you're in a competition with both your peers and the test makers determined to place you orderly among them. If you want to climb your way to the next level you can't count on a friendly scale or lucky breaks; you have to be better prepared to conquer whatever comes your way. And we can help with that.

Have any questions or comments? Please post them below!