LSAT Test Composition: How and Why Difficulty Varies

Struggling with Consistency?

One question that arises frequently is, “Why am I not consistent in how I score on a certain section?” For example, a student may see his or her score in Logical Reasoning move around by a few questions with each practice test. Or, a student with consistent overall scores notices that sometimes she performs well on Reading Comprehension, other times Logical Reasoning, and other times Games. So, she ends up scoring about the same each time, but the way she gets there—that is, the correct number of answers coming from each section—seems to vary each time. Why does this happen?

Variability in scoring has many possible causes: how much sleep you got the night before, what you ate for breakfast, the room temperature, noise levels in the room, and so on. One factor that isn’t supposed to cause score variability is overall test difficulty. While this element does vary from test to test, the test scoring scale attempts to neutralize the effect of test difficulty on your final score (see The LSAT Scoring Scale Explained for more information on how this works).

However, while the test makers can account for overall difficulty by adjusting the scoring scale, they cannot account for the effects that variances in the difficulty of each section have on the overall performance of each test taker. To explain this idea, let’s talk a bit about how each LSAT is constructed.

Test Construction

While all LSATs strive to have a relatively similar level of overall difficulty, how they arrive at that difficulty varies. Let’s consider some released LSAT difficulty data from LSAC to show how the test is constructed, and then draw some conclusions.

Over the last several years, LSAC has released explanations to exactly four separate LSATs (February 1996, February 1997, February 1999, and February 2000) and in these explanations they have included an evaluation of question difficulty on a 1 (easy) to 5 (very difficult) scale. Because this information comes from the people who make the test (and have access to the results of all test takers), this information can provide some insight into the overall difficulty of each section.

The difficulty level statistics also allow us to create a composite of the overall average difficulty of each test, and two of the tests happen to have been rated at the exact same level of overall difficulty: the February 1997 LSAT and the February 1999 LSAT. Let’s examine those two LSATs and see how their composition would affect different test takers. Here are the average difficulty statistics for each test:

What the Numbers Tell Us

Let’s start with the two Logical Reasoning sections of each test. While the average difficulty of the two sections on each section is similar (February 1997 = 3.02; February 1999 = 3.00), the February 1999 LSAT is actually the more dangerous of the two tests. This is because one of its sections (LR Section #1), has a relatively high level of difficulty at 3.31. While this is offset by a relatively easier second section, many students would feel the extra difficulty in section #1, which could throw them off. The February 1997 LSAT features two very similar sections, neither of which has an especially high or low level of difficulty. These two sections would “feel” fairly standard, and psychologically would not be likely to set off any alarms.

The Games and Reading Comp sections of these two tests differ even more. February 1997 has a medium-difficulty Logic Games section and a hard Reading Comp section, whereas February 1999 has a very hard Logic Games section and a medium-difficulty Reading Comp section. These two tests, then, present very different tasks depending on your skills. If you are weak at Games but good at Reading Comp, you would prefer to be presented with a test like February 1997. On the other hand, if you are great at Games and struggle with Reading Comprehension, you would prefer to encounter the February 1999 LSAT.

Of course, the problem with the LSAT is that there is no way to know beforehand what the specific composition of your test will be. Yes, you will know the types of sections, the time limits, etc., but you will not know if you are facing a hard or easy Games section, and so on.

The Takeaway

1. When practicing, you will see a natural variance in your individual section scores as you encounter tests with differing compositions. For example, on a hard Reading Comp test, you will miss more, but maybe you will perform better on the Games section. On the next test you take the composition will likely be somewhat different, and your scores will move a bit accordingly.
2. If you go take the actual LSAT and have a subpar performance, one of the causes could be that you ran into a test that was not suited for your particular strengths and weaknesses. If you choose to sit for the LSAT again, one of the possible advantages is that you may very well get a test that is better suited to your abilities. Thus, you would most likely produce a score more in line with your expectations.

In any event, although the test makers would love to be able to consistently produce tests of perfectly identical difficulty (the holy grail of test making), it is impossible to do so.