One of the most common questions I get from you guys new to the LSAT is: "Which LSAT's month is the easiest/hardest?"
Anyone who knows anything will tell you, "They're all the same. No month's LSAT is particularly easy or difficult."
If you're especially savvy, you won't be satisfied with that. You'll look at my
.
1. include my analysis of the raw score conversion charts, which supports the claim that December exams consistently have the easiest "curve" and June exams consistently have the hardest "curve."
2. include my lengthy email conversation with the blog reader who brought this to my attention.
I should mention right off the bat that the differences we're talking about are only a point or two out of 180. Additionally, I still think that the June exam is the best for admissions purposes (see
However, the differences covered in this blog post are consistent for the past 8 years (and in some cases, beyond that). Even an average difference of a point or two is significant.
Analyzing the Past 8 Years (aka how do you know I'm not making this up?)First, I did a month-by-month comparison of the raw score conversion charts for the past 8 years of exams: PrepTest 37=June 2002 through PrepTest 59=Dec 2009 (present). I analyzed the June, September/October, and December exams on 5 data points.
The following is the average number of questions you could answer incorrectly (by month) and still achieve scaled scores of 160, 165, 170, 172, and 180, respectively, over the past 8 years:
In case you can't see the image, here's that data in text form:
Jun: 24.125, 16.5, 10, 8, 1.5
S/O: 24.875, 16.5, 10.25, 8.25, 1.75
Dec: 26.25, 18.125, 11.375, 9.25, 2(I didn't examine any data points between 173 and 179 because each exam lacked at least one of these scores. In other words, there were too many cases where there was no raw score that converted to one of those scores out of 180.)
In all cases for averaged raw score conversions over this period (for these data points), one could answer a greater number of questions incorrectly on the December exam than on either the June or the Sept/Oct exams, yet still achieve the same score out of 180.
In 4 out of 5 cases, the Sep/Oct exam was slightly "easier" than June, as well. In the other case, they were perfectly tied.
To put it another way, in 4 out of 5 cases, the June exam required the most correct answers to achieve a particular scaled score. In the other case, it was perfectly tied with Sep/Oct.
How Big Is This Trend? Does It Also Hold For The 8 Years Before That?To determine this, I also analyzed PrepTest 11=June 1994 through PrepTest 36=December 2001 by month: June, September/October, and December on 2 data points, just to see if the general trend held true in the 8 years prior to June 02:
The following is the average number of questions you could answer incorrectly over that period (by month) and still achieve scaled scores of 160 and 170, respectively:
In case you can't see the image, here's that data in text form:
Jun: 27.875, 12
S/O: 29.125, 12.875
Dec: 27.625, 14.125These findings are somewhat surprising, given what I found for June 02-December 09 (above).
From 1994-2001, it was
even easier, on average, to get a 170 in December than in June than from 2002-2009 (The older period had a difference of 2.125 raw score points at the 170 data point, while the more recent period only had an average difference of 1.375 for the 170 data point.)
In other words, the June exam was not only still the toughest to get a 170 on in this period, but it was even tougher to get a 170 in June over this period than in the more recent period.
I also found that the September/October exam's "easiness" was closer to December than it had been in the more recent period.
However, my
most surprising finding for this period: it was actually a bit easier to get a 160 in September/October than in either June or December, a trend that certainly hasn't held true in the past 8 years.
How Do February LSAT Conversions Compare To Those of Other Months?After all this analysis of June, Sep/Oct, and Dec, I started wondering how February exams compare. Unfortunately, no February LSATs have been released since 2000, so our sample size is both older and smaller than it otherwise would have been.
However, I did what I could. I looked specifically at the conversion charts for nearly every exam from February 1994 - December 2000. 7 February exams were released over this period. (I excluded the entire year of 1998 because that year's February exam was not released.)
I didn't compare the February exam data with current exam data because it currently takes more questions correct to get a particular scaled score (out of 180) across the board than it did in the past (
data).
The following is the average number of questions you could answer incorrectly over that period (by month) and still achieve scaled scores of 160 and 170, respectively:
In case you can't see the image, here's that data in text form:
Feb: 27.166, 12.333
Jun: 27.833, 11.833
S/O: 29.5, 12.833
Dec: 27.667, 14.166At the 160 data point, the Feb exam was the most difficult (required the most questions correct to get a 160). At the 170 data point, it was the second most difficult.
Of course, as we know from looking at the entire 8-year period from 1994-2001 period (previous section), what was true of the 160 data point was not true of the present day.
We have no way of knowing whether Feb exams have continued to be relatively difficult, of course, since they're no longer released. However, it's still something to keep in mind.
***
The following email exchange includes some off-the-cuff hypothesizing about the reasons that December exams consistently allow one to have a greater number of incorrect answers, yet still achieve the same scaled score. (The data above also raises questions about why June exams consistently require one to have a fewer number of incorrect answers to achieve the same scaled score.)
Unfortunately, we have more questions than answers as to "why."
Is it because the December tests are consistently harder and June tests are consistently easier?
Looking at the exams, it doesn't seem that way. Without a large sample size, it's difficult to say. All we can say is that difficulty of particular exams and questions is, to a certain extent, subjective.
Additionally, one would think LSAC aims to make each exam of equal difficulty to avoid too much variation in the raw score conversion charts. After all, LSAC wants to maintain the equivalency of scores from different exams.
Is it because the December/June pools of LSAT-takers are "different" in some way? Maybe.
Is it because LSAC abducted Elvis? Maybe.
Any hypothesis about it is just that - a guess.
As I've said before, statistics isn't my thing - it's much easier for me to take averages, as I did above, than to tell you the reason the numbers appear as they do - that's a whole different ball game.
I've asked LSAC to shed some light on these questions. Here's part of LSAC's response:
"The differences you describe are very small and represent the type of minor fluctuation we expect to observe."
I still think the differences are important enough to warrant this blog post.
My emails with the blog reader (Christopher) who brought this to my attention:
Christopher:I read your posts about the LSAT "curve" (that's not really a curve) and then looked at the raw score conversion charts - it seems to me from quick analysis that the December LSAT consistently seems to be "easier."
Easier is a relative term I suppose, but let's say we look only at the upper end of the scores - ie. 170-180. It's hard to get an exact comparison since there are so many blanks in the upper ranges from year to year but
it seems that consistently in a given year with the December test, you can afford to get more questions incorrect to achieve the same scaled score.
Let's say we look at 180 and 172 which are both uninterrupted (no blanks) since June 2002. Basically in every instance, you could afford to get more wrong in December than in June (granted the differential is only 1-2 points). 2005 seems to be an odd year, but for the rest if you pick a score between 172-180 where there are three data points, overwhelmingly it seems to indicate that December is more forgiving.
I guess you could make the argument that the December test is in fact "harder" and thus someone who scored 94/101 in Dec '09 would most likely score 96/101 in Jun '09 (achieving 176 on both tests) - BUT given a small chance of human error (you pick the right answer, but fill in the wrong bubble) or let's say you run out of time and leave the last question on every section blank no matter how easy or hard it may be - aren't you better off taking the December test if you're aiming for 175-180?
Me: For the last 8 years, at least, the data supports what you suggested.
It would certainly be worthwhile to take in December if one's primary goal were to safely achieve high scores - less punishment for bubbling errors, or for any errors at all, of course. I would expect someone scoring at that level wouldn't have significant time issues, though.
There are considerations that, generally speaking, might lead one to avoid December, though. An admissions-related consideration is that Dec is rather late in the cycle to apply to a T14 school, especially for T5 schools. Of course, a 175-180 would more than eliminate any drawback of applying that late. However, if something goes wrong in December, you're basically out of luck for that cycle (for many top schools).
(You could always take in December and apply in the following fall, but most people don't plan that far ahead, and most aren't willing to wait that long.)
At the same time, though, if you're capable of getting 175-180 in Dec, you can probably also get it in Feb, June, or Sept/Oct. At the same time, better safe than sorry, though.
Christopher:All the points you mention are definitely true if you have other concerns than just scoring high - i.e. admissions/timing concerns. My question was more just specifically if your intent was to try and get as high a score as possible (and timing was less of an issue).
Thinking more about this - I wonder if it's due to the fact that more people take the test in December but the ratio of high scorers to low scorers doesn't scale equivalently at the same rate.
i.e. if the ratios were the same, and when the number of test takers doubled it was as if everyone grew a twin with the exact same scoring ability, then it would make no difference which month you took the test in.
However, conversely (and what the data would seem to suggest, although you wouldn't be able to prove it) - maybe when twice as many people take the test in December, there's a disproportionately increased number of "average test-takers", but less (as a percentage of the total) "high-scoring" test takers. Therefore if you were a "high scorer" it would be in your benefit to take December because there are a smaller percentage of people who are at your ability or better.
This latter thought is just a hypothesis - not sure how valid it is given that I did a quick glance at scores in the ranges around 130 and it still seems that "Dec" is easier.
Me:
If you look at the
data from LSAC on the number of test-takers for each exam, you'll find that the September/October exam is the most popular, by far.
I hypothesize that there are fewer strong test-takers in the December pool because it's late in the cycle. Perhaps a lot of the weaker test-takers who take, or planned to take, the September/October exam retake it in December. Generally, the stronger test-takers from Sept/Oct wouldn't need to retake because they did fine.
Christopher:
I'm inclined to agree with your hypothesis about December test-takers. I think it's a combination of what you mentioned + the fact that (for college kids) Sept allows for summer prep whereas Dec doesn't. Also, Dec runs into the problem of conflicting with exam study.
Additionally, under our current tough economic conditions, I would guess a lot of people may not think about going to law/grad school until they realize that finding a job is harder than it seems. For May graduates they may not realize this until the summer winds down and the end of the year approaches and suddenly they find themselves in a position where they want to take the LSAT, GMAT etc. "just to leave their options" open. Once again though, unfortunately there's really no way to prove this, though.
Photo by bensonkua / CC BY-SA 2.0
The LSAT accommodations process is a difficult one. LSAC requires test-takers to jump through a series of hoops, whether their disabilities are physical or mental. As a result, LSAC has faced a series of lawsuits.
