Easiest LSAT Curve: December | Hardest LSAT Curve: June

LSAT Blog Easiest LSAT Curve June Feb Oct DecOne of the most common questions I get from you guys new to the LSAT is: "Which LSAT's month is the easiest/hardest?"

Anyone who knows anything will tell you, "They're all the same. No month's LSAT is particularly easy or difficult."

You then ask, "But what about the curve?"

Answer: "It's not actually curved. It's equated."

If you're especially savvy, you won't be satisfied with that. You'll look at my LSAT PrepTest Raw Score Conversion Charts and calculations of what it takes to get an LSAT score of 160 or 170.

Using that data, you'll find that the December exam consistently has the easiest "curve," and the June exam consistently has the hardest.

In this blog post, I do two things:

1. include my analysis of the raw score conversion charts, which supports the claim that December exams consistently have the easiest "curve" and June exams consistently have the hardest "curve."

2. include my lengthy email conversation with the blog reader who brought this to my attention.

I should mention right off the bat that the differences we're talking about are only a point or two out of 180. Additionally, I still think that the June exam is the best for admissions purposes (see February vs. June LSAT and June vs. October LSAT.)

However, the differences covered in this blog post are consistent for the past 8 years (and in some cases, beyond that). Even an average difference of a point or two is significant.


Analyzing the Past 8 Years (aka how do you know I'm not making this up?)
First, I did a month-by-month comparison of the raw score conversion charts for the past 8 years of exams: PrepTest 37=June 2002 through PrepTest 59=Dec 2009 (present). I analyzed the June, September/October, and December exams on 5 data points.

The following is the average number of questions you could answer incorrectly (by month) and still achieve scaled scores of 160, 165, 170, 172, and 180, respectively, over the past 8 years:


LSAT Blog December Curve Comparison Averages 2002-2009






In case you can't see the image, here's that data in text form:

Jun: 24.125, 16.5, 10, 8, 1.5
S/O: 24.875, 16.5, 10.25, 8.25, 1.75
Dec: 26.25, 18.125, 11.375, 9.25, 2


(I didn't examine any data points between 173 and 179 because each exam lacked at least one of these scores. In other words, there were too many cases where there was no raw score that converted to one of those scores out of 180.)

In all cases for averaged raw score conversions over this period (for these data points), one could answer a greater number of questions incorrectly on the December exam than on either the June or the Sept/Oct exams, yet still achieve the same score out of 180.

In 4 out of 5 cases, the Sep/Oct exam was slightly "easier" than June, as well. In the other case, they were perfectly tied.

To put it another way, in 4 out of 5 cases, the June exam required the most correct answers to achieve a particular scaled score. In the other case, it was perfectly tied with Sep/Oct.


How Big Is This Trend? Does It Also Hold For The 8 Years Before That?

To determine this, I also analyzed PrepTest 11=June 1994 through PrepTest 36=December 2001 by month: June, September/October, and December on 2 data points, just to see if the general trend held true in the 8 years prior to June 02:

The following is the average number of questions you could answer incorrectly over that period (by month) and still achieve scaled scores of 160 and 170, respectively:


LSAT Blog December Curve Comparison Averages 1994-2001





In case you can't see the image, here's that data in text form:

Jun: 27.875, 12
S/O: 29.125, 12.875
Dec: 27.625, 14.125


These findings are somewhat surprising, given what I found for June 02-December 09 (above).

From 1994-2001, it was even easier, on average, to get a 170 in December than in June than from 2002-2009 (The older period had a difference of 2.125 raw score points at the 170 data point, while the more recent period only had an average difference of 1.375 for the 170 data point.)

In other words, the June exam was not only still the toughest to get a 170 on in this period, but it was even tougher to get a 170 in June over this period than in the more recent period.

I also found that the September/October exam's "easiness" was closer to December than it had been in the more recent period.

However, my most surprising finding for this period: it was actually a bit easier to get a 160 in September/October than in either June or December, a trend that certainly hasn't held true in the past 8 years.


How Do February LSAT Conversions Compare To Those of Other Months?

After all this analysis of June, Sep/Oct, and Dec, I started wondering how February exams compare. Unfortunately, no February LSATs have been released since 2000, so our sample size is both older and smaller than it otherwise would have been.

However, I did what I could. I looked specifically at the conversion charts for nearly every exam from February 1994 - December 2000. 7 February exams were released over this period. (I excluded the entire year of 1998 because that year's February exam was not released.)

I didn't compare the February exam data with current exam data because it currently takes more questions correct to get a particular scaled score (out of 180) across the board than it did in the past (data).

The following is the average number of questions you could answer incorrectly over that period (by month) and still achieve scaled scores of 160 and 170, respectively:








In case you can't see the image, here's that data in text form:

Feb: 27.166, 12.333
Jun: 27.833, 11.833
S/O: 29.5, 12.833
Dec: 27.667, 14.166


At the 160 data point, the Feb exam was the most difficult (required the most questions correct to get a 160). At the 170 data point, it was the second most difficult.

Of course, as we know from looking at the entire 8-year period from 1994-2001 period (previous section), what was true of the 160 data point was not true of the present day.

We have no way of knowing whether Feb exams have continued to be relatively difficult, of course, since they're no longer released. However, it's still something to keep in mind.
***

The following email exchange includes some off-the-cuff hypothesizing about the reasons that December exams consistently allow one to have a greater number of incorrect answers, yet still achieve the same scaled score. (The data above also raises questions about why June exams consistently require one to have a fewer number of incorrect answers to achieve the same scaled score.)

Unfortunately, we have more questions than answers as to "why."


Is it because the December tests are consistently harder and June tests are consistently easier?

Looking at the exams, it doesn't seem that way. Without a large sample size, it's difficult to say. All we can say is that difficulty of particular exams and questions is, to a certain extent, subjective.

Additionally, one would think LSAC aims to make each exam of equal difficulty to avoid too much variation in the raw score conversion charts. After all, LSAC wants to maintain the equivalency of scores from different exams.

Is it because the December/June pools of LSAT-takers are "different" in some way? Maybe.

Is it because LSAC abducted Elvis? Maybe.

Any hypothesis about it is just that - a guess.


As I've said before, statistics isn't my thing - it's much easier for me to take averages, as I did above, than to tell you the reason the numbers appear as they do - that's a whole different ball game.

I've asked LSAC to shed some light on these questions. Here's part of LSAC's response:

"The differences you describe are very small and represent the type of minor fluctuation we expect to observe."

I still think the differences are important enough to warrant this blog post.


My emails with the blog reader (Christopher) who brought this to my attention:


Christopher:
I read your posts about the LSAT "curve" (that's not really a curve) and then looked at the raw score conversion charts - it seems to me from quick analysis that the December LSAT consistently seems to be "easier."

Easier is a relative term I suppose, but let's say we look only at the upper end of the scores - ie. 170-180. It's hard to get an exact comparison since there are so many blanks in the upper ranges from year to year but it seems that consistently in a given year with the December test, you can afford to get more questions incorrect to achieve the same scaled score.

Let's say we look at 180 and 172 which are both uninterrupted (no blanks) since June 2002. Basically in every instance, you could afford to get more wrong in December than in June (granted the differential is only 1-2 points). 2005 seems to be an odd year, but for the rest if you pick a score between 172-180 where there are three data points, overwhelmingly it seems to indicate that December is more forgiving.

I guess you could make the argument that the December test is in fact "harder" and thus someone who scored 94/101 in Dec '09 would most likely score 96/101 in Jun '09 (achieving 176 on both tests) - BUT given a small chance of human error (you pick the right answer, but fill in the wrong bubble) or let's say you run out of time and leave the last question on every section blank no matter how easy or hard it may be - aren't you better off taking the December test if you're aiming for 175-180?


Me:
For the last 8 years, at least, the data supports what you suggested.

It would certainly be worthwhile to take in December if one's primary goal were to safely achieve high scores - less punishment for bubbling errors, or for any errors at all, of course. I would expect someone scoring at that level wouldn't have significant time issues, though.

There are considerations that, generally speaking, might lead one to avoid December, though. An admissions-related consideration is that Dec is rather late in the cycle to apply to a T14 school, especially for T5 schools. Of course, a 175-180 would more than eliminate any drawback of applying that late. However, if something goes wrong in December, you're basically out of luck for that cycle (for many top schools).

(You could always take in December and apply in the following fall, but most people don't plan that far ahead, and most aren't willing to wait that long.)

At the same time, though, if you're capable of getting 175-180 in Dec, you can probably also get it in Feb, June, or Sept/Oct. At the same time, better safe than sorry, though.


Christopher:
All the points you mention are definitely true if you have other concerns than just scoring high - i.e. admissions/timing concerns. My question was more just specifically if your intent was to try and get as high a score as possible (and timing was less of an issue).

Thinking more about this - I wonder if it's due to the fact that more people take the test in December but the ratio of high scorers to low scorers doesn't scale equivalently at the same rate.

i.e. if the ratios were the same, and when the number of test takers doubled it was as if everyone grew a twin with the exact same scoring ability, then it would make no difference which month you took the test in.

However, conversely (and what the data would seem to suggest, although you wouldn't be able to prove it) - maybe when twice as many people take the test in December, there's a disproportionately increased number of "average test-takers", but less (as a percentage of the total) "high-scoring" test takers. Therefore if you were a "high scorer" it would be in your benefit to take December because there are a smaller percentage of people who are at your ability or better.

This latter thought is just a hypothesis - not sure how valid it is given that I did a quick glance at scores in the ranges around 130 and it still seems that "Dec" is easier.


Me:
If you look at the data from LSAC on the number of test-takers for each exam, you'll find that the September/October exam is the most popular, by far.

I hypothesize that there are fewer strong test-takers in the December pool because it's late in the cycle. Perhaps a lot of the weaker test-takers who take, or planned to take, the September/October exam retake it in December. Generally, the stronger test-takers from Sept/Oct wouldn't need to retake because they did fine.


Christopher:
I'm inclined to agree with your hypothesis about December test-takers. I think it's a combination of what you mentioned + the fact that (for college kids) Sept allows for summer prep whereas Dec doesn't. Also, Dec runs into the problem of conflicting with exam study.

Additionally, under our current tough economic conditions, I would guess a lot of people may not think about going to law/grad school until they realize that finding a job is harder than it seems. For May graduates they may not realize this until the summer winds down and the end of the year approaches and suddenly they find themselves in a position where they want to take the LSAT, GMAT etc. "just to leave their options" open. Once again though, unfortunately there's really no way to prove this, though.

Photo by bensonkua / CC BY-SA 2.0



18 comments:

  1. Interesting point. Without doing any research I always had a gut feeling that an equated scoring system would result in the least popular exam also be the easiest exam to get a high score on.

    Although I see now that it's probably more accurate to say that the least popular among the high scorers (or the most popular amongst the low scorers) would be the easiest exam to get a high score on.

    I concur with what was said above and feel that the December exam most likley is the easiest exam to get a high score on since it's logically the least popular option.

    ReplyDelete
  2. Great question, John - the fish represents surprise = "huh?" (wide eyes, open mouth)

    I also just happen to think it's a pretty cool photo.

    ReplyDelete
  3. Did you get any answer back from the LSAC?

    ReplyDelete
  4. I'm including LSAC's response below and bolding the relevant part.

    "This is in response to your inquiry regarding conversion table differences you have noted across the four LSAT administrations that occur during each testing year.

    As I'm sure you know, a statistical process called test equating is carried out for every form of the LSAT to adjust for minor differences in difficulty between different forms of the test. Specifically, the item response theory ( IRT) true score equating method is applied to convert raw scores (the number correct) for each administration to a common 120 to 180 scale. A detailed description of this methodology can be found in Lord, F. M. (1980), Applications of Item Response Theory to Practical Testing Problems, Hillsdale, NJ: Erlbaum. The equating process assures that a particular LSAT scaled score reflects the same level of ability regardless of any slight differences in difficulty between different forms of the test. The equating process does not differ across the four customary LSAT administrations that occur each testing year. The differences you describe are very small and represent the type of minor fluctuation we expect to observe.

    I hope you find this information to be helpful. Thank you for your interest in the LSAT."

    ReplyDelete
  5. I know this is an old post now, but I've just come across it. One question -- isn't it possible that the tests that allow more wrong answers to get a particular score are actually *harder* not easier. That is, if we just compare December and June exams, and find the Dec. exams allow for more wrong answers to get a particular score, couldn't it be that the June tests tend to be easier, so test-takers simply get fewer answers wrong?

    ReplyDelete
  6. I find the previous poster's point to be valid. Harder tests should allow for more wrong answers...the interesting point is...why should the December test consistently come back with allowing more wrong answers??...my theory is that they want to give people who apply late a better chance at getting accepted vs. the early birds.

    ReplyDelete
  7. @ nick,

    That is insane to consider...

    Do you actually believe that LSAC manipulates the December test to allow for 1.5 more questions wrong on average to compensate for possible albeit likely chance that those applicants are applying later in the cycle? There is no real, non-aectodal proof that an early start on the cycle os objectively advantageous.

    ReplyDelete
  8. I am an LSAT instructor and I also agree that the June curve is generally a harder curve --- HOWEVER, before deciding that this means that June is the hardest test- take this into consideration: my company breaks down each released LSAT question and rates it on a scale of 1-4 (4 being the hardest)... those tests which have an "easier curve" generally have anywhere from 20-40 of the 4 star questions; those tests with easier curves generally have 1-6 of the 4 star questions. LSAC has stated that they curve in order to make up for any difference in "hardness" from test to test. Thus it is reasonable to assume that an easier curve actually means a harder question set, and a harder curve actually means an easier question set.

    ReplyDelete
  9. When LSAC says a curve may make up for the differences in hardness fro test to test, does that mean for example the experiment section was harder in one test than another? For example, if a test taker has two reading comprehension sections (1 experimental)are they curved with other test takers that also had reading comprehension?

    ReplyDelete
  10. I'm no expert on curves. but shouldn't we compare the actually levels of difficulty among tests before we reach any conclusion on whether a meaner curve could be translated to a harder test?

    ReplyDelete
  11. Clearly, the fact that one can get more questions wrong on the December exam, relative to the June exam, on average, indicates that the December exam is relatively HARDER. Moreover, the fact that less people take the December exam has no significant effect on the questions that compose it; the questions that compose it may well have been field tested on administrations in months other than December.

    ReplyDelete
    Replies
    1. I dunno. Obviously, test-takers performed poorly on December exams relative to the June exams, but like Steve wrote, this may have more to do with the makeup of the pool of test-takers for the December administrations than the difficulty of the questions. If it's always a bunch of morons that take the December LSAT that would certainly account of the generosity of the Dec curve.

      Delete
  12. I'm inclined to believe, similar to a couple of earlier statements made, that the December LSAT has a higher number of people re-taking the test due to a desire to improve a score from one of the tests earlier in the year, which makes it reasonable to believe that, given a desire for a higher score, the re-takers tend to be people who didn't score high the first time, and will (perhaps) still not be a high scorer in December (even if they do better than the first time). The December LSAT is the last chance for anyone who's serious about admission the following Fall to bolster his or her score. I also agree with the "current students like the timing of the June and October" hypothesis, which I infer is meant to suggest that students perform better on the LSAT than those who have been out of school for a while (I don't know if that is true, though). Additionally, students who are both high-scoring and the most motivated, may take an earlier LSAT to get the advantage on the rolling admissions process.

    If the date I'm posting this jumped out at you, yes, the December 2012 LSAT is tomorrow, and I'm someone who is retaking it (took it last June). I scored around the 50th percentile, and I'm looking to improve that. So I'm in the same category of those I hypothesized about.

    ReplyDelete
  13. But doesn't all of this assume that when they create the December LSAT they are only drawing from the experimental section of other December LSATs in order to have the makeup of the test-takers influence their gauge of what is and is not a difficult question?

    From my understanding, the LSAC equates by seeing how past LSAT populations scored on the experimental questions and uses that as a gauge to determine its curve for future LSAT administrations.

    ReplyDelete
  14. the key to the lsat is u gotta put in the hours, its that simple. more time spent studying = better score, everything else - and i mean everything - is irrelevent. scored 77th perecntalie june and 94th october, retaking december hoping to get 99th percentile

    ReplyDelete
    Replies
    1. Wow that's impressive! Good job! When you say put in the hours do you mean just full on practice exams back to back, or a mix of reading supplements and strategizing in addition to practice exams.

      Delete
  15. Hi,

    I was wondering what you thought about January exams. This is when I'm planning on taking it.

    ReplyDelete