This post is Part 4 of the "The LSAT Curve" series. The series starts with The LSAT Curve | Test-Equating at LSAC.Deciding which questions are "difficult"
Difficulty is all relative, right?
One way to make a question difficult is to include less-obvious conditional indicator words (using "if" and "then" kinda give the game away).
Another way is to make the question about a boring topic that few test-takers know about, like morality, aestheticism, or brown dwarf stars.
The issue, however, is that it's not always clear how difficult a question actually is in practice. The experimental section allows LSAC to determine how tens of thousands of test-takers perform on its latest questions.
Let's assume that LSAC gave a particular Logical Reasoning section to a bunch of test-takers on a given administration of the LSAT.
If only a small percentage of test-takers get question #17 right, and these are primarily the same test-takers who scored 170+ on the 4 sections of the exam that counted, then LSAC can safely assume that this is a question with a "Difficulty Rating" of 5.
If a large percentage of test-takers get question #3 right, and it's mainly the sub-140-scorers who get it wrong, then LSAC can safely assume this is a question with a "Difficulty Rating" of 1.
However, if a small percentage of test-takers get question #5 right, and these test-takers are mainly the sub-140-scorers, then LSAC can safely assume that something's very wrong with this question. This question is unlikely to make it into any part of the scored exam, at least, not in its current form. This question just isn't doing its job.
Similarly, if a large percentage of test-takers get question #20 right, but the 170+-scorers aren't getting it right, then something's probably wrong with this question. This question isn't doing its job either.
Cases where questions aren't doing their job are probably rare. LSAC's people generally know what they're doing, but it's worth thinking about the fact that LSAC trusts the opinions of its top scorers. Since they get the greatest number of questions right, most of them probably know what they're doing when it comes to the LSAT (or they're just really lucky).
***
Next week, we move on to Part 5: Creating the LSAT's Raw Score Conversion Chart (aka, the Curve)
Want to start at the beginning? Begin with The LSAT Curve | Test-Equating at LSAC.
Photo by sea-turtle / CC BY-NC-ND 2.0
Could you do a blog post about questions 15-21 on any given Logical Reasoning section?
ReplyDeleteI know that you have discussed how to approach these sections by doing the first part, last part, and then the middle, but I think it would be helpful to maybe just discuss what makes these questions so much harder than those surrounding them.
On all of my practice sessions, about one LR section a day while reading the LR Bible, I'm still having a noticibly more difficult time with this group of questions. I know they are harder and I know with more practice and studying I will get better but I would just like to know your thoughts on how the LSAT authors write these questions to be more difficult.
I know you don't usually answer direct questions about specific problems but I was hoping this would be different.
Thanks for all of your good work by the way. Your site is a life saver.