The Experimental Section and Difficulty of LSAT Questions

This post is Part 3 of the "The LSAT Curve" series. The series starts with The LSAT Curve | Test-Equating at LSAC.

"The LSAT is equated so that a test score obtained in the current year is comparable to scores obtained in previous years." - LSAC (Executive Summary)

Test-equating requires pre-testing.

After LSAC's elves write individual LSAT questions, they compile these questions into various 35-minute sections. If you've taken the LSAT before, you've already completed one of these sections as the hated "experimental section." In LSAC language, this is the "pretest" section where new questions are tested to:

provide test development staff with statistical information about each question, and with information about possibly ambiguous or misleading information in the question or in one or more of the answer choices. If problems are identified, either the question is discarded or it is revised and pretested again. All questions that pass the quality standards of a pretest administration are placed in the LSAT test question item bank. New test sections are assembled by selecting questions from this LSAT item bank. Each fully assembled test section is administered on one or more separate occasions for the purpose of pre-equating the new form.

Pre-equating is a statistical method used to adjust for minor fluctuations in the difficulty of different test forms so that a test taker is neither advantaged nor disadvantaged by the particular form that is given. Following each pre-equating administration, the statistical information about each question is reviewed to assure that the data support that the question is of appropriate difficulty, discriminates higher ability test takers from lower ability test takers, is unambiguous, and has a single best answer. When the test is given at a regular LSAT administration, but before final scoring is completed, statistical analysis is conducted one last time. Each question is evaluated using the same criteria that were applied following the pretesting and pre-equating administrations. If a problem is found, the question is eliminated from the test before final scoring and reporting are accomplished.

(Source: Page 2 of Policies and Procedures Governing Challenges to Law School Admission Test Questions. I divided this excerpt into two paragraphs. Just like some Reading Comp passages, it lacked paragraph breaks.)

Most of us know LSAC pretests questions in order to avoid using flawed questions that will later be withdrawn from scoring. This is what they mean by "is unambiguous and has a single best answer." (second para, second sentence)

However, the other parts of that particular sentence are worth noting.

Questions have various levels of difficulty
- LSAC is careful to make sure "the question is of appropriate difficulty" and "discriminates higher ability test takers from lower ability test takers."

LSAC wants to have a certain number of super-easy, easy, medium, difficult, and super-difficult questions on each exam (as part of the test-equating process).

It's not enough to just make a bunch of super-difficult questions and say whoever answers them right deserves to get into Harvard Law School.

How would you distinguish the students who got those questions wrong from each other?

For law schools, it's not enough to separate the 175+-scorers from everyone else. You also have to separate the 170-scorers from the 165-scorers from the 160-scorers, etc.

If you make every single question very difficult, some test-takers will get them all right, but most will just end up guessing. Obviously, LSAC won't know whether a test-taker guessed or not on a given question. However, if most test-takers end up guessing, the LSAT will no longer be a good predictor of law school performance (which is what it's supposed to be, after all), and the LSAT won't be able to adequately distinguish a good test-taker from a decent one from a bad one.

By including questions of various levels of difficulty, the LSAT meaningfully separates test-takers into multiple ability levels - not just 175+ and "everyone else."

For insight into how LSAC views the difficulty of various questions, check out the SuperPrep book's explanations (which are written by LSAC). After each question, you'll see a "Difficulty Rating" of anywhere from 1 to 5.

***

Next week, we move on to Part 4: LSAT Question Difficulty Ratings

Want to start at the beginning? Begin with The LSAT Curve | Test-Equating at LSAC.

Photo by practical owl / CC BY-NC 2.0