Law School Admission Council | Senior Test Specialist

Who exactly are the people who make the LSAT, and how do they think? What's going through their minds as they write it?

To answer these questions, I found a survey completed by Stephen W. Luebke, a Senior Test Specialist (formerly Director of Test Development), at the Law School Admission Council.

These excerpts will tell you about his responsibilities and background:

Job Title and Principal Duties:

Director of Test Development. Develop test questions and test forms for the Law School Admission Test, a major standardized admissions test required for applicants to most U.S. and Canadian law schools. Acquire test questions. Review, revise, rewrite, edit, and process test questions. Assemble and review test forms. Review and reply to challenges to test questions. Monitor statistical performance of test questions. Hire and oversee staff doing similar work. Participate in test-related research and in test planning and development with psychometricians.


Non-philosophical Background Pertinent to Your Job:

Some knowledge of statistics or educational measurement is useful in this job, but not necessary -- the necessary knowledge can be acquired on the job. I did some graduate course work in psychometrics while working.


How You Obtained Your Job:

I conducted a search for "education-related" jobs for which my graduate study and teaching experience provided an appropriate background. I had held several such positions since leaving teaching. I found an ad for a position at LSAC in the Chronicle for Higher Education. The initial position involved reviewing reading passages and handling copyright issues, but I was quickly moved into a management position and then became director of Test Development. After some reorganization my position became Senior Test Specialist.


Personal Characteristics and Philosophical Skills You Use in Your Present Position:

Reviewing, revising, and editing test questions draw heavily on the analytical skills taught in analytic philosophy -- close reading and analysis of texts, careful drawing of implications, identifying ambiguities and category mistakes. Since much of the LSAT consists of reasoning questions, my specific training in logic and informal logic was directly applicable, along with the general philosophical skill of argument analysis. Working with reading comprehension questions calls upon philosophical skill in understanding and analyzing texts. Other skills used include the ability to see multiple readings and multiple sides of an argument and a sensitivity to issues of fairness and the concerns of various population groups. Writing and editing skills and experience writing questions for classroom tests -- particularly multiple-choice questions -- are directly applicable to writing and revising questions, although for high-stakes admissions tests the standards are much higher than those usually applied in classroom tests. The job draws so heavily and directly on philosophical skills and training that one of my colleagues likes to call what we do "applied philosophy."


Comments:

Training in analytic philosophy, informal logic, and philosophy of language seem most directly applicable to reasoning testing. The major tasks in reviewing test questions are to make sure that they are clear and unambiguous, test for the appropriate skill, and have one and only one best answer. The job is intellectually challenging and many interesting philosophical questions arise in reviewing test questions.