Last week, I interviewed Stephen Harris, former LSAT question-writer and author of Mastering Logic Games. (Yes, he's written hundreds of the questions that appear in your books of LSAT PrepTests.)
Our discussion follows.
You can also:
1. Read ALL of my interviews with him (more than 5!)
***
1. You mentioned in our last interview that you worked on a freelance basis and that ACT/LSAC only bought the items (questions) they liked. What were you paid per item accepted, and, once you got the hang of things, how many items would you typically write for every accepted item (e.g. 1 out of 5)?
The LSAT format changed in the early 1990s, and I started writing items late in 1992. At first the pay rate was $75 for each accepted LR item, but it went up to $85 per item at some point, and that’s what it was when I stopped writing LSAT items in fall of 1997. I imagine that it has gone up a bit, but I will say that, compared to most other item writing gigs, that’s a pretty high rate even today.
An LR item writing assignment consisted of 10 items of various types – two weakeners, one assumption, etc. I probably averaged somewhere around 8.7 accepted items per assignment. Most writers I knew who averaged much fewer, say 6 or so out of 10, didn’t write many items, or didn’t write for long.
2. How is the item-writing process different today than when you worked as an item-writer? Does LSAC still use freelancers, or is it in-house?
I haven’t worked on the LSAT in a long time, but as far as I know it’s pretty much the same as it was back then. In fact, relying on contract item writers is now probably the industry norm. Since about 2000, the testing industry has exploded, and lots of folks who worked on the LSAT went to other places to oversee freelance item writers for other tests. I’m pretty sure that most tests work on the freelance model to a large degree. The GRE may be an exception; ETS may produce that one primarily in-house.
3. In our last interview, you mentioned that LSAC sent you to a training workshop where they gave you a guide covering all Logical Reasoning item types, and a list of what not-to-do. Can you elaborate on what each contained? What sort of feedback did LSAC offer on improving submitted items?
LSAC’s item writing guide was quite helpful. It made some general points about content, style, item stems, etc., and then they worked through exemplars of each item type. I seem to remember pointers for constructing good distractors, tips for disguising correct answers, etc. You quickly internalize most of it, so I don’t remember much about the details. But I remember thinking that somebody put a lot of work into it, and that it was helpful.
After each assignment was reviewed by my editor I had a phone call with him, and he would give me good, detailed feedback on all the items. I was given a clear reason why any rejected item was found unsuitable, and the accepted items that could have used some extra polish were discussed with me as well. It was in everybody’s interest to make the process as efficient as possible. I will say that, given my experience working on other tests, the LSAT was especially good in the feedback department.
4. What's the formal process by which an LSAT question goes from being an item-writer's draft of a question to becoming part of an actual scored exam?
Here’s how the process works, as best I understand it: first, an item writer sends the items to the testing company, which I believe for the LSAT is still ACT. Editors pick the ones they like, make whatever changes they think are appropriate, and then send the items to LSAC. The items receive another level of editing/review and then are placed on experimental test sections. The tests are administered and statistics are gathered. Then some of the items (the “good” ones) go to real test forms, while others go back for more editing and then another shot on an experimental section. [Ed. Read my series on the test-equating process.]
5. How are Reading Comprehension passages chosen?
I’m not exactly sure how RC passages are chosen for the LSAT; it’s probably pretty idiosyncratic. I am sure that writing assignments specify the general type of passage – natural science, humanities, etc., but after that it’s probably up to the writer. In each released LSAT test, you’ll find references to the articles that the RC passages are based on, so that will give you an idea of the kinds of sources that are used.
6. Are Logical Reasoning passages based on actual scholarship? A lot of them seem like real arguments.
One of the points that I do remember from the LR writer’s guide is that, to the extent the stimulus makes factual claims, they should be true, or at least reflect the current state of research in a field. So yes, many stimuli are based on real scholarship. But a stimulus might discuss a hypothetical vaccine, for example, that is not based on anything factual. So while it is the case that many LR items are based on actual scholarship, many are not. Often the basis for an item is a reasoning style, or a type of error, and then the item writer is simply looking for a topic to cloak the idea in. This is at least in part because item writing assignments specify the items that the writer needs to come up with by task, rather than by topic.
7. Would you agree that Logic Games have generally become easier over the years? Why don't we see pattern games anymore, and why do "rare" game-types show up so sporadically?
Well, I will say that preparing for logic games has gotten easier over the years, and I think this is really the phenomenon that your reader is remarking on. More specifically, if one were to grab your basic, pretty smart off-the-street person who wanted to go to law school and give her an AR section from an old test, and then a new one, she’d do about the same, on average. [Ed. Analytical Reasoning = Logic Games].
But, for someone who is actually studying to take the LSAT, there are definite differences. As your reader points out, the range of games seems to have shrunk significantly; virtually all recent games are instances of just a few types. But these games aren’t intrinsically easier than the less frequently seen games, in my opinion. Rather, the point is that one who prepares today has a smaller “strike zone” than test takers in the past, to use a baseball metaphor, and that mastery of a few game types is more likely to translate into success today than it used to.
8. Do you see the LSAT's emphasis moving in any particular direction now or in the future? Do you see LSAC making any major changes in any section, like the addition of Comparative Reading in June 2007?
I’m not sure where the LSAT is headed. I do know that they considered making it a computer-based test at one point, and that they were toying with the idea of items that were auditory - played through headphones - rather than written. For whatever reason, perhaps the cost of using testing centers, they decided to stick with the paper-and-pencil test. A safe assumption: the LSAT won’t change much, or quickly at least, unless people start complaining about it (even more than they do now).
9. Do you believe the LSAT to be a test of innate skill or something people can learn to master?
This is an interesting question. On any reasonable sense of “innate,” the LSAT does not test innate skills; they are all cultural skills. Sure, certain abilities tend to make one better or worse suited for possessing these skills; a good short-term memory, for instance, is undoubtedly helpful on the AR section. But all of the skills tested on the LSAT are cultural and acquirable, not innate. One interesting point that the question presupposes, but that I think is especially important, is that the LSAT is a test of skills, “know-how” rather than “know-that.” The LSAT is a lot more like tying your shoes, or playing a game of cards, than recalling chemistry facts - with each item, getting the correct answer is a matter of what you do, not what you know.
Now for some people certain skills come “naturally,” we like to say. Some folks are just really good at throwing a ball, others have a knack for long division. The rest of us, after some effort, can eventually learn the skill in question.
When it comes to the LSAT, some people are undoubtedly “naturals,” but pretty much anybody who wants to can master the relevant skills. It just involves a different, more reflective procedure for some of us than for others. Specifically, for us “learners” the key is to take a complex task that a natural LSAT test-taker performs intuitively, and to break it into its components so that we can learn the task piece-by-piece, until with practice and repetition the complex task becomes second nature, like tying our shoes. I’ve tried this approach with hundreds of LSAT students, and several shoe-tying kids, with great results all around.
10. Do you think certain groups of people are at a greater disadvantage preparing for/taking the test than others, particularly those of lower socioeconomic status?
Well, this is a complicated issue, and “disadvantage” is a funny word. There is no question that LSAT scores are correlated with family income, which I guess means socio-economic status. There are thousands of individual exceptions to this trend, but as generalizations go it’s pretty reliable. So, yeah, when it comes to taking the LSAT low socio-economic status probably puts one at a disadvantage, statistically speaking. But this is hardly unique to the LSAT, first of all.
Second, this doesn’t mean that the test is biased against those of low socio-economic status, any more than soccer and chess are biased against people who don’t grow up playing them. All of the skills necessary for the LSAT are acquirable by any literate person comfortable with English, although (regardless of one’s income level) the less familiar one is with these skills to begin with, the more work it will typically require to master them. But this is no different from most other aspects of life, and way “fairer” than some.
Consider, for example, how relevant height is to being good at basketball. And height is something that a person has virtually no control over. By contrast, test takers have much more control over whether they possess the key attributes conducive to success on the LSAT, and any initial disadvantage faced by a particular individual is surmountable, through study of the relevant skills.
One thing I’m pretty sure of: LSAC goes out of its way to make sure that the LSAT lacks cultural bias. Items are screened by representatives of several groups to ensure this. When I wrote items there was even a guy whose job it was to make sure that the test didn’t discriminate against Canadians.
11. Aside from completing lots of LSAT PrepTests and getting your book, of course, what are your general LSAT prep recommendations?
This is an open-ended question but I’ll mention a few points that I think are really important.
i. When you study, focus on a particular skill per study session. The more focused the better – not LR, but assumption items, for example; or not working through complete AR sections, but just setting up a bunch of different grouping games one after another, without worrying about solving the items at that time.
ii. The process of elimination is very important; obviously for AR, but in a different, more procedural way for LR and RC. With these latter two, it is almost always a good approach to try to eliminate three answer choices first, and then to go back and select the correct one from the remaining two.
On a related note: with LR and RC, the point of studying items is not just to identify the correct answers, but especially to understand the reasons why the others are incorrect, and to learn the general “distractor” strategies that will help you become more efficient in identifying these in the future. Repetition is key – look at the same items over and over. It is much better to be extremely familiar with several hundred items than to have a passing familiarity with a few thousand.
iii. After you’ve taken a few practice tests, the value of taking additional test plummets, from an improvement perspective. The key to improvement is working on individual skills in isolation, not mindlessly plowing through a hundred items at a time, hoping to have done better this time than the last. Obviously, one wants to take enough practice tests to be comfortable on test day. But taking tests is not generally an efficient use of study time.
iv. Practice with real LSAT items. They are cheap, and there is no substitute for the real thing.
Photo by lwr
The LSAT format changed in the early 1990s, and I started writing items late in 1992. At first the pay rate was $75 for each accepted LR item, but it went up to $85 per item at some point, and that’s what it was when I stopped writing LSAT items in fall of 1997. I imagine that it has gone up a bit, but I will say that, compared to most other item writing gigs, that’s a pretty high rate even today.
An LR item writing assignment consisted of 10 items of various types – two weakeners, one assumption, etc. I probably averaged somewhere around 8.7 accepted items per assignment. Most writers I knew who averaged much fewer, say 6 or so out of 10, didn’t write many items, or didn’t write for long.
2. How is the item-writing process different today than when you worked as an item-writer? Does LSAC still use freelancers, or is it in-house?
I haven’t worked on the LSAT in a long time, but as far as I know it’s pretty much the same as it was back then. In fact, relying on contract item writers is now probably the industry norm. Since about 2000, the testing industry has exploded, and lots of folks who worked on the LSAT went to other places to oversee freelance item writers for other tests. I’m pretty sure that most tests work on the freelance model to a large degree. The GRE may be an exception; ETS may produce that one primarily in-house.
3. In our last interview, you mentioned that LSAC sent you to a training workshop where they gave you a guide covering all Logical Reasoning item types, and a list of what not-to-do. Can you elaborate on what each contained? What sort of feedback did LSAC offer on improving submitted items?
LSAC’s item writing guide was quite helpful. It made some general points about content, style, item stems, etc., and then they worked through exemplars of each item type. I seem to remember pointers for constructing good distractors, tips for disguising correct answers, etc. You quickly internalize most of it, so I don’t remember much about the details. But I remember thinking that somebody put a lot of work into it, and that it was helpful.
After each assignment was reviewed by my editor I had a phone call with him, and he would give me good, detailed feedback on all the items. I was given a clear reason why any rejected item was found unsuitable, and the accepted items that could have used some extra polish were discussed with me as well. It was in everybody’s interest to make the process as efficient as possible. I will say that, given my experience working on other tests, the LSAT was especially good in the feedback department.
4. What's the formal process by which an LSAT question goes from being an item-writer's draft of a question to becoming part of an actual scored exam?
Here’s how the process works, as best I understand it: first, an item writer sends the items to the testing company, which I believe for the LSAT is still ACT. Editors pick the ones they like, make whatever changes they think are appropriate, and then send the items to LSAC. The items receive another level of editing/review and then are placed on experimental test sections. The tests are administered and statistics are gathered. Then some of the items (the “good” ones) go to real test forms, while others go back for more editing and then another shot on an experimental section. [Ed. Read my series on the test-equating process.]
5. How are Reading Comprehension passages chosen?
I’m not exactly sure how RC passages are chosen for the LSAT; it’s probably pretty idiosyncratic. I am sure that writing assignments specify the general type of passage – natural science, humanities, etc., but after that it’s probably up to the writer. In each released LSAT test, you’ll find references to the articles that the RC passages are based on, so that will give you an idea of the kinds of sources that are used.
6. Are Logical Reasoning passages based on actual scholarship? A lot of them seem like real arguments.
One of the points that I do remember from the LR writer’s guide is that, to the extent the stimulus makes factual claims, they should be true, or at least reflect the current state of research in a field. So yes, many stimuli are based on real scholarship. But a stimulus might discuss a hypothetical vaccine, for example, that is not based on anything factual. So while it is the case that many LR items are based on actual scholarship, many are not. Often the basis for an item is a reasoning style, or a type of error, and then the item writer is simply looking for a topic to cloak the idea in. This is at least in part because item writing assignments specify the items that the writer needs to come up with by task, rather than by topic.
7. Would you agree that Logic Games have generally become easier over the years? Why don't we see pattern games anymore, and why do "rare" game-types show up so sporadically?
Well, I will say that preparing for logic games has gotten easier over the years, and I think this is really the phenomenon that your reader is remarking on. More specifically, if one were to grab your basic, pretty smart off-the-street person who wanted to go to law school and give her an AR section from an old test, and then a new one, she’d do about the same, on average. [Ed. Analytical Reasoning = Logic Games].
But, for someone who is actually studying to take the LSAT, there are definite differences. As your reader points out, the range of games seems to have shrunk significantly; virtually all recent games are instances of just a few types. But these games aren’t intrinsically easier than the less frequently seen games, in my opinion. Rather, the point is that one who prepares today has a smaller “strike zone” than test takers in the past, to use a baseball metaphor, and that mastery of a few game types is more likely to translate into success today than it used to.
8. Do you see the LSAT's emphasis moving in any particular direction now or in the future? Do you see LSAC making any major changes in any section, like the addition of Comparative Reading in June 2007?
I’m not sure where the LSAT is headed. I do know that they considered making it a computer-based test at one point, and that they were toying with the idea of items that were auditory - played through headphones - rather than written. For whatever reason, perhaps the cost of using testing centers, they decided to stick with the paper-and-pencil test. A safe assumption: the LSAT won’t change much, or quickly at least, unless people start complaining about it (even more than they do now).
9. Do you believe the LSAT to be a test of innate skill or something people can learn to master?
This is an interesting question. On any reasonable sense of “innate,” the LSAT does not test innate skills; they are all cultural skills. Sure, certain abilities tend to make one better or worse suited for possessing these skills; a good short-term memory, for instance, is undoubtedly helpful on the AR section. But all of the skills tested on the LSAT are cultural and acquirable, not innate. One interesting point that the question presupposes, but that I think is especially important, is that the LSAT is a test of skills, “know-how” rather than “know-that.” The LSAT is a lot more like tying your shoes, or playing a game of cards, than recalling chemistry facts - with each item, getting the correct answer is a matter of what you do, not what you know.
Now for some people certain skills come “naturally,” we like to say. Some folks are just really good at throwing a ball, others have a knack for long division. The rest of us, after some effort, can eventually learn the skill in question.
When it comes to the LSAT, some people are undoubtedly “naturals,” but pretty much anybody who wants to can master the relevant skills. It just involves a different, more reflective procedure for some of us than for others. Specifically, for us “learners” the key is to take a complex task that a natural LSAT test-taker performs intuitively, and to break it into its components so that we can learn the task piece-by-piece, until with practice and repetition the complex task becomes second nature, like tying our shoes. I’ve tried this approach with hundreds of LSAT students, and several shoe-tying kids, with great results all around.
10. Do you think certain groups of people are at a greater disadvantage preparing for/taking the test than others, particularly those of lower socioeconomic status?
Well, this is a complicated issue, and “disadvantage” is a funny word. There is no question that LSAT scores are correlated with family income, which I guess means socio-economic status. There are thousands of individual exceptions to this trend, but as generalizations go it’s pretty reliable. So, yeah, when it comes to taking the LSAT low socio-economic status probably puts one at a disadvantage, statistically speaking. But this is hardly unique to the LSAT, first of all.
Second, this doesn’t mean that the test is biased against those of low socio-economic status, any more than soccer and chess are biased against people who don’t grow up playing them. All of the skills necessary for the LSAT are acquirable by any literate person comfortable with English, although (regardless of one’s income level) the less familiar one is with these skills to begin with, the more work it will typically require to master them. But this is no different from most other aspects of life, and way “fairer” than some.
Consider, for example, how relevant height is to being good at basketball. And height is something that a person has virtually no control over. By contrast, test takers have much more control over whether they possess the key attributes conducive to success on the LSAT, and any initial disadvantage faced by a particular individual is surmountable, through study of the relevant skills.
One thing I’m pretty sure of: LSAC goes out of its way to make sure that the LSAT lacks cultural bias. Items are screened by representatives of several groups to ensure this. When I wrote items there was even a guy whose job it was to make sure that the test didn’t discriminate against Canadians.
11. Aside from completing lots of LSAT PrepTests and getting your book, of course, what are your general LSAT prep recommendations?
This is an open-ended question but I’ll mention a few points that I think are really important.
i. When you study, focus on a particular skill per study session. The more focused the better – not LR, but assumption items, for example; or not working through complete AR sections, but just setting up a bunch of different grouping games one after another, without worrying about solving the items at that time.
ii. The process of elimination is very important; obviously for AR, but in a different, more procedural way for LR and RC. With these latter two, it is almost always a good approach to try to eliminate three answer choices first, and then to go back and select the correct one from the remaining two.
On a related note: with LR and RC, the point of studying items is not just to identify the correct answers, but especially to understand the reasons why the others are incorrect, and to learn the general “distractor” strategies that will help you become more efficient in identifying these in the future. Repetition is key – look at the same items over and over. It is much better to be extremely familiar with several hundred items than to have a passing familiarity with a few thousand.
iii. After you’ve taken a few practice tests, the value of taking additional test plummets, from an improvement perspective. The key to improvement is working on individual skills in isolation, not mindlessly plowing through a hundred items at a time, hoping to have done better this time than the last. Obviously, one wants to take enough practice tests to be comfortable on test day. But taking tests is not generally an efficient use of study time.
iv. Practice with real LSAT items. They are cheap, and there is no substitute for the real thing.
Photo by lwr
Just wanted to congratulate you on the interview. It was one of the better things I've read about the LSAT in a while. I thought the advice to focus on skills vs grinding through PTs to be particularly good and often ignored.
ReplyDeleteI read through this interview, as well. Though I am a 1L, I am still interested in LSAT lore. This is great stuff.
ReplyDeleteThese two interviews give a fantastic behind-the-scenes look at the LSAT. Thanks so much!
ReplyDeleteI'd like to add a correction/clarification: Stephen mentions "the testing company, which I believe for the LSAT is still ACT". The LSAC has never contracted out the LSAT to any "testing company". Perhaps he got it confused with the GMAT, which ACT did take over in 2004?
Stephens last piece of advice to only use real LSAT questions is spot-on. I would take it a step further and say that, as much as possible and especially once students start to become more proficient, they should use only recent preptests. Preptests from 15-20 years ago have a different "feel" to them that is invisible at first but becomes increasingly apparent as the students' "LSAT palate" becomes more refined. By the time students reach the level of "LSAT Aficionado" in the weeks leading up to the test, those differences can become distracting.
This preference is incorporated in your study schedules, but I wanted to draw a bit more attention to it.
Great work as always! Keep it up!
-Law
Glad you all enjoyed the interviews!
ReplyDelete@Law - I forwarded the 2nd paragraph of your comment to Dr. Harris. He responded:
"I'm not sure about the term "contracting out," but I when I wrote LSAT items, I was paid by ACT, even though LSAC ultimately took these items and used them on the test. The LSAT item writing workshop I attended was in Iowa City, at ACT's campus; and the LSAC guys (who I had previously met at LSAC's headquarters, and who gave me a ride from the airport to the workshop) were the one's conducting it. I was later offered a job by ACT as an editor for LSAT items. I didn't take it, but a grad school buddy of mine did, and he still worked for ACT last time we talked, about five years ago. Funny that the commenter mentions the GMAT, because that was the test my buddy was working on for ACT when we talked last, and I just assumed from our conversation that ACT's role in the two tests (LSAT and GMAT) was the same. But I must admit that I really don't know the details."
Thanks for the fantastic interview! Very eye-opening.
ReplyDeleteIn response to question 9:
ReplyDelete"On any reasonable sense of “innate,” the LSAT does not test innate skills; they are all cultural skills. Sure, certain abilities tend to make one better or worse suited for possessing these skills; a good short-term memory, for instance, is undoubtedly helpful on the AR section. But all of the skills tested on the LSAT are cultural and acquirable, not innate."
Compare this to the answer to question 10:
"One thing I’m pretty sure of: LSAC goes out of its way to make sure that the LSAT lacks cultural bias."
Is the LSAT a test of acculturation or isn't it?