Home > Company > News > eNewsletter > Educator's Voice

Educator's Voice

Volume 7, Issue 5
May 10, 2006

Creating quality multiple choice questions

We all took many multiple choice tests in school, and therefore we tend to believe that we know how to construct good test items. For most of us, this belief is wrong. Not only because exposure does not guarantee know-how, but because most of the tests we've been exposed to were horribly constructed. That's right, I said it. The fact is, most of us have had very limited or no training on constructing test items, and the same is true for those before us.

The intention of this article is to set out some conventional wisdom for the construction of multiple-choice questions. Much literature has been dedicated to the science of test theory, and as educators I encourage you to pursue a greater understanding and appreciation of field. It is a fascinating, yet complex, field and an important facet of your role as an educator.

It will be useful to establish some basic terminology prior to discussing multiple choice questions (hereto referred to as MCQs). An MCQ has two basic components: a stem and associated alternatives. The stem is an introductory question or incomplete statement at the beginning of each item, and it is followed by three or more alternatives that consist of the correct answer and distracters — the incorrect, but tempting options.

Planning and strategy

1. Time
If you've never created MCQs before, you're in for a big surprise. MCQ development is a very time-consuming task. Not only does it take time to brainstorm on what to write, you must also take into consideration the amount of time for review and revision. Add or even multiply more time if your MCQs are being reviewed by other faculty or administrators (which I recommend).

MCQ development is detailed work, and every word of every question is critical. Therefore, you can't wait until the last minute to start writing. The result of rushing the MCQ development process will likely be an unreliable, ambiguous and invalid exam in which both you and your students will pay the consequences. Procrastination is tempting when it comes to MCQ development, but it should be treated as one of the most important aspects of your course development efforts, especially if exams are a larger part of your grading strategy. Therefore, plan to dedicate a considerable amount of time to construct, review and revise your MCQs.

2. Validation
Poorly written multiple choice items elicit complaints from students that the questions are confusing, ambiguous or were not covered in the course content. The validity of MCQs can be tested by having knowledgeable people, such as colleagues and teaching assistants, review the items with much scrutiny. If possible, a group of people should be involved in the preparation of MCQs—both in generating the questions and particularly in trialing the questions. In the trialing process, a group of colleagues should work through the MCQs without knowing the answers. Group discussion of their results should follow, and the questions that generated the most incorrect responses should be reconstructed or discarded.

3. Learning objectives are your best friend
Pure and simple: your test questions must directly correlate to your course learning objectives. Learning objectives are the foundation for which all instructional materials should be written, and not having them is like trying to build a house without a blueprint.

A strategy I often use is what I refer to as "triangulation." In essence, I map my test questions to a particular learning objective as well as the specific course content that covers that specific objective. Not only does this strategy help ensure the validity of my MCQs, but it also ensures that my learning objectives and course content are in alignment as well. Therefore, the quality of both my MCQs and course content are directly correlated to my learning objectives. With rock-solid, well-written learning objectives as my foundation, MCQ test development is a much less painful process, and the validity of my MCQs are easily proven from both my course content and learning objectives.

4. Start from the end
Another strategy that I like to use is to write my MCQ test items prior to writing any course content. This allows me to focus strictly on how my MCQs map to my learning objectives. This process also helps me refine, add or delete learning objectives before I begin to create any course content. In essence, I'm checking and validating my blueprint before I start building my house. Moreover, this strategy works very well when developing courses in conjunction with other faculty in that it is much easier to come to consensus. Instead of being mired in pages and pages of course content, reviewers are focused by looking at objective-level materials only.

Writing question stems

The "stem" of an MCQ poses a problem or presents a question. The basic rule for stem writing is that students should be able to understand the question without reading it several times and without having to read all the options. Below are some guidelines for writing MCQs:

1. Use clear, straightforward language in the stem.
Questions that are constructed using complex wording may become a test of reading comprehension rather than an assessment of whether the student knows the subject matter. The problem in the stem should be stated clearly, use straightforward vocabulary and not include unnecessary or complex wording.

Poor Example #1:
As the level of fertility approaches its nadir, what is the most likely ramification for the citizenry of a developing nation?

1. a decrease in the labor force participation rate of women
2. a dispersing effect on population concentration
3. a downward trend in the youth dependency ratio
4. a broader base in the population pyramid
5. an increased infant mortality rate

Improved Question #1:
A major decline in fertility in a developing nation is likely to produce:

1. a decrease in the labor forces participation rate of women
2. a dispersing effect on population concentration
3. a downward trend in the youth dependency ratio
4. a broader base in the population pyramid
5. an increased infant mortality rate

The stem of the poor example above is awkwardly written, and therefore somewhat confusing. The stem of the improved question is concise and to the point.

Poor Example #2:
Cells of one kind belong to a particular group performing a specialized duty. We call this group of cells a tissue. All of us have different kinds of tissues in our bodies. Which of the following would be classified as epithelial tissue?

1. adenoids
2. cartilage
3. tendons
4. mucous membranes
5. tonsils

Improved Question #2: Which of the following would be classified as epithelial tissue?

1. adenoids
2. cartilage
3. tendons
4. mucous membranes
5. tonsils

The first three sentences of the stem in the poor example are poorly written and also completely unnecessary. The stem of the improved question is concise and to the point.

2. Provide a single, definite statement to be completed or answered by one of the several given choices.

Poor Example:
Cell islets:

1. are made up of thousands of smaller units called monosaccharides
2. contain ducts
3. are created during photosynthesis
4. produce insulin

Improved Question: The cell islets of the pancreas secrete the substance called:

1. trypsin
2. polysaccharides
3. insulin
4. tryptophan
5. amyloplasts

In the poor example above, there is no sense from the stem what the question is asking. It is merely a collection of true/false statements with a common stem. The improved question clearly identifies the question and offers the student a set of homogeneous alternatives. A good stem can be definitively answered without providing alternatives.

3. Avoid unnecessary and irrelevant material. The stem should only contain information that is pertinent to the question. Do not use the stem as an opportunity to teach or include statements that are not needed to answer the question. The stem should not be tricky or misleading.

Poor Example: Paul Muldoon, an Irish postmodern poet who uses experimental and playful language, uses which poetic genre in "Why Brownlee Left"?

1. sonnet
2. elegy
3. narrative poem
4. dramatic monologue
5. haiku

Improved Question: Paul Muldoon uses which poetic genre in "Why Brownlee Left"?

1. sonnet
2. elegy
3. narrative poem
4. dramatic monologue
5. haiku

The poor example above contains material irrelevant to the question.

4. State the stem in positive form whenever possible.
Negatives, such as "no," "not" and "except" should be used sparingly, and avoided altogether if possible. If negative words must be used, capitalize, underscore or embolden them so that the examinee will take notice.

Poor Example: Which of the following is not a symptom of osteoporosis?

1. decreased bone density
2. frequent bone fractures
3. raised body temperature
4. lower back pain

Improved Question:
Which of the following is a symptom of osteoporosis?

1. decreased bone density
2. raised body temperature
3. hair loss
4. painful joints

The poor example above contains a negative. The improved question stem is phrased in positive form.

5. Include as much of the question in the stem as possible. If the same words or phrases are repeated in all or most of the alternatives, the stem should be rewritten to include the repetitious material.

Poor Example:
In objective testing, the term objective:

1. refers to the method of identifying the learning outcomes
2. refers to the method of scoring the answers
3. refers to the method of selecting the course content
4. refers to the method of presenting the problem

Improved Question:
In objective testing, the term objective refers to the method of:

1. identifying the learning outcomes
2. scoring the answers
3. selecting the course content
4. presenting the problem

The alternatives of the poor question include the repeated use of "refers to the method of…." The improved question stem incorporates as much of the question as possible to avoid unnecessary redundancy in the alternatives.

Writing alternatives

Writing quality alternatives is, in my opinion, more challenging than writing the stem. Coming up with "quality" alternatives is very difficult. Most will find this to be the most frustrating and time-consuming component of MCQ development, yet it is the most critical. Why? Because you need to make your MCQs challenging yet fair. Additionally, you can't underestimate how clever students can be—especially those who are unprepared and will therefore put forth a lot of effort to look for the "best guess." A test-wise examinee will exploit every little oversight of inferior test writing. The poorer job you do, the more your test will be exploited by the examinees, and therefore the less reliable your MCQ exam.

Below are a variety of alternative writing guidelines.

1. Ensure that there is only one clearly correct response (for single response MCQs).

Poor Example:
Which of the following texts is considered to represent the pinnacle of modernist achievement?

1. The Waste Land
2. Middlemarch
3. "Ode to a Nightingale"
4. Ulysses
5. Ethan Frome

Improved Question:
Which of the following texts represents one of the high points of modernist achievement?

1. The Waste Land
2. Middlemarch
3. "Ode to a Nightingale"
4. Ethan Frome
5. "My Last Duchess"

Both "a" and "d" could be considered to be correct responses for the poor example.

2. Make all alternatives grammatically consistent with the stem and parallel in form.
Make sure to check the wording of the stem so that it is grammatically consistent with the alternatives. An experienced test taker will quickly notice inconsistencies and can therefore easily eliminate some alternatives.

Poor Example: A fertile area in the desert in which the water table reaches the ground surface is called an:

1. mirage
2. oasis
3. water hole
4. polder

Improved Question:
A fertile area in the desert in which the water table reaches the ground surface is called a/an:

1. mirage
2. oasis
3. water hole
4. polder

The poor example uses the article "an," which identifies choice "b" as the correct response. Ending the stem with "a/an" improves the question.

3. Avoid providing common verbal associations to words in the stem.
In addition to grammatical inconsistencies, providing common verbal associations to words in the stem are also very obvious clues that savvy test takers can pick up on.

Poor Example:
What is the best way to locate an attorney for assistance?

1. contact an attorney referral service
2. contact a police officer
3. Look for a newspaper ad
4. Wait until you go to jail

Improved Question:
What is the best way to locate an attorney for assistance?

1. ask a police officer
2. call a bail bonding agency
3. contact a legal referral service
4. look for an ad in the phonebook

Note that in the poor example, the word "attorney" is in both the stem and correct response.

4. Keep the relative length of all alternatives consistent.
Another common pitfall of MCQ development is providing the examinee clues based on the relative length of the alternatives. More often than not, correct responses are usually longer or shorter in length compared to their associated distracters. Therefore, all alternatives should be of approximately the same length to eliminate length as a clue.

Poor Example: One advantage of multiple choice questions over essay questions is that they:

1. provide for a more extensive sampling of course content
2. focus on more complex outcomes
3. require less time to score
4. rely more on recall

Improved Question:
One advantage of multiple choice questions over essay questions is that they:

1. place greater emphasis on the recall of factual information
2. provide for a more extensive sampling of course content
3. require less time for test preparation and scoring
4. provide for the measurement of complex learning outcomes

Note that in the poor example, alternative "a" is a more verbose and robust alternative—which provides the examinee a good clue to the fact that this is the correct response. Conversely, the alternatives for the improved question are consistent in length and therefore eliminate length as a clue.

5. Use only possible and attractive alternatives as distracters.
One of the most frequent mistakes made by test developers (generally due to laziness) is providing unreasonable distracters. Often, completely ridiculous alternatives are provided (you've seen this before) that are so off-the-wall that it is simply incomprehensible.

The distracters must be attractive to examinees who lack knowledge of the material so that they cannot easily eliminate alternatives (hence the word "distract"). Therefore, the distracters need to be both logically consistent with the stem as well as represent common errors.

Poor Example:
Subtract:
  8032
- 5743

1. 2289
2. 2288
3. 2378
4. 2378
5. 3378

Improved Question:
Subtract:
  8032
- 5743

1. 2289
2. 2389
3. 3399
4. 3711

In the poor example, the distracters are all fairly unreasonable. If the examinee successfully subtracts the 3 from the 2 (which is 9 when numbers are borrowed), there is only one alternative that ends in 9—giving away the answer without having to complete the entire calculation. Therefore, alternatives "b," "c," "d" and "e" are not plausible distracters. Moreover, there is no logical way to calculate this problem to obtain these results.

However, the alternatives for the improved question are all plausible distracters in that they are all the result of common errors. For alternative "b", if the examinee fails to change the 0 to 9, 2389 is a plausible alternative. For alternative "c" 3399 is plausible if the examinee fails to decrease the two digits borrowed from. For alternative "d" 3711 is the result if you subtracted 8032 from 5743.

6. Avoid overlapping alternatives.
This is more often an oversight than it is intentional. Surprisingly, not all examinees catch on to this. Can you identify what is wrong with the following MCQ?

Poor Example:
Due to budget cutbacks, the university library now subscribes to fewer than _?_ periodicals.

1. 25,000
2. 20,000
3. 15,000
4. 10,000

Do you see the problem? A savvy examinee would know that given the alternatives, "a" would always be correct.

7. Avoid using "All of the above" and "None of the above."
It is tempting to use "all of the above" or "none of the above," and these are often used to the point of abuse. These alternatives are also frequently misused as they often appear as an alternative that is obviously not the correct response. Moreover, "all of the above" or "none of the above" alternatives are easily exploited by test-wise examinees who will recognize one as the correct choice by identifying only two correct or one incorrect alternatives, respectively. Obviously, the students' chances of guessing the correct answer improves as they use these techniques.

Poor Example:
Which of the following factors must be considered in computing basal energy requirements?

1. height
2. weight
3. age
4. all of the above

Improved Question:
Which of the following factors must be considered in computing basal energy requirements?

1. age only
2. weight only
3. height and weight only
4. age, height and weight

8. Avoid using "always" and "never" in the stem.
Savvy examinees love these types of questions for the same reasons they love "all of the above" and "none of the above" alternatives. Many examinees are smart enough to rule such universal statements out of consideration.

9. Provide a sufficient number of alternatives.
An MCQ, by default, must have at least three alternatives (two alternatives would make it a True/False question). Historically, most people have chosen four or five alternatives as it seems like a reasonable number. Moreover, it is assumed that the greater number of alternatives leads to greater difficulty. However, this is not necessarily true. Although a greater number of distracters lessens the ability of the examinee to correctly guess the correct response, a "good" MCQ can be achieved with as few as three alternatives, if the guidelines provided in this document are applied.

Conclusion

As you can see, creating MCQs is not as simple as it may first seem. It is a very detail-oriented effort that is very time consuming and frustrating for many. However, the effort you put into MCQ development will result in an exam that is both reliable and valid.

I strongly encourage you to review MCQs that you or your department have developed. Chances are you will make some interesting discoveries that can lead to healthy discussion with your peers. How well do the questions align with the course content and learning objectives? Are the question stems easy to understand? Are the alternatives for each question plausible? Do the alternatives provide too many clues for the examinees? Although time consuming, reviewing your MCQ tests and honestly answering these questions are well worth your effort.

       —Eric Rohrer, M.S.Ed.

Resources

Haladyna, T.M. & Downing, S.M. (1989). "Validity of a Taxonomy of Multiple-Choice Item-Writing Rules." Applied Measurement in Education, 2 (1), 51-78.

Osterlind, S. J. (1998). "Constructing test items: Multiple-choice, constructed response, performance and other formats." Boston MA: Kluwer Academic Publishers.

Smith, J.K. (1982). "Converging on correct answers: A peculiarity of multiple-choice items." Journal of Educational Measurement, 19, pp 211-220.

Wesman, A.G. (1971). "Writing the test item." In R.L. Thorndike (Ed.) Educational Measurement (1st ed, pp 99-111). Washington, DC: American Council on Education.

TIP

Working in the full-screen editor view

Many of us, when working in Author view to develop or edit a homepage or content item, like to work in Edit full-screen view (accessed by clicking the link in the bottom right of the page). We find the larger screen view facilitates working online for longer periods of time.

There is an issue, however, that some of us encounter. If you are working with Windows XP, Service Pack 2, you will not have immediate visual access to the Save & Close and Cancel buttons in the Edit full-screen view.

There is a ready solution—panic on your part is not necessary. When you complete what you are working on, simply press the F11 key on your keyboard. The Save & Close and Cancel buttons will appear in their standard location in the bottom right of your screen. You may now save your work and exit to the standard Author view of the page.

       —Ken Switzer, Ph.D.