Writing Effective Test Questions

Writing test questions is a critically important and challenging skill. Unfortunately, few instructors have been taught how to design assessments or write test questions.

Well written tests motivate students, reinforce learning, and assess content mastery. A good test provides feedback on teaching and helps identify poorly communicated information or critical bottlenecks. Sadly, poorly written tests not only frustrate students, they can actually be a learning detriment as students may focus on the wrong content or ineffective, short-term study strategies.

As you construct your tests, consider what types of questions are most appropriate for your content. The idea is to test the students’ mastery of content, not how well they can take a test. Each type of question has its own advantages and disadvantages.

• Can be autocorrected
• Easy to collect data
• Tests recall of facts
• Can test large amount of content

• Does not test for higher order knowledge
• Learners have 50/50 chance by guessing
• Need a large number of questions for reliability

Multiple Choice

• Can be autocorrected
• Easy to collect data
• Versatile. Can test fact recall, understanding, synthesis, analysis, and application of learning

• Challenging to write good questions
• Difficult to come up with plausible distractors
• Does not test organization of knowledge

Short Answer

• Easy to write questions
• Tests knowledge of specific facts and details
• Minimizes guessing
• Students must know the answer rather than recognizing the answer

• Difficult to auto grade
• Can overemphasize facts
• Difficult to write questions with only one correct answer
• Can generate “Guess what I’m thinking” type questions

Essay or Long Answer

• Tests for complex understanding
• Easy to write
• Stimulates deeper studying and learning strategies
• Tests ability to organize, opine, and use originality

• Limits the amount of content that can be tested
• Grading can be highly subjective and unreliable
• Time consuming to grade
• May have to defend grading strategy

Tips on Testing

• Fairness: Content mastery should be measured in a way that does not give advantages to irrelevant factors. Test questions should reflect the learning objectives that were clearly communicated to students. Point values should be determined before the test is given.
• Accuracy: Never use a test written by someone else without checking it for accuracy. Publisher questions may save time, but they can be poorly written and answers may be incorrect. Always proofread. Even a small mistake, like misnumbering the questions, can cause big headaches when grading.
• Objectivity: No test can be truly objective. Thinking that one type of question is automatically objective and another is undoubtedly subjective is a false assumption. Bias can creep into any type of test question.
• Frequency:Frequent testing leads to better learning. The retrieval effect (aka the testing effect) means that the more often students are required to remember something, the better they know it. Frequent testing also provides multiple data sources and helps minimize the effects of an off day.
• Reliability:The more items a test has, the more reliable it is. In a short test, a few incorrect answers can have significant impact while a few incorrect answers in a long test makes minimal difference in the outcome.
• Mixed methods: A test with a mix of types of questions minimizes student weakness with a particular method of testing.
• Instructions:Provide clear, concise written instructions. It is often helpful to provide an example of an exemplary answer.
• Accommodations: Think ahead about any accommodations you may be required to give for students with learning or physical disabilities. Consider what accommodations are reasonable. These might include giving extra time, allowing the use of tools like dictionaries or calculators, providing a separate testing location, etc.

Writing Multiple Choice Questions

Multiple choice questions are versatile because they can test factual recall as well as levels of understanding, synthesis, analysis, and application of learning. Multiple choice questions, however, are also one of the most difficult types of questions to write.

A multiple choice question is constructed from a problem or question, known as the stem, and list of suggested answer choices, known as alternatives. The alternatives include one correct or best alternative, which is the answer, and incorrect or inferior alternatives, known as distractors (Clay, 2001). Since questions and alternatives can sometimes be misleading, consider having a colleague take the test before your students do.

One way to construct multiple choice questions that require deeper level of thinking is to create a scenario. Provide a description of a situation, a series of graphs or other data that would be appropriate to your discipline. Then, develop a series of questions that relate back to that material. These questions can require students to apply concepts, combine data, make a prediciton or diagnosis, analyze relationships between data or synthesize information.

Writing an Effective Stem

• The stem must be clear and unambiguous. Any vague terms, like normally, usually, possibly, may, should be avoided.
• State the stem as a direct question rather than an incomplete statement. The stem should be meaningful by itself. Avoid stems such as: Which of the following is a true statement?
• Eliminate unnecessary information or excessive verbiage in the stem.
• Include in the stem any words that are repeated in every alternative.
• All alternatives should be plausible. The purpose distractors is to assess which students have achieved the learning. Implausible alternatives do not accomplish that goal.
• Use at least four alternatives to reduce the impact of guessing, but no more than six. Eliminate complex choices such as: “A and B” “B and C” “C if A is false” etc.
• Alternatives should be mutually exclusive. Watch out for overlapping categories like: "A.1-2” “B. 2-4” etc. If the answer is 2, which selection is correct? A or B?
• Randomly distribute the correct answer among the positions throughout the test. There are testing software programs that will do this automatically.
• Avoid use of all of the above or none of the above. If you do use them, do not make them the correct response all of the time.
• Be cautious in using negative terms in the stem or choices. If you use a negative, capitalize the word for emphasis. For example: Which of the following is NOT an indication of osteoporosis?
• Make alternatives roughly equivalent in length and in a similar format (i.e., phrases, sentences, etc.). Ensure your alternatives are free from clues for the correct answer.

Writing Effective Alternatives

Use questions which encourage evaluation of “higher order” thinking such as interpretation, analysis and synthesis, not just simple recall of medical facts (Loo, 2017).

Example of a recall question of a medical fact:

Which of the following types of cells lack functioning mitochondria?

1. erythrocyte
2. hepatocyte
3. myocardiocyte
4. astrocyte

Example of a question that requires an explanation or interpretation:

The absence of which of the following is the most likely explanation for why a mature red block cell (RBC) is unable to carry out beta-oxidation of fatty acids?

1. mitochondria
2. endoplasmic reticulum
3. Golgi apparatus
4. intracellular oxygen

The use of tables and figures facilitates “higher order” thinking questions (Loo, 2017).

Example of a question requiring integration and synthesis:

Which of the following pulmonary function tests is most consistent with a patient with idiopathic pulmonary fibrosis (IPF)?

FEV₁ = Forced expiratory volume in 1 second; FVC = Forced vital capacity; TLC = Total lung capacity;
DL_CO = Diffusion lung capacity for carbon monoxide

Writing Essay or Short-Answer Questions

Essay or short-answer questions are the easiest to write, but the most time consuming to grade. Use these types of questions when you want to assess the students’ ability to organize knowledge, give an opinion, or use original thinking.

• Construct the question so the approach you want students to take is clear. Words like discuss or explain are vague and open to wildly different interpretation.
• Avoid giving a selection of questions. This reduces reliability. If students answer different questions, they are actually taking a completely different test. To score students accurately, they all must perform the same tasks.
• Write questions which are comprehensive, rather than focusing on small units of content.
• To test the depth of background knowledge, require students to provide supporting evidence for claims and assertions.
• Communicate how you will respond to technical errors such as misspellings, grammar, etc.
• Identify the relative value of each question.

Writing True/False Questions

Due to the high probability of guessing, true/false questions are not reliable and not recommended (Case & Swanson, 2002). They are also surprisingly difficult to write because they must be phrased in absolute terms.

References

• Case, S. M., & Swanson, D. B. (2002). Constructing written test questions for the basic and clinical sciences. National Board of Medical Examiners. Retrieved from http://www.nbme.org/PDF/ItemWriting_2003/2003IWGwhole.pdf
• Clay, B. (2001, October). Is this a trick question? A short guide to writing effective test questions. Kansas Curriculum Center. Retrieved from https://www.k-state.edu/ksde/alp/resources/Handout-Module6.pdf
• Loo, L. (2017, February 19). Guidelines for well written MCQ (multiple-choice questions).

For more information or support contact: facultydevelopment@medsch.ucr.edu.