Order effects occur when questions are poorly ordered and slight changes in the order of questions causes participants to answer differently. These types of problems in questionnaire design commonly lead to bias or issues with data reliability and validity.
In order to get the most reliable data from your questionnaire, your participants need to be able to consider each question and respond thoughtfully. The more mental strain your questionnaire causes, the more likely you will have threats to reliability. The sections below will highlight some common problems in question ordering, the effects that they have on participant responses, and how to fix them.
Serial Order Effects
Serial order effects are related to problems in ordering different types of questions. They can threaten the reliability of your data when participants don’t thoughtfully answer each question.
There are two key main reasons a serial order effect would occur:
- There are too many hard or open-ended questions in a row, which leads the participant to lose motivation to continue.
- There are too many simple questions in a row, which causes participants to rush through a questionnaire and not give thoughtful answers.
One way to minimize the impact of these problems on your questionnaire is to make sure that you are varying question types throughout different sections of your survey. Serial order effects are one of the reasons that you should start your survey with easy to answer, engaging questions, but it is also important to continue to pay attention to other places where these order effects might arise in your questionnaire.
Semantic Order Effects
Semantic order effects are related to problems in ordering the content or meaning of questions. They can threaten the reliability of your data when participants respond to one question differently in different contexts.
Semantic order effects may be more complicated to notice. They typically arise when related questions are not grouped together, or when specific questions are asked as a preface to more general ones. These two ordering problems create a semantic order effect when they change the meaning of the questions and therefore the participant’s response.
Use the navigation arrows on the slides below to elaborate the two different types of semantic order effects–contrast and assimilation–and how to recognize them:
In order to minimize the impact of these problems in your questionnaire, make sure that you have ordered your questions from general to specific so that each question is presented in the appropriate context. You should also group similar questions so that the participant recalls memories or information relevant to the question at hand while they answer.
