Survey design: Use of positively vs. negatively worded questions

An employee engagement survey is a critical part of your employee listening strategy. 

To design a robust survey that achieves its key objectives, you need to consider multiple perspectives and gather varied sources of information to inform the survey design:

Use of positively vs. negatively worded questions

There has been a strong academic advocate for the use of both positively and negatively worded questions in surveys as best practice, and these are the arguments and assumptions behind such a practice.

For instance, to find out if a resource website is useful for employees in their day-to-day activities.

🟢 Positively worded: The website is easy to use.

🔴 Negatively worded: It is difficult to find what I need on this website.

Theoretical arguments for alternating positive and negatively worded questions:

• Acquiescence Bias - People are more likely to agree with a statement than disagree with it 
• Extremity (vs Moderacy) Response Bias - The tendency to choose extreme vs neutral response categories on a rating scale
• Inattentional Bias - To prevent survey respondents from not paying attention to the meaning of the survey
• Broader Construct - To widen the way respondents think and arrange their beliefs related to the construct
  

Assumption underlying the use of negatively worded questions:

• Equivalence - Reverse-coding the negatively worded questions for scoring assumes that agreeing to a positively worded statement = disagreeing with its negatively worded counterpart
  
  

However, there is empirical evidence and counter-arguments that using negatively worded questions does not actually mitigate the issues of a consistently aligned positively worded question set, and in fact, introduces more errors, contaminates scores, and prevents the use of data meaningfully.

Findings include:

• Challenges in Preventing Biases - Having negatively worded items does not prevent acquiescence nor ERB and introduces new errors that prevent the data from being used effectively
• Cognitive Load & Fatigue - Survey respondents’ cognitive load may be affected by how you word the negative statements and can make the judgment process more difficult - this leads to a greater chance of errors due to fatigue
• Inattentional Bias - Inattentional bias may not be prevented when using negatively worded questions and may even occur, especially with longer surveys
• Confusion - Negatively worded items had lower consistency in responding than positively worded items, with also more people responding “Don’t Know”
• Non-equivalence - Respondents react differently to regular forms of a word (e.g.,  encourage) vs. its polar opposite (e.g., discourage) which means that positively vs. negatively worded questions may not be the same

Instead of using a mix of positively and negatively worded questions, our recommendations are:

• Convey the importance of survey to mitigate acquiescence

Do so via a live interaction, such as a video call (e.g., townhall) or face-to-face event (e.g., dinner & dance) and avoid doing it over asynchronous communication as people can ignore or neglect

• Be concrete about the implications of survey results for each and every employee in the organization

State clear expectations such as impact on growth, company initiatives, etc.

• Educate employees about all the potential biases they may have when doing the survey

Include information about why they should overcome these biases (e.g., acquiescence) and how it benefits them, their colleagues and the entire organization

• Schedule the survey at a non-peak season to mitigate inattention

This will allow them to have more cognitive resources to do the survey more effectively

---

To date, employee engagement surveys remain one of the most popular ways to measure engagement and gather feedback. 

A well-designed survey will prompt employees to consider the big picture of their workplace, as well as provide a true reflection of their work experiences.