Survey Data: a drop in the ocean

By Nicole Barrett, Psychologist

Jan 04, 2016

Have you noticed an increase in companies applying survey methods to gather feedback on employee opinion? It might be a questionnaire to measure engagement, safety attitudes or workplace culture. Now I am an absolute advocate for collecting data, but how do you make sense of the results and what can be done to verify if the numbers are real?

There is no doubt that surveys are helpful, however, there are limitations that can mislead if the method is adopted without the appropriate due diligence and caution.

On the upside, surveys reach a wide and geographically dispersed audience. You can cut and dice the results to compare demographic groups, teams or roles and they are very effective when establishing a measurable baseline. On the flip side, surveys are diabolically risky if the results are used in isolation to inform corporate decision making and actions.

Take the 2004 Snorre Alpha for example, where there was an uncontrolled gas blow out under the seabed. Fortunately, of the 216 people on the platform, no-one was injured. What's interesting is that just 6 months' prior, the organisation performed a safety survey and was assessed as a 'generative' culture. Yes, that's right... they came up roses! Yet, the post incident investigation interviews revealed significant issues with the company's culture (1). Whilst some level of hindsight bias might account for the discrepancy, this incident is a classic illustration of just how dangerous relying on survey data alone can be.

I'd like to believe that the inspiration for this growing desire for feedback is linked with a sense of corporate responsibility to engage and listen to employees - a highly valuable exercise. Yet, too often, survey processes are poorly designed and executed with superficial efforts to act on the results. Warning! This can and will do more harm than good.

The survey design in itself is problematic - if you don't ask the right questions you won't get the right answers.

I recently reviewed a report that claimed to measure project performance and safety. I was gobsmacked! 170 pages of colourful graphs with the spread of scores for each survey item along with pages and pages of verbatim responses to the open text questions. Not the slightest attempt to theme or deliver any meaningful interpretation of the results. Seriously? Where would you start on understanding the essence of the findings never mind how to develop a remedial response. My bet is that this very expensive report is collecting dust on a shelf somewhere. A waste of time, money and the worthy contribution made by employees which invariably leads to apathy and lost motivation to participate in future.

When adopting survey methods, be aware and consider:

• A good survey will have construct validity. That is, there are sound theoretical dimensions for interpretation as opposed to a random collection of statements or question items.
• There is a very real potential for impression management or unconscious biases to respond in ways that represent how things 'ought' to be rather than what actually is.
• If the process is managed internally, the person/team may find it difficult to remain objective when interpreting the results.
• In more toxic environments, people doubt anonymity which often influences response rates and can skew the results.
• Participating in survey processes raises employee expectations that something will happen and failure to give feedback invites scepticism about the leaderships commitment.

A few tips when designing your future survey process:

• Be crystal clear on the purpose of your survey, what you are setting out to measure and how you intend to use the results.
• Select a survey that has stand up psychometric properties sitting behind the tool or at least recognise when a newly designed survey is under development.
• Combine the survey with qualitative interview and/or focus group data to verify and add meaning with a richer understanding of the 'what' and the 'why' a particular score has occurred.
• Be prepared to feedback the results - the good the bad and the ugly.
• Engage the workforce in top down and bottom up problem solving.
• Communicate what you are doing and visibly act on the results.