+ Reply to Thread
Results 1 to 9 of 9

Thread: What to do when survey participants answer all '4's?

  1. #1
    Points: 970, Level: 16
    Level completed: 70%, Points required for next Level: 30

    Posts
    5
    Thanks
    4
    Thanked 0 Times in 0 Posts

    What to do when survey participants answer all '4's?




    I reached out to members of my organization to rank how they felt the company was performing on various topics, about 35 questions in total. Several individuals are responding using the same response to each of these questions (using a scale similar to Likert's but with different descriptors). They may throw a few in that differ by one level.

    Is there a logical way to either exclude these or weight others' responses more heavily? We plan to take some average of all the responses to identify key performance concerns as a whole.

  2. #2
    Omega Contributor
    Points: 38,281, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,990
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: What to do when survey participants answer all '4's?

    I don't get if you want a representative sample that you would not include all data.


    Did this happen for all of the response groups (only picked 1, 2, 3, or 4)?
    Stop cowardice, ban guns!

  3. The Following User Says Thank You to hlsmith For This Useful Post:

    Jane M (07-09-2015)

  4. #3
    Points: 970, Level: 16
    Level completed: 70%, Points required for next Level: 30

    Posts
    5
    Thanks
    4
    Thanked 0 Times in 0 Posts

    Re: What to do when survey participants answer all '4's?

    The sampling population included various members, including individuals that are not knowledgeable to respond to certain questions. We gave them the option to opt out but also stressed that we would like responses to each question if possible.

    The concern is that the perception of performance will be under/overstated.

    Also, the concern is that someone who answers mostly '4's, let's say, has done so because they were either uncomfortable responding or did not have the time to devote to the survey. We will have less than 20 surveys in the end, so the validity of the sampling is of question.

    Appreciate the comments!

  5. #4
    Omega Contributor
    Points: 38,281, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,990
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: What to do when survey participants answer all '4's?

    Are you able to determine which surveys belong to these people, "The sampling population included various members, including individuals that are not knowledgeable to respond to certain questions"?


    Or are you just guessing?
    Stop cowardice, ban guns!

  6. #5
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: What to do when survey participants answer all '4's?

    An old problem in surveys is that respondents tend to give the same answer to every question. This might reflect the fact that respondents are not interested in the survey or it might mean that they are very positive generally about the topic (respondents often like to give positive answers which is probably tied to a well known habit of giving socially desirable answers) something I have seen in my own analysis. It is also true you have a very small group you are sampling so variation is likely less common to be caught.

    With such a small population you would be better served to do qualitative interviews than a survey. You will get a lot more useable data that way.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  7. The Following User Says Thank You to noetsi For This Useful Post:

    Jane M (07-09-2015)

  8. #6
    Omega Contributor
    Points: 38,281, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,990
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: What to do when survey participants answer all '4's?

    I agree with noetsi qualitative suggestion. You may also have issues with the narrow range of response options (i.e., 1-4).
    Stop cowardice, ban guns!

  9. #7
    Points: 970, Level: 16
    Level completed: 70%, Points required for next Level: 30

    Posts
    5
    Thanks
    4
    Thanked 0 Times in 0 Posts

    Re: What to do when survey participants answer all '4's?

    In response to all the comments so far:

    I can group the members of the population based on certain criteria, for example by department, and I'll likely use summary statistics in such a manner.

    We've proceeded with a quantitative study (based on decisions outside my control ) using 5 choices but agree that a secondary study utilizing a qualitative approach and requiring open-ended responses is the way to go. For now, I'm stuck having to work with analyzing what I have.

    Just didn't know if there was a statistical way of quantifying or dealing with the data.

    Thanks all!

  10. #8
    Omega Contributor
    Points: 38,281, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,990
    Thanks
    397
    Thanked 1,185 Times in 1,146 Posts

    Re: What to do when survey participants answer all '4's?

    Survey data can be tricky as you are finding out
    Stop cowardice, ban guns!

  11. The Following User Says Thank You to hlsmith For This Useful Post:

    Jane M (07-09-2015)

  12. #9
    Fortran must die
    Points: 58,790, Level: 100
    Level completed: 0%, Points required for next Level: 0
    noetsi's Avatar
    Posts
    6,532
    Thanks
    692
    Thanked 915 Times in 874 Posts

    Re: What to do when survey participants answer all '4's?


    Given that a qualitative model probably fits your needs better, I think your best option with such few respondents and the type of responses you have gotten is descriptive statistics not regression. Regression is a powerful tool, but it makes assumptions and requires a certain minimum response set (in practice if not always in theory). Sometimes it is better to go with the "simpler' model...

    In exploratory research/analysis qualitative or descriptive is commonly better IMHO as the first step.

    An issue with going beyond 4 or 5 levels is that while methodologically it may have advantages you run the real risk that customers won't really understand the differences between the levels. You can ask them to rank something on a hundred point scale but if they don't understand what 77 is compared to 90 what have you really gained. False assurances about your measurement I would argue

    I agree with hlsmith that survey data can be very very difficult. And many courses and discussions of it focus on regression or methods not on the substantive elements such as how you can generalize, and the type of issues you have raised. A good book in survey design or design of experiment is really useful.... (you may well have these already). Commonly the statistics is the easy element of a design.
    "Very few theories have been abandoned because they were found to be invalid on the basis of empirical evidence...." Spanos, 1995

  13. The Following User Says Thank You to noetsi For This Useful Post:

    Jane M (07-09-2015)

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats