+ Reply to Thread
Results 1 to 10 of 10

Thread: Themes in Survey Response, maybe PCA

  1. #1
    Omega Contributor
    Points: 38,374, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,998
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Themes in Survey Response, maybe PCA




    I have a sample of 50 respondents to a cross-sectional (identical) survey administer at four time points. The instrument asks the potential respondents to report their confidence level (on a 5-point Likert scale) performing 12 tasks.


    I assume that I could perform a cronbach alpha on each time point and report on whether certain items have responses that correlate, right? But I was wondering if there was a way for me to use all four time points and see if there are certain questions that respondents are not confident performing, so patterns. I am not sure if I would be able to generalize that the responses have a comparable latent construct or not between items or what my options are.


    So far I have only collapsed the Likert scale to binary and looked at question specific trends across the study period, etc. Lastly, all respondents to all questions had a near maximum score by the last time point.


    Thanks!
    Stop cowardice, ban guns!

  2. #2
    TS Contributor
    Points: 22,418, Level: 93
    Level completed: 7%, Points required for next Level: 932
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Themes in Survey Response, maybe PCA

    Do you mean you are following these 50 people at 4 timepoints? Or that at each timepoint you get 50 different people?
    for all your psychometric needs! https://psychometroscar.wordpress.com/about/

  3. #3
    Devorador de queso
    Points: 95,819, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,935
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Themes in Survey Response, maybe PCA

    But like... are we really the same "person" at any other point in time. Whoa man... things just got philosophical.
    I don't have emotions and sometimes that makes me very sad.

  4. #4
    Omega Contributor
    Points: 38,374, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,998
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Themes in Survey Response, maybe PCA

    Spunky, exactly the same 50 people at each of the four time points. So: 50, 50, 50, 50. No study intervention, just passage of time.


    Dason, you are correct that we are not the same person at each time point. In my field we talk about the moment a study is over time (the over dimension: time) keeps moving so everything is actually different, so can you generalize results to a new sample from the ever changing population. But I don't care about that in this post. Though you are closure to the equator now, so your irradiation of components are denaturing faster, so you aren't component-ly the same bot.
    Stop cowardice, ban guns!

  5. #5
    TS Contributor
    Points: 22,418, Level: 93
    Level completed: 7%, Points required for next Level: 932
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Themes in Survey Response, maybe PCA

    Thing is it seems to me like you'd like to probably do a more exploratory type of analysis. Maybe plotting the items over time and seeing if they change or if the trend flattens or something?

    I don't think Psychometrics can offer any specific insight at this point given the kind of question you're asking.

    Now, if you're making the claim that the latent construct changes over time and you would like to model it, then you would end up doing some type of structural equation model or item response theory... but at N=50 well... I kind of think you're stuck with basic classical test theory type analyses.
    for all your psychometric needs! https://psychometroscar.wordpress.com/about/

  6. #6
    Devorador de queso
    Points: 95,819, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Awards:
    Posting AwardCommunity AwardDiscussion EnderFrequent Poster
    Dason's Avatar
    Location
    Tampa, FL
    Posts
    12,935
    Thanks
    307
    Thanked 2,629 Times in 2,245 Posts

    Re: Themes in Survey Response, maybe PCA

    Quote Originally Posted by hlsmith View Post
    Though you are closure to the equator now, so your irradiation of components are denaturing faster, so you aren't component-ly the same bot.
    I also got that one upgrade to help prevent corrosion due to all the salt water!
    I don't have emotions and sometimes that makes me very sad.

  7. #7
    Omega Contributor
    Points: 38,374, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,998
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Themes in Survey Response, maybe PCA

    So do I need to run factor analysis on the baseline data (time 0) to say, hey people suck at these 4 things. I guess I am also thinking about whether certain people have low confidence on similar items. So if I am no good at this logic, so I will also be poor on this one as well.


    I am content with my current results, which are fairly descriptive - but I feel like the may be an opportunity to better describe an underlying pattern. Any input would be appreciated.
    Stop cowardice, ban guns!

  8. #8
    Omega Contributor
    Points: 38,374, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,998
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Themes in Survey Response, maybe PCA

    And more humidity in general as well as bugs getting into your works. You should take out some more insurance to help provide for Gideon in the case of catastrophic damage or the blue screen of death some day.
    Stop cowardice, ban guns!

  9. #9
    TS Contributor
    Points: 22,418, Level: 93
    Level completed: 7%, Points required for next Level: 932
    spunky's Avatar
    Location
    vancouver, canada
    Posts
    2,135
    Thanks
    166
    Thanked 537 Times in 431 Posts

    Re: Themes in Survey Response, maybe PCA

    Quote Originally Posted by hlsmith View Post
    So do I need to run factor analysis on the baseline data (time 0) to say, hey people suck at these 4 things. I guess I am also thinking about whether certain people have low confidence on similar items. So if I am no good at this logic, so I will also be poor on this one as well.
    Factor analysis will not really tell you that. Factor analysis will be trying to answer more the question of "how many distinct (albeit related) types of latent abilities are needed to accurately explain these results?" It doesn't really factor difficulty in its initial concept.

    That's why I was thinking if you take the route of classical test theory approach you can use the proportion-correct total score as an indicator of whether or not learning is occurring. Where basically the idea is that if you see the proportion of correct answers increase over time, then you can assume people are becoming more proficient.

    Again, N = 50 scares me for factor analysis. The number people have kind of agreed upon in the Psychometric world about sample needed for factor analysis is around N>200.

    Plus you mentioned towards the end most people are getting things correct, right? (all respondents to all questions had a near maximum score by the last time point.) Items that cannot accurately discriminate among people are bad for factor analysis. If everyone's getting the item correct then you're going to have very little variance which translates into lower-than-expected correlations.
    for all your psychometric needs! https://psychometroscar.wordpress.com/about/

  10. #10
    Omega Contributor
    Points: 38,374, Level: 100
    Level completed: 0%, Points required for next Level: 0
    hlsmith's Avatar
    Location
    Not Ames, IA
    Posts
    6,998
    Thanks
    398
    Thanked 1,186 Times in 1,147 Posts

    Re: Themes in Survey Response, maybe PCA


    SSPPUUNNKKYY, Teach me what you mean here:


    classical test theory approach you can use the proportion-correct total score as an indicator of whether or not learning is occurring
    I had treated the outcome as binary (correct/not correct, given your lingo) and modeled time, which had a positive association. Or what I deemed a positive trend. What are you referring to, something similar? Awe me with the Social Sciences here.
    Stop cowardice, ban guns!

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats