+ Reply to Thread
Results 1 to 8 of 8

Thread: single sample or multiple samples?

  1. #1
    Points: 382, Level: 7
    Level completed: 64%, Points required for next Level: 18

    Posts
    11
    Thanks
    1
    Thanked 0 Times in 0 Posts

    single sample or multiple samples?




    Let's say I run an experiment where I want to measure reaction times to some stimuli. The experiment consists of 100 trials.

    I collect data from 30 participants, and each participant carries out his/her 100 RT trials.

    In this case, do I have 30 samples, or do I have 1 sample?

    Typically with that sort of data I suppose we don't care about the individual trials (assuming they are all identical), so we average them such that each participant now has a single datapoint (the average of his/her 100 trials). Does this change things?

    The thing that confuses me is that when running an experiment we refer to our 30 participants as a sample. But when studying statistics it feels much more to me as though in this case we are collecting 30 samples.

    Thanks

  2. #2
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Re: single sample or multiple samples?

    You've stumbled on something quite important. The approach were we try to model the type of structure you are hinting at, is called "multilevel modelling" (also: "mixed-effect model" and a bunch of other confusing names).

    I'm not really sure whether this approach offers advantages when trials are truly identical. But it seems to me that they rarely are (maybe on a very simple task, say where you distinguish "X" from "O" in an "oddball" task). More typically, they are assumed to be some random sample from a population of possible trials. For instance, I used to do some psycholinguistics stuff a long time, and there we often had words with a certain property. There, it may be very useful to model your item as a random factor.

    Another thing to consider is that even with identical trials, there are generally training and/or fatigue effects over time, and that these differ by subject. By modelling these, we can reduce unexplained variance and increase power.

    Here is a paper on this topic within the domain of psycholinguistics. It assumes items as a random factor, so it's not a perfect match, but it helps to explain the concepts.
    Last edited by Junes; 10-30-2016 at 06:38 PM.

  3. #3
    Points: 382, Level: 7
    Level completed: 64%, Points required for next Level: 18

    Posts
    11
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: single sample or multiple samples?

    Thanks for your reply but it doesn't really clear things up for me - I am just trying to learn the basic stats concepts here. Although what you mentioned about modeling items (trials, I suppose?) as random factors to reduce unexplained variance seems quite interesting and I will definitely keep it in mind. It's just a step beyond what I'm trying to think of here.

  4. #4
    TS Contributor
    Points: 17,749, Level: 84
    Level completed: 80%, Points required for next Level: 101
    Karabiner's Avatar
    Location
    FC Schalke 04, Germany
    Posts
    2,540
    Thanks
    56
    Thanked 639 Times in 601 Posts

    Re: single sample or multiple samples?

    *snip* (mistake)

  5. #5
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Re: single sample or multiple samples?

    Quote Originally Posted by fffrost View Post
    Thanks for your reply but it doesn't really clear things up for me - I am just trying to learn the basic stats concepts here. Although what you mentioned about modeling items (trials, I suppose?) as random factors to reduce unexplained variance seems quite interesting and I will definitely keep it in mind. It's just a step beyond what I'm trying to think of here.
    OK, no problem. Well, the thing is that before you average, you do have "multiple samples", so to speak. And this is a problem because the methods in common use (linear regression, ANOVA, etc.) all assume independent errors (the part of the reaction time that our model can't explain). If you just have a big list of trials from different subjects, then the errors will not be independent: the trials of one particular subject will have correlated errors (our model does well or badly in predicting all of that subject's trials, so the errors will be similar).

    So, just leaving all in one big dataset won't do. We can average over trials to get a mean for each subject. And then do, for instance, an ANOVA.

    Like I said, this may be fine if trials are truly identical (for each level of my variable, I get exactly the same stimulus every time and have to respond the same way every time). There are some simple reaction time experiments where this is applicable.

    However, in my experience, this is not a common situation in psychology. More likely is that our variable represents a class of things, of which we have items. For instance, we might test whether people are faster to respond to happy words ("joy", "bliss", "laughing") than to sad words ("down", "depressed", "crying"). I just made this up, but this is quite a common situation in some fields of psychology.

    So, there is a difference between "trials" and "items". Trials are the "slots" in which we fit the items. Now the thing is, we can't assume these items are identical. For instance, "bliss" is an uncommon word, and we know that uncommon words take longer to recognize. Similar things may happen in other fields of psychology (an "X" might be easier to process by the visual system than an "O", etc.).

    There are two reasons why we might want to take into account these differences in items:
    1) In general, the more information in a model, the more variance we can explain and the better it performs
    2) We may be interested in generalizing from the items to the class. For instance, we are interested in the effect of "happy" words in general, not so much in the effect of "bliss", "laughing", etc.

    Remember, we can't do an analysis on the big pile of data. So we need some other way to do it. The way people used to do to this (and probably still do), is do two analyses: first average by item and do an ANOVA (or something else), then average by subject and do an ANOVA (or something else). However, multilevel modeling, which is somewhat more complex, offers a more powerful tool.

    So, in conclusion: if items are truly identical, then it's probably not a big deal to average them. If they are not identical and you want to generalize from your items, you can do the averaging thing (simpler but less powerful) or use multilevel modeling (bit more difficult but more powerful).

  6. The Following User Says Thank You to Junes For This Useful Post:

    fffrost (10-31-2016)

  7. #6
    Points: 382, Level: 7
    Level completed: 64%, Points required for next Level: 18

    Posts
    11
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: single sample or multiple samples?

    Thanks a lot for the answer! This helped a lot and I will keep this all in mind. I think this issue is something very common to psychology, but it seems like textbooks don't like to go into these details.

    The reason why I was wondering this in the first place is because of the standard error. In psychology, you learn that you collect data from "a sample" of participants. Then you pick up a textbook and it talks about collecting data from multiple samples and then from there calculating the standard error. I'm just wondering what this really means for the standard error if we only have one sample of data, which seems to be the standard practice in most psychological research.

  8. #7
    Points: 6,387, Level: 52
    Level completed: 19%, Points required for next Level: 163
    Junes's Avatar
    Location
    Netherlands
    Posts
    246
    Thanks
    17
    Thanked 25 Times in 20 Posts

    Re: single sample or multiple samples?

    You're welcome.

    As for the standard error and multiple samples, what's typically done is not collecting data from multiple samples, but inferring the standard error from the sample.

    The standard error is the standard deviation of the sample mean. Now if we have only one sample, we only observe 1 mean, so how would we know its standard deviation? Well, the math shows that the SE = SD/sqrt(n). So if we have our standard deviation of the sample, we can estimate the standard error (note that since we have a sample, our estimate of the population SD and therefore the SE may be slightly off).

    So, typically, the standard error is inferred, rather than calculated from data from multiple samples.

  9. #8
    Points: 382, Level: 7
    Level completed: 64%, Points required for next Level: 18

    Posts
    11
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Re: single sample or multiple samples?


    I see, it all makes more sense now Thanks again.

+ Reply to Thread

           




Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts






Advertise on Talk Stats