alternative/opposite to piecemeal statistical approach?

#1
For not normally distributed data, I used Kruskal-Wallis test to investigate the statistical significance between different variables. I performed an exercise with three different intensities (different weights w1, w2, w3), then with one weight at three different speeds (s1, s2, s3). The readings were observed from 3 different points(p1, p2, p3).
I opted for statistical significance among p1, p2, p3 at (w1 and w2 and w3) and at (s1 and s2 and s3). then i opted statistical significance among w1, w2, w3 for p1 and p2 and p3. then i opted statistical significance among s1, s2, s3 for p1 and p2 and p3. So, there are 36 independent Kruskal-Wallis tests.
After that, Mann-Whitney test was performed for pairs (post hoc analysis).
I got comment on the test that, "the piecemeal statistical approach, consisting of a very large number of comparisons made between dependent variables during different conditions, without corrections for multiple testing, renders it probable that “significant” results may well be due to chance."
Can someone please suggest where I am wrong, and how and where to adjust the p value?
 

Karabiner

TS Contributor
#2
How large is your sample size?

Is this really a complete between-subjects design (e.g. the three different weights were given to three distinct groups of people, are p1 p2 p3 three distinct groups)? What does "The readings were observed from 3 different points(p1, p2, p3)" mean?

I do not completely understand what you mean by "I opted for statistical significance among p1, p2, p3 at (w1 and w2 and w3" What precisely did you do there - did you perform three Kruskal-Wallis Tests to compare groups p1 / p2 / p3, within the w1 condition, and then you compared groups p1 / p2 / p3 within the w2 condition etc.?

With kind regards

Karabiner
 
#3
@Karabiner Thanks for the attention.
sample size was 25 (data was collected from 25 subjects).

Is this really a complete between-subjects design (e.g. the three different weights were given to three distinct groups of people, are p1 p2 p3 three distinct groups)?
There is no group of people. each person perform exercise with three different weights, (30%, 50% and 70% of their maximum handling weight).
What does "The readings were observed from 3 different points(p1, p2, p3)" mean?
It mean data was collected from three different muscles namely P1, P2 and P3. I put it that way so that a person from only statistics background may understand.

I do not completely understand what you mean by "I opted for statistical significance among p1, p2, p3 at (w1 and w2 and w3" What precisely did you do there - did you perform three Kruskal-Wallis Tests to compare groups p1 / p2 / p3, within the w1 condition, and then you compared groups p1 / p2 / p3 within the w2 condition etc.?
Yes, u understand it correctly. I performed Kruskal-Wallis Test among P1/ P2/P3 for w1, then P1/ P2/P3 for w2, and then P1/ P2/P3 for w3. Then P1/ P2/P3 for s1, then P1/ P2/P3 for s2 and then P1/ P2/P3 for s3. Then w1/w2/w3 for P1, then w1/w2/w3 for P2 and then w1/w2/w3 for P3 etc etc.

I hope now the question is a bit more clear.
 

Karabiner

TS Contributor
#4
You cannot perfom Kruskal-Wallis tests or U-tests with your data.
These tests deal with the comparison of independent groups, not
with the comparison of measurements taken from the same group
under different conditions.

If I understand your design correctely, then you have n subjects who
were measured 27 times (3 weights x 3 speeds x 3 muscles)? This
could be analysed using a repeated measures analysis of variance
with three within-subject factors.

With kind regards

Karabiner
 
#5
@Karabiner Thanks for the response.
then you have n subjects who
were measured 27 times (3 weights x 3 speeds x 3 muscles)?
The test was performed 6 times (3 different speeds and 3 different intensities) only. For each exercise, the data was collected from three different muscles at the very same time. I believe that the three muscles are independent (this is what i want to prove). Repeated ANOVA is performed for data from same source at different times.
Repeated ANOVA could be used for compaing w1, w2, w3 for P1 etc etc, but i'd seen people using Kruskal-Wallis test for this purpose.

Can you please clarify should i use Kruskal-Wallis test for comparisons among P1, P2, P3 at different conditions and repeated ANOVA for comparison among P1 collected at w1,w2,w3 etc etc?
 

Karabiner

TS Contributor
#6
The test was performed 6 times (3 different speeds and 3 different intensities) only. For each exercise, the data was collected from three different muscles at the very same time. I believe that the three muscles are independent (this is what i want to prove).
I suppose that the same dependent variable was measured at each muscle?
What exactely is your research question, what does it mean that you want to prove
that the 3 muscles are independent? Do you assume e.g. that the effect of
speed and or of intisity can only be seen in one or two of these muscles?

Repeated ANOVA is performed for data from same source at different times.
Repeated measures ANOVA is performed if the same measurement is taken from the same
subjects several times, e.g. if eyesight is measured from a person's left eye as well as from his
right eye.

Repeated ANOVA could be used for compaing w1, w2, w3 for P1 etc etc, but i'd seen people using Kruskal-Wallis test for this purpose.
I am afraid that I do not quite understand what you mean.
If the 3 measurements (i.e. at 3 muscles) are taken from
the same persons, then it is a within-subject design and
you cannot use Kruskal-Wallis. But again (see above), I m
not sure what the reserach question is.

With kind regards

Karabiner
 
#7
Repeated measures ANOVA is performed if the same measurement is taken from the same
subjects several times, e.g. if eyesight is measured from a person's left eye as well as from his
right eye.
But what if eye-sight is measured several times using different lenses? Will we still employ repeated ANOVA in that case as well?
let me give u an example that might clear some questions. you take three points/regions (p1, p2, p3) on tongue. you hypothesized that at different points, the taste buds are different. then u put different things on each point simultaneously and ask the subject whether he feels any difference in taste for each point. suppose u tested with 3 different sweet (s1, s2, s3) items and three different bitter (w1, w2, w3) items. Lets assume, u get the results in the form of numbers.
I want to compare p1, p2, p3 for s1 and s2 and s3. (three independent tests, p1, p2, p3 with s1, then p1, p2, p3 with s2 and so and so forth), the same with w1, w2, w3 (these tests were performed among P1, p2, p3 for different tastes). After that, I want to check whether the three sweet things are statistically different from each other. For that, i compared s1, s2, s3 for p1. then s1, s2, s3 for p2 and so and so forth. similar for w1, w2, w3.
Now, which test (ANOVA/repeated ANOVA, Kruskal-Wallis or any other) to be performed for which test.

Hope this analogy helps u in understanding the problem.
 

Karabiner

TS Contributor
#8
But what if eye-sight is measured several times using different lenses? Will we still employ repeated ANOVA in that case as well?
The probability of any left eye entering the study is dependent on a
specific right eye (the one from the very same subject) being in the
study, therefore these are not independent observations and a repeated
measures approach is appropriate.

Now, which test (ANOVA/repeated ANOVA, Kruskal-Wallis or any other) to be performed for which test.
You have only one group of subjects, all of them were measured at all locations included,
and under all conditions included. This is completey a repeated-measures design. Nowhere
can I see any justification for using an analysis for independent groups, such as K-W.

With kind regards

Karabiner
 
#9
Nowhere
can I see any justification for using an analysis for independent groups, such as K-W.
Thanks Sir.
I am not arguing but trying to understand.
Now I got the basic idea that i need to do repeated measure of ANOVA.
As you can see from my discussion, i know only basic level of statistics. i tried to go through some books but its not easy to grab the concept (as this is not my field).
I am observing 4 different values. (in the taste analogy,
Lets assume, u get the results in the form of numbers.
these "values" are actually 4 different variables. It means for each experiment and each point, i will get 4 values. these 4 variables are independent of one another.
Will you please tell which test should i perform, in which order and will it be three-way ANOVA or two-way?

Thanking you.
 

Karabiner

TS Contributor
#10
these "values" are actually 4 different variables. It means for each experiment and each point, i will get 4 values. these 4 variables are independent of one another.
Please describe these variables and their measurements in detail.
I have neither an idea whether these are categorical, ordinal or interval
scaled variables, nor what they represent.

With kind regards

Karabiner
 
#11
Please describe these variables and their measurements in detail.
The four variables are, 1. normalized amplitude (it represents the intensity of sweetness in our example, e.g. more sweet mean higher amplitude)
2. median frequency, 3. mean frequency, 4, instantaneous frequency. The frequency is extracted from recorded signal from each point. There is no direct interpretation of frequency (definitely we can explain that from some other prospect but it is irrelevant in stat analysis).
nor what they represent.
They simply represents the characteristics of the signals.
and the variables are scaled.
 

Karabiner

TS Contributor
#12
As far as I can see, you can do four repeated-measures ANOVAs.
Or, if the dependent variables jointly represent a hypothetical
construct, a repeated-measures ANOVA with all DVs and type
of DV as repeated-measures factor. Or, a Mixed Model, which
requires a bit more experience, though.

With kind regards

Karabiner
 
#13
Thanks, I will do four repeated measures of ANOVA (as recommended by you).
Next thing, what about between P1, P2, P3 at s1.
As discussed above, P1, P2, P3 were collected simultaneously. It seems it should be K-W. Please correct me if I'm wrong even at this thing :p
 

Karabiner

TS Contributor
#14
Sorry, I do not quite understand. We already discussed that measurements
which were taken from the same subject represent dependent observations
and cannot be compared by using tests for independent groups. Since you
insist on using K-W H test here, I am inclined to assume that I misunderstood
your design and/or that I am missing an important aspect of your data.
I suppose that repeating the same statements once again would be meaningless.

Good luck for your research endeavours!

Karabiner