What test to use? Too many nominal variables?

[ktchris]

Hey to all,

I have data for 150 participants and the following variables:
- Perceived to be fastest: Which interaction mode was the fastest one? (Question with 5 options and the user selects one): Desktop, Smartphone, Tablet, VirtualReality (VR), AugmentedReality (AR)
- Actual time (in sec) for each of the five interaction modes e.g. TimeforDesktop, TimeforSmarpthone etc

It is a within-subjects design: all 150 participants answered the question and their time was measured for all 5 interaction modes. I attach an excerpt of the data.

I want to explore the following hypothesis: Is there any difference between participants perception of which interaction mode was fastest and their actual times in these interaction modes.

I have thought to recode the dataset as in the following: create a new variable that stores the fastest actual interaction time. In this way, I have two variables: PerceivedFastest, ActualFastest that are nominal (?) and can take one of the five values for the interaction mode. Could this be input for a Chi-squared test, and if yes how can I enter the data? Should I do something different to explore the abovementioned hypothesis?

Thanks in advance

 

[Karabiner]

Your question "ist there any difference between 'perceived' and 'actual' "
is answered if n=1 participant shows such a difference. That is trivial.
So I wonder what you really are searching for.

With kind regards

Karabiner

 

[ktchris]

Thanks for your answer Karabiner.

I want to see if there is a difference between what people say about what they did (perceived) and what we actually measure (observed/actual). The hypothesis above is meant to be an H0 hypothesis, so I disagree that even if n=1 participant shows such difference it is anwered. We need an inferential statistical analysis for this. The question is which one and whether I need to recode the data in a different way.

To take an other example, suppose we had two trials of performing a task with a software and we measured time to operationalize task efficiency. If n=1 participant did not show a decrease in time would you argue that people in general do not increase their efficiency while performing the same task?

Best regards

 

[Karabiner]

Generalization is not "most people do....". Generalization refers here to whether a statement
about the population from which the sample data are drawn can be rejected. For example
"H0: In the population, the difference between actual and perceived is exactely zero".
This hypothesis can be rejected as soon as 1 observation in the sample destroys the exact
zero difference. The generalization is: in the population, the difference is not exactely
0.0000 . And statistical significance has nothing to do with "a relevant proportion shows that
behaviour". It is just a statement whether p(H0|Data) is below a certain threshhold.

If you are interested to reject some other point, or better composite H0, you can
test something like "a considerable proporion of the population, at least 20%, shows
a difference between perceived and actual" by using a H0: "the proportion of differences
is <= 20% in the population".

Another option would be to calculate the percentage of participants with a difference
between perceived and actual, and construct a 95% confidence interval. Those intervals
are rountinely mis-interpreted (as containing the "true" proportion with 95% probability),
but apart from that, they give a pretty impression of how variable resuls can be.

With kind regards

Karabiner

 

[ktchris]

I have some trouble following the "this hypothesis can be rejected as soon as 1 observation in the sample destroys the exact zero difference.". In the example I provided, wouldn't you have trial as independent variable (two levels), task time as dependent and use a dependent t-test (assuming assumptions are met) to test for the "H0: There is no effect of experience on task time" or more technically "H0: In the population, the difference between the means of the task times across two trials is zero"? If one participant did not have exactly the same times across the two trials could we claim that the hypothesis can be rejected?

Maybe the problem here is that I have not clarified the indepedent and dependent variables. It seems that both variables (perceived, actual) are dependent and I do not have any independent? I wanted to have the interaction mode (5 levels) as independent and for each such mode to see if users of a specific software application (the population) have significant differences between perceived and actual.

Regarding the percentage you say, how can I specify the threshold for such a comparison (i.e., why 20% and not 10%?). For that hypothesis, what test should I take, a Chi-squared test? Any idea how I would enter the input for such an analysis in SPSS?

Best Regards