Which statistical test to use to compare 3 measure instruments?

[lycanthropy]

Hello everybody!

I have to compare two fitnesstrackers: W1 and W2 and select the most accurate one, the most performant one in measuring HeartRate data.

For that, I measured for each of my 10 participants their heartrate using: an Electrocardiogramm (ECG), W1 and W2.
Now for each of my participants, I have 4 same size column data (the time of the data, the ECG data, W1 and W2 data).

The ECG data are for me the reference: those data are the 'true' ones. I want to compare W1 data and W2 data with my ECG data and select the one that is the most accurate in measuring Heartrate.

What I did:
for each of my participant, I have more than a 100 points
- I did a nomality test to each of my intsrument and they all follow a normal distribution.
- I calculate the RMSE, the variance and the mean between ECG and W1/W2. But I don't know what or how to conclude.

I tried actually a lot of statistical test, and I don't know what to conclude or which one I should choose:
- should I use linear regression between ECG and W1/2? I think linear regression is more a prediction statistical test, and i don't think I should use it. I ploted the scatter plot and it doesn't look very linear.
- A student test (or t-test?). It's a comparaison test and I also don't know how to conclude with my result: I have 2 independant variables (ECG and W1 or ECG and W2) that follow a normal distribution with not equal variances.
- Should I use some kind of equilavence test? because I want to see which one (W1 or W2) is more 'equivalent' to ECG? but which one?
- I saw some ANOVA method but I don't know if I should use it and how...

I'm a bit lost...

 

[Miner]

In the case where you are comparing two measurement devices, the typical approach is to use a Bland-Altman plot. With three measurement devices, you may want to use a Youden plot. Another possible approach is to calculate and compare the differences (i.e., W1-ECG vs. W2-ECG) using a 2-sample t-test and a test for equal variances.

 

[Alex MacMillan]

A repeated measures ANOVA could be good to look up - bear in mind you have 'clustered' data here. Your data is clustered within patients.

 

[lycanthropy]

Hello everybody! Thank you for your answers.

I tried to use a Bland-Altman plot, after reading more about this method, it does seems to be what i'm searching for.
So I'm using SPSS and to plot the Bland Altman plot, I created new varibales : the difference ( ECG-W1) and the mean ((ECG+W1)/2).

I then did a t-test on the difference and SPSS told me that the p-values is significant. I read that you shouldn't do a Bland Altman plot if the difference is significant and I don't understand why ?

I still wanted to see how it looks like so I ploted difference with mean, and I have this graph:
I can see those parallel lines. Does it mean the repartition of the points are not uniforme? Or it means there is some kind of proportionality? or systematic errors? I don't know how to interpret this graph. I also read that I could do some log transformation to rectify uniformity, so I did it and I obtained the same graph. Thank you very much for your help!

 

[Miner]

The Bland-Altman plot may be considered unnecessary if the t-test is significant, but the Bland-Altman plot can help you understand WHY the difference is significant. You plot is a great example. The t-test did not give you any idea that the lines existed, but the plot does. Read this about Ellis Ott and Plot the data!

The plots on a Bland-Altman plot are typically randomly distributed. This pattern indicates that there is something systemic influencing the results. I would evaluate what each line has in common. For example do they correspond to your participants? Once you determine that correspondence, we can figure out the reason for the slope.

 

[Miner]

If you can share your data, I could help you. I have a lot of experience with measurement systems.

 

[lycanthropy]

I attached a text file where you can see my data:
there are 3 columns: timestamp, ECG_data and W1_data. Those are the data for one participant only. I think that I should do an analysis individually for each participant. Then there are two columns: Difference (=ECG-W1) and Mean.


First, I just ploted the representation of the heartrate for each instrument in function of time in excel:


I also did a normality test on ECG_data and W1_data, and I concluded that they follow a normal distribution and I ploted a scatter plot: just W1 in function of ECG, and I could already see a repeated pattern:

Then I calculated the Difference and Mean and plotted the Bland Altman plot:



Thank you for the link to: Ellis Ott and Plot the data, i'm going to read it and a thousand thanks for helping me !

 

[Miner]

I just had enough time to verify that the diagonal lines are caused by the lower resolution of the W1 data. W1 is only precise to the whole number level, so this shows up as the diagonal lines. You can verify this by selecting one of the lines at a time and looking at the W1 values in that line.

When I finish teaching today's class, I will do some further analysis.

 

[Miner]

I ran the Bland-Altman plot and noticed that the W1 had problems with the higher ranges, so I also ran a gage linearity and bias study (attached) that more clearly shows the problem of negative bias increasing with higher heart rates. I would look into the algorithm used for each. For example: is the ECG measuring peak values while W1 is averaging?