Bland-Altman plot interpretation

#1
Good afternoon,

I am trying to compare humidity levels taken with our old hygrometers vs. our new one.

I did some research over the internet, as well as in this forum and found that the Bland-Altman plot would be the best choice to compare measurements (level of agreements) between two instruments and see if the new one performs as good as the old one.

However one I plot my data (at least 1000 measurements / hygrometers) it give me this "weird" plot. While most of the data points are within the agreements limits, the shape / curve bothers me.

I tried looking around to see if I could find an explination but no luck.

If someone could please help me, or at least reassure me that it is normal (or not) it would be great.

Thank you

Alexia
 
#2
I'm sorry again, but I forgot to mention that I also did run a paired t test between the two sets of data and they show a statistically significant different, which adds more questioning as weather my plots makes sense or not (in other words, car my two hygrometers gives me two different measurements, but have an agreements between the two?)

Thank you again

Alexia
 

Miner

TS Contributor
#5
Presuming that you collected your data in time order sequence, there appears to be strong autocorrelation between successive measurements out to 75 lags. I extracted every 76th measurement, reran the analysis and attached the results.

At low measurements, the two devices are in close agreement, but at values greater than 0.1 the bias increases then levels out.
 

rogojel

TS Contributor
#6
hi,
if you do a simple scatterplot old vs. new values I think it will show a very irregular behaviour for the new measurement . e.g. for old values in the range of 0.6 to 0.7 or 1 to 1.6 you can get two or more, wildly different, new measurement values. I would say that the two systems are not comparable at all.

regards
 

Miner

TS Contributor
#7
Again, I am assuming that the data are in time series order when creating this plot. It appears that the two instruments track each other at a macro level, but the new instrument might be less sensitive to micro level changes.
 

rogojel

TS Contributor
#8
hi,
between measurements 400 and 800 roughly the old values jump across a range of 0.4 whereas the new ones are pretty much constant. So the old instrument must have picked up something else then the new one (or was simply defecive)?
 
#9
Hi Miner,

First thank you for your reply, and indeed the data collected are in a time order sequence. What I forgot to mention is that the "exact" value is not important but the "trend" or relative value (if it goes up or down and if so did it go up by how much).

I do seem to understand what you did, but just to make sure I understand you say that if I take any successive measurement up to 75, meaning for example the 70th, the 114th, ... there is a strong autocorrelation between the two, correct?

Also, am I allowed to do this, or normally I have to take all the data.

Thank you again

Alexia
 

Miner

TS Contributor
#10
That is correct. The value of the 70th measurement is dependent on the first measurement. Most statistical methods assume independence of the measurements, so by taking sample 1, 76, etc. you have independent measurements.
 
#11
Hi Rogojel,

Thank you for replying.

While the two measurement were taking at different times, and most importantly under different conditions, it is a bit normal that the exact value are different. However, the important point is that both go in the same direction throughout the time.

Knowing that do you think it is still adequate to do the Altman plot, and if so while the measurement are exactly the same can we still conclude there is a "good" agreement or not.

Thank you

Alexia
 

rogojel

TS Contributor
#13
Hi Rogojel,

Thank you for replying.

While the two measurement were taking at different times, and most importantly under different conditions, it is a bit normal that the exact value are different. However, the important point is that both go in the same direction throughout the time.

Knowing that do you think it is still adequate to do the Altman plot, and if so while the measurement are exactly the same can we still conclude there is a "good" agreement or not.

Thank you

Alexia
Hi,
I think, the problem is, that they definitely do not go in the same direction, e.g. at roughly the points between 700 and 750 the old values drop by a lot, the new ones grow and then drop a little. If the directions were roughly the same you would expect a more or less linear cloud in the scatterplot old vs. new - e.g. high old values linked to high new values etc. This is definitely not the case here .

regards
 

Miner

TS Contributor
#14
I thought there appeared to be a delayed response from the new instrument, so I ran a cross correlation with the extracted data set. There is a delayed response between the instruments at roughly lag 76 of the original data set.
 
#15
Thank you all for your response.

I went on and followed your steps and it seems going well. However, I do have another question. I heard on a youtube video that the Bland-Altman as well as a Linear regression has "no points (of doing)" if the p value is less than 0.05 (which means there is a statistically significant difference between the two groups) is it true?

I looked around and saw some people that still do it even if the p-value is less than 0.05.

If I understand my two set of measurements are statistically different but do correlate and agree closely. Is that correct to say that?.

Thank you again Miner and Rorojel I really appreciate it
 

Miner

TS Contributor
#16
I don't understand your first comment. Do you mean "There is no point in doing" these tests if there is a statistical difference between them?

Regarding your second comment, the measurements are statistically different, they do correlate, but whether they agree closely is a practical decision you must make.
 
#17
Hi Miner,

Sorry for the late reply.

For the first question yes that is correct (sorry if I wasn't clear) but my question was ´is there a point or is it acceptable (are we allowed) to do these test if there is a statistical difference between them?

Thank you
 

rogojel

TS Contributor
#18
hi,
in many cases people do the Bland-Altmann test to prove that there is no difference between the old and the new measurement method. But, of course, if there is a significant difference berween the methods, we would want to know.

regarda