Analysing a designed experiment - is the theory wrong?

rogojel

TS Contributor
#1
hi,
just had a scary thought . in all industrial DoEs we definitely have measurement error on the IVs. However I have always seen the OLS as the proposed method to analyse them. Is this an error? Should we actually use a Deming regression or some more sophisticated method?

If the variance of the X measurements is less then the variance of the Y measurement this might be ok, but I have never seen this comparison be actually performed or even proposed.

regards
 

hlsmith

Omega Contributor
#2
How do you know the measurements are off? Is there a validation variable collected. If you know the bias, there may be a sensitivity analysis to incorporate.

What is this variance ratio you mentioned?
 

rogojel

TS Contributor
#3
Hi,
they are generally not off pe se but have their own measurement variance so I habe no way to know whether the level thatbwas set is the level actually achieved. This should be incorporated in the analysis - actually I can not see an application of multiple fegression where this would not be the case.
 

hlsmith

Omega Contributor
#4
Well issues are typically selection bias, measurement error, confounding, and model misclassification. Your concerns are that there may be bias? I can remember what Deming reg entails. Would it help.

I think you need to think about if there is systematic error such as a pattern in the mismeasure or if it was random.
 

Jake

Cookie Scientist
#5
If all you want to know about is the relationship between the X measurements and Y, then there is no need to correct for measurement error. We need only to assume that future X measurements will contain about as much error as the current X measurements. But if your interest is in the relationship between Y and the underlying X "true scores"--that is, the underlying thing supposedly being measured--then correcting for measurement error would be appropriate. In science we are more often interested in the latter, but in industrial experiments you may well only care about the former.
 

spunky

Smelly poop man with doo doo pants.
#6
If all you want to know about is the relationship between the X measurements and Y, then there is no need to correct for measurement error. We need only to assume that future X measurements will contain about as much error as the current X measurements. But if your interest is in the relationship between Y and the underlying X "true scores"--that is, the underlying thing supposedly being measured--then correcting for measurement error would be appropriate. In science we are more often interested in the latter, but in industrial experiments you may well only care about the former.
Taking on the psychometric perspective of measurement error, I see...

...

...

well, then I guess I can't really add anything else, LOL.

But I am curious, though... is there any reason as far as why you would make a distinction between, say the process of measurement and (statistical) analysis in science VS in industry? I mean, as a business owner I could see myself interested in making sure I can measure things (sales, amount of product distributed, etc.) as accurately as I can.
 

hlsmith

Omega Contributor
#7
So you are not able to measure with the exactitude fully desired. I guess it comes down to your purpose and if there is any concern toward systematic error. What are your thoughts?
 

rogojel

TS Contributor
#8
If all you want to know about is the relationship between the X measurements and Y, then there is no need to correct for measurement error. We need only to assume that future X measurements will contain about as much error as the current X measurements.
Hi,
I could imagine several scenarios here : e.g. I could measure X precisely at a given moment but I can not keep it constant during the experiment - imagine it is a reaction temperature that is regulated by a thermostat for instance. I set the value and the "real "temperature will go up and down according to the algorithm the thermostat follows. So my original X is just a rough approximation of what I actually do - but I would consider it a fixed given value in the OLS regression.

The other alternative would be a measurement that really has a lot of variation - like measuring viscosity for instance. My "real" input could be anywhere in a wide range of possible values due to measurement variation (it is not a bias BTW) I just do not know the real value. Again, the traditional DoE analysis with OLS would just assume the X has the value we measured.

Of course, in both cases we would be interested in the relationship between the real X and the real Y, the measured /set values being more or less random.
 

rogojel

TS Contributor
#9
But I am curious, though... is there any reason as far as why you would make a distinction between, say the process of measurement and (statistical) analysis in science VS in industry? I mean, as a business owner I could see myself interested in making sure I can measure things (sales, amount of product distributed, etc.) as accurately as I can.
:) It is a bit more complex then that but that would be a loong story. If you report the measurement to a customer for instance, then maybe not. If the accurate measurement costs a lot more and you can not control the process as accurately then again maybe not ...
 

hlsmith

Omega Contributor
#11
I could see real world temperatures always being variable given where it was taken and ambient characteristics. It isn't measurement error given the variable being recorded, but I still think it falls under bias, because it is not a recording of the true measure of the underlying phenomena. We just don't like hearing the word bias and our minds cringe and associate it with systematic errors. But if I want to know the temp on something given an experiment (equation) and I can't not control for everything my errors result in a bias. Think about this under the guise of the bias/variance tradeoff.


I digress. The big thing is that I believe this always happens in experiments, but most move forward with standard approaches and just document these limitation in their write-up. I am off to check out GG's link.
 

Miner

TS Contributor
#12
rogojel,

You work in manufacturing, correct? What follows, is correct for industrial statistics. I'm not certain whether it can generalize to other branches of statistics.

In industrial statistics, DOEs are used for 4 purposes:
  1. Screening: to identify the few significant factors that affect a process
  2. Modeling: to create a mathematical model describing the relationship between process factors and outputs
  3. Optimizing: to use the model to minimize/maximize/target the process output
  4. Robust optimizing (desensitizing): to make the process output less sensitive to variation in the factors
In both Screening and Robust Optimizing, you are typically looking for moderate to large effects and the errors in measurements (EIM) issue is too small to worry about. If they were large enough to impact the results and decisions made, you have other problems to worry about and the effects are too small to be of practical value.

For Modeling and Optimizing, the EIM issue might cause a practical problem if your model needs to be very precise. In my personal experience, this issue was never a problem in the manufacturing environment. However, I have seen several product design applications where this could have caused some practical issues. These were situations where the model needed to be very precise and was used by the product's control software.